Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp6977216imu; Mon, 3 Dec 2018 05:58:48 -0800 (PST) X-Google-Smtp-Source: AFSGD/WSG93KuiB7csaCKt5EOMrQIVe0cI3eAM+khYVLiYLHHFQ1YbEf2GqD4CnWgdRxCjFkp/UP X-Received: by 2002:a62:1a8e:: with SMTP id a136mr16476071pfa.76.1543845528557; Mon, 03 Dec 2018 05:58:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543845528; cv=none; d=google.com; s=arc-20160816; b=jCUa+onVxUUQMnSPrCjR2fxx1U0oRkM3mLBnXLC0PlVToS1Aeib8rqApvB5ep8vpPF 0y3TT1OQuh6t8kxH3vbewIUGb6kC/RWtoRpNG4mCnOLwtVXsybj4GFYkTfzKShkK0G2C iEuWd7A5DeH2V0kcD1tSyES5jHLzLlAfrTPKwI2hPQM95Ws9NXzS05a9jxL/YCGlcf31 X+9+G76iWZWhURIXl2rK3nQBPKiViGmDz9yQadrxEyFNo0Gjn00PLUiiyOjzspWPYvVf X8XpidgikMY5D1Gu/UlmbmFexyHlYtPkB0Ao+xIKCQbqReS965F9MunVP98wSrUajLK4 arZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:user-agent:in-reply-to :content-disposition:mime-version:references:reply-to:subject:cc:to :from:date; bh=qVDAO19K5vWPkcR/lhKU5a4QDVURqRaL4kWQFU7efiM=; b=Le7HWJmY/kLXvzPA9K9wBup66+l+j0mbpHiF60o6MBPe+1/UUjNFZ40O6ATH90/jAq ySGoBAqXNjk6eNMlV2Hx+T+SRVoOf1LRNed5VqOVBBtUIiNYkJwdp1i7q1cgSIUMnNT+ tG+HWwtvFbdecmLC2E1qbNX6YpDPkSCR981m6gUQdJPw3vHux4nMrvAPWYdJaCYr+afT hM75QpPiE7MIVbja1oEDe4ohc6uPxhemAACSTm3It36xm/B03Bsc+NiCW5KgKITo31To fmp4R4XEac1EFwjoF00b+dq7nj16Gxy5Jz7+FlkDoduw1DBTkgsUsDnAok2ASUChr7+l cv7g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g17si11270706pgi.578.2018.12.03.05.58.33; Mon, 03 Dec 2018 05:58:48 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726581AbeLCN5u (ORCPT + 99 others); Mon, 3 Dec 2018 08:57:50 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:38088 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726146AbeLCN5u (ORCPT ); Mon, 3 Dec 2018 08:57:50 -0500 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wB3DsGAA091701 for ; Mon, 3 Dec 2018 08:56:42 -0500 Received: from e16.ny.us.ibm.com (e16.ny.us.ibm.com [129.33.205.206]) by mx0b-001b2d01.pphosted.com with ESMTP id 2p54vy2vva-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Dec 2018 08:56:41 -0500 Received: from localhost by e16.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Dec 2018 13:56:41 -0000 Received: from b01cxnp22033.gho.pok.ibm.com (9.57.198.23) by e16.ny.us.ibm.com (146.89.104.203) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Dec 2018 13:56:38 -0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wB3Dubq719398776 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Dec 2018 13:56:37 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 44B1EB2066; Mon, 3 Dec 2018 13:56:37 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0FCF6B2065; Mon, 3 Dec 2018 13:56:36 +0000 (GMT) Received: from paulmck-ThinkPad-W541 (unknown [9.80.206.214]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 3 Dec 2018 13:56:36 +0000 (GMT) Received: by paulmck-ThinkPad-W541 (Postfix, from userid 1000) id E178016C5DF5; Mon, 3 Dec 2018 05:56:38 -0800 (PST) Date: Mon, 3 Dec 2018 05:56:38 -0800 From: "Paul E. McKenney" To: "He, Bo" Cc: Steven Rostedt , "linux-kernel@vger.kernel.org" , "josh@joshtriplett.org" , "mathieu.desnoyers@efficios.com" , "jiangshanlai@gmail.com" , "Zhang, Jun" , "Xiao, Jin" , "Zhang, Yanmin" Subject: Re: rcu_preempt caused oom Reply-To: paulmck@linux.ibm.com References: <20181129130647.GG4170@linux.ibm.com> <20181129142712.GA16607@linux.ibm.com> <20181130144317.GQ4170@linux.ibm.com> <20181130101631.618d955f@gandalf.local.home> <20181130164920.GR4170@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 18120313-0072-0000-0000-000003D326C4 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00010165; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000270; SDB=6.01126388; UDB=6.00582089; IPR=6.00906523; MB=3.00024426; MTD=3.00000008; XFM=3.00000015; UTC=2018-12-03 13:56:40 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18120313-0073-0000-0000-00004A50D774 Message-Id: <20181203135638.GG4170@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-12-03_08:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812030131 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Dec 03, 2018 at 07:44:03AM +0000, He, Bo wrote: > Thanks, we have run the test for the whole weekend and not reproduce the issue, so we confirm the CONFIG_RCU_BOOST can fix the issue. Very good, that is encouraging. Perhaps I should think about making CONFIG_RCU_BOOST=y the default for CONFIG_PREEMPT in mainline, at least for architectures for which rt_mutexes are implemented. > We have enabled the rcupdate.rcu_cpu_stall_timeout=7 and also set panic on rcu stall and will see if we can see the panic, will keep you posed with the test results. > echo 1 > /proc/sys/kernel/panic_on_rcu_stall Looking forward to seeing what is going on! Of course, to reproduce, you will need to again build with CONFIG_RCU_BOOST=n. Thanx, Paul > -----Original Message----- > From: Paul E. McKenney > Sent: Saturday, December 1, 2018 12:49 AM > To: He, Bo > Cc: Steven Rostedt ; linux-kernel@vger.kernel.org; josh@joshtriplett.org; mathieu.desnoyers@efficios.com; jiangshanlai@gmail.com; Zhang, Jun ; Xiao, Jin ; Zhang, Yanmin > Subject: Re: rcu_preempt caused oom > > On Fri, Nov 30, 2018 at 03:18:58PM +0000, He, Bo wrote: > > Here is the kernel cmdline: > > Thank you! > > > Kernel command line: androidboot.acpio_idx=0 > > androidboot.bootloader=efiwrapper-02_03-userdebug_kernelflinger-06_03- > > userdebug androidboot.diskbus=00.0 androidboot.verifiedbootstate=green > > androidboot.bootreason=power-on androidboot.serialno=R1J56L6006a7bb > > g_ffs.iSerialNumber=R1J56L6006a7bb no_timer_check noxsaves > > reboot_panic=p,w i915.hpd_sense_invert=0x7 mem=2G nokaslr nopti > > ftrace_dump_on_oops trace_buf_size=1024K intel_iommu=off gpt > > loglevel=4 androidboot.hardware=gordon_peak > > firmware_class.path=/vendor/firmware relative_sleep_states=1 > > enforcing=0 androidboot.selinux=permissive cpu_init_udelay=10 > > androidboot.android_dt_dir=/sys/bus/platform/devices/ANDR0001:00/prope > > rties/android/ pstore.backend=ramoops memmap=0x1400000$0x50000000 > > ramoops.mem_address=0x50000000 ramoops.mem_size=0x1400000 > > ramoops.record_size=0x4000 ramoops.console_size=0x1000000 > > ramoops.ftrace_size=0x10000 ramoops.dump_oops=1 vga=current > > i915.modeset=1 drm.atomic=1 i915.nuclear_pageflip=1 > > drm.vblankoffdelay= > > And no sign of any suppression of RCU CPU stall warnings. Hmmm... > It does take more than 21 seconds to OOM? Or do things happen faster than that? If they do happen faster than that, then on approach would be to add something like this to the kernel command line: > > rcupdate.rcu_cpu_stall_timeout=7 > > This would set the stall timeout to seven seconds. Note that timeouts less than three seconds are silently interpreted as three seconds. > > Thanx, Paul > > > -----Original Message----- > > From: Steven Rostedt > > Sent: Friday, November 30, 2018 11:17 PM > > To: Paul E. McKenney > > Cc: He, Bo ; linux-kernel@vger.kernel.org; > > josh@joshtriplett.org; mathieu.desnoyers@efficios.com; > > jiangshanlai@gmail.com; Zhang, Jun ; Xiao, Jin > > ; Zhang, Yanmin > > Subject: Re: rcu_preempt caused oom > > > > On Fri, 30 Nov 2018 06:43:17 -0800 > > "Paul E. McKenney" wrote: > > > > > Could you please send me your list of kernel boot parameters? They > > > usually appear near the start of your console output. > > > > Or just: cat /proc/cmdline > > > > -- Steve > > >