Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933799AbcKPRar (ORCPT ); Wed, 16 Nov 2016 12:30:47 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:41800 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932995AbcKPRam (ORCPT ); Wed, 16 Nov 2016 12:30:42 -0500 Date: Wed, 16 Nov 2016 09:30:36 -0800 From: "Paul E. McKenney" To: Paul Menzel Cc: linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, Josh Triplett , dvteam@molgen.mpg.de Subject: Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node` Reply-To: paulmck@linux.vnet.ibm.com References: <24c226a5-1a4a-173e-8b4e-5107a2baac04@molgen.mpg.de> <28a9fabb-c9fe-c865-016a-467a4d5e2a34@molgen.mpg.de> <20161108170340.GB4127@linux.vnet.ibm.com> <6c717122-e671-b086-77ed-4b3c26398564@molgen.mpg.de> <20161108183938.GD4127@linux.vnet.ibm.com> <9f87f8f0-9d0f-f78f-8dca-993b09b19a69@molgen.mpg.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <9f87f8f0-9d0f-f78f-8dca-993b09b19a69@molgen.mpg.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16111617-0016-0000-0000-0000052EBE43 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006089; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000189; SDB=6.00781544; UDB=6.00377019; IPR=6.00559056; BA=6.00004886; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00013346; XFM=3.00000011; UTC=2016-11-16 17:30:38 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16111617-0017-0000-0000-000034B17F04 Message-Id: <20161116173036.GK3612@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-11-16_06:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1609300000 definitions=main-1611160262 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4412 Lines: 101 On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > Dear Linux folks, > > > On 11/08/16 19:39, Paul E. McKenney wrote: > >On Tue, Nov 08, 2016 at 06:38:18PM +0100, Paul Menzel wrote: > >>On 11/08/16 18:03, Paul E. McKenney wrote: > >>>On Tue, Nov 08, 2016 at 01:22:28PM +0100, Paul Menzel wrote: > >> > >>>>Could you please help me shedding some light into the messages below? > >>>> > >>>>With Linux 4.4.X, these messages were not seen. When updating to > >>>>Linux 4.8.4, and Linux 4.8.6 they started to appear. In that > >>>>version, we enabled several CGROUP options. > >>>> > >>>>>$ dmesg -T > >>>>>[…] > >>>>>[Mon Nov 7 15:09:45 2016] INFO: rcu_sched detected stalls on CPUs/tasks: > >>>>>[Mon Nov 7 15:09:45 2016] 3-...: (493 ticks this GP) idle=515/140000000000000/0 softirq=5504423/5504423 fqs=13876 > >>>>>[Mon Nov 7 15:09:45 2016] (detected by 5, t=60002 jiffies, g=1363193, c=1363192, q=268508) > >>>>>[Mon Nov 7 15:09:45 2016] Task dump for CPU 3: > >>>>>[Mon Nov 7 15:09:45 2016] kswapd1 R running task 0 87 2 0x00000008 > >>>>>[Mon Nov 7 15:09:45 2016] ffffffff81aabdfd ffff8810042a5cb8 ffff88080ad34000 ffff88080ad33dc8 > >>>>>[Mon Nov 7 15:09:45 2016] ffff88080ad33d00 0000000000003501 0000000000000000 0000000000000000 > >>>>>[Mon Nov 7 15:09:45 2016] 0000000000000000 0000000000000000 0000000000022316 000000000002bc9f > >>>>>[Mon Nov 7 15:09:45 2016] Call Trace: > >>>>>[Mon Nov 7 15:09:45 2016] [] ? __schedule+0x21d/0x5b0 > >>>>>[Mon Nov 7 15:09:45 2016] [] ? shrink_node+0xbf/0x1c0 > >>>>>[Mon Nov 7 15:09:45 2016] [] ? kswapd+0x315/0x5f0 > >>>>>[Mon Nov 7 15:09:45 2016] [] ? mem_cgroup_shrink_node+0x90/0x90 > >>>>>[Mon Nov 7 15:09:45 2016] [] ? kthread+0xc4/0xe0 > >>>>>[Mon Nov 7 15:09:45 2016] [] ? ret_from_fork+0x1f/0x40 > >>>>>[Mon Nov 7 15:09:45 2016] [] ? kthread_worker_fn+0x160/0x160 > >>>> > >>>>Even after reading `stallwarn.txt` [1], I don’t know what could > >>>>cause this. All items in the backtrace seem to belong to the Linux > >>>>kernel. > >>>> > >>>>There is also nothing suspicious in the monitoring graphs during that time. > >>> > >>>If you let it be, do you get a later stall warning a few minutes later? > >>>If so, how does the stack trace compare? > >> > >>With Linux 4.8.6 this is the only occurrence since yesterday. > >> > >>With Linux 4.8.3, and 4.8.4 the following stack traces were seen. > > > >Looks to me like one or both of the loops in shrink_node() need > >an cond_resched_rcu_qs(). > > Thank you for the pointer. I haven’t had time yet to look into it. In theory, it is quite straightforward, as shown by the patch below. In practice, the MM guys might wish to call cond_resched_rcu_qs() less frequently, but I will leave that to their judgment. My guess is that the overhead of the cond_resched_rcu_qs() is way down in the noise, but I have been surprised in the past. Anyway, please give this patch a try and let me know how it goes. Thanx, Paul ------------------------------------------------------------------------ commit 1a5595eec6c034c27e1c826a93292240bfea934e Author: Paul E. McKenney Date: Wed Nov 16 09:26:28 2016 -0800 mm: Prevent shrink_node() RCU CPU stall warnings This commit adds a couple cond_resched_rcu_qs() calls in the inner loop in shrink_node() in order to prevent RCU CPU stall warnings. Reported-by: Paul Menzel Signed-off-by: Paul E. McKenney diff --git a/mm/vmscan.c b/mm/vmscan.c index 744f926af442..0d3b5f5a04ef 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2529,8 +2529,11 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) unsigned long scanned; if (mem_cgroup_low(root, memcg)) { - if (!sc->may_thrash) + if (!sc->may_thrash) { + /* Prevent CPU CPU stalls. */ + cond_resched_rcu_qs(); continue; + } mem_cgroup_events(memcg, MEMCG_LOW, 1); } @@ -2565,6 +2568,7 @@ static bool shrink_node(pg_data_t *pgdat, struct scan_control *sc) mem_cgroup_iter_break(root, memcg); break; } + cond_resched_rcu_qs(); /* Prevent CPU CPU stalls. */ } while ((memcg = mem_cgroup_iter(root, memcg, &reclaim))); /*