Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933862AbcLAMks (ORCPT ); Thu, 1 Dec 2016 07:40:48 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:60041 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758599AbcLAMkl (ORCPT ); Thu, 1 Dec 2016 07:40:41 -0500 Date: Thu, 1 Dec 2016 04:40:24 -0800 From: "Paul E. McKenney" To: Peter Zijlstra Cc: Michal Hocko , Donald Buczek , Paul Menzel , dvteam@molgen.mpg.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Josh Triplett Subject: Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node` Reply-To: paulmck@linux.vnet.ibm.com References: <29196f89-c35e-f79d-8e4d-2bf73fe930df@molgen.mpg.de> <20161130110944.GD18432@dhcp22.suse.cz> <20161130115320.GO3924@linux.vnet.ibm.com> <20161130131910.GF18432@dhcp22.suse.cz> <20161130142955.GS3924@linux.vnet.ibm.com> <20161130163820.GQ3092@twins.programming.kicks-ass.net> <20161130170557.GK18432@dhcp22.suse.cz> <20161130175015.GR3092@twins.programming.kicks-ass.net> <20161130194019.GF3924@linux.vnet.ibm.com> <20161201053035.GC3092@twins.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161201053035.GC3092@twins.programming.kicks-ass.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16120112-0020-0000-0000-00000A679B59 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006173; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000193; SDB=6.00787826; UDB=6.00381135; IPR=6.00565503; BA=6.00004935; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00013502; XFM=3.00000011; UTC=2016-12-01 12:40:25 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16120112-0021-0000-0000-000057BB4E26 Message-Id: <20161201124024.GB3924@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-12-01_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1609300000 definitions=main-1612010221 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3223 Lines: 75 On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 11:40:19AM -0800, Paul E. McKenney wrote: > > > > See commit: > > > > > > 4a81e8328d37 ("rcu: Reduce overhead of cond_resched() checks for RCU") > > > > > > Someone actually wrote down what the problem was. > > > > Don't worry, it won't happen again. ;-) > > > > OK, so the regressions were in the "open1" test of Anton Blanchard's > > "will it scale" suite, and were due to faster (and thus more) grace > > periods rather than path length. > > > > I could likely counter the grace-period speedup by regulating the rate > > at which the grace-period machinery pays attention to the rcu_qs_ctr > > per-CPU variable. Actually, this looks pretty straightforward (famous > > last words). But see patch below, which is untested and probably > > completely bogus. > > Possible I suppose. Didn't look too hard at it. > > > > > > Also, I seem to have missed, why are we going through this again? > > > > > > > > Well, the point I've brought that up is because having basically two > > > > APIs for cond_resched is more than confusing. Basically all longer in > > > > kernel loops do cond_resched() but it seems that this will not help the > > > > silence RCU lockup detector in rare cases where nothing really wants to > > > > schedule. I am really not sure whether we want to sprinkle > > > > cond_resched_rcu_qs at random places just to silence RCU detector... > > > > > > Right.. now, this is obviously all PREEMPT=n code, which therefore also > > > implies this is rcu-sched. > > > > > > Paul, now doesn't rcu-sched, when the grace-period has been long in > > > coming, try and force it? And doesn't that forcing include prodding CPUs > > > with resched_cpu() ? > > > > It does in the v4.8.4 kernel that Boris is running. It still does in my > > -rcu tree, but only after an RCU CPU stall (something about people not > > liking IPIs). I may need to do a resched_cpu() halfway to stall-warning > > time or some such. > > Sure, we all dislike IPIs, but I'm thinking this half-way point is > sensible, no point in issuing user visible annoyance if indeed we can > prod things back to life, no? > > Only if we utterly fail to make it respond should we bug the user with > our failure.. Sold! ;-) I will put together a patch later today. My intent is to hold off on the "upgrade cond_resched()" patch, one step at a time. Longer term, I do very much like the idea of having cond_resched() do both scheduling and RCU quiescent states, assuming that this avoids performance pitfalls. > > > I'm thinking not, because if it did, that would make cond_resched() > > > actually schedule, which would then call into rcu_note_context_switch() > > > which would then make RCU progress, no? > > > > Sounds plausible, but from what I can see some of the loops pointed > > out by Boris's stall-warning messages don't have cond_resched(). > > There was another workload that apparently worked better when moved from > > cond_resched() to cond_resched_rcu_qs(), but I don't know what kernel > > version was running. > > Egads.. cursed if you do, cursed if you dont eh.. Almost like this was real life! ;-) Thanx, Paul