Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752892AbbDBIs4 (ORCPT ); Thu, 2 Apr 2015 04:48:56 -0400 Received: from foss.arm.com ([217.140.101.70]:39732 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752315AbbDBIsu (ORCPT ); Thu, 2 Apr 2015 04:48:50 -0400 Date: Thu, 2 Apr 2015 09:49:32 +0100 From: Morten Rasmussen To: Jason Low Cc: Preeti U Murthy , "peterz@infradead.org" , "mingo@kernel.org" , Daniel Lezcano , "riel@redhat.com" , "vincent.guittot@linaro.org" , "srikar@linux.vnet.ibm.com" , "pjt@google.com" , "benh@kernel.crashing.org" , "efault@gmx.de" , "linux-kernel@vger.kernel.org" , "iamjoonsoo.kim@lge.com" , "svaidy@linux.vnet.ibm.com" , "tim.c.chen@linux.intel.com" Subject: Re: sched: Improve load balancing in the presence of idle CPUs Message-ID: <20150402084932.GY18994@e105550-lin.cambridge.arm.com> References: <1427741729.5694.24.camel@j-VirtualBox> <551A5CCE.70008@linux.vnet.ibm.com> <1427828056.2492.24.camel@j-VirtualBox> <551B9514.80701@linux.vnet.ibm.com> <20150401170418.GX18994@e105550-lin.cambridge.arm.com> <1427945434.2556.26.camel@j-VirtualBox> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1427945434.2556.26.camel@j-VirtualBox> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2829 Lines: 60 On Thu, Apr 02, 2015 at 04:30:34AM +0100, Jason Low wrote: > On Wed, 2015-04-01 at 18:04 +0100, Morten Rasmussen wrote: > > On Wed, Apr 01, 2015 at 07:49:56AM +0100, Preeti U Murthy wrote: > > > > > > On 04/01/2015 12:24 AM, Jason Low wrote: > > > > On Tue, 2015-03-31 at 14:07 +0530, Preeti U Murthy wrote: > > > >> Hi Jason, > > > >> > > > >> On 03/31/2015 12:25 AM, Jason Low wrote: > > > >>> Hi Preeti, > > > >>> > > > >>> I noticed that another commit 4a725627f21d converted the check in > > > >>> nohz_kick_needed() from idle_cpu() to rq->idle_balance, causing a > > > >>> potentially outdated value to be used if this cpu is able to pull tasks > > > >>> using rebalance_domains(), and nohz_kick_needed() directly returning > > > >>> false. > > > >> > > > >> I see that rebalance_domains() will be run at the end of the scheduler > > > >> tick interrupt handling. trigger_load_balance() only sets the softirq, > > > >> it does not call rebalance_domains() immediately. So the call graph > > > >> would be: > > > > > > > > Oh right, since that only sets the softirq, this wouldn't be the issue, > > > > though we would need these changes if we were to incorporate any sort of > > > > nohz_kick_needed() logic into the nohz_idle_balance() code path correct? > > > > > > I am sorry I don't quite get this. Can you please elaborate? > > > > I think the scenario is that we are in nohz_idle_balance() and decide to > > bail out because we have pulled some tasks, but before leaving > > nohz_idle_balance() we want to check if more balancing is necessary > > using nohz_kick_needed() and potentially kick somebody to continue. > > > Note that the balance cpu is currently skipped in nohz_idle_balance(), > > but if it wasn't the scenario would be possible. > > This scenario would also be possible if we call rebalance_domains() > first again. Yes. > I'm wondering if adding the nohz_kick_needed(), ect... in > nohz_idle_balance() can address the 10 second latency issue while still > calling rebalance_domains() first, since it seems more ideal to try > balancing on the current awake CPU first, as you also have mentioned I believe it could. That is where I was going with the chain-of-kicks idea. I think the main cause of the unacceptable you are observing is due to nohz_kicks only being issued at the tick. So if the balancer pulls for itself first and bails out the next kick won't be issued until the next tick or even multiple ticks later depending on nohz.next_balance. I haven't figured out if there is a reason for delaying the next nohz_idle_balance() though. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/