Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932309AbbEUS7p (ORCPT ); Thu, 21 May 2015 14:59:45 -0400 Received: from mail-wg0-f45.google.com ([74.125.82.45]:35783 "EHLO mail-wg0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932249AbbEUS7n (ORCPT ); Thu, 21 May 2015 14:59:43 -0400 Message-ID: <1432234778.3254.59.camel@gmail.com> Subject: Re: [PATCH 4/4] nohz: Set isolcpus when nohz_full is set From: Mike Galbraith To: Frederic Weisbecker Cc: "Paul E. McKenney" , Afzal Mohammed , Sasha Levin , Ingo Molnar , LKML , Chris Metcalf , "Rafael J . Wysocki" , Peter Zijlstra , Dave Jones , Thomas Gleixner , Oleg Nesterov , Ingo Molnar , Rik van Riel , Martin Schwidefsky Date: Thu, 21 May 2015 20:59:38 +0200 In-Reply-To: <20150521130621.GA14324@lerouge> References: <1430928266-24888-1-git-send-email-fweisbec@gmail.com> <1430928266-24888-5-git-send-email-fweisbec@gmail.com> <55579CE0.5060801@gmail.com> <1431840650.3222.78.camel@gmail.com> <20150520203809.GA2940@afzalpc> <20150520210026.GC6776@linux.vnet.ibm.com> <20150521121246.GA4723@afzalpc> <20150521125759.GL6776@linux.vnet.ibm.com> <20150521130621.GA14324@lerouge> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.12.11 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3737 Lines: 75 On Thu, 2015-05-21 at 15:06 +0200, Frederic Weisbecker wrote: > On Thu, May 21, 2015 at 05:57:59AM -0700, Paul E. McKenney wrote: > > On Thu, May 21, 2015 at 05:42:46PM +0530, Afzal Mohammed wrote: > > > Hi, > > > > > > On Wed, May 20, 2015 at 02:00:26PM -0700, Paul E. McKenney wrote: > > > > > > > > > Given that kernel initiated association to isolcpus, a user turning > > > > > > NO_HZ_FULL_ALL on had better not have much generic load to manage. If > > > > > > > > > > On a quad-core desktop system with NO_HZ_FULL_ALL, hackbench took 3x > > > > > time as compared to w/o this patch, except boot cpu every one else > > > > > jobless. Though NO_HZ_FULL_ALL (afaik) is not meant for generic load, > > > > > it was working fine, but not after this - it is now like a single core > > > > > system. > > > > > > > > I have to ask... What is your use case? What are you wanting NO_HZ_FULL > > > > to do for you? > > > > > > I was just playing NO_HZ_FULL with tip-[sched,timers]-* changes. > > > > > > Thought that shutting down ticks as much as possible would be > > > beneficial to normal loads too, though it has been mentioned to be used > > > for specialized loads. Seems like drawbacks due to it weigh against > > > normal loads, but haven't so far observed any (on a laptop with normal > > > activities) before this change. > > > > Indeed, NO_HZ_FULL is special purpose. You normally would select > > NO_HZ_FULL_ALL only on a system intended for heavy compute without > > normal-workload distractions or for some real-time systems. For mixed > > workloads, you would build with NO_HZ_FULL (but not NO_HZ_FULL_ALL) and > > use the boot parameters to select which CPUs are to be running the > > specialized portion of the workload. > > > > And you would of course need to lead enough CPUs running normally to > > handle the non-specialized portion of the workload. > > > > This sort of thing has traditionally required specialized kernels, > > so the cool thing here is that we can make Linux do it. Though, as > > you noticed, careful configuration is still required. > > > > Seem reasonable? > > That said if he saw a big performance regression after applying these patches, > then there is likely a problem in the patchset. Well it could be due to that mode > which loops on full dynticks before resuming to userspace. Indeed when that is > enabled, I expect real throughput issues on workloads doing lots of kernel <-> > userspace roundtrips. We just need to make sure this thing only works when requested. > > Anyway, I need to look at the patchset. I think it's a mistake to make any rash assumption and/or mandate that the user WILL use nohz_full CPUs immediately, or even at all for that matter. nohz_full currently is nothing but a CPU attribute, period, nothing more, nothing less. The overhead that comes with the it is the only thing that suggests that the set really really should be used exclusively for isolated compute loads, and even then, they had better be pure userspace due to the hefty border crossing overhead. Let that overhear be seriously reduced, as per the Ingo/Andy discussion, and presto, the things become a very interesting to dynamic sets. You can really really use them as you see fit, and on the fly, it doesn't cost you an arm and a leg just to turn it on. The only restriction being the static set declaration, that you have to manage until that too goes away. I see no reason to turn nohz_full into a rigid construct. -Mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/