Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751939AbbEQFbI (ORCPT ); Sun, 17 May 2015 01:31:08 -0400 Received: from mail-wi0-f196.google.com ([209.85.212.196]:34701 "EHLO mail-wi0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751857AbbEQFax (ORCPT ); Sun, 17 May 2015 01:30:53 -0400 Message-ID: <1431840650.3222.78.camel@gmail.com> Subject: Re: [PATCH 4/4] nohz: Set isolcpus when nohz_full is set From: Mike Galbraith To: Sasha Levin Cc: Frederic Weisbecker , Ingo Molnar , LKML , Chris Metcalf , "Rafael J . Wysocki" , Peter Zijlstra , Dave Jones , Thomas Gleixner , Oleg Nesterov , "Paul E . McKenney" , Ingo Molnar , Rik van Riel , Martin Schwidefsky Date: Sun, 17 May 2015 07:30:50 +0200 In-Reply-To: <55579CE0.5060801@gmail.com> References: <1430928266-24888-1-git-send-email-fweisbec@gmail.com> <1430928266-24888-5-git-send-email-fweisbec@gmail.com> <55579CE0.5060801@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.12.11 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3188 Lines: 67 On Sat, 2015-05-16 at 15:39 -0400, Sasha Levin wrote: > On 05/06/2015 12:04 PM, Frederic Weisbecker wrote: > > From: Chris Metcalf > > > > nohz_full is only useful with isolcpus also set, since otherwise the > > scheduler has to run periodically to try to determine whether to steal > > work from other cores. > > > > Accordingly, when booting with nohz_full=xxx on the command line, we > > should act as if isolcpus=xxx was also set, and set (or extend) the > > isolcpus set to include the nohz_full cpus. > > > > Acked-by: Mike Galbraith ["thumbs up!"] > > Acked-by: Rik van Riel > > Acked-by: Peter Zijlstra (Intel) > > Signed-off-by: Chris Metcalf > > Cc: Peter Zijlstra (Intel) > > Cc: Paul E. McKenney > > Cc: Rafael J. Wysocki > > Cc: Martin Schwidefsky > > Cc: Mike Galbraith > > Cc: Ingo Molnar > > Cc: Rik van Riel > > Signed-off-by: Frederic Weisbecker > > Hi folks, > > I've noticed a regression in my testing a few days ago and bisected it down to > this patch. I was seeing frequent soft lockups/RCU lockups and the load of the > testing VMs would go beyond 400-500 (on 32 VCPU guests) - note I'm booting them > with nohz_full=1-27. > > This patch sort of explains the behaviour I was seeing now: most of the cores > are no longer being used by the scheduler, and the remaining cores can't deal > with the load imposed on them which results in "lockups" which are really just > the CPUs being unable to keep up. > > I always thought that nohz_full without isolcpus meant that the cores would > be available to the scheduler, but it won't interfere if there is one task > running on them. It seems that this patch changed that behaviour. > > Did I misunderstand that? Yeah, tying nohz_full set to isolcpus set up an initial condition that you have to tear down with cpusets if you want those cpus returned to the general purpose pool. I had considered the kernel setting initial state to be an optimization, but have reconsidered. You may have misunderstood somewhat though, if you do not explicitly isolate the nohz_full set from the scheduler via either isolcpus or cpusets, there is no exclusion from load balancing, the scheduler may place additional tasks on a nohz_full cpu to resolve load imbalance. Given that kernel initiated association to isolcpus, a user turning NO_HZ_FULL_ALL on had better not have much generic load to manage. If he/she does not have CPUSETS enabled, or should Rik's patch rendering isolcpus immutable be merged, he/she would quickly discover that the generic box has been transformed into an utterly inflexible specialist incapable of performing mundane tasks ever again. -Mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/