Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752015AbbGMVpn (ORCPT ); Mon, 13 Jul 2015 17:45:43 -0400 Received: from mail-la0-f54.google.com ([209.85.215.54]:34552 "EHLO mail-la0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751115AbbGMVpk (ORCPT ); Mon, 13 Jul 2015 17:45:40 -0400 MIME-Version: 1.0 In-Reply-To: <55A4271B.9040506@ezchip.com> References: <1436817481-8732-1-git-send-email-cmetcalf@ezchip.com> <1436817481-8732-2-git-send-email-cmetcalf@ezchip.com> <55A4271B.9040506@ezchip.com> From: Andy Lutomirski Date: Mon, 13 Jul 2015 14:45:19 -0700 Message-ID: Subject: Re: [PATCH v4 1/5] nohz_full: add support for "cpu_isolated" mode To: Chris Metcalf Cc: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Christoph Lameter , Viresh Kumar , "linux-doc@vger.kernel.org" , Linux API , "linux-kernel@vger.kernel.org" Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3477 Lines: 77 On Mon, Jul 13, 2015 at 2:01 PM, Chris Metcalf wrote: > On 07/13/2015 04:40 PM, Andy Lutomirski wrote: >> >> On Mon, Jul 13, 2015 at 12:57 PM, Chris Metcalf >> wrote: >>> >>> The existing nohz_full mode makes tradeoffs to minimize userspace >>> interruptions while still attempting to avoid overheads in the >>> kernel entry/exit path, to provide 100% kernel semantics, etc. >>> >>> However, some applications require a stronger commitment from the >>> kernel to avoid interruptions, in particular userspace device >>> driver style applications, such as high-speed networking code. >>> >>> This change introduces a framework to allow applications to elect >>> to have the stronger semantics as needed, specifying >>> prctl(PR_SET_CPU_ISOLATED, PR_CPU_ISOLATED_ENABLE) to do so. >>> Subsequent commits will add additional flags and additional >>> semantics. >> >> I thought the general consensus was that this should be the default >> behavior and that any associated bugs should be fixed. > > > I think it comes down to dividing the set of use cases in two: > > - "Regular" nohz_full, as used to improve performance and limit > interruptions, possibly for power benefits, etc. But, stray > interrupts are not particularly bad, and you don't want to take > extreme measures to avoid them. > > - What I'm calling "cpu_isolated" mode where when you return to > userspace, you expect that by God, the kernel doesn't interrupt you > again, and if it does, it's a flat-out bug. > > There are a few things that cpu_isolated mode currently does to > accomplish its goals that are pretty heavy-weight: > > Processes are held in kernel space until ticks are quiesced; this is > not necessarily what every nohz_full task wants. If a task makes a > kernel call, there may well be arbitrary timer fallout, and having a > way to select whether or not you are willing to take a timer tick after > return to userspace is pretty important. Then shouldn't deferred work be done immediately in nohz_full mode regardless? What is this delayed work that's being done? > > Likewise, there are things that you may want to do on return to > userspace that are designed to prevent further interruptions in > cpu_isolated mode, even at a possible future performance cost if and > when you return to the kernel, such as flushing the per-cpu free page > list so that you won't be interrupted by an IPI to flush it later. > Why not just kick the per-cpu free page over to whatever cpu is monitoring your RCU state, etc? That should be very quick. > If you're arguing that the cpu_isolated semantic is really the only > one that makes sense for nohz_full, my sense is that it might be > surprising to many of the folks who do nohz_full work. But, I'm happy > to be wrong on this point, and maybe all the nohz_full community is > interested in making the same tradeoffs for nohz_full generally that > I've proposed in this patch series just for cpu_isolated? nohz_full is currently dog slow for no particularly good reasons. I suspect that the interrupts you're seeing are also there for no particularly good reasons as well. Let's fix them instead of adding new ABIs to work around them. --Andy -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/