Date: Tue, 19 Dec 2017 10:01:46 -0600 (CST)
From: Christopher Lameter <cl@linux.com>
To: Peter Zijlstra <peterz@infradead.org>
cc: Frederic Weisbecker <frederic@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Chris Metcalf <cmetcalf@mellanox.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Luiz Capitulino <lcapitulino@redhat.com>,
        "Paul E . McKenney" <paulmck@linux.vnet.ibm.com>,
        Ingo Molnar <mingo@kernel.org>, Wanpeng Li <kernellwp@gmail.com>,
        Mike Galbraith <efault@gmx.de>, Rik van Riel <riel@redhat.com>
Subject: Re: [PATCH 4/5] sched/isolation: Residual 1Hz scheduler tick
 offload
In-Reply-To: <20171219091911.tg2k4w7mgv2bcmeb@hirez.programming.kicks-ass.net>
Message-ID: <alpine.DEB.2.20.1712190957390.16727@nuc-kabylake>
References: <1513653838-31314-1-git-send-email-frederic@kernel.org> <1513653838-31314-5-git-send-email-frederic@kernel.org> <20171219091911.tg2k4w7mgv2bcmeb@hirez.programming.kicks-ass.net>
User-Agent: Alpine 2.20 (DEB 67 2015-01-07)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1422
Lines: 29

On Tue, 19 Dec 2017, Peter Zijlstra wrote:

> On Tue, Dec 19, 2017 at 04:23:57AM +0100, Frederic Weisbecker wrote:
> > When a CPU runs in full dynticks mode, a 1Hz tick remains in order to
> > keep the scheduler stats alive. However this residual tick is a burden
> > for Real-Time tasks that can't stand no interruption at all.
>
> I'm not sure that is accurate. RT doesn't necessarily have anything much
> to so with this. The tick is per definition very deterministic and thus
> should not be a problem.

Depends what one means by RT. Certainly if you want bare metal performance
then the tick is a problem. There are numerous loads sensitive to cpu
denial through the scheduler tick for a few microseconds. MPI and software
that rendevouz frequently in HPC are one example. Here at my workplace we
regular monitor these effects and would like a system that runs as clean
as possible.

> > Adding the boot parameter "isolcpus=nohz_offload" will now outsource
> > these scheduler ticks to the global workqueue so that a housekeeping CPU
> > handles that tick remotely.
>
> The global workqueue sounds horrific; surely you want at least one such
> housekeeping CPU per node or something ?

Well that determines how many processors you can free from the OS noise.
Having one workqueue per node sacrifices one core more to the operating
system. But it would increase OS performance. So maybe we can have that
configurable?