Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 18 May 2023 10:37:18 -0400
From:   Phil Auld <pauld@redhat.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     linux-kernel@vger.kernel.org, Juri Lelli <juri.lelli@redhat.com>,
        Ingo Molnar <mingo@redhat.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Ben Segall <bsegall@google.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Mel Gorman <mgorman@suse.de>
Subject: Re: [PATCH RESEND] sched/nohz: Add HRTICK_BW for using cfs bandwidth
 with nohz_full
Message-ID: <20230518143718.GC110197@lorien.usersys.redhat.com>
References: <20230518132038.3534728-1-pauld@redhat.com>
 <20230518134746.GB2860939@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230518134746.GB2860939@hirez.programming.kicks-ass.net>
Precedence: bulk

On Thu, May 18, 2023 at 03:47:46PM +0200 Peter Zijlstra wrote:
> On Thu, May 18, 2023 at 09:20:38AM -0400, Phil Auld wrote:
> > CFS bandwidth limits and NOHZ full don't play well together.  Tasks
> > can easily run well past their quotas before a remote tick does
> > accounting.  This leads to long, multi-period stalls before such
> > tasks can run again.  Use the hrtick mechanism to set a sched
> > tick to fire at remaining_runtime in the future if we are on
> > a nohz full cpu, if the task has quota and if we are likely to
> > disable the tick (nr_running == 1).  This allows for bandwidth
> > accounting before tasks go too far over quota.
> > 
> > A number of container workloads use a dynamic number of real
> > nohz tasks but also have other work that is limited which ends
> > up running on the "spare" nohz cpus.  This is an artifact of
> > having to specify nohz_full cpus at boot. Adding this hrtick
> > resolves the issue of long stalls on these tasks.
> > 
> > Add the sched_feat HRTICK_BW off by default to allow users to
> > enable this only when needed.
> 
> OMG; so because NOHZ_FULL configuration sucks, we add hacks on?
>

I suppose one could make that argument. The HRTICK mechanism is already
in place and used similarly for DL (and that also benefits nohz workloads).

I don't see NOHZ_FULL configuration getting better anytime soon, although
I think efforts are being made in that direction. 

This seemed to be a sane way to handle what are effectively conflicting
requirements.  Stalling a task to the point the host gets rebooted is
pretty painful.  Maybe if we could fail the tick_stop test in this
case that would work but that would keep all the ticks whereas this
tries to respect the request for nohz as much as possible. 

Thanks for taking a look :)

Cheers,
Phil
--