Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Wed, 30 Mar 2022 15:14:40 -0400
From:   Phil Auld <pauld@redhat.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Steven Rostedt <rostedt@goodmis.org>,
        Chengming Zhou <zhouchengming@bytedance.com>, mingo@redhat.com,
        juri.lelli@redhat.com, vincent.guittot@linaro.org,
        dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, linux-kernel@vger.kernel.org,
        duanxiongchun@bytedance.com, songmuchun@bytedance.com,
        Frederic Weisbecker <fweisbec@gmail.com>
Subject: Re: [External] Re: [PATCH] sched/fair: fix broken bandwidth control
 with nohz_full
Message-ID: <20220330191439.GC17246@pauld.bos.csb>
References: <20220328110751.39987-1-zhouchengming@bytedance.com>
 <20220328132047.GD8939@worktop.programming.kicks-ass.net>
 <c0465b27-328a-1288-fb50-d4c6321c8b4d@bytedance.com>
 <YkHRmv/OcABIB0wP@hirez.programming.kicks-ass.net>
 <6fc49cff-f8a1-8b09-5a25-a64e5d07d258@bytedance.com>
 <YkHal1m3pnxGoQ1Y@hirez.programming.kicks-ass.net>
 <20220328124454.08ab6126@gandalf.local.home>
 <20220330182327.GO8939@worktop.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220330182327.GO8939@worktop.programming.kicks-ass.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

Hi Peter,

On Wed, Mar 30, 2022 at 08:23:27PM +0200 Peter Zijlstra wrote:
> On Mon, Mar 28, 2022 at 12:44:54PM -0400, Steven Rostedt wrote:
> > On Mon, 28 Mar 2022 17:56:07 +0200
> > Peter Zijlstra <peterz@infradead.org> wrote:
> > 
> > > > echo $$ > test/cgroup.procs
> > > > taskset -c 1 bash -c "while true; do let i++; done"  --> will be throttled  
> > > 
> > > Ofcourse.. I'm arguing that bandiwdth control and NOHZ_FULL are somewhat
> > > mutually exclusive, use-case wise. So I really don't get why you'd want
> > > them both.
> > 
> > Is it?
> > 
> > One use case I can see for having both is for having a deadline task that
> > needs to get something done in a tight deadline. NOHZ_FULL means "do not
> > interrupt this task when it is the top priority task on the CPU and is
> > running in user space".
> 
> This is absolute batshit.. It means no such thing. We'll happily wake
> another task to this CPU and re-enable the tick any instant.
> 
> Worse; the use-case at hand pertains to cfs bandwidth control, which
> pretty much guarantees there *will* be an interrupt.

The problem is (at least in some cases) that container orchestration userspace
code allocates a whole CPU by setting quota == period.  Or 3 cpus as 3*period etc.

In cases where an isolated task is expected to run uninterrupted (only task in 
the system affined to that cpu, nohz_full, nocbs etc) you can end up with it
getting throttled even though it theoritically has enough bandwidth for the full
cpu and therefore should never get throttled. 

There are radio network setups where the packet processing is isolated
like this but the system as a whole is managed by container orchestration so 
everything has cfs bandwidth quotas set.

I don't think generally the bandwidth controls in these cases are used for 
CPU sharing (quota < period). I agree that doesn't make much sense with NOHZ_FULL
and won't work right. 

It's doled out as full cpu(s) in these cases.

Thats not a VM case so is likely different from the one that started this thread
but I thought I should mention it.


Cheers,
Phil

> 
> > Why is it mutually exclusive to have a deadline task that does not want to
> > be interrupted by timer interrupts?
> 
> This has absolutely nothing to do with deadline tasks, nada, noppes.
> 
> > Just because the biggest pushers of NOHZ_FULL is for those that are running
> > RT tasks completely in user space and event want to fault if it ever goes
> > into the kernel, doesn't mean that's the only use case.
> 
> Because there's costs associated with the whole thing. system entry/exit
> get far more expensive. It just doesn't make much sense to use NOHZ_FULL
> if you're not absoultely limiting system entry.
> 
> > Chengming brought up VMs. That's a case to want to control the bandwidth,
> > but also not interrupt them with timer interrupts when they are running as
> > the top priority task on a CPU.
> 
> It's CFS, there is nothing top priority about that.
> 

--