Received-SPF: pass (google.com: domain of linux-kernel+bounces-166408-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Thu, 2 May 2024 07:45:35 -0400
From: Phil Auld <pauld@redhat.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Yury Norov <yury.norov@gmail.com>,
	Ankit Jain <ankit-aj.jain@broadcom.com>, linux@rasmusvillemoes.dk,
	akpm@linux-foundation.org, linux-kernel@vger.kernel.org,
	juri.lelli@redhat.com, ajay.kaher@broadcom.com,
	alexey.makhalov@broadcom.com, vasavi.sirnapalli@broadcom.com,
	Paul Turner <pjt@google.com>, Ingo Molnar <mingo@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Daniel Bristot de Oliveira <bristot@redhat.com>,
	Valentin Schneider <vschneid@redhat.com>
Subject: Re: [PATCH] lib/cpumask: Boot option to disable tasks distribution
 within cpumask
Message-ID: <20240502114535.GA86496@lorien.usersys.redhat.com>
References: <20240430090431.1619622-1-ankit-aj.jain@broadcom.com>
 <ZjE3C9UgeZR02Jyy@yury-ThinkPad>
 <20240502084349.GW30852@noisy.programming.kicks-ass.net>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240502084349.GW30852@noisy.programming.kicks-ass.net>

Hi Peter,

On Thu, May 02, 2024 at 10:43:49AM +0200 Peter Zijlstra wrote:
> On Tue, Apr 30, 2024 at 11:23:07AM -0700, Yury Norov wrote:
> > On Tue, Apr 30, 2024 at 02:34:31PM +0530, Ankit Jain wrote:
> > > commit 46a87b3851f0 ("sched/core: Distribute tasks within affinity masks")
> > > and commit 14e292f8d453 ("sched,rt: Use cpumask_any*_distribute()")
> > > introduced the logic to distribute the tasks within cpumask upon initial
> > > wakeup.
> > 
> > So let's add the authors in CC list?
> > 
> > > For Telco RAN deployments, isolcpus are a necessity to cater to
> > > the requirement of low latency applications. These isolcpus are generally
> > > tickless so that high priority SCHED_FIFO tasks can execute without any
> > > OS jitter. Since load balancing is disabled on isocpus, any task
> > > which gets placed on these CPUs can not be migrated on its own.
> > > For RT applications to execute on isolcpus, a guaranteed kubernetes pod
> > > with all isolcpus becomes the requirement and these RT applications are
> > > affine to execute on a specific isolcpu within the kubernetes pod.
> > > However, there may be some non-RT tasks which could also schedule in the
> > > same kubernetes pod without being affine to any specific CPU(inherits the
> > > pod cpuset affinity).
> > 
> > OK... It looks like adding scheduler maintainers is also a necessity to
> > cater here...
> 
> So 14e292f8d453 is very specifically only using sched_domain_span(), and
> if you're using partitioned CPUs they should not show up there.
> 
> As to 46a87b3851f0, if you're explicitly creating tasks with an affinity
> masks that spans your partition then you're getting what you ask for.

I think you are skipping some details. We've also asked for no load
balancing and this spreading is a form of load balancing. So that's
not getting what was asked for.

And the tasks being created with this affinity are not being explicitly
created thusly. It's implicit. Anything exec'd into the container
(==cgroup with cpuset set) gets the cpu affinity spanning the cpus in
the container. There are layers. It's not someone using "taskset bash"
at a command prompt.

> 
> In fact, I already explained this to you earlier, so why are you
> suggesting horrible hacks again? This behaviour toggle you suggest is
> absolutely unacceptable.
> 
> I even explained what the problem was and where to look for solutions.
> 
> https://lkml.kernel.org/r/20231011135238.GG6337@noisy.programming.kicks-ass.net
>

I was not aware of the history here. Thanks. I'll look into that. At
first blush it's not obvious how that helps but I've been wrong before...


Cheers,
Phil

--