Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 24 Mar 2020 13:50:36 -0300
From:   Marcelo Tosatti <mtosatti@redhat.com>
To:     Chris Friesen <chris.friesen@windriver.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org,
        Christoph Lameter <cl@linux.com>,
        Jim Somerville <Jim.Somerville@windriver.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Peter Zijlstra <peterz@infradead.org>
Subject: Re: [PATCH v2] isolcpus: affine kernel threads to specified cpumask
Message-ID: <20200324165036.GC28165@fuller.cnet>
References: <20200323135414.GA28634@fuller.cnet>
 <87k13boxcn.fsf@nanos.tec.linutronix.de>
 <af285c22-2a3f-5aa6-3fdb-27fba73389bd@windriver.com>
 <87imiuq0cg.fsf@nanos.tec.linutronix.de>
 <20200324152016.GA25422@fuller.cnet>
 <f40c828e-5168-bd08-69b8-f7a54fd53f42@windriver.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f40c828e-5168-bd08-69b8-f7a54fd53f42@windriver.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Tue, Mar 24, 2020 at 09:56:26AM -0600, Chris Friesen wrote:
> I hadn't been keeping up with all the changes to the "isolcpus" boot arg.
> Given how it's been extended, I agree that it seems the logical place to
> deal with this.  Patch seems okay to me, but I've got a couple of nits in
> the message portion.
> 
> If I want to specify both no_kthreads and managed_irq it then something like
> "isolcpus=managed_irq,no_kthreads,2-16" would work?

Yes.

> On 3/24/2020 9:20 AM, Marcelo Tosatti wrote:
> > 
> > This is a kernel enhancement to configure the cpu affinity of kernel
> > threads via kernel boot option isolcpus=no_kthreads,<isolcpus_params>,<cpulist>
> 
> https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt
> says that "isolcpus" is deprecated.  Are we un-deprecating it?  Or is it
> only really deprecated for the "domain" option?

I don't think its deprecated (see the recent inclusion of managed_irq,
and the suggestion from Thomas to extend it).

Will send another patch to remove that sentence.

> > When this option is specified, the cpumask is immediately applied upon
> > thread launch. This does not affect kernel threads that specify cpu
> > and node.
> > 
> > This allows CPU isolation (that is not allowing certain threads
> > to execute on certain CPUs) without using the isolcpus=domain parameter,
> > making it possible to enable load balancing on such CPUs
> > during runtime (see
> 
> I think you're missing the rest of the sentence here.

Right.

> > Note-1: this is based off on Wind River's patch at
> > https://github.com/starlingx-staging/stx-integ/blob/master/kernel/kernel-std/centos/patches/affine-compute-kernel-threads.patch
> > 
> > Difference being that this patch is limited to modifying
> > kernel thread cpumask: Behaviour of other threads can
> > be controlled via cgroups or sched_setaffinity.
> > 
> > Note-2: MontaVista's patch was based off Christoph Lameter's patch at
> > https://lwn.net/Articles/565932/ with the only difference being
> > the kernel parameter changed from kthread to kthread_cpus.
> 
> Wind River, not MontaVista.  I know all us embedded linux folks look the
> same...

Doh^2.

> 
> > 
> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> > 
> > ---
> > 
> > v2: use isolcpus= subcommand (Thomas Gleixner)
> > 
> >   Documentation/admin-guide/kernel-parameters.txt |    8 ++++++++
> >   include/linux/cpumask.h                         |    5 +++++
> >   include/linux/sched/isolation.h                 |    1 +
> >   init/main.c                                     |    1 +
> >   kernel/cpu.c                                    |   13 +++++++++++++
> >   kernel/kthread.c                                |    4 ++--
> >   kernel/sched/isolation.c                        |    6 ++++++
> >   7 files changed, 36 insertions(+), 2 deletions(-)
> > 
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index c07815d230bc..7318e3057383 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -1959,6 +1959,14 @@
> >   			  the CPU affinity syscalls or cpuset.
> >   			  <cpu number> begins at 0 and the maximum value is
> >   			  "number of CPUs in system - 1".
> > +			  When using cpusets, use the isolcpus option no_kthreads
> > +			  to avoid creation of kernel threads on isolated CPUs.
> > +
> > +			no_kthreads
> > +			  Adjust the CPU affinity mask of unbound kernel threads to
> > +			  not contain CPUs on the isolated list. This complements
> > +			  the isolation provided by the cpusets mechanism described
> > +			  above.
> 
> It also complements the "managed_irq" option below.  In many cases I'd
> expect the same set of CPUs to be isolated from both irqs and kernel
> threads.
> 
> 
> Chris

Agree, will fix in -v3.