Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 12 Jan 2021 16:51:06 +0100
From:   Peter Zijlstra <peterz@infradead.org>
To:     Jens Axboe <axboe@kernel.dk>
Cc:     Valentin Schneider <valentin.schneider@arm.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Lai Jiangshan <jiangshanlai@gmail.com>,
        linux-kernel@vger.kernel.org, Qian Cai <cai@redhat.com>,
        Vincent Donnefort <vincent.donnefort@arm.com>,
        Dexuan Cui <decui@microsoft.com>,
        Lai Jiangshan <laijs@linux.alibaba.com>,
        Paul McKenney <paulmck@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [PATCH -tip V3 0/8] workqueue: break affinity initiatively
Message-ID: <X/3FaoV0K/U7ZuY4@hirez.programming.kicks-ass.net>
References: <20201226025117.2770-1-jiangshanlai@gmail.com>
 <X/hGHNGB9fltElWB@hirez.programming.kicks-ass.net>
 <87o8hv7pnd.fsf@nanos.tec.linutronix.de>
 <X/wv7+PP8ywNYmIS@hirez.programming.kicks-ass.net>
 <X/yH9+MGa1JCNZ8x@hirez.programming.kicks-ass.net>
 <jhj7doj1dr1.mognet@arm.com>
 <f5b0f650-5dc1-c67d-f548-340e7ca54d41@kernel.dk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f5b0f650-5dc1-c67d-f548-340e7ca54d41@kernel.dk>
Precedence: bulk

On Tue, Jan 12, 2021 at 07:57:26AM -0700, Jens Axboe wrote:
> On 1/11/21 12:21 PM, Valentin Schneider wrote:
> > On 11/01/21 18:16, Peter Zijlstra wrote:
> >> Sadly it appears like io_uring() uses kthread_create_on_cpu() without
> >> then having any hotplug crud on, so that needs additinoal frobbing.
> >>
> > 
> > I noticed that as well sometime ago, and I believed then (still do) this
> > usage is broken. I don't think usage of kthread_create_on_cpu() outside
> > of smpboot makes sense, because without any hotplug step to park the
> > thread, its affinity can end up being reset after its dedicated CPU gets
> > offlined.
> > 
> > I'm clueless about io_uring, but if it *actually* has a good reason to
> > use some pcpu kthreads (it seems it doesn't have to be on all CPUs?),
> > then it needs to register some hotplug step to park them / do something
> > sensible on hotplug.
> 
> For io_uring, it's purely used by the SQPOLL mode, which sets aside a
> kernel thread for submissions so the application doesn't have to do
> anything but write new SQE entries to submit. The thread then notices
> these, and submits them. There's an option to affinitize that thread
> to a single CPU, which often makes sense for setups like that. Think
> of it like a strong hint.

OK, that matches what I could make of it earlier today.

> Things aren't going to break if this CPU goes away and we end up being
> affinitized to some other CPU, though it is suboptimal. So I guess we
> might need some notifiers to ensure that we reset the CPU back again
> if it's gone offline+online again? I can take a look at that.

Indeed, that would make sense.

Thanks!