Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230606093135.GA9077@didi-ThinkCentre-M930t-N000> <ZIJbvCcaqjzNteWs@slm.duckdns.org>
In-Reply-To: <ZIJbvCcaqjzNteWs@slm.duckdns.org>
From:   Yuanhan Zhang <zyhtheonly@gmail.com>
Date:   Fri, 9 Jun 2023 14:28:19 +0800
Message-ID: <CAEQmJ=gYe=d53HHC1xW_epmPmmddA4J28SHybwGmQzUZgxZovg@mail.gmail.com>
Subject: Re: [PATCH] workqueue: introduce queue_work_cpumask to queue work
 onto a given cpumask
To:     Tejun Heo <tj@kernel.org>
Cc:     jiangshanlai@gmail.com, linux-kernel@vger.kernel.org,
        pmladek@suse.com, zyhtheonly@yeah.net, zwp10758@gmail.com,
        tiozhang@didiglobal.com, fuyuanli@didiglobal.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

// I resend this to put it into the same thread, sorry for the confusion.

> Can you elaborate the intended use cases?

Hi Tejun,

Thanks for your reply! Please let me use myself as an example to explain th=
is.

In my scenario, I have 7 cpus on my machine (actually it is uma, so
queue_work_node
or using UNBOUND do not works for me), and for some unlucky reasons
there are always some irqs running on cpu 0 and cpu 6, since I'm using arm6=
4
with irqs tuning into FIFO threads, those threaded irqs are always running =
on
cpu 0 and 6 too (for affinity). And this would not be fixed easily in
short terms :(

So in order to help async init for better boot times for my devices,
I'd like to prevent
works from running on cpu 0 and 6. With queue_work_cpumask(), it would be s=
imply
done by:

...
cpumask_clear_cpu(0, cpumask);  // actually I use sysfs to parse my cpumask
cpumask_clear_cpu(6, cpumask);
queue_work_cpumask(cpumask, my_wq, &my_work->work);
...


> The code seems duplicated too. Could you do a little refactoring and make
> they (queue_work_cpumask() & queue_work_node()) share some code?

Hi Lai,

Thanks for your advice!

I do the refactoring in PATCH v2, there are some changes:
1. removed WARN_ONCE in previous code
  1). queue_work_node works well in UNBOUND since we have unbound_pwq_by_no=
de()
       in __queue_work() to choose the right node.
  2). queue_work_cpumask does not work in UNBOUND since list
numa_pwq_tbl is designed
       to be per numa node. I comment on this in this patch.
2. remove the previous workqueue_select_cpu_near and let queue_work_node() =
use
    queue_work_on() and queue_work_cpumask().

I test this patch with 100,000 queue_work_cpumask() &
queue_work_node() with randomly
inputs cpumask & node, it works as expected on my machines (80 cores
x86_64 & 7 cores ARM64
& 16 cores ARM64).

Please help review, thanks a lot!

Thanks,
Tio Zhang

Tejun Heo <tj@kernel.org> =E4=BA=8E2023=E5=B9=B46=E6=9C=889=E6=97=A5=E5=91=
=A8=E4=BA=94 06:52=E5=86=99=E9=81=93=EF=BC=9A
>


Tejun Heo <tj@kernel.org> =E4=BA=8E2023=E5=B9=B46=E6=9C=889=E6=97=A5=E5=91=
=A8=E4=BA=94 06:52=E5=86=99=E9=81=93=EF=BC=9A
>
> On Tue, Jun 06, 2023 at 05:31:35PM +0800, Tio Zhang wrote:
> > Introduce queue_work_cpumask to queue work on a "random" CPU onto a giv=
en
> > cpumask. It would be helpful when devices/modules want to assign works =
on
> > different cpusets but do not want to maintain extra workqueues, since t=
hey
> > have to alloc different workqueues and set different
> > workqueue_attrs->cpumask in the past times.
> >
> > For now only available for unbound workqueues, We will try to further
> > patch it.
> > And default to the first CPU that is in the intersection of the cpumask
> > given and the online cpumask.
> > The only exception is if the CPU is local in the cpuset we will just us=
e
> > the current CPU.
> >
> > The implementation and comments are referenced from
> > 'commit 8204e0c1113d ("workqueue: Provide queue_work_node to queue work
> > near a given NUMA node")'
>
> Can you elaborate the intended use cases?
>
> Thanks.
>
> --
> tejun