Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\))
Subject: Re: [PATCH v2] nvme/tcp: Add support to set the tcp worker cpu
 affinity
From:   Li Feng <fengli@smartx.com>
In-Reply-To: <3e45f600db2049c4986fd8bb6aea69f4@AcuMS.aculab.com>
Date:   Mon, 17 Apr 2023 11:31:08 +0800
Cc:     Hannes Reinecke <hare@suse.de>, Keith Busch <kbusch@kernel.org>,
        Jens Axboe <axboe@fb.com>, Christoph Hellwig <hch@lst.de>,
        Sagi Grimberg <sagi@grimberg.me>,
        "open list:NVM EXPRESS DRIVER" <linux-nvme@lists.infradead.org>,
        linux-kernel <linux-kernel@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <A5745886-81C6-4BB3-848C-D4A7E47E720A@smartx.com>
References: <20230413062339.2454616-1-fengli@smartx.com>
 <20230413132941.2489795-1-fengli@smartx.com>
 <94d6a76c-8ad1-bda1-6336-e9f5fa3a6168@suse.de>
 <CAHckoCxcmNC++AXELmnCVZNjpHcaOQWOGcjia=NBCnOA7S7EeQ@mail.gmail.com>
 <3e45f600db2049c4986fd8bb6aea69f4@AcuMS.aculab.com>
To:     David Laight <David.Laight@ACULAB.COM>
Precedence: bulk


> 2023=E5=B9=B44=E6=9C=8816=E6=97=A5 =E4=B8=8A=E5=8D=885:06=EF=BC=8CDavid =
Laight <David.Laight@ACULAB.COM> =E5=86=99=E9=81=93=EF=BC=9A
>=20
> From: Li Feng
>> Sent: 14 April 2023 10:35
>>>=20
>>> On 4/13/23 15:29, Li Feng wrote:
>>>> The default worker affinity policy is using all online cpus, e.g. =
from 0
>>>> to N-1. However, some cpus are busy for other jobs, then the =
nvme-tcp will
>>>> have a bad performance.
>>>>=20
>>>> This patch adds a module parameter to set the cpu affinity for the =
nvme-tcp
>>>> socket worker threads.  The parameter is a comma separated list of =
CPU
>>>> numbers.  The list is parsed and the resulting cpumask is used to =
set the
>>>> affinity of the socket worker threads.  If the list is empty or the
>>>> parsing fails, the default affinity is used.
>>>>=20
> ...
>>> I am not in favour of this.
>>> NVMe-over-Fabrics has _virtual_ queues, which really have no
>>> relationship to the underlying hardware.
>>> So trying to be clever here by tacking queues to CPUs sort of works =
if
>>> you have one subsystem to talk to, but if you have several where =
each
>>> exposes a _different_ number of queues you end up with a quite
>>> suboptimal setting (ie you rely on the resulting cpu sets to =
overlap,
>>> but there is no guarantee that they do).
>>=20
>> Thanks for your comment.
>> The current io-queues/cpu map method is not optimal.
>> It is stupid, and just starts from 0 to the last CPU, which is not =
configurable.
>=20
> Module parameters suck, and passing the buck to the user
> when you can't decide how to do something isn't a good idea either.
>=20
> If the system is busy pinning threads to cpus is very hard to
> get right.
>=20
> It can be better to set the threads to run at the lowest RT
> priority - so they have priority over all 'normal' threads
> and also have a very sticky (but not fixed) cpu affinity so
> that all such threads tends to get spread out by the scheduler.
> This all works best if the number of RT threads isn't greater
> than the number of physical cpu.
>=20
> 	David
>=20
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, =
MK1 1PT, UK
> Registration No: 1397386 (Wales)

Hi David,

RT priority can=E2=80=99t solve the cross numa access issue.
If the user doesn=E2=80=99t know how to configure this affinity, just =
keep it default.

Cross numa is not a obvious issue on X86{_64}, but it=E2=80=99s a =
significant issue
on aarch64 with multiple numa nodes.

Thanks.