Received-SPF: pass (google.com: domain of linux-kernel+bounces-43822-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Naohiro Aota <Naohiro.Aota@wdc.com>
To: Tejun Heo <tj@kernel.org>
CC: "jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kernel-team@meta.com" <kernel-team@meta.com>
Subject: Re: Re: Re: [PATCHSET wq/for-6.8] workqueue: Implement system-wide
 max_active for unbound workqueues
Thread-Topic: Re: Re: [PATCHSET wq/for-6.8] workqueue: Implement system-wide
 max_active for unbound workqueues
Thread-Index: AQHaMxW+fup8mSSkJ02yqXaFIZNHL7DKm++AgArgQYCABQnoAIACkwmAgBTHq4A=
Date: Tue, 30 Jan 2024 02:24:47 +0000
Message-ID: <a4obzmueffpsmvlzfe64oksxdzyknxacxb2kkeytwzjtlzhz6r@w4lyfr6vrrp7>
References: <20231220072529.1036099-1-tj@kernel.org>
 <e3r47rru6go5fqxl5issvduzzmsxrtkefigrkfcnqiuouxm467@72hfzpblzr36>
 <ZaCMkV_pjPfhZmrn@mtj.duckdns.org>
 <znrr4upkwnaehoifwcfuhk25ddv4kriyhrymqawcvnxseampml@drwl73kklgmq>
 <ZabvdYTNhj6fiHgl@slm.duckdns.org>
In-Reply-To: <ZabvdYTNhj6fiHgl@slm.duckdns.org>
Accept-Language: ja-JP, en-US
Content-Language: en-US
wdcipoutbound: EOP-TRUE
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?vqxu4Z+MEFDHE/K9cex9Kullzdbzk5ceGzCxNFcGoMmGZX4jA7DhJO/Al382?=
 =?us-ascii?Q?4KKXhgh1Az4aekbyyqzTZdVhMbxRCwn0b9c7keu/uynYGu3ILqmc3UMm4Hpj?=
 =?us-ascii?Q?yk4DQkOz9OjxCAqISbNgY6kHQnED1gPcyIWywmhyu474ae6FaY2WyinsIpve?=
 =?us-ascii?Q?O65FzWCaKAzqlaqTGS+7hV8ih3weGEGexRA7wOic5vmLYZ2oRxA8j/SNKmEj?=
 =?us-ascii?Q?EWEkLzte4WDxq/aD1rotWoKud8u0fPHIlKh74LoDeEV96N71AU6ok4sBi7GW?=
 =?us-ascii?Q?DhSJNH1VPTtovvOv7CAtZxEg8m1V0DvDaCslbRwQ9F7+7ruqR2zExRVNsc6W?=
 =?us-ascii?Q?qo4v41U6bA5RmNN4SupPoXKbEUxFNJ9oHfCIrbYFDuxuDbDbaznvuTKCMvJ1?=
 =?us-ascii?Q?/PzMBf9vKil2hYjPiiulWVA0Z9gjZ52tH33iSh8z0OXAkE6/JuCmCIhHiFPM?=
 =?us-ascii?Q?aGpoBjsjpvzXovpoU4s2ScM0sZWCI1AOAJ3sFphC8JgjV+WkIC8Mh5OeQO0O?=
 =?us-ascii?Q?B9quTV1j8U3pLnQy2ymFNBT/TZw6dpicKlAvAq2nz89wTVejt3glSpTm9+Oz?=
 =?us-ascii?Q?DVdH4Rfg94YOj96TUSzo6RXyShceGIV9xeGc2xjk+XkOVPhNPr5eUrT++X3L?=
 =?us-ascii?Q?OD/f45gIKmJ7/ZOdukCzECJAzWB70WxWvG6v+w9qdel+BvrpKXKr8aYPvGJ5?=
 =?us-ascii?Q?rYA8q7a0PuPlYBQo8V5KGOTKMIzld142Yvil51600PmIAiePpSrBOf3e26Ry?=
 =?us-ascii?Q?R4tbfICUGRerlsWwu01W2HasHFUtL7BP4FbwPa76VRrbBNkW8xe9LhT7WgZB?=
 =?us-ascii?Q?vI+61nWawVMb+TILk+k2IUlNN2NXW7clTVVL0PYGt+Wsg1ng6CisgUCfBbIP?=
 =?us-ascii?Q?urR544ndX8SKBPJ1APt1oWjDseoWs9fCsgL/FWZFcn15BpXVOjduDuQvf8dd?=
 =?us-ascii?Q?n2MnNARCAexuQiYFJKc2q7s/TuizxknSMQZ/GigKAaQcw7RUFXdcpdxxK4mj?=
 =?us-ascii?Q?+A/pYWivziOA27TdpA7Mj6UNNwc74tCSyWxj8vNYCkJcHAeOmK2BT9f2H/KN?=
 =?us-ascii?Q?ASg5aZE3dA+KOp3C/Vmw7H63BZE3FqufFpAtQlpYgexyuWvg0Rs9sGp55QhK?=
 =?us-ascii?Q?//+s7MCvdtdPDoY+5MIwQE3X3XaD4E9xCIlNQmYCd5hYlRHcDGg9xvKGbMYz?=
 =?us-ascii?Q?ebUgKh1iTzNSoL+Cr0bhLvnL73GGtckeu1YCshQMY/IpbtcRNVItKi8GABDx?=
 =?us-ascii?Q?phcc1gLu9piUQuz/eB+Wkdw36HLrR2qS0iRHiK0lqep8uOYkMUyw714eWD0I?=
 =?us-ascii?Q?e/fcaQyyhVEnc58boYaQ4/IlAG8ddz4NPwg2gLqFcHGDhR3+ePQJHzssUUzQ?=
 =?us-ascii?Q?cq1XkEsPsHFzPEVKfXVt2VBW82bjQ3sXPmdFw2l0FM63DwQxYuSkTCfFRaBo?=
 =?us-ascii?Q?EdHiExL4/sLs7neONlP291ltNvBdIKhCLPZ/ucPJwnxBiw5F5QMS/ldMpQcW?=
 =?us-ascii?Q?UH1iLuB+lndUysv8MRGfap8ZnjjtdFgvS/igoQh6nwchgwES4LMcMaHNS7Vq?=
 =?us-ascii?Q?q98pwg1JcfBcHDIZtUdYju8PF4CXwJiN5Ha8x2ltCmhJQOzIThNz/8iSC3v5?=
 =?us-ascii?Q?oA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5615616D9953A94D83E85EDCF085D4CF@namprd04.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	Rm0uhhGtNwh1qcWHNAiv9Tn0iLRDyB5XHjK5VoPPGhsPz8Yf6528pdhU8ZBDeSeXrpbZJhlpZWytl9BjQj6qoWsu4QYF54tZkIY9KfEGGsVj+0HrXxIBcsljHF9x8FHNOqMaPTaRMwZjJnM9TJTqhpx5+8uwjgC9WSR8xtAm+X1zhNU9mjb8hc8shRw+/OSeHbYCneEmluA3nDg6q+klCONx+Ms8810fBoRo7FZAq7aQAdLEUrl2tO+fBrWTRQNm9uEwIY8prVQ3QcwYWwLrRKgVWIGHAlKau2wNuYgsHzpQ5kwT6k/m1tG7ppbLU1e+ske2K6oPfe2LM/rZk0tdLkQPHQIQJ4abRmS+nQH/nWZ5QSp2rsct6hT0/7oYlcMmc1UNlE0YfXWQQ8LKxXDu569nD7q+pYgsQ+z6fPSbZwN26x61rMkIOc21AwvsjZEngPE8KFN+U+BFgoAZ82VA1tYsjaTLE3hpPtZAWjKYDDac19LHnnLCymCuPF1TQAfDCnDX8IMYQJLWD79d7s/NNckIt+Bq/rl7kuASk9DUduojgGQFaf+leqq5oUKdB8+6paVLGJovwTCzH+3gXhRzHBQP0ihSD8DP4PB/SkESTEnOAz0jBzy05eOHZMfkDW2Q
X-OriginatorOrg: wdc.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR04MB7776.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b5250821-7b06-4f5d-455a-08dc213aa115
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Jan 2024 02:24:47.3563
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RWQ1Mm+jMVneWp19T2AMmx240XcrtjnG4LOz+26PzLc3wIXTaavJoNZFFPxMmgcA+/yQsiXVEzmJmkgaWLCj7g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR04MB7673

Hello,

I'm sorry for the late response.

On Tue, Jan 16, 2024 at 11:04:53AM -1000, Tejun Heo wrote:
> Hello,
>=20
> On Mon, Jan 15, 2024 at 05:46:07AM +0000, Naohiro Aota wrote:
> > CPU: Intel(R) Xeon(R) Platinum 8260 CPU, 96 cores
> > NUMA nodes: 2
> > RAM: 1024 GB
> >=20
> > However, for another benchmark experiment I'm doing, I booted the machi=
ne
> > with "numa=3Doff mem=3D16G" kernel command-line. I admit this is an unu=
sual
> > setup...
>=20
> So, does that end up using only memory from one node while making the ker=
nel
> unaware of NUMA topology?

Yes.

>=20
> > On that machine, I create a fresh btrfs with "mkfs.btrfs -d raid0 -m ra=
id0
> > <devices>" with 6 SSD devices. And, I run the following command on the =
FS.
> >=20
> > fio --group_reporting --eta=3Dalways --eta-interval=3D30s --eta-newline=
=3D30s \
> >     --rw=3Dwrite --fallocate=3Dnone \
> >     --direct=3D1 --ioengine=3Dlibaio --iodepth=3D32 \
> >     --filesize=3D100G \
> >     --blocksize=3D64k \
> >     --time_based --runtime=3D300s \
> >     --end_fsync=3D1 \
> >     --directory=3D${MNT} \
> >     --name=3Dwriter --numjobs=3D32
> >=20
> > tools/workqueue/wq_dump.py output is pasted at the
> > bottom. "btrfs-endio-write" is the workqueue, which had many workers on=
 the
> > unpatched kernel.
>=20
> If so, I'm not sure how meaningful the result is. e.g. The perf would dep=
end
> heavily on random factors like which threads end up on which node and so =
on.
> Sure, if we're slow because we're creating huge number of concurrent
> workers, that's still a problem but comparing relatively small perf delta
> might not be all that meaningful. How much is the result variance in that
> setup?

Yeah, that is true. I conducted the benchmark 30 times, and the sample stan=
dard
deviation is 320.30. They ranged as follow.
Min 1732 MiB/s - Max 2565 MiB/s
Mean: 2212.3 MiB/s Sample stddev 320.30

Comparing to that, here is the result on the baseline.
Min 1113 MiB/s - Max 1498 MiB/s
Mean: 1231.85 Sample stddev 104.31

For a reference, a result on reverted case is as follow:
Min 2211 MiB/s - Max 2506 MiB/s
Mean 2372.23 MiB/s Sample stddev 82.49

So, the patched one is indeed better than the baseline. Even the worst case
on patched version is better than the best on baseline. And, as you
mentioned. patched version has far larger variance than baseline and
reverted one.

>=20
> > FYI, without the kernel command-line (i.e, numa=3Don and all RAM availa=
ble as
> > usual), as shown below, your patch series (v1) improved the performance
> > significantly. It is even better than the reverted case.
> >=20
> > - misc-next, numa=3Don
> >   WRITE: bw=3D1121MiB/s (1175MB/s), 1121MiB/s-1121MiB/s (1175MB/s-1175M=
B/s), io=3D332GiB (356GB), run=3D303030-303030msec
> > - misc-next+wq patches, numa=3Don
> >   WRITE: bw=3D2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291M=
B/s), io=3D667GiB (717GB), run=3D312806-312806msec
> > - misc-next+wq reverted, numa=3Don
> >   WRITE: bw=3D1557MiB/s (1633MB/s), 1557MiB/s-1557MiB/s (1633MB/s-1633M=
B/s), io=3D659GiB (708GB), run=3D433426-433426msec
>=20
> That looks pretty good, right?

Yes, it is so good. Since the numa=3Doff case is quite unusual and it has a
large variance, I believe this patch series is a good improvement.

>=20
> Thanks.
>=20
> --=20
> tejun=