Received-SPF: pass (google.com: domain of linux-kernel+bounces-44905-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Sender: Tejun Heo <htejun@gmail.com>
Date: Tue, 30 Jan 2024 06:11:16 -1000
From: Tejun Heo <tj@kernel.org>
To: Naohiro Aota <Naohiro.Aota@wdc.com>
Cc: "jiangshanlai@gmail.com" <jiangshanlai@gmail.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kernel-team@meta.com" <kernel-team@meta.com>
Subject: Re: Re: Re: [PATCHSET wq/for-6.8] workqueue: Implement system-wide
 max_active for unbound workqueues
Message-ID: <ZbkfpKv7CQs2u9RH@slm.duckdns.org>
References: <20231220072529.1036099-1-tj@kernel.org>
 <e3r47rru6go5fqxl5issvduzzmsxrtkefigrkfcnqiuouxm467@72hfzpblzr36>
 <ZaCMkV_pjPfhZmrn@mtj.duckdns.org>
 <znrr4upkwnaehoifwcfuhk25ddv4kriyhrymqawcvnxseampml@drwl73kklgmq>
 <ZabvdYTNhj6fiHgl@slm.duckdns.org>
 <a4obzmueffpsmvlzfe64oksxdzyknxacxb2kkeytwzjtlzhz6r@w4lyfr6vrrp7>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <a4obzmueffpsmvlzfe64oksxdzyknxacxb2kkeytwzjtlzhz6r@w4lyfr6vrrp7>

Hello,

On Tue, Jan 30, 2024 at 02:24:47AM +0000, Naohiro Aota wrote:
> > If so, I'm not sure how meaningful the result is. e.g. The perf would depend
> > heavily on random factors like which threads end up on which node and so on.
> > Sure, if we're slow because we're creating huge number of concurrent
> > workers, that's still a problem but comparing relatively small perf delta
> > might not be all that meaningful. How much is the result variance in that
> > setup?
> 
> Yeah, that is true. I conducted the benchmark 30 times, and the sample standard
> deviation is 320.30. They ranged as follow.
> Min 1732 MiB/s - Max 2565 MiB/s
> Mean: 2212.3 MiB/s Sample stddev 320.30
> 
> Comparing to that, here is the result on the baseline.
> Min 1113 MiB/s - Max 1498 MiB/s
> Mean: 1231.85 Sample stddev 104.31
> 
> For a reference, a result on reverted case is as follow:
> Min 2211 MiB/s - Max 2506 MiB/s
> Mean 2372.23 MiB/s Sample stddev 82.49
> 
> So, the patched one is indeed better than the baseline. Even the worst case
> on patched version is better than the best on baseline. And, as you
> mentioned. patched version has far larger variance than baseline and
> reverted one.

Yeah, the average being similar while the variance being way larger makes
sense. Before the revert, it's spraying things across the machine. After,
per run, the execution is more sticky, so you basically end up amplifying
the varince.

> > > FYI, without the kernel command-line (i.e, numa=on and all RAM available as
> > > usual), as shown below, your patch series (v1) improved the performance
> > > significantly. It is even better than the reverted case.
> > > 
> > > - misc-next, numa=on
> > >   WRITE: bw=1121MiB/s (1175MB/s), 1121MiB/s-1121MiB/s (1175MB/s-1175MB/s), io=332GiB (356GB), run=303030-303030msec
> > > - misc-next+wq patches, numa=on
> > >   WRITE: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=667GiB (717GB), run=312806-312806msec
> > > - misc-next+wq reverted, numa=on
> > >   WRITE: bw=1557MiB/s (1633MB/s), 1557MiB/s-1557MiB/s (1633MB/s-1633MB/s), io=659GiB (708GB), run=433426-433426msec
> > 
> > That looks pretty good, right?
> 
> Yes, it is so good. Since the numa=off case is quite unusual and it has a
> large variance, I believe this patch series is a good improvement.

Great to hear.

Thanks.

-- 
tejun