Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Sender: Tejun Heo <htejun@gmail.com>
Date:   Thu, 22 Jun 2023 09:23:30 -1000
From:   Tejun Heo <tj@kernel.org>
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     open list <linux-kernel@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: contention on pwq->pool->lock under heavy NFS workload
Message-ID: <ZJSfsvFtC_d265M1@slm.duckdns.org>
References: <38FA0353-5303-4A3D-86A5-EF1E989CD497@oracle.com>
 <ZJNrht3NlLyPn2A0@slm.duckdns.org>
 <CF5AF0E6-5213-489D-87CD-8E8325A6560F@oracle.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CF5AF0E6-5213-489D-87CD-8E8325A6560F@oracle.com>
Precedence: bulk

Hello,

On Thu, Jun 22, 2023 at 03:45:18PM +0000, Chuck Lever III wrote:
> The good news:
> 
> On stock 6.4-rc7:
> 
> fio 8k [r=108k,w=46.9k IOPS]
> 
> On the affinity-scopes-v2 branch (with no other tuning):
> 
> fio 8k [r=130k,w=55.9k IOPS]

Ah, okay, that's probably coming from per-cpu pwq. Didn't expect that to
make that much difference but that's nice.

> The bad news:
> 
> pool->lock is still the hottest lock on the system during the test.
> 
> I'll try some of the alternate scope settings this afternoon.

Yeah, in your system, there's still gonna be one pool shared across all
CPUs. SMT or CPU may behave better but it might make sense to add a way to
further segment the scope so that e.g. one can split a cache domain N-ways.

Thanks.

-- 
tejun