Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Tejun Heo <tj@kernel.org>
CC:     open list <linux-kernel@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: contention on pwq->pool->lock under heavy NFS workload
Thread-Topic: contention on pwq->pool->lock under heavy NFS workload
Thread-Index: AQHZpFS9Q4z69UfUdkyWw4ptMMUQRK+VxgUAgAEyXAA=
Date:   Thu, 22 Jun 2023 15:45:18 +0000
Message-ID: <CF5AF0E6-5213-489D-87CD-8E8325A6560F@oracle.com>
References: <38FA0353-5303-4A3D-86A5-EF1E989CD497@oracle.com>
 <ZJNrht3NlLyPn2A0@slm.duckdns.org>
In-Reply-To: <ZJNrht3NlLyPn2A0@slm.duckdns.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?s47eMeAokGCoKP8WGezt8pT2LwMWD1meiMzMg8GaxLyT41rkhjRXwsptSaFD?=
 =?us-ascii?Q?2pCnAVQbTMfWpvrtIGsT/ClOMgv06NjwyEN3NjbWHl41eS2GAiuelj+40gLL?=
 =?us-ascii?Q?sv5dIG7ZBVz85Uk/u0Cy/sq910D+HvEm3aH9cw8rdd+5N8yOmifqcI+lHoAG?=
 =?us-ascii?Q?s9XVG6VcRMZfvQOfd66YHe+GJDJpKbEPUl3Rt8lPW46nnnWhhTJAk7d0Dy8X?=
 =?us-ascii?Q?Eb0SEjL1x7L2P5bx5HX8U5luLcFnF2DrO0yVZF6Kx5GsIBGbzo5FAaPKduYt?=
 =?us-ascii?Q?v0vsAPLyHA2941UQcb9DCjlVaSKbbvsoEy00znd9IOMdVssxBjU4iBg5Cg12?=
 =?us-ascii?Q?Ex+5SDbCZG+hn3rEliCC0s3NOkA88yBZDpphAIo9mfK83TWUnAKzuaqZD5Bz?=
 =?us-ascii?Q?jvk5jMWmJTy+NOQwZh7CVND3pxgSyl6vXzoHuaPr4yiFunLtqO7jqP4QP09y?=
 =?us-ascii?Q?WlZhuhL7JGYT55zesIvty3OrmFAXhMtneKvHHk/NJGUy96GpnyjM4ARaJCHu?=
 =?us-ascii?Q?wRjqL6xIilPfe7bVtLs6Lpl1qMNI7fh8nmb6yy2o85VdZt+A/lnzLK7R5QGl?=
 =?us-ascii?Q?KYQr7fDc4G1GR+PgbRd/m4Xs3/W+NfKbq3jTRxvIW+VpsG3AmQ+LrF95xGAC?=
 =?us-ascii?Q?BBVfsVPjnkqbEmebzotgeMhKp+dsNcG81K7uguZpbCqMOwldcEAetWqM7uYX?=
 =?us-ascii?Q?0rhLUefEvJmh+IVMd0EmsX0t9jfiUdLa1Q19gBtOU5i5khlZSjIMyk6q9NRH?=
 =?us-ascii?Q?rfaQ6kb+9i59/KyQm9AgFo0XSIlDU7hD+DWfrrB9CMIdJ/TC3KO28HDK4iEt?=
 =?us-ascii?Q?gNyXpsKeqnlOUUk6RdnCk2h92DvEglwQRJuncCmzmf4fGqP273+MPE6KfX0x?=
 =?us-ascii?Q?VlbTXNK6Mpaf8JVn+8f7xNij0sVm7WOD+aeZDtLrpcDmyvH5rQkdAF/Hj343?=
 =?us-ascii?Q?uIs2YbtQy3wNpC3YRHRMdkyLoWDwtGRhRBIky0SV/CgPzTLixHzVwkhz6aF+?=
 =?us-ascii?Q?76xKKr+hYbzA8LgpSRGKvXev3vwi1w8UTQ2vERhJyQQ6Y8yTVNaQUtOweJzh?=
 =?us-ascii?Q?erfF/pEY4KdfRuSf//QVXMST9BzmXQqHWsvEkZe7Ghz+inw7dcAyyXteY/Fk?=
 =?us-ascii?Q?yGMhAmlLW5lCcWOtqd7PxAT4E44ijqCcuoSEZs8ShB2qmKalr4CZdg9yIlY+?=
 =?us-ascii?Q?jQgba6032s5wHvR6soudpzLCZuEBJAEbbz1mlOnv0zbOPa9gc9M+nko/QIhR?=
 =?us-ascii?Q?sRi6LsN2Bb9RP5TdnnSJM5mJjb92OPNEt+DB4NRoHj9lqUBatvdnio/rIgdi?=
 =?us-ascii?Q?63rUtAiKiXqiE3BLjx6N4C+hxLbpwETf+0432l56zoRwZCLd6sNjIfwjGv/J?=
 =?us-ascii?Q?N0Hs328zw5MN4q99dLXuKQuQqP7sBSdFMYUTFfCoBROLJ+Er4ahCd2Hour3a?=
 =?us-ascii?Q?ovQj9xlbhwQKjKThET5VKmmJd48WrsE7TpXOmiu5swDS9ZsVFJybpUhVrWrv?=
 =?us-ascii?Q?OtgrCPrbwu1UiDya8Pv6JJlNbgW96UaGsrZNWspUGUoD2jPPbN2PipXlyMqR?=
 =?us-ascii?Q?fYaIh1BjVIoZYqD+6+eYcPQ1nRmGWg4YeP27Mgc7ipak6zMT6r2XLQKLBNaa?=
 =?us-ascii?Q?Qw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7A30CEF6AD513D4BA38B11AC676F7B6C@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: krF8eus6x3O3PSZFY/Pdav0j6klTdi+Vl/HStXufacBDo0qtqrWM+pjy+Vl5gLOsm0u49WdJBIMefOudD+MHFh6lnVPkXmMIlIlvuA0R3Mwd6rQ9qn48XvHnylmJbli/JVWWxyyrb+tx9J+G3kXgCz9Al1IDY221ZbgFbGAUSm2CSg2xzC6bRULeyC+wIhTPnvJXOv1MZgmCuBmkBqcR3bcx6Jbrwrdq6dDb5ocQAStRH6lMLxfkWPpOvqgar3rFIdyE/v21MNLWKyWWrbpbtRY73Q+VKyubhLsc4uaicHAejnIpI9seMM9KmaFB0VgM17Z0rqLwRW+wXppeVa/w8U4KpT11QGBxJFASDrKbJSnele0dbo+N6GnMsQRmnhHIfok4+1uDlmLpPs9Rg0X2iwgXXQvFZIGKU+ZWJcrZObJKuX98vUphCN7hrsLhXVAhLE0JJtC2xQDVea7Ue4Qoo75L8+GaoINPKtF9Vb+nezQmNbUIIsQTTVaxmD1lfAn4zYPKjb0qdKnJiGqpilLjNywN11QfCF4Q3ls783wM+y9+KfY2tNNb1KHc5P7/2gFgtxM6HvYpZ6uRVvufoCe3qIUBngMpf0/7+KL1Eexd+c1EHOfsln3rc25vR2E9ZvDUrJ9MxKqEnWqlndL671vS5yPkbDD9/9h3uDQLa5AK+A6AxyN+OpibpAPA+f/Lcz8PlTO0jd+5H+qbUDzmPSu4EvBQJR9QLCDOXiiXDjvHLDYxakJHoTGkdwrfjlaEbcT9rh7tSLnNB12irQ9JYoCmKVA3NDhDiQEfMEywSNEqUDQ=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 34f35f11-39ee-4018-be50-08db7337ae75
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jun 2023 15:45:18.9819
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: vrJw5jHzsELkIkAcGMkbQhH9nPmD0O6AwEi7/9HuBomkcMMaMnuag1sCJfaIUJaubiBP2rM/u4wskwa/rZleTg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB7683
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-06-22_10,2023-06-22_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 spamscore=0 phishscore=0 suspectscore=0 malwarescore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2306220133
X-Proofpoint-GUID: e7BSW8I6sBqC0cUzc-kQKHGTmW38Oq3V
X-Proofpoint-ORIG-GUID: e7BSW8I6sBqC0cUzc-kQKHGTmW38Oq3V
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 21, 2023, at 5:28 PM, Tejun Heo <tj@kernel.org> wrote:
>=20
> Hello,
>=20
> On Wed, Jun 21, 2023 at 03:26:22PM +0000, Chuck Lever III wrote:
>> lock_stat reports that the pool->lock kernel/workqueue.c:1483 is the hig=
hest
>> contended lock on my test NFS client. The issue appears to be that the t=
hree
>> NFS-related workqueues, rpciod_workqueue, xprtiod_workqueue, and nfsiod =
all
>> get placed in the same worker_pool, so they have to fight over one pool =
lock.
>>=20
>> I notice that ib_comp_wq is allocated with the same flags, but I don't s=
ee
>> significant contention there, and a trace_printk in __queue_work shows t=
hat
>> work items queued on that WQ seem to alternate between at least two diff=
erent
>> worker_pools.
>>=20
>> Is there a preferred way to ensure the NFS WQs get spread a little more =
fairly
>> amongst the worker_pools?
>=20
> Can you share the output of lstopo on the test machine?
>=20
> The following branch has pending workqueue changes which makes unbound
> workqueues finer grained by default and a lot more flexible in how they'r=
e
> segmented.
>=20
> git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git affinity-scopes-v=
2
>=20
> Can you please test with the brnach? If the default doesn't improve the
> situation, you can set WQ_SYSFS on the affected workqueues and change the=
ir
> scoping by writing to /sys/devices/virtual/WQ_NAME/affinity_scope. Please
> take a look at
>=20
> https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git/tree/Documentat=
ion/core-api/workqueue.rst?h=3Daffinity-scopes-v2#n350
>=20
> for more details.

The good news:

On stock 6.4-rc7:

fio 8k [r=3D108k,w=3D46.9k IOPS]

On the affinity-scopes-v2 branch (with no other tuning):

fio 8k [r=3D130k,w=3D55.9k IOPS]


The bad news:

pool->lock is still the hottest lock on the system during the test.


I'll try some of the alternate scope settings this afternoon.


--
Chuck Lever