Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Tejun Heo <tj@kernel.org>
CC:     open list <linux-kernel@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: contention on pwq->pool->lock under heavy NFS workload
Thread-Topic: contention on pwq->pool->lock under heavy NFS workload
Thread-Index: AQHZpFS8u3GiMvOyOE6KS73MPu0Bsg==
Date:   Wed, 21 Jun 2023 15:26:22 +0000
Message-ID: <38FA0353-5303-4A3D-86A5-EF1E989CD497@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?D9RkrKoGvgialNFNS6tJMMATP0V4G8cyPZcgFN+kXwx2OEV/NpnYNMnLoLX3?=
 =?us-ascii?Q?xUhNpHcPhIE5K98vSCruVk+QfUAaYx6eNFipl/aR8YnFp3I7GSDn5BffAkFA?=
 =?us-ascii?Q?r5Sxr+T3nWX46y+qaNl06CliZQQXhTctR3/tsXO/Ec1keXkiOGNijLq60RSl?=
 =?us-ascii?Q?6VX+hwznDLquJLd3iC4qY4LIO6HvFYTgTWUPg5/wO2Xi8yEVAnKClqIQufmC?=
 =?us-ascii?Q?pItncm9n66cpbLKHPvU+q9eZETfAY52TZ8WTmtQFnJCRjAqHm2sLxeUQlBty?=
 =?us-ascii?Q?n2djVVdgIx8c3Zjr5wxphwAumVK2TM7OT+xpXgoDlEDZJ8O518Cho3WvWLZp?=
 =?us-ascii?Q?Xk2ZhTblgqbMOpV+FFokfzRyUBd1hwsO5KGVPFtnkJsWVy9VCrZFktfNmA6g?=
 =?us-ascii?Q?ZnkEik3DxpjgWnh990vu9bXH6tH485cXFOeBmUF1b7f8UhpAzFq8Qx1GirqE?=
 =?us-ascii?Q?ggelK+4ItogOF4GMVISV3CvTSxcMu/aHh7pK5xEgIG3rQd8LU+PDSoBP+1pp?=
 =?us-ascii?Q?gU+JhI5SDKfHPIpe1+eXr/4BEQrffpbq0vcmSKVo1M07ZgOF7ajOfDzmqZe9?=
 =?us-ascii?Q?KywqPD6/cnNl1mvaFFwNr1WZxy+tWKl99Ns8V3j287NTSykbf97Bs9EBNPML?=
 =?us-ascii?Q?Wqp6ccGrqYX+INX1SR1u9GuaEX3oKj/FqhkOMwnVDZSzoi5U6M2nbb7zjHcd?=
 =?us-ascii?Q?xBAu3ood8z+vPIt4B8GIYVavjol+MrYd+tp7lxTL9R/YTKPbPRt19csrtpwt?=
 =?us-ascii?Q?q3O0CQwW2c9NGpokrXB0DM59NVyYDpzRyRIvwGwr+DgYSuWcQ+Xi+UI3fFDy?=
 =?us-ascii?Q?dvxyBcLWqbvD7sc9Pxnv3Vakep/wbJm7bg6S/1ymNJ3NKS5CKCkC6+7QgE+W?=
 =?us-ascii?Q?HEaYpGg/9R8hURrxkONowdmkF0p/q7AqVeFCZPTHeN60/98NhHuVsjLEXBct?=
 =?us-ascii?Q?gxpDAozc6g8lMN0VU85EeCvDSvSObPWQazBrV7Oh23jJKQ8I0g/np+A9q6oa?=
 =?us-ascii?Q?FUw/d/dGpG5GXDdVW2aB+Z50o7GDZm1m9oIBzQ9ldj7ycerLIauOotIcaRmI?=
 =?us-ascii?Q?DaMOcfs2s9oOyL8MDScS0yn1O2IYlpFnE2uyLvLDkVckXXkA1RR3ER/Y3ZK+?=
 =?us-ascii?Q?lw9jEnCFogGZvbhpIbOgKWvlfnRQOMniHuVpi8zTLr2sh6U/TxfoPvcjEeRe?=
 =?us-ascii?Q?gljSV1xzTi6rzIy/7trBiuK9HYAtRKsPtNCS1rvPjaW7HwRzWBD0qRecxR0t?=
 =?us-ascii?Q?vSWTodvoqsnf9BoZ/tuN0U9TbBAfinB6lfztx7ir0CEFHYWkJNDUCGRDB7E9?=
 =?us-ascii?Q?HZEhDGpX/w8vfKc8oDq0CYdbLg/8wOMMcvqjqC9lzNrljrBTRRxoR2UlsioS?=
 =?us-ascii?Q?U9br5s9r6P4BV1b5+xsQ6Zithv2uaCj+ULbvRctC2w/C5iaBUXg6IAuK5IUm?=
 =?us-ascii?Q?7jUWZQByvkeC1pu6qYlHGncmv2ruu2dQe3uiZa3TVisHEgYK9gS3O2twxzLN?=
 =?us-ascii?Q?w3KjF8EJWgMNgD+VTpYVEd7FBq+zEiQ59YAi+1x+Y2D6hN6XZ/THsRgLcPG1?=
 =?us-ascii?Q?/eXdExrH7N6XL7sCHN3IJK1+nGeNv+3UYhZKNDpECIrEaKZKuH3UU6HK0VeU?=
 =?us-ascii?Q?rA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <AA6BCD8C993CAF4CB41732DB7CF3097F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: r3s3BRA3xQGqxqxUly17Zv90xgraUHpbGzh4qcwRnxb/hiK/uzA+xE+jIK28YXw3SK/NAuYhmoAU8PPP/+xaq1cMlvu/vcxqkEV+sIKWzvJ1BnXZSzGEMSMXE6+OoESWd2nocGYDeyB9P2KeBOg2CS+TP6uA7VAmO2m7lhAT5Q6CD54oAuEk8OLL5x4r7Itvp8978cSr0d8b5watp1PS2G5fw9H8nwYPxuXl2O56QpXl2CrQWZRv7FNy0RcSHQywQq26Ks/A2XrlAW7TkPVQ+TlRz/0xOglmBVe2BT4psrSiaQIxaqhQucQq7Bqp+R/k4yv1HEqy1MAHhzZxitL2C/Cz5sc33InMdFXvjpZWcmJl7iQdPPZr47m6GtdVJlup9DCUv2MlF/cviXBr1wScLZcojgN0hsqB6Vl5w2gtP3Z3IlzXgw73unKHXcyVUh3N87z/3yF/0RRAWNA7JpF7CLtJsST9WfmLZgs/MYJiWB14w79nRkzguSDdGzhu1/V/ZeLCicARlUfwPB/Pb4CaPAkzxAx1lS7AFG7VHvnCLxnBVs7c/NmU/xpYtj53xfo22QyE6K0RjGmQvMqJGPqcb9Ja/0EA4L8kjwT1m3aKXoILLqr/6RuNSohy5wQ8cbMELnANlRzYCNbpgtTQV65qXdm1td7LvGY1ONIVYDT5rn7XMIkfJqsYBrAfPCs9EToTzq9cMSh4Uj7NHHAOaPHDF2V6HAMu5+4aa5z9Fy5mXhs/hfHYxI7MZFLjCT95gDEKqEFFtQ9ULp63YJOBw5KzQzQ/jv3YG3mbOR54XnnPMbs=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c19418ab-e591-4ecf-16f1-08db726bded1
X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jun 2023 15:26:22.8264
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: uP9PE7xljy4E0jfdPTfqJwsaYdrGDqSuyYe4ZaoQewN3mu0AwwzjvQUsyRpTz7Fl7MXOb/r/WzsskXwM+JxlbA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7091
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-06-21_08,2023-06-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 spamscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2306210130
X-Proofpoint-GUID: VSEmmlUbNBr8rDTZtMkx0kyvku4sH7Mg
X-Proofpoint-ORIG-GUID: VSEmmlUbNBr8rDTZtMkx0kyvku4sH7Mg
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Tejun-

lock_stat reports that the pool->lock kernel/workqueue.c:1483 is the highes=
t
contended lock on my test NFS client. The issue appears to be that the thre=
e
NFS-related workqueues, rpciod_workqueue, xprtiod_workqueue, and nfsiod all
get placed in the same worker_pool, so they have to fight over one pool loc=
k.

I notice that ib_comp_wq is allocated with the same flags, but I don't see
significant contention there, and a trace_printk in __queue_work shows that
work items queued on that WQ seem to alternate between at least two differe=
nt
worker_pools.

Is there a preferred way to ensure the NFS WQs get spread a little more fai=
rly
amongst the worker_pools?

Thanks in advance.

--
Chuck Lever