Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Tejun Heo <tj@kernel.org>
CC:     open list <linux-kernel@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: contention on pwq->pool->lock under heavy NFS workload
Thread-Topic: contention on pwq->pool->lock under heavy NFS workload
Thread-Index: AQHZpFS9Q4z69UfUdkyWw4ptMMUQRK+VxgUAgAEyXACAAD0DAIAABE6A
Date:   Thu, 22 Jun 2023 19:39:05 +0000
Message-ID: <36C8F75A-7C84-4D86-A721-6BCD6001CAFF@oracle.com>
References: <38FA0353-5303-4A3D-86A5-EF1E989CD497@oracle.com>
 <ZJNrht3NlLyPn2A0@slm.duckdns.org>
 <CF5AF0E6-5213-489D-87CD-8E8325A6560F@oracle.com>
 <ZJSfsvFtC_d265M1@slm.duckdns.org>
In-Reply-To: <ZJSfsvFtC_d265M1@slm.duckdns.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?R27cAMfPuz+QDuuWO3uk5pHuRt9vOrDQJpAgaM5xdT6NfMfey8CfPDxHDYwZ?=
 =?us-ascii?Q?+KuUb4AoxTToonbtpZOl5RqEpw3wc5rLOQxxpmZ5M3E38DdDq2s8PFx+lbAx?=
 =?us-ascii?Q?MR6NjskVlot8Pp0oc4kGvXVvKRFVM/0cFvRHFf/6wikxULuKdbLHy7mmnr0E?=
 =?us-ascii?Q?QRLmfS0Ug6u6hJyDRkOwK8YQy8s/LMmOZUuJ0i957UmyKZeQLYs43Yfod6W4?=
 =?us-ascii?Q?W+znyBLmFc/23zP8XftidJkGML5O655jIsNUvWyDBoCZ45JFsyrvPHAsfHDI?=
 =?us-ascii?Q?yzABRzx6h5VyeaXwQJZ4yytnfqndijFtfpzoprcUcDRSBJk407BdOq4vN/KX?=
 =?us-ascii?Q?NUIpTgVAAzvd+CYP64nGBVCBCxnnupz9Grbdx1zbtp+ysvGAMNUUEajde+D8?=
 =?us-ascii?Q?dCu3S4/2PceBF63i7tsdiAlDuO66fb2DNAIkB0G08lyX0CZZ9vR/TFXscxVh?=
 =?us-ascii?Q?CvEqtut182s55XKau75SwKUgm//4jcz4h4jjYLAV4zYfZZCNIQoWlpLpAU6e?=
 =?us-ascii?Q?+ibARTbwdly1AD+DaBIcrL+WsmsRY069vi0VvVmrO7R4hNooSFYA5Vo5ORS1?=
 =?us-ascii?Q?71djV9TqFCYQbqhOCSR3aoI9kXrMGHfAcjviUOCDex6Tm9Jt4u/2UKIMG9jV?=
 =?us-ascii?Q?Zi9IbCT5JNtiQisMS2P/fL30bINUxNzdV6P3NZtkqyUYSRF2zadXMpP7aRw4?=
 =?us-ascii?Q?eMOzhrTVJreQLAWPLDKZzJVMf4EVi8UYSdNHVun1Dms0ttqemicQa5c4XwVV?=
 =?us-ascii?Q?/lP/AVAarBYFI3A1+LxiMAWqBbIc7fHvqlSYO23j9kN619D0sAgetSgzvmiz?=
 =?us-ascii?Q?Z3T3OSVltvHRqObM05INvJJ0hjPGRpxt1TLTZPgpSt/A8DBcEIe6hZwJjL+u?=
 =?us-ascii?Q?EPLD9U32cyKs+FwSifuFp0zAbe59bViqh6wVaXY7iaSWATJ8GaCe7DRDmks7?=
 =?us-ascii?Q?IFj4WHo9VWjLPmMQuZQ4u/wDQzsXcJmVMuLyiWX3DjDJwSl3gM0iZsSKbv2m?=
 =?us-ascii?Q?CXV1sfII+/ru3H4LNftoJWamz8YCeTInGJLkBBRYkdlB1zHggqnO1U7fy+0y?=
 =?us-ascii?Q?C0gmNPijeimJR4BP3c3sI81n+qCzecCppDyAoY4NEpvI4YQTsYe4+tPWsThI?=
 =?us-ascii?Q?dLojjEjkI0JzFCafFUmKnNL41Fu2k3L7cDv4po4vCXVd4itZbkqgavNN125X?=
 =?us-ascii?Q?3TxFoYhJ9kTYWd3/z+NcOTXNjlEXwFn8DmZmC3BSbY/+uXHuq4mloTbIMip5?=
 =?us-ascii?Q?CFWLsKNW0R1I+aczvasd0uTZmlerFNmblJLgFiGQ3Kw3KYNljwn3BLP24/OL?=
 =?us-ascii?Q?D3JlOfazecmevh6BVML9V8ZKfA9kvTVk0iaGeiA1Gcm35d5T+3LUGNYomm30?=
 =?us-ascii?Q?hBpT88nh36PjdKheV3r9llVtLvCd/4dunkPfvjYSPqisKzFLT1FH6sacErSN?=
 =?us-ascii?Q?RroDLGfSIjHr6GjEYm1AvavxsTwYTJCS64TfAGYt7svnYitEXiF7gXP8QFzR?=
 =?us-ascii?Q?R0rC3UeSSmLQJG3cK+4oAOTmzOrltM7WgZ+JIDf+D78kjw0SE+ejDgmkilmW?=
 =?us-ascii?Q?PrUVj1jENtn/SDzUDn3L8g90lglWhcBjkUA/k3IubAbaVGGKlsFzHi1OeYxY?=
 =?us-ascii?Q?Ew=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7E353C41E7F26C4DA00856A7CB341E8A@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: /kWGEaSOJT4sx36owsRvnU02zYJkp+/s9fNRW8aFdjICumCDzpNR6wXMHtces6vLA1VQWZ8wP81nnTv/ESUtUzeLpHd51Iqq/lPqZtYiR97W6RiBIFMUlkPW+13wOMkE7FSdlJONtSLI6oHA/H6uaBv6capLsqh4MQAvqdfDB0rwo4zUAoP4WTANfy60C8rsShTj4uSKQG2ybW17mouVJPyKXRhGjmsgNyOEWRyX2gExTIWhOLndUv21HUUX4v7Gw67WF2z9qGJvDYgWQnaM3LJtMx493GYHdCxYEqXotbR52ICOX01Oh3qUW+FpAUGst6n2NKNIJd3fKUwqR0dYSCqQ9ImT8OtYSh8q0bFC7kjFuF19sSTe4pExbAuNNezWI/2VG5kb9SNb3DxiHOWcfyfw/kPvqmL7CtYBqHN9h5ElAhbZEpz+0m2stWr98MVL5BApz2TsV3cemAEeq2qalSsIH+Hcm9D4JfPTXhJgf9GXHbwjWl+UOI8Q5Oq8b1ANNPCxEjws539OQVuhdr4O9PdOUQ+CVhfZ9HYP7601noDuFTeIGqZycDv68dhMkCCQXLZYjJmVz0EdX7ZoG0qkJSdAQStolf8+6/k3ZrHTRF46uB9ULSUR/A7pakSThrhNycaWL5eSCB221RGPqQl59WXNzuOYIK9FJTHysFL4OFB4uWEgZkuUaJVNDKdP4AQfYLcTKfKvGZ/Mix48z8gg4Rc94MGOUJBY0fu2frZ9S5/IWHrIVfOLEL/5U+QznwTFMfrP3XDo5N1oj4EsheRpQzf35vVCIDvAjDZwqUnAf5I=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3c379399-31b1-4f86-9ede-08db7358570d
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jun 2023 19:39:05.7445
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: FBqxuCBsubMzzXSZDTW/o78o5NjB/WHYIm9UE4Ik1Xi1JjcDRHLyo6zvqjfv2JnqKOMlLNS/c4lYeH0Nw2/4GA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB7379
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-06-22_14,2023-06-22_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 suspectscore=0
 adultscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000
 definitions=main-2306220168
X-Proofpoint-GUID: RmvXN9YkYqTQsyVUXjDMta3_xim_JBvr
X-Proofpoint-ORIG-GUID: RmvXN9YkYqTQsyVUXjDMta3_xim_JBvr
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 22, 2023, at 3:23 PM, Tejun Heo <tj@kernel.org> wrote:
>=20
> Hello,
>=20
> On Thu, Jun 22, 2023 at 03:45:18PM +0000, Chuck Lever III wrote:
>> The good news:
>>=20
>> On stock 6.4-rc7:
>>=20
>> fio 8k [r=3D108k,w=3D46.9k IOPS]
>>=20
>> On the affinity-scopes-v2 branch (with no other tuning):
>>=20
>> fio 8k [r=3D130k,w=3D55.9k IOPS]
>=20
> Ah, okay, that's probably coming from per-cpu pwq. Didn't expect that to
> make that much difference but that's nice.

"cpu" and "smt" work equally well on this system.

"cache", "numa", and "system" work equally poorly.

I have HT disabled, and there's only one NUMA node, so
the difference here is plausible.


>> The bad news:
>>=20
>> pool->lock is still the hottest lock on the system during the test.
>>=20
>> I'll try some of the alternate scope settings this afternoon.
>=20
> Yeah, in your system, there's still gonna be one pool shared across all
> CPUs. SMT or CPU may behave better but it might make sense to add a way t=
o
> further segment the scope so that e.g. one can split a cache domain N-way=
s.

If there could be more than one pool to choose from, then these
WQs would not be hitting the same lock. Alternately, finding a
lockless way to queue the work on a pool would be a huge win.


--
Chuck Lever