Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Jeff Layton <jlayton@kernel.org>
CC:     Chuck Lever <cel@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v1 0/6] Fix some lock contention in the NFS server's DRC
Thread-Topic: [PATCH v1 0/6] Fix some lock contention in the NFS server's DRC
Thread-Index: AQHZsnxbV8dyPdC2OEuC2hGjvlROW6+y+MMAgAAMcQA=
Date:   Mon, 10 Jul 2023 13:47:49 +0000
Message-ID: <0959A996-3FB0-4920-AB5C-BE84A7339DE9@oracle.com>
References: <168891733570.3964.15456501153247760888.stgit@manet.1015granger.net>
 <35ca6665f3cbedb60b904bfe4e74ca37c32af985.camel@kernel.org>
In-Reply-To: <35ca6665f3cbedb60b904bfe4e74ca37c32af985.camel@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RwUkuFr1AHOymM1v45kgEF3a7Ulby6KXTp8sHXon4z+yxfej/t2VUrxQElTU?=
 =?us-ascii?Q?1p8EtZPA0S1rE5aFBCcvYqbnNDmARj1u2/NAxsTWJcTscVZUphpQv+3px5LB?=
 =?us-ascii?Q?jTlfFF8152/CFq1/hEbm8e67VSACk6xug21i/Aw09Y95NN+/8qm1U2xcMlfJ?=
 =?us-ascii?Q?/OJ1ArmXskOTlRmKL1QeomC1sv0RmASFOggK8h+2Eyr+144cypTIS7iQlGtd?=
 =?us-ascii?Q?bLqgX+2IdGcPAxjsuYWFZBmq0uOLn7EhIHaG/UAjmy2HHlXPzl7M/uF5sJRv?=
 =?us-ascii?Q?dJQxO/7GLah5PpglItWPHRIz8AuD+m9ZvnENlmAU9v83vF1LX8hGWJlOTtBh?=
 =?us-ascii?Q?HZvmDHEVoehbxZ8YL8NTYi+buwWcF/+Mogn3yDwH18wzwsbBS0xW4tBrGkVi?=
 =?us-ascii?Q?Hx3uHYHp2XO1DX4Y3f6f4dQ6PLnEwKp6QlkVGEqCch6W4zmOL+gnDhpJZPIC?=
 =?us-ascii?Q?VAZ76QhaxgHVP2SHKZxJjcD8+H7VEm33J9IJXLN7/dDSGLIokn1ekJdRvrSG?=
 =?us-ascii?Q?kptlPz+HvefBW6MQVJGgQcwzWC1HTp80MYSDxDujCj6vOJTY+BgZyofKeyv8?=
 =?us-ascii?Q?aI3Fd2LuF7FA25VOpEwHYVwmqFakVGvLAApUP8QBPpcMAN7dslhJXE19DKQC?=
 =?us-ascii?Q?1uv390NSktABrhlNQGvfun4Q4Qlixb4iSDC+UweDUNnlCLUg8NkAVO/aG2So?=
 =?us-ascii?Q?AQXex3STt8mO8fpzqzjV8tl9/vY5zfnr0D7C+n1TmJc29HxKF9sNhvbhZfIU?=
 =?us-ascii?Q?XVadLnBwjvk+ZipF7EGOUeqGhgZga0ZKttSEUyUGbepYk4qHokOPnUVLQquL?=
 =?us-ascii?Q?wvz/d0J01Sea4IiqfF8XrQY9lAgLwjz55KuR++QefoFFgyZWrEu8hNkmMDAb?=
 =?us-ascii?Q?X3Or488F18yswlj9qYVqLm9WKgy0Ia18NFOENCA9oG4pxCkI6kUemt7exU4P?=
 =?us-ascii?Q?RDTP4QARVeWoeNgW/pg7rcvlwwHDVuvQmNtn3ETw7LnkafFSNiBEq2Buwut1?=
 =?us-ascii?Q?bkei8PFi9h20bMPMmbXOzywZthbB4aMMEinFVEVz4ups7Hr9R2/L2vlnbLRY?=
 =?us-ascii?Q?LXUcrf5Y7CM6RtzhsVmCcePTrSC5iC4Quw75MCLcA50MMuEHWO5HwdQ/p6fo?=
 =?us-ascii?Q?SvzxFv2hpV9mlksqiikdDLJCjkN0pmd/pW1mXBv51t4LEGYMVITQhT04tfm1?=
 =?us-ascii?Q?mTp/cipBSjgVx7X0NotSUaP3A/VaNZV0blCAU1KtmjYolW6gJcOicnu/Om/Q?=
 =?us-ascii?Q?mevCl7lXzTdWjkeqRIRGCyqahkkf9+x+udCyBmJYuuP1ZEwLl7ZoDrMa8veg?=
 =?us-ascii?Q?PJb5fIPDN+5Yn9dGDpTOpzMjf+D35Wq6L0Ul7MVKECdWtJd5FwgXFf6mDI0x?=
 =?us-ascii?Q?rcoG0zxleymSZd/xxxCdA+9Kw1tWVFhHVmhhzqvVHF0R6ts8AsNgKgUsuMc+?=
 =?us-ascii?Q?98jMUeHWHdKKw6zkAY8WZylTpAmHE8/aFmlgqZR/1AQ0X4u+aYtDcCShA/Tb?=
 =?us-ascii?Q?fSNkLZFJan25jla7W2Oo2W2xNAAL+L1bde7ctH939+JanmQR4VTy6BzWzhWc?=
 =?us-ascii?Q?INqWGhjYl6PtB6A92eAr+JZf/NwMA49GdgYxQ+MSBog5jCLz+3ktmUC1FZ3q?=
 =?us-ascii?Q?wg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1448A7C7BB75C94C82C66B04E87967E9@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 49F0ej0zhnMvKGj13jpS3TyAeRmACAMn4Z4tWfThuVBMNwOyLxuiB69WXyfX2cOQiL755MK51pR/8XOescnkInG/f4KAWXXw97iZxqDiUs3nc56P8RA1g9t8ncTvTa3VgBuhiMHKwKf4oWFjhNxjNUwhRKTFlC6B9Pk+rB5FXs+LGMCFQQqJx6vdvIVMfyxmo5nkr/eQshsbb5fS0XL+LoGRKc+UEqML8FF4p8k9DZwZrratWu5h6t8B/4PU662UOK4qQJnFgQXRMEVs3061GKo+Xl5rYeEjdqdl7OYvB/+xksL5Wj+DqvMdG7z6w/4O5c7VDIan6wjMLHzJBB+OAL4CSkR/cWvmzyklm0Xy/46ywN/ZVqtGXKy/Ltk26Y24+dGVaVWhxSy2Bd7wiWR5AWV4Ivzfmtkm5OPVUwL50ICnHDSC2jq17r3Mg8Rbsc1n/zr4p4Tj/TggswxRDfX7tjlk4JePw1GFsqr7n4J2nCnfZfuq244bvmrgOz6c4uvPYdEckBA9hNw22qHSaWI8IlqSYcvM+knJgk1PzuA/s0Fvq6EsaapKUQP12hjx2HRtYi55NnT331YGT4ELQyhHhGOboJIOvnBmwaIrKXFIfiMPPmJ0GMERZE+3b5xjJdTsQ4uKnm0HawaAV1OUs6uA88CbgFPl1Vc+knZ5dUEZuGhQwh2g9/coLRxR52YrkERIFvoRLF8aWexQr6eXkev6E1PFS9j66ZFLHNMrqKW/89mau28oRLKOJxhE7kWETHu4ZTEk3XBNshvgTdAmr+HzXeBjjbAMvfp/PCxuwYKpGis=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b5598fb5-20bc-4f95-33e3-08db814c3fe3
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jul 2023 13:47:49.2119
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: fgn5h69+PhTvD8j6O4+rIFCxQjA5mIiVRhC3v3UU5bh3gAUTX4bMV7L5B5cJKVdCLMAyf0ccJNB9R0NZ0xKEhQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5087
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-10_10,2023-07-06_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=983
 adultscore=0 mlxscore=0 spamscore=0 phishscore=0 malwarescore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2307100124
X-Proofpoint-GUID: 3Df7iEI0nV_jXVMZfjZD6bWzxiEBkUvU
X-Proofpoint-ORIG-GUID: 3Df7iEI0nV_jXVMZfjZD6bWzxiEBkUvU
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 10, 2023, at 9:03 AM, Jeff Layton <jlayton@kernel.org> wrote:
>=20
> On Sun, 2023-07-09 at 11:45 -0400, Chuck Lever wrote:
>> This series optimizes DRC scalability by freeing cache objects only
>> once the hash bucket lock is no longer held. There are a couple of
>> related clean-ups to go along with this optimization.
>>=20
>=20
>=20
> The conventional wisdom that I've always heard is that a kfree under
> spinlock is generally no big deal. It can't block and is usually quite
> fast. Are you able to measure any performance delta from this set?

Yes, a couple of percent better throughput with a tmpfs export and
a fast transport, which is not a common use case, granted.

I think the difference is that, after this change, we're holding
each bucket lock for a shorter period of time. That's enough to
reduce some lock contention.

But I also like the shrinker improvements.


>> ---
>>=20
>> Chuck Lever (6):
>>      NFSD: Refactor nfsd_reply_cache_free_locked()
>>      NFSD: Rename nfsd_reply_cache_alloc()
>>      NFSD: Replace nfsd_prune_bucket()
>>      NFSD: Refactor the duplicate reply cache shrinker
>>      NFSD: Remove svc_rqst::rq_cacherep
>>      NFSD: Rename struct svc_cacherep
>>=20
>>=20
>> fs/nfsd/cache.h            |   8 +-
>> fs/nfsd/nfscache.c         | 203 ++++++++++++++++++++++++-------------
>> fs/nfsd/nfssvc.c           |  10 +-
>> fs/nfsd/trace.h            |  26 ++++-
>> include/linux/sunrpc/svc.h |   1 -
>> 5 files changed, 165 insertions(+), 83 deletions(-)
>>=20
>> --
>> Chuck Lever
>>=20
>=20
> --=20
> Jeff Layton <jlayton@kernel.org>


--
Chuck Lever