Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     "trondmy@kernel.org" <trondmy@kernel.org>
CC:     Bruce Fields <bfields@redhat.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH] nfsd: Reduce contention for the nfsd_file nf_rwsem
Thread-Topic: [PATCH] nfsd: Reduce contention for the nfsd_file nf_rwsem
Thread-Index: AQHXY9BFZBFmWOpSekG1W92QMIYbjKsaD6qA
Date:   Fri, 18 Jun 2021 17:59:13 +0000
Message-ID: <C7ADB3D9-D626-4DC2-91FD-E35D62605002@oracle.com>
References: <20210617232652.264884-1-trondmy@kernel.org>
In-Reply-To: <20210617232652.264884-1-trondmy@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?7KibRCn52SLuaroy5Hm3uJkCG2c6g6j/Rx3HlGG4pkICgY4MvdJ3RUxIWHWg?=
 =?us-ascii?Q?OaJo2u41T/lzdVoxcrRjZUfIt8zcn/qkpc7oDd+XsgkQ8++BVE0MacbR9L8t?=
 =?us-ascii?Q?EMHp7bMvpaKY6dIfKdrYsU6AT/KepKALYqaahCOwlT4yvwnieCztnWhFDEpD?=
 =?us-ascii?Q?V0dcP+rqBY+Xabi2kgnb9SU9XfIIspBNeFH4OIYibLyjRRinhSHB2DN6qJj5?=
 =?us-ascii?Q?6WV6tHMmQ24fKK1UStuv/2EeE3F8sccYPGyVorNLkU6QO31HyS6wsBbCvuwr?=
 =?us-ascii?Q?BZXnbKy3PgdNH0mv8iWC1gqI7PwrEgikEwUgmsdTD1hAemdmVLj/C2CauWbU?=
 =?us-ascii?Q?1OFpaC0Q7e94HYzzHjPpoi98CIFcxSjckkXYbBO62AvsVVcZI6E2IiXV3F9L?=
 =?us-ascii?Q?4tCYaMBHn3GvzdsW4eW9ACDmnG5NadpuN45FgX/l0IFFIHZB19ntUsUl/7dJ?=
 =?us-ascii?Q?Xzch+NzGWZAK/EGeAXXx5pfBdD1lqpcylc14oTSIfy2Xm5NTyEcAMwjSQlfe?=
 =?us-ascii?Q?jCPzBYjL0ML+b9b+v/vjp01pqcu1KIGpT1h/7rHLMzpSvk45nmDXjaf/7Npv?=
 =?us-ascii?Q?v3Kce9t9nrBo/9UdBqG3BdqaeI96ryuo/Z9QHtQOxdZiOOvbD5d91yXpcJAY?=
 =?us-ascii?Q?/oxrQZnPNt2lKDJBmgiCNMHCeankxFivB+LOof0Ljtc8CrVm5XWuHHMCZKMK?=
 =?us-ascii?Q?16Tz4TafosODyiG//RAhinP93PJ+oOO4UDI6/swj0KcvbEWFaLJkVG4FtrQO?=
 =?us-ascii?Q?S2WccU8fW+0DsvzazaxCvfAJ7PG6XrsNk2fDmt0/icmym0XbGz06dP0cwKPi?=
 =?us-ascii?Q?E4kPBksEhvlhZKoYeF0QgNTVisTVRL/8UnB89eQGWSdLEJ3Yz5es/czts7I2?=
 =?us-ascii?Q?4hvGpDNncyG8jsElHppM5JXwylMOLDy7VPx/EVbWBT/F8arIV0mE+VNn3OFA?=
 =?us-ascii?Q?jfjMknflEwcCeF3O3w2e2jvV/tL9wyAtqNoktnuPhY1KuFOWR4dEuaJ8wV+P?=
 =?us-ascii?Q?q9ZEvD+Ix9UmFi+7EVrTOFnkNQX8erOFAojMqzNLTsV3c0I/P84XBEi/c5A3?=
 =?us-ascii?Q?I7cBZqXCFLjPTgJW2L4WapM4NbqobbUudzRj99AMf1SXGaR40PkyyfkSIWg2?=
 =?us-ascii?Q?Rx70LAMhLxly7o8gOdZygWdmHcUKfZ2rsccrB4OoOx5jgqLnzaL8yS0J1Yw9?=
 =?us-ascii?Q?wXQolt0JhfzsL7RMNLkOxs81jY5hDC7Eq2NDTCCqo1bFxwFxYvDxBrqcVB7W?=
 =?us-ascii?Q?1MHv3g6ALBMUm335nuG1kkDtvQchXA/pe9BiVNcxorF+sm6luZd97ABN1gSm?=
 =?us-ascii?Q?0NrVa+VSIuZTZBt53ClfI7Jy?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <DE3C9A88BDD51D41A940344CD46FE828@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b9ff3914-1941-4532-b4d6-08d93282c85d
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jun 2021 17:59:13.7680
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: WdrFatVen5tyFzZR2PFGglXwxBb3++iVA9cfD7Qa7K+rOLElIAmYQ9LivIyDT2sPQAouWgdxt96re24CfsH2BA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2920
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10019 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0
 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 adultscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2104190000 definitions=main-2106180104
X-Proofpoint-ORIG-GUID: -0ZSoOuMII5RCgjLdZjTAyLUi67-KxUl
X-Proofpoint-GUID: -0ZSoOuMII5RCgjLdZjTAyLUi67-KxUl
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 17, 2021, at 7:26 PM, trondmy@kernel.org wrote:
>=20
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
>=20
> When flushing out the unstable file writes as part of a COMMIT call, try
> to perform most of of the data writes and waits outside the semaphore.
>=20
> This means that if the client is sending the COMMIT as part of a memory
> reclaim operation, then it can continue performing I/O, with contention
> for the lock occurring only once the data sync is finished.
>=20
> Fixes: 5011af4c698a ("nfsd: Fix stable writes")
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>

I'll toss this onto my server over the weekend for some testing.


> ---
> fs/nfsd/vfs.c | 18 ++++++++++++++++--
> 1 file changed, 16 insertions(+), 2 deletions(-)
>=20
> diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
> index 15adf1f6ab21..46485c04740d 100644
> --- a/fs/nfsd/vfs.c
> +++ b/fs/nfsd/vfs.c
> @@ -1123,6 +1123,19 @@ nfsd_write(struct svc_rqst *rqstp, struct svc_fh *=
fhp, loff_t offset,
> }
>=20
> #ifdef CONFIG_NFSD_V3
> +static int
> +nfsd_filemap_write_and_wait_range(struct nfsd_file *nf, loff_t offset,
> +				  loff_t end)
> +{
> +	struct address_space *mapping =3D nf->nf_file->f_mapping;
> +	int ret =3D filemap_fdatawrite_range(mapping, offset, end);
> +
> +	if (ret)
> +		return ret;
> +	filemap_fdatawait_range_keep_errors(mapping, offset, end);
> +	return 0;
> +}
> +
> /*
>  * Commit all pending writes to stable storage.
>  *
> @@ -1153,10 +1166,11 @@ nfsd_commit(struct svc_rqst *rqstp, struct svc_fh=
 *fhp,
> 	if (err)
> 		goto out;
> 	if (EX_ISSYNC(fhp->fh_export)) {
> -		int err2;
> +		int err2 =3D nfsd_filemap_write_and_wait_range(nf, offset, end);
>=20
> 		down_write(&nf->nf_rwsem);
> -		err2 =3D vfs_fsync_range(nf->nf_file, offset, end, 0);
> +		if (!err2)
> +			err2 =3D vfs_fsync_range(nf->nf_file, offset, end, 0);
> 		switch (err2) {
> 		case 0:
> 			nfsd_copy_boot_verifier(verf, net_generic(nf->nf_net,
> --=20
> 2.31.1
>=20

--
Chuck Lever