Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: still seeing single client NFS4ERR_DELAY / CB_RECALL
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <20200810190729.GB13266@fieldses.org>
Date:   Mon, 10 Aug 2020 16:01:00 -0400
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <00CAA5B7-418E-4AB5-AE08-FE2F87B06795@oracle.com>
References: <139C6BD7-4052-4510-B966-214ED3E69D61@oracle.com>
 <20200809202739.GA29574@fieldses.org> <20200809212531.GB29574@fieldses.org>
 <227E18E8-5A45-47E3-981C-549042AFB391@oracle.com>
 <20200810190729.GB13266@fieldses.org>
To:     Bruce Fields <bfields@fieldses.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Aug 10, 2020, at 3:07 PM, Bruce Fields <bfields@fieldses.org> =
wrote:
>=20
> Thanks for the test results:
>=20
> On Mon, Aug 10, 2020 at 02:21:34PM -0400, Chuck Lever wrote:
>> For these results I've switched to sec=3Dsys so the test completes =
faster.
>>=20
>> NFSv3/sys: 953.37user 5101.96system 14:13.78elapsed 709%CPU =
(0avgtext+0avgdata 107160maxresident)k
>>=20
>> NFSv4.1/sys: 953.64user 5202.27system 17:54.51elapsed 572%CPU =
(0avgtext+0avgdata 107204maxresident)k
>>=20
>> NFSv4.0/sys unpatched: 965.44user 5406.75system 36:10.72elapsed =
293%CPU (0avgtext+0avgdata 107252maxresident)k
>>=20
>> NFSv4.0/sys with fix: 968.38user 5359.18system 30:50.38elapsed =
341%CPU (0avgtext+0avgdata 107140maxresident)k
>=20
> Well, that didn't work!
>=20
> So maybe it's write opens that are the problem in this case.  The =
below
> should mostly revert to pre-94415b06eb8a behavior in the 4.0 case, so =
if
> this doesn't fix it then I was wrong about the cause....
>=20
> --b.
>=20
> commit 0e94ee0b6f11
> Author: J. Bruce Fields <bfields@redhat.com>
> Date:   Sun Aug 9 17:11:59 2020 -0400
>=20
>    nfsd4: don't grant delegations on 4.0 create opens
>=20
>    Chuck reported a major slowdown running the git regression suite =
over
>    NFSv4.0.
>=20
>    In the 4.0 case, the server has no way to identify which client =
most
>    metadata-modifying operations come from.  So, for example, the =
common
>    pattern of an create or write open followed by a setattr is likely =
to
>    result in an immediate break in the 4.0 case.
>=20
>    It's probably not worth giving out delegations on 4.0 write or =
create
>    opens.
>=20
>    Reported-by: Chuck Lever <chuck.lever@oracle.com>
>    Signed-off-by: J. Bruce Fields <bfields@redhat.com>
>=20
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index fdba971d06c3..0d51d1751592 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -5096,6 +5096,19 @@ nfs4_open_delegation(struct svc_fh *fh, struct =
nfsd4_open *open,
> 				goto out_no_deleg;
> 			if (!cb_up || !(oo->oo_flags & =
NFS4_OO_CONFIRMED))
> 				goto out_no_deleg;
> +			if (clp->cl_minorversion)
> +				break;
> +			/*
> +			 * In the absence of sessions, most operations
> +			 * that modify metadata (like setattr) can't
> +			 * be linked to the client sending them, so
> +			 * will result in a delegation break.  That's
> +			 * especially likely for write and create opens:
> +			 */
> +			if (open->op_share_access & =
NFS4_SHARE_ACCESS_WRITE)
> +				goto out_no_deleg;
> +			if (open->op_create =3D=3D NFS4_OPEN_CREATE)
> +				goto out_no_deleg;
> 			break;
> 		default:
> 			goto out_no_deleg;

Roughly the same result with this patch as with the first one. The
first one is a little better. Plus, I think the Solaris NFS server
hands out write delegations on v4.0, and I haven't heard of a
significant issue there. It's heuristics may be different, though.

So, it might be that NFSv4.0 has always run significantly slower. I
will have to try a v5.4 or older server to see.

Also, instead of timing, I should count forward channel RPCs and
callbacks, or perhaps the number of DELAY responses.

Don't touch that dial!


--
Chuck Lever