Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: still seeing single client NFS4ERR_DELAY / CB_RECALL
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <20200809212531.GB29574@fieldses.org>
Date:   Mon, 10 Aug 2020 14:21:34 -0400
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <227E18E8-5A45-47E3-981C-549042AFB391@oracle.com>
References: <139C6BD7-4052-4510-B966-214ED3E69D61@oracle.com>
 <20200809202739.GA29574@fieldses.org> <20200809212531.GB29574@fieldses.org>
To:     Bruce Fields <bfields@fieldses.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Aug 9, 2020, at 5:25 PM, Bruce Fields <bfields@fieldses.org> wrote:
>=20
> On Sun, Aug 09, 2020 at 04:27:39PM -0400, Bruce Fields wrote:
>> On Sun, Aug 09, 2020 at 01:11:36PM -0400, Chuck Lever wrote:
>>> Hi Bruce-
>>>=20
>>> I noticed that one of my tests takes about 4x longer on NFSv4.0 than
>>> it does on NFSv3 or NFSv4.[12]. As an initial stab at the cause, I'm
>>> seeing lots of these:
>>=20
>> Oops, looks obvious in retrospect, but I didn't think of it.
>>=20
>> In the 4.1+ case, sessions mean that we know which client every
>> operation comes from.
>>=20
>> In the 4.0 case that only works for operations that take a stateid or
>> something else we can link back to a client.
>>=20
>> That means in the 4.0 case delegations are less useful, since they =
have
>> to be broken on any (non-truncating) setattr, any link/unlink, etc.,
>> even if the operation comes from the same client--the server doesn't
>> have a way to know that.
>>=20
>> Maybe the change to give out read delegations even on write opens
>> probably just isn't worth in the 4.0 case.
>=20
> Untested, but maybe this?--b.
>=20
> commit 2102ac0b68f3
> Author: J. Bruce Fields <bfields@redhat.com>
> Date:   Sun Aug 9 17:11:59 2020 -0400
>=20
>    nfsd4: don't grant delegations on 4.0 create opens
>=20
>    Chuck reported a major slowdown running the git regression suite =
over
>    NFSv4.0.
>=20
>    In the 4.0 case, the server has no way to identify which client =
most
>    metadata-modifying operations come from.  So, for example, the =
common
>    pattern of a a setattr is likely to result in an immediate break in =
the
>    4.0 case.
>=20
>    It's probably not worth giving out delegations on 4.0 creates.
>=20
>    Reported-by: Chuck Lever <chuck.lever@oracle.com>
>    Signed-off-by: J. Bruce Fields <bfields@redhat.com>
>=20
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index fdba971d06c3..ce2d052b3f64 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -5096,6 +5096,16 @@ nfs4_open_delegation(struct svc_fh *fh, struct =
nfsd4_open *open,
> 				goto out_no_deleg;
> 			if (!cb_up || !(oo->oo_flags & =
NFS4_OO_CONFIRMED))
> 				goto out_no_deleg;
> +			/*
> +			 * In the absence of sessions, most operations
> +			 * that modify metadata (like setattr) can't
> +			 * be linked to the client sending them, so
> +			 * will result in a delegation break.  That's
> +			 * especially likely for create opens:
> +			 */
> +			if (clp->cl_minorversion =3D=3D 0 &&
> +					open->op_create =3D=3D =
NFS4_OPEN_CREATE)
> +				goto out_no_deleg;
> 			break;
> 		default:
> 			goto out_no_deleg;


For these results I've switched to sec=3Dsys so the test completes =
faster.

NFSv3/sys: 953.37user 5101.96system 14:13.78elapsed 709%CPU =
(0avgtext+0avgdata 107160maxresident)k

NFSv4.1/sys: 953.64user 5202.27system 17:54.51elapsed 572%CPU =
(0avgtext+0avgdata 107204maxresident)k


NFSv4.0/sys unpatched: 965.44user 5406.75system 36:10.72elapsed 293%CPU =
(0avgtext+0avgdata 107252maxresident)k

NFSv4.0/sys with fix: 968.38user 5359.18system 30:50.38elapsed 341%CPU =
(0avgtext+0avgdata 107140maxresident)k


--
Chuck Lever