Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
From:   "NeilBrown" <neilb@suse.de>
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     "Dave Chinner" <david@fromorbit.com>,
        "Trond Myklebust" <trondmy@hammerspace.com>,
        "Chuck Lever" <chuck.lever@oracle.com>, linux-nfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org
Subject: Re: sporadic hangs on generic/186
In-reply-to: <20220407014939.GC1242@fieldses.org>
References: <20220406195424.GA1242@fieldses.org>,
 <20220407001453.GE1609613@dread.disaster.area>,
 <164929126156.10985.11316778982526844125@noble.neil.brown.name>,
 <164929437439.10985.5253499040284089154@noble.neil.brown.name>,
 <20220407014939.GC1242@fieldses.org>
Date:   Thu, 07 Apr 2022 14:23:00 +1000
Message-id: <164930538057.10985.13909676315036428067@noble.neil.brown.name>
Precedence: bulk

On Thu, 07 Apr 2022, J. Bruce Fields wrote:
> On Thu, Apr 07, 2022 at 11:19:34AM +1000, NeilBrown wrote:
> > I had a look through the various places where alloc can now fail.
> >=20
> > I think xdr_alloc_bvec() in xprt_sent_pagedata() is the most likely
> > cause of a problem here.  I don't think an -ENOMEM from there is caught,
> > so it could likely filter up to NFS and result in the message you got.
> >=20
> > I don't think we can easily handle failure there.  We need to stay with
> > GFP_KERNEL rely on PF_MEMALLOC to make forward progress for
> > swap-over-NFS.
> >=20
> > Bruce: can you change that one line back to GFP_KERNEL and see if the
> > problem goes away?
>=20
> Like this?  Sure--might take me a day or two to run the tests and get
> results back.--b.
>=20
> diff --git a/net/sunrpc/socklib.c b/net/sunrpc/socklib.c
> index 05b38bf68316..506627dc9a0f 100644
> --- a/net/sunrpc/socklib.c
> +++ b/net/sunrpc/socklib.c
> @@ -223,7 +223,7 @@ static int xprt_send_pagedata(struct socket *sock, stru=
ct msghdr *msg,
>  {
>  	int err;
> =20
> -	err =3D xdr_alloc_bvec(xdr, rpc_task_gfp_mask());
> +	err =3D xdr_alloc_bvec(xdr, GFP_KERNEL);
>  	if (err < 0)
>  		return err;
> =20
>=20

That looks right.

I instrumented my kernel to deliberately fail 10% of the time, and I got
lots of

nfs: RPC call returned error 12

so I'm fairly sure this explains that message.
But you say the hangs were only occasionally accompanied by the message,
so it probably doesn't explain the hangs.

NeilBrown