Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
From:   "NeilBrown" <neilb@suse.de>
To:     "Dave Chinner" <david@fromorbit.com>,
        Trond Myklebust <trondmy@hammerspace.com>
Cc:     "J. Bruce Fields" <bfields@fieldses.org>,
        "Chuck Lever" <chuck.lever@oracle.com>, linux-nfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org
Subject: Re: sporadic hangs on generic/186
In-reply-to: <20220407001453.GE1609613@dread.disaster.area>
References: <20220406195424.GA1242@fieldses.org>,
 <20220407001453.GE1609613@dread.disaster.area>
Date:   Thu, 07 Apr 2022 10:27:41 +1000
Message-id: <164929126156.10985.11316778982526844125@noble.neil.brown.name>
Precedence: bulk

On Thu, 07 Apr 2022, Dave Chinner wrote:
> On Wed, Apr 06, 2022 at 03:54:24PM -0400, J. Bruce Fields wrote:
> > In the last couple days I've started getting hangs on xfstests
> > generic/186 on upstream.  I also notice the test completes after 10+
> > hours (usually it takes about 5 minutes).  Sometimes this is accompanied
> > by "nfs: RPC call returned error 12" on the client.
> 
> #define ENOMEM          12      /* Out of memory */
> 
> So either the client or the server is running out of memory
> somewhere?

Probably the client.  There are a bunch of changes recently which add
__GFP_NORETRY to memory allocations from PF_WQ_WORKERs because that can
result in deadlocks when swapping over NFS.
This means that kmalloc request that previously never failed (because
GFP_KERNEL never fails for kernel threads I think) can now fail.  This
has tickled one bug that I know of.  There are likely to be more.

The RPC code should simply retry these allocations after a short delay. 
HZ/4 is the number that is used in a couple of places.  Possibly there
are more places that need to handle -ENOMEM with rpc_delay().

NeilBrown