Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 11 May 2022 14:38:05 +0200
From:   Greg KH <gregkh@linuxfoundation.org>
To:     Wolfgang Walter <linux@stwm.de>
Cc:     stable@vger.kernel.org, Trond Myklebust <trondmy@gmail.com>,
        Chuck Lever <chuck.lever@oracle.com>,
        linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: 5.4.188 and later: massive performance regression with nfsd
Message-ID: <YnuuLZe6h80KCNhd@kroah.com>
References: <f8d9b9112607df4807fba8948ac6e145@stwm.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f8d9b9112607df4807fba8948ac6e145@stwm.de>
Precedence: bulk

On Wed, May 11, 2022 at 12:03:13PM +0200, Wolfgang Walter wrote:
> Hi,
> 
> starting with 5.4.188 wie see a massive performance regression on our
> nfs-server. It basically is serving requests very very slowly with cpu
> utilization of 100% (with 5.4.187 and earlier it is 10%) so that it is
> unusable as a fileserver.
> 
> The culprit are commits (or one of it):
> 
> c32f1041382a88b17da5736886da4a492353a1bb "nfsd: cleanup
> nfsd_file_lru_dispose()"
> 628adfa21815f74c04724abc85847f24b5dd1645 "nfsd: Containerise filecache
> laundrette"
> 
> (upstream 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63 and
> 9542e6a643fc69d528dfb3303f145719c61d3050)
> 
> If I revert them in v5.4.192 the kernel works as before and performance is
> ok again.
> 
> I did not try to revert them one by one as any disruption of our nfs-server
> is a severe problem for us and I'm not sure if they are related.
> 
> 5.10 and 5.15 both always performed very badly on our nfs-server in a
> similar way so we were stuck with 5.4.
> 
> I now think this is because of 36ebbdb96b694dd9c6b25ad98f2bbd263d022b63
> and/or 9542e6a643fc69d528dfb3303f145719c61d3050 though I didn't tried to
> revert them in 5.15 yet.

Odds are 5.18-rc6 is also a problem?

If so, I'll just wait for the fix to get into Linus's tree as this does
not seem to be a stable-tree-only issue.

thanks,

greg k-h