Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: Adventures in NFS re-exporting
Date:   Mon, 28 Sep 2020 15:49:49 +0000
From:   Frank van der Linden <fllinden@amazon.com>
To:     Daire Byrne <daire@dneg.com>
CC:     Chuck Lever <chuck.lever@oracle.com>,
        bfields <bfields@fieldses.org>,
        linux-nfs <linux-nfs@vger.kernel.org>,
        linux-cachefs <linux-cachefs@redhat.com>
Message-ID: <20200928154949.GA14702@dev-dsk-fllinden-2c-c1893d73.us-west-2.amazon.com>
References: <943482310.31162206.1599499860595.JavaMail.zimbra@dneg.com>
 <20200915172140.GA32632@fieldses.org>
 <2001715792.39705019.1600358470997.JavaMail.zimbra@dneg.com>
 <20200917190931.GA6858@fieldses.org>
 <20200917202303.GA29892@dev-dsk-fllinden-2c-c1893d73.us-west-2.amazon.com>
 <76A4DC7D-D4F7-4A17-A67D-282E8522132A@oracle.com>
 <1790619463.44171163.1600892707423.JavaMail.zimbra@dneg.com>
 <20200923210157.GA1650@dev-dsk-fllinden-2c-c1893d73.us-west-2.amazon.com>
 <108670779.52656705.1601110822013.JavaMail.zimbra@dneg.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <108670779.52656705.1601110822013.JavaMail.zimbra@dneg.com>
User-Agent: Mutt/1.11.4 (2019-03-13)
Precedence: bulk

On Sat, Sep 26, 2020 at 10:00:22AM +0100, Daire Byrne wrote:
> 
> 
> ----- On 23 Sep, 2020, at 22:01, Frank van der Linden fllinden@amazon.com wrote:
> > It's entirely possible that my patch introduces a refcounting error - it was
> > intended as a proof-of-concept on how to fix the LRU locking issue for v4
> > open file caching (while keeping it enabled) - which is why I didn't
> > "formally" send it in.
> >
> > Having said that, I don't immediately see the problem.
> >
> > Maybe try it without the rhashtable patch, that is much less of an
> > optimization.
> >
> > The problem would have to be nf_ref as part of nfsd_file, or fi_ref as part
> > of nfs4_file. If it's the latter, it's probably the rhashtable change.
> 
> Thanks Frank; I think you are right in that it seems to be a problem with the rhashtable patch. Another 48 hours using the same workload with just the main patch and I have not seen the same issue again so far.
> 
> Also, it still has the effect of reducing the CPU usage dramatically such that there are plenty of cores still left idle. This is actually helping us buy some more time while we fix our obviously broken software so that it doesn't open/close so crazily.
> 
> So, many thanks for that.

Cool. I'm glad the "don't put v4 files on the LRU list" works as intended for
you. The rhashtable patch was more of an afterthought, and obviously has an
issue. It did provide some extra gains, so I'll see if I can find the problem
if I get some time.

Bruce - if you want me to 'formally' submit a version of the patch, let me
know. Just disabling the cache for v4, which comes down to reverting a few
commits, is probably simpler - I'd be able to test that too.

- Frank