Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAOv1SKCmdtchm5Z2NU80o49tkrHpAkPFaHKj4-vLDN5bZNCz-Q@mail.gmail.com>
 <162846730406.22632.14734595494457390936@noble.neil.brown.name>
 <CAOv1SKBZ7sGBnvG9-M+De+s=CfU=H_GBs4hJah1E4ks+NSMyPw@mail.gmail.com>
 <CAOv1SKCUM5cGuXWAc7dsXtbmPMATqd245juC+S9gVXHWiZsvmQ@mail.gmail.com>
 <162855893202.12431.3423894387218130632@noble.neil.brown.name>
 <CAOv1SKAaSbfw53LWCCrvGCHESgdtCf5h275Zkzi9_uHkqnCrdg@mail.gmail.com> <162882238416.1695.4958036322575947783@noble.neil.brown.name>
In-Reply-To: <162882238416.1695.4958036322575947783@noble.neil.brown.name>
From:   Mike Javorski <mike.javorski@gmail.com>
Date:   Thu, 12 Aug 2021 19:53:00 -0700
Message-ID: <CAOv1SKB_dsam7P9pzzh_SKCtA8uE9cyFdJ=qquEfhLT42-szPA@mail.gmail.com>
Subject: Re: NFS server regression in kernel 5.13 (tested w/ 5.13.9)
To:     NeilBrown <neilb@suse.de>
Cc:     linux-nfs@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

The "semi-known-good" has been the client. I tried updating the server
multiple times to a 5.13 kernel and each time had to downgrade to the
last 5.12 kernel that ArchLinux released (5.12.15) to stabilize
performance. At each attempt, the client was running the same 5.13
kernel that was being deployed to the server. I never downgraded the
client.

Thank you for the scripts and all the details, I will test things out
this weekend when I can dedicate time to it.

- mike

On Thu, Aug 12, 2021 at 7:39 PM NeilBrown <neilb@suse.de> wrote:
>
> On Fri, 13 Aug 2021, Mike Javorski wrote:
> > Neil:
> >
> > Apologies for the delay, your message didn't get properly flagged in my email.
>
> :-)
>
> >
> > To answer your questions, both client (my Desktop PC) and server (my
> > NAS) are running ArchLinux; client w/ current kernel (5.13.9), server
> > w/ current or alternate testing kernels (see below).
>
> So the bug could be in the server or the client.  I assume you are
> careful to test a client against a know-good server, or a server against
> a known-good client.
>
> >                                                                 I
> > intend to spend some time this weekend attempting to get the tcpdump.
> > My initial attempts wound up with 400+Mb files which would be
> > difficult to ship and use for diagnostics.
>
> Rather than you sending me the dump, I'll send you the code.
>
> Run
>   tshark -r filename -d tcp.port==2049,rpc -Y 'tcp.port==2049 && rpc.time > 1'
>
> This will ensure the NFS traffic is actually decoded as NFS and then
> report only NFS(rpc) replies that arrive more than 1 second after the
> request.
> You can add
>
>     -T fields -e frame.number -e rpc.time
>
> to find out what the actual delay was.
>
> If it reports any, that will be interesting.  Try with a larger time if
> necessary to get a modest number of hits.  Using editcap and the given
> frame number you can select out 1000 packets either side of the problem
> and that should compress to be small enough to transport.
>
> However it might not find anything.  If the reply never arrives, you'll
> never get a reply with a long timeout.  So we need to check that
> everything got a reply...
>
>  tshark -r filename -t tcp.port==2049,rpc  \
>    -Y 'tcp.port==2049 && rpc.msg == 0' -T fields \
>    -e rpc.xid -e frame.number | sort > /tmp/requests
>
>  tshark -r filename -t tcp.port==2049,rpc  \
>    -Y 'tcp.port==2049 && rpc.msg == 1' -T fields \
>    -e rpc.xid -e frame.number | sort > /tmp/replies
>
>  join -a1 /tmp/requests /tmp/replies | awk 'NF==2'
>
> This should list the xid and frame number of all requests that didn't
> get a reply.  Again, editcap can extract a range of frames into a file of
> manageable size.
>
> Another possibility is that requests are getting replies, but the reply
> says "NFS4ERR_DELAY"
>
>  tshark -r filename -t tcp.port==2049,rpc -Y nfs.nfsstat4==10008
>
> should report any reply with that error code.
>
> Hopefully something there will be interesting.
>
> NeilBrown