Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAOv1SKCmdtchm5Z2NU80o49tkrHpAkPFaHKj4-vLDN5bZNCz-Q@mail.gmail.com>
 <162846730406.22632.14734595494457390936@noble.neil.brown.name>
In-Reply-To: <162846730406.22632.14734595494457390936@noble.neil.brown.name>
From:   Mike Javorski <mike.javorski@gmail.com>
Date:   Sun, 8 Aug 2021 17:28:44 -0700
Message-ID: <CAOv1SKBZ7sGBnvG9-M+De+s=CfU=H_GBs4hJah1E4ks+NSMyPw@mail.gmail.com>
Subject: Re: NFS server regression in kernel 5.13 (tested w/ 5.13.9)
To:     NeilBrown <neilb@suse.de>
Cc:     linux-nfs@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Appreciate the insights Neil. I will make an attempt to
trigger/capture the fault if I can.

- mike

On Sun, Aug 8, 2021 at 5:01 PM NeilBrown <neilb@suse.de> wrote:
>
> On Mon, 09 Aug 2021, Mike Javorski wrote:
> > I have been experiencing nfs file access hangs with multiple release
> > versions of the 5.13.x linux kernel. In each case, all file transfers
> > freeze for 5-10 seconds and then resume. This seems worse when reading
> > through many files sequentially.
>
> A particularly useful debugging tool for NFS freezes is to run
>
>   rpcdebug -m rpc -c all
>
> while the system appears frozen.  As you only have a 5-10 second window
> this might be tricky.
> Setting or clearing debug flags in the rpc module (whether they are
> already set or not) has a side effect if listing all RPC "tasks" which a
> waiting for a reply.  Seeing that task list can often be useful.
>
> The task list appears in "dmesg" output.  If there are not tasks
> waiting, nothing will be written which might lead you to think it didn't
> work.
>
> As Chuck hinted, tcpdump is invaluable for this sort of problem.
>   tcpdump -s 0 -w /tmp/somefile.pcap port 2049
>
> will capture NFS traffic.  If this can start before a hang, and finish
> after, it may contain useful information.  Doing that in a way that
> doesn't create an enormous file might be a challenge.  It would help if
> you found a way trigger the problem.  Take note of the circumstances
> when it seems to happen the most.  If you can only produce a large file,
> we can probably still work with it.
>   tshark -r /tmp/somefile.pcap
> will report the capture one line per packet.  You can look for the
> appropriate timestamp, note the frame numbers, and use "editcap"
> to extract a suitable range of packets.
>
> NeilBrown
>
>
> >
> > My server:
> > - Archlinux w/ a distribution provided kernel package
> > - filesystems exported with "rw,sync,no_subtree_check,insecure" options
> >
> > Client:
> > - Archlinux w/ latest distribution provided kernel (5.13.9-arch1-1 at writing)
> > - nfs mounted via /net autofs with "soft,nodev,nosuid" options
> > (ver=4.2 is indicated in mount)
> >
> > I have tried the 5.13.x kernel several times since the first arch
> > release (most recently with 5.13.9-arch1-1), all with similar results.
> > Each time, I am forced to downgrade the linux package to a 5.12.x
> > kernel (5.12.15-arch1 as of writing) to clear up the transfer issues
> > and stabilize performance. No other changes are made between tests. I
> > have confirmed the freezing behavior using both ext4 and btrfs
> > filesystems exported from this server.
> >
> > At this point I would appreciate some guidance in what to provide in
> > order to diagnose and resolve this issue. I don't have a lot of kernel
> > debugging experience, so instruction would be helpful.
> >
> > - mike
> >
> >