Received-SPF: pass (google.com: domain of linux-nfs+bounces-3535-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <20240429152537.212958-6-cel@kernel.org> <efc4f895-3a45-4b44-a47d-532896526458@linux.dev>
 <90F6A893-5315-4E53-B54E-1CF8D7D4AC4D@oracle.com> <f49907ea-cedc-44b7-9ffc-30c265731f3e@linux.dev>
 <675A3584-6086-45D4-9B31-F7F572394144@oracle.com> <6d483d75-5866-4c4e-8d86-c89a2b27f5e7@linux.dev>
 <F2726F77-06F9-4DB4-B2A8-97F21B045A6F@oracle.com> <CAD=hENdL3v6gMpU7SBdkLjcyuEhzCvTRxt3+N+8m6ybuVKGHwA@mail.gmail.com>
 <953940CF-98DE-4727-8E32-066CC4B3E8B2@oracle.com>
In-Reply-To: <953940CF-98DE-4727-8E32-066CC4B3E8B2@oracle.com>
From: Zhu Yanjun <zyjzyj2000@gmail.com>
Date: Mon, 3 Jun 2024 18:54:11 +0200
Message-ID: <CAD=hENdTMVXB-3w3j3WDMBjUcuE7eRWZ1itZcc8==szWCe4KbQ@mail.gmail.com>
Subject: Re: [RFC PATCH 0/4] NFS: Fix another 'check_flush_dependency' splat
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: Linux NFS Mailing List <linux-nfs@vger.kernel.org>, 
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>, 
	"kdevops@lists.linux.dev" <kdevops@lists.linux.dev>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 3, 2024 at 5:59=E2=80=AFPM Chuck Lever III <chuck.lever@oracle.=
com> wrote:
>
>
>
> > On Jun 2, 2024, at 2:14=E2=80=AFPM, Zhu Yanjun <zyjzyj2000@gmail.com> w=
rote:
> >
> > On Sun, Jun 2, 2024 at 5:40=E2=80=AFPM Chuck Lever III <chuck.lever@ora=
cle.com> wrote:
> >>
> >>
> >>> On Apr 30, 2024, at 10:45=E2=80=AFAM, Zhu Yanjun <zyjzyj2000@gmail.co=
m> wrote:
> >>>
> >>> On 30.04.24 16:13, Chuck Lever III wrote:
> >>>> It is possible to add rxe as a second option in kdevops,
> >>>> but siw has worked for our purposes so far, and the NFS
> >>>> test matrix is already enormous.
> >>>
> >>> Thanks. If rxe can be as a second option in kdevops, I will make test=
s with kdevops to check rxe work well or not in the future kernel version.
> >>
> >> As per our recent discussion, I have added rxe as a second
> >> software RDMA option in kdevops. Proof of concept:
> >
> > Thanks a lot. I am very glad to know that rxe is treated as a second
> > software RDMA option in kdeops.
> > And I also checked the commit related with this feature. It is very
> > complicated and huge.
>
> I split this into four smaller patches, HTH.
>
>
> > I hope rxe can work well in kdeops.
> > So I can also use kdeops to verify rxe and rdma subsystems.  Thanks a
> > lot your efforts.
> >
> >>
> >>  https://github.com/chucklever/kdevops/tree/add-rxe-support
> >>
> >> But basic rping testing is not working (with 6.10-rc1 kernels)
> >> in this set-up. It's missing something...
> >
> > Just now I made tests with the latest rdma-core (rping is included in
> > rdma-core) and 6.10-rc1 kernels. rping can work well.
> >
> > Normally rping works as a basic tool to verify if rxe works well or
> > not.  If rping can not work well, normally I will do the followings:
> > 1. rping -s -a 127.0.0.1
> >    rping -c -a 127.0.0.1 -C 3 -d -v
> >    This will verify whether rxe is configured correctly or not.
>
> I don't have rxe set up on loopback, so I substituted the host's
> configured Ethernet IP.
>
> The tests works on the NFS server, but the rping client hangs
> on the NFS client (both running v6.10-rc1).
>
> I rebooted in to the Fedora 39 stock kernel, and the rping tests
> pass.
>
> However, when I try to run fstests with NFS/RDMA using rxe, the
> client kernel reports a soft CPU lock-up, and top shows this:
>
>     115 root      20   0       0      0      0 R  99.3   0.0   1:03.50 kw=
orker/u8:5+rxe_wq

rxe_wq is introduced in the commit 9b4b7c1f9f54 "RDMA/rxe: Add
workqueue support for rxe tasks".
And this commit is merged into kernel v6.4-rc2-1-g9b4b7c1f9f54.

And the Fedora 39 stock kernel is kernel 6.5. So maybe some commits
between 6.5 and 6.10 introduce this problem.

>
> So I think this is enough to show that the Ansible parts of this
> change are working as expected. I can push this to kdevops now
> if there are no objections, and someone (maybe you, maybe me) can
> sort out the rxe specific issues later.

Thanks. After I can reproduce this problem in my local host, I am very
glad to delve into this problem. Perhaps it will take me a long time
since I do not have a good host to deploy kdeops.

To be honest, perhaps "git bisec" can find the commit that introduce
this problem. If you can find the commit, we can fix this problem very
quickly^_^

Thanks,
Zhu Yanjun

>
>
> > 2. ping -c 3 server_ip on client host.
> >    This will verify whether the client host can connect to the server
> > host or not.
> > 3. rping -s -a server_ip
> >    rping -c -a server_ip -C 3 -d -v
> >    1) shutdown firewall
> >    2) tcpdump -ni xxxx to capture udp packets
> > Normally the above steps can find out the errors in rxe client/server.
> > Hope the above can help to find out the errors.
> >
> > Zhu Yanjun
> >
> >>
> >> --
> >> Chuck Lever
> >>
> >>
>
> --
> Chuck Lever
>
>