Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: NFS over RDMA issues on Linux 5.4
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <8a1087d3-9add-dfe1-da0c-edab74fcca51@rothenpieler.org>
Date:   Mon, 3 Aug 2020 12:24:21 -0400
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <CE6C02CE-3EEB-4834-B499-376BC6020A17@oracle.com>
References: <8a1087d3-9add-dfe1-da0c-edab74fcca51@rothenpieler.org>
To:     Timo Rothenpieler <timo@rothenpieler.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

Hi Timo-

> On Aug 3, 2020, at 11:05 AM, Timo Rothenpieler <timo@rothenpieler.org> =
wrote:
>=20
> Hello,
>=20
> I have just deployed a new system with Mellanox ConnectX-4 VPI EDR IB =
cards and wanted to setup NFS over RDMA on it.
>=20
> However, while mounting the FS over RDMA works fine, actually using it =
results in the following messages absolutely hammering dmesg on both =
client and server:
>=20
>> =
https://gist.github.com/BtbN/9582e597b6581f552fa15982b0285b80#file-server-=
log
>=20
> The spam only stops once I forcibly reboot the client. The filesystem =
gets nowhere during all this. The retrans counter in nfsstat just keeps =
going up, nothing actually gets done.
>=20
> This is on Linux 5.4.54, using nfs-utils 2.4.3.
> The mlx5 driver had enhanced-mode disabled in order to enable IPoIB =
connected mode with an MTU of 65520.
>=20
> Normal NFS 4.2 over tcp works perfectly fine on this setup, it's only =
when I mount via rdma that things go wrong.
>=20
> Is this an issue on my end, or did I run into a bug somewhere here?
> Any pointers, patches and solutions to test are welcome.

I haven't seen that failure mode here, so best I can recommend is
keep investigating. I've copied linux-rdma in case they have any
advice.

--
Chuck Lever