Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: NFS over RDMA issues on Linux 5.4
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <20200804134642.GC4432@unreal>
Date:   Tue, 4 Aug 2020 09:53:08 -0400
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <45BA86D8-52A3-407E-83BE-27343C0182C5@oracle.com>
References: <8a1087d3-9add-dfe1-da0c-edab74fcca51@rothenpieler.org>
 <CE6C02CE-3EEB-4834-B499-376BC6020A17@oracle.com>
 <20200804093635.GA4432@unreal>
 <92a5a932-b843-eed3-555e-7557ccc1f308@rothenpieler.org>
 <20200804122557.GB4432@unreal>
 <DAF6EFDA-5863-4887-B495-0BE3CA714209@oracle.com>
 <d41ac40e-8974-0a44-2b9f-bede74619935@rothenpieler.org>
 <F6BDB1B6-6315-47C9-A589-F4A57E25B2C5@oracle.com>
 <20200804134642.GC4432@unreal>
To:     Leon Romanovsky <leon@kernel.org>,
        Timo Rothenpieler <timo@rothenpieler.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Aug 4, 2020, at 9:46 AM, Leon Romanovsky <leon@kernel.org> wrote:
>=20
> On Tue, Aug 04, 2020 at 09:12:55AM -0400, Chuck Lever wrote:
>>=20
>>=20
>>> On Aug 4, 2020, at 9:08 AM, Timo Rothenpieler =
<timo@rothenpieler.org> wrote:
>>>=20
>>> On 04.08.2020 14:49, Chuck Lever wrote:
>>>> Timo, I tend to think this is not a configuration issue.
>>>> Do you know of a known working kernel?
>>>=20
>>> This is a brand new system, it's never been running with any kernel =
older than 5.4, and downgrading it to 4.19 or something else while in =
operation is unfortunately not easily possible. For a client it would =
definitely not be out of the question, but the main nfs server I cannot =
easily downgrade.
>>>=20
>>> Also keep in mind that the dmesg spam happens on both server and =
client simultaneously.
>>=20
>> Let's start with the client only, since restarting it seems to clear =
the problem.
>=20
> It is client because according to the server CQE errors, it is =
Remote_Invalid_Request_Error
> with "9.7.5.2.2 NAK CODES" from IBTA.

Thanks! OK, then let's use ftrace.

Timo, can you install trace-cmd on your client? Then:

1. # trace-cmd record -e rpcrdma -e sunrpc

2. Trigger the problem

3. Control-C the trace-cmd, and copy the trace.dat file to another =
system

4. reboot your client

Then send me your trace.dat. You don't have to cc the mailing lists.


--
Chuck Lever