Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Timo Rothenpieler <timo@rothenpieler.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgA
Date:   Mon, 17 May 2021 16:27:29 +0000
Message-ID: <72ECF9E1-1F6E-44AF-850C-536BED898DDD@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
In-Reply-To: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4BED40FAF015D84A8849A3BC764A6702@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 098cfa39-0491-466a-fc8e-08d91950aa38
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 May 2021 16:27:29.1982
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Mn1xZlKX7gagQbKmCAs0lZzWNM+Un4MRhGpoL6oQtnCxOtre4rtCjaPcDLtdn/7VQPxUnsR4N3Dgo6R+pn8oAQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3858
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9987 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 mlxlogscore=999
 mlxscore=0 adultscore=0 malwarescore=0 suspectscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2105170112
X-Proofpoint-GUID: cHUM_DF5WyaNyN-UWV00_vzVVVritOC7
X-Proofpoint-ORIG-GUID: cHUM_DF5WyaNyN-UWV00_vzVVVritOC7
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9987 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 impostorscore=0 mlxscore=0
 mlxlogscore=999 adultscore=0 malwarescore=0 priorityscore=1501
 phishscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 clxscore=1011
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2104190000 definitions=main-2105170112
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hello Timo-

> On May 16, 2021, at 1:29 PM, Timo Rothenpieler <timo@rothenpieler.org> wr=
ote:
>=20
> This has happened 3 times so far over the last couple months, and I do no=
t have a clear way to reproduce it.
> It happens under moderate load, when lots of nodes read and write from th=
e server. Though not in any super intense way. Just normal program executio=
n, writing of light logs, and other standard tasks.
>=20
> The issues on the clients manifest in a multitude of ways. Most of the ti=
me, random IO operations just fail, rarely hang indefinitely and make the p=
rocess unkillable.
> Another example would be: "Failed to remove '.../.nfs00000000007b03af0000=
0001': Device or resource busy"
>=20
> Once a client is in that state, the only way to get it back into order is=
 a reboot.
>=20
> On the server side, a single error cqe is dumped each time this problem h=
appened. So far, I always rebooted the server as well, to make sure everyth=
ing is back in order. Not sure if that is strictly necessary.
>=20
>> [561889.198889] infiniband mlx5_0: dump_cqe:272:(pid 709): dump error cq=
e
>> [561889.198945] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0=
0
>> [561889.198984] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0=
0
>> [561889.199023] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0=
0
>> [561889.199061] 00000030: 00 00 00 00 00 00 88 13 08 00 01 13 07 47 67 d=
2
>=20
>> [985074.602880] infiniband mlx5_0: dump_cqe:272:(pid 599): dump error cq=
e
>> [985074.602921] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0=
0
>> [985074.602946] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0=
0
>> [985074.602970] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0=
0
>> [985074.602994] 00000030: 00 00 00 00 00 00 88 13 08 00 01 46 f2 93 0b d=
3
>=20
>> [1648894.168819] infiniband ibp1s0: dump_cqe:272:(pid 696): dump error c=
qe
>> [1648894.168853] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00
>> [1648894.168878] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00
>> [1648894.168903] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00
>> [1648894.168928] 00000030: 00 00 00 00 00 00 88 13 08 00 01 08 6b d2 b9 =
d3

I'm hoping Leon can get out his magic decoder ring and tell us if
these CQE dumps contain a useful WC status code.

Meanwhile, you could try 5.11 or 5.12 on your NFS server to see if
the problem persists.


> These all happened under different Versions of the 5.10 Kernel. The last =
one under 5.10.32 today.
>=20
> Switching all clients to TCP seems to make NFS works perfectly reliable.
>=20
> I'm not sure how to read those error dumps, so help there would be apprec=
iated.
>=20
> Could this be similar to spurious issues you get with UDP, where dropped =
packages cause havoc? Though I would not expect heavy load on IB to cause a=
n error cqe to be logged.

The underlying RDMA connection for RPC/RDMA is reliable (lossless),
so I don't see an immediate relation to how RPC/UDP might behave.


--
Chuck Lever