Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Timo Rothenpieler <timo@rothenpieler.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <aglo@umich.edu>,
        Dai Ngo <dai.ngo@oracle.com>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgAgAATb4CANuhZgIBOXZcAgAAVhICAADVLgIAASYeAgAAo1oCAAAOTXYAA6JWAgAADg4CAAANlAIAAA4eAgAAZh4CAABNpAIABnm2AgAX5FgCABmbVAIBtyicAgAA5xwCAADMmAIAADYKA
Date:   Fri, 29 Oct 2021 19:06:08 +0000
Message-ID: <855BA036-FAF3-404B-90B7-D95AF3A96567@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
 <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org>
 <a28b403e-42cf-3189-a4db-86d20da1b7aa@rothenpieler.org>
 <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com>
 <c8025457-7376-e1b7-bd6c-e5c6ee5d9ce7@rothenpieler.org>
 <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org>
 <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com>
 <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com>
 <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org>
 <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com>
 <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org>
 <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
 <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com>
 <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org>
 <716B2A38-9705-41D7-969B-665EF90156C7@oracle.com>
 <D55A1FA5-71D4-4D37-9B88-E1733BEDBE47@oracle.com>
 <60273c2e-e946-25fb-68af-975f793e73d2@rothenpieler.org>
 <2445C26E-7D96-4E77-8079-98B865CC4C57@oracle.com>
 <774c4839-0165-e660-bbc4-9a8814192f26@rothenpieler.org>
In-Reply-To: <774c4839-0165-e660-bbc4-9a8814192f26@rothenpieler.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Kkx0HhNVffRaQQudd86aipMyKku0+801YiLlN33zAK3DYv/btd2AAcL2SAn5?=
 =?us-ascii?Q?mOjw9zLZArXuTbErcnyU33W6U3dJ2v8AmCXsm0ktfN1UM9+ANU87MLGoZIaq?=
 =?us-ascii?Q?YYjlyi/oxsFbC07DwcIojpPTD/QW3xPBugOu4iUD/ODdaDuwVPR/Iue0Gu5i?=
 =?us-ascii?Q?tTM7yxk1Wl3+WoALx17QXLGQo5yMy9nNHicEGJgu2MkkXMfuio1T1v1MriAC?=
 =?us-ascii?Q?qxZ2j+TsiFP3xGmqhefibP/IGaRMMFXz5veVlRFs99+tcdx3mnmJ32hmqU4j?=
 =?us-ascii?Q?2+3XeVLi2ceREURcS1mixhZTWGei3slVBLNNaPH0yE1CiR09EaxVRhN9viqB?=
 =?us-ascii?Q?B7i0eizH2GUW9yO4gaD3Tcc1NBxhOOVRzfrT64J/xwWeqhIbLeaBsMDh8eix?=
 =?us-ascii?Q?9U0w4aQ4Suitvbn8WR8RsS5JzL8Tr0RVRvw6ksOVPbjUl2B9GNCew7SnsKUC?=
 =?us-ascii?Q?Dxo8+dvuSHSWvqS/8LCyfZt4myhZNuuKprYJUsIbplifB0rYIBzXrkYJ190H?=
 =?us-ascii?Q?SIeCyeHqQJwsKusX4QlJtAA2KcJOJ9lq4N586ZQaDMnDS7/uxZMj+EeCbjEE?=
 =?us-ascii?Q?oGjE3rJvXe9f5pBq55rclSmxBpO9THmiJ09ySYtqG2MhbgB/f0kerr+1ERzG?=
 =?us-ascii?Q?MQsCe+aBnmIFlrwO2jBNC90bkF3xLdLiw2cuWdP7PLO5CF8LU9i+h3mOrEQY?=
 =?us-ascii?Q?0RtSok95aaxZAQh51+N42W2veqIbS/Kd5O9tzTYH0Bx6fnGDz/Eej8SK81X5?=
 =?us-ascii?Q?faQhjiXdNkB0+nOnKczVuvtoxps91zpU2P2rG3nE2Hch38bgpaxZ+9CKzRJC?=
 =?us-ascii?Q?Y4RCwUHvBei4nS8YWGX6QjqWw+aZJffRvVGSxB0lyj7mR8Y7yWO6WjvBnvoa?=
 =?us-ascii?Q?ZLn7u50c49pVBZ/W5kjjBhciw+Fg5lINBI6vDNytoKRj+/dGTTcJAh1InZpm?=
 =?us-ascii?Q?/DzSMKsKtCEWMGaa+1WLoP4qAfB8Nazz8H+IygOK5j8xVoFZ+8mkVgm4O/gH?=
 =?us-ascii?Q?PdAXY7yTjWl/AmbIYPIWb/I7gBUauroQcz9p+ojvUxrIEhbNJVZKKQczWDUy?=
 =?us-ascii?Q?XiMs2uF3smOHXjwDNHijpBXTeyuPUkQjOX1upf80MSSAFbPKgLzIwL5Yhe7B?=
 =?us-ascii?Q?jy6Q69gVIS/jfSDghJoFPoMCWiwsQAifeS8KdM6VPTC4GDD7QIz2RfwbkTJS?=
 =?us-ascii?Q?k/4C5vqGgwcVk01QU/BK/or0mYnlYMtcfpSt51FI7O6ls+Y1WBj4zss6Or/u?=
 =?us-ascii?Q?5qJ+2rxuCYiqUcB1K4kFgkxbfmefSFtoT7bMPVKRVlFpj+Plk8QuxPnGGYyn?=
 =?us-ascii?Q?uiM6AneyqHA6rXLSRNewQWsS5vIApiGRHDLB8PZ9LOzu/lGkzz6RuoRF7jg/?=
 =?us-ascii?Q?Y5pFHmm+n6z7/qaf857JJObOH9Q4VaAOUpwaWOpnMKLoExsCfZnUMdpRKhA6?=
 =?us-ascii?Q?Ggb3+hpcjue1D7HZWZmv0hDOEP2aeRUFellqMZaXa11k/gMP/ZL3TKZoxpBZ?=
 =?us-ascii?Q?UXA73V3U9Tbde+/2IzxpCN2sxoB/pnhCRDAEYB2m5eOumKdiaiGOQZhQGVNv?=
 =?us-ascii?Q?ehR5eDVHI/aSU3HCaDYOvGkWdr3MOgcrUbvcpW2AyKeLaMPVN7m9jOnZ4wPm?=
 =?us-ascii?Q?7xluwDegaL2rVQhQUtF08Qs=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C8D66737A2719E419BAF3EB0502EB2EE@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 932ae5cf-1c55-4977-63aa-08d99b0f2a1e
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2021 19:06:08.2657
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: trjTEqSn3mGvD8ysqydn8U1hCfMTNEd7AEF9QhuVbQXgv42Yo+R6bh6nwagBQ39L2QjGHdf9lzcVEBX4LrCXcQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2920
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10152 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0
 adultscore=0 suspectscore=0 bulkscore=0 mlxscore=0 spamscore=0
 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2110290104
X-Proofpoint-ORIG-GUID: MlhhfEm5Xsof3dsdt6DKQNs0o0ie0cqK
X-Proofpoint-GUID: MlhhfEm5Xsof3dsdt6DKQNs0o0ie0cqK
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Oct 29, 2021, at 2:17 PM, Timo Rothenpieler <timo@rothenpieler.org> wr=
ote:
>=20
> On 29.10.2021 17:14, Chuck Lever III wrote:
>> Hi Timo-
>>> On Oct 29, 2021, at 7:47 AM, Timo Rothenpieler <timo@rothenpieler.org> =
wrote:
>>>=20
>>> On 20/08/2021 17:12, Chuck Lever III wrote:
>>>> OK, I think the issue with this reproducer was resolved
>>>> completely with 6820bf77864d.
>>>> I went back and reviewed the traces from when the client got
>>>> stuck after a long uptime. This looks very different from
>>>> what we're seeing with 6820bf77864d. It involves CB_PATH_DOWN
>>>> and BIND_CONN_TO_SESSION, which is a different scenario. Long
>>>> story short, I don't think we're getting any more value by
>>>> leaving 6820bf77864d reverted.
>>>> Can you re-apply that commit on your server, and then when
>>>> the client hangs again, please capture with:
>>>> # trace-cmd record -e nfsd -e sunrpc -e rpcrdma
>>>> I'd like to see why the client's BIND_CONN_TO_SESSION fails
>>>> to repair the backchannel session.
>>>=20
>>> Happened again today, after a long time of no issues.
>>> Still on 5.12.19, since the system did not have a chance for a bigger m=
aintenance window yet.
>>>=20
>>> Attached are traces from both client and server, while the client is tr=
ying to do the usual xfs_io copy_range.
>>> The system also has a bunch of other users and nodes working on it at t=
his time, so there's a good chance for unrelated noise in the traces.
>>>=20
>>> The affected client is 10.110.10.251.
>>> Other clients are working just fine, it's only this one client that's a=
ffected.
>>>=20
>>> There was also quite a bit of heavy IO work going on on the Cluster, wh=
ich I think coincides with the last couple times this happened as well.<nfs=
trace.tar.xz>
>> Thanks for the report. We believe this issue has been addressed in v5.15=
-rc:
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi=
t/?id=3D02579b2ff8b0becfb51d85a975908ac4ab15fba8
>=20
> 5.15 is a little too bleeding edge for my comfort to roll out on a produc=
tion system.
> But the patch applies cleanly on top of 5.12.19. So I pulled it and am no=
w running the resulting kernel on all clients and the server(s).

Yup, that's the best we can do for now. Thanks for testing!


> Hopefully won't see this happen again from now on, thanks!
>=20
>=20
> Timo

--
Chuck Lever