Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Timo Rothenpieler <timo@rothenpieler.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <aglo@umich.edu>,
        Dai Ngo <dai.ngo@oracle.com>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgAgAATb4CANuhZgIBOXZcAgAAVhICAADVLgIAASYeAgAAo1oCAAAOTXYAA6JWAgAADg4CAAANlAIAAA4eAgAAZh4CAABNpAIABnm2AgAX5FgCABmbVAA==
Date:   Fri, 20 Aug 2021 15:12:04 +0000
Message-ID: <D55A1FA5-71D4-4D37-9B88-E1733BEDBE47@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
 <72ECF9E1-1F6E-44AF-850C-536BED898DDD@oracle.com>
 <e12c24fd-beaf-31ce-cb49-36ad32bd22b3@rothenpieler.org>
 <daae674a-84d4-de36-336d-693dc582e3ef@rothenpieler.org>
 <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org>
 <a28b403e-42cf-3189-a4db-86d20da1b7aa@rothenpieler.org>
 <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com>
 <c8025457-7376-e1b7-bd6c-e5c6ee5d9ce7@rothenpieler.org>
 <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org>
 <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com>
 <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com>
 <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org>
 <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com>
 <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org>
 <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
 <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com>
 <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org>
 <716B2A38-9705-41D7-969B-665EF90156C7@oracle.com>
In-Reply-To: <716B2A38-9705-41D7-969B-665EF90156C7@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?XDWkDY3hlhHWufBZQqp6Nk28EDbjyE9b20bU/3hLf0N3XSdbxv9j7t/iRnqo?=
 =?us-ascii?Q?Oj0+jUw8su8gjzk++NS7/EsJfRzN+hDyutVcZd0sOTpbaJcaHhifVhEcOzJ/?=
 =?us-ascii?Q?Sn8d17204J1GmjqVOkVKcv3tXpUWku+SeNI7Lc8m2wBpvcHWL/U09wiSsYA6?=
 =?us-ascii?Q?ZkXssVUghCGgtVYsFEgSkivaZj8Ra1SX4SwF5Dy4Dh2NmSrVjL+jrrYPAnsK?=
 =?us-ascii?Q?4BNTq5nulGOiIuA/o5+sGW+dlg5CK4uIRMRCSINkLqUFeAz/cGZb/H5V5QGd?=
 =?us-ascii?Q?eSBKXMYJurEWs5hy8NgwjJss2kTK8RPH/TJkEnZM6kmB+KLgSIv9BA7h+W8d?=
 =?us-ascii?Q?GGdkEbvtc/vV/ZwHQmhDnS6VZ/n4F5sGkb/1op3VXvJmic7RMLazatgLuh6s?=
 =?us-ascii?Q?KMVgSBY0obPWVU8ST9mrp9aNtm5r388ufbDD18XcxKo4KqysHy5XbKoYNvni?=
 =?us-ascii?Q?Zn1ivQFlMozbTgpsbFRa1qT8Q/b1dCAqDSrKC+qCWfNJ6yG4+avaU3yWybeb?=
 =?us-ascii?Q?+ZwxEZVeG4zvPq2fenM3BJ0kWJ31IdHrKRHUCZ7PiOlefr1UfvhvmjsyKkDI?=
 =?us-ascii?Q?URjzGJ6ix71q0PpvjH1MgTbs9zqxylB9v6SxVaDqwpyF/Cu7XbNLhj7owRQB?=
 =?us-ascii?Q?0GO3RFIyRqbfZIVGBFkCAfM0HhCYcUYX4jn4qH7Yt4x+4rJDqzfB3CT7qbQ5?=
 =?us-ascii?Q?8tLFX1zdkKjs1D756IQvLsnWM9Xe7/vrL5Wn4c3qgqD0NyKauFo3iJ/xllzm?=
 =?us-ascii?Q?81ng2wqmoCA5dEVKzeBUf4QYBWSwROln3obqvfzkSOmyRatCBc1EtxUUrHYW?=
 =?us-ascii?Q?scXZMgxQOPAv6NjyLYJjKQAfsd13rWaKLw4RFavNfzTJRqJcDu8ckaiW+NRr?=
 =?us-ascii?Q?PPnvpbd4BaZIO/34L/6n9Pxn5YG7HGMJCvs4a9NypKAo8/xTePcvK3lllLW/?=
 =?us-ascii?Q?SVEqshDq98bLL7AMllxjPAuPPj+degBmSKTZfQIh908l/MPp3O3ozwf6MOWx?=
 =?us-ascii?Q?bO4RCRbuvZBtB+IIvIP2G6W61IoNVgGs+p+ro383NP0D6mOT9qbzukScWylY?=
 =?us-ascii?Q?ljsl4xk2wfhg+nC8sXlaSBNsjs1L/z5UQREs/vRpjPnzHhounzAxN6jjrkl4?=
 =?us-ascii?Q?ng6Ai7wddR2kZmKj7UMWrV/Ds4rt/5AgH8BMAnPYIM+TKQVdemGnBBtecDF/?=
 =?us-ascii?Q?nXg5CGyNjc2RGKAV2Usg1zzmgcgk7YYK2pILW+la3ooNT2popzbHXnNhc9N5?=
 =?us-ascii?Q?rRptnXkdav5KyafmMFaGKvtt4VEuRehk5SWvc5jjC4WeZ+a8JCRFAvZtmly8?=
 =?us-ascii?Q?ksYc3iIAQ9t3SG8Sz1+aJPBx?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <36B47CF77DCB1649AA0B978397DCBFF7@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1cacc97c-b6ab-4c3d-3dfd-08d963ecdec0
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Aug 2021 15:12:04.9033
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: baZyAn2omDcbDqD2oMYK63uz/st4o1M87gMwREsUUpl/4Khm42BaSNGQ6ZUMNd/TqLiHB/PMEmeGXNcjFiAHYw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3413
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10082 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0
 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000
 definitions=main-2108200086
X-Proofpoint-ORIG-GUID: rVQpJ6Oo78nUzy_R5T4AKHWgyupwD7-C
X-Proofpoint-GUID: rVQpJ6Oo78nUzy_R5T4AKHWgyupwD7-C
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 16, 2021, at 9:26 AM, Chuck Lever III <chuck.lever@oracle.com> wro=
te:
>=20
>> On Aug 12, 2021, at 2:13 PM, Timo Rothenpieler <timo@rothenpieler.org> w=
rote:
>>=20
>> On 11.08.2021 19:30, Chuck Lever III wrote:
>>>> On Aug 11, 2021, at 12:20 PM, Timo Rothenpieler <timo@rothenpieler.org=
> wrote:
>>>>=20
>>>> resulting dmesg and trace logs of both client and server are attached.
>>>>=20
>>>> Test procedure:
>>>>=20
>>>> - start tracing on client and server
>>>> - mount NFS on client
>>>> - immediately run 'xfs_io -fc "copy_range testfile" testfile.copy' (wh=
ich succeeds)
>>>> - wait 10~15 minutes for the backchannel to time out (still running 5.=
12.19 with the fix for that reverted)
>>>> - run xfs_io command again, getting stuck now
>>>> - let it sit there stuck for a minute, then cancel it
>>>> - run the command again
>>>> - while it's still stuck, finished recording the logs and traces
>>> The server tries to send CB_OFFLOAD when the offloaded copy
>>> completes, but finds the backchannel transport is not connected.
>>> The server can't report the problem until the client sends a
>>> SEQUENCE operation, but there's really no other traffic going
>>> on, so it just waits.
>>> The client eventually sends a singleton SEQUENCE to renew its
>>> lease. The server replies with the SEQ4_STATUS_BACKCHANNEL_FAULT
>>> flag set at that point. Client's recovery is to destroy that
>>> session and create a new one. That appears to be successful.
>>=20
>> If it re-created the session and the backchannel, shouldn't that mean th=
at after I cancel the first stuck xfs_io command, and try it again immediat=
ely (before the backchannel had a chance to timeout again) it should work?
>=20
> I would guess that yes, subsequent COPY_OFFLOAD requests
> should work unless the backchannel has already timed out
> again.
>=20
> I was about to use your reproducer myself, but a storm
> came through on Thursday and knocked out my power and
> internet. I'm still waiting for restoration.
>=20
> Once power is restored I can chase this a little more
> efficiently in my lab.

OK, I think the issue with this reproducer was resolved
completely with 6820bf77864d.

I went back and reviewed the traces from when the client got
stuck after a long uptime. This looks very different from
what we're seeing with 6820bf77864d. It involves CB_PATH_DOWN
and BIND_CONN_TO_SESSION, which is a different scenario. Long
story short, I don't think we're getting any more value by
leaving 6820bf77864d reverted.

Can you re-apply that commit on your server, and then when
the client hangs again, please capture with:

# trace-cmd record -e nfsd -e sunrpc -e rpcrdma

I'd like to see why the client's BIND_CONN_TO_SESSION fails
to repair the backchannel session.

--
Chuck Lever