Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Timo Rothenpieler <timo@rothenpieler.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <aglo@umich.edu>,
        Dai Ngo <dai.ngo@oracle.com>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgAgAATb4CANuhZgIBOXZcAgAAVhICAADVLgIAASYeAgAAo1oCAAAOTXYAA6JWAgAADg4CAAANlAIAAA4eAgAAZh4CAABNpAIABnm2AgAX5FgA=
Date:   Mon, 16 Aug 2021 13:26:29 +0000
Message-ID: <716B2A38-9705-41D7-969B-665EF90156C7@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
 <72ECF9E1-1F6E-44AF-850C-536BED898DDD@oracle.com>
 <e12c24fd-beaf-31ce-cb49-36ad32bd22b3@rothenpieler.org>
 <daae674a-84d4-de36-336d-693dc582e3ef@rothenpieler.org>
 <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org>
 <a28b403e-42cf-3189-a4db-86d20da1b7aa@rothenpieler.org>
 <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com>
 <c8025457-7376-e1b7-bd6c-e5c6ee5d9ce7@rothenpieler.org>
 <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org>
 <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com>
 <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com>
 <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org>
 <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com>
 <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org>
 <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
 <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com>
 <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org>
In-Reply-To: <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?nYLINPjobbWVNcCUT0zxi2wymeh4F8Utp6b0Q8ILq1ThycTAHk3WxYYoZ60B?=
 =?us-ascii?Q?mPKUuVTuRdlqQKi/bmCqngweLbQTPvASuqZITkR7L8GyKJ6/nVBBeznrQSa0?=
 =?us-ascii?Q?TlL5r2fVl9nn3KUVcf/X9ZDaNJN1b3vj3/cqmIHQ4kxmOM5RiDg/u27ceI1L?=
 =?us-ascii?Q?RIgNJdxHc/DGBi3YN8aUu0JIZDMAIwzTC2fImVubu+9onXG9UwQShRumrh+O?=
 =?us-ascii?Q?1fHWpiLFPnKBKVb8CANnEWd5u8huaSegWta0wPcUfooBpmrN0wEqP20V62ck?=
 =?us-ascii?Q?FTO4CitzHcIOreL5LBcbDSAHp4u2/6BQqvuonUdVGHeMGBkIIAVM8fktiuPM?=
 =?us-ascii?Q?EyHFI5BKRpydcmxMSLkt24PYG7wH2g6ggy08QzJbkUOteGHQnTJBxmDXppHI?=
 =?us-ascii?Q?UnWELeksJIg0O0ai0QMyXo6hOV4GhsQzY85j2bnVDjdvHQZIwz5McI3glvSh?=
 =?us-ascii?Q?dQnT0riNhqbu5LzoCKrJn/fuF1sSIppdud0srPmW+EhKBPpllATrJCrh/T9+?=
 =?us-ascii?Q?18QwbKgWIRDNtrNY+TMk9DpOZsL4LEffva/9pXSAaFTg43PBbUoy/+aYwxnl?=
 =?us-ascii?Q?jhtGcNU7f1fZgQqfpilZNbau/YJu4vLN8n7SPpKiyjNd5/6f8eTAx0/8ncgs?=
 =?us-ascii?Q?8W/00s4cBs8IFpg8IFVVCebIW2LEo9SOZTyMfu1pgOUxMsg6Vtk/RChU2nu8?=
 =?us-ascii?Q?6nEdF8V7av0kiqoNaViZU3UZfRQE77ox8eCMjNnQZfRLtoOZk6i+ysLBf+fp?=
 =?us-ascii?Q?xxan9cnrqcX4NuvuAxT1+kVLDIM4qGkbvz3PjfJia4ucWRNa2rQLWckx+qIu?=
 =?us-ascii?Q?o7dENxZAQ9riYf3LSDV9sZrbfF6gycYjug2ruBZgrBfFI2J2OsfOEwIupqJW?=
 =?us-ascii?Q?BKC8IkBfrq+KQAr9VhmCj67DgH3Q+z3LZN30VdY0htUKIpc7IBtSjC4kP+1g?=
 =?us-ascii?Q?/2yF8sYgB5IMJBPaQmclectH/FvfxNTY7Qj4FTofYyT1VZEsaHscaDP5s+LR?=
 =?us-ascii?Q?k6pSLueCoD/xTqg3gpdaGTFdN9VbSJ5QtBjRoHHKjbYPq86HZTQYnlPYR5CG?=
 =?us-ascii?Q?0a/ATGZugINxoi5gnRaynOMifpUjtxohKydYA2bVInGZquP6tHIrkueipUnK?=
 =?us-ascii?Q?bHP0vwcoIGzr3JvOh5O5MEwi3FqP8E+r5X95BgcVIVVGByjiINAGRCzsW4Um?=
 =?us-ascii?Q?Dcl4BdvIpWYrrVZ/EvitJojg2EP7DO+ZR5fHLnUuxz0vzjTBVKLONksf7yTr?=
 =?us-ascii?Q?JFjsdUj2qKVR/BOeQmkdaeOe0kNSl+onWegIGnUOCkehadgnVa5ApCG/hVRS?=
 =?us-ascii?Q?PovaQgrhHgDKRel9G5cYYl0khPuUZ5C5qIxyo3YvBr7GbfTId3ivDdmlQi6J?=
 =?us-ascii?Q?FfMWkSzDhlsE5m3CRTBEV2X+v6ywRmw0eUelcFvtfaiEImi/MQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <00446C63CCC2B748800EDE30FEDBF131@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 971f70f4-4e45-40dc-eaf2-08d960b974ab
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Aug 2021 13:26:29.0732
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: oSKh5ZQRP2n6sISCE/4poaMlGdkwOkm9xSX7LwE+jmASiaWzSEft7aedHLnk8wSUycAl3DRViMyy4KkXQ5DtNQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3222
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10077 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0
 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000
 definitions=main-2108160084
X-Proofpoint-ORIG-GUID: ENZZ8OFwh_d1qQeg5kyI0DfK5dnRmjxS
X-Proofpoint-GUID: ENZZ8OFwh_d1qQeg5kyI0DfK5dnRmjxS
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 12, 2021, at 2:13 PM, Timo Rothenpieler <timo@rothenpieler.org> wr=
ote:
>=20
> On 11.08.2021 19:30, Chuck Lever III wrote:
>>> On Aug 11, 2021, at 12:20 PM, Timo Rothenpieler <timo@rothenpieler.org>=
 wrote:
>>>=20
>>> resulting dmesg and trace logs of both client and server are attached.
>>>=20
>>> Test procedure:
>>>=20
>>> - start tracing on client and server
>>> - mount NFS on client
>>> - immediately run 'xfs_io -fc "copy_range testfile" testfile.copy' (whi=
ch succeeds)
>>> - wait 10~15 minutes for the backchannel to time out (still running 5.1=
2.19 with the fix for that reverted)
>>> - run xfs_io command again, getting stuck now
>>> - let it sit there stuck for a minute, then cancel it
>>> - run the command again
>>> - while it's still stuck, finished recording the logs and traces
>> The server tries to send CB_OFFLOAD when the offloaded copy
>> completes, but finds the backchannel transport is not connected.
>> The server can't report the problem until the client sends a
>> SEQUENCE operation, but there's really no other traffic going
>> on, so it just waits.
>> The client eventually sends a singleton SEQUENCE to renew its
>> lease. The server replies with the SEQ4_STATUS_BACKCHANNEL_FAULT
>> flag set at that point. Client's recovery is to destroy that
>> session and create a new one. That appears to be successful.
>=20
> If it re-created the session and the backchannel, shouldn't that mean tha=
t after I cancel the first stuck xfs_io command, and try it again immediate=
ly (before the backchannel had a chance to timeout again) it should work?

I would guess that yes, subsequent COPY_OFFLOAD requests
should work unless the backchannel has already timed out
again.

I was about to use your reproducer myself, but a storm
came through on Thursday and knocked out my power and
internet. I'm still waiting for restoration.

Once power is restored I can chase this a little more
efficiently in my lab.


> Cause that's explicitly not the case, once the backchannel initially time=
s out, all subsequent commands get stuck, even if the system is seeing othe=
r work on the NFS mount being done in parallel, and no matter how often I r=
e-try and how long I wait in between or with it stuck.
>=20
>> But the server doesn't send another CB_OFFLOAD to let the client
>> know the copy is complete, so the client hangs.
>> This seems to be peculiar to COPY_OFFLOAD, but I wonder if the
>> other CB operations suffer from the same "failed to retransmit
>> after the CB path is restored" issue. It might not matter for
>> some of them, but for others like CB_RECALL, that could be
>> important.
>> --
>> Chuck Lever

--
Chuck Lever