Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Timo Rothenpieler <timo@rothenpieler.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <aglo@umich.edu>,
        Dai Ngo <dai.ngo@oracle.com>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgAgAATb4CANuhZgIBOXZcAgAAVhICAADVLgIAASYeAgAAo1oCAAAOTXYAA6JWAgAADg4CAAANlAIAAA4eAgAAZh4CAABNpAA==
Date:   Wed, 11 Aug 2021 17:30:23 +0000
Message-ID: <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
 <72ECF9E1-1F6E-44AF-850C-536BED898DDD@oracle.com>
 <e12c24fd-beaf-31ce-cb49-36ad32bd22b3@rothenpieler.org>
 <daae674a-84d4-de36-336d-693dc582e3ef@rothenpieler.org>
 <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org>
 <a28b403e-42cf-3189-a4db-86d20da1b7aa@rothenpieler.org>
 <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com>
 <c8025457-7376-e1b7-bd6c-e5c6ee5d9ce7@rothenpieler.org>
 <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org>
 <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com>
 <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com>
 <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org>
 <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com>
 <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org>
 <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
In-Reply-To: <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?olXO1RTt/aNuLIyatgm/SvzGjPe6kLpp2AWn33BvQ59QTIcqwkuo/iEefsq0?=
 =?us-ascii?Q?b0dIXfv3UTdn4tcRtrNAv+AoGxepZl462vLYGZcfr8b3tzBj+hDBrKALA60h?=
 =?us-ascii?Q?ujJk+vF6shHtLYCFeHJ37VQME0+pUmyFuoPiKyXkoLdGVqJTZyQrqx/lBSbF?=
 =?us-ascii?Q?QWDfpUxY5Nk7/+J5pvcmd9zz3FgSRSlmjns6wZXkTcrnkJVhq8h2DI3L8Arw?=
 =?us-ascii?Q?sRHbJqlWHgO9RZ47j7zkq9ocnJtwYOCGwfY6Q1FlXiCVix8DnZPzmqn/ZgKq?=
 =?us-ascii?Q?Gm/FJR6NNenoN4sl69wXucrF9LwoFzrJ5BWoloZFUJt7apza3BNtVpQWENsP?=
 =?us-ascii?Q?42+rcAUyVhn11/wux1KJRyexZHqzSSoD7e08FFHcJ4MweXYx1ZfwC2gRB3ov?=
 =?us-ascii?Q?bmmJnBKEc8cgnU14AYvxMbvGl5wKNt+IUBXZ1DJhbS3MRw8XW5s2dhQZ9IuU?=
 =?us-ascii?Q?tQwE/Xr++AYkK7T8yQHdkVfEv6V/H0uKcBhR4YRXZbJTt8OlP66/ATrg08p+?=
 =?us-ascii?Q?NqON6O4PgWY4Q/rcyTHgJrt8CMU5onoYiYsk3pvNuZIYeO8uxq1djm625r+7?=
 =?us-ascii?Q?2O549Q+NPKifVjpKRqt6ZQyiK3Nvu54ljl2ntvqSUnrrRhlsdzWRf3GlqrOk?=
 =?us-ascii?Q?/R2VfJAswRlbyMQyMwo5MLP8Nz0Ngr3fWHgRgQH2n/yvV1DWwl29Bs9RhC0s?=
 =?us-ascii?Q?2saAYazvGxmu7IrEYEt8KVIbHfz/3nRyPC8SP0/msUZR7mpvOwG2LpPZ40ir?=
 =?us-ascii?Q?Fmbit5uBiJxaZlCu0HbVms1mEiQJzisYoR5DdhZF0i4gba0/1w4A8rapSNDb?=
 =?us-ascii?Q?aMG6MboHGfREkZCVXAAv8+s8Sqpzu6qQ1Eva3dbLhcfjDTgokeBXTOMAfDM6?=
 =?us-ascii?Q?D9Tl86JhkJHir/bSzNFqzePvOT2KhmOvkRNZrUAME3VUMCnwjJgMH9ZaDBYO?=
 =?us-ascii?Q?XkcMAIh1I9csG0/M4eKfX6DCzZuNcySCe0py6Na+NKYgjs0xIH7ptd7+lumT?=
 =?us-ascii?Q?IvXCWSaIbqOEGe+19EH1B6whca8WpNOq8uDz6FtN97zR4sWf2f1fP0axOF5k?=
 =?us-ascii?Q?FT/qDKH5DPclIZmvLoQDXjo3Ts7YhylhqI4dDdRSXo9ZdRu1aes272w23OC3?=
 =?us-ascii?Q?fnBD39KKB36LkqCSf+qsTmVBEuStpzmXpYZ4S22WMPJDyeohXI+OgwKs2f7I?=
 =?us-ascii?Q?8YrFu5ZkSszd/DkUDLbeXVMIITPc0mnSweODp2fFBn3dFuatJSNdTNiPR8Wp?=
 =?us-ascii?Q?3ZUyb0QRzRva7TOKcRrJjo2z4wO4v1Nxf2v63lN7LDzRbGEj9VP57WsUP42X?=
 =?us-ascii?Q?2zJPpvn8jhBJiTZHKiz0x8zmW8Cim3fWsjXw7cMccDw4BA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8C65B302A66C3A498D6078ED106573CA@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c32883ab-25b1-4b16-321b-08d95cedb316
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Aug 2021 17:30:23.0230
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: I3ODZqDX2BC5vpUsi6i09ggZCdpzGPTPTbFHMpItFshcZ/tK6l5jrlErjfD1Fj2fWYV8dF8i1rSZK8sxi7eXjw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3464
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10073 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 spamscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2107140000 definitions=main-2108110119
X-Proofpoint-ORIG-GUID: n0Hltwoc6vInOpCzVoaUen9mtguFU0gJ
X-Proofpoint-GUID: n0Hltwoc6vInOpCzVoaUen9mtguFU0gJ
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 11, 2021, at 12:20 PM, Timo Rothenpieler <timo@rothenpieler.org> w=
rote:
>=20
> resulting dmesg and trace logs of both client and server are attached.
>=20
> Test procedure:
>=20
> - start tracing on client and server
> - mount NFS on client
> - immediately run 'xfs_io -fc "copy_range testfile" testfile.copy' (which=
 succeeds)
> - wait 10~15 minutes for the backchannel to time out (still running 5.12.=
19 with the fix for that reverted)
> - run xfs_io command again, getting stuck now
> - let it sit there stuck for a minute, then cancel it
> - run the command again
> - while it's still stuck, finished recording the logs and traces

The server tries to send CB_OFFLOAD when the offloaded copy
completes, but finds the backchannel transport is not connected.

The server can't report the problem until the client sends a
SEQUENCE operation, but there's really no other traffic going
on, so it just waits.

The client eventually sends a singleton SEQUENCE to renew its
lease. The server replies with the SEQ4_STATUS_BACKCHANNEL_FAULT
flag set at that point. Client's recovery is to destroy that
session and create a new one. That appears to be successful.

But the server doesn't send another CB_OFFLOAD to let the client
know the copy is complete, so the client hangs.

This seems to be peculiar to COPY_OFFLOAD, but I wonder if the
other CB operations suffer from the same "failed to retransmit
after the CB path is restored" issue. It might not matter for
some of them, but for others like CB_RECALL, that could be
important.


--
Chuck Lever