Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dai Ngo <dai.ngo@oracle.com>
CC:     Timo Rothenpieler <timo@rothenpieler.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <aglo@umich.edu>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgAgAATb4CANuhZgIBOXZcAgAAVhICAADVLgIAASYeAgAAo1oCAAAOTXYAA6JWAgAADg4CAAANlAIAAA4eAgAAZh4CAABNpAIABnm2AgAgMfgCAAAwogIAAEeKAgAACzoCAASrTgA==
Date:   Wed, 18 Aug 2021 16:55:32 +0000
Message-ID: <0233D762-14A0-4BC5-91DC-5ECB455E908F@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
 <daae674a-84d4-de36-336d-693dc582e3ef@rothenpieler.org>
 <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org>
 <a28b403e-42cf-3189-a4db-86d20da1b7aa@rothenpieler.org>
 <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com>
 <c8025457-7376-e1b7-bd6c-e5c6ee5d9ce7@rothenpieler.org>
 <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org>
 <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com>
 <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com>
 <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org>
 <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com>
 <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org>
 <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
 <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com>
 <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org>
 <7B44D7FD-9D0F-4A0D-ABE9-E295072D953F@oracle.com>
 <97fe445e-6c4f-b7cf-fa39-fd0c4222a89e@rothenpieler.org>
 <baf4f3b0-6717-8e3d-efd5-fa471ae8e7e3@oracle.com>
 <94a7f911-84bd-f250-526e-054649a00f23@oracle.com>
In-Reply-To: <94a7f911-84bd-f250-526e-054649a00f23@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?2vboPGfFApT+59MdtRCSdGgpEensquC3+ulE9ykKBInsBrahruj08q+fsNP0?=
 =?us-ascii?Q?VHeAPszkBP+ZxgbknmU6G4TikP7nbTkn0rmEKCPcoQjN3N7v9LnxtXdrVIqB?=
 =?us-ascii?Q?bXI2ekSqyKBBtPZp+kc57yTsKaUUzowgjTfolA2UeFvOWzd4uYKVUlssXxIZ?=
 =?us-ascii?Q?yB+tBvqmzUzCYKdAYcIt1WRRuG7iUZ/+rViXo7yaAjpevYj+sk7ZQDhdx1L8?=
 =?us-ascii?Q?zrGkNC/rpRMKfbXIDXNCSllIP90ujCZvy68TKyzXLROnt9B39htDU2iscdUa?=
 =?us-ascii?Q?tzENOYV0N6kiIMVWObknezCcU7NajXFoX8GThHA0F79QT5UuJD812AmJzbQ8?=
 =?us-ascii?Q?TRcJz2lUZMULRkYegnCl7tCb2hyOfFUDlciOSCH+K7wX4xEtgjKY/pL2llOo?=
 =?us-ascii?Q?nCuxpsTY36wCKSGcEpuB5wC8Xkl0QoEA5oFcEZm3kX6XGQiTU/99LViDi52q?=
 =?us-ascii?Q?eZbn/FXREL4kPQ///dDxpAS9ieWzkCOvw8NWInSpFM/7NXcP9pK7CoCr+8qV?=
 =?us-ascii?Q?AjMUAuv2OKWMITZ6ihoQESsh57gNB3dIUTXvbJUCl8P2Eirp/V9EHM3dNTVk?=
 =?us-ascii?Q?KCGgFvRiBHq5TPz6TdusrwvmYI/PY9g/jSUYC91uuYnSdwMwq/vFvIoyAJfj?=
 =?us-ascii?Q?CED6LC5LuFnbyCC1/Ich3d15jFFaTa/HoZvqrkeg41+oiHo5MD7XGNxUXLnY?=
 =?us-ascii?Q?PZ/c7ZWHWvUMCzC4Pee8UySsFVxMQgfudLB0TyEEKI7wRQs5cFMXGKoVRuCZ?=
 =?us-ascii?Q?YQIloJ6TtKESd0yawBub+22OHMK3OvllnIL+m9h3VFcy+2MwtnWk2TFnkDtX?=
 =?us-ascii?Q?tW7p9B0dERdb1uqI/n3R7bCg+FrofmIEzQFzGnIXzl3nmGcIMs9EeQMUVseh?=
 =?us-ascii?Q?g870lEpnzju+mIRP8Y0SrvQLkiEToewIWOIWmVDRyuZZ5qkF+tfsV9z/evIz?=
 =?us-ascii?Q?coWv5awhX2ZqRyx5Cluluij0us4JriFeWdbvb7ZviFeSjI7gQ0ocgApTLnbm?=
 =?us-ascii?Q?TsScCShvbh26ePYFuPTTj+lLnunpuov5yKj/SeQnIigTfwX3ydB9T/REwnYp?=
 =?us-ascii?Q?jIWaptAXx1tvOycnhtnzF9MhHrOXk9a2EfH3f79eKA6nBajVz7WFC2tIR+7v?=
 =?us-ascii?Q?tyKYkw06CuzxFzKYb1c6oDyulM9IQeLeg7f5PqKy+PlJY1nmWMZYUC3PT+LT?=
 =?us-ascii?Q?+DpWkJQ3Uy1t0bkVPXGi+QUJMAvVy5X1jWfJF+UOW2j7vLWjvU2Yhd6/09eG?=
 =?us-ascii?Q?6POCXf+GB+Fv1rXWMGIvKYJ6DcywHSfFmvLIRcMhwFg/H+DN0wrW0wFf+TpE?=
 =?us-ascii?Q?NUBc8PRQIjwbfP2KgAwuuJ6f?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <337ED81272030241A0698CC9E773EDF9@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 12662c8c-e98e-4ad1-4e57-08d96268fdbe
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Aug 2021 16:55:32.1510
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Q6Xa0Mn8qX7wTFItB9530xeyAenabGzl/GHK8O4ENnIbFkK4m81Fxdn+QFgFyoftAdlJ7lE+RgogMSQPwb2dgA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2487
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10080 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0
 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000
 definitions=main-2108180106
X-Proofpoint-ORIG-GUID: dMQQ7obBTtvq6bdJepkw22UErde6fqet
X-Proofpoint-GUID: dMQQ7obBTtvq6bdJepkw22UErde6fqet
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 17, 2021, at 7:05 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>=20
>=20
> On 8/17/21 3:55 PM, dai.ngo@oracle.com wrote:
>>=20
>> On 8/17/21 2:51 PM, Timo Rothenpieler wrote:
>>> On 17.08.2021 23:08, Chuck Lever III wrote:
>>>> I tried reproducing this using your 'xfs_io -fc "copy_range
>>>> testfile" testfile.copy' reproducer, but couldn't.
>>>>=20
>>>> A network capture shows that the client tries CLONE first. The
>>>> server reports that's not supported, so the client tries COPY.
>>>> The COPY works, and the reply shows that the COPY was synchro-
>>>> nous. Thus there's no need for a callback, and I'm not tripping
>>>> over backchannel misbehavior.
>>>=20
>>> Make sure the testfile is of sufficient size. I'm not sure what the thr=
eshold is, but if it's too small, it'll just do a synchronous copy for me a=
s well.
>>> I'm using a 50MB file in my tests.
>>=20
>> The threshold for intra-server copy is (2*rsize) of the source server.
>=20
> The threshold for inter-server copy is (14*rsize). This restriction is to=
 be removed in the next merge.
>=20
> e-Dai
>=20
>> Any copy smaller than this is done synchronously.

Thank you, Dai, that helped. I was able to reproduce with a 200MB
source file on an xfs-backed export. Looking at the issue now.


--
Chuck Lever