Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Timo Rothenpieler <timo@rothenpieler.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Olga Kornievskaia <aglo@umich.edu>,
        Dai Ngo <dai.ngo@oracle.com>
Subject: Re: Spurious instability with NFSoRDMA under moderate load
Thread-Topic: Spurious instability with NFSoRDMA under moderate load
Thread-Index: AQHXSnp8j8N6G2OhyUK9Nr2RtJxnoarn3hgAgAATb4CANuhZgIBOXZcAgAAVhICAADVLgIAASYeAgAAo1oCAAAOTXYAA6JWAgAADg4CAAANlAIAAA4eAgAAZh4CAABNpAIABnm2AgAX5FgCABmbVAIBtyicAgAA5xwA=
Date:   Fri, 29 Oct 2021 15:14:42 +0000
Message-ID: <2445C26E-7D96-4E77-8079-98B865CC4C57@oracle.com>
References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org>
 <e12c24fd-beaf-31ce-cb49-36ad32bd22b3@rothenpieler.org>
 <daae674a-84d4-de36-336d-693dc582e3ef@rothenpieler.org>
 <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org>
 <a28b403e-42cf-3189-a4db-86d20da1b7aa@rothenpieler.org>
 <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com>
 <c8025457-7376-e1b7-bd6c-e5c6ee5d9ce7@rothenpieler.org>
 <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org>
 <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com>
 <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com>
 <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org>
 <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com>
 <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org>
 <cc2f55cd-57d4-d7c3-ed83-8b81ea60d821@rothenpieler.org>
 <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com>
 <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org>
 <716B2A38-9705-41D7-969B-665EF90156C7@oracle.com>
 <D55A1FA5-71D4-4D37-9B88-E1733BEDBE47@oracle.com>
 <60273c2e-e946-25fb-68af-975f793e73d2@rothenpieler.org>
In-Reply-To: <60273c2e-e946-25fb-68af-975f793e73d2@rothenpieler.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?JWrLDT8NboLtHoGI5SMwt8vH8PsmEHtRcaFVQPBkWlqjYfozCdXjItt929Gz?=
 =?us-ascii?Q?RXwFNLg/aTQ5U/LyXZJn6Q26qTMZRf4Iiz/lQ0Ot99SJVoiCR0AzamIDyoth?=
 =?us-ascii?Q?XbcVWd3c2LzciY6Bhbnk2ofggCR6H6LlnFhvHjgIhAeu1j8U580X5PQS40y0?=
 =?us-ascii?Q?L1eOGahthA2dmp73v5moRG9vghNNX9zPGx+o8qOGBljwEAsoUondHwAnpALZ?=
 =?us-ascii?Q?/s9gaXx2d/ZYbeiT0+1aRaZY4v9ZoqTf4k+07W2yPDDx9DNoHBl6amLQuppH?=
 =?us-ascii?Q?LwLnN+IirM3ayz4DTQ/ASVM1fNRtACvOX9Gv//TPKMKCB5Er+1gxyUhSvdqN?=
 =?us-ascii?Q?m+1AWTO80e3Z2dbIUT8Cw7c8cR2pl8ktWaHDWCFWQBide2N3u5yOGm7Z3MFT?=
 =?us-ascii?Q?DOOYQff0BvfDDBvQUuG1Iflg3YYO6NVSbXaxjORxii1lol05pW9DVt5zMG5S?=
 =?us-ascii?Q?XhJB/FVKuWRwzZpY4Vdnz2yz0eJScuy6B5GHPhMWDB56A85AgzOeNz5jeaOl?=
 =?us-ascii?Q?1mp4HAlJlxfATmijCptKGTrxIFKaAH5pLGYFzEjSpYbcOnAr2QCF73B4+esP?=
 =?us-ascii?Q?gGz4+TEn+BoNgdJbxI+A3l+lxtAlruAtLUMyVUREPNV4viTN4urEk4nO3YZL?=
 =?us-ascii?Q?+ldyAJQYsHW4gF5NZ+P/KqbTewsf+PYU8MZx2HlwXzddSO/GKJBdqVRoa9zh?=
 =?us-ascii?Q?KTwQ/UPRiRyLrJkvMefq9xVwbr94gTy1cA+9VKfUkXYFAiQNOalyhyR25D8R?=
 =?us-ascii?Q?A85aXuLG6FPD984FzRY4UvHiryJAJt6eNOhRl0OVMzPbXImORTZghF2gVKgL?=
 =?us-ascii?Q?Bb+4fBV8G63HEynkvsL/rMIGM1O2HyjcTLRl/jkI0mgJlGL3gvCkyEVJIiFJ?=
 =?us-ascii?Q?D8GClhDFo7on50QiHotYU2bS4C1EML3XiHooaBw87E6VNQ1Ye6on17IFjqKY?=
 =?us-ascii?Q?WgHgMQ4yHwuGusgG+RHN9kmF1y0Yn9w98vvhV2zmkQc3WnJj56Xj1VFaNZfm?=
 =?us-ascii?Q?0lzr3FSzDQFHk+EC0DYnD57BkM8DVMBfle1VSW+oaYvZu+/KQn+LC5qx3AfY?=
 =?us-ascii?Q?GGO7le56lPvQgRbi/Kg3LyBAz1qZITfpVTpqak1nh1x8aqmmIo0WL7cXySD7?=
 =?us-ascii?Q?ylkUUZbJ/M7QxAwSdqkVPjaqYZLTe3cAanxkDJFr9Xa4zICJwByd2acCnGlG?=
 =?us-ascii?Q?QDsyJYjsLmsd64e2aj+wBoqhit1hR1txyfEZ3Ssk0FLvG5EDIftBFBKdjyyT?=
 =?us-ascii?Q?n38Utm5u5eukn2/GnuGPzi0gZf9xpWaAOZKTe1boRWAAKNi/iERDiDe78AIG?=
 =?us-ascii?Q?f0AM12he+akwk/cTHpafdDAgqcEtwruqB7S/dpte3LRz22WYnqt5TBR5UXyt?=
 =?us-ascii?Q?n2nTTwx1+HjH5xOhjYwIKDzR/9/QRDzvGwMmVDGaHnAQ0pzF0gHHIJUvlJvP?=
 =?us-ascii?Q?99lUSoelGWdTldrFewofO6SE/Za9vyejNgPxdjETKRfjPxddoIy0igaOmUJS?=
 =?us-ascii?Q?tDZ+CkEMyhbaFld/2Z23+G14Ge8fzkmJ3aL8pO7P4Kosh/2oh2im7ZzX20SU?=
 =?us-ascii?Q?kFtyaHzkBQ4IkOb+pRkSPWCV6P5TL2gQvIgjrp7d3lSsLVBtrczPsWXw5aiW?=
 =?us-ascii?Q?Tbu1AWU7PwKFu8RF+JEzmWM=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4969C44B77BA024CB52D9C8893C7D1A5@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f386e408-df4d-44b1-0967-08d99aeed5a6
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2021 15:14:42.5534
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: iLm4N1JV+0mAeFP/4SUyKbLi4AhhE2/AY0EY7KR8OLCiMr1jRgqTHkZ5kjDaWGb+8CzWWsrNDUNf9G7rMRZdbg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4050
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10151 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999
 bulkscore=0 phishscore=0 mlxscore=0 spamscore=0 adultscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2110290087
X-Proofpoint-GUID: -P7lStxsBFvBKMDwFWB0Gsqf4QYI8oar
X-Proofpoint-ORIG-GUID: -P7lStxsBFvBKMDwFWB0Gsqf4QYI8oar
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Timo-

> On Oct 29, 2021, at 7:47 AM, Timo Rothenpieler <timo@rothenpieler.org> wr=
ote:
>=20
> On 20/08/2021 17:12, Chuck Lever III wrote:
>> OK, I think the issue with this reproducer was resolved
>> completely with 6820bf77864d.
>> I went back and reviewed the traces from when the client got
>> stuck after a long uptime. This looks very different from
>> what we're seeing with 6820bf77864d. It involves CB_PATH_DOWN
>> and BIND_CONN_TO_SESSION, which is a different scenario. Long
>> story short, I don't think we're getting any more value by
>> leaving 6820bf77864d reverted.
>> Can you re-apply that commit on your server, and then when
>> the client hangs again, please capture with:
>> # trace-cmd record -e nfsd -e sunrpc -e rpcrdma
>> I'd like to see why the client's BIND_CONN_TO_SESSION fails
>> to repair the backchannel session.
>=20
> Happened again today, after a long time of no issues.
> Still on 5.12.19, since the system did not have a chance for a bigger mai=
ntenance window yet.
>=20
> Attached are traces from both client and server, while the client is tryi=
ng to do the usual xfs_io copy_range.
> The system also has a bunch of other users and nodes working on it at thi=
s time, so there's a good chance for unrelated noise in the traces.
>=20
> The affected client is 10.110.10.251.
> Other clients are working just fine, it's only this one client that's aff=
ected.
>=20
> There was also quite a bit of heavy IO work going on on the Cluster, whic=
h I think coincides with the last couple times this happened as well.<nfstr=
ace.tar.xz>

Thanks for the report. We believe this issue has been addressed in v5.15-rc=
:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?=
id=3D02579b2ff8b0becfb51d85a975908ac4ab15fba8


--
Chuck Lever