Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Jonathan Woithe <jwoithe@just42.net>
CC:     Bruce Fields <bfields@fieldses.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [Bug report] Recurring oops, 5.15.x, possibly during or soon
 after client mount
Thread-Topic: [Bug report] Recurring oops, 5.15.x, possibly during or soon
 after client mount
Thread-Index: AQHYCTZE+ENVwkeetkGrNbALtTtuGqxioWgAgAEb9gCAAMFGgA==
Date:   Sat, 15 Jan 2022 19:46:06 +0000
Message-ID: <927EED04-840E-4DA6-B2B1-B604A7577B4E@oracle.com>
References: <20220114103901.GA22009@marvin.atrad.com.au>
 <C7A57602-4DDD-4952-BA38-03F819DBD296@oracle.com>
 <20220115081420.GB8808@marvin.atrad.com.au>
In-Reply-To: <20220115081420.GB8808@marvin.atrad.com.au>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?zm/VQXHzwap3OdRnrvNXll548d5Y+0Jp/R31SFT0xfAFJ9T9EMZUcGX6Wuqb?=
 =?us-ascii?Q?HwBDjOc4pNNnQ/rRcas+Xw5Fp0JHsfKJApp+B/XJBrP5qfcqxMR59rpA7KZh?=
 =?us-ascii?Q?mPMHyEjN9b7TcPIrEuN8KhGmwN7+os9MRbCKhFdhC3pUCUDOx8R7b/2lDxf6?=
 =?us-ascii?Q?PVkarlzFvamkrqbEbCBhVIdgEvdi8MsTKsUIiayWt/Wd7nVvBAYKU1u60s/S?=
 =?us-ascii?Q?B2eDOrim65hhv73pv0X8l3RgOjB7F0MXQni7D1blC3VOCMA3KEO/FnRdaJYP?=
 =?us-ascii?Q?TXwwJyCWf9XcjPYxEiNDs9YujeJo0MIc+4Be3Xfqm66wSheBMg8OqJ1GnM1O?=
 =?us-ascii?Q?jj4zUS11hZg1qfA/W+Q9/l8yt+dTv/Uvcqavl75cCvSseCEMpjR9qdH42Odu?=
 =?us-ascii?Q?hzwKbOYlVWg52krYeSoYYU4Y81nFTP6lPZO7VfMW877NBLk11B+Sx9ANNFbb?=
 =?us-ascii?Q?Hhj6t4XV3G9WXFvu4n+gOxG7mZMywFj/BEmV+j95/kfdRv+PK7zlnKzPqp+i?=
 =?us-ascii?Q?llqKiwtzRaogG6skbQ9At2G+3sEGc6xgkeIgRwdoIbZJ0S/4bdOcuRZvwU+7?=
 =?us-ascii?Q?DppQ8gyH0C7lmVcRVx0uI/md12/JcQTB5ZtoX//Vg1+ByULo2xhKdCkmnwgv?=
 =?us-ascii?Q?a0ZgrUaOxPFnvyWFYi02U5n8qC/DY3DWSbmBiryvVbmZO+oj/gZetCsf9b/M?=
 =?us-ascii?Q?rR47/VpF35jzHPJXRrtz4KL0Y9gNAsMxp+YkvgsuH1jXHIx44Kc85ElEtHG2?=
 =?us-ascii?Q?Y+3eVnq0R8aMtaCO0USpzrKDhUGlgh6EKvt1J94tJ4lprpKIE6SQrf0bHsPb?=
 =?us-ascii?Q?x1pYNMfeJXjaCDD4w1NgcO7oyGqsB6VtH8TleCWbaa8T21oMdpwCpeXCNdss?=
 =?us-ascii?Q?4ipSKjejbP06W1TZe1SpE6+J9D3MNUERhHwTFt/PZGCVhUASLhE20U2KrlLK?=
 =?us-ascii?Q?RXKdjNZUgxD+Xa8bpt1pAmSGTm0BorheEuxP1lp4y//G0BdWG6HbKi1BUE8O?=
 =?us-ascii?Q?Fi40XifjJmdfZfoKtr9/Jj77clo97Sn+qAB62/NgCsjGwenIcUEDE+3CWF9c?=
 =?us-ascii?Q?afiV7+liMK3hgfTPM/P8e6gVkHvowhGbk664RRIywtwbxTjI8M6pqyinPhKO?=
 =?us-ascii?Q?Ijq14mbQ/mhBrhqM4+LszMFtHNzyDXa5MXvChAbSOOIx13WzZr0TCsPCNG+A?=
 =?us-ascii?Q?dYT8GUOQaTf7DEPqBO9xy+Y9C6vsPPlWQi8BFwJFoHRfTj6lrIvAbY9Qmrhb?=
 =?us-ascii?Q?FalrsaUkS9fHk5HyWm65v7G1JFDp2A0c17qaZpKg3hgpR338HSNhdYC/mWfT?=
 =?us-ascii?Q?ccomsawawkTbdPlIkEk+rQBrzdeOxfXWdV/CW/1ur+NPX+LXvtXrcO4X8h7R?=
 =?us-ascii?Q?VZu44mwE97wNhyjvmWjUmSqGDrLI3LfWd5HBkwEZJFJdgjTT2XT0Z/FPWv3G?=
 =?us-ascii?Q?yYn1e2CJdz9W4u4L2kzILVWDaoIYzw6RE7N1b4+tGSahLtghsWoqUKf4RBZt?=
 =?us-ascii?Q?OlwYX0N8AH+G3IeT5EddxDdqm4t7FmDrjVCzKTnh5+yJtWUT/YLl9+MHCmi4?=
 =?us-ascii?Q?iK2FdCoGXXG+UsTg+SGhSCcj4jKtcbZiKSMN4Rra1e2Sgj7QiI8H4Ki4QjOH?=
 =?us-ascii?Q?sCsWjVU40og7EUdqfWqFDFU=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7185EBDB0C1E0F44BE8BD5A0E5A37506@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR10MB4858.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 72c027e2-63df-44bf-41f1-08d9d85fac17
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Jan 2022 19:46:06.9552
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: k6nUdOy6ZE43Sxg38y5NGeZLAlpbRdOae7/2klqs9c4QCGDdu54HyPoqs1UfhmDMVW9WEqMwkHqr1No6S+AnYw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3143
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10228 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0
 mlxlogscore=999 malwarescore=0 bulkscore=0 suspectscore=0 spamscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2201150128
X-Proofpoint-ORIG-GUID: 6QSAwsoDUM-f-jN67q6r_cCBYq7d9RJl
X-Proofpoint-GUID: 6QSAwsoDUM-f-jN67q6r_cCBYq7d9RJl
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 15, 2022, at 3:14 AM, Jonathan Woithe <jwoithe@just42.net> wrote:
>=20
> Hi Chuck
>=20
> Thanks for your response.
>=20
> On Fri, Jan 14, 2022 at 03:18:01PM +0000, Chuck Lever III wrote:
>>> Recently we migrated an NFS server from a 32-bit environment running=20
>>> kernel 4.14.128 to a 64-bit 5.15.x kernel.  The NFS configuration remai=
ned
>>> unchanged between the two systems.
>>>=20
>>> On two separate occasions since the upgrade (5 Jan under 5.15.10, 14 Ja=
n
>>> under 5.15.12) the kernel has oopsed at around the time that an NFS cli=
ent
>>> machine is turned on for the day.  On both occasions the call trace was
>>> essentially identical.  The full oops sequence is at the end of this em=
ail.=20
>>> The oops was not observed when running the 4.14.128 kernel.
>>>=20
>>> Is there anything more I can provide to help track down the cause of th=
e
>>> oops?
>>=20
>> A possible culprit is 7f024fcd5c97 ("Keep read and write fds with each
>> nlm_file"), which was introduced in or around v5.15.  You could try a
>> simple test and back the server down to v5.14.y to see if the problem
>> persists.
>=20
> I could do this, but only perhaps on Monday when I'm next on site.  It ma=
y
> take a while to get an answer though, since it seems we hit the fault onl=
y
> around once every 2 weeks.  Since it's a production server we are of cour=
se
> limited in the things I can do.
>=20
> I *may* be able to set up another system as an NFS server and hit that wi=
th
> repeated mount requests.  That could help reduce the time we have to wait
> for an answer.

Given the callback information you provided, I believe that the problem
is due to a client reboot, not a mount request. The callback shows the
crash occurs while your server is processing an SM_NOTIFY request from
one of your clients.


> Is it worth considering a revert of 7f024fcd5c97?  I guess it depends on =
how
> many later patches depended on it.

You can try reverting 7f024fcd5c97, but as I recall there are some
subsequent changes that depend on that one.

--
Chuck Lever