Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Jonathan Woithe <jwoithe@just42.net>,
        Bruce Fields <bfields@fieldses.org>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [Bug report] Recurring oops, 5.15.x, possibly during or soon
 after client mount
Thread-Topic: [Bug report] Recurring oops, 5.15.x, possibly during or soon
 after client mount
Thread-Index: AQHYCTZE+ENVwkeetkGrNbALtTtuGqxioWgA
Date:   Fri, 14 Jan 2022 15:18:01 +0000
Message-ID: <C7A57602-4DDD-4952-BA38-03F819DBD296@oracle.com>
References: <20220114103901.GA22009@marvin.atrad.com.au>
In-Reply-To: <20220114103901.GA22009@marvin.atrad.com.au>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?7Qwj5LT55MyOFb2aVdd60wFmor+sfTH8w6DTC6bIaefBDq2kwMb4NsWlHP1Z?=
 =?us-ascii?Q?VOuA6jmi4oSKgxdkN5ICBIGHa4fyV8LPHA8y4nl0cw5aFa0XSSariwK7EsoX?=
 =?us-ascii?Q?5DgyurzIkTG5vNxsnW2iX8vSgl7y2NjW5pzx+gbMZHLNBDKzvoMPzoKo7f3G?=
 =?us-ascii?Q?ZIufg6xsT27lS8tIOs/zhJyOAABWvfCzSJFLuDPZGCppeM4Hv365G5lNepCb?=
 =?us-ascii?Q?HprLKFwugYVTLevrR99qbfRQFrYtb/D7MIjdMQWI4sIhZ+udR93C3XboGALk?=
 =?us-ascii?Q?gZKdYx+HMviIUB2DQG83ZxBjuvS70nHfNJSJMRq6rAkofWgyqObB3JoSu7t2?=
 =?us-ascii?Q?I02F/xQMITStel3i9IpXHjaRVr7WCPATkSVYgKjEcsdWGbinxpjs5W/WkR5z?=
 =?us-ascii?Q?lQBgERbhIclb/VbWRSblqDLyWalln8DnHzbNyTj8ISBlq2u7koA9wXBU2ZN7?=
 =?us-ascii?Q?p6RAk/ePtKmOjb+QeiWa0ivK1lPLnlzgXvfnBJaTVAwykVHWOpC/qCT0t2vS?=
 =?us-ascii?Q?frAHORjgf+oVy/scvkwKQNAe9IFmbExiNWEbrswBhm0zidoHh7esjtk6nWC5?=
 =?us-ascii?Q?VJ392yN38PAtDCh0cJ9LQqCl5+5NQ7056v97KcSQcCztVCbrh2UH6BHt0so7?=
 =?us-ascii?Q?AX1rLmixSDSee4Cj6Lx5UhT7QSKiQU76SJsP4Mx2QWL2ALsMTEQltCIWJDQs?=
 =?us-ascii?Q?MAqH+0CFXaH1g/pOleYTrYNL5XLUBP7CSC7AucLp2J1gUEIYCO5mSS4zLbsw?=
 =?us-ascii?Q?4SWN6qldpoRmsNrfQJfPwenJ/JRduX6axbcnROV/MqYgC+mPELrPHphqgXI5?=
 =?us-ascii?Q?nA/GQtPzoZyki8d38IHJo+HqWjlmrbo3bQkgExy2Dr4KDWqnacAGGJAeb6c3?=
 =?us-ascii?Q?kx8PRfHEzgMMZhvNqOVmdYqywsPViAtjYzb8AwJCUzuMlBp6+5hpNePLTiIz?=
 =?us-ascii?Q?eQWzXh27uv0UnmYjvbQhDrfagyCLCgDq7XLJKzc4ehHJiaMC/o66/fQXXBbp?=
 =?us-ascii?Q?AxY7Lt47i9rSMZm8j/eYgU0eJynDI3LiUcOGQMKBUiM+gcESV3irUYh5zD/W?=
 =?us-ascii?Q?2ZKTcQezh7PYDVqPLS+ZWkC5O5zc50Zbpk6XGrcuZIaBGz/4reNBkJigFWzA?=
 =?us-ascii?Q?nUJonyXwtwdL/U8NziXSDuAe8ZKbhCa7IQHE3dxlRfQ2YceEftLAXUlE721S?=
 =?us-ascii?Q?i/8+WP8zqCTBZvWeu+aKU7uygdMAu2Oqy/NItC5HcRV9/1a8WPI2oaeEXiUN?=
 =?us-ascii?Q?XBy38fvYcnuHQfEx/YE66HaHudP2HBBV8yDt+sXDJcNMxE8PYwtsHuAslLBA?=
 =?us-ascii?Q?AdnYu8TOqup0zL37VWkI//sRAiWkzKvsI57ZdwmR64X5j7OedyrJHsa4z7PC?=
 =?us-ascii?Q?cVZ9W7lDyrZ7xdLUTWceerY3btVgT0VgCdf8JgzcuEcaVZis9UND5r0gXklZ?=
 =?us-ascii?Q?5BcDhLlyG3ShNGeo4/fhZODDoNF4hSZ/AlbLlv8e0xtpLApCIFuWRQVYxt3K?=
 =?us-ascii?Q?a1xVNBgib44M2rC8xCFdQKf1z+I8fzMpez3lr1iH72gnH7s3bzJ5EAtxNgD/?=
 =?us-ascii?Q?iqeB9A3zMnNoWRsWsmSBMP8pCPwrcTCC5G5VSGImnGwWhiNKQNDP3u5QpI5B?=
 =?us-ascii?Q?olC3f3gaMJIjA9kCD407eQU=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <AB1BA3F5C3882B4AB11CBCE81A263AF4@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH0PR10MB4858.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9842bca8-d13b-4aab-36bf-08d9d7710e02
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Jan 2022 15:18:01.4832
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 7kf70TpJjg88ZBrEsOx6Q05KEtLY4WNCY+snpN6Q1h6venkCxYk++hBjO1bSbA0nqJlayvIYc5G0/ObmBSFubg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5647
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10226 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 phishscore=0
 mlxlogscore=999 spamscore=0 bulkscore=0 malwarescore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201140102
X-Proofpoint-GUID: uiAT3yaeFKg8BgvaSFiou0n8lCtDZVxu
X-Proofpoint-ORIG-GUID: uiAT3yaeFKg8BgvaSFiou0n8lCtDZVxu
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Jonathan-

> On Jan 14, 2022, at 5:39 AM, Jonathan Woithe <jwoithe@just42.net> wrote:
>=20
> Hi all
>=20
> Recently we migrated an NFS server from a 32-bit environment running=20
> kernel 4.14.128 to a 64-bit 5.15.x kernel.  The NFS configuration remaine=
d
> unchanged between the two systems.
>=20
> On two separate occasions since the upgrade (5 Jan under 5.15.10, 14 Jan
> under 5.15.12) the kernel has oopsed at around the time that an NFS clien=
t
> machine is turned on for the day.  On both occasions the call trace was
> essentially identical.  The full oops sequence is at the end of this emai=
l.=20
> The oops was not observed when running the 4.14.128 kernel.
>=20
> Is there anything more I can provide to help track down the cause of the
> oops?

A possible culprit is 7f024fcd5c97 ("Keep read and write fds with each nlm_=
file"),
which was introduced in or around v5.15. You could try a simple test and ba=
ck
the server down to v5.14.y to see if the problem persists.

Otherwise, Bruce, can you have a look at this?


> Regards
>  jonathan
>=20
> Oops under 5.15.12:
>=20
> Jan 14 08:48:30 nfssvr kernel: BUG: kernel NULL pointer dereference, addr=
ess: 0000000000000110
> Jan 14 08:48:30 nfssvr kernel: #PF: supervisor read access in kernel mode
> Jan 14 08:48:30 nfssvr kernel: #PF: error_code(0x0000) - not-present page
> Jan 14 08:48:30 nfssvr kernel: Oops: 0000 [#1] PREEMPT SMP PTI
> Jan 14 08:48:30 nfssvr kernel: CPU: 0 PID: 2935 Comm: lockd Not tainted 5=
.15.12 #1
> Jan 14 08:48:30 nfssvr kernel: Hardware name:  /DG31PR, BIOS PRG3110H.86A=
.0038.2007.1221.1757 12/21/2007
> Jan 14 08:48:30 nfssvr kernel: RIP: 0010:vfs_lock_file+0x5/0x30
> Jan 14 08:48:30 nfssvr kernel: Code: ff ff 41 89 c4 85 c0 0f 84 42 ff ff =
ff e9 f8 fe ff ff 0f 0b e8 2c bc d2 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 =
0f 1f 44 00 00 <48> 8b 47 28 49 89 d0 48 8b 80 98 00 00 00 48 85 c0 74 05 e=
9 f3 dc
> Jan 14 08:48:30 nfssvr kernel: RSP: 0018:ffffa478401a3c38 EFLAGS: 0001024=
6
> Jan 14 08:48:30 nfssvr kernel: RAX: 7fffffffffffffff RBX: 00000000000000e=
8 RCX: 0000000000000000
> Jan 14 08:48:30 nfssvr kernel: RDX: ffffa478401a3c40 RSI: 000000000000000=
6 RDI: 00000000000000e8
> Jan 14 08:48:30 nfssvr kernel: RBP: ffff946ead1ecc00 R08: ffff946f88ab100=
0 R09: ffff946f88b33a00
> Jan 14 08:48:30 nfssvr kernel: R10: 0000000000000000 R11: 000000000000000=
0 R12: ffffffffa657ff30
> Jan 14 08:48:30 nfssvr kernel: R13: ffff946e99df7c40 R14: ffff946e82fb051=
0 R15: ffff946ead1ecc00
> Jan 14 08:48:30 nfssvr kernel: FS:  0000000000000000(0000) GS:ffff946fabc=
00000(0000) knlGS:0000000000000000
> Jan 14 08:48:30 nfssvr kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008=
0050033
> Jan 14 08:48:30 nfssvr kernel: CR2: 0000000000000110 CR3: 000000010083a00=
0 CR4: 00000000000006f0
> Jan 14 08:48:30 nfssvr kernel: Call Trace:
> Jan 14 08:48:30 nfssvr kernel:  <TASK>
> Jan 14 08:48:30 nfssvr kernel:  nlm_unlock_files+0x6e/0xb0
> Jan 14 08:48:30 nfssvr kernel:  ? __skb_recv_udp+0x198/0x330
> Jan 14 08:48:30 nfssvr kernel:  ? _raw_spin_lock+0x13/0x2e
> Jan 14 08:48:30 nfssvr kernel:  ? nlmsvc_traverse_blocks+0x36/0x120
> Jan 14 08:48:30 nfssvr kernel:  ? preempt_count_add+0x68/0xa0
> Jan 14 08:48:30 nfssvr kernel:  nlm_traverse_files+0x152/0x280
> Jan 14 08:48:30 nfssvr kernel:  nlmsvc_free_host_resources+0x27/0x40
> Jan 14 08:48:30 nfssvr kernel:  nlm_host_rebooted+0x23/0x90
> Jan 14 08:48:30 nfssvr kernel:  nlmsvc_proc_sm_notify+0xae/0x110
> Jan 14 08:48:30 nfssvr kernel:  ? nlmsvc_decode_reboot+0x8b/0xc0
> Jan 14 08:48:30 nfssvr kernel:  nlmsvc_dispatch+0x89/0x180
> Jan 14 08:48:30 nfssvr kernel:  svc_process_common+0x3ce/0x6f0
> Jan 14 08:48:30 nfssvr kernel:  ? lockd_inet6addr_event+0xf0/0xf0
> Jan 14 08:48:30 nfssvr kernel:  svc_process+0xb7/0xf0
> Jan 14 08:48:30 nfssvr kernel:  lockd+0xca/0x1b0
> Jan 14 08:48:30 nfssvr kernel:  ? preempt_count_add+0x68/0xa0
> Jan 14 08:48:30 nfssvr kernel:  ? _raw_spin_lock_irqsave+0x19/0x40
> Jan 14 08:48:30 nfssvr kernel:  ? set_grace_period+0x90/0x90
> Jan 14 08:48:30 nfssvr kernel:  kthread+0x141/0x170
> Jan 14 08:48:30 nfssvr kernel:  ? set_kthread_struct+0x40/0x40
> Jan 14 08:48:30 nfssvr kernel:  ret_from_fork+0x22/0x30
> Jan 14 08:48:30 nfssvr kernel:  </TASK>
> Jan 14 08:48:30 nfssvr kernel: Modules linked in: tun nf_nat_ftp nf_connt=
rack_ftp xt_REDIRECT xt_nat xt_conntrack xt_tcpudp xt_NFLOG nfnetlink_log n=
fnetlink iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipta=
ble_filter ip_tables x_tables ipv6 hid_generic usbhid hi
> Jan 14 08:48:30 nfssvr kernel: CR2: 0000000000000110
> Jan 14 08:48:30 nfssvr kernel: ---[ end trace f8f28acee6f24340 ]---
> Jan 14 08:48:30 nfssvr kernel: RIP: 0010:vfs_lock_file+0x5/0x30
> Jan 14 08:48:30 nfssvr kernel: Code: ff ff 41 89 c4 85 c0 0f 84 42 ff ff =
ff e9 f8 fe ff ff 0f 0b e8 2c bc d2 00 66 66 2e 0f 1f 84 00 00 00 00 00 90 =
0f 1f 44 00 00 <48> 8b 47 28 49 89 d0 48 8b 80 98 00 00 00 48 85 c0 74 05 e=
9 f3 dc
> Jan 14 08:48:30 nfssvr kernel: RSP: 0018:ffffa478401a3c38 EFLAGS: 0001024=
6
> Jan 14 08:48:30 nfssvr kernel: RAX: 7fffffffffffffff RBX: 00000000000000e=
8 RCX: 0000000000000000
> Jan 14 08:48:30 nfssvr kernel: RDX: ffffa478401a3c40 RSI: 000000000000000=
6 RDI: 00000000000000e8
> Jan 14 08:48:30 nfssvr kernel: RBP: ffff946ead1ecc00 R08: ffff946f88ab100=
0 R09: ffff946f88b33a00
> Jan 14 08:48:30 nfssvr kernel: R10: 0000000000000000 R11: 000000000000000=
0 R12: ffffffffa657ff30
> Jan 14 08:48:30 nfssvr kernel: R13: ffff946e99df7c40 R14: ffff946e82fb051=
0 R15: ffff946ead1ecc00
> Jan 14 08:48:30 nfssvr kernel: FS:  0000000000000000(0000) GS:ffff946fabc=
00000(0000) knlGS:0000000000000000
> Jan 14 08:48:30 nfssvr kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008=
0050033
> Jan 14 08:48:30 nfssvr kernel: CR2: 0000000000000110 CR3: 000000010083a00=
0 CR4: 00000000000006f0

--
Chuck Lever