Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Thorsten Leemhuis <regressions@leemhuis.info>
CC:     Jeff Layton <jlayton@kernel.org>, Robert Dinse <nanook@eskimo.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        "regressions@lists.linux.dev" <regressions@lists.linux.dev>
Subject: Re: Bug 216560 - Kernel 6.0.0 Use after Free Error in NFS code
Thread-Topic: Bug 216560 - Kernel 6.0.0 Use after Free Error in NFS code
Thread-Index: AQHY3V5yhuX6s0vXrEqqnTT8ytFPh64JI/CA
Date:   Tue, 11 Oct 2022 12:45:28 +0000
Message-ID: <E0E7F585-3D01-4A76-B972-DF3D8ACD8B36@oracle.com>
References: <2b693b7b-aadd-5353-154c-37a175fc5b99@leemhuis.info>
In-Reply-To: <2b693b7b-aadd-5353-154c-37a175fc5b99@leemhuis.info>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?QkQmIKUt9BfXDU3553Kexop1ZXPSv/PlrJexhcbScQ1oXwjRXquI3uE+BFZe?=
 =?us-ascii?Q?3v3gMeomcSzWKa+RfDUBXxUt1lJQJLw6p3d1XlrdkZqOYlZ1foxZc1iBLd3e?=
 =?us-ascii?Q?KALOVfQUQ8nSfs3TKZoIgJv5MPVILHsns98z7cyAjFUE/rEzaFik8GPtr9aW?=
 =?us-ascii?Q?qyLo03/U+lRfYHlhfE99gIoru00NT2W6jTbaXYTeiVutPZcj9MrdVYNtsr3T?=
 =?us-ascii?Q?2Ht1lHbJbr2p+o3jFK//q35eulFwT920/dqWPpIlCAsVbgNxPRAaQkdXoOLz?=
 =?us-ascii?Q?S5q1saEfbu6w+ocpzZsfFRkdXoJ01lJMLnxaZRtDC0Iq7KF3fZKtyUG5T3aS?=
 =?us-ascii?Q?/IlEkscRTDiV5dD+lrfejYGxULaYhfHnukGcInfjY6pVL4nGGtIOPLNfmjZP?=
 =?us-ascii?Q?WHYq5m0EaGOn80T6P+luYoSEgbvKJdiYw1qGE9akP8Kl1kiXFlGKp3gLi5Yx?=
 =?us-ascii?Q?IrESb0p5KIYxKM/WEPrZOxYhJ7GDfKPULeSJunVkH+Bm1GVufhLjoHyKreTo?=
 =?us-ascii?Q?N2/5gl33oYewiSlmYt/TNxGE1DS0g1BjOIeguz0QkFnLLZoPX7w0rfONbD4k?=
 =?us-ascii?Q?8JSdGoa4jYaUZPDA7V+/X0eGduf0lBTFeZ/bfXz125jjiM9473UEX/kBtoD5?=
 =?us-ascii?Q?12UweaxxnW+I758yxq86XSZEjEHk3ic7g83ox0Drv6E4wwZXO9FG92wfE20N?=
 =?us-ascii?Q?7+O+21BKVnwtS7CRvn2OJGpqUjPMl5+vnKr7Bh8DI4qWK9ygfwF9JPfYPWTO?=
 =?us-ascii?Q?0D7EuPuXA71giZemPP1TPtVW443nC8Rd32lCJ3FK3Z3fwBUoyDWi4Guv3hHj?=
 =?us-ascii?Q?fwqkukozGMowx9RKIrJkEf3+YFnFAuu2iN7n9o9UtnvgEi1T1G13gP32XOvZ?=
 =?us-ascii?Q?8vHOL5mWwuhBti/48fuV0hvWbewKbJRW4M9Rb8tVRP/JiZpNW9Px3JbpKogd?=
 =?us-ascii?Q?HOXi0YQhkCQNm3P7W76FoylyiLZFaImLvE6ZXMPRTDDIOsO65iGtRXW8OGFD?=
 =?us-ascii?Q?jVfGtrRkydn9RT9Mbz81hE+w+V14rmvMPpiSo98MRd9J2y3e7bQCChWVHLLJ?=
 =?us-ascii?Q?0j8sh457an28y7HaLyRpKGryxa/iYj9pCIaRZR2Gmcfwca9mW9PDQdEqEg7k?=
 =?us-ascii?Q?qbNbRCRSA97dBGs7Zu/LLmm5t5NBmF2PjZInVsO1nb0nrScEZc609w7tl73R?=
 =?us-ascii?Q?LSk+xLuk7iJvBU8I7CEjlNuFbB+Y3Tpi9axVv8zQCdA4+V1FJkr+ZsOU8z58?=
 =?us-ascii?Q?1vkRog8BB88lSSPhoLFK3UkzK0WQz671uKM92+Ao5ORqDWJYEbBbU9nD7+Ir?=
 =?us-ascii?Q?+FY4XjJDFOG8vGMWllJla5WxcQQC/t9HJXQ+DLAptGl6JTLAYZccB7pNWczE?=
 =?us-ascii?Q?qX05VyAL1UUfKd0r1PcfgSvFWafeVBxPNA+r+i5kNmuipIETgX9fBeb1Llv8?=
 =?us-ascii?Q?bW/WVzMBzm8BPPRYzuk8BlLwMvSmnStAjPL0+4cCyeS+EYVCrKJvYAuv6xEC?=
 =?us-ascii?Q?1FwXkz4Fci3ru0WRpFhhEIPefWqJdlWlK1+jIstI7bq6U2kChhJI2imYii5q?=
 =?us-ascii?Q?tKrrSghP1b6DW1xnH0hIt7oDJrokxAwlNT5ZOc5uN9eAJANZ/CrOKuVs7eCX?=
 =?us-ascii?Q?4g=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D12E4F59C2DC2645A8E0439F5733BF4D@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6012687d-fc6b-45ef-32fa-08daab8679ea
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Oct 2022 12:45:28.5392
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: BkzWiVMe3D0E54j2i9ldX5JYzwC5vD7Ouky0Fi8saOfbPfmXPuKZBv5yx83G5BIUN277XWSv5ygdT+4P2+PhPw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR10MB6064
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-11_07,2022-10-11_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0
 phishscore=0 mlxscore=0 malwarescore=0 spamscore=0 mlxlogscore=999
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2209130000 definitions=main-2210110072
X-Proofpoint-GUID: __ZgSCoLOv4P22jgm8ef4exyx8UIUERW
X-Proofpoint-ORIG-GUID: __ZgSCoLOv4P22jgm8ef4exyx8UIUERW
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Oct 11, 2022, at 6:44 AM, Thorsten Leemhuis <regressions@leemhuis.info=
> wrote:
>=20
> Hi, this is your Linux kernel regression tracker speaking.
>=20
> I noticed a regression report in bugzilla.kernel.org. As many (most?)
> kernel developer don't keep an eye on it, I decided to forward it by
> mail. Quoting from https://bugzilla.kernel.org/show_bug.cgi?id=3D216560 :
>=20
>> reated attachment 302961 [details]
>> Config file used ot compile 6.0.0
>>=20
>> NFS Use after Free Call Trace.
>> This is on an i7-6850k based server that is running Ubuntu 22.04 and ser=
ves as an NFS server as well as virtual machine host.  It is equipped with =
128GB of RAM and both SATA hard drives and PCI flash drives for storage.  T=
he configuration file used to compile this kernel is attached as config-6.0=
.0.  Anything else you need please let me know.  Compiled with gcc 12.2.0.
>>=20
>>=20
>> [45414.551153] ------------[ cut here ]------------
>> [45414.551158] refcount_t: underflow; use-after-free.
>> [45414.551169] WARNING: CPU: 11 PID: 2289 at lib/refcount.c:87 refcount_=
dec_not_one+0x68/0x70
>> [45414.551177] Modules linked in: vhost_net vhost vhost_iotlb tap tcp_di=
ag inet_diag unix_diag xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf=
_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defr=
ag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink rpcsec_gss_krb5 nfsv4 nfs fscach=
e netfs bridge stp llc binfmt_misc nls_iso8859_1 quota_v2 quota_tree snd_hd=
a_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel inte=
l_rapl_msr snd_intel_dspcfg snd_intel_sdw_acpi intel_rapl_common intel_unco=
re_frequency intel_uncore_frequency_common snd_hda_codec snd_hda_core x86_p=
kg_temp_thermal intel_powerclamp snd_hwdep snd_pcm snd_seq_midi snd_seq_mid=
i_event snd_rawmidi snd_seq snd_seq_device snd_timer eeepc_wmi mei_me kvm_i=
ntel snd asus_wmi kvm platform_profile ledtrig_audio iTCO_wdt intel_pmc_bxt=
 sparse_keymap rapl intel_cstate mei intel_wmi_thunderbolt iTCO_vendor_supp=
ort soundcore wmi_bmof mac_hid sch_fq_codel nct6775 nct6775_core hwmon_vid =
coretemp nfsd parport_pc ppdev
>> [45414.551237]  auth_rpcgss nfs_acl lp lockd parport grace sunrpc ramoop=
s reed_solomon efi_pstore pstore_blk pstore_zone ip_tables x_tables autofs4=
 btrfs blake2b_generic zstd_compress raid456 async_raid6_recov async_memcpy=
 async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear =
dm_mirror dm_region_hash dm_log raid1 nouveau video i2c_algo_bit drm_ttm_he=
lper ttm drm_display_helper cec rc_core drm_kms_helper syscopyarea sysfillr=
ect sysimgblt fb_sys_fops raid10 crct10dif_pclmul crc32_pclmul ghash_clmuln=
i_intel mxm_wmi aesni_intel crypto_simd cryptd drm nvme e1000e i2c_i801 ahc=
i lpc_ich i2c_smbus r8169 xhci_pci nvme_core libahci xhci_pci_renesas realt=
ek wmi
>> [45414.551282] CPU: 11 PID: 2289 Comm: nfsd Not tainted 6.0.0 #1
>> [45414.551284] Hardware name: ASUS All Series/X99-E, BIOS 2101 07/10/201=
9
>> [45414.551286] RIP: 0010:refcount_dec_not_one+0x68/0x70
>> [45414.551288] Code: d3 6e 00 83 e3 01 74 0f 48 8b 5d f8 b8 01 00 00 00 =
c9 c3 cc cc cc cc 48 c7 c7 38 1a 64 99 c6 05 36 65 53 01 01 e8 bf c4 6a 00 =
<0f> 0b eb da 0f 1f 40 00 55 48 89 e5 41 54 49 89 f4 53 48 89 fb e8
>> [45414.551290] RSP: 0018:ffffa6f041a5bb80 EFLAGS: 00010282
>> [45414.551292] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000=
00000027
>> [45414.551293] RDX: ffff89763fce0568 RSI: 0000000000000001 RDI: ffff8976=
3fce0560
>> [45414.551295] RBP: ffffa6f041a5bb88 R08: 0000000000000000 R09: ffffa6f0=
41a5bb18
>> [45414.551296] R10: 00000000ffffe664 R11: 0000000000000665 R12: ffff8963=
d049b1a0
>> [45414.551297] R13: ffff8963d049b1a0 R14: 0000000000000000 R15: ffff8972=
18d09e90
>> [45414.551298] FS:  0000000000000000(0000) GS:ffff89763fcc0000(0000) knl=
GS:0000000000000000
>> [45414.551300] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [45414.551301] CR2: 00007fea43472718 CR3: 000000015686a005 CR4: 00000000=
003726e0
>> [45414.551303] Call Trace:
>> [45414.551304]  <TASK>
>> [45414.551308]  __nfsd_file_close_inode+0x1ca/0x210 [nfsd]
>> [45414.551340]  nfsd_file_fsnotify_handle_event+0x85/0x120 [nfsd]
>> [45414.551362]  fsnotify_handle_inode_event.isra.0+0x6f/0xe0
>> [45414.551365]  fsnotify+0x615/0x9b0
>> [45414.551368]  vfs_unlink+0x22d/0x2d0
>> [45414.551372]  nfsd_unlink+0x128/0x2f0 [nfsd]
>> [45414.551394]  nfsd4_remove+0x52/0x100 [nfsd]
>> [45414.551411]  nfsd4_proc_compound+0x431/0x6f0 [nfsd]
>> [45414.551425]  ? nfs4svc_decode_compoundargs+0x380/0x410 [nfsd]
>> [45414.551440]  nfsd_dispatch+0x15a/0x250 [nfsd]
>> [45414.551451]  svc_process_common+0x295/0x600 [sunrpc]
>> [45414.551485]  ? nfsd_svc+0x3f0/0x3f0 [nfsd]
>> [45414.551504]  ? nfsd_shutdown_threads+0xa0/0xa0 [nfsd]
>> [45414.551522]  svc_process+0xba/0x100 [sunrpc]
>> [45414.551544]  nfsd+0xd1/0x190 [nfsd]
>> [45414.551555]  kthread+0xd1/0xf0
>> [45414.551558]  ? kthread_complete_and_exit+0x20/0x20
>> [45414.551560]  ret_from_fork+0x22/0x30
>> [45414.551563]  </TASK>
>> [45414.551563] ---[ end trace 0000000000000000 ]---
>=20
> See the ticket for more details.
>=20
> Side note: I wonder if these two are related, but this is not my area of
> expertise, so be warned that it might be something totally different:
> https://lore.kernel.org/all/166486048770.14457.133971372966856907@noble.n=
eil.brown.name/
> https://bugzilla.linux-nfs.org/show_bug.cgi?id=3D394

The reported trace looks similar to 394, but this is a new
and unfamiliar failure mode (to my eyes).

As Jeff said we are actively working on filecache refcounting
issues. The challenge is that none of the bug reports include
a reliable reproducer.

I should also mention that we believe some or all of these
issues predate the filecache overhaul done in 6.0, so I'm
not ready to confirm this as a regression in that kernel.
We've seen similar failures reported in kernels as old as
5.15.

Thus my intention is to pursue a conservative approach for
now, and fix this for v6.2 rather than pushing it into v6.1.


> BTW, let me use this mail to also add the report to the list of tracked
> regressions to ensure it's doesn't fall through the cracks:
>=20
> #regzbot introduced: v5.19..v6.0
> https://bugzilla.kernel.org/show_bug.cgi?id=3D216560
> #regzbot ignore-activity
>=20
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>=20
> P.S.: As the Linux kernel's regression tracker I deal with a lot of
> reports and sometimes miss something important when writing mails like
> this. If that's the case here, don't hesitate to tell me in a public
> reply, it's in everyone's interest to set the public record straight.

--
Chuck Lever