LinuxLists.cc - linux 5.14.3: free_user_ns causes NULL pointer dereference

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

Jordan Glover <[email protected]> writes:

> Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window
>
> https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u

So that warning comes from:

void dec_ucount(struct ucounts *ucounts, enum ucount_type type)
{
struct ucounts *iter;
for (iter = ucounts; iter; iter = iter->ns->ucounts) {
long dec = atomic_long_dec_if_positive(&iter->ucount[type]);
WARN_ON_ONCE(dec < 0);
}
put_ucounts(ucounts);
}

Which certainly looks like a reference count bug. It could also be a
memory stomp somewhere close.

Do you have any idea what else was going on? This location is the
symptom but not the actual cause.

Eric

>
> Logs below:
>
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 26546 at kernel/ucount.c:253 dec_ucount+0x43/0x50
> Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_magn_3d hid_sensor_rotation hid_sensor_accel_3d hid_sensor_gyro_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader nft_counter intel_ishtp_hid snd_hda_codec_hdmi intel_rapl_msr xt_mark ipt_REJECT nf_reject_ipv4 snd_ctl_led xt_LOG snd_hda_codec_conexant nf_log_syslog snd_hda_codec_generic xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv4 mei_hdcp snd_hda_intel nft_compat wmi_bmof nf_tables intel_rapl_common libcrc32c think_lmi intel_tcc_cooling snd_intel_dspcfg firmware_attributes_class nfnetlink iwlmvm
> intel_wmi_thunderbolt mac80211 x86_pkg_temp_thermal snd_hda_codec intel_powerclamp coretemp libarc4 vfat fat kvm_intel rapl intel_cstate snd_hwdep intel_uncore iwlwifi snd_hda_core mousedev joydev snd_pcm psmouse cfg80211 snd_timer mei_me ucsi_acpi wacom intel_ish_ipc intel_xhci_usb_role_switch mei intel_pch_thermal typec_ucsi roles typec intel_ishtp wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore rfkill tpm_crb i2c_hid_acpi i2c_hid acpi_pad tpm_tis mac_hid tpm_tis_core pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd xhci_pci rtsx_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec
> drm kvm irqbypass
> CPU: 1 PID: 26546 Comm: kworker/1:1 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
> Workqueue: events free_user_ns
> RIP: 0010:dec_ucount+0x43/0x50
> Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
> RSP: 0018:ffffa82cc2bd7e60 EFLAGS: 00010297
> RAX: 0000000000000000 RBX: ffffa2f53298ee50 RCX: ffffa2f3c0061000
> RDX: ffffa2f3c0061020 RSI: ffffffffffffffff RDI: ffffa2f3c0061000
> RBP: ffffa2f53298ebe0 R08: 0000000000000020 R09: 0000000000000000
> R10: 0000000000000001 R11: 0000000000000000 R12: ffffa2f3c0061000
> R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> FS: 0000000000000000(0000) GS:ffffa2f599680000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000628f892be9f8 CR3: 000000002880e004 CR4: 00000000003706e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> free_user_ns+0x73/0x110
> process_one_work+0x1e1/0x380
> worker_thread+0x50/0x3a0
> ? rescuer_thread+0x360/0x360
> kthread+0x127/0x150
> ? set_kthread_struct+0x40/0x40
> ret_from_fork+0x22/0x30
> ---[ end trace eb7a8d38b64b2d3a ]---
> BUG: kernel NULL pointer dereference, address: 00000000000001e8
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> Oops: 0000 [#1] SMP PTI
> CPU: 1 PID: 26546 Comm: kworker/1:1 Tainted: G W 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
> Workqueue: events free_user_ns
> RIP: 0010:dec_ucount+0x32/0x50
> Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
> RSP: 0018:ffffa82cc2bd7e60 EFLAGS: 00010297
> RAX: 0000000000000000 RBX: ffffa2f53298ee50 RCX: ffffa2f3c0061000
> RDX: ffffa2f3c0061020 RSI: ffffffffffffffff RDI: ffffa2f3c0061000
> RBP: ffffa2f53298ebe0 R08: 0000000000000020 R09: 0000000000000000
> R10: 0000000000000001 R11: 0000000000000000 R12: ffffa2f3c0061000
> R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> FS: 0000000000000000(0000) GS:ffffa2f599680000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000001e8 CR3: 000000002880e004 CR4: 00000000003706e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> free_user_ns+0x73/0x110
> process_one_work+0x1e1/0x380
> worker_thread+0x50/0x3a0
> ? rescuer_thread+0x360/0x360
> kthread+0x127/0x150
> ? set_kthread_struct+0x40/0x40
> ret_from_fork+0x22/0x30

2021-09-15 22:49:10

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:

> Jordan Glover [email protected] writes:
>
> > Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window
> >
> > https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u
>
> So that warning comes from:
>
> void dec_ucount(struct ucounts *ucounts, enum ucount_type type)
>
> {
>
> struct ucounts *iter;
>
> for (iter = ucounts; iter; iter = iter->ns->ucounts) {
>
> long dec = atomic_long_dec_if_positive(&iter->ucount[type]);
>
> WARN_ON_ONCE(dec < 0);
> }
> put_ucounts(ucounts);
>
>
> }
>
> Which certainly looks like a reference count bug. It could also be a
>
> memory stomp somewhere close.
>
> Do you have any idea what else was going on? This location is the
>
> symptom but not the actual cause.
>
> Eric

I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.

Maybe it's some random fluke. I'll reply if I hit it again.

Jordan

2021-09-15 23:46:23

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wed, Sep 15, 2021 at 4:42 PM Jordan Glover
<[email protected]> wrote:
>
> On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:
>
> > Jordan Glover [email protected] writes:
> >
> > > Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window
> > >
> > > https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u
> >
> > So that warning comes from:
> >
> > void dec_ucount(struct ucounts *ucounts, enum ucount_type type)
> >
> > {
> >
> > struct ucounts *iter;
> >
> > for (iter = ucounts; iter; iter = iter->ns->ucounts) {
> >
> > long dec = atomic_long_dec_if_positive(&iter->ucount[type]);
> >
> > WARN_ON_ONCE(dec < 0);
> > }
> > put_ucounts(ucounts);
> >
> >
> > }
> >
> > Which certainly looks like a reference count bug. It could also be a
> >
> > memory stomp somewhere close.
> >
> > Do you have any idea what else was going on? This location is the
> >
> > symptom but not the actual cause.
> >
> > Eric
>
> I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
>
> Maybe it's some random fluke. I'll reply if I hit it again.

I have been able to steadily reproduce this for a while. But I haven't
had time to look into it. I'd appreciate any help.

[ 1324.741762] ==================================================================
[ 1324.749200] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
[ 1324.755198] Write of size 8 at addr ffffff808f2f0120 by task kworker/7:2/1817
[ 1324.762523]
[ 1324.764063] CPU: 7 PID: 1817 Comm: kworker/7:2 Not tainted 5.14.0-lockdep+ #7
[ 1324.771396] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
[ 1324.778098] Workqueue: events free_user_ns
[ 1324.782320] Call trace:
[ 1324.784845] dump_backtrace+0x0/0x42c
[ 1324.788614] show_stack+0x24/0x30
[ 1324.792026] dump_stack_lvl+0xd0/0x100
[ 1324.795884] print_address_description+0x30/0x304
[ 1324.800725] kasan_report+0x190/0x1d8
[ 1324.804494] kasan_check_range+0x1ac/0x1bc
[ 1324.808708] __kasan_check_write+0x44/0x54
[ 1324.812923] dec_ucount+0x50/0xd8
[ 1324.816334] free_user_ns+0x1b0/0x288
[ 1324.820102] process_one_work+0x7b4/0x10ec
[ 1324.824318] worker_thread+0x800/0xcf4
[ 1324.828176] kthread+0x2a8/0x358
[ 1324.831500] ret_from_fork+0x10/0x18
[ 1324.835182]
[ 1324.836720] Allocated by task 5167:
[ 1324.840312] kasan_save_stack+0x38/0x68
[ 1324.844261] __kasan_slab_alloc+0x6c/0x88
[ 1324.848387] kmem_cache_alloc+0x234/0x47c
[ 1324.852505] getname_flags+0xb4/0x3b8
[ 1324.856273] __arm64_sys_unlink+0x4c/0x68
[ 1324.860400] invoke_syscall+0xd4/0x2c8
[ 1324.864258] el0_svc_common+0x124/0x200
[ 1324.868199] do_el0_svc_compat+0x54/0x64
[ 1324.872235] el0_svc_compat+0x24/0x34
[ 1324.876003] el0t_32_sync_handler+0xc0/0xf0
[ 1324.880307] el0t_32_sync+0x19c/0x1a0
[ 1324.884078]
[ 1324.885616] Freed by task 5167:
[ 1324.888848] kasan_save_stack+0x38/0x68
[ 1324.892797] kasan_set_track+0x28/0x3c
[ 1324.896653] kasan_set_free_info+0x28/0x4c
[ 1324.900865] ____kasan_slab_free+0x118/0x164
[ 1324.905259] __kasan_slab_free+0x18/0x28
[ 1324.909296] kmem_cache_free+0x1c0/0x4b8
[ 1324.913325] putname+0xd0/0x140
[ 1324.916554] do_unlinkat+0x518/0x57c
[ 1324.920233] __arm64_sys_unlink+0x58/0x68
[ 1324.924359] invoke_syscall+0xd4/0x2c8
[ 1324.928215] el0_svc_common+0x124/0x200
[ 1324.932164] do_el0_svc_compat+0x54/0x64
[ 1324.936195] el0_svc_compat+0x24/0x34
[ 1324.939963] el0t_32_sync_handler+0xc0/0xf0
[ 1324.944259] el0t_32_sync+0x19c/0x1a0
[ 1324.948029]
[ 1324.949567] The buggy address belongs to the object at ffffff808f2f0040
[ 1324.949567] which belongs to the cache names_cache of size 4096
[ 1324.962498] The buggy address is located 224 bytes inside of
[ 1324.962498] 4096-byte region [ffffff808f2f0040, ffffff808f2f1040)
[ 1324.974635] The buggy address belongs to the page:
[ 1324.979561] page:fffffffe023cbc00 refcount:1 mapcount:0
mapping:0000000000000000 index:0xffffff808f2f23c0 pfn:0x10f2f0
[ 1324.990536] head:fffffffe023cbc00 order:3 compound_mapcount:0
compound_pincount:0
[ 1324.998220] flags: 0x8000000000010200(slab|head|zone=2)
[ 1325.003596] raw: 8000000000010200 fffffffe02b59a08 fffffffe0247a208
ffffff808036ad80
[ 1325.011549] raw: ffffff808f2f23c0 0000000000070006 00000001ffffffff
0000000000000000
[ 1325.019494] page dumped because: kasan: bad access detected
[ 1325.025214]
[ 1325.026751] Memory state around the buggy address:
[ 1325.031680] ffffff808f2f0000: fc fc fc fc fc fc fc fc fa fb fb fb
fb fb fb fb
[ 1325.039093] ffffff808f2f0080: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1325.046506] >ffffff808f2f0100: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1325.053920] ^
[ 1325.058311] ffffff808f2f0180: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1325.065725] ffffff808f2f0200: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1325.073153] ==================================================================

[ 7358.532846] ==================================================================
[ 7358.540369] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
[ 7358.546429] Write of size 8 at addr ffffff800af71120 by task
kworker/3:1/27623
[ 7358.553880]
[ 7358.555458] CPU: 3 PID: 27623 Comm: kworker/3:1 Not tainted
5.14.0-lockdep+ #13
[ 7358.563013] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
[ 7358.569761] Workqueue: events free_user_ns
[ 7358.574039] Call trace:
[ 7358.576595] dump_backtrace+0x0/0x42c
[ 7358.580402] show_stack+0x24/0x30
[ 7358.583842] dump_stack_lvl+0xd0/0x100
[ 7358.587732] print_address_description+0x30/0x304
[ 7358.592608] kasan_report+0x190/0x1d8
[ 7358.596405] kasan_check_range+0x1ac/0x1bc
[ 7358.600643] __kasan_check_write+0x44/0x54
[ 7358.604876] dec_ucount+0x50/0xd8
[ 7358.608317] free_user_ns+0x1b0/0x288
[ 7358.612106] process_one_work+0x7b4/0x10ec
[ 7358.616354] worker_thread+0x800/0xcf4
[ 7358.620234] kthread+0x2a8/0x358
[ 7358.623585] ret_from_fork+0x10/0x18
[ 7358.627297]
[ 7358.628861] Allocated by task 26538:
[ 7358.632556] kasan_save_stack+0x38/0x68
[ 7358.636537] __kasan_kmalloc+0x9c/0xb8
[ 7358.640424] kmem_cache_alloc_trace+0x2a4/0x370
[ 7358.645104] fscrypt_setup_filename+0x3bc/0x754
[ 7358.649793] ext4_find_entry+0x80/0x240
[ 7358.653774] __ext4_unlink+0x88/0x594
[ 7358.657563] ext4_unlink+0x458/0x99c
[ 7358.661263] vfs_unlink+0x22c/0x340
[ 7358.664877] do_unlinkat+0x358/0x57c
[ 7358.668586] __arm64_sys_unlinkat+0xa8/0xc4
[ 7358.672920] invoke_syscall+0xd4/0x2c8
[ 7358.676808] el0_svc_common+0x124/0x200
[ 7358.680781] do_el0_svc_compat+0x54/0x64
[ 7358.684835] el0_svc_compat+0x24/0x34
[ 7358.688640] el0t_32_sync_handler+0xc0/0xf0
[ 7358.692973] el0t_32_sync+0x19c/0x1a0
[ 7358.696772]
[ 7358.698331] Freed by task 26538:
[ 7358.701668] kasan_save_stack+0x38/0x68
[ 7358.705646] kasan_set_track+0x28/0x3c
[ 7358.709533] kasan_set_free_info+0x28/0x4c
[ 7358.713772] ____kasan_slab_free+0x118/0x164
[ 7358.718186] __kasan_slab_free+0x18/0x28
[ 7358.722240] kfree+0x2f8/0x500
[ 7358.725403] ext4_find_entry+0x19c/0x240
[ 7358.729469] __ext4_unlink+0x88/0x594
[ 7358.733265] ext4_unlink+0x458/0x99c
[ 7358.736968] vfs_unlink+0x22c/0x340
[ 7358.740587] do_unlinkat+0x358/0x57c
[ 7358.744285] __arm64_sys_unlinkat+0xa8/0xc4
[ 7358.748610] invoke_syscall+0xd4/0x2c8
[ 7358.752495] el0_svc_common+0x124/0x200
[ 7358.756464] do_el0_svc_compat+0x54/0x64
[ 7358.760522] el0_svc_compat+0x24/0x34
[ 7358.764320] el0t_32_sync_handler+0xc0/0xf0
[ 7358.768653] el0t_32_sync+0x19c/0x1a0
[ 7358.772453]
[ 7358.774015] Last potentially related work creation:
[ 7358.779053] kasan_save_stack+0x38/0x68
[ 7358.783025] kasan_record_aux_stack+0xdc/0x108
[ 7358.787619] call_rcu+0x180/0xed4
[ 7358.791061] __ip6_del_rt+0xfc/0x130
[ 7358.794771] ip6_route_del+0xa74/0xb74
[ 7358.798653] inet6_rtm_delroute+0x31c/0x3e4
[ 7358.802986] rtnetlink_rcv_msg+0x3d0/0x928
[ 7358.807242] netlink_rcv_skb+0x164/0x2ac
[ 7358.811311] rtnetlink_rcv+0x24/0x30
[ 7358.815025] netlink_unicast+0x328/0x518
[ 7358.819083] netlink_sendmsg+0x5c0/0x934
[ 7358.823142] sock_sendmsg+0xb8/0xdc
[ 7358.826764] __sys_sendto+0x218/0x3e4
[ 7358.830557] __arm64_sys_send+0xac/0xc8
[ 7358.834526] invoke_syscall+0xd4/0x2c8
[ 7358.838413] el0_svc_common+0x124/0x200
[ 7358.842382] do_el0_svc_compat+0x54/0x64
[ 7358.846443] el0_svc_compat+0x24/0x34
[ 7358.850245] el0t_32_sync_handler+0xc0/0xf0
[ 7358.854578] el0t_32_sync+0x19c/0x1a0
[ 7358.858382]
[ 7358.859941] Second to last potentially related work creation:
[ 7358.865862] kasan_save_stack+0x38/0x68
[ 7358.869850] kasan_record_aux_stack+0xdc/0x108
[ 7358.874452] insert_work+0x60/0x22c
[ 7358.878072] __queue_work+0xae8/0xe74
[ 7358.881866] queue_work_on+0x100/0x148
[ 7358.885752] drm_atomic_helper_commit+0x18c/0x1f4
[ 7358.890615] drm_atomic_nonblocking_commit+0xc4/0xdc
[ 7358.895749] drm_mode_atomic_ioctl+0x89c/0xaa0
[ 7358.900353] drm_ioctl_kernel+0x15c/0x260
[ 7358.904505] drm_ioctl+0x460/0x828
[ 7358.908029] drm_compat_ioctl+0x1bc/0x230
[ 7358.912176] __arm64_compat_sys_ioctl+0x1b0/0x1c4
[ 7358.917037] invoke_syscall+0xd4/0x2c8
[ 7358.920919] el0_svc_common+0x124/0x200
[ 7358.924887] do_el0_svc_compat+0x54/0x64
[ 7358.928946] el0_svc_compat+0x24/0x34
[ 7358.932735] el0t_32_sync_handler+0xc0/0xf0
[ 7358.937065] el0t_32_sync+0x19c/0x1a0
[ 7358.940853]
[ 7358.942408] The buggy address belongs to the object at ffffff800af71100
[ 7358.942408] which belongs to the cache kmalloc-256 of size 256
[ 7358.955282] The buggy address is located 32 bytes inside of
[ 7358.955282] 256-byte region [ffffff800af71100, ffffff800af71200)
[ 7358.967268] The buggy address belongs to the page:
[ 7358.972209] page:fffffffe002bdc00 refcount:1 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x8af70
[ 7358.981790] head:fffffffe002bdc00 order:3 compound_mapcount:0
compound_pincount:0
[ 7358.989493] flags: 0x10200(slab|head|zone=0)
[ 7358.993922] raw: 0000000000010200 fffffffe021e0c08 fffffffe02960008
ffffff808000c980
[ 7359.001898] raw: 0000000000000000 0000000000200020 00000001ffffffff
0000000000000000
[ 7359.009867] page dumped because: kasan: bad access detected
[ 7359.015607]
[ 7359.017162] Memory state around the buggy address:
[ 7359.022105] ffffff800af71000: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7359.029543] ffffff800af71080: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7359.036981] >ffffff800af71100: fa fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7359.044414] ^
[ 7359.048818] ffffff800af71180: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7359.056256] ffffff800af71200: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7359.063695] ==================================================================

2021-09-15 23:50:00

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover <[email protected]> wrote:
>
> I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
>
> Maybe it's some random fluke. I'll reply if I hit it again.

Heh, it jut happened again. This time closing firefox alone had such effect:

------------[ cut here ]------------
WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x50
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops cec drm kvm irqbypass
CPU: 1 PID: 351 Comm: kworker/1:3 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x43/0x50
Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056ffceff6b10 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
free_user_ns+0x73/0x110
process_one_work+0x1e1/0x380
worker_thread+0x50/0x3a0
? rescuer_thread+0x360/0x360
kthread+0x127/0x150
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
---[ end trace ff45ac39689f43c1 ]---
BUG: kernel NULL pointer dereference, address: 00000000000001e8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 351 Comm: kworker/1:3 Tainted: G W 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
Workqueue: events free_user_ns
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
free_user_ns+0x73/0x110
process_one_work+0x1e1/0x380
worker_thread+0x50/0x3a0
? rescuer_thread+0x360/0x360
kthread+0x127/0x150
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops cec drm kvm irqbypass
CR2: 00000000000001e8
---[ end trace ff45ac39689f43c2 ]---
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

2021-09-16 17:45:19

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

Jordan Glover <[email protected]> writes:

> On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover <[email protected]> wrote:
>>
>> I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
>>
>> Maybe it's some random fluke. I'll reply if I hit it again.
>
> Heh, it jut happened again. This time closing firefox alone had such
> effect:

Ok. It looks like he have a couple of folks seeing issues here.

I thought we had all of the issues sorted out for the release of v5.14,
but it looks like there is still some little bug left.

If Alex doesn't beat me to it I will see if I can come up with a
debugging patch to make it easy to help track down where the reference
count is going wrong. It will be a little bit as my brain is mush at
the moment.

Eric

> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x50
> Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
> snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
> fb_sys_fops cec drm kvm irqbypass
> CPU: 1 PID: 351 Comm: kworker/1:3 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
> Workqueue: events free_user_ns
> RIP: 0010:dec_ucount+0x43/0x50
> Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
> RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
> RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
> RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
> RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
> R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
> R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000056ffceff6b10 CR3: 0000000147a0e005 CR4: 00000000003706e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> free_user_ns+0x73/0x110
> process_one_work+0x1e1/0x380
> worker_thread+0x50/0x3a0
> ? rescuer_thread+0x360/0x360
> kthread+0x127/0x150
> ? set_kthread_struct+0x40/0x40
> ret_from_fork+0x22/0x30
> ---[ end trace ff45ac39689f43c1 ]---
> BUG: kernel NULL pointer dereference, address: 00000000000001e8
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> Oops: 0000 [#1] SMP PTI
> CPU: 1 PID: 351 Comm: kworker/1:3 Tainted: G W 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
> Workqueue: events free_user_ns
> RIP: 0010:dec_ucount+0x32/0x50
> Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
> RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
> RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
> RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
> RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
> R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
> R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
> free_user_ns+0x73/0x110
> process_one_work+0x1e1/0x380
> worker_thread+0x50/0x3a0
> ? rescuer_thread+0x360/0x360
> kthread+0x127/0x150
> ? set_kthread_struct+0x40/0x40
> ret_from_fork+0x22/0x30
> Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
> snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
> fb_sys_fops cec drm kvm irqbypass
> CR2: 00000000000001e8
> ---[ end trace ff45ac39689f43c2 ]---
> RIP: 0010:dec_ucount+0x32/0x50
> Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
> RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
> RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
> RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
> RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
> R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
> R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

2021-09-17 06:18:52

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Thu, Sep 16, 2021 at 12:30:44PM -0500, Eric W. Biederman wrote:
> Jordan Glover <[email protected]> writes:
>
> > On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover <[email protected]> wrote:
> >>
> >> I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
> >>
> >> Maybe it's some random fluke. I'll reply if I hit it again.
> >
> > Heh, it jut happened again. This time closing firefox alone had such
> > effect:
>
> Ok. It looks like he have a couple of folks seeing issues here.
>
> I thought we had all of the issues sorted out for the release of v5.14,
> but it looks like there is still some little bug left.
>
> If Alex doesn't beat me to it I will see if I can come up with a
> debugging patch to make it easy to help track down where the reference
> count is going wrong. It will be a little bit as my brain is mush at
> the moment.

I'm poking it right now, but so far without much success.

The dec_ucount() happened free_user_ns(). There we only decrease the
UCOUNT_USER_NAMESPACES in the user_ns tree. Perhaps somewhere someone
forgot to do create_user_ns() ?

> Eric
>
> > ------------[ cut here ]------------
> > WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x50
> > Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
> > snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
> > fb_sys_fops cec drm kvm irqbypass
> > CPU: 1 PID: 351 Comm: kworker/1:3 Not tainted 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
> > Workqueue: events free_user_ns
> > RIP: 0010:dec_ucount+0x43/0x50
> > Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
> > RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
> > RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
> > RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
> > RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
> > R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
> > R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> > FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 000056ffceff6b10 CR3: 0000000147a0e005 CR4: 00000000003706e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > Call Trace:
> > free_user_ns+0x73/0x110
> > process_one_work+0x1e1/0x380
> > worker_thread+0x50/0x3a0
> > ? rescuer_thread+0x360/0x360
> > kthread+0x127/0x150
> > ? set_kthread_struct+0x40/0x40
> > ret_from_fork+0x22/0x30
> > ---[ end trace ff45ac39689f43c1 ]---
> > BUG: kernel NULL pointer dereference, address: 00000000000001e8
> > #PF: supervisor read access in kernel mode
> > #PF: error_code(0x0000) - not-present page
> > Oops: 0000 [#1] SMP PTI
> > CPU: 1 PID: 351 Comm: kworker/1:3 Tainted: G W 5.14.3 #1 c719caf0c6c208968387ed83e3061ac05d0faf2f
> > Workqueue: events free_user_ns
> > RIP: 0010:dec_ucount+0x32/0x50
> > Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
> > RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
> > RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
> > RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
> > RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
> > R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
> > R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> > FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > Call Trace:
> > free_user_ns+0x73/0x110
> > process_one_work+0x1e1/0x380
> > worker_thread+0x50/0x3a0
> > ? rescuer_thread+0x360/0x360
> > kthread+0x127/0x150
> > ? set_kthread_struct+0x40/0x40
> > ret_from_fork+0x22/0x30
> > Modules linked in: nft_ct nft_fib_ipv4 nft_fib wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ccm algif_aead des_generic libdes ecb algif_skcipher cmac md4 algif_hash af_alg hid_sensor_custom_intel_hinge hid_sensor_als hid_sensor_gyro_3d hid_sensor_accel_3d hid_sensor_rotation hid_sensor_magn_3d hid_sensor_trigger industrialio_triggered_buffer hid_sensor_iio_common kfifo_buf industrialio hid_sensor_custom hid_sensor_hub cros_ec_ishtp cros_ec intel_ishtp_loader intel_ishtp_hid intel_rapl_msr nft_counter xt_mark ipt_REJECT nf_reject_ipv4 mei_hdcp intel_rapl_common xt_LOG nf_log_syslog intel_tcc_cooling x86_pkg_temp_thermal think_lmi wmi_bmof xt_addrtype firmware_attributes_class xt_tcpudp intel_powerclamp xt_conntrack nf_conntrack nf_defrag_ipv4 snd_hda_codec_hdmi nft_compat intel_wmi_thunderbolt nf_tables libcrc32c coretemp iwlmvm snd_ctl_led nfnetlink
> > snd_hda_codec_conexant mac80211 snd_hda_codec_generic libarc4 vfat snd_hda_intel kvm_intel fat iwlwifi snd_intel_dspcfg rapl intel_cstate joydev snd_hda_codec mousedev intel_uncore snd_hwdep snd_hda_core psmouse snd_pcm snd_timer cfg80211 mei_me wacom ucsi_acpi typec_ucsi mei intel_pch_thermal intel_xhci_usb_role_switch intel_ish_ipc roles intel_ishtp typec wmi thinkpad_acpi ledtrig_audio platform_profile snd soundcore tpm_crb rfkill i2c_hid_acpi tpm_tis tpm_tis_core i2c_hid mac_hid acpi_pad pkcs8_key_parser fuse zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 usbhid dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod rtsx_pci_sdmmc mmc_core serio_raw atkbd libps2 crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd rtsx_pci xhci_pci xhci_pci_renesas i8042 serio kvmgt mdev vfio_iommu_type1 vfio i915 i2c_algo_bit intel_gtt ttm agpgart video drm_kms_helper syscopyarea sysfillrect sysimgblt
> > fb_sys_fops cec drm kvm irqbypass
> > CR2: 00000000000001e8
> > ---[ end trace ff45ac39689f43c2 ]---
> > RIP: 0010:dec_ucount+0x32/0x50
> > Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 0d fd ff ff 0f 0b eb e7 66
> > RSP: 0018:ffffaa06c08bbe60 EFLAGS: 00010297
> > RAX: 0000000000000000 RBX: ffff894ecb0c35a0 RCX: ffff894e25cdfcc0
> > RDX: ffff894e25cdfce0 RSI: ffffffffffffffff RDI: ffff894e25cdfcc0
> > RBP: ffff894ee393d380 R08: 0000000000000020 R09: ffff894ee393d5f0
> > R10: ffff894f617fd000 R11: 0000000000031678 R12: ffff894e25cdfcc0
> > R13: 00000000ffffffff R14: 0000000000000000 R15: 0000000000000001
> > FS: 0000000000000000(0000) GS:ffff894f59680000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00000000000001e8 CR3: 0000000147a0e005 CR4: 00000000003706e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>

--
Rgrds, legion

2021-09-18 00:50:27

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

Yu Zhao <[email protected]> writes:

> On Wed, Sep 15, 2021 at 4:42 PM Jordan Glover
> <[email protected]> wrote:
>>
>> On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:
>>
>> > Jordan Glover [email protected] writes:
>> >
>> > > Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window
>> > >
>> > > https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u
>> >
>> > So that warning comes from:
>> >
>> > void dec_ucount(struct ucounts *ucounts, enum ucount_type type)
>> >
>> > {
>> >
>> > struct ucounts *iter;
>> >
>> > for (iter = ucounts; iter; iter = iter->ns->ucounts) {
>> >
>> > long dec = atomic_long_dec_if_positive(&iter->ucount[type]);
>> >
>> > WARN_ON_ONCE(dec < 0);
>> > }
>> > put_ucounts(ucounts);
>> >
>> >
>> > }
>> >
>> > Which certainly looks like a reference count bug. It could also be a
>> >
>> > memory stomp somewhere close.
>> >
>> > Do you have any idea what else was going on? This location is the
>> >
>> > symptom but not the actual cause.
>> >
>> > Eric
>>
>> I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
>>
>> Maybe it's some random fluke. I'll reply if I hit it again.
>
> I have been able to steadily reproduce this for a while. But I haven't
> had time to look into it. I'd appreciate any help.

It would be very helpful if you could look farther back in your logs and
see if you can also see:
WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x5

Or anything else preceding the use-after-free.

I am inclined to think they are the same issue but without seeing the
WARN_ON_ONCE I can't safely conclude that.

Eric

2021-09-18 02:33:21

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Fri, Sep 17, 2021 at 10:17 AM Eric W. Biederman
<[email protected]> wrote:
>
> Yu Zhao <[email protected]> writes:
>
> > On Wed, Sep 15, 2021 at 4:42 PM Jordan Glover
> > <[email protected]> wrote:
> >>
> >> On Wednesday, September 15th, 2021 at 9:02 PM, <[email protected]> wrote:
> >>
> >> > Jordan Glover [email protected] writes:
> >> >
> >> > > Hi, recently I hit system freeze after I was closing few containerized apps on my system. As for now it occurred only once on linux 5.14.3. I think it maybe be related to "Count rlimits in each user namespace" patchset merged during 5.14 window
> >> > >
> >> > > https://lore.kernel.org/all/257aa5fb1a7d81cf0f4c34f39ada2320c4284771.1619094428.git.legion@kernel.org/T/#u
> >> >
> >> > So that warning comes from:
> >> >
> >> > void dec_ucount(struct ucounts *ucounts, enum ucount_type type)
> >> >
> >> > {
> >> >
> >> > struct ucounts *iter;
> >> >
> >> > for (iter = ucounts; iter; iter = iter->ns->ucounts) {
> >> >
> >> > long dec = atomic_long_dec_if_positive(&iter->ucount[type]);
> >> >
> >> > WARN_ON_ONCE(dec < 0);
> >> > }
> >> > put_ucounts(ucounts);
> >> >
> >> >
> >> > }
> >> >
> >> > Which certainly looks like a reference count bug. It could also be a
> >> >
> >> > memory stomp somewhere close.
> >> >
> >> > Do you have any idea what else was going on? This location is the
> >> >
> >> > symptom but not the actual cause.
> >> >
> >> > Eric
> >>
> >> I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
> >>
> >> Maybe it's some random fluke. I'll reply if I hit it again.
> >
> > I have been able to steadily reproduce this for a while. But I haven't
> > had time to look into it. I'd appreciate any help.
>
> It would be very helpful if you could look farther back in your logs and
> see if you can also see:
> WARNING: CPU: 1 PID: 351 at kernel/ucount.c:253 dec_ucount+0x43/0x5
>
> Or anything else preceding the use-after-free.
>
> I am inclined to think they are the same issue but without seeing the
> WARN_ON_ONCE I can't safely conclude that.

It was either the WARN_ON_ONCE or the KASAN before my kernel crashed.
(KASAN is on for both cases.)

[ 3049.540734] ------------[ cut here ]------------^M
[ 3049.545557] WARNING: CPU: 0 PID: 8369 at kernel/ucount.c:253
dec_ucount+0xb0/0xd8^M
[ 3049.553293] Modules linked in: vhost_vsock vhost vhost_iotlb
vmw_vsock_virtio_transport_common vsock rfcomm algif_hash
algif_skcipher af_alg uinput uvcvideo videobuf2_vmalloc venus_enc
venus_dec videobuf2_dma_contig videobuf2_memops cros_ec_typec typec
hci_uart btqca bluetooth ecdh_generic ecc venus_core v4l2_mem2mem
videobuf2_v4l2 videobuf2_common qcom_q6v5_mss ipa qcom_pil_info
qcom_q6v5 qcom_common xt_MASQUERADE rmtfs_mem fuse ath10k_snoc
qmi_helpers ath10k_core ath mac80211 lzo_rle lzo_compress cfg80211
zram smsc smsc95xx usbnet mii joydev^M
[ 3049.603135] CPU: 0 PID: 8369 Comm: kworker/0:2 Not tainted
5.14.0-lockdep+ #4^M
[ 3049.610506] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)^M
[ 3049.617230] Workqueue: events free_user_ns^M
[ 3049.621494] pstate: a0400009 (NzCv daif +PAN -UAO -TCO BTYPE=--)^M
[ 3049.627694] pc : dec_ucount+0xb0/0xd8^M
[ 3049.631489] lr : dec_ucount+0x50/0xd8^M
[ 3049.635280] sp : ffffffc019ff7b50^M
[ 3049.638707] x29: ffffffc019ff7b50 x28: ffffffd5b5d8b710 x27:
ffffff80884cae90^M
[ 3049.646083] x26: dfffffc000000000 x25: ffffffd5b5de98a0 x24:
0000000000000001^M
[ 3049.653460] x23: ffffff80d2597100 x22: 0000000000000000 x21:
dfffffc000000000^M
[ 3049.660838] x20: ffffff80d2597120 x19: ffffff80d2597100 x18:
1ffffff01b4b3f48^M
[ 3049.668210] x17: 1ffffff01b4b3f47 x16: 0000000000000000 x15:
0000000000000000^M
[ 3049.675585] x14: 0000000000000000 x13: 000000002b0cc34b x12:
0000000024966d0f^M
[ 3049.682960] x11: 0000000000000000 x10: dfffffc000000001 x9 :
0000000000000000^M
[ 3049.690334] x8 : ffffff80d259791f x7 : 0000000000000000 x6 :
ffffffd5b3de71c4^M
[ 3049.697709] x5 : 0000000000000000 x4 : 0000000000000000 x3 :
ffffffd5b3391ca4^M
[ 3049.705086] x2 : 0000000000000001 x1 : 0000000000000008 x0 :
0000000000000001^M
[ 3049.712459] Call trace:^M
[ 3049.715003] dec_ucount+0xb0/0xd8^M
[ 3049.718443] free_user_ns+0x1b0/0x288^M
[ 3049.722242] process_one_work+0x7b4/0x10ec^M
[ 3049.726485] worker_thread+0x800/0xcf4^M
[ 3049.730377] kthread+0x2a8/0x358^M
[ 3049.733736] ret_from_fork+0x10/0x18^M
[ 3049.737445] irq event stamp: 0^M
[ 3049.740612] hardirqs last enabled at (0): [<0000000000000000>] 0x0^M
[ 3049.747092] hardirqs last disabled at (0): [<ffffffd5b331413c>]
copy_process+0xcb0/0x2a54^M
[ 3049.755521] softirqs last enabled at (0): [<ffffffd5b3314164>]
copy_process+0xcd8/0x2a54^M
[ 3049.763954] softirqs last disabled at (0): [<0000000000000000>] 0x0^M
[ 3049.770432] ---[ end trace fa55518c981c0c5d ]---^M

2021-09-28 13:42:45

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Thursday, September 16th, 2021 at 5:30 PM, <[email protected]> wrote:

> Jordan Glover [email protected] writes:
>
> > On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover [email protected] wrote:
> >
> > > I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
> > >
> > > Maybe it's some random fluke. I'll reply if I hit it again.
> >
> > Heh, it jut happened again. This time closing firefox alone had such
> > effect:
>
> Ok. It looks like he have a couple of folks seeing issues here.
> I thought we had all of the issues sorted out for the release of v5.14,
> but it looks like there is still some little bug left.
>
> If Alex doesn't beat me to it I will see if I can come up with a
> debugging patch to make it easy to help track down where the reference
> count is going wrong. It will be a little bit as my brain is mush at
> the moment.
>
> Eric

As the issue persist in 5.14.7 I would be very interested in such patch.

For now the thing is mostly reproducible when I close several tabs in ff then
close the browser in short period of time. When I close tabs then wait out
a bit then close the browser it doesn't happen so I guess some interrupted
cleanup triggers it.

Jordan

2021-09-29 19:30:32

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Tue, Sep 28, 2021 at 01:40:48PM +0000, Jordan Glover wrote:
> On Thursday, September 16th, 2021 at 5:30 PM, <[email protected]> wrote:
>
> > Jordan Glover [email protected] writes:
> >
> > > On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover [email protected] wrote:
> > >
> > > > I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
> > > >
> > > > Maybe it's some random fluke. I'll reply if I hit it again.
> > >
> > > Heh, it jut happened again. This time closing firefox alone had such
> > > effect:
> >
> > Ok. It looks like he have a couple of folks seeing issues here.
> > I thought we had all of the issues sorted out for the release of v5.14,
> > but it looks like there is still some little bug left.
> >
> > If Alex doesn't beat me to it I will see if I can come up with a
> > debugging patch to make it easy to help track down where the reference
> > count is going wrong. It will be a little bit as my brain is mush at
> > the moment.
> >
> > Eric
>
> As the issue persist in 5.14.7 I would be very interested in such patch.
>
> For now the thing is mostly reproducible when I close several tabs in ff then
> close the browser in short period of time. When I close tabs then wait out
> a bit then close the browser it doesn't happen so I guess some interrupted
> cleanup triggers it.

I'm still investigating, but I would like to rule out one option.
Could you check out the patch?

diff --git a/kernel/ucount.c b/kernel/ucount.c
index bb51849e6375..f23f906f4f62 100644
--- a/kernel/ucount.c
+++ b/kernel/ucount.c
@@ -201,11 +201,14 @@ void put_ucounts(struct ucounts *ucounts)
{
unsigned long flags;

- if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) {
+ spin_lock_irqsave(&ucounts_lock, flags);
+ if (atomic_dec_and_test(&ucounts->count)) {
hlist_del_init(&ucounts->node);
spin_unlock_irqrestore(&ucounts_lock, flags);
kfree(ucounts);
+ return;
}
+ spin_unlock_irqrestore(&ucounts_lock, flags);
}

static inline bool atomic_long_inc_below(atomic_long_t *v, int u)

--
Rgrds, legion

2021-09-29 21:41:04

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wednesday, September 29th, 2021 at 5:36 PM, Alexey Gladkov <[email protected]> wrote:

> On Tue, Sep 28, 2021 at 01:40:48PM +0000, Jordan Glover wrote:
>
> > On Thursday, September 16th, 2021 at 5:30 PM, [email protected] wrote:
> >
> > > Jordan Glover [email protected] writes:
> > >
> > > > On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover [email protected] wrote:
> > > >
> > > > > I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
> > > > >
> > > > > Maybe it's some random fluke. I'll reply if I hit it again.
> > > >
> > > > Heh, it jut happened again. This time closing firefox alone had such
> > > >
> > > > effect:
> > >
> > > Ok. It looks like he have a couple of folks seeing issues here.
> > >
> > > I thought we had all of the issues sorted out for the release of v5.14,
> > >
> > > but it looks like there is still some little bug left.
> > >
> > > If Alex doesn't beat me to it I will see if I can come up with a
> > >
> > > debugging patch to make it easy to help track down where the reference
> > >
> > > count is going wrong. It will be a little bit as my brain is mush at
> > >
> > > the moment.
> > >
> > > Eric
> >
> > As the issue persist in 5.14.7 I would be very interested in such patch.
> >
> > For now the thing is mostly reproducible when I close several tabs in ff then
> >
> > close the browser in short period of time. When I close tabs then wait out
> >
> > a bit then close the browser it doesn't happen so I guess some interrupted
> >
> > cleanup triggers it.
>
> I'm still investigating, but I would like to rule out one option.
>
> Could you check out the patch?

Thx, I added it to my kernel and will report in few days.
Does this patch try to fix the issue or make it easier to track?

Jordan

> diff --git a/kernel/ucount.c b/kernel/ucount.c
>
> index bb51849e6375..f23f906f4f62 100644
>
> --- a/kernel/ucount.c
>
> +++ b/kernel/ucount.c
>
> @@ -201,11 +201,14 @@ void put_ucounts(struct ucounts *ucounts)
>
> {
>
> unsigned long flags;
>
> - if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) {
>
>
>
> - spin_lock_irqsave(&ucounts_lock, flags);
>
>
> - if (atomic_dec_and_test(&ucounts->count)) {
>
> hlist_del_init(&ucounts->node);
>
> spin_unlock_irqrestore(&ucounts_lock, flags);
> kfree(ucounts);
>
>
> - return;
> }
>
>
> - spin_unlock_irqrestore(&ucounts_lock, flags);
>
>
>
> }
>
> static inline bool atomic_long_inc_below(atomic_long_t *v, int u)
>
> ---------------------------------------------------------------------
>
> Rgrds, legion

2021-09-30 13:30:17

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wed, Sep 29, 2021 at 09:39:06PM +0000, Jordan Glover wrote:
> > I'm still investigating, but I would like to rule out one option.
> >
> > Could you check out the patch?
>
>
> Thx, I added it to my kernel and will report in few days.
> Does this patch try to fix the issue or make it easier to track?

I suspect the error is caused by a race between allow_ucounts() and
put_ucounts(). I think this patch could solve the problem.

--
Rgrds, legion

2021-09-30 22:58:23

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Thu, Sep 30, 2021 at 7:06 AM Alexey Gladkov <[email protected]> wrote:
>
> On Wed, Sep 29, 2021 at 09:39:06PM +0000, Jordan Glover wrote:
> > > I'm still investigating, but I would like to rule out one option.
> > >
> > > Could you check out the patch?
> >
> >
> > Thx, I added it to my kernel and will report in few days.
> > Does this patch try to fix the issue or make it easier to track?
>
> I suspect the error is caused by a race between allow_ucounts() and
> put_ucounts(). I think this patch could solve the problem.

Thanks for your help. Still can reproduce the problem with the change suggested.

[ 7761.885966] ==================================================================
[ 7761.893462] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
[ 7761.899491] Write of size 8 at addr ffffff80c537b140 by task
kworker/u16:3/10303
[ 7761.907110]
[ 7761.908668] CPU: 0 PID: 10303 Comm: kworker/u16:3 Not tainted
5.14.0-lockdep+ #1
[ 7761.916289] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
[ 7761.923021] Workqueue: netns cleanup_net
[ 7761.927106] Call trace:
[ 7761.929648] dump_backtrace+0x0/0x42c
[ 7761.933442] show_stack+0x24/0x30
[ 7761.936878] dump_stack_lvl+0xd0/0x100
[ 7761.940763] print_address_description+0x30/0x304
[ 7761.945628] kasan_report+0x190/0x1d8
[ 7761.949418] kasan_check_range+0x1ac/0x1bc
[ 7761.953655] __kasan_check_write+0x44/0x54
[ 7761.957891] dec_ucount+0x50/0xd8
[ 7761.961334] cleanup_net+0x630/0x718
[ 7761.965036] process_one_work+0x7b4/0x10ec
[ 7761.969274] worker_thread+0x800/0xcf4
[ 7761.973152] kthread+0x2a8/0x358
[ 7761.976496] ret_from_fork+0x10/0x18
[ 7761.980201]
[ 7761.981761] Allocated by task 4840:
[ 7761.985366] kasan_save_stack+0x38/0x68
[ 7761.989342] __kasan_kmalloc+0x9c/0xb8
[ 7761.993222] kmem_cache_alloc_trace+0x2a4/0x370
[ 7761.997905] alloc_ucounts+0x150/0x374
[ 7762.001787] set_cred_ucounts+0x198/0x248
[ 7762.005935] __sys_setresuid+0x31c/0x4f8
[ 7762.009993] __arm64_sys_setresuid+0x84/0x98
[ 7762.014410] invoke_syscall+0xd4/0x2c8
[ 7762.018292] el0_svc_common+0x124/0x200
[ 7762.022265] do_el0_svc_compat+0x54/0x64
[ 7762.026325] el0_svc_compat+0x24/0x34
[ 7762.030124] el0t_32_sync_handler+0xc0/0xf0
[ 7762.034451] el0t_32_sync+0x19c/0x1a0
[ 7762.038241]
[ 7762.039799] Freed by task 0:
[ 7762.042778] kasan_save_stack+0x38/0x68
[ 7762.046747] kasan_set_track+0x28/0x3c
[ 7762.050625] kasan_set_free_info+0x28/0x4c
[ 7762.054857] ____kasan_slab_free+0x118/0x164
[ 7762.059277] __kasan_slab_free+0x18/0x28
[ 7762.063339] kfree+0x2f8/0x500
[ 7762.066505] put_ucounts+0x11c/0x134
[ 7762.070209] put_cred_rcu+0x1bc/0x35c
[ 7762.074006] rcu_core+0xa68/0x1b20
[ 7762.077538] rcu_core_si+0x1c/0x28
[ 7762.081061] __do_softirq+0x4bc/0xedc
[ 7762.084851]
[ 7762.086401] The buggy address belongs to the object at ffffff80c537b100
[ 7762.086401] which belongs to the cache kmalloc-256 of size 256
[ 7762.099267] The buggy address is located 64 bytes inside of
[ 7762.099267] 256-byte region [ffffff80c537b100, ffffff80c537b200)
[ 7762.111248] The buggy address belongs to the page:
[ 7762.116185] page:fffffffe0314de00 refcount:1 mapcount:0
mapping:0000000000000000 index:0xffffff80c537ad00 pfn:0x145378
[ 7762.127180] head:fffffffe0314de00 order:3 compound_mapcount:0
compound_pincount:0
[ 7762.134881] flags: 0x8000000000010200(slab|head|zone=2)
[ 7762.140286] raw: 8000000000010200 fffffffe02799408 fffffffe02020808
ffffff808000c980
[ 7762.148263] raw: ffffff80c537ad00 0000000000200004 00000001ffffffff
0000000000000000
[ 7762.156228] page dumped because: kasan: bad access detected
[ 7762.161974]
[ 7762.163532] Memory state around the buggy address:
[ 7762.168475] ffffff80c537b000: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.175915] ffffff80c537b080: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.183346] >ffffff80c537b100: fa fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7762.190774] ^
[ 7762.196258] ffffff80c537b180: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7762.203689] ffffff80c537b200: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7762.211125] ==================================================================

2021-10-03 20:03:04

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wednesday, September 29th, 2021 at 5:36 PM, Alexey Gladkov <[email protected]> wrote:

> On Tue, Sep 28, 2021 at 01:40:48PM +0000, Jordan Glover wrote:
>
> > On Thursday, September 16th, 2021 at 5:30 PM, [email protected] wrote:
> >
> > > Jordan Glover [email protected] writes:
> > >
> > > > On Wednesday, September 15th, 2021 at 10:42 PM, Jordan Glover [email protected] wrote:
> > > >
> > > > > I had about 2 containerized (flatpak/bubblewrap) apps (browser + music player) running . I quickly closed them with intent to shutdown the system but instead get the freeze and had to use magic sysrq to reboot. System logs end with what I posted and before there is nothing suspicious.
> > > > >
> > > > > Maybe it's some random fluke. I'll reply if I hit it again.
> > > >
> > > > Heh, it jut happened again. This time closing firefox alone had such
> > > >
> > > > effect:
> > >
> > > Ok. It looks like he have a couple of folks seeing issues here.
> > >
> > > I thought we had all of the issues sorted out for the release of v5.14,
> > >
> > > but it looks like there is still some little bug left.
> > >
> > > If Alex doesn't beat me to it I will see if I can come up with a
> > >
> > > debugging patch to make it easy to help track down where the reference
> > >
> > > count is going wrong. It will be a little bit as my brain is mush at
> > >
> > > the moment.
> > >
> > > Eric
> >
> > As the issue persist in 5.14.7 I would be very interested in such patch.
> >
> > For now the thing is mostly reproducible when I close several tabs in ff then
> >
> > close the browser in short period of time. When I close tabs then wait out
> >
> > a bit then close the browser it doesn't happen so I guess some interrupted
> >
> > cleanup triggers it.
>
> I'm still investigating, but I would like to rule out one option.
>
> Could you check out the patch?
>
> diff --git a/kernel/ucount.c b/kernel/ucount.c
>
> index bb51849e6375..f23f906f4f62 100644
>
> --- a/kernel/ucount.c
>
> +++ b/kernel/ucount.c
>
> @@ -201,11 +201,14 @@ void put_ucounts(struct ucounts *ucounts)
>
> {
>
> unsigned long flags;
>
> - if (atomic_dec_and_lock_irqsave(&ucounts->count, &ucounts_lock, flags)) {
>
>
>
> - spin_lock_irqsave(&ucounts_lock, flags);
>
>
> - if (atomic_dec_and_test(&ucounts->count)) {
>
> hlist_del_init(&ucounts->node);
>
> spin_unlock_irqrestore(&ucounts_lock, flags);
> kfree(ucounts);
>
>
> - return;
> }
>
>
> - spin_unlock_irqrestore(&ucounts_lock, flags);
>
>
>
> }
>
> static inline bool atomic_long_inc_below(atomic_long_t *v, int u)
>
> ---------------------------------------------------------------------
>
> Rgrds, legion

I'm still able to reproduce the issue with above patch although situation
changed/improved a bit as now I have to close tabs and browser really fast
to hit it which means it's more unlikely to happen during real usage.

On the other hand the kernel logging cuts off much earlier, just after few
lines:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 20387 at kernel/ucount.c:256 dec_ucount+0x43/0x50
Modules linked in: ...

Jordan

2021-10-04 22:56:38

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

Adding Rune Kleveland to the discussion as he also seems to have
reproduced the issue.

Alex and I have been starring at the code and the reports and this
bug is hiding well. Here is what we have figured out so far.

Both the warning from free_user_ns calling dec_ucount that Jordan Glover
reported and the KASAN error that Yu Zhao has reported appear to have
the same cause. Using a ucounts structure after it has been freed and
reallocated as something else.

I have just skimmed through the recent report from Rune Kleveland
and it appears also to be a use after free. Especially since the
second failure in the log is slub complaining about trying to free
the ucounts data structure.

We looked through the users of put_ucounts and we don't see any obvious
buggy users that would be freeing the data structure early.

Alex has tried to reproduce this so far is not having any luck.
Folks can you tell what compiler versions you are using and share your
kernel config with us? That might help.

The little debug diff below is my guess of what is happening. If the
folks who can reproduce this issue can try the patch below and let me
know if the warnings fire that would be appreciated. It is still not
enough to track down the bug but at least it will confirm my current
hypothesis about how things look before there is a use of memory after
it is freed.

Thank you,
Eric

diff --git a/kernel/cred.c b/kernel/cred.c
index f784e08c2fbd..e7ffaa3cf5a6 100644
--- a/kernel/cred.c
+++ b/kernel/cred.c
@@ -120,6 +120,12 @@ static void put_cred_rcu(struct rcu_head *rcu)
if (cred->group_info)
put_group_info(cred->group_info);
free_uid(cred->user);
+#if 1
+ if ((cred->ucounts == cred->user_ns->ucounts) &&
+ (atomic_read(&cred->ucounts->count) == 1)) {
+ WARN_ONCE(1, "put_cred_rcu: ucount count 1\n");
+ }
+#endif
if (cred->ucounts)
put_ucounts(cred->ucounts);
put_user_ns(cred->user_ns);
diff --git a/kernel/exit.c b/kernel/exit.c
index 91a43e57a32e..60fd88b34c1a 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -743,6 +743,13 @@ void __noreturn do_exit(long code)
if (unlikely(!tsk->pid))
panic("Attempted to kill the idle task!");

+#if 1
+ if ((tsk->cred->ucounts == tsk->cred->user_ns->ucounts) &&
+ (atomic_read(tsk->cred->ucounts->count) == 1)) {
+ WARN_ONCE(1, "do_exit: ucount count 1\n");
+ }
+#endif
+
/*
* If do_exit is called because this processes oopsed, it's possible
* that get_fs() was left as KERNEL_DS, so reset it to USER_DS before

2021-10-04 23:51:07

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Mon, Oct 4, 2021 at 11:19 AM Eric W. Biederman <[email protected]> wrote:
>
> [email protected] (Eric W. Biederman) writes:
>
> > Adding Rune Kleveland to the discussion as he also seems to have
> > reproduced the issue.
> >
> > Alex and I have been starring at the code and the reports and this
> > bug is hiding well. Here is what we have figured out so far.
> >
> > Both the warning from free_user_ns calling dec_ucount that Jordan Glover
> > reported and the KASAN error that Yu Zhao has reported appear to have
> > the same cause. Using a ucounts structure after it has been freed and
> > reallocated as something else.
> >
> > I have just skimmed through the recent report from Rune Kleveland
> > and it appears also to be a use after free. Especially since the
> > second failure in the log is slub complaining about trying to free
> > the ucounts data structure.
> >
> > We looked through the users of put_ucounts and we don't see any obvious
> > buggy users that would be freeing the data structure early.
> >
> > Alex has tried to reproduce this so far is not having any luck.
> > Folks can you tell what compiler versions you are using and share your
> > kernel config with us? That might help.

Thanks. Per your request:

$ aarch64-cros-linux-gnu-clang -v
Chromium OS 12.0_pre422132_p20210405-r9 clang version 13.0.0
(/var/tmp/portage/sys-devel/llvm-12.0_pre422132_p20210405-r9/work/llvm-12.0_pre422132_p20210405/clang
cd442157cff4aad209ae532cbf031abbe10bc1df)
Target: aarch64-cros-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation:
/usr/bin/../lib/gcc/aarch64-cros-linux-gnu/10.2.0
Found candidate GCC installation:
/usr/bin/../lib64/gcc/aarch64-cros-linux-gnu/10.2.0
Selected GCC installation: /usr/bin/../lib64/gcc/aarch64-cros-linux-gnu/10.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64

Kernel config attached.

Attachments:

kernel.config (185.50 kB)

2021-10-05 00:17:19

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

[email protected] (Eric W. Biederman) writes:

> Adding Rune Kleveland to the discussion as he also seems to have
> reproduced the issue.
>
> Alex and I have been starring at the code and the reports and this
> bug is hiding well. Here is what we have figured out so far.
>
> Both the warning from free_user_ns calling dec_ucount that Jordan Glover
> reported and the KASAN error that Yu Zhao has reported appear to have
> the same cause. Using a ucounts structure after it has been freed and
> reallocated as something else.
>
> I have just skimmed through the recent report from Rune Kleveland
> and it appears also to be a use after free. Especially since the
> second failure in the log is slub complaining about trying to free
> the ucounts data structure.
>
> We looked through the users of put_ucounts and we don't see any obvious
> buggy users that would be freeing the data structure early.
>
> Alex has tried to reproduce this so far is not having any luck.
> Folks can you tell what compiler versions you are using and share your
> kernel config with us? That might help.
>
> The little debug diff below is my guess of what is happening. If the
> folks who can reproduce this issue can try the patch below and let me
> know if the warnings fire that would be appreciated. It is still not
> enough to track down the bug but at least it will confirm my current
> hypothesis about how things look before there is a use of memory after
> it is freed.

Bah. Scratch that test patch. I just double checked myself and
cred->ucounts and cred->user_ns->ucounts should never be equal,
as the user namespace is counted in it's parent user namespace.

That observation now tells me I have a parent user namespace that went
corrupt.

Back to the drawing board.

> Thank you,
> Eric
>
> diff --git a/kernel/cred.c b/kernel/cred.c
> index f784e08c2fbd..e7ffaa3cf5a6 100644
> --- a/kernel/cred.c
> +++ b/kernel/cred.c
> @@ -120,6 +120,12 @@ static void put_cred_rcu(struct rcu_head *rcu)
> if (cred->group_info)
> put_group_info(cred->group_info);
> free_uid(cred->user);
> +#if 1
> + if ((cred->ucounts == cred->user_ns->ucounts) &&
> + (atomic_read(&cred->ucounts->count) == 1)) {
> + WARN_ONCE(1, "put_cred_rcu: ucount count 1\n");
> + }
> +#endif
> if (cred->ucounts)
> put_ucounts(cred->ucounts);
> put_user_ns(cred->user_ns);
> diff --git a/kernel/exit.c b/kernel/exit.c
> index 91a43e57a32e..60fd88b34c1a 100644
> --- a/kernel/exit.c
> +++ b/kernel/exit.c
> @@ -743,6 +743,13 @@ void __noreturn do_exit(long code)
> if (unlikely(!tsk->pid))
> panic("Attempted to kill the idle task!");
>
> +#if 1
> + if ((tsk->cred->ucounts == tsk->cred->user_ns->ucounts) &&
> + (atomic_read(tsk->cred->ucounts->count) == 1)) {
> + WARN_ONCE(1, "do_exit: ucount count 1\n");
> + }
> +#endif
> +
> /*
> * If do_exit is called because this processes oopsed, it's possible
> * that get_fs() was left as KERNEL_DS, so reset it to USER_DS before

2021-10-06 06:25:19

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Tue, Oct 5, 2021 at 8:12 PM Hillf Danton <[email protected]> wrote:
>
> On Thu, 30 Sep 2021 16:27:34 -0600 Yu Zhao wrote:
> >On Thu, Sep 30, 2021 at 7:06 AM Alexey Gladkov <[email protected]> wrote:
> >>
> >> On Wed, Sep 29, 2021 at 09:39:06PM +0000, Jordan Glover wrote:
> >> > > I'm still investigating, but I would like to rule out one option.
> >> > >
> >> > > Could you check out the patch?
> >> >
> >> >
> >> > Thx, I added it to my kernel and will report in few days.
> >> > Does this patch try to fix the issue or make it easier to track?
> >>
> >> I suspect the error is caused by a race between allow_ucounts() and
> >> put_ucounts(). I think this patch could solve the problem.
> >
> >Thanks for your help. Still can reproduce the problem with the change suggested.
> >
> >[ 7761.885966] ==================================================================
> >[ 7761.893462] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
> >[ 7761.899491] Write of size 8 at addr ffffff80c537b140 by task
> >kworker/u16:3/10303
> >[ 7761.907110]
> >[ 7761.908668] CPU: 0 PID: 10303 Comm: kworker/u16:3 Not tainted
> >5.14.0-lockdep+ #1
> >[ 7761.916289] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
> >[ 7761.923021] Workqueue: netns cleanup_net
> >[ 7761.927106] Call trace:
> >[ 7761.929648] dump_backtrace+0x0/0x42c
> >[ 7761.933442] show_stack+0x24/0x30
> >[ 7761.936878] dump_stack_lvl+0xd0/0x100
> >[ 7761.940763] print_address_description+0x30/0x304
> >[ 7761.945628] kasan_report+0x190/0x1d8
> >[ 7761.949418] kasan_check_range+0x1ac/0x1bc
> >[ 7761.953655] __kasan_check_write+0x44/0x54
> >[ 7761.957891] dec_ucount+0x50/0xd8
> >[ 7761.961334] cleanup_net+0x630/0x718
> >[ 7761.965036] process_one_work+0x7b4/0x10ec
> >[ 7761.969274] worker_thread+0x800/0xcf4
> >[ 7761.973152] kthread+0x2a8/0x358
> >[ 7761.976496] ret_from_fork+0x10/0x18
> >[ 7761.980201]
> >[ 7761.981761] Allocated by task 4840:
> >[ 7761.985366] kasan_save_stack+0x38/0x68
> >[ 7761.989342] __kasan_kmalloc+0x9c/0xb8
> >[ 7761.993222] kmem_cache_alloc_trace+0x2a4/0x370
> >[ 7761.997905] alloc_ucounts+0x150/0x374
> >[ 7762.001787] set_cred_ucounts+0x198/0x248
> >[ 7762.005935] __sys_setresuid+0x31c/0x4f8
> >[ 7762.009993] __arm64_sys_setresuid+0x84/0x98
> >[ 7762.014410] invoke_syscall+0xd4/0x2c8
> >[ 7762.018292] el0_svc_common+0x124/0x200
> >[ 7762.022265] do_el0_svc_compat+0x54/0x64
> >[ 7762.026325] el0_svc_compat+0x24/0x34
> >[ 7762.030124] el0t_32_sync_handler+0xc0/0xf0
> >[ 7762.034451] el0t_32_sync+0x19c/0x1a0
> >[ 7762.038241]
> >[ 7762.039799] Freed by task 0:
> >[ 7762.042778] kasan_save_stack+0x38/0x68
> >[ 7762.046747] kasan_set_track+0x28/0x3c
> >[ 7762.050625] kasan_set_free_info+0x28/0x4c
> >[ 7762.054857] ____kasan_slab_free+0x118/0x164
> >[ 7762.059277] __kasan_slab_free+0x18/0x28
> >[ 7762.063339] kfree+0x2f8/0x500
> >[ 7762.066505] put_ucounts+0x11c/0x134
> >[ 7762.070209] put_cred_rcu+0x1bc/0x35c
> >[ 7762.074006] rcu_core+0xa68/0x1b20
> >[ 7762.077538] rcu_core_si+0x1c/0x28
> >[ 7762.081061] __do_softirq+0x4bc/0xedc
> >[ 7762.084851]
> >[ 7762.086401] The buggy address belongs to the object at ffffff80c537b100
> >[ 7762.086401] which belongs to the cache kmalloc-256 of size 256
> >[ 7762.099267] The buggy address is located 64 bytes inside of
> >[ 7762.099267] 256-byte region [ffffff80c537b100, ffffff80c537b200)
> >[ 7762.111248] The buggy address belongs to the page:
> >[ 7762.116185] page:fffffffe0314de00 refcount:1 mapcount:0
> >mapping:0000000000000000 index:0xffffff80c537ad00 pfn:0x145378
> >[ 7762.127180] head:fffffffe0314de00 order:3 compound_mapcount:0
> >compound_pincount:0
> >[ 7762.134881] flags: 0x8000000000010200(slab|head|zone=2)
> >[ 7762.140286] raw: 8000000000010200 fffffffe02799408 fffffffe02020808
> >ffffff808000c980
> >[ 7762.148263] raw: ffffff80c537ad00 0000000000200004 00000001ffffffff
> >0000000000000000
> >[ 7762.156228] page dumped because: kasan: bad access detected
> >[ 7762.161974]
> >[ 7762.163532] Memory state around the buggy address:
> >[ 7762.168475] ffffff80c537b000: fc fc fc fc fc fc fc fc fc fc fc fc
> >fc fc fc fc
> >[ 7762.175915] ffffff80c537b080: fc fc fc fc fc fc fc fc fc fc fc fc
> >fc fc fc fc
> >[ 7762.183346] >ffffff80c537b100: fa fb fb fb fb fb fb fb fb fb fb fb
> >fb fb fb fb
> >[ 7762.190774] ^
> >[ 7762.196258] ffffff80c537b180: fb fb fb fb fb fb fb fb fb fb fb fb
> >fb fb fb fb
> >[ 7762.203689] ffffff80c537b200: fc fc fc fc fc fc fc fc fc fc fc fc
> >fc fc fc fc
> >[ 7762.211125] ==================================================================
>
> Could you please check if it is due to count underflow? Given nothing wrong
> on the other side based on the efforts
> "We looked through the users of put_ucounts and we don't see any obvious buggy
> users that would be freeing the data structure early."

Thanks, Hillf.

Still can reproduce the problem without triggering the warning.

[ 7965.795189] ==================================================================
[ 7965.802642] BUG: KASAN: use-after-free in dec_ucount+0x50/0xd8
[ 7965.808653] Write of size 8 at addr ffffff8090294920 by task kworker/7:0/7002
[ 7965.815990]
[ 7965.817536] CPU: 7 PID: 7002 Comm: kworker/7:0 Not tainted 5.14.0-lockdep+ #2
[ 7965.824870] Hardware name: Google Lazor (rev3+) with KB Backlight (DT)
[ 7965.831576] Workqueue: events free_user_ns
[ 7965.835810] Call trace:
[ 7965.838338] dump_backtrace+0x0/0x42c
[ 7965.842119] show_stack+0x24/0x30
[ 7965.845540] dump_stack_lvl+0xd0/0x100
[ 7965.849401] print_address_description+0x30/0x304
[ 7965.854248] kasan_report+0x190/0x1d8
[ 7965.858016] kasan_check_range+0x1ac/0x1bc
[ 7965.862236] __kasan_check_write+0x44/0x54
[ 7965.866463] dec_ucount+0x50/0xd8
[ 7965.869876] free_user_ns+0x1b0/0x288
[ 7965.873645] process_one_work+0x7b4/0x10ec
[ 7965.877871] worker_thread+0x800/0xcf4
[ 7965.881729] kthread+0x2a8/0x358
[ 7965.885057] ret_from_fork+0x10/0x18
[ 7965.888746]
[ 7965.890288] Allocated by task 19167:
[ 7965.893976] kasan_save_stack+0x38/0x68
[ 7965.897923] __kasan_kmalloc+0x9c/0xb8
[ 7965.901782] __kmalloc+0x2c8/0x3c0
[ 7965.905287] kvmalloc_node+0x64/0x104
[ 7965.909065] v4l2_ctrl_new+0x478/0xc6c
[ 7965.912923] v4l2_ctrl_new_std+0x280/0x374
[ 7965.917136] venc_ctrl_init+0x4c4/0x7c8 [venus_enc]
[ 7965.922170] venc_open+0x218/0x668 [venus_enc]
[ 7965.926748] v4l2_open+0x158/0x234
[ 7965.930255] chrdev_open+0x17c/0x37c
[ 7965.933942] do_dentry_open+0x504/0xa44
[ 7965.937889] vfs_open+0x84/0x98
[ 7965.941124] path_openat+0x1980/0x1fac
[ 7965.944991] do_filp_open+0x19c/0x2c0
[ 7965.948767] do_sys_openat2+0xb8/0x298
[ 7965.952628] do_sys_open+0x124/0x15c
[ 7965.956309] __arm64_compat_sys_openat+0xa4/0xbc
[ 7965.961060] invoke_syscall+0xd4/0x2c8
[ 7965.964920] el0_svc_common+0x124/0x200
[ 7965.968873] do_el0_svc_compat+0x54/0x64
[ 7965.972915] el0_svc_compat+0x24/0x34
[ 7965.976688] el0t_32_sync_handler+0xc0/0xf0
[ 7965.980999] el0t_32_sync+0x19c/0x1a0
[ 7965.984781]
[ 7965.986330] Last potentially related work creation:
[ 7965.991342] kasan_save_stack+0x38/0x68
[ 7965.995294] kasan_record_aux_stack+0xdc/0x108
[ 7965.999866] insert_work+0x60/0x22c
[ 7966.003458] __queue_work+0xae8/0xe74
[ 7966.007223] queue_work_on+0x100/0x148
[ 7966.011076] drm_atomic_helper_commit+0x18c/0x1f4
[ 7966.015917] drm_atomic_nonblocking_commit+0xc4/0xdc
[ 7966.021018] drm_mode_atomic_ioctl+0x89c/0xaa0
[ 7966.025590] drm_ioctl_kernel+0x15c/0x260
[ 7966.029720] drm_ioctl+0x460/0x828
[ 7966.033229] drm_compat_ioctl+0x1bc/0x230
[ 7966.037370] __arm64_compat_sys_ioctl+0x1b0/0x1c4
[ 7966.042216] invoke_syscall+0xd4/0x2c8
[ 7966.046071] el0_svc_common+0x124/0x200
[ 7966.050015] do_el0_svc_compat+0x54/0x64
[ 7966.054053] el0_svc_compat+0x24/0x34
[ 7966.057819] el0t_32_sync_handler+0xc0/0xf0
[ 7966.062119] el0t_32_sync+0x19c/0x1a0
[ 7966.065884]
[ 7966.067424] Second to last potentially related work creation:
[ 7966.073329] kasan_save_stack+0x38/0x68
[ 7966.077271] kasan_record_aux_stack+0xdc/0x108
[ 7966.081837] insert_work+0x60/0x22c
[ 7966.085434] __queue_work+0xae8/0xe74
[ 7966.089202] queue_work_on+0x100/0x148
[ 7966.093060] drm_atomic_helper_commit+0x18c/0x1f4
[ 7966.097900] drm_atomic_nonblocking_commit+0xc4/0xdc
[ 7966.103000] drm_mode_atomic_ioctl+0x89c/0xaa0
[ 7966.107572] drm_ioctl_kernel+0x15c/0x260
[ 7966.111697] drm_ioctl+0x460/0x828
[ 7966.115200] drm_compat_ioctl+0x1bc/0x230
[ 7966.119323] __arm64_compat_sys_ioctl+0x1b0/0x1c4
[ 7966.124161] invoke_syscall+0xd4/0x2c8
[ 7966.128016] el0_svc_common+0x124/0x200
[ 7966.131962] do_el0_svc_compat+0x54/0x64
[ 7966.135993] el0_svc_compat+0x24/0x34
[ 7966.139762] el0t_32_sync_handler+0xc0/0xf0
[ 7966.144067] el0t_32_sync+0x19c/0x1a0
[ 7966.147837]
[ 7966.149374] The buggy address belongs to the object at ffffff8090294900
[ 7966.149374] which belongs to the cache kmalloc-256 of size 256
[ 7966.162218] The buggy address is located 32 bytes inside of
[ 7966.162218] 256-byte region [ffffff8090294900, ffffff8090294a00)
[ 7966.174174] The buggy address belongs to the page:
[ 7966.179101] page:fffffffe0240a400 refcount:1 mapcount:0
mapping:0000000000000000 index:0xffffff8090294900 pfn:0x110290
[ 7966.190079] head:fffffffe0240a400 order:3 compound_mapcount:0
compound_pincount:0
[ 7966.197760] flags: 0x8000000000010200(slab|head|zone=2)
[ 7966.203137] raw: 8000000000010200 fffffffe002f3c08 fffffffe023f9008
ffffff808000c980
[ 7966.211086] raw: ffffff8090294900 0000000000200005 00000001ffffffff
0000000000000000
[ 7966.219034] page dumped because: kasan: bad access detected
[ 7966.224763]
[ 7966.226300] Memory state around the buggy address:
[ 7966.231228] ffffff8090294800: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7966.238641] ffffff8090294880: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7966.246055] >ffffff8090294900: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7966.253466] ^
[ 7966.257849] ffffff8090294980: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 7966.265267] ffffff8090294a00: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 7966.272678] ==================================================================

2021-10-07 13:32:26

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Wednesday, October 6th, 2021 at 2:12 AM, Hillf Danton <[email protected]> wrote:

> Could you please check if it is due to count underflow? Given nothing wrong
>
> on the other side based on the efforts
>
> "We looked through the users of put_ucounts and we don't see any obvious buggy
>
> users that would be freeing the data structure early."
>
> Thanks
>
> Hillf
>
> --- linux-5.14.4/kernel/ucount.c
>
> +++ b/kernel/ucount.c
>
> @@ -152,7 +152,10 @@ static void hlist_add_ucounts(struct uco
>
> struct ucounts *get_ucounts(struct ucounts *ucounts)
>
> {
>
> - if (ucounts && atomic_add_negative(1, &ucounts->count)) {
>
> - if (!ucounts)
>
> - return NULL;
>
>
> - WARN_ON(!atomic_read(&ucounts->count));
>
> - if (atomic_add_negative(1, &ucounts->count)) {
>
> put_ucounts(ucounts);
> ucounts = NULL;
>
>
> }
>
> --
>

For me above patch changed slightly the printed output. Now the warning
comes from 'cleanup_net' instead of 'free_user_ns'. My system was also
still responsive after the bug occurred which didn't happen previously.
I can't say if this means anything or if this is result of above patch
or instability of my reproducer.

------------[ cut here ]------------
WARNING: CPU: 2 PID: 27643 at kernel/ucount.c:256 dec_ucount+0x43/0x50
Modules linked in: <cut>
CPU: 2 PID: 27643 Comm: kworker/u8:3 Not tainted 5.14.9 #1 0274f3d0712a6dadc9a2cf8341ae333de732a31a
Workqueue: netns cleanup_net
RIP: 0010:dec_ucount+0x43/0x50
Code: 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 48 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 fd fc ff ff <0f> 0b eb e7 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 f8 48
RSP: 0018:ffffb34fc34cfe30 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa448eec5f3b0 RCX: ffffa447cfe1f540
RDX: ffffa447cfe1f580 RSI: ffffffffffffffff RDI: ffffa447c445c780
RBP: ffffa448eec5f380 R08: 0000000000000040 R09: ffffa44a196ac040
R10: 00000000001436be R11: 0000000000000259 R12: ffffb34fc34cfe10
R13: ffffb34fc34cfe40 R14: 00000000ffffffff R15: ffffa448eec5d414
FS: 0000000000000000(0000) GS:ffffa44a19700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000072a95d359030 CR3: 000000000b20e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
cleanup_net+0x2e2/0x370
process_one_work+0x1e1/0x380
worker_thread+0x50/0x3a0
? rescuer_thread+0x360/0x360
kthread+0x127/0x150
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
---[ end trace e5fdc3317f00d0e8 ]---
BUG: kernel NULL pointer dereference, address: 00000000000001e8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] SMP PTI
CPU: 2 PID: 27643 Comm: kworker/u8:3 Tainted: G W 5.14.9 #1 0274f3d0712a6dadc9a2cf8341ae333de732a31a
Workqueue: netns cleanup_net
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 fd fc ff ff 0f 0b eb e7 66
RSP: 0018:ffffb34fc34cfe30 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa448eec5f3b0 RCX: ffffa447cfe1f540
RDX: ffffa447cfe1f580 RSI: ffffffffffffffff RDI: ffffa447c445c780
RBP: ffffa448eec5f380 R08: 0000000000000040 R09: ffffa44a196ac040
R10: 00000000001436be R11: 0000000000000259 R12: ffffb34fc34cfe10
R13: ffffb34fc34cfe40 R14: 00000000ffffffff R15: ffffa448eec5d414
FS: 0000000000000000(0000) GS:ffffa44a19700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 000000000b20e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
cleanup_net+0x2e2/0x370
process_one_work+0x1e1/0x380
worker_thread+0x50/0x3a0
? rescuer_thread+0x360/0x360
kthread+0x127/0x150
? set_kthread_struct+0x40/0x40
ret_from_fork+0x22/0x30
Modules linked in: <cut>
CR2: 00000000000001e8
---[ end trace e5fdc3317f00d0e9 ]---
RIP: 0010:dec_ucount+0x32/0x50
Code: 74 34 89 f6 48 89 f9 4c 8d 04 f5 20 00 00 00 4a 8d 14 01 48 8b 02 48 89 c6 48 83 ee 01 78 1c f0 48 0f b1 32 75 f0 48 8b 41 10 <48> 8b 88 e8 01 00 00 48 85 c9 75 d9 e9 fd fc ff ff 0f 0b eb e7 66
RSP: 0018:ffffb34fc34cfe30 EFLAGS: 00010297
RAX: 0000000000000000 RBX: ffffa448eec5f3b0 RCX: ffffa447cfe1f540
RDX: ffffa447cfe1f580 RSI: ffffffffffffffff RDI: ffffa447c445c780
RBP: ffffa448eec5f380 R08: 0000000000000040 R09: ffffa44a196ac040
R10: 00000000001436be R11: 0000000000000259 R12: ffffb34fc34cfe10
R13: ffffb34fc34cfe40 R14: 00000000ffffffff R15: ffffa448eec5d414
FS: 0000000000000000(0000) GS:ffffa44a19700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001e8 CR3: 000000000b20e005 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

2021-10-11 16:28:00

[permalink] [raw]

Subject: Re: linux 5.14.3: free_user_ns causes NULL pointer dereference

On Mon, Oct 04, 2021 at 12:19:16PM -0500, Eric W. Biederman wrote:
> [email protected] (Eric W. Biederman) writes:
>
> > Adding Rune Kleveland to the discussion as he also seems to have
> > reproduced the issue.
> >
> > Alex and I have been starring at the code and the reports and this
> > bug is hiding well. Here is what we have figured out so far.
> >
> > Both the warning from free_user_ns calling dec_ucount that Jordan Glover
> > reported and the KASAN error that Yu Zhao has reported appear to have
> > the same cause. Using a ucounts structure after it has been freed and
> > reallocated as something else.
> >
> > I have just skimmed through the recent report from Rune Kleveland
> > and it appears also to be a use after free. Especially since the
> > second failure in the log is slub complaining about trying to free
> > the ucounts data structure.
> >
> > We looked through the users of put_ucounts and we don't see any obvious
> > buggy users that would be freeing the data structure early.
> >
> > Alex has tried to reproduce this so far is not having any luck.
> > Folks can you tell what compiler versions you are using and share your
> > kernel config with us? That might help.
> >
> > The little debug diff below is my guess of what is happening. If the
> > folks who can reproduce this issue can try the patch below and let me
> > know if the warnings fire that would be appreciated. It is still not
> > enough to track down the bug but at least it will confirm my current
> > hypothesis about how things look before there is a use of memory after
> > it is freed.
>
> Bah. Scratch that test patch. I just double checked myself and
> cred->ucounts and cred->user_ns->ucounts should never be equal,
> as the user namespace is counted in it's parent user namespace.
>
> That observation now tells me I have a parent user namespace that went
> corrupt.
>
> Back to the drawing board.
>
>
> > Thank you,
> > Eric
> >
> > diff --git a/kernel/cred.c b/kernel/cred.c
> > index f784e08c2fbd..e7ffaa3cf5a6 100644
> > --- a/kernel/cred.c
> > +++ b/kernel/cred.c
> > @@ -120,6 +120,12 @@ static void put_cred_rcu(struct rcu_head *rcu)
> > if (cred->group_info)
> > put_group_info(cred->group_info);
> > free_uid(cred->user);
> > +#if 1
> > + if ((cred->ucounts == cred->user_ns->ucounts) &&
> > + (atomic_read(&cred->ucounts->count) == 1)) {
> > + WARN_ONCE(1, "put_cred_rcu: ucount count 1\n");
> > + }
> > +#endif
> > if (cred->ucounts)
> > put_ucounts(cred->ucounts);
> > put_user_ns(cred->user_ns);
> > diff --git a/kernel/exit.c b/kernel/exit.c
> > index 91a43e57a32e..60fd88b34c1a 100644
> > --- a/kernel/exit.c
> > +++ b/kernel/exit.c
> > @@ -743,6 +743,13 @@ void __noreturn do_exit(long code)
> > if (unlikely(!tsk->pid))
> > panic("Attempted to kill the idle task!");
> >
> > +#if 1
> > + if ((tsk->cred->ucounts == tsk->cred->user_ns->ucounts) &&
> > + (atomic_read(tsk->cred->ucounts->count) == 1)) {
> > + WARN_ONCE(1, "do_exit: ucount count 1\n");
> > + }
> > +#endif
> > +
> > /*
> > * If do_exit is called because this processes oopsed, it's possible
> > * that get_fs() was left as KERNEL_DS, so reset it to USER_DS before
>

It looks like I found a fairly reliable reproducer. I'm able to reproduce
the problem on ARCH=um.

I added a bit of debugging and it looks like the namespace has a reference
count problem:

[ 6.970000] ------------[ cut here ]------------
[ 6.970000] WARNING: CPU: 0 PID: 18 at kernel/ucount.c:257 dec_ucount+0x86/0xfc
[ 6.970000] dec_ucounts: accessing freed ucounts: 0000000061202780
^^^^^^^^^^^^^^^^

This is a pointer to the freed ucounts.

[ 6.970000] Modules linked in:
[ 6.970000] CPU: 0 PID: 18 Comm: kworker/u2:1 Tainted: G W 5.15.0-rc3-00082-ga81d1c67022e-dirty #14
[ 6.970000] Workqueue: netns cleanup_net
[ 6.970000] Stack:
[ 6.970000] 800bfca0 602693cb 603d3767 00000000
[ 6.970000] 6048c52d 00000009 603d23b5 60046b28
[ 6.970000] 800bfcc0 603d37a7 603d3767 800bfd20
[ 6.970000] Call Trace:
[ 6.970000] [<603d23b5>] ? _printk+0x0/0xa1
[ 6.970000] [<600238df>] show_stack+0x14b/0x15a
[ 6.970000] [<602693cb>] ? dump_stack_print_info+0xeb/0xfa
[ 6.970000] [<603d3767>] ? dump_stack+0x0/0x45
[ 6.970000] [<603d23b5>] ? _printk+0x0/0xa1
[ 6.970000] [<60046b28>] ? __local_bh_enable_ip+0x0/0xf8
[ 6.970000] [<603d37a7>] dump_stack+0x40/0x45
[ 6.970000] [<603d3767>] ? dump_stack+0x0/0x45
[ 6.970000] [<60043633>] __warn+0x101/0x149
[ 6.970000] [<603d1559>] warn_slowpath_fmt+0xde/0xec
[ 6.970000] [<6003953b>] ? set_signals+0x0/0x48
[ 6.970000] [<60265087>] ? free_object+0x1d/0x73
[ 6.970000] [<603d147b>] ? warn_slowpath_fmt+0x0/0xec
[ 6.970000] [<60083a2d>] ? destroy_rcu_head+0x27/0x29
[ 6.970000] [<60083ba3>] ? __wait_rcu_gp+0x12d/0x14f
[ 6.970000] [<600846bb>] ? rcu_barrier+0x0/0x40
[ 6.970000] [<6006471d>] dec_ucount+0x86/0xfc
[ 6.970000] [<60064697>] ? dec_ucount+0x0/0xfc
[ 6.970000] [<6031221f>] cleanup_net+0x2cb/0x326
[ 6.970000] [<60039468>] ? unblock_signals+0x0/0xd3
[ 6.970000] [<60059daa>] process_one_work+0x1e1/0x2df
[ 6.970000] [<60039445>] ? block_signals+0x0/0x23
[ 6.970000] [<6003947c>] ? unblock_signals+0x14/0xd3
[ 6.970000] [<6005658e>] ? move_linked_works+0x0/0x8a
[ 6.970000] [<6005a49c>] worker_thread+0x293/0x3c0
[ 6.970000] [<600566bc>] ? set_pf_worker+0x0/0x6d
[ 6.970000] [<6005ecf2>] ? arch_local_irq_save+0x0/0x29
[ 6.970000] [<60045960>] ? do_exit+0x0/0x93e
[ 6.970000] [<6005a209>] ? worker_thread+0x0/0x3c0
[ 6.970000] [<60060554>] kthread+0x171/0x179
[ 6.970000] [<6002230e>] new_thread_handler+0x8e/0xbf
[ 6.970000] ---[ end trace 02feadf6f35abde2 ]---
[ 6.970000]
[ 6.970000] Modules linked in:
[ 6.970000] Pid: 18, comm: kworker/u2:1 Tainted: G W 5.15.0-rc3-00082-ga81d1c67022e-dirty
[ 6.970000] RIP: 0033:[<000000006006476f>]
[ 6.970000] RSP: 00000000800bfe20 EFLAGS: 00010246
[ 6.970000] RAX: 0000000000000000 RBX: 0000000000000048 RCX: 0000000000000000
[ 6.970000] RDX: 00000000612027c8 RSI: 00000000ffffffff RDI: 0000000000000009
[ 6.970000] RBP: 00000000800bfe40 R08: 0000000060551bc8 R09: 0000000000015920
[ 6.970000] R10: 00000000600203e0 R11: 0000000000015920 R12: 0000000061202780
[ 6.970000] R13: 000000006057bca6 R14: 0000000061202780 R15: 0000000060046b28
[ 6.970000] Kernel panic - not syncing: Segfault with no mm
[ 6.970000] CPU: 0 PID: 18 Comm: kworker/u2:1 Tainted: G W 5.15.0-rc3-00082-ga81d1c67022e-dirty #14
[ 6.970000] Workqueue: netns cleanup_net
[ 6.970000] Stack:
[ 6.970000] 800bfe48 612d69c0 800bfe70 60064697
[ 6.970000] 800bfeb0 6031221f ffffffff800bfe90 605735c0
[ 6.970000] 60039468 ffff8d00 800bfe70 800bfe70
[ 6.970000] Call Trace:
[ 6.970000] [<60064697>] ? dec_ucount+0x0/0xfc
[ 6.970000] [<6031221f>] cleanup_net+0x2cb/0x326
[ 6.970000] [<60039468>] ? unblock_signals+0x0/0xd3
[ 6.970000] [<60059daa>] process_one_work+0x1e1/0x2df
[ 6.970000] [<60039445>] ? block_signals+0x0/0x23
[ 6.970000] [<6003947c>] ? unblock_signals+0x14/0xd3
[ 6.970000] [<6005658e>] ? move_linked_works+0x0/0x8a
[ 6.970000] [<6005a49c>] worker_thread+0x293/0x3c0
[ 6.970000] [<600566bc>] ? set_pf_worker+0x0/0x6d
[ 6.970000] [<6005ecf2>] ? arch_local_irq_save+0x0/0x29
[ 6.970000] [<60045960>] ? do_exit+0x0/0x93e
[ 6.970000] [<6005a209>] ? worker_thread+0x0/0x3c0
[ 6.970000] [<60060554>] kthread+0x171/0x179
[ 6.970000] [<6002230e>] new_thread_handler+0x8e/0xbf

This is where this ucounts was created:

[ 5.530000] ------------[ cut here ]------------
[ 5.530000] WARNING: CPU: 0 PID: 216 at kernel/ucount.c:187 alloc_ucounts+0x174/0x189
[ 5.530000] allooc_ucounts: new ucounts ucounts=0000000061202780
[ 5.530000] Modules linked in:
[ 5.530000] CPU: 0 PID: 216 Comm: unshare Tainted: G W 5.15.0-rc3-00082-ga81d1c67022e-dirty #14
[ 5.530000] Stack:
[ 5.530000] 8045bb00 602693cb 603d3767 00000000
[ 5.530000] 6048c52d 00000009 603d23b5 6063d388
[ 5.530000] 8045bb20 603d37a7 603d3767 8045bb80
[ 5.530000] Call Trace:
[ 5.530000] [<603d23b5>] ? _printk+0x0/0xa1
[ 5.530000] [<600238df>] show_stack+0x14b/0x15a
[ 5.530000] [<602693cb>] ? dump_stack_print_info+0xeb/0xfa
[ 5.530000] [<603d3767>] ? dump_stack+0x0/0x45
[ 5.530000] [<603d23b5>] ? _printk+0x0/0xa1
[ 5.530000] [<603d37a7>] dump_stack+0x40/0x45
[ 5.530000] [<603d3767>] ? dump_stack+0x0/0x45
[ 5.530000] [<60043633>] __warn+0x101/0x149
[ 5.530000] [<60039468>] ? unblock_signals+0x0/0xd3
[ 5.530000] [<603d1559>] warn_slowpath_fmt+0xde/0xec
[ 5.530000] [<60274613>] ? memcmp+0x0/0x2f
[ 5.530000] [<60039445>] ? block_signals+0x0/0x23
[ 5.530000] [<603d147b>] ? warn_slowpath_fmt+0x0/0xec
[ 5.530000] [<600ff06d>] ? arch_local_irq_save+0x0/0x29
[ 5.530000] [<600ff8d7>] ? slab_post_alloc_hook+0x5b/0x1ae
[ 5.530000] [<60039445>] ? block_signals+0x0/0x23
[ 5.530000] [<600645e2>] alloc_ucounts+0x174/0x189
[ 5.530000] [<600645f7>] ? inc_ucount+0x0/0xa0
[ 5.530000] [<60064618>] inc_ucount+0x21/0xa0
[ 5.530000] [<6012f6e0>] alloc_mnt_ns+0x53/0x193
[ 5.530000] [<600ff06d>] ? arch_local_irq_save+0x0/0x29
[ 5.530000] [<601334c6>] ? copy_mnt_ns+0x0/0x29e
[ 5.530000] [<60133570>] copy_mnt_ns+0xaa/0x29e
[ 5.530000] [<600609a6>] create_new_namespaces+0x7c/0x2b4
[ 5.530000] [<60060ed7>] unshare_nsproxy_namespaces+0xb1/0xcb
[ 5.530000] [<600b1111>] ? unshare_userns+0x4c/0x78
[ 5.530000] [<60042dcb>] ksys_unshare+0x179/0x345
[ 5.530000] [<60042fb4>] sys_unshare+0x1d/0x21
[ 5.530000] [<60025b0a>] handle_syscall+0x8e/0xbe
[ 5.530000] [<6003c471>] userspace+0x3cf/0x536
[ 5.530000] [<600223e0>] fork_handler+0xa1/0xa3
[ 5.530000] ---[ end trace 02feadf6f35abdd7 ]---

--
Rgrds, legion

2021-10-12 17:32:30