Hi,
after enabling KASAN literally I was bombarded with messages about
slab-use-after-free in drm_sched_get_cleanup_job.
All messages has similar backtrace:
[ 1138.492091] ==================================================================
[ 1138.492104] BUG: KASAN: slab-use-after-free in
drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492120] Read of size 8 at addr ffff88815adf04c0 by task sdma1/749
[ 1138.492130] CPU: 29 PID: 749 Comm: sdma1 Tainted: G W L
------- --- 6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
#1
[ 1138.492136] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
[ 1138.492141] Call Trace:
[ 1138.492145] <TASK>
[ 1138.492150] dump_stack_lvl+0x72/0xc0
[ 1138.492159] print_report+0xcf/0x670
[ 1138.492169] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492181] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492193] kasan_report+0xa4/0xe0
[ 1138.492200] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492215] drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1138.492229] drm_sched_main+0x643/0x990 [gpu_sched]
[ 1138.492245] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1138.492259] ? __pfx_autoremove_wake_function+0x10/0x10
[ 1138.492269] ? __kthread_parkme+0xc1/0x1f0
[ 1138.492277] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1138.492288] kthread+0x29e/0x340
[ 1138.492294] ? __pfx_kthread+0x10/0x10
[ 1138.492301] ret_from_fork+0x2c/0x50
[ 1138.492314] </TASK>
[ 1138.492320] Allocated by task 10867:
[ 1138.492323] kasan_save_stack+0x2f/0x50
[ 1138.492329] kasan_set_track+0x21/0x30
[ 1138.492334] __kasan_kmalloc+0x8b/0x90
[ 1138.492339] amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
[ 1138.493112] drm_file_alloc+0x46e/0x880
[ 1138.493120] drm_open_helper+0x161/0x460
[ 1138.493126] drm_open+0x1e7/0x5c0
[ 1138.493131] drm_stub_open+0x24d/0x400
[ 1138.493138] chrdev_open+0x215/0x620
[ 1138.493144] do_dentry_open+0x5f1/0x1000
[ 1138.493149] path_openat+0x1b3d/0x28a0
[ 1138.493156] do_filp_open+0x1bd/0x400
[ 1138.493161] do_sys_openat2+0x140/0x420
[ 1138.493167] __x64_sys_openat+0x11f/0x1d0
[ 1138.493173] do_syscall_64+0x5b/0x80
[ 1138.493179] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1138.493189] Freed by task 10867:
[ 1138.493193] kasan_save_stack+0x2f/0x50
[ 1138.493199] kasan_set_track+0x21/0x30
[ 1138.493205] kasan_save_free_info+0x2a/0x50
[ 1138.493210] __kasan_slab_free+0x107/0x1a0
[ 1138.493216] slab_free_freelist_hook+0x11e/0x1d0
[ 1138.493221] __kmem_cache_free+0xbc/0x2e0
[ 1138.493227] amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
[ 1138.493963] drm_file_free.part.0+0x638/0xb70
[ 1138.493969] drm_release+0x1ea/0x470
[ 1138.493975] __fput+0x213/0x9e0
[ 1138.493981] task_work_run+0x11b/0x200
[ 1138.493987] exit_to_user_mode_prepare+0x23a/0x260
[ 1138.493994] syscall_exit_to_user_mode+0x16/0x50
[ 1138.494001] do_syscall_64+0x67/0x80
[ 1138.494006] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1138.494015] The buggy address belongs to the object at ffff88815adf0000
which belongs to the cache kmalloc-4k of size 4096
[ 1138.494021] The buggy address is located 1216 bytes inside of
freed 4096-byte region [ffff88815adf0000, ffff88815adf1000)
[ 1138.494030] The buggy address belongs to the physical page:
[ 1138.494034] page:0000000017cd5a82 refcount:1 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x15adf0
[ 1138.494041] head:0000000017cd5a82 order:3 entire_mapcount:0
nr_pages_mapped:0 pincount:0
[ 1138.494046] flags:
0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 1138.494055] raw: 0017ffffc0010200 ffff88810004d040 dead000000000122
0000000000000000
[ 1138.494061] raw: 0000000000000000 0000000000040004 00000001ffffffff
0000000000000000
[ 1138.494065] page dumped because: kasan: bad access detected
[ 1138.494071] Memory state around the buggy address:
[ 1138.494076] ffff88815adf0380: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494080] ffff88815adf0400: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494084] >ffff88815adf0480: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494088] ^
[ 1138.494092] ffff88815adf0500: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494095] ffff88815adf0580: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1138.494099] ==================================================================
[ 1138.494140] Disabling lock debugging due to kernel taint
First thing I checked RAM for sure that it is definitely not a
hardware problem. I started investigating which application triggered
this message. I started to notice that it usually happens when any
game starts in the steam client. So I came to the culprit of the
problem is the compilation of shaders. For reproduction we should have
a kernel with enabled KASAN and a steam client. After allowing
background processing of Vulkan shaders the issue happens.
Unfortunately it does not happen immediately as you may expect.
I record four screencasts and uploaded them on to youtube [1-4] for
proving that definitely processing of Vulkan shaders culprit here but
this may happens after some time.
[1] https://youtu.be/hFuWlGTbetg
[2] https://youtu.be/TAOqoYUYA5Q
[3] https://youtu.be/9SZNXWhwAZs
[4] https://youtu.be/W9gtYyL-2pg
I do not know which information would be also useful. Please ask me.
I attached a full kernel log and config if someone would be interested
to see it.
--
Best Regards,
Mike Gavrilov.
Hi Mikhail,
Am 20.03.23 um 13:05 schrieb Mikhail Gavrilov:
> Hi,
> after enabling KASAN literally I was bombarded with messages about
> slab-use-after-free in drm_sched_get_cleanup_job.
mhm, interesting.
> All messages has similar backtrace:
> [ 1138.492091] ==================================================================
> [ 1138.492104] BUG: KASAN: slab-use-after-free in
> drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492120] Read of size 8 at addr ffff88815adf04c0 by task sdma1/749
>
> [ 1138.492130] CPU: 29 PID: 749 Comm: sdma1 Tainted: G W L
> ------- --- 6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
> #1
> [ 1138.492136] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
> [ 1138.492141] Call Trace:
> [ 1138.492145] <TASK>
> [ 1138.492150] dump_stack_lvl+0x72/0xc0
> [ 1138.492159] print_report+0xcf/0x670
> [ 1138.492169] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492181] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492193] kasan_report+0xa4/0xe0
> [ 1138.492200] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492215] drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492229] drm_sched_main+0x643/0x990 [gpu_sched]
> [ 1138.492245] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1138.492259] ? __pfx_autoremove_wake_function+0x10/0x10
> [ 1138.492269] ? __kthread_parkme+0xc1/0x1f0
> [ 1138.492277] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1138.492288] kthread+0x29e/0x340
> [ 1138.492294] ? __pfx_kthread+0x10/0x10
> [ 1138.492301] ret_from_fork+0x2c/0x50
> [ 1138.492314] </TASK>
>
> [ 1138.492320] Allocated by task 10867:
> [ 1138.492323] kasan_save_stack+0x2f/0x50
> [ 1138.492329] kasan_set_track+0x21/0x30
> [ 1138.492334] __kasan_kmalloc+0x8b/0x90
> [ 1138.492339] amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
> [ 1138.493112] drm_file_alloc+0x46e/0x880
> [ 1138.493120] drm_open_helper+0x161/0x460
> [ 1138.493126] drm_open+0x1e7/0x5c0
> [ 1138.493131] drm_stub_open+0x24d/0x400
> [ 1138.493138] chrdev_open+0x215/0x620
> [ 1138.493144] do_dentry_open+0x5f1/0x1000
> [ 1138.493149] path_openat+0x1b3d/0x28a0
> [ 1138.493156] do_filp_open+0x1bd/0x400
> [ 1138.493161] do_sys_openat2+0x140/0x420
> [ 1138.493167] __x64_sys_openat+0x11f/0x1d0
> [ 1138.493173] do_syscall_64+0x5b/0x80
> [ 1138.493179] entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1138.493189] Freed by task 10867:
> [ 1138.493193] kasan_save_stack+0x2f/0x50
> [ 1138.493199] kasan_set_track+0x21/0x30
> [ 1138.493205] kasan_save_free_info+0x2a/0x50
> [ 1138.493210] __kasan_slab_free+0x107/0x1a0
> [ 1138.493216] slab_free_freelist_hook+0x11e/0x1d0
> [ 1138.493221] __kmem_cache_free+0xbc/0x2e0
> [ 1138.493227] amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
> [ 1138.493963] drm_file_free.part.0+0x638/0xb70
> [ 1138.493969] drm_release+0x1ea/0x470
> [ 1138.493975] __fput+0x213/0x9e0
> [ 1138.493981] task_work_run+0x11b/0x200
> [ 1138.493987] exit_to_user_mode_prepare+0x23a/0x260
> [ 1138.493994] syscall_exit_to_user_mode+0x16/0x50
> [ 1138.494001] do_syscall_64+0x67/0x80
> [ 1138.494006] entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1138.494015] The buggy address belongs to the object at ffff88815adf0000
> which belongs to the cache kmalloc-4k of size 4096
> [ 1138.494021] The buggy address is located 1216 bytes inside of
> freed 4096-byte region [ffff88815adf0000, ffff88815adf1000)
>
> [ 1138.494030] The buggy address belongs to the physical page:
> [ 1138.494034] page:0000000017cd5a82 refcount:1 mapcount:0
> mapping:0000000000000000 index:0x0 pfn:0x15adf0
> [ 1138.494041] head:0000000017cd5a82 order:3 entire_mapcount:0
> nr_pages_mapped:0 pincount:0
> [ 1138.494046] flags:
> 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
> [ 1138.494055] raw: 0017ffffc0010200 ffff88810004d040 dead000000000122
> 0000000000000000
> [ 1138.494061] raw: 0000000000000000 0000000000040004 00000001ffffffff
> 0000000000000000
> [ 1138.494065] page dumped because: kasan: bad access detected
>
> [ 1138.494071] Memory state around the buggy address:
> [ 1138.494076] ffff88815adf0380: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494080] ffff88815adf0400: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494084] >ffff88815adf0480: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494088] ^
> [ 1138.494092] ffff88815adf0500: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494095] ffff88815adf0580: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494099] ==================================================================
> [ 1138.494140] Disabling lock debugging due to kernel taint
>
> First thing I checked RAM for sure that it is definitely not a
> hardware problem. I started investigating which application triggered
> this message. I started to notice that it usually happens when any
> game starts in the steam client. So I came to the culprit of the
> problem is the compilation of shaders. For reproduction we should have
> a kernel with enabled KASAN and a steam client. After allowing
> background processing of Vulkan shaders the issue happens.
> Unfortunately it does not happen immediately as you may expect.
> I record four screencasts and uploaded them on to youtube [1-4] for
> proving that definitely processing of Vulkan shaders culprit here but
> this may happens after some time.
>
> [1] https://youtu.be/hFuWlGTbetg
> [2] https://youtu.be/TAOqoYUYA5Q
> [3] https://youtu.be/9SZNXWhwAZs
> [4] https://youtu.be/W9gtYyL-2pg
>
> I do not know which information would be also useful. Please ask me.
>
> I attached a full kernel log and config if someone would be interested
> to see it.
That looks like a reference counting issue to me.
I'm going to take a look, but we have already fixed one of those recently.
Probably best that you try this on drm-fixes, just to double check that
this isn't the same issue.
Thanks,
Christian.
On Tue, Mar 21, 2023 at 11:47 PM Christian König
<[email protected]> wrote:
>
> Hi Mikhail,
>
> That looks like a reference counting issue to me.
>
> I'm going to take a look, but we have already fixed one of those recently.
>
> Probably best that you try this on drm-fixes, just to double check that
> this isn't the same issue.
>
Hi Christian,
you meant this branch?
$ git clone -b drm-fixes git://anongit.freedesktop.org/drm/drm linux-drm
If yes I just checked and unfortunately see this issue unfixed there.
[ 1984.295833] ==================================================================
[ 1984.295876] BUG: KASAN: slab-use-after-free in
drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.295898] Read of size 8 at addr ffff88814cadc4c0 by task sdma1/764
[ 1984.295924] CPU: 12 PID: 764 Comm: sdma1 Tainted: G W L
6.3.0-rc3-drm-fixes+ #1
[ 1984.295937] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
[ 1984.295951] Call Trace:
[ 1984.295963] <TASK>
[ 1984.295975] dump_stack_lvl+0x72/0xc0
[ 1984.295991] print_report+0xcf/0x670
[ 1984.296007] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296030] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296047] kasan_report+0xa4/0xe0
[ 1984.296118] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296149] drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
[ 1984.296175] drm_sched_main+0x643/0x990 [gpu_sched]
[ 1984.296204] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1984.296222] ? __pfx_autoremove_wake_function+0x10/0x10
[ 1984.296290] ? __kthread_parkme+0xc1/0x1f0
[ 1984.296304] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
[ 1984.296321] kthread+0x29e/0x340
[ 1984.296334] ? __pfx_kthread+0x10/0x10
[ 1984.296501] ret_from_fork+0x2c/0x50
[ 1984.296518] </TASK>
[ 1984.296539] Allocated by task 12194:
[ 1984.296552] kasan_save_stack+0x2f/0x50
[ 1984.296566] kasan_set_track+0x21/0x30
[ 1984.296578] __kasan_kmalloc+0x8b/0x90
[ 1984.296590] amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
[ 1984.297051] drm_file_alloc+0x46e/0x880
[ 1984.297064] drm_open_helper+0x161/0x460
[ 1984.297076] drm_open+0x1e7/0x5c0
[ 1984.297089] drm_stub_open+0x24d/0x400
[ 1984.297107] chrdev_open+0x215/0x620
[ 1984.297125] do_dentry_open+0x5f1/0x1000
[ 1984.297146] path_openat+0x1b3d/0x28a0
[ 1984.297164] do_filp_open+0x1bd/0x400
[ 1984.297180] do_sys_openat2+0x140/0x420
[ 1984.297197] __x64_sys_openat+0x11f/0x1d0
[ 1984.297213] do_syscall_64+0x5b/0x80
[ 1984.297231] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1984.297266] Freed by task 12195:
[ 1984.297284] kasan_save_stack+0x2f/0x50
[ 1984.297303] kasan_set_track+0x21/0x30
[ 1984.297323] kasan_save_free_info+0x2a/0x50
[ 1984.297343] __kasan_slab_free+0x107/0x1a0
[ 1984.297361] slab_free_freelist_hook+0x11e/0x1d0
[ 1984.297373] __kmem_cache_free+0xbc/0x2e0
[ 1984.297385] amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
[ 1984.297821] drm_file_free.part.0+0x638/0xb70
[ 1984.297834] drm_release+0x1ea/0x470
[ 1984.297845] __fput+0x213/0x9e0
[ 1984.297857] task_work_run+0x11b/0x200
[ 1984.297869] exit_to_user_mode_prepare+0x23a/0x260
[ 1984.297883] syscall_exit_to_user_mode+0x16/0x50
[ 1984.297896] do_syscall_64+0x67/0x80
[ 1984.297907] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1984.298033] Last potentially related work creation:
[ 1984.298044] kasan_save_stack+0x2f/0x50
[ 1984.298057] __kasan_record_aux_stack+0x97/0xb0
[ 1984.298075] __call_rcu_common.constprop.0+0xf8/0x1af0
[ 1984.298095] amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
[ 1984.298557] amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
[ 1984.299055] amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
[ 1984.299624] drm_ioctl_kernel+0x1f8/0x3d0
[ 1984.299637] drm_ioctl+0x4c1/0xaa0
[ 1984.299649] amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
[ 1984.300083] __x64_sys_ioctl+0x12d/0x1a0
[ 1984.300097] do_syscall_64+0x5b/0x80
[ 1984.300109] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1984.300135] Second to last potentially related work creation:
[ 1984.300149] kasan_save_stack+0x2f/0x50
[ 1984.300167] __kasan_record_aux_stack+0x97/0xb0
[ 1984.300185] __call_rcu_common.constprop.0+0xf8/0x1af0
[ 1984.300203] amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
[ 1984.300692] amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
[ 1984.301133] amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
[ 1984.301577] drm_ioctl_kernel+0x1f8/0x3d0
[ 1984.301598] drm_ioctl+0x4c1/0xaa0
[ 1984.301610] amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
[ 1984.302043] __x64_sys_ioctl+0x12d/0x1a0
[ 1984.302056] do_syscall_64+0x5b/0x80
[ 1984.302068] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1984.302090] The buggy address belongs to the object at ffff88814cadc000
which belongs to the cache kmalloc-4k of size 4096
[ 1984.302103] The buggy address is located 1216 bytes inside of
freed 4096-byte region [ffff88814cadc000, ffff88814cadd000)
[ 1984.302129] The buggy address belongs to the physical page:
[ 1984.302141] page:00000000103c1864 refcount:1 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x14cad8
[ 1984.302155] head:00000000103c1864 order:3 entire_mapcount:0
nr_pages_mapped:0 pincount:0
[ 1984.302168] flags:
0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 1984.302184] raw: 0017ffffc0010200 ffff88810004d040 ffffea00311f0200
dead000000000002
[ 1984.302196] raw: 0000000000000000 0000000000040004 00000001ffffffff
0000000000000000
[ 1984.302208] page dumped because: kasan: bad access detected
[ 1984.302230] Memory state around the buggy address:
[ 1984.302241] ffff88814cadc380: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302252] ffff88814cadc400: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302263] >ffff88814cadc480: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302274] ^
[ 1984.302285] ffff88814cadc500: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302296] ffff88814cadc580: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 1984.302307] ==================================================================
[ 1984.302339] Disabling lock debugging due to kernel taint
--
Best Regards,
Mike Gavrilov.
Am 23.03.23 um 20:05 schrieb Mikhail Gavrilov:
> On Tue, Mar 21, 2023 at 11:47 PM Christian König
> <[email protected]> wrote:
>> Hi Mikhail,
>>
>> That looks like a reference counting issue to me.
>>
>> I'm going to take a look, but we have already fixed one of those recently.
>>
>> Probably best that you try this on drm-fixes, just to double check that
>> this isn't the same issue.
>>
> Hi Christian,
> you meant this branch?
> $ git clone -b drm-fixes git://anongit.freedesktop.org/drm/drm linux-drm
Yeah, that one
Thanks for the info, looks like this isn't fixed.
Christian.
>
> If yes I just checked and unfortunately see this issue unfixed there.
>
> [ 1984.295833] ==================================================================
> [ 1984.295876] BUG: KASAN: slab-use-after-free in
> drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.295898] Read of size 8 at addr ffff88814cadc4c0 by task sdma1/764
>
> [ 1984.295924] CPU: 12 PID: 764 Comm: sdma1 Tainted: G W L
> 6.3.0-rc3-drm-fixes+ #1
> [ 1984.295937] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
> [ 1984.295951] Call Trace:
> [ 1984.295963] <TASK>
> [ 1984.295975] dump_stack_lvl+0x72/0xc0
> [ 1984.295991] print_report+0xcf/0x670
> [ 1984.296007] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296030] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296047] kasan_report+0xa4/0xe0
> [ 1984.296118] ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296149] drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1984.296175] drm_sched_main+0x643/0x990 [gpu_sched]
> [ 1984.296204] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1984.296222] ? __pfx_autoremove_wake_function+0x10/0x10
> [ 1984.296290] ? __kthread_parkme+0xc1/0x1f0
> [ 1984.296304] ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1984.296321] kthread+0x29e/0x340
> [ 1984.296334] ? __pfx_kthread+0x10/0x10
> [ 1984.296501] ret_from_fork+0x2c/0x50
> [ 1984.296518] </TASK>
>
> [ 1984.296539] Allocated by task 12194:
> [ 1984.296552] kasan_save_stack+0x2f/0x50
> [ 1984.296566] kasan_set_track+0x21/0x30
> [ 1984.296578] __kasan_kmalloc+0x8b/0x90
> [ 1984.296590] amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
> [ 1984.297051] drm_file_alloc+0x46e/0x880
> [ 1984.297064] drm_open_helper+0x161/0x460
> [ 1984.297076] drm_open+0x1e7/0x5c0
> [ 1984.297089] drm_stub_open+0x24d/0x400
> [ 1984.297107] chrdev_open+0x215/0x620
> [ 1984.297125] do_dentry_open+0x5f1/0x1000
> [ 1984.297146] path_openat+0x1b3d/0x28a0
> [ 1984.297164] do_filp_open+0x1bd/0x400
> [ 1984.297180] do_sys_openat2+0x140/0x420
> [ 1984.297197] __x64_sys_openat+0x11f/0x1d0
> [ 1984.297213] do_syscall_64+0x5b/0x80
> [ 1984.297231] entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.297266] Freed by task 12195:
> [ 1984.297284] kasan_save_stack+0x2f/0x50
> [ 1984.297303] kasan_set_track+0x21/0x30
> [ 1984.297323] kasan_save_free_info+0x2a/0x50
> [ 1984.297343] __kasan_slab_free+0x107/0x1a0
> [ 1984.297361] slab_free_freelist_hook+0x11e/0x1d0
> [ 1984.297373] __kmem_cache_free+0xbc/0x2e0
> [ 1984.297385] amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
> [ 1984.297821] drm_file_free.part.0+0x638/0xb70
> [ 1984.297834] drm_release+0x1ea/0x470
> [ 1984.297845] __fput+0x213/0x9e0
> [ 1984.297857] task_work_run+0x11b/0x200
> [ 1984.297869] exit_to_user_mode_prepare+0x23a/0x260
> [ 1984.297883] syscall_exit_to_user_mode+0x16/0x50
> [ 1984.297896] do_syscall_64+0x67/0x80
> [ 1984.297907] entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.298033] Last potentially related work creation:
> [ 1984.298044] kasan_save_stack+0x2f/0x50
> [ 1984.298057] __kasan_record_aux_stack+0x97/0xb0
> [ 1984.298075] __call_rcu_common.constprop.0+0xf8/0x1af0
> [ 1984.298095] amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
> [ 1984.298557] amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
> [ 1984.299055] amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
> [ 1984.299624] drm_ioctl_kernel+0x1f8/0x3d0
> [ 1984.299637] drm_ioctl+0x4c1/0xaa0
> [ 1984.299649] amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
> [ 1984.300083] __x64_sys_ioctl+0x12d/0x1a0
> [ 1984.300097] do_syscall_64+0x5b/0x80
> [ 1984.300109] entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.300135] Second to last potentially related work creation:
> [ 1984.300149] kasan_save_stack+0x2f/0x50
> [ 1984.300167] __kasan_record_aux_stack+0x97/0xb0
> [ 1984.300185] __call_rcu_common.constprop.0+0xf8/0x1af0
> [ 1984.300203] amdgpu_bo_list_put+0x1a4/0x1f0 [amdgpu]
> [ 1984.300692] amdgpu_cs_parser_fini+0x293/0x5a0 [amdgpu]
> [ 1984.301133] amdgpu_cs_ioctl+0x4f2a/0x5630 [amdgpu]
> [ 1984.301577] drm_ioctl_kernel+0x1f8/0x3d0
> [ 1984.301598] drm_ioctl+0x4c1/0xaa0
> [ 1984.301610] amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
> [ 1984.302043] __x64_sys_ioctl+0x12d/0x1a0
> [ 1984.302056] do_syscall_64+0x5b/0x80
> [ 1984.302068] entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1984.302090] The buggy address belongs to the object at ffff88814cadc000
> which belongs to the cache kmalloc-4k of size 4096
> [ 1984.302103] The buggy address is located 1216 bytes inside of
> freed 4096-byte region [ffff88814cadc000, ffff88814cadd000)
>
> [ 1984.302129] The buggy address belongs to the physical page:
> [ 1984.302141] page:00000000103c1864 refcount:1 mapcount:0
> mapping:0000000000000000 index:0x0 pfn:0x14cad8
> [ 1984.302155] head:00000000103c1864 order:3 entire_mapcount:0
> nr_pages_mapped:0 pincount:0
> [ 1984.302168] flags:
> 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
> [ 1984.302184] raw: 0017ffffc0010200 ffff88810004d040 ffffea00311f0200
> dead000000000002
> [ 1984.302196] raw: 0000000000000000 0000000000040004 00000001ffffffff
> 0000000000000000
> [ 1984.302208] page dumped because: kasan: bad access detected
>
> [ 1984.302230] Memory state around the buggy address:
> [ 1984.302241] ffff88814cadc380: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302252] ffff88814cadc400: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302263] >ffff88814cadc480: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302274] ^
> [ 1984.302285] ffff88814cadc500: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302296] ffff88814cadc580: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1984.302307] ==================================================================
> [ 1984.302339] Disabling lock debugging due to kernel taint
>
On Fri, Mar 24, 2023 at 7:37 PM Christian König
<[email protected]> wrote:
>
> Yeah, that one
>
> Thanks for the info, looks like this isn't fixed.
>
> Christian.
>
Hi,
glad to see that "BUG: KASAN: slab-use-after-free in
drm_sched_get_cleanup_job+0x47b/0x5c0" was fixed in 6.3-rc5.
For history it would be good to know the commit which fixes this issue.
I waited for this moment because I know other one issue which was also
found by KASAN santiniser.
BUG: KASAN: null-ptr-deref in drm_sched_job_cleanup+0x96/0x290 [gpu_sched]
Read of size 4 at addr 0000000000000078 by task GameThread/23915
CPU: 10 PID: 23915 Comm: GameThread Tainted: G W L
------- --- 6.3.0-0.rc5.42.fc39.x86_64+debug #1
Hardware name: System manufacturer System Product Name/ROG STRIX
X570-I GAMING, BIOS 4601 02/02/2023
Call Trace:
<TASK>
dump_stack_lvl+0x72/0xc0
kasan_report+0xa4/0xe0
? drm_sched_job_cleanup+0x96/0x290 [gpu_sched]
kasan_check_range+0x104/0x1b0
drm_sched_job_cleanup+0x96/0x290 [gpu_sched]
? __pfx_drm_sched_job_cleanup+0x10/0x10 [gpu_sched]
? slab_free_freelist_hook+0x11e/0x1d0
? amdgpu_cs_parser_fini+0x363/0x5a0 [amdgpu]
amdgpu_job_free+0x40/0x1b0 [amdgpu]
amdgpu_cs_parser_fini+0x3c9/0x5a0 [amdgpu]
? __pfx_amdgpu_cs_parser_fini+0x10/0x10 [amdgpu]
amdgpu_cs_ioctl+0x3d9/0x5630 [amdgpu]
? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
? mark_lock+0x101/0x16e0
? __lock_acquire+0xe54/0x59f0
? __pfx_lock_release+0x10/0x10
? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
drm_ioctl_kernel+0x1f8/0x3d0
? __pfx_drm_ioctl_kernel+0x10/0x10
drm_ioctl+0x4c1/0xaa0
? __pfx_amdgpu_cs_ioctl+0x10/0x10 [amdgpu]
? __pfx_drm_ioctl+0x10/0x10
? _raw_spin_unlock_irqrestore+0x62/0x80
? lockdep_hardirqs_on+0x7d/0x100
? _raw_spin_unlock_irqrestore+0x4b/0x80
amdgpu_drm_ioctl+0xce/0x1b0 [amdgpu]
__x64_sys_ioctl+0x12d/0x1a0
do_syscall_64+0x5c/0x90
? do_syscall_64+0x68/0x90
? lockdep_hardirqs_on+0x7d/0x100
? do_syscall_64+0x68/0x90
? do_syscall_64+0x68/0x90
? lockdep_hardirqs_on+0x7d/0x100
? do_syscall_64+0x68/0x90
? do_syscall_64+0x68/0x90
? lockdep_hardirqs_on+0x7d/0x100
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7fe97a50881d
Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00
00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2
3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
RSP: 002b:000000007c35d3f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 000000007c35d6e8 RCX: 00007fe97a50881d
RDX: 000000007c35d4d0 RSI: 00000000c0186444 RDI: 00000000000000ae
RBP: 000000007c35d440 R08: 00007fe8fc0f0970 R09: 000000007c35d490
R10: 000000007fb79000 R11: 0000000000000246 R12: 000000007c35d4d0
R13: 00000000c0186444 R14: 00000000000000ae R15: 00007fe8fc0f0900
</TASK>
I know at least 3 games which 100% triggering this bug:
- Cyberpunk 2077
- Forza Horizon 4
- Forza Horizon 5
We would continue to discuss it here or better create a new thread
(for someone who is also faced with this issue could easily find a
solution on the internet)?
A full kernel log as usual attached here.
--
Best Regards,
Mike Gavrilov.