Hello,
syzbot found the following issue on:
HEAD commit: 534293368afa Merge tag 'kbuild-fixes-v6.3' of git://git.ke..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11a0c26ec80000
kernel config: https://syzkaller.appspot.com/x/.config?x=85fa20c996a2e46d
dashboard link: https://syzkaller.appspot.com/bug?extid=774c29891415ab0fd29d
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
Unfortunately, I don't have any reproducer for this issue yet.
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/ec0e584af797/disk-53429336.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/423a67f52f94/vmlinux-53429336.xz
kernel image: https://storage.googleapis.com/syzbot-assets/8409b9716faa/bzImage-53429336.xz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]
==================================================================
BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
memcg_slab_free_hook mm/slab.h:587 [inline]
__cache_free mm/slab.c:3373 [inline]
__do_kmem_cache_free mm/slab.c:3577 [inline]
kmem_cache_free+0x105/0x280 mm/slab.c:3602
__d_free fs/dcache.c:298 [inline]
dentry_free fs/dcache.c:375 [inline]
__dentry_kill+0x422/0x4a0 fs/dcache.c:621
dentry_kill+0x8d/0x1e0
dput+0x118/0x1f0 fs/dcache.c:913
__fput+0x3bf/0x570 fs/file_table.c:329
____fput+0x15/0x20 fs/file_table.c:349
task_work_run+0x123/0x160 kernel/task_work.c:179
resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
__syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x63/0xcd
read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
obj_stock_flush_required mm/memcontrol.c:3319 [inline]
drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
try_charge mm/memcontrol.c:2837 [inline]
mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
__sys_setsockopt+0x1c3/0x230 net/socket.c:2271
__do_sys_setsockopt net/socket.c:2282 [inline]
__se_sys_setsockopt net/socket.c:2279 [inline]
__x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
value changed: 0xffff8881382d52c0 -> 0xffff888138893740
Reported by Kernel Concurrency Sanitizer on:
CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
==================================================================
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
On Fri, 21 Apr 2023 at 16:33, syzbot
<[email protected]> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 534293368afa Merge tag 'kbuild-fixes-v6.3' of git://git.ke..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11a0c26ec80000
> kernel config: https://syzkaller.appspot.com/x/.config?x=85fa20c996a2e46d
> dashboard link: https://syzkaller.appspot.com/bug?extid=774c29891415ab0fd29d
> compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/ec0e584af797/disk-53429336.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/423a67f52f94/vmlinux-53429336.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/8409b9716faa/bzImage-53429336.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: [email protected]
Here this:
stock->cached_objcg = NULL;
runs concurrently with:
if (stock->cached_objcg) {
memcg = obj_cgroup_memcg(stock->cached_objcg);
If I am reading this correctly, this can cause a NULL deref, if
cached_objcg is reset between the check and the use.
> ==================================================================
> BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
>
> write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> memcg_slab_free_hook mm/slab.h:587 [inline]
> __cache_free mm/slab.c:3373 [inline]
> __do_kmem_cache_free mm/slab.c:3577 [inline]
> kmem_cache_free+0x105/0x280 mm/slab.c:3602
> __d_free fs/dcache.c:298 [inline]
> dentry_free fs/dcache.c:375 [inline]
> __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> dentry_kill+0x8d/0x1e0
> dput+0x118/0x1f0 fs/dcache.c:913
> __fput+0x3bf/0x570 fs/file_table.c:329
> ____fput+0x15/0x20 fs/file_table.c:349
> task_work_run+0x123/0x160 kernel/task_work.c:179
> resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> try_charge mm/memcontrol.c:2837 [inline]
> mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> __do_sys_setsockopt net/socket.c:2282 [inline]
> __se_sys_setsockopt net/socket.c:2279 [inline]
> __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> value changed: 0xffff8881382d52c0 -> 0xffff888138893740
>
> Reported by Kernel Concurrency Sanitizer on:
> CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> ==================================================================
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at [email protected].
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/00000000000058b63f05f9d98811%40google.com.
KCSAN found an issue in obj_stock_flush_required():
stock->cached_objcg can be reset between the check and dereference:
==================================================================
BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
memcg_slab_free_hook mm/slab.h:587 [inline]
__cache_free mm/slab.c:3373 [inline]
__do_kmem_cache_free mm/slab.c:3577 [inline]
kmem_cache_free+0x105/0x280 mm/slab.c:3602
__d_free fs/dcache.c:298 [inline]
dentry_free fs/dcache.c:375 [inline]
__dentry_kill+0x422/0x4a0 fs/dcache.c:621
dentry_kill+0x8d/0x1e0
dput+0x118/0x1f0 fs/dcache.c:913
__fput+0x3bf/0x570 fs/file_table.c:329
____fput+0x15/0x20 fs/file_table.c:349
task_work_run+0x123/0x160 kernel/task_work.c:179
resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
__syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x63/0xcd
read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
obj_stock_flush_required mm/memcontrol.c:3319 [inline]
drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
try_charge mm/memcontrol.c:2837 [inline]
mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
__sys_setsockopt+0x1c3/0x230 net/socket.c:2271
__do_sys_setsockopt net/socket.c:2282 [inline]
__se_sys_setsockopt net/socket.c:2279 [inline]
__x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
value changed: 0xffff8881382d52c0 -> 0xffff888138893740
Reported by Kernel Concurrency Sanitizer on:
CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
Fix it by reading the cached_objcg with READ_ONCE().
Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
Reported-by: [email protected]
Reported-by: Dmitry Vyukov <[email protected]>
Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
Signed-off-by: Roman Gushchin <[email protected]>
---
mm/memcontrol.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 5abffe6f8389..9426a1ddc190 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3314,10 +3314,11 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock)
static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
struct mem_cgroup *root_memcg)
{
+ struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
struct mem_cgroup *memcg;
- if (stock->cached_objcg) {
- memcg = obj_cgroup_memcg(stock->cached_objcg);
+ if (objcg) {
+ memcg = obj_cgroup_memcg(objcg);
if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
return true;
}
--
2.40.0
On Fri, Apr 21, 2023 at 10:41 AM Roman Gushchin
<[email protected]> wrote:
>
> KCSAN found an issue in obj_stock_flush_required():
> stock->cached_objcg can be reset between the check and dereference:
>
> ==================================================================
> BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
>
> write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> memcg_slab_free_hook mm/slab.h:587 [inline]
> __cache_free mm/slab.c:3373 [inline]
> __do_kmem_cache_free mm/slab.c:3577 [inline]
> kmem_cache_free+0x105/0x280 mm/slab.c:3602
> __d_free fs/dcache.c:298 [inline]
> dentry_free fs/dcache.c:375 [inline]
> __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> dentry_kill+0x8d/0x1e0
> dput+0x118/0x1f0 fs/dcache.c:913
> __fput+0x3bf/0x570 fs/file_table.c:329
> ____fput+0x15/0x20 fs/file_table.c:349
> task_work_run+0x123/0x160 kernel/task_work.c:179
> resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> try_charge mm/memcontrol.c:2837 [inline]
> mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> __do_sys_setsockopt net/socket.c:2282 [inline]
> __se_sys_setsockopt net/socket.c:2279 [inline]
> __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> value changed: 0xffff8881382d52c0 -> 0xffff888138893740
>
> Reported by Kernel Concurrency Sanitizer on:
> CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
>
> Fix it by reading the cached_objcg with READ_ONCE().
IIUC reading an outdated objcg is fine here, and the real problem is a
potential NULL dereference, which isn't shown by this report. Is this
correct?
>
> Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> Reported-by: [email protected]
> Reported-by: Dmitry Vyukov <[email protected]>
> Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> Signed-off-by: Roman Gushchin <[email protected]>
> ---
> mm/memcontrol.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 5abffe6f8389..9426a1ddc190 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3314,10 +3314,11 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock)
> static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
> struct mem_cgroup *root_memcg)
> {
> + struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
> struct mem_cgroup *memcg;
>
> - if (stock->cached_objcg) {
> - memcg = obj_cgroup_memcg(stock->cached_objcg);
> + if (objcg) {
> + memcg = obj_cgroup_memcg(objcg);
> if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
> return true;
> }
> --
> 2.40.0
>
>
On Fri, Apr 21, 2023 at 03:16:02PM -0700, Yosry Ahmed wrote:
> On Fri, Apr 21, 2023 at 10:41 AM Roman Gushchin
> <[email protected]> wrote:
> >
> > KCSAN found an issue in obj_stock_flush_required():
> > stock->cached_objcg can be reset between the check and dereference:
> >
> > ==================================================================
> > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> >
> > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > memcg_slab_free_hook mm/slab.h:587 [inline]
> > __cache_free mm/slab.c:3373 [inline]
> > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > __d_free fs/dcache.c:298 [inline]
> > dentry_free fs/dcache.c:375 [inline]
> > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > dentry_kill+0x8d/0x1e0
> > dput+0x118/0x1f0 fs/dcache.c:913
> > __fput+0x3bf/0x570 fs/file_table.c:329
> > ____fput+0x15/0x20 fs/file_table.c:349
> > task_work_run+0x123/0x160 kernel/task_work.c:179
> > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> >
> > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > try_charge mm/memcontrol.c:2837 [inline]
> > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > __do_sys_setsockopt net/socket.c:2282 [inline]
> > __se_sys_setsockopt net/socket.c:2279 [inline]
> > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> >
> > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> >
> > Reported by Kernel Concurrency Sanitizer on:
> > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> >
> > Fix it by reading the cached_objcg with READ_ONCE().
>
> IIUC reading an outdated objcg is fine here, and the real problem is a
> potential NULL dereference, which isn't shown by this report. Is this
> correct?
My understanding of the problem:
1) we are reading stock->cached_objcg and checking for being NULL, it's not NULL.
2) we're reading it again and dereferencing it without a check.
If stock->cached_objcg is zeroed between 1) and 2), we're in trouble.
The fix makes sure we're dereferencing exactly what we check.
>
> >
> > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > Reported-by: [email protected]
> > Reported-by: Dmitry Vyukov <[email protected]>
> > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > Signed-off-by: Roman Gushchin <[email protected]>
> > ---
> > mm/memcontrol.c | 5 +++--
> > 1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 5abffe6f8389..9426a1ddc190 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -3314,10 +3314,11 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock)
> > static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
> > struct mem_cgroup *root_memcg)
> > {
> > + struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
> > struct mem_cgroup *memcg;
> >
> > - if (stock->cached_objcg) {
> > - memcg = obj_cgroup_memcg(stock->cached_objcg);
> > + if (objcg) {
> > + memcg = obj_cgroup_memcg(objcg);
> > if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
> > return true;
> > }
> > --
> > 2.40.0
> >
> >
On Fri, Apr 21, 2023 at 3:55 PM Roman Gushchin <[email protected]> wrote:
>
> On Fri, Apr 21, 2023 at 03:16:02PM -0700, Yosry Ahmed wrote:
> > On Fri, Apr 21, 2023 at 10:41 AM Roman Gushchin
> > <[email protected]> wrote:
> > >
> > > KCSAN found an issue in obj_stock_flush_required():
> > > stock->cached_objcg can be reset between the check and dereference:
> > >
> > > ==================================================================
> > > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> > >
> > > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > > memcg_slab_free_hook mm/slab.h:587 [inline]
> > > __cache_free mm/slab.c:3373 [inline]
> > > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > > __d_free fs/dcache.c:298 [inline]
> > > dentry_free fs/dcache.c:375 [inline]
> > > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > > dentry_kill+0x8d/0x1e0
> > > dput+0x118/0x1f0 fs/dcache.c:913
> > > __fput+0x3bf/0x570 fs/file_table.c:329
> > > ____fput+0x15/0x20 fs/file_table.c:349
> > > task_work_run+0x123/0x160 kernel/task_work.c:179
> > > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> > > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> > > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > >
> > > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > > try_charge mm/memcontrol.c:2837 [inline]
> > > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > > __do_sys_setsockopt net/socket.c:2282 [inline]
> > > __se_sys_setsockopt net/socket.c:2279 [inline]
> > > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > >
> > > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> > >
> > > Reported by Kernel Concurrency Sanitizer on:
> > > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> > >
> > > Fix it by reading the cached_objcg with READ_ONCE().
> >
> > IIUC reading an outdated objcg is fine here, and the real problem is a
> > potential NULL dereference, which isn't shown by this report. Is this
> > correct?
>
> My understanding of the problem:
> 1) we are reading stock->cached_objcg and checking for being NULL, it's not NULL.
> 2) we're reading it again and dereferencing it without a check.
>
> If stock->cached_objcg is zeroed between 1) and 2), we're in trouble.
>
> The fix makes sure we're dereferencing exactly what we check.
I see. This matches my understanding. The bug report does not show a
NULL dereference but it shows a data race that can very well lead to
one.
FWIW,
Reviewed-by: Yosry Ahmed <[email protected]>
>
> >
> > >
> > > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > > Reported-by: [email protected]
> > > Reported-by: Dmitry Vyukov <[email protected]>
> > > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > > Signed-off-by: Roman Gushchin <[email protected]>
> > > ---
> > > mm/memcontrol.c | 5 +++--
> > > 1 file changed, 3 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > index 5abffe6f8389..9426a1ddc190 100644
> > > --- a/mm/memcontrol.c
> > > +++ b/mm/memcontrol.c
> > > @@ -3314,10 +3314,11 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock)
> > > static bool obj_stock_flush_required(struct memcg_stock_pcp *stock,
> > > struct mem_cgroup *root_memcg)
> > > {
> > > + struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
> > > struct mem_cgroup *memcg;
> > >
> > > - if (stock->cached_objcg) {
> > > - memcg = obj_cgroup_memcg(stock->cached_objcg);
> > > + if (objcg) {
> > > + memcg = obj_cgroup_memcg(objcg);
> > > if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
> > > return true;
> > > }
> > > --
> > > 2.40.0
> > >
> > >
> On Apr 22, 2023, at 01:40, Roman Gushchin <[email protected]> wrote:
>
> KCSAN found an issue in obj_stock_flush_required():
> stock->cached_objcg can be reset between the check and dereference:
>
> ==================================================================
> BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
>
> write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> memcg_slab_free_hook mm/slab.h:587 [inline]
> __cache_free mm/slab.c:3373 [inline]
> __do_kmem_cache_free mm/slab.c:3577 [inline]
> kmem_cache_free+0x105/0x280 mm/slab.c:3602
> __d_free fs/dcache.c:298 [inline]
> dentry_free fs/dcache.c:375 [inline]
> __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> dentry_kill+0x8d/0x1e0
> dput+0x118/0x1f0 fs/dcache.c:913
> __fput+0x3bf/0x570 fs/file_table.c:329
> ____fput+0x15/0x20 fs/file_table.c:349
> task_work_run+0x123/0x160 kernel/task_work.c:179
> resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> try_charge mm/memcontrol.c:2837 [inline]
> mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> __do_sys_setsockopt net/socket.c:2282 [inline]
> __se_sys_setsockopt net/socket.c:2279 [inline]
> __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> value changed: 0xffff8881382d52c0 -> 0xffff888138893740
>
> Reported by Kernel Concurrency Sanitizer on:
> CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
>
> Fix it by reading the cached_objcg with READ_ONCE().
>
> Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> Reported-by: [email protected]
> Reported-by: Dmitry Vyukov <[email protected]>
> Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> Signed-off-by: Roman Gushchin <[email protected]>
Acked-by: Muchun Song <[email protected]>
Thanks.
On Sun, 23 Apr 2023 at 04:26, Muchun Song <[email protected]> wrote:
> > On Apr 22, 2023, at 01:40, Roman Gushchin <[email protected]> wrote:
> >
> > KCSAN found an issue in obj_stock_flush_required():
> > stock->cached_objcg can be reset between the check and dereference:
> >
> > ==================================================================
> > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> >
> > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > memcg_slab_free_hook mm/slab.h:587 [inline]
> > __cache_free mm/slab.c:3373 [inline]
> > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > __d_free fs/dcache.c:298 [inline]
> > dentry_free fs/dcache.c:375 [inline]
> > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > dentry_kill+0x8d/0x1e0
> > dput+0x118/0x1f0 fs/dcache.c:913
> > __fput+0x3bf/0x570 fs/file_table.c:329
> > ____fput+0x15/0x20 fs/file_table.c:349
> > task_work_run+0x123/0x160 kernel/task_work.c:179
> > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> >
> > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > try_charge mm/memcontrol.c:2837 [inline]
> > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > __do_sys_setsockopt net/socket.c:2282 [inline]
> > __se_sys_setsockopt net/socket.c:2279 [inline]
> > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> >
> > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> >
> > Reported by Kernel Concurrency Sanitizer on:
> > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> >
> > Fix it by reading the cached_objcg with READ_ONCE().
> >
> > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > Reported-by: [email protected]
> > Reported-by: Dmitry Vyukov <[email protected]>
> > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > Signed-off-by: Roman Gushchin <[email protected]>
>
> Acked-by: Muchun Song <[email protected]>
>
> Thanks.
This improves things, but strictly speaking the write side also needs
WRITE_ONCE. Ordering is always a game of two. It's not possible to
order things on one side, if the other side messes up the ordering.
On Sun, Apr 23, 2023 at 11:51 PM Dmitry Vyukov <[email protected]> wrote:
>
> On Sun, 23 Apr 2023 at 04:26, Muchun Song <[email protected]> wrote:
> > > On Apr 22, 2023, at 01:40, Roman Gushchin <[email protected]> wrote:
> > >
> > > KCSAN found an issue in obj_stock_flush_required():
> > > stock->cached_objcg can be reset between the check and dereference:
> > >
> > > ==================================================================
> > > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> > >
> > > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > > memcg_slab_free_hook mm/slab.h:587 [inline]
> > > __cache_free mm/slab.c:3373 [inline]
> > > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > > __d_free fs/dcache.c:298 [inline]
> > > dentry_free fs/dcache.c:375 [inline]
> > > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > > dentry_kill+0x8d/0x1e0
> > > dput+0x118/0x1f0 fs/dcache.c:913
> > > __fput+0x3bf/0x570 fs/file_table.c:329
> > > ____fput+0x15/0x20 fs/file_table.c:349
> > > task_work_run+0x123/0x160 kernel/task_work.c:179
> > > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> > > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> > > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > >
> > > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > > try_charge mm/memcontrol.c:2837 [inline]
> > > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > > __do_sys_setsockopt net/socket.c:2282 [inline]
> > > __se_sys_setsockopt net/socket.c:2279 [inline]
> > > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > >
> > > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> > >
> > > Reported by Kernel Concurrency Sanitizer on:
> > > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> > >
> > > Fix it by reading the cached_objcg with READ_ONCE().
> > >
> > > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > > Reported-by: [email protected]
> > > Reported-by: Dmitry Vyukov <[email protected]>
> > > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > > Signed-off-by: Roman Gushchin <[email protected]>
> >
> > Acked-by: Muchun Song <[email protected]>
> >
> > Thanks.
>
> This improves things, but strictly speaking the write side also needs
> WRITE_ONCE. Ordering is always a game of two. It's not possible to
> order things on one side, if the other side messes up the ordering.
>
It looks like most other accesses use memcg_stock.stock_lock for
synchronization. Based on the output of obj_stock_flush_required()
we call drain_local_stock(), which acquires that lock as well. Should
we refactor the code to extend the lock section to cover both
obj_stock_flush_required() and drain_local_stock()?
IIUC this may unify the synchronization handling and
READ_ONCE/WRITE_ONCE may no longer be needed. This should also avoid
any inaccuracies (e.g. unnecessary flushes) that may happen if the
cached objcg changes between obj_stock_flush_required() and
drain_local_stock().
Did I miss anything here?
On Mon, Apr 24, 2023 at 2:13 AM Yosry Ahmed <[email protected]> wrote:
>
> On Sun, Apr 23, 2023 at 11:51 PM Dmitry Vyukov <[email protected]> wrote:
> >
> > On Sun, 23 Apr 2023 at 04:26, Muchun Song <[email protected]> wrote:
> > > > On Apr 22, 2023, at 01:40, Roman Gushchin <[email protected]> wrote:
> > > >
> > > > KCSAN found an issue in obj_stock_flush_required():
> > > > stock->cached_objcg can be reset between the check and dereference:
> > > >
> > > > ==================================================================
> > > > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> > > >
> > > > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > > > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > > > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > > > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > > > memcg_slab_free_hook mm/slab.h:587 [inline]
> > > > __cache_free mm/slab.c:3373 [inline]
> > > > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > > > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > > > __d_free fs/dcache.c:298 [inline]
> > > > dentry_free fs/dcache.c:375 [inline]
> > > > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > > > dentry_kill+0x8d/0x1e0
> > > > dput+0x118/0x1f0 fs/dcache.c:913
> > > > __fput+0x3bf/0x570 fs/file_table.c:329
> > > > ____fput+0x15/0x20 fs/file_table.c:349
> > > > task_work_run+0x123/0x160 kernel/task_work.c:179
> > > > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> > > > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > > > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> > > > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > > > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > >
> > > > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > > > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > > > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > > > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > > > try_charge mm/memcontrol.c:2837 [inline]
> > > > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > > > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > > > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > > > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > > > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > > > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > > > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > > > __do_sys_setsockopt net/socket.c:2282 [inline]
> > > > __se_sys_setsockopt net/socket.c:2279 [inline]
> > > > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > > > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > >
> > > > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> > > >
> > > > Reported by Kernel Concurrency Sanitizer on:
> > > > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> > > >
> > > > Fix it by reading the cached_objcg with READ_ONCE().
> > > >
> > > > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > > > Reported-by: [email protected]
> > > > Reported-by: Dmitry Vyukov <[email protected]>
> > > > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > > > Signed-off-by: Roman Gushchin <[email protected]>
> > >
> > > Acked-by: Muchun Song <[email protected]>
> > >
> > > Thanks.
> >
> > This improves things, but strictly speaking the write side also needs
> > WRITE_ONCE. Ordering is always a game of two. It's not possible to
> > order things on one side, if the other side messes up the ordering.
> >
>
> It looks like most other accesses use memcg_stock.stock_lock for
> synchronization. Based on the output of obj_stock_flush_required()
> we call drain_local_stock(), which acquires that lock as well. Should
> we refactor the code to extend the lock section to cover both
> obj_stock_flush_required() and drain_local_stock()?
>
> IIUC this may unify the synchronization handling and
> READ_ONCE/WRITE_ONCE may no longer be needed. This should also avoid
> any inaccuracies (e.g. unnecessary flushes) that may happen if the
> cached objcg changes between obj_stock_flush_required() and
> drain_local_stock().
>
> Did I miss anything here?
Yes, drain_local_stock only works on local cpu and
obj_stock_flush_required can touch the stock of all the cpus.
The patch is good but I agree with Dmitry that we should add the
WRITE_ONCE as well.
On Mon, Apr 24, 2023 at 10:10 AM Shakeel Butt <[email protected]> wrote:
>
> On Mon, Apr 24, 2023 at 2:13 AM Yosry Ahmed <[email protected]> wrote:
> >
> > On Sun, Apr 23, 2023 at 11:51 PM Dmitry Vyukov <[email protected]> wrote:
> > >
> > > On Sun, 23 Apr 2023 at 04:26, Muchun Song <[email protected]> wrote:
> > > > > On Apr 22, 2023, at 01:40, Roman Gushchin <[email protected]> wrote:
> > > > >
> > > > > KCSAN found an issue in obj_stock_flush_required():
> > > > > stock->cached_objcg can be reset between the check and dereference:
> > > > >
> > > > > ==================================================================
> > > > > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> > > > >
> > > > > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > > > > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > > > > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > > > > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > > > > memcg_slab_free_hook mm/slab.h:587 [inline]
> > > > > __cache_free mm/slab.c:3373 [inline]
> > > > > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > > > > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > > > > __d_free fs/dcache.c:298 [inline]
> > > > > dentry_free fs/dcache.c:375 [inline]
> > > > > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > > > > dentry_kill+0x8d/0x1e0
> > > > > dput+0x118/0x1f0 fs/dcache.c:913
> > > > > __fput+0x3bf/0x570 fs/file_table.c:329
> > > > > ____fput+0x15/0x20 fs/file_table.c:349
> > > > > task_work_run+0x123/0x160 kernel/task_work.c:179
> > > > > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
> > > > > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > > > > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
> > > > > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > > > > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > > > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > > >
> > > > > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > > > > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > > > > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > > > > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > > > > try_charge mm/memcontrol.c:2837 [inline]
> > > > > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > > > > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > > > > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > > > > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > > > > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > > > > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > > > > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > > > > __do_sys_setsockopt net/socket.c:2282 [inline]
> > > > > __se_sys_setsockopt net/socket.c:2279 [inline]
> > > > > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > > > > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > > > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > > > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > > >
> > > > > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> > > > >
> > > > > Reported by Kernel Concurrency Sanitizer on:
> > > > > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzkaller-00387-g534293368afa #0
> > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
> > > > >
> > > > > Fix it by reading the cached_objcg with READ_ONCE().
> > > > >
> > > > > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > > > > Reported-by: [email protected]
> > > > > Reported-by: Dmitry Vyukov <[email protected]>
> > > > > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > > > > Signed-off-by: Roman Gushchin <[email protected]>
> > > >
> > > > Acked-by: Muchun Song <[email protected]>
> > > >
> > > > Thanks.
> > >
> > > This improves things, but strictly speaking the write side also needs
> > > WRITE_ONCE. Ordering is always a game of two. It's not possible to
> > > order things on one side, if the other side messes up the ordering.
> > >
> >
> > It looks like most other accesses use memcg_stock.stock_lock for
> > synchronization. Based on the output of obj_stock_flush_required()
> > we call drain_local_stock(), which acquires that lock as well. Should
> > we refactor the code to extend the lock section to cover both
> > obj_stock_flush_required() and drain_local_stock()?
> >
> > IIUC this may unify the synchronization handling and
> > READ_ONCE/WRITE_ONCE may no longer be needed. This should also avoid
> > any inaccuracies (e.g. unnecessary flushes) that may happen if the
> > cached objcg changes between obj_stock_flush_required() and
> > drain_local_stock().
> >
> > Did I miss anything here?
>
> Yes, drain_local_stock only works on local cpu and
> obj_stock_flush_required can touch the stock of all the cpus.
Oh right, for other cpus this is infeasible as we schedule the work to
be done later.
>
> The patch is good but I agree with Dmitry that we should add the
> WRITE_ONCE as well.
Agreed. I guess WRITE_ONCE will be needed in multiple places that
update the cached objcg.
Thanks.