Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <00000000000058b63f05f9d98811@google.com> <20230421174054.3434533-1-roman.gushchin@linux.dev>
 <DB1562BC-CAD0-45B4-8C6D-92E583C128DF@linux.dev> <CACT4Y+Yy9a8R_oiumdZiE-hwAqrDV63VAWgbrE5nFEk9w-JuPQ@mail.gmail.com>
 <CAJD7tkY4VH3CDoZHjOTWvCCPXikRScNjwPsV9hf001X_y6i+FA@mail.gmail.com> <CALvZod4bX+vPLAXb5J4TW=BWDfeweuB_h+B1GgHED39N=b=SbA@mail.gmail.com>
In-Reply-To: <CALvZod4bX+vPLAXb5J4TW=BWDfeweuB_h+B1GgHED39N=b=SbA@mail.gmail.com>
From:   Yosry Ahmed <yosryahmed@google.com>
Date:   Mon, 24 Apr 2023 14:10:45 -0700
Message-ID: <CAJD7tkZ+Nz5Hh=CfvViaVyprsp9xgoJn3f8GimkJxqsfqeZeVA@mail.gmail.com>
Subject: Re: [PATCH] mm: kmem: fix a NULL pointer dereference in obj_stock_flush_required()
To:     Shakeel Butt <shakeelb@google.com>
Cc:     Dmitry Vyukov <dvyukov@google.com>,
        Roman Gushchin <roman.gushchin@linux.dev>,
        Muchun Song <muchun.song@linux.dev>,
        Linux Memory Management List <linux-mm@kvack.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Michal Hocko <mhocko@kernel.org>, linux-kernel@vger.kernel.org,
        syzbot+774c29891415ab0fd29d@syzkaller.appspotmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, Apr 24, 2023 at 10:10=E2=80=AFAM Shakeel Butt <shakeelb@google.com>=
 wrote:
>
> On Mon, Apr 24, 2023 at 2:13=E2=80=AFAM Yosry Ahmed <yosryahmed@google.co=
m> wrote:
> >
> > On Sun, Apr 23, 2023 at 11:51=E2=80=AFPM Dmitry Vyukov <dvyukov@google.=
com> wrote:
> > >
> > > On Sun, 23 Apr 2023 at 04:26, Muchun Song <muchun.song@linux.dev> wro=
te:
> > > > > On Apr 22, 2023, at 01:40, Roman Gushchin <roman.gushchin@linux.d=
ev> wrote:
> > > > >
> > > > > KCSAN found an issue in obj_stock_flush_required():
> > > > > stock->cached_objcg can be reset between the check and dereferenc=
e:
> > > > >
> > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > > > BUG: KCSAN: data-race in drain_all_stock / drain_obj_stock
> > > > >
> > > > > write to 0xffff888237c2a2f8 of 8 bytes by task 19625 on cpu 0:
> > > > > drain_obj_stock+0x408/0x4e0 mm/memcontrol.c:3306
> > > > > refill_obj_stock+0x9c/0x1e0 mm/memcontrol.c:3340
> > > > > obj_cgroup_uncharge+0xe/0x10 mm/memcontrol.c:3408
> > > > > memcg_slab_free_hook mm/slab.h:587 [inline]
> > > > > __cache_free mm/slab.c:3373 [inline]
> > > > > __do_kmem_cache_free mm/slab.c:3577 [inline]
> > > > > kmem_cache_free+0x105/0x280 mm/slab.c:3602
> > > > > __d_free fs/dcache.c:298 [inline]
> > > > > dentry_free fs/dcache.c:375 [inline]
> > > > > __dentry_kill+0x422/0x4a0 fs/dcache.c:621
> > > > > dentry_kill+0x8d/0x1e0
> > > > > dput+0x118/0x1f0 fs/dcache.c:913
> > > > > __fput+0x3bf/0x570 fs/file_table.c:329
> > > > > ____fput+0x15/0x20 fs/file_table.c:349
> > > > > task_work_run+0x123/0x160 kernel/task_work.c:179
> > > > > resume_user_mode_work include/linux/resume_user_mode.h:49 [inline=
]
> > > > > exit_to_user_mode_loop+0xcf/0xe0 kernel/entry/common.c:171
> > > > > exit_to_user_mode_prepare+0x6a/0xa0 kernel/entry/common.c:203
> > > > > __syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inlin=
e]
> > > > > syscall_exit_to_user_mode+0x26/0x140 kernel/entry/common.c:296
> > > > > do_syscall_64+0x4d/0xc0 arch/x86/entry/common.c:86
> > > > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > > >
> > > > > read to 0xffff888237c2a2f8 of 8 bytes by task 19632 on cpu 1:
> > > > > obj_stock_flush_required mm/memcontrol.c:3319 [inline]
> > > > > drain_all_stock+0x174/0x2a0 mm/memcontrol.c:2361
> > > > > try_charge_memcg+0x6d0/0xd10 mm/memcontrol.c:2703
> > > > > try_charge mm/memcontrol.c:2837 [inline]
> > > > > mem_cgroup_charge_skmem+0x51/0x140 mm/memcontrol.c:7290
> > > > > sock_reserve_memory+0xb1/0x390 net/core/sock.c:1025
> > > > > sk_setsockopt+0x800/0x1e70 net/core/sock.c:1525
> > > > > udp_lib_setsockopt+0x99/0x6c0 net/ipv4/udp.c:2692
> > > > > udp_setsockopt+0x73/0xa0 net/ipv4/udp.c:2817
> > > > > sock_common_setsockopt+0x61/0x70 net/core/sock.c:3668
> > > > > __sys_setsockopt+0x1c3/0x230 net/socket.c:2271
> > > > > __do_sys_setsockopt net/socket.c:2282 [inline]
> > > > > __se_sys_setsockopt net/socket.c:2279 [inline]
> > > > > __x64_sys_setsockopt+0x66/0x80 net/socket.c:2279
> > > > > do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> > > > > do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
> > > > > entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > > >
> > > > > value changed: 0xffff8881382d52c0 -> 0xffff888138893740
> > > > >
> > > > > Reported by Kernel Concurrency Sanitizer on:
> > > > > CPU: 1 PID: 19632 Comm: syz-executor.0 Not tainted 6.3.0-rc2-syzk=
aller-00387-g534293368afa #0
> > > > > Hardware name: Google Google Compute Engine/Google Compute Engine=
, BIOS Google 03/02/2023
> > > > >
> > > > > Fix it by reading the cached_objcg with READ_ONCE().
> > > > >
> > > > > Fixes: bf4f059954dc ("mm: memcg/slab: obj_cgroup API")
> > > > > Reported-by: syzbot+774c29891415ab0fd29d@syzkaller.appspotmail.co=
m
> > > > > Reported-by: Dmitry Vyukov <dvyukov@google.com>
> > > > > Link: https://lore.kernel.org/linux-mm/CACT4Y+ZfucZhM60YPphWiCLJr=
6+SGFhT+jjm8k1P-a_8Kkxsjg@mail.gmail.com/T/#t
> > > > > Signed-off-by: Roman Gushchin <roman.gushchin@linux.dev>
> > > >
> > > > Acked-by: Muchun Song <songmuchun@bytedance.com>
> > > >
> > > > Thanks.
> > >
> > > This improves things, but strictly speaking the write side also needs
> > > WRITE_ONCE. Ordering is always a game of two. It's not possible to
> > > order things on one side, if the other side messes up the ordering.
> > >
> >
> > It looks like most other accesses use memcg_stock.stock_lock for
> > synchronization. Based on the output of obj_stock_flush_required()
> > we call drain_local_stock(), which acquires that lock as well. Should
> > we refactor the code to extend the lock section to cover both
> > obj_stock_flush_required() and drain_local_stock()?
> >
> > IIUC this may unify the synchronization handling and
> > READ_ONCE/WRITE_ONCE may no longer be needed. This should also avoid
> > any inaccuracies (e.g. unnecessary flushes) that may happen if the
> > cached objcg changes between obj_stock_flush_required() and
> > drain_local_stock().
> >
> > Did I miss anything here?
>
> Yes, drain_local_stock only works on local cpu and
> obj_stock_flush_required can touch the stock of all the cpus.

Oh right, for other cpus this is infeasible as we schedule the work to
be done later.

>
> The patch is good but I agree with Dmitry that we should add the
> WRITE_ONCE as well.

Agreed. I guess WRITE_ONCE will be needed in multiple places that
update the cached objcg.

Thanks.