Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 1 Nov 2022 21:55:29 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     Hyeonggon Yoo <42.hyeyoo@gmail.com>
CC:     John Thomson <lists@johnthomson.fastmail.com.au>,
        Vlastimil Babka <vbabka@suse.cz>,
        Andrew Morton <akpm@linux-foundation.org>,
        "Christoph Lameter" <cl@linux.com>,
        Pekka Enberg <penberg@kernel.org>,
        David Rientjes <rientjes@google.com>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>,
        Roman Gushchin <roman.gushchin@linux.dev>,
        Dmitry Vyukov <dvyukov@google.com>,
        "Jonathan Corbet" <corbet@lwn.net>,
        Andrey Konovalov <andreyknvl@gmail.com>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kasan-dev@googlegroups.com" <kasan-dev@googlegroups.com>,
        Robin Murphy <robin.murphy@arm.com>,
        John Garry <john.garry@huawei.com>,
        Kefeng Wang <wangkefeng.wang@huawei.com>,
        Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
        John Crispin <john@phrozen.org>,
        "Matthias Brugger" <matthias.bgg@gmail.com>,
        "linux-mips@vger.kernel.org" <linux-mips@vger.kernel.org>
Subject: Re: [PATCH v6 1/4] mm/slub: enable debugging memory wasting of
 kmalloc
Message-ID: <Y2ElURkvmGD5csMc@feng-clx>
References: <af2ba83d-c3f4-c6fb-794e-c2c7c0892c44@suse.cz>
 <Y180l6zUnNjdCoaE@feng-clx>
 <c4285caf-277c-45fd-8fc7-8a1d61685ce8@app.fastmail.com>
 <Y1+0sbQ3R4DB46NX@feng-clx>
 <9b71ae3e-7f53-4c9e-90c4-79d3d649f94c@app.fastmail.com>
 <Y2DReuPHZungAGsU@feng-clx>
 <53b53476-bb1e-402e-9f65-fd7f0ecf94c2@app.fastmail.com>
 <Y2DngwUc7cLB0dG7@hyeyoo>
 <29271a2b-cf19-4af9-bfe5-5bcff8a23fda@app.fastmail.com>
 <Y2D4D52h5VVa8QpE@hyeyoo>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y2D4D52h5VVa8QpE@hyeyoo>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?n2JLZMEqxpOYRgB4O+BMpWHBKdB/OuWHte78WumTz/bxXm/OEIBNsxuktJ4e?=
 =?us-ascii?Q?ApRc3x9ydExDYp7CoXcP6JvDsn533H0UHoExy0LpqdpTI0HoKONEU+FydOkB?=
 =?us-ascii?Q?tONfwMtub2KYJtLtE1UJ0avJWdqXRdfJTBOiIBWQgNeqjzregw1as2T9I2O4?=
 =?us-ascii?Q?BVgkgk1HR8UweaLF82yhbi3GELZrmY+l6mhFgtTroH/AoxbGR/s/yWA/KSXe?=
 =?us-ascii?Q?9muj+2qTfYSIhiD6ChoyxvOD+8DYcAZDL/4h8ElXozQBVubHhU+oyS1xGelg?=
 =?us-ascii?Q?zX6uixJJbUh/hCV0ybEfyuzBg7/hjjy1SNkXKsp+mrS4cKCcPV6BMnKXOGZA?=
 =?us-ascii?Q?7QPZi6wnW9wQc0zRLfUCx2RUozEuL9u1k2KhZflN39JaTKgaA0UuaUaBIbRs?=
 =?us-ascii?Q?GO81LrcN5Uy4urK/kRQNt/FJPmU0CEV0UcP0DbkHjHbnXDRWnNd6CaBMXRG5?=
 =?us-ascii?Q?HaEBquAuBcBCEXJyuqtCiTjAKV1aHG256kXz6fgGsiZThpjJTlXGhn4cIxfY?=
 =?us-ascii?Q?X9s2ExA4NorcfkhEGRS4fNY8R1fJClzXds+I+vm5lvePiKolJurmsxcB26aU?=
 =?us-ascii?Q?SRuNJJ9De7AogyfU3h5acF2AeHBKzMR6opKDyM7iNyByg8EP2olXyqIJEhEa?=
 =?us-ascii?Q?MbBscDq1aJAbxmHF3iMINzU3oL4rDTji6XaqsxLsuGtx3wwAzptHqF5usFnL?=
 =?us-ascii?Q?QKgVV00G2LSt9D00PuTS4A1f3vFJGO9WCRha0kkApNobwxfpEfDhT9RMeyKd?=
 =?us-ascii?Q?7n0nPAlEJljwQRPuX9DjoAdA2B9q0OwEhS7qKUBfwNqPt6D6LETJrTAZIetz?=
 =?us-ascii?Q?Xm1bWgzwBJAO83syi6ossNpmUXY4qSTV8iyKSlHBCsf35wMO1r6ksiRB/Gum?=
 =?us-ascii?Q?zESz2/Sl0WTZVmXxkIpvi6Vsiha7DNc0XlntDjvhZrmiPU4AgKGY3wH+WqTI?=
 =?us-ascii?Q?XM2XbmFRf2uBORHpqyanI4WQET4FatJcqhg97/fa7RyEGGA16VQ7sPODWpgh?=
 =?us-ascii?Q?7QFXjN7VKMpn2uTY2e0sYxzFpsjaS9V5Jrl6spHF95G0AM1DPvmeV1zk0jOK?=
 =?us-ascii?Q?i5Hiw3bTuU6pUcJ3+ZWymb9YPiaY0LStmKM24Nne4nUU4mHSqPfk2q+4bEd7?=
 =?us-ascii?Q?tO6RtkKBSGyZ0xdu42C9GukGnYswuIHwvj9fsLWMDzh0isW7mSTZK0RcTNfB?=
 =?us-ascii?Q?XcXYR08Kt0V0xR9JEIN8jTGRT62dmasYPLxg3rqjfx9QOH7385GOkRrobihN?=
 =?us-ascii?Q?Kck0bjDhCLI9kp7+jB7YnImq6A1PX/qxnvTqnRB2Vx63TTKOyLjwGFn/8Jol?=
 =?us-ascii?Q?fL2DEHYFPanz2kVjhWUbrv67AiZTnXJzqrvxSLIJbt4eOidhhJ+/MjApK3yP?=
 =?us-ascii?Q?P0bEUGQH7QP+o2XJxEHg2178lMM6WxESDy1P3Ap2wVRrZWJcViPz89Cs5UX6?=
 =?us-ascii?Q?0GzjJafVj/p6+F+f8vWKPmQoCh4HxdaYE7Gqo2cW8mxu4u3YQW/BsvhrU5dW?=
 =?us-ascii?Q?xTLOI7rXw1RS++wuCSdWJyyDQCxTt6qT/beEXIVQbPOHkPpavlHf/g/kTrf6?=
 =?us-ascii?Q?2RMD5m1D8R9GXWBvGo3QZ1W6FPaOmwgrNqGzoUIR?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d6799d13-6bf7-4383-cdb0-08dabc10c2a0
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2022 13:55:40.1549
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EC1WrfK5wkNSRq7+lgjw7H/40AgYk+p6RTJC+5767hi0wRMaUMJbd0ziPGFTNyjcBManTbKFYwIE/9sSKLZOUg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5243
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 01, 2022 at 06:42:23PM +0800, Hyeonggon Yoo wrote:
> On Tue, Nov 01, 2022 at 10:33:32AM +0000, John Thomson wrote:
> > On Tue, 1 Nov 2022, at 09:31, Hyeonggon Yoo wrote:
> > > On Tue, Nov 01, 2022 at 09:20:21AM +0000, John Thomson wrote:
> > >> On Tue, 1 Nov 2022, at 07:57, Feng Tang wrote:
> > >> > Hi Thomson,
> > >> >
> > >> > Thanks for testing!
> > >> >
> > >> > + mips maintainer and mail list. The original report is here
> > >> >
> > >> > https://lore.kernel.org/lkml/becf2ac3-2a90-4f3a-96d9-a70f67c66e4a@app.fastmail.com/
> > >>
> > >> I am guessing my issue comes from __kmem_cache_alloc_lru accessing s->object_size when (kmem_cache) s is NULL?
> > >> If that is the case, this change is not to blame, it only exposes the issue?
> > >> 
> > >> I get the following dmesg (note very early NULL kmem_cache) with the below change atop v6.1-rc3:
> > >> 
> > >> transfer started ......................................... transfer ok, time=2.02s
> > >> setting up elf image... OK
> > >> jumping to kernel code
> > >> zimage at:     80B842A0 810B4EFC
> > >> 
> > >> Uncompressing Linux at load address 80001000
> > >> 
> > >> Copy device tree to address  80B80EE0
> > >> 
> > >> Now, booting the kernel...
> > >> 
> > >> [    0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #61 SMP Tue Nov  1 18:04:13 AEST 2022
> > >> [    0.000000] slub: kmem_cache_alloc called with kmem_cache: 0x0
> > >> [    0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache: 0x0
> > >> [    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
> > >> [    0.000000] printk: bootconsole [early0] enabled
> > >> [    0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
> > >> [    0.000000] MIPS: machine is MikroTik RouterBOARD 760iGS
> > >> 
> > >> normal boot
> > >> 
> > >> 
> > >> diff --git a/mm/slub.c b/mm/slub.c
> > >> index 157527d7101b..10fcdf2520d2 100644
> > >> --- a/mm/slub.c
> > >> +++ b/mm/slub.c
> > >> @@ -3410,7 +3410,13 @@ static __always_inline
> > >>  void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> > >>  			     gfp_t gfpflags)
> > >>  {
> > >> -	void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
> > >> +	void *ret;
> > >> +	if (IS_ERR_OR_NULL(s)) {
> > >> +		pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s);
> > >> +		ret = slab_alloc(s, lru, gfpflags, _RET_IP_, 0);
> > >> +	} else {
> > >> +		ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
> > >> +	}
> > >>  
> > >>  	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
> > >>  
> > >> @@ -3419,6 +3425,8 @@ void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> > >>  
> > >>  void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
> > >>  {
> > >> +	if (IS_ERR_OR_NULL(s))
> > >> +		pr_warn("slub: kmem_cache_alloc called with kmem_cache: %pSR\n", s);
> > >>  	return __kmem_cache_alloc_lru(s, NULL, gfpflags);
> > >>  }
> > >>  EXPORT_SYMBOL(kmem_cache_alloc);
> > >> @@ -3426,6 +3434,8 @@ EXPORT_SYMBOL(kmem_cache_alloc);
> > >>  void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> > >>  			   gfp_t gfpflags)
> > >>  {
> > >> +	if (IS_ERR_OR_NULL(s))
> > >> +		pr_warn("slub: __kmem_cache_alloc_lru called with kmem_cache: %pSR\n", s);
> > >>  	return __kmem_cache_alloc_lru(s, lru, gfpflags);
> > >>  }
> > >>  EXPORT_SYMBOL(kmem_cache_alloc_lru);
> > >> 
> > >> 
> > >> Any hints on where kmem_cache_alloc would be being called from this early?
> > >> I will start looking from /init/main.c around pr_notice("%s", linux_banner);
> > >
> > > Great. Would you try calling dump_stack(); when we observed s == NULL?
> > > That would give more information about who passed s == NULL to these
> > > functions.
> > >
> > 
> > With the dump_stack() in place:
> > 
> > Now, booting the kernel...
> > 
> > [    0.000000] Linux version 6.1.0-rc3+ (john@john) (mipsel-buildroot-linux-gnu-gcc.br_real (Buildroot 2021.11-4428-g6b6741b) 12.2.0, GNU ld (GNU Binutils) 2.39) #62 SMP Tue Nov  1 19:49:52 AEST 2022
> > [    0.000000] slub: __kmem_cache_alloc_lru called with kmem_cache ptr: 0x0
> > [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.1.0-rc3+ #62
> > [    0.000000] Stack : 810fff78 80084d98 80889d00 00000004 00000000 00000000 80889d5c 80c90000
> > [    0.000000]         80920000 807bd380 8089d368 80923bd3 00000000 00000001 80889d08 00000000
> > [    0.000000]         00000000 00000000 807bd380 8084bd51 00000002 00000002 00000001 6d6f4320
> > [    0.000000]         00000000 80c97ce9 80c97d14 fffffffc 807bd380 00000000 00000003 00000dc0
> > [    0.000000]         00000000 a0000000 80910000 8110a0b4 00000000 00000020 80010000 80010000
> > [    0.000000]         ...
> > [    0.000000] Call Trace:
> > [    0.000000] [<80008260>] show_stack+0x28/0xf0
> > [    0.000000] [<8070cdc0>] dump_stack_lvl+0x60/0x80
> > [    0.000000] [<801c1428>] kmem_cache_alloc+0x5c0/0x740
> > [    0.000000] [<8092856c>] prom_soc_init+0x1fc/0x2b4
> > [    0.000000] [<80928060>] prom_init+0x44/0xf0
> > [    0.000000] [<80929214>] setup_arch+0x4c/0x6a8
> > [    0.000000] [<809257e0>] start_kernel+0x88/0x7c0
> > [    0.000000] 
> > [    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
> 
> setup_arch() is too early to use slab allocators.
> I think slab received NULL pointer because kmalloc is not initialized.
> 
> It seems arch/mips/ralink/mt7621.c is using slab too early.

Cool! it is finally root caused :) Thanks!

The following patch should solve it and give it a warning message, though
I'm not sure if there is other holes.  

Thanks,
Feng

---
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 33b1886b06eb..429c21b7ecbc 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1043,7 +1043,14 @@ size_t __ksize(const void *object)
 #ifdef CONFIG_TRACING
 void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size)
 {
-	void *ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE,
+	void *ret;
+
+	if (unlikely(ZERO_OR_NULL_PTR(s))) {
+		WARN_ON_ONCE(1);
+		return s;
+	}
+
+	ret = __kmem_cache_alloc_node(s, gfpflags, NUMA_NO_NODE,
 					    size, _RET_IP_);
 
 	trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags, NUMA_NO_NODE);
diff --git a/mm/slub.c b/mm/slub.c
index 157527d7101b..85d24bb6eda7 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3410,8 +3410,14 @@ static __always_inline
 void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
 			     gfp_t gfpflags)
 {
-	void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
+	void *ret;
 
+	if (unlikely(ZERO_OR_NULL_PTR(s))) {
+		WARN_ON_ONCE(1);
+		return s;
+	}
+
+	ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
 	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfpflags, NUMA_NO_NODE);
 
 	return ret;