Received-SPF: pass (google.com: domain of linux-kernel+bounces-33757-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <ac0ce290-58a7-42b2-a7e4-72e241717c63@suse.cz>
Date: Mon, 22 Jan 2024 18:13:13 +0100
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/3] mm/slub: directly load freelist from cpu partial slab
 in the likely case
Content-Language: en-US
To: Chengming Zhou <zhouchengming@bytedance.com>,
 "Christoph Lameter (Ampere)" <cl@linux.com>
Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Joonsoo Kim
 <iamjoonsoo.kim@lge.com>, Pekka Enberg <penberg@kernel.org>,
 Andrew Morton <akpm@linux-foundation.org>,
 Roman Gushchin <roman.gushchin@linux.dev>,
 David Rientjes <rientjes@google.com>, linux-mm@kvack.org,
 linux-kernel@vger.kernel.org
References: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com>
 <20240117-slab-misc-v1-1-fd1c49ccbe70@bytedance.com>
 <76641777-1918-2b29-b6aa-bda9b5467aa3@gentwo.org>
 <412b8618-0941-4d9d-85df-ee480695e7f7@bytedance.com>
 <a2132c63-99a5-7fa2-9f2a-cccf6b40fe9e@linux.com>
 <e81d914b-8718-4dbb-a2d8-d5298fe66d1a@bytedance.com>
From: Vlastimil Babka <vbabka@suse.cz>
In-Reply-To: <e81d914b-8718-4dbb-a2d8-d5298fe66d1a@bytedance.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 1/19/24 04:53, Chengming Zhou wrote:
> On 2024/1/19 06:14, Christoph Lameter (Ampere) wrote:
>> On Thu, 18 Jan 2024, Chengming Zhou wrote:
>> 
>>> So get_freelist() has two cases to handle: cpu slab and cpu partial list slab.
>>> The latter is NOT frozen, so need to remove "VM_BUG_ON(!new.frozen)" from it.
>> 
>> Right so keep the check if it is the former?
>> 
> 
> Ok, I get it. Maybe like this:

I think that's just too ugly for a VM_BUG_ON(). I'd just remove the check
and be done with that.

I have a somewhat different point. You reused get_freelist() but in fact
it's more like freeze_slab(), but that one uses slab_update_freelist() and
we are under the local_lock so we want the cheaper __slab_update_freelist(),
which get_freelist() has and I guess that's why you reused that one.

However get_freelist() also assumes it can return NULL if the freelist is
empty. If that's possible to happen on the percpu partial list, we should
not "goto load_freelist;" but rather create a new label above that, above
the "if (!freelist) {" block that handles the case.

If that's not possible to happen (needs careful audit) and we have guarantee
that slabs on percpu partial list must have non-empty freelist, then we
probably instead want a new __freeze_slab() variant that is like
freeze_slab(), but uses __slab_update_freelist() and probably also has
VM_BUG_ON(!freelist) before returning it?

> 
> diff --git a/mm/slub.c b/mm/slub.c
> index 2ef88bbf56a3..7fa9dbc2e938 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3313,7 +3313,7 @@ __update_cpu_freelist_fast(struct kmem_cache *s,
>   *
>   * If this function returns NULL then the slab has been unfrozen.
>   */
> -static inline void *get_freelist(struct kmem_cache *s, struct slab *slab)
> +static inline void *get_freelist(struct kmem_cache *s, struct slab *slab, int frozen)
>  {
>         struct slab new;
>         unsigned long counters;
> @@ -3326,7 +3326,7 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab)
>                 counters = slab->counters;
> 
>                 new.counters = counters;
> -               VM_BUG_ON(!new.frozen);
> +               VM_BUG_ON(frozen && !new.frozen);
> 
>                 new.inuse = slab->objects;
>                 new.frozen = freelist != NULL;
> @@ -3440,7 +3440,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
>         if (freelist)
>                 goto load_freelist;
> 
> -       freelist = get_freelist(s, slab);
> +       freelist = get_freelist(s, slab, 1);
> 
>         if (!freelist) {
>                 c->slab = NULL;
> @@ -3498,18 +3498,19 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
> 
>                 slab = slub_percpu_partial(c);
>                 slub_set_percpu_partial(c, slab);
> -               local_unlock_irqrestore(&s->cpu_slab->lock, flags);
> -               stat(s, CPU_PARTIAL_ALLOC);
> 
> -               if (unlikely(!node_match(slab, node) ||
> -                            !pfmemalloc_match(slab, gfpflags))) {
> -                       slab->next = NULL;
> -                       __put_partials(s, slab);
> -                       continue;
> +               if (likely(node_match(slab, node) &&
> +                          pfmemalloc_match(slab, gfpflags))) {
> +                       c->slab = slab;
> +                       freelist = get_freelist(s, slab, 0);
> +                       stat(s, CPU_PARTIAL_ALLOC);
> +                       goto load_freelist;
>                 }
> 
> -               freelist = freeze_slab(s, slab);
> -               goto retry_load_slab;
> +               local_unlock_irqrestore(&s->cpu_slab->lock, flags);
> +
> +               slab->next = NULL;
> +               __put_partials(s, slab);
>         }
>  #endif