Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Message-ID: <e0647e65-d348-35a0-bc5a-66d1ad02ea53@suse.cz>
Date:   Wed, 29 Nov 2023 14:25:32 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: [RFC v2 2/7] mm, slub: add opt-in slub_percpu_array
Content-Language: en-US
To:     Hyeonggon Yoo <42.hyeyoo@gmail.com>
Cc:     "Liam R. Howlett" <Liam.Howlett@oracle.com>,
        Matthew Wilcox <willy@infradead.org>,
        Suren Baghdasaryan <surenb@google.com>,
        Christoph Lameter <cl@linux.com>,
        David Rientjes <rientjes@google.com>,
        Pekka Enberg <penberg@kernel.org>,
        Joonsoo Kim <iamjoonsoo.kim@lge.com>,
        Roman Gushchin <roman.gushchin@linux.dev>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, patches@lists.linux.dev
References: <20230810163627.6206-9-vbabka@suse.cz>
 <20230810163627.6206-11-vbabka@suse.cz>
 <CAB=+i9TSMVURktFvr7sAt4T2BdaUvsWFapAjTZNtk0AKS01O9A@mail.gmail.com>
 <077e8e97-e88f-0b8e-2788-4031458be090@suse.cz>
 <CAB=+i9SQBhYBjOSHDeqgJJ5YARqZCS3oUUutzr4m+2V+ZvySpg@mail.gmail.com>
From:   Vlastimil Babka <vbabka@suse.cz>
In-Reply-To: <CAB=+i9SQBhYBjOSHDeqgJJ5YARqZCS3oUUutzr4m+2V+ZvySpg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 11/29/23 01:46, Hyeonggon Yoo wrote:
> On Wed, Nov 29, 2023 at 2:37 AM Vlastimil Babka <vbabka@suse.cz> wrote:
> 
>> >> @@ -4060,6 +4201,45 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
>> >>  }
>> >>  EXPORT_SYMBOL(kmem_cache_alloc_bulk);
>> >>
>> >> +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count,
>> >> +               gfp_t gfp)
>> >> +{
>> >> +       struct slub_percpu_array *pca;
>> >> +       void *objects[32];
>> >> +       unsigned int used;
>> >> +       unsigned int allocated;
>> >> +
>> >> +       if (!s->cpu_array)
>> >> +               return -EINVAL;
>> >> +
>> >> +       /* racy but we don't care */
>> >> +       pca = raw_cpu_ptr(s->cpu_array);
>> >> +
>> >> +       used = READ_ONCE(pca->used);
>> >
>> > Hmm for the prefill to be meaningful,
>> > remote allocation should be possible, right?
>>
>> Remote in what sense?
> 
> TL;DR) What I wanted to ask was:
> "How pre-filling a number of objects works when the pre-filled objects
> are not shared between CPUs"
> 
> IIUC the prefill is opportunistically filling the array so (hopefully)
> expecting there are
> some objects filled in it.

Yes.

> Let's say CPU X calls kmem_cache_prefill_percpu_array(32) and all 32
> objects are filled into CPU X's array.
> But if CPU Y can't allocate from CPU X's array (which I referred to as
> "remote allocation"), the semantics differ from
> the maple tree's perspective because preallocated objects were shared
> between CPUs before, but now it's not?

The assumption is that the operation will prefill on CPU X and then consume
it also on X, because shortly after prefill it will enter some restricted
context (i.e. spin_lock_irqsave or whatnot) that prevents it from migrating.
That's not guaranteed of course, but migration in a bad moment and
subsequent depleted array should be rare enough that we'll just handle it in
the slow paths, and if it results in dipping into reserves, it won't be too
disruptive.

> Thanks!
> 
> --
> Hyeonggon