Received-SPF: pass (google.com: domain of linux-kernel+bounces-85654-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <ed828ce7-2046-4884-ab1f-d7bff3c0a714@efficios.com>
Date: Wed, 28 Feb 2024 15:01:55 -0500
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 00/30] RSEQ node id and mm concurrency id extensions
Content-Language: en-US
To: Marco Elver <elver@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>, linux-kernel@vger.kernel.org,
 Thomas Gleixner <tglx@linutronix.de>, "Paul E . McKenney"
 <paulmck@kernel.org>, Boqun Feng <boqun.feng@gmail.com>,
 "H . Peter Anvin" <hpa@zytor.com>, Paul Turner <pjt@google.com>,
 linux-api@vger.kernel.org, Christian Brauner <brauner@kernel.org>,
 Florian Weimer <fw@deneb.enyo.de>, David.Laight@aculab.com,
 carlos@redhat.com, Peter Oskolkov <posk@posk.io>,
 Alexander Mikhalitsyn <alexander@mihalicyn.com>,
 Chris Kennelly <ckennelly@google.com>, dvyukov@google.com
References: <20221122203932.231377-1-mathieu.desnoyers@efficios.com>
 <Zd-AfDcQ-r04CMXk@elver.google.com>
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
In-Reply-To: <Zd-AfDcQ-r04CMXk@elver.google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 2024-02-28 13:50, Marco Elver wrote:
> Hi Mathieu, all,
> 
> On Tue, Nov 22, 2022 at 03:39PM -0500, Mathieu Desnoyers wrote:
>> Extend the rseq ABI to expose NUMA node ID, mm_cid, and mm_numa_cid
>> fields.
>>
>> The NUMA node ID field allows implementing a faster getcpu(2) in libc.
>>
>> The per-memory-map concurrency id (mm_cid) [1] allows ideal scaling
>> (down or up) of user-space per-cpu data structures. The concurrency ids
>> allocated within a memory map are tracked by the scheduler, which takes
>> into account the number of concurrently running threads, thus implicitly
>> considering the number of threads, the cpu affinity, the cpusets
>> applying to those threads, and the number of logical cores on the
>> system.
>>
>> The NUMA-aware concurrency id (mm_numa_cid) is similar to the mm_cid,
>> except that it keeps track of the NUMA node ids with which each cid has
>> been associated. On NUMA systems, when a NUMA-aware concurrency ID is
>> observed by user-space to be associated with a NUMA node, it is
>> guaranteed to never change NUMA node unless a kernel-level NUMA
>> configuration change happens. This is useful for NUMA-aware per-cpu data
>> structures running in environments where a process or a set of processes
>> belonging to cpuset are pinned to a set of cores which belong to a
>> subset of the system's NUMA nodes.
> [...]
> 
> Just out of curiosity: is anyone aware of any libraries that have
> started using CIDs? It looks like the cost of CID assignment is always
> paid (even though it should be small), I'm trying to understand if after
> 1.5 years there are common libraries that have started using it and what
> their exact usecase is.

Hi Marco,

AFAIK the only project using the mm_cid concept I know of today is
tcmalloc. It's very useful to scale data structures such as memory
allocator arenas to the number of concurrently running threads
within a process without having to rely on heuristics on the
user-space side.

I have plans to migrate LTTng-UST to per-ipc-namespace NUMA-aware
mm_cid as well (after I get around to submit this extension into the
Linux kernel) for user-space ring buffers over shared memory, but my
current focus has been on pushing support for extensible RSEQ into
GNU libc for the past year or so.

We are getting there though:

https://sourceware.org/pipermail/libc-alpha/2024-February/154390.html

Once we have this key piece in place within GNU libc, it will become
easier to extend rseq further because the libc will adapt to the extended
feature set.

Note that the overhead of the mm_cid assignment within the scheduler
should be negligible after
commit 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid").

Another thing we've actively been working on is to get the "librseq"
project [1] in shape so a copy the librseq headers can be integrated
into the GNU libc project as internal header files. So basically
librseq will become a GNU libc upstream. This will facilitate
implementation of rseq critical section within GNU libc. One of
the possible use-cases will be to move the GNU libc malloc
implementation to per-mm_cid arenas.

> 
> I'm aware that TCMalloc was the inspiration for vCPUs [1], then renamed to
> CIDs, but am wondering if other users are out there.

I'd be curious to learn about those as well.

I suspect that the lack of official release of librseq critical section
helper headers may contribute to the fact that few applications use advanced
rseq features at this point.

Thanks,

Mathieu

[1] https://git.kernel.org/pub/scm/libs/librseq/librseq.git/

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com