Received-SPF: pass (google.com: domain of linux-kernel+bounces-187979-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <156dc43f-fdcf-4643-83d9-b374452b0929@efficios.com>
Date: Thu, 23 May 2024 16:39:33 -0400
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2 0/1] Add FUTEX_SPIN operation
To: =?UTF-8?Q?Andr=C3=A9_Almeida?= <andrealmeid@igalia.com>,
 Peter Zijlstra <peterz@infradead.org>
Cc: linux-kernel@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
 "Paul E . McKenney" <paulmck@kernel.org>, Boqun Feng <boqun.feng@gmail.com>,
 "H . Peter Anvin" <hpa@zytor.com>, Paul Turner <pjt@google.com>,
 linux-api@vger.kernel.org, Christian Brauner <brauner@kernel.org>,
 Florian Weimer <fw@deneb.enyo.de>, David.Laight@ACULAB.COM,
 carlos@redhat.com, Peter Oskolkov <posk@posk.io>,
 Alexander Mikhalitsyn <alexander@mihalicyn.com>,
 Chris Kennelly <ckennelly@google.com>, Ingo Molnar <mingo@redhat.com>,
 Darren Hart <dvhart@infradead.org>, Davidlohr Bueso <dave@stgolabs.net>,
 libc-alpha@sourceware.org, Steven Rostedt <rostedt@goodmis.org>,
 Jonathan Corbet <corbet@lwn.net>, Noah Goldstein <goldstein.w.n@gmail.com>,
 longman@redhat.com, kernel-dev@igalia.com
References: <20240523200704.281514-1-andrealmeid@igalia.com>
From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Content-Language: en-US
In-Reply-To: <20240523200704.281514-1-andrealmeid@igalia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 2024-05-23 16:07, André Almeida wrote:
> Hi,
> 
> In the last LPC, Mathieu Desnoyers and I presented[0] a proposal to extend the
> rseq interface to be able to implement spin locks in userspace correctly. Thomas
> Gleixner agreed that this is something that Linux could improve, but asked for
> an alternative proposal first: a futex operation that allows to spin a user
> lock inside the kernel. This patchset implements a prototype of this idea for
> further discussion.
> 
> With FUTEX2_SPIN flag set during a futex_wait(), the futex value is expected to
> be the TID of the lock owner. Then, the kernel gets the task_struct of the
> corresponding TID, and checks if it's running. It spins until the futex
> is awaken, the task is scheduled out or if a timeout happens.  If the lock owner
> is scheduled out at any time, then the syscall follows the normal path of
> sleeping as usual. The user input is masked with FUTEX_TID_MASK so we have some
> bits to play.
> 
> If the futex is awaken and we are spinning, we can return to userspace quickly,
> avoid the scheduling out and in again to wake from a futex_wait(), thus
> speeding up the wait operation. The user input is masked with FUTEX_TID_MASK so
> we have some bits to play.
> 
> Christian Brauner suggested using pidfd to avoid race conditions, and I will
> implement that in the next patch iteration. I benchmarked the implementation
> measuring the time required to wait for a futex for a simple loop using the code
> at [2]. In my setup, the total wait time for 1000 futexes using the spin method
> was almost 10% lower than just using the normal futex wait:
> 
> 	Testing with FUTEX2_SPIN | FUTEX_WAIT
> 	Total wait time: 8650089 usecs
> 
> 	Testing with FUTEX_WAIT
> 	Total wait time: 9447291 usecs
> 
> However, as I played with how long the lock owner would be busy, the
> benchmark results of spinning vs no spinning would match, showing that the
> spinning will be effective for some specific scheduling scenarios, but depending
> on the wait time, there's no big difference either spinning or not.
> 
> [0] https://lpc.events/event/17/contributions/1481/
> 
> You can find a small snippet to play with this interface here:
> 
> [1] https://gist.github.com/andrealmeid/f0b8c93a3c7a5c50458247c47f7078e1

What exactly are you trying to benchmark here ? I've looked at this toy
program, and I suspect that most of the delay you observe is due to
initial scheduling of a newly cloned thread, because this is what is
repeatedly being done in the delay you measure.

I would recommend to change this benchmark program to measure something
meaningful, e.g.:

- N threads repeatedly contending on a lock, until a "stop" flag is set,
- run for "duration" seconds, after which main() sets a "stop" flag.
- delay loop of "work_delay" us within the lock critical section,
- delay loop of "inactive_delay" us between locking attempts,
- measure the time it takes to grab the lock, report stats on this,
- measure the total number of operations done within the given
   "duration".
- report statistics on the number of operations per thread to see
   the impact on fairness,

The run the program with the following constraints:

- Pin one thread per core, with nb thread <= nb cores. This should
   be a best case scenario for spinning.
- Pin all threads to a single core. when nb threads > nb cores, this
   should be the worse scenario for spinning.
- Groups things between those two extremes to see how things evolve.

I would not be surprised that if you measure relevant delays, you will
observe much better speedups than what you currently have.

Thanks,

Mathieu

> 
> Changelog:
> 
> v1: - s/PID/TID
>      - masked user input with FUTEX_TID_MASK
>      - add benchmark tool to the cover letter
>      - dropped debug prints
>      - added missing put_task_struct()
> 
> André Almeida (1):
>    futex: Add FUTEX_SPIN operation
> 
>   include/uapi/linux/futex.h |  2 +-
>   kernel/futex/futex.h       |  6 ++-
>   kernel/futex/waitwake.c    | 78 +++++++++++++++++++++++++++++++++++++-
>   3 files changed, 82 insertions(+), 4 deletions(-)
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com