Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1761836AbbEESZO (ORCPT ); Tue, 5 May 2015 14:25:14 -0400 Received: from mail.efficios.com ([78.47.125.74]:49502 "EHLO mail.efficios.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752771AbbEESZK (ORCPT ); Tue, 5 May 2015 14:25:10 -0400 Date: Tue, 5 May 2015 18:25:12 +0000 (UTC) From: Mathieu Desnoyers To: josh@joshtriplett.org Cc: Andrew Morton , linux-kernel@vger.kernel.org, KOSAKI Motohiro , Steven Rostedt , Nicholas Miell , Linus Torvalds , Ingo Molnar , Alan Cox , Lai Jiangshan , Stephen Hemminger , Thomas Gleixner , Peter Zijlstra , David Howells , Pranith Kumar , Michael Kerrisk Message-ID: <2080861989.43911.1430850312443.JavaMail.zimbra@efficios.com> In-Reply-To: <20150504213047.GA8109@cloud> References: <1430773213-8981-1-git-send-email-mathieu.desnoyers@efficios.com> <1430773213-8981-2-git-send-email-mathieu.desnoyers@efficios.com> <20150504213047.GA8109@cloud> Subject: Re: [PATCH v17 1/2] sys_membarrier(): system-wide memory barrier (generic, x86) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [24.114.81.217] X-Mailer: Zimbra 8.0.7_GA_6021 (ZimbraWebClient - FF37 (Linux)/8.0.7_GA_6021) Thread-Topic: sys_membarrier(): system-wide memory barrier (generic, x86) Thread-Index: b1v4JO8zaxWGOvgSsnrLU8XvfgOHkA== Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4957 Lines: 112 ----- Original Message ----- > On Mon, May 04, 2015 at 05:00:12PM -0400, Mathieu Desnoyers wrote: > > * Benchmarks > > > > On Intel Xeon E5405 (8 cores) > > (one thread is calling sys_membarrier, the other 7 threads are busy > > looping) > > > > 1000 non-expedited sys_membarrier calls in 33s = 33 milliseconds/call. > > > > * User-space user of this system call: Userspace RCU library > > > > Both the signal-based and the sys_membarrier userspace RCU schemes > > permit us to remove the memory barrier from the userspace RCU > > rcu_read_lock() and rcu_read_unlock() primitives, thus significantly > > accelerating them. These memory barriers are replaced by compiler > > barriers on the read-side, and all matching memory barriers on the > > write-side are turned into an invocation of a memory barrier on all > > active threads in the process. By letting the kernel perform this > > synchronization rather than dumbly sending a signal to every process > > threads (as we currently do), we diminish the number of unnecessary wake > > ups and only issue the memory barriers on active threads. Non-running > > threads do not need to execute such barrier anyway, because these are > > implied by the scheduler context switches. > > > > Results in liburcu: > > > > Operations in 10s, 6 readers, 2 writers: > > > > memory barriers in reader: 1701557485 reads, 3129842 writes > > signal-based scheme: 9825306874 reads, 5386 writes > > sys_membarrier: 7992076602 reads, 220 writes > > > > The dynamic sys_membarrier availability check adds some overhead to > > the read-side compared to the signal-based scheme, but besides that, > > with the expedited scheme, we can see that we are close to the read-side > > performance of the signal-based scheme. However, this non-expedited > > sys_membarrier implementation has a much slower grace period than signal > > and memory barrier schemes. > > > > An expedited version of this system call can be added later on to speed > > up the grace period. Its implementation will likely depend on reading > > the cpu_curr()->mm without holding each CPU's rq lock. > > So, I realize that there's a lot of history tied up in the previous 16 > versions and associated mail threads. However, can you please summarize > in the commit message what the benefit of merging this version is? > Because from the text above, from liburcu's perspective, it appears to > be strictly worse in performance than the signal-based scheme. > > There are other non-performance reasons why it might make sense to > include this; for instance, signals don't play nice with libraries, with > other processes you might inject yourself into for tracing purposes, or > with general sanity. However, the explanation for those use cases and > how membarrier() improves them needs to go in the commit message, rather > than only in the collective memory and mail archives of people who have > discussed this patch series. > > (My apologies if the if the explanation is in the commit message and > I've just missed it.) I will add info about signals vs libraries, which appears to be missing from the commit message: "Besides diminishing the number of wake-ups, one major advantage of the membarrier system call over the signal-based scheme is that it does not need to reserve a signal. This plays much more nicely with libraries, and with processes injected into for tracing purposes, for which we cannot expect that signals will be unused by the application." The commit message already point out that sys_membarrier diminishes the number of unnecessary wake-ups sent to other threads compared to the signal-based approach. I re-ran those tests on urcu master branch with a slightly modified version of the sys_membarrier scheme too: a version which assumes that sys_membarrier is always available. We can then compare apples to apples performance-wise between signal and membarrier approaches: Results in liburcu: Operations in 10s, 6 readers, 2 writers: memory barriers in reader: 1701557485 reads, 3129842 writes signal-based scheme: 9830061167 reads, 6700 writes sys_membarrier: 9952759104 reads, 425 writes sys_membarrier (dyn. check): 7970328887 reads, 425 writes It shows that sys_membarrier read-side actually performs slightly better than the signal-based scheme, in the absence of dynamic check for syscall availability. This could be enhanced in userspace eventually if we decide to implement self-modifying code upon feature detection in liburcu. I'll update the commit message with this new table. Thanks! Mathieu > > - Josh Triplett > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/