Date: Mon, 31 Jul 2017 11:00:19 -0700
From: Dave Watson <davejwatson@fb.com>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
CC: Avi Kivity <avi@scylladb.com>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
        maged michael <maged.michael@gmail.com>,
        Andrew Hunter <ahh@google.com>, gromer <gromer@google.com>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Peter Zijlstra <peterz@infradead.org>
Subject: Re: Udpated sys_membarrier() speedup patch, FYI
Message-ID: <20170731180019.GA74975@dhcp-172-20-173-153.dhcp.thefacebook.com>
References: <20170727181250.GA20183@linux.vnet.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20170727181250.GA20183@linux.vnet.ibm.com>
User-Agent: Mutt/1.6.0 (2016-04-01)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jul 2017 18:00:28.4880 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1753
X-OriginatorOrg: fb.com
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-07-31_06:,,
 signatures=0
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 1823
Lines: 51

Hi Paul, 

Thanks for looking at this again!

On 07/27/17 11:12 AM, Paul E. McKenney wrote:
> Hello!
> 
> But my main question is whether the throttling shown below is acceptable
> for your use cases, namely only one expedited sys_membarrier() permitted
> per scheduling-clock period (1 millisecond on many platforms), with any
> excess being silently converted to non-expedited form.  The reason for
> the throttling is concerns about DoS attacks based on user code with a
> tight loop invoking this system call.

We've been using sys_membarrier for the last year or so in a handful
of places with no issues.  Recently we made it an option in our hazard
pointers implementation, giving us something with performance between
hazard pointers and RCU:

https://github.com/facebook/folly/blob/master/folly/experimental/hazptr/hazptr-impl.h#L555

Currently hazard pointers tries to free retired memory the same thread
that did the retire(), so the latency spiked for workloads that were
retire() heavy.   For the moment we dropped back to using mprotect
hacks.

I've tested Mathieu's v4 patch, it works great.  We currently don't
have any cases where we need SHARED. 

I also tested the rate-limited version, while better than the current
non-EXPEDITED SHARED version, we still hit the slow path pretty often.
I agree with other commenters that returning an error code instead of
silently slowing down might be better.

> +	case MEMBARRIER_CMD_SHARED_EXPEDITED:
> +		if (num_online_cpus() > 1) {
> +			static unsigned long lastexp;
> +			unsigned long j;
> +
> +			j = jiffies;
> +			if (READ_ONCE(lastexp) == j) {
> +				synchronize_sched();
> +				WRITE_ONCE(lastexp, j);

It looks like this update of lastexp should be in the other branch?

> +			} else {
> +				synchronize_sched_expedited();
> +			}
> +		}
> +		return 0;