Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751537AbdIRT3n (ORCPT ); Mon, 18 Sep 2017 15:29:43 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:45263 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751390AbdIRT3k (ORCPT ); Mon, 18 Sep 2017 15:29:40 -0400 Date: Mon, 18 Sep 2017 12:29:35 -0700 From: "Paul E. McKenney" To: Alan Stern Cc: peterz@infradead.org, mathieu.desnoyers@efficios.com, will.deacon@arm.com, luto@kernel.org, mpe@ellerman.id.au, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, davejwatson@fb.com, maged.michael@gmail.com Subject: Re: Rough notes from sys_membarrier() lightning BoF Reply-To: paulmck@linux.vnet.ibm.com References: <20170917223608.GA14577@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17091819-0056-0000-0000-000003CB0F99 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007757; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000230; SDB=6.00918924; UDB=6.00461645; IPR=6.00699151; BA=6.00005595; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00017199; XFM=3.00000015; UTC=2017-09-18 19:29:38 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17091819-0057-0000-0000-0000080212B5 Message-Id: <20170918192935.GW3521@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-09-18_08:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1709180275 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3317 Lines: 74 On Mon, Sep 18, 2017 at 03:04:21PM -0400, Alan Stern wrote: > On Sun, 17 Sep 2017, Paul E. McKenney wrote: > > > Hello! > > > > Rough notes from our discussion last Thursday. Please reply to the > > group with any needed elaborations or corrections. > > > > Adding Andy and Michael on CC since this most closely affects their > > architectures. Also adding Dave Watson and Maged Michael because > > the preferred approach requires that processes wanting to use the > > lightweight sys_membarrier() do a registration step. > > > > Thanx, Paul > > > > ------------------------------------------------------------------------ > > > > Problem: > > > > 1. The current sys_membarrier() introduces an smp_mb() that > > is not otherwise required on powerpc. > > > > 2. The envisioned JIT variant of sys_membarrier() assumes that > > the return-to-user instruction sequence handling any change > > to the usermode instruction stream, and Andy Lutomirski's > > upcoming changes invalidate this assumption. It is believed > > that powerpc has a similar issue. > > > E. Require that threads register before using sys_membarrier() for > > private or JIT usage. (The historical implementation using > > synchronize_sched() would continue to -not- require registration, > > both for compatibility and because there is no need to do so.) > > > > For x86 and powerpc, this registration would set a TIF flag > > on all of the current process's threads. This flag would be > > inherited by any later thread creation within that process, and > > would be cleared by fork() and exec(). When this TIF flag is set, > > Why a TIF flag, and why clear it during fork()? If a process registers > to use private expedited sys_membarrier, shouldn't that apply to > threads it will create in the future just as much as to threads it has > already created? The reason for a TIF flag is to keep this per-architecture, as only powerpc and x86 need it. The reason for clearing it during fork() is that fork() creates a new process initially having but a single thread, which might or might not use sys_membarrier(). Usually not, as most instances of fork() are quickly followed by exec(). In addition, if we give an error for unregistered use of private sys_membarrier(), clearing on fork() gets an unambiguous error instead of a silent likely failure (due to libraries being confused by the fork()). That said, pthread_create() should preserve the flag, as the new thread will be part of this same process. > > the return-to-user path would execute additional code that would > > ensure that ordering and newly JITed code was handled correctly. > > We believe that checks for these TIF flags could be combined with > > existing checks to avoid adding any overhead in the common case > > where the process was not using these sys_membarrier() features. > > > > For all other architecture, the registration step would be > > a no-op. > > Don't we want to fail private expedited sys_membarrier calls if the > process hasn't registered for them? This requires the registration > call to set a flag for the process, even on architectures where no > additional memory barriers are actually needed. It can't be a no-op. Good point, and we did discuss that. Color me forgetful!!! Thanx, Paul