Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp2093265imm; Fri, 7 Sep 2018 10:40:00 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbvoF3GAa55UJhuQiWeRxzL2YYLB8B/kwSJePl6zLielK53DEW7RWa9nJDwkC6VCS72nn1C X-Received: by 2002:a17:902:42c3:: with SMTP id h61-v6mr9216942pld.319.1536342000039; Fri, 07 Sep 2018 10:40:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536342000; cv=none; d=google.com; s=arc-20160816; b=L45bTc0xhY4u41TD/miVZlqWEG3K8wHrchIVmuKzP94U2ELuItqnwK6qp7dT1tZQ8N lRpt9oMiupqJ6UDAzsPk287nVa/sCM7PoqRPG71Jt8G000KNzp3LBXot+QOHB+AdanLZ 0x16pIbTuJVJoeVJKoG1UGoP5hfP79SCcmRdvs4MFhm0YyvPl/jtVldvco57ojbDnwVo +kt64LuayEGhQH0TsDwxT9dNuNAsD9p53rxMAMKYO3HR6xGJhM2dcNLM+y9PP3LewgE3 rLH92jDQrEQDa5h1gkLi8LlwYaC1VF4rHlmsIBhNjMhF3zaBaUaFH+NaT38A+k2atU77 8QKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:message-id:in-reply-to :subject:cc:to:from:date; bh=whlJFXvEd2ng0H/HQ5eH+gF1sX8H16YdCkGJZoUzT+c=; b=sG4yp715o/vmSQS22hMNDfhdEkw9kcmR3fTmpmeZ1SzFKMxZ7vJCUD/aMOykWElGug uxHOxxmkSvSQzVsPoY7A0vLP3hGOjyfxw1KMI06K7Y4+BbRCkvFYX0dB88rGe/Ro3+7n o3pTAIxrgq5Vsh0cK/C9IwPGJXn0MMIr6I+OlOibQswtEc+8h7h6tvbxKTtoYCMMAoZT EzJZLNGCpD0/HjGxaAWSJW67gqwJpoxuJWJjDixK1rPtaCZXFy0psa8M7WnFdLGum63H WUFuy0M1FalH3HGPqK/OA5Yr4tE7YEnk7nEzfyVN7Dp3GaIz95CELjdJCIJeF2hUB0NP fqOQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e4-v6si9349643pgk.630.2018.09.07.10.39.44; Fri, 07 Sep 2018 10:39:59 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727093AbeIGWUc (ORCPT + 99 others); Fri, 7 Sep 2018 18:20:32 -0400 Received: from iolanthe.rowland.org ([192.131.102.54]:56760 "HELO iolanthe.rowland.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1726614AbeIGWUc (ORCPT ); Fri, 7 Sep 2018 18:20:32 -0400 Received: (qmail 4939 invoked by uid 2102); 7 Sep 2018 13:38:33 -0400 Received: from localhost (sendmail-bs@127.0.0.1) by localhost with SMTP; 7 Sep 2018 13:38:33 -0400 Date: Fri, 7 Sep 2018 13:38:33 -0400 (EDT) From: Alan Stern X-X-Sender: stern@iolanthe.rowland.org To: Daniel Lustig cc: Will Deacon , Andrea Parri , Andrea Parri , "Paul E. McKenney" , Kernel development list , , , , , , , Jade Alglave , Luc Maranget , , Palmer Dabbelt Subject: Re: [PATCH RFC LKMM 1/7] tools/memory-model: Add extra ordering for locks and remove it for ordinary release/acquire In-Reply-To: <20990a3f-1507-c98b-f14e-2f5241319d8c@nvidia.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 7 Sep 2018, Daniel Lustig wrote: > On 9/7/2018 9:09 AM, Will Deacon wrote: > > On Fri, Sep 07, 2018 at 12:00:19PM -0400, Alan Stern wrote: > >> On Thu, 6 Sep 2018, Andrea Parri wrote: > >> > >>>> Have you noticed any part of the generic code that relies on ordinary > >>>> acquire-release (rather than atomic RMW acquire-release) in order to > >>>> implement locking constructs? > >>> > >>> There are several places in code where the "lock-acquire" seems to be > >>> provided by an atomic_cond_read_acquire/smp_cond_load_acquire: I have > >>> mentioned one in qspinlock in this thread; qrwlock and mcs_spinlock > >>> provide other examples (grep for the primitives...). > >>> > >>> As long as we don't consider these primitive as RMW (which would seem > >>> odd...) or as acquire for which "most people expect strong ordering" > >>> (see above), these provides other examples for the _gap_ I mentioned. > >> > >> Okay, now I understand your objection. It does appear that on RISC-V, > >> if nowhere else, the current implementations of qspinlock, qrwlock, > >> etc. will not provide "RCtso" ordering. > >> > >> The discussions surrounding this topic have been so lengthy and > >> confusing that I have lost track of any comments Palmer or Daniel may > >> have made concerning this potential problem. > >> > >> One possible resolution would be to define smp_cond_load_acquire() > >> specially on RISC-V so that it provided the same ordering guarantees as > >> RMW-acquire. (Plus adding a comment in the asm-generic/barrier.h > >> pointing out the necessity for the stronger guarantee on all > >> architectures.) > >> > >> Another would be to replace the usages of atomic/smp_cond_load_acquire > >> in the locking constructs with a new function that would otherwise be > >> the same but would provide the ordering guarantee we want. > >> > >> Do you think either of these would be an adequate fix? > > > > I didn't think RISC-V used qspinlock or qrwlock, so I'm not sure there's > > actually anything to fix, is there? > > > > Will > > I've also lost track of whether the current preference is or is not for > RCtso, or in which subset of cases RCtso is currently preferred. For > whichever cases do in fact need to be RCtso, the RISC-V approach would > still be the same as what I've written in the past, as far as I can > tell [1]. The patch which Paul plans to send in for the next merge window makes the LKMM require RCtso ordering for spinlocks, and by extension, for all locking operations. As I understand it, the current RISC-V implementation of spinlocks does provide this ordering. We have discussed creating another patch for the LKMM which would require RMW-acquire/ordinary-release also to have RCtso ordering. Nobody has written the patch yet, but it would be straightfoward. The rationale is that many types of locks are implemented in terms of RMW-acquire, so if the locks are required to be RCtso then so should the lower-level operations they are built from. Will feels strongly (and Linus agrees) that the LKMM should not require ordinary acquire and release to be any stronger than RCpc. The issue that Andrea raised has to do with qspinlock, qrwlock, and mcs_spinlock, which are implemented using smp_cond_load_acquire() instead of RMW-acquire. This provides only the ordering properties of smp_load_acquire(), namely RCpc, which means that qspinlocks etc. might not be RCtso. Since we do want locks to be RCtso, the question is how to resolve this discrepancy. > In a nutshell, if a data structure uses only atomics with .aq/.rl, > RISC-V provides RCtso already anyway. If a data structure uses fences, > or mixes fences and atomics, we can replace a "fence r,rw" or a > "fence rw,w" with a "fence.tso" (== fence r,rw + fence rw,w) as > necessary, at the cost of some amount of performance. > > I suppose the answer to the question of whether smp_cond_load_acquire() > needs to change depends on where exactly RCtso is needed, and which > data structures actually use that vs. some other macro. > > Does that answer your question Alan? Does it make sense? On all other architectures, as far as I know, smp_cond_load_acquire() is in fact RCtso. Any changes would only be needed on RISC-V. A quick grep of the kernel source (not quite up-to-date, unfortunately) turns up only the following additional usages of smp_cond_load_acquire(): It is used in kernel/smp.c for csd_lock(); I don't know what that is meant for. It is also used in the scheduler core (kernel/sched/core.c). I don't know what ordering requirements the scheduler has for it, but Peter does. There's a usage in drivers/iommu/arm-smmu-v3.c, but no comment to explain why it is needed. To tell the truth, I'm not aware of any code in the kernel that actually _needs_ RCtso ordering for locks, but Peter and Will are quite firm that it should be required. Linus would actually like locks to be RCsc, but he recognizes that this would incur a noticeable performance penalty on Power so he'll settle for RCtso. I'm not in a position to say whether smp_cond_load_acquire() should be changed, but hopefully this information will help others to make that determination. Alan > [1] https://lore.kernel.org/lkml/11b27d32-4a8a-3f84-0f25-723095ef1076@nvidia.com/ > > Dan