Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: i56a14606:Fastmail
User-Agent: Cyrus-JMAP/3.7.0-alpha0-935-ge4ccd4c47b-fm-20220914.001-ge4ccd4c4
Mime-Version: 1.0
Message-Id: <b7379459-989f-446d-9d1d-b381a8550de1@www.fastmail.com>
In-Reply-To: <PH0PR12MB548166865BF446C7F232DCE1DC519@PH0PR12MB5481.namprd12.prod.outlook.com>
References: <20220915050106.650813-1-parav@nvidia.com>
 <96457b14-e196-4f29-be9a-7fa25ac805d9@www.fastmail.com>
 <PH0PR12MB5481192DB7B5C6E19683D514DC499@PH0PR12MB5481.namprd12.prod.outlook.com>
 <a8ee97f5-b92f-47a6-9b50-197974738ff7@www.fastmail.com>
 <PH0PR12MB548113D13F9E9CE4D5206514DC499@PH0PR12MB5481.namprd12.prod.outlook.com>
 <9ae25893-f19f-4186-a19a-7fc55d9295ed@www.fastmail.com>
 <PH0PR12MB548166865BF446C7F232DCE1DC519@PH0PR12MB5481.namprd12.prod.outlook.com>
Date:   Fri, 23 Sep 2022 20:40:12 +0200
From:   "Arnd Bergmann" <arnd@arndb.de>
To:     "Parav Pandit" <parav@nvidia.com>,
        "stern@rowland.harvard.edu" <stern@rowland.harvard.edu>,
        "parri.andrea@gmail.com" <parri.andrea@gmail.com>,
        "Will Deacon" <will@kernel.org>,
        "Peter Zijlstra" <peterz@infradead.org>,
        "boqun.feng@gmail.com" <boqun.feng@gmail.com>,
        "Nicholas Piggin" <npiggin@gmail.com>,
        "dhowells@redhat.com" <dhowells@redhat.com>,
        "j.alglave@ucl.ac.uk" <j.alglave@ucl.ac.uk>,
        "luc.maranget@inria.fr" <luc.maranget@inria.fr>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        "akiyks@gmail.com" <akiyks@gmail.com>,
        "Dan Lustig" <dlustig@nvidia.com>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>,
        "Jonathan Corbet" <corbet@lwn.net>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Linux-Arch <linux-arch@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>
Subject: Re: [PATCH] locking/memory-barriers.txt: Improve documentation for writel()
 usage
Content-Type: text/plain
Precedence: bulk

On Fri, Sep 23, 2022, at 5:55 PM, Parav Pandit wrote:
>> From: Arnd Bergmann <arnd@arndb.de>
>> Sent: Friday, September 16, 2022 12:09 AM

>> > And I sort of see above pattern in two drivers, and it is not good.
>> > It ends up doing dsb(st) on arm64, while needed barrier is only
>> > dmb(oshst).
>> >
>> > So to fix those two drivers, it is better to first avoid wmb()
>> > documentation reference when referring to writel().
>> 
>> Yes, this suggestion is correct. On x86 and a few others, I think it's even
>> worse when wmb() is an expensive barrier, while writel() is the same as
>> writel_relaxed() and the barrier is implied by the MMIO access.
>> 
>> It might help to spell this out and say that writel() is always preferred over
>> wmb()+writel_relaxed().
>> 
> True.
>
>> Site note: there are several other problems with wmb()+__raw_writel(),
>> which on many architectures does not guarantee any atomicity of the access
>> (a word store could get split into four byte stores), breaks endianess
>> assumptions and may still not provide the correct barrier semantics.
>>
> Hmm. So far didn't observe this on arm64, x86_64, ppc64 yet.
> May be because the address is aligned to 8 bytes, we don't see the byte stores?

It's complicated. On some architectures (but not the ones you list),
__raw_writel() is a pointer dereference, so the compiler is allowed
to use whichever instruction it wants. Depending on the CPU it
is building for, gcc can decide to split up those stores when it
sees a pointer that was assigned from pointer with lesser alignment
(which is undefined behavior in C). If the pointer is actually
unaligned but gcc does not see this, then powerpc and arm will
trigger an alignment exception for an MMIO location even on CPUs
that can work with unaligned data on normal RAM.

> mlx5_write64() variant to use writeX() and avoid wmb() post the 
> documentation update is good start.

Ok, fair enough, as long as the loop function is not timing
critical itself.

    Arnd