Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Sat, 20 Apr 2019 10:26:15 +1000
From:   Nicholas Piggin <npiggin@gmail.com>
Subject: Re: [PATCH] Documentation: atomic_t.txt: Explain ordering provided by
 smp_mb__{before,after}_atomic()
To:     paulmck@linux.ibm.com, Peter Zijlstra <peterz@infradead.org>
Cc:     LKMM Maintainers -- Akira Yokosawa <akiyks@gmail.com>,
        Andrea Parri <andrea.parri@amarulasolutions.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        David Howells <dhowells@redhat.com>,
        Daniel Lustig <dlustig@nvidia.com>,
        Jade Alglave <j.alglave@ucl.ac.uk>,
        Kernel development list <linux-kernel@vger.kernel.org>,
        Luc Maranget <luc.maranget@inria.fr>,
        Alan Stern <stern@rowland.harvard.edu>,
        Will Deacon <will.deacon@arm.com>
References: <Pine.LNX.4.44L0.1904191312200.1406-100000@iolanthe.rowland.org>
        <20190419180017.GP4038@hirez.programming.kicks-ass.net>
        <20190419182620.GF14111@linux.ibm.com>
In-Reply-To: <20190419182620.GF14111@linux.ibm.com>
MIME-Version: 1.0
User-Agent: astroid/0.14.0 (https://github.com/astroidmail/astroid)
Message-Id: <1555719429.t9n8gkf70y.astroid@bobo.none>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Paul E. McKenney's on April 20, 2019 4:26 am:
> On Fri, Apr 19, 2019 at 08:00:17PM +0200, Peter Zijlstra wrote:
>> On Fri, Apr 19, 2019 at 01:21:45PM -0400, Alan Stern wrote:
>> > Index: usb-devel/Documentation/atomic_t.txt
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> > --- usb-devel.orig/Documentation/atomic_t.txt
>> > +++ usb-devel/Documentation/atomic_t.txt
>> > @@ -171,7 +171,10 @@ The barriers:
>> >    smp_mb__{before,after}_atomic()
>> > =20
>> >  only apply to the RMW ops and can be used to augment/upgrade the orde=
ring
>> > -inherent to the used atomic op. These barriers provide a full smp_mb(=
).
>> > +inherent to the used atomic op. Unlike normal smp_mb() barriers, they=
 order
>> > +only the RMW op itself against the instructions preceding the
>> > +smp_mb__before_atomic() or following the smp_mb__after_atomic(); they=
 do
>> > +not order instructions on the other side of the RMW op at all.
>>=20
>> Now it is I who is confused; what?
>>=20
>> 	x =3D 1;
>> 	smp_mb__before_atomic();
>> 	atomic_add(1, &a);
>> 	y =3D 1;
>>=20
>> the stores to both x and y will be ordered as if an smp_mb() where
>> there. There is no order between a and y otoh.
>=20
> Let's look at x86.  And a slightly different example:
>=20
> 	x =3D 1;
> 	smp_mb__before_atomic();
> 	atomic_add(1, &a);
> 	r1 =3D y;
>=20
> The atomic_add() asm does not have the "memory" constraint, which is
> completely legitimate because atomic_add() does not return a value,
> and thus guarantees no ordering.  The compiler is therefore within
> its rights to transform the code into the following:
>=20
> 	x =3D 1;
> 	smp_mb__before_atomic();
> 	r1 =3D y;
> 	atomic_add(1, &a);
>=20
> But x86's smp_mb__before_atomic() is just a compiler barrier, and
> x86 is further allowed to reorder prior stores with later loads.
> The CPU can therefore execute this code as follows:
>=20
> 	r1 =3D y;
> 	x =3D 1;
> 	smp_mb__before_atomic();
> 	atomic_add(1, &a);
>=20
> So in general, the ordering is guaranteed only to the atomic itself,
> not to accesses on the other side of the atomic.

That's interesting. I don't think that's what all our code expects.
I had the same idea as Peter.

IIRC the primitive was originally introduced exactly so x86 would not
need to have the unnecessary hardware barrier with sequences like

  smp_mb();
  ...
  atomic_inc(&v);

The "new" semantics are a bit subtle. One option might be just to
replace it entirely with atomic_xxx_mb() ?

Thanks,
Nick

=