Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 28 Jan 2021 16:26:00 +0100
From:   Peter Zijlstra <peterz@infradead.org>
To:     Alexander Sverdlin <alexander.sverdlin@nokia.com>
Cc:     Paul Burton <paul.burton@imgtec.com>, linux-mips@vger.kernel.org,
        Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
        Will Deacon <will@kernel.org>,
        Boqun Feng <boqun.feng@gmail.com>,
        Ingo Molnar <mingo@redhat.com>, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 3/6] MIPS: Octeon: qspinlock: Flush write buffer
Message-ID: <YBLXiG3UxLNDB5xK@hirez.programming.kicks-ass.net>
References: <20210127203627.47510-1-alexander.sverdlin@nokia.com>
 <20210127203627.47510-4-alexander.sverdlin@nokia.com>
 <YBHqeXPMILg+352D@hirez.programming.kicks-ass.net>
 <f492b29b-32a4-b4d4-a5d3-0d0dd8a56b5a@nokia.com>
 <YBKha2GRFWyYp+Lz@hirez.programming.kicks-ass.net>
 <f8344f35-cdf1-ec3d-dcd0-7bfc392ef6e0@nokia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f8344f35-cdf1-ec3d-dcd0-7bfc392ef6e0@nokia.com>
Precedence: bulk

On Thu, Jan 28, 2021 at 01:13:03PM +0100, Alexander Sverdlin wrote:
> Hi!
> 
> On 28/01/2021 12:35, Peter Zijlstra wrote:
> >> My point was that original MIPS spinlocks had this write-buffer-flush and
> >> it got lost on the conversion to qspinlocks. The referenced commit just
> >> allows to see the last MIPS-specific implementation before deletion.
> > Hardware that needs a store-buffer flush after release is highly suspect
> > and needs big and explicit comments. Not vague hints.
> 
> I have a feeling that you are not going to suggest the comments for the code
> and one has to guess what is it you have in mind?

I've no insight in the specific microarch that causes this weirdness, so
it's very hard for me to suggest something here.

Find inspiration in the loongson commit.

> Do you think the proper approach would be to undelete MIPS spinlocks and
> make these broken qspinlocks a configurable option for MIPS? I don't even
> mind if they will be default option for those not interested in performance
> or latency.

qspinlock really isn't the only generic code that relies on this. I
would seriously consider doing the loongson-v3 thing, possibly also
adding that nudge_writes() thing to your smp_store_release(), you
already have it in __clear_bit_unlock().

It would then look something like:


/*
 * Octeon is special; it does not do read speculation, therefore an
 * smp_wmb() is sufficient to generate {load,store}->{store} order
 * required for RELEASE. It however has store-buffer weirdness
 * that requires an additional smp_wmb() (which is a completion barrier
 * for them) to flush the store-buffer, otherwise visibility of the
 * store can be arbitrarily delayed, also see __SYNC_loongson3_war.
 */
#define __smp_store_release(p, v) \
do { \
	compiletime_assert_atomic_type(*p); \
	__smp_wmb(); \
	WRITE_ONCE(*p, v); \
	__smp_wmb(); \
} while (0)

/*
 * Octeon also likes to retain stores, see __SYNC_loongson3_war.
 */
#define cpu_relax()	__smp_wmb();


Or something...