Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Andy Lutomirski <luto@amacapital.net>
Mime-Version: 1.0 (1.0)
Subject: Re: [RFC please help] membarrier: Rewrite sync_core_before_usermode()
Date:   Tue, 5 Jan 2021 08:20:51 -0800
Message-Id: <7BFAB97C-1949-46A3-A1E2-DFE108DC7D5E@amacapital.net>
References: <20210105132623.GB11108@willie-the-truck>
Cc:     Andy Lutomirski <luto@kernel.org>,
        Nicholas Piggin <npiggin@gmail.com>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        X86 ML <x86@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
        Benjamin Herrenschmidt <benh@kernel.crashing.org>,
        Catalin Marinas <catalin.marinas@arm.com>,
        linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
        LKML <linux-kernel@vger.kernel.org>,
        linuxppc-dev <linuxppc-dev@lists.ozlabs.org>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Paul Mackerras <paulus@samba.org>,
        stable <stable@vger.kernel.org>
In-Reply-To: <20210105132623.GB11108@willie-the-truck>
To:     Will Deacon <will@kernel.org>
Precedence: bulk


> On Jan 5, 2021, at 5:26 AM, Will Deacon <will@kernel.org> wrote:
>=20
> =EF=BB=BFHi Andy,
>=20
> Sorry for the slow reply, I was socially distanced from my keyboard.
>=20
>> On Mon, Dec 28, 2020 at 04:36:11PM -0800, Andy Lutomirski wrote:
>> On Mon, Dec 28, 2020 at 4:11 PM Nicholas Piggin <npiggin@gmail.com> wrote=
:
>>>> +static inline void membarrier_sync_core_before_usermode(void)
>>>> +{
>>>> +     /*
>>>> +      * XXX: I know basically nothing about powerpc cache management.
>>>> +      * Is this correct?
>>>> +      */
>>>> +     isync();
>>>=20
>>> This is not about memory ordering or cache management, it's about
>>> pipeline management. Powerpc's return to user mode serializes the
>>> CPU (aka the hardware thread, _not_ the core; another wrongness of
>>> the name, but AFAIKS the HW thread is what is required for
>>> membarrier). So this is wrong, powerpc needs nothing here.
>>=20
>> Fair enough.  I'm happy to defer to you on the powerpc details.  In
>> any case, this just illustrates that we need feedback from a person
>> who knows more about ARM64 than I do.
>=20
> I think we're in a very similar boat to PowerPC, fwiw. Roughly speaking:
>=20
>  1. SYNC_CORE does _not_ perform any cache management; that is the
>     responsibility of userspace, either by executing the relevant
>     maintenance instructions (arm64) or a system call (arm32). Crucially,
>     the hardware will ensure that this cache maintenance is broadcast
>     to all other CPUs.

Is this guaranteed regardless of any aliases?  That is, if I flush from one C=
PU at one VA and then execute the same physical address from another CPU at a=
 different VA, does this still work?

>=20
>  2. Even with all the cache maintenance in the world, a CPU could have
>     speculatively fetched stale instructions into its "pipeline" ahead of
>     time, and these are _not_ flushed by the broadcast maintenance instruc=
tions
>     in (1). SYNC_CORE provides a means for userspace to discard these stal=
e
>     instructions.
>=20
>  3. The context synchronization event on exception entry/exit is
>     sufficient here. The Arm ARM isn't very good at describing what it
>     does, because it's in denial about the existence of a pipeline, but
>     it does have snippets such as:
>=20
>    (s/PE/CPU/)
>       | For all types of memory:
>       | The PE might have fetched the instructions from memory at any time=

>       | since the last Context synchronization event on that PE.
>=20
>     Interestingly, the architecture recently added a control bit to remove=

>     this synchronisation from exception return, so if we set that then we'=
d
>     have a problem with SYNC_CORE and adding an ISB would be necessary (an=
d
>     we could probable then make kernel->kernel returns cheaper, but I
>     suspect we're relying on this implicit synchronisation in other places=

>     too).
>=20

Is ISB just a context synchronization event or does it do more?

On x86, it=E2=80=99s very hard to tell that MFENCE does any more than LOCK, b=
ut it=E2=80=99s much slower.  And we have LFENCE, which, as documented, does=
n=E2=80=99t appear to have any semantics at all.  (Or at least it didn=E2=80=
=99t before Spectre.)

> Are you seeing a problem in practice, or did this come up while trying to
> decipher the semantics of SYNC_CORE?

It came up while trying to understand the code and work through various bugs=
 in it.  The code was written using something approximating x86 terminology,=
 but it was definitely wrong on x86 (at least if you believe the SDM, and I h=
aven=E2=80=99t convinced any architects to say otherwise).

Thanks!

>=20
> Will