Received-SPF: pass (google.com: domain of linux-kernel+bounces-171626-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Precedence: bulk
MIME-Version: 1.0
References: <20240411-tso-v1-0-754f11abfbff@marcan.st> <20240411132853.GA26481@willie-the-truck>
 <87seythqct.fsf@draig.linaro.org>
In-Reply-To: <87seythqct.fsf@draig.linaro.org>
From: Ard Biesheuvel <ardb@kernel.org>
Date: Tue, 7 May 2024 16:52:30 +0200
Message-ID: <CAMj1kXFqG7D2Q_T_NXZ-y3NYOjK6d8bP8ihJTeFz8TUJ77W7tw@mail.gmail.com>
Subject: Re: [PATCH 0/4] arm64: Support the TSO memory model
To: =?UTF-8?B?QWxleCBCZW5uw6ll?= <alex.bennee@linaro.org>
Cc: Will Deacon <will@kernel.org>, Hector Martin <marcan@marcan.st>, 
	Catalin Marinas <catalin.marinas@arm.com>, Marc Zyngier <maz@kernel.org>, 
	Mark Rutland <mark.rutland@arm.com>, Zayd Qumsieh <zayd_qumsieh@apple.com>, 
	Justin Lu <ih_justin@apple.com>, Ryan Houdek <Houdek.Ryan@fex-emu.org>, 
	Mark Brown <broonie@kernel.org>, Mateusz Guzik <mjguzik@gmail.com>, 
	Anshuman Khandual <anshuman.khandual@arm.com>, Oliver Upton <oliver.upton@linux.dev>, 
	Miguel Luis <miguel.luis@oracle.com>, Joey Gouly <joey.gouly@arm.com>, 
	Christoph Paasch <cpaasch@apple.com>, Kees Cook <keescook@chromium.org>, 
	Sami Tolvanen <samitolvanen@google.com>, Baoquan He <bhe@redhat.com>, 
	Joel Granados <j.granados@samsung.com>, Dawei Li <dawei.li@shingroup.cn>, 
	Andrew Morton <akpm@linux-foundation.org>, Florent Revest <revest@chromium.org>, 
	David Hildenbrand <david@redhat.com>, Stefan Roesch <shr@devkernel.io>, Andy Chiu <andy.chiu@sifive.com>, 
	Josh Triplett <josh@joshtriplett.org>, Oleg Nesterov <oleg@redhat.com>, Helge Deller <deller@gmx.de>, 
	Zev Weiss <zev@bewilderbeest.net>, Ondrej Mosnacek <omosnace@redhat.com>, 
	Miguel Ojeda <ojeda@kernel.org>, linux-arm-kernel@lists.infradead.org, 
	linux-kernel@vger.kernel.org, Asahi Linux <asahi@lists.linux.dev>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, 7 May 2024 at 12:24, Alex Benn=C3=A9e <alex.bennee@linaro.org> wrot=
e:
>
> Will Deacon <will@kernel.org> writes:
>
> > Hi Hector,
> >
> > On Thu, Apr 11, 2024 at 09:51:19AM +0900, Hector Martin wrote:
> >> x86 CPUs implement a stricter memory modern than ARM64 (TSO). For this
> >> reason, x86 emulation on baseline ARM64 systems requires very expensiv=
e
> >> memory model emulation. Having hardware that supports this natively is
> >> therefore very attractive. Such hardware, in fact, exists. This series
> >> adds support for userspace to identify when TSO is available and
> >> toggle it on, if supported.
> >
> > I'm probably going to make myself hugely unpopular here, but I have a
> > strong objection to this patch series as it stands. I firmly believe
> > that providing a prctl() to query and toggle the memory model to/from
> > TSO is going to lead to subtle fragmentation of arm64 Linux userspace.
> >
> > It's not difficult to envisage this TSO switch being abused for native
> > arm64 applications:
> >
> >   * A program no longer crashes when TSO is enabled, so the developer
> >     just toggles TSO to meet a deadline.
> >
> >   * Some legacy x86 sources are being ported to arm64 but concurrency
> >     is hard so the developer just enables TSO to (mostly) avoid thinkin=
g
> >     about it.
> >
> >   * Some binaries in a distribution exhibit instability which goes away
> >     in TSO mode, so a taskset-like program is used to run them with TSO
> >     enabled.
>
> These all just seem like cases of engineers hiding from their very real
> problems. I don't know if its really the kernels place to avoid giving
> them the foot gun. Would it assuage your concerns at all if we set a
> taint flag so bug reports/core dumps indicated we were in a
> non-architectural memory mode?
>
> > In all these cases, we end up with native arm64 applications that will
> > either fail to load or will crash in subtle ways on CPUs without the TS=
O
> > feature. Assuming that the application cannot be fixed, a better
> > approach would be to recompile using stronger instructions (e.g.
> > LDAR/STLR) so that at least the resulting binary is portable. Now, it's
> > true that some existing CPUs are TSO by design (this is a perfectly
> > valid implementation of the arm64 memory model), but I think there's a
> > big difference between quietly providing more ordering guarantees than
> > software may be relying on and providing a mechanism to discover,
> > request and ultimately rely upon the stronger behaviour.
>
> I think the main use case here is for emulation. When we run x86-on-arm
> in QEMU we do currently insert lots of extra barrier instructions on
> every load and store. If we can probe and set a TSO mode I can assure
> you we'll do the right thing ;-)
>

Without a public specification of what TSO mode actually entails,
deciding which of those barriers can be dropped is not going to be as
straight-forward as you make it out to be.

Apple's TSO mode is vertically integrated with Rosetta, which means
that TSO mode provides whatever Rosetta needs to run x86 code
correctly, and that it could mean different things on different
generations of the micro-architecture. And whether Apple's TSO is the
same as Fujitsu's is anyone's guess afaik.

Running a game and seeing it perform better is great, but it is not
the kind of rigor we usually attempt to apply when adding support for
architectural features. Hopefully, there will be some architectural
support for this in the future, but without any spec that defines the
memory model it implements, I am not convinced we should merge this.