Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: Candidate Linux ABI for Intel AMX and hypothetical new related features
From:   Andy Lutomirski <luto@amacapital.net>
In-Reply-To: <CAJvTdK=QbPRtZ9zPgu8c9tqxOtaG3apo7u4BBTXP0--qVWA5ig@mail.gmail.com>
Cc:     Greg KH <gregkh@linuxfoundation.org>,
        Andy Lutomirski <luto@kernel.org>,
        "Bae, Chang Seok" <chang.seok.bae@intel.com>,
        Dave Hansen <dave.hansen@intel.com>, X86 ML <x86@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        libc-alpha <libc-alpha@sourceware.org>,
        Florian Weimer <fweimer@redhat.com>,
        Rich Felker <dalias@libc.org>, Kyle Huey <me@kylehuey.com>,
        Keno Fischer <keno@juliacomputing.com>,
        Linux API <linux-api@vger.kernel.org>
Date:   Mon, 29 Mar 2021 11:16:52 -0700
Message-Id: <5F98327E-8EC4-455E-B9E1-74D2F13578C5@amacapital.net>
References: <CAJvTdK=QbPRtZ9zPgu8c9tqxOtaG3apo7u4BBTXP0--qVWA5ig@mail.gmail.com>
To:     Len Brown <lenb@kernel.org>
Precedence: bulk


> On Mar 29, 2021, at 8:47 AM, Len Brown <lenb@kernel.org> wrote:
>=20
> =EF=BB=BFOn Sat, Mar 27, 2021 at 5:58 AM Greg KH <gregkh@linuxfoundation.o=
rg> wrote:
>>> On Fri, Mar 26, 2021 at 11:39:18PM -0400, Len Brown wrote:
>>> Hi Andy,
>>> Say a mainline links with a math library that uses AMX without the
>>> knowledge of the mainline.
>=20
> sorry for the confusion.
>=20
> mainline =3D main().
>=20
> ie. the part of the program written by you, and not the library you linked=
 with.
>=20
> In particular, the library may use instructions that main() doesn't know e=
xist.

If we pretend for a bit that AMX were a separate device instead of a part of=
 the CPU, this would be a no brainer: something would be responsible for ope=
ning a device node or otherwise requesting access to the device.=20

Real AMX isn=E2=80=99t so different. Programs acquire access either by sysca=
ll or by a fault, they use it, and (hopefully) they release it again using T=
ILERELEASE. The only thing special about it is that, supposedly, acquiring a=
nd releasing access (at least after the first time) is quite fast.  But hold=
ing access is *not* free =E2=80=94 despite all your assertions to the contra=
ry, the kernel *will* correctly context switch it to avoid blowing up power c=
onsumption, and this will have overhead.

We=E2=80=99ve seen the pattern of programs thinking that, just because somet=
hing is a CPU insn, it=E2=80=99s free and no thought is needed before using i=
t. This happened with AVX and AVX512, and it will happen again with AMX. We *=
still* have a significant performance regression in the kernel due to screwi=
ng up the AVX state machine, and the only way I know about any of the detail=
s is that I wrote silly test programs to try to reverse engineer the nonsens=
ical behavior of the CPUs.

I might believe that Intel has figured out how to make a well behaved XSTATE=
 feature after Intel demonstrates at least once that it=E2=80=99s possible. =
 That means full documentation of all the weird issues, no new special cases=
, and the feature actually making sense in the context of XSTATE.  This has n=
ot happened.  Let=E2=80=99s list all of them:

- SSE.  Look for all the MXCSR special cases in the pseudocode and tell me w=
ith a straight face that this one works sensibly.

- AVX.  Also has special cases in the pseudocode. And has transition issues t=
hat are still problems and still not fully documented. L

- AVX2.  Horrible undocumented performance issues.  Otherwise maybe okay?

- MPX: maybe the best example, but the compat mode part got flubbed and it=E2=
=80=99s MPX.

- PKRU: Should never have been in XSTATE. (Also, having WRPKRU in the ISA wa=
s a major mistake, now unfixable, that seriously limits the usefulness of th=
e whole feature.  I suppose Intel could release PKRU2 with a better ISA and d=
eprecate the original PKRU, but I=E2=80=99m not holding my breath.)

- AVX512: Yet more uarch-dependent horrible performance issues, and Intel ha=
s still not responded about documentation.  The web is full of people specul=
ating differently about when, exactly, using AVX512 breaks performance. This=
 is NAKked in kernel until docs arrive. Also, it broke old user programs.  I=
f we had noticed a few years ago, AVX512 enablement would have been reverted=
.

- AMX: This mess.

The current system of automatic user enablement does not work. We need somet=
hing better.=