From:   Thomas Gleixner <tglx@linutronix.de>
To:     Andy Lutomirski <luto@kernel.org>,
        Asit K Mallick <asit.k.mallick@intel.com>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "Brown, Len" <len.brown@intel.com>,
        "Peter Zijlstra (Intel)" <peterz@infradead.org>,
        the arch/x86 maintainers <x86@kernel.org>,
        Borislav Petkov <bp@alien8.de>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "Bae, Chang Seok" <chang.seok.bae@intel.com>,
        Arjan van de Ven <arjan@linux.intel.com>
Cc:     Andrew Cooper <andrew.cooper3@citrix.com>
Subject: Re: Revisiting XFD-based AMX and heterogenous systems
In-Reply-To: <0f03a9e2-4611-4b5c-962f-93a7e1d2bde8@www.fastmail.com>
References: <0f03a9e2-4611-4b5c-962f-93a7e1d2bde8@www.fastmail.com>
Date:   Mon, 15 Nov 2021 22:12:52 +0100
Message-ID: <8735nx6s2z.ffs@tglx>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

Andy,

On Mon, Nov 15 2021 at 11:59, Andy Lutomirski wrote:
> So I suggest that we go back and switch to the XCR0 model.  Tasks will
> start out with AMX clear in XCR0.  If they want AMX, they issue a
> prctl asking for AMX, AMX gets set in XCR0, and the tasks need to be
> able to tolerate the XCR0 change.

We can do that, but that still want's XFD for avoiding allocating large
buffers for all tasks in such a process which never use that feature.

Aside of that as we all know context switching XCR0 sucks.

> Then, if Intel ever wants to expose the full Alder Lake physical
> capabilities and support efficiency cores and AVX-512 on the same
> boot, we can have a mode in which tasks start with AVX-512 clear in
> XCR0 and can opt in with prctl.  This will require HPC-like apps to be
> recompiled or run with a special wrapper bit will otherwise expose the
> full HW capabilities. (Of course this assumes that Intel sets up MSRs
> or ucode or whatever to support this.)

If software needs to be recompiled or wrapped anyway then Intel can just
provide XFD support for AVX512 if it wants to expose this at runtime on
those CPUs.

As that needs to be implemented for AMX anyway the logical consequence
for user space is:

    available = arch_prctl(ARCH_GET_XCOMP_SUPP);        // Same as XCR0
    permitted = arch_prctl(ARCH_GET_XCOMP_PERM);        // XRC0 & permission bits

and work from there. If done with future XFD support for other features
than AMX in mind (even retroactively added for AVX512) then this should
be straight forward to adjust.

For the kernel adding XFD for AVX512 even conditionally based on a CPUID
bit is pretty straight forward now. It needs a small change to the way
how we distinguish XFD based and unconditional features, but that's
trivial effort compared to going for XCR0 switching with all its
downsides.

Thanks,

        tglx