Received-SPF: pass (google.com: domain of linux-kernel+bounces-128532-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Tue, 2 Apr 2024 20:00:11 +0100
From: Mark Brown <broonie@kernel.org>
To: Mark Rutland <mark.rutland@arm.com>
Cc: Vineet Gupta <vineetg@rivosinc.com>, Will Deacon <will@kernel.org>,
	Andrew Waterman <andrew@sifive.com>,
	Palmer Dabbelt <palmer@rivosinc.com>,
	lkml <linux-kernel@vger.kernel.org>,
	Dave Martin <Dave.Martin@arm.com>
Subject: Re: ARM SVE ABI: kernel dropping SVE/SME state on syscalls
Message-ID: <d78b800f-669c-4fa1-9e48-f01d69f08ae0@sirena.org.uk>
References: <bc21c53f-23b9-4596-b757-fc7707f91f24@rivosinc.com>
 <ZgxKXy6r0X7Whu5U@FVFF77S0Q05N>
Precedence: bulk
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="YNznzJAAiYA0oAGE"
Content-Disposition: inline
In-Reply-To: <ZgxKXy6r0X7Whu5U@FVFF77S0Q05N>


--YNznzJAAiYA0oAGE
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Tue, Apr 02, 2024 at 07:11:43PM +0100, Mark Rutland wrote:
> On Wed, Mar 27, 2024 at 05:30:00PM -0700, Vineet Gupta wrote:

> > If I'm reading the arm64 code correctly, SVE state is unconditionally
> > (for any syscall whatsoever) dropped in following code path:

> > el0_svc
> >     fp_user_discard

Yes.

> Surely you *must* save/restore VTYPE in the signal frame? Otherwise the signal
> handler can't make any syscall whatsoever, or it's responsible for saving and
> restoring VTYPE in userspace, which doesn't seem right.

The signal handler would also be unable to use any instructions that
affect the vector state (we've got SVE based memcpy() implementations on
arm64...), and there would be issues with anything that returns to a
context other than the one where the signal was originally deliverered.

> > How do you guys handle this for SVE/SME ? One way would be to not do the
> > discard in rt_sigreturn codepath, but I don't see that - granted I'm not
> > too familiar with arch/arm64/*/**

> IIUC this works on arm64 because we'll save all the original state when we
> deliver the signal, then restore that state *after* entry to the rt_sigreturn()
> syscall.

Yes, that's it exactly.  All the process state including the SVE and SME
state is written into the signal frame along when the signal is
delivered, then on signal return we restore everything from the signal
frame overwriting any changes that happened while the signal handler was
running including the effects of entering the kernel to do the signal
return.

> > Other thing I wanted to ask is, have there been any perf implications of
> > this ABI decision: as in if this was other way around, userspace (and/or
> > compilers) could potentially leverage the fact that SVE/SME state would
> > still be valid past a syscall - and won't have to reload/resetup etc.

> I believe Mark Brown has made some changes recently to try to avoid some of
> that impact. He might be able to comment on that.

The optimisation work I've been doing has all been around avoiding
enabling traps when we discard the SVE state rather than to do with the
discarding of SVE state, the initial implementation for SVE disabled
traps very eagerly which wasn't ideal.

For arm64 our PCS specifies that the SVE specific state becomes unknown
over any standard function call, this means that any optimisation in
userspace that relied on state being preserved would need to be outside
the PCS which would be *relatively* niche though it's definitely
something that could be done especially for non-C code.  The advantage
is that we can stop saving and restoring the full SVE state for
processes that have stopped using SVE, avoiding overhead there.  The
kernel ABI basically models syscalls as C function calls, though it is
stricter in that it specifies the values after a syscall rather than
allowing them to be undefined which does impose a small performance
overhead.

Our SME ABI similarly follows the PCS for standard C functions.  Our
standard PCS for SME requires that functions be called with SME disabled
so when we exit streaming mode over syscall we're just enforcing that,
we will preserve ZA (the matrix register) since it is controlled
separately and that's broadly how the PCS works.

--YNznzJAAiYA0oAGE
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAmYMVboACgkQJNaLcl1U
h9DIEQf/VAmj4Tl4pzYhdQuF4W1Q2utS6FbdB9duKy3TDDhYEcDYBvySmAxK8Oj/
VAJT2rpBj0VwkB+azR/pGdLzvD9f6gcH2yWP1TVS/Q0bfYWwVJ1CuGpNqyXjD3eV
xu6n7uKiNuCezF+w9x4d8KEAADCA3smQdkNLVQOVp5vobxMReSGNpHwuLxChwJNG
D+P6Hes7SAXOY/qYOClaCGRHWcXmHnTu8loJhk+6xTF/Oqit0R7whmOPxb1LOlgB
Ec6HpwXefFtfIThgS46gmPR6yPxyBdLNsFCL5rwdessjz+wfe+hkfwTHsXpsVy7E
ELsWfZUYOYlzbuBOSsLVuqryVWwqBA==
=CB25
-----END PGP SIGNATURE-----

--YNznzJAAiYA0oAGE--