Received-SPF: pass (google.com: domain of linux-kernel+bounces-122235-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <bc21c53f-23b9-4596-b757-fc7707f91f24@rivosinc.com>
Date: Wed, 27 Mar 2024 17:30:00 -0700
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
From: Vineet Gupta <vineetg@rivosinc.com>
Subject: ARM SVE ABI: kernel dropping SVE/SME state on syscalls
To: Will Deacon <will@kernel.org>, Mark Rutland <mark.rutland@arm.com>
Cc: Andrew Waterman <andrew@sifive.com>, Palmer Dabbelt
 <palmer@rivosinc.com>, lkml <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Hi Will, Marc,

In the RISC-V land we are hitting an issue and need some help
understanding the SVE ABI about dropping the state on syscalls (and its
implications etc - in hindsight)

If I'm reading the arm64 code correctly, SVE state is unconditionally
(for any syscall whatsoever) dropped in following code path:

el0_svc
    fp_user_discard

The RISC-V Vector ABI mandates something similar and kernel implements
something similar.

    2023-06-29 9657e9b7d253 riscv: Discard vector state on syscalls  

However in recent testing with RISC-V vector builds we are running into
an issue when this just doesn't work.

Just for some background, RISC-V vector instructions relies on
additional state in a VTYPE register which is setup using an apriori
VSETVLI insn.
So consider the following piece of code:

   3ff80:    cc787057              vsetivli    zero,16,e8,mf2,ta,ma    
<-- sets up VTYPE
   3ff84:    44d8                    lw    a4,12(s1)
   3ff86:    449c                    lw    a5,8(s1)
   3ff88:    06f75563              bge    a4,a5,3fff2
   3ff8c:    02010087              vle8.v    v1,(sp)
   3ff90:    020980a7              vse8.v    v1,(s3)   <-- Vector store
instruction
Here's the sequence of events that's causing the issue 1. The vector
store instruction (in say bash) takes a page fault, enters kernel.
2. In PF return path, a SIGCHLD signal is pending (a bash sub-shell
which exited, likely on different cpu).
3. kernel resumes in userspace signal handler which ends up making an
rt_sigreturn syscall - and which as specified discards the V state (and
makes VTYPE reg invalid).
4. When sigreturn finally returns to original Vector store instruction,
invalid VTYPE triggers an Illegal instruction which causes a SIGILL (as
state was discarded above).

So there is no way dropping syscall state would work here.

How do you guys handle this for SVE/SME ? One way would be to not do the
discard in rt_sigreturn codepath, but I don't see that - granted I'm not
too familiar with arch/arm64/*/**

Other thing I wanted to ask is, have there been any perf implications of
this ABI decision: as in if this was other way around, userspace (and/or
compilers) could potentially leverage the fact that SVE/SME state would
still be valid past a syscall - and won't have to reload/resetup etc.

Thanks,
-Vineet