Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <8fe9cfaf-2cbc-8de6-3928-067de9113bfc@rivosinc.com>
Date:   Wed, 14 Dec 2022 12:07:03 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: RISCV Vector unit disabled by default for new task (was Re:
 [PATCH v12 17/17] riscv: prctl to enable vector commands)
Content-Language: en-US
To:     Palmer Dabbelt <palmer@dabbelt.com>, fweimer@redhat.com,
        Andrew Waterman <andrew@sifive.com>, stillson@rivosinc.com,
        Paul Walmsley <paul.walmsley@sifive.com>, anup@brainfault.org,
        atishp@atishpatra.org, guoren@kernel.org,
        Conor Dooley <conor.dooley@microchip.com>,
        greentime.hu@sifive.com, vincent.chen@sifive.com,
        andy.chiu@sifive.com, arnd@kernel.org,
        linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org,
        bjorn@kernel.org, libc-alpha@sourceware.org,
        christoph.muellner@vrull.eu, Aaron Durbin <adurbin@rivosinc.com>,
        linux@rivosinc.com
References: <mhng-975b1d7b-7b3f-4e88-804c-8b22787f9588@palmer-ri-x1c9>
 <e9cf13a9-561a-3438-00f0-41fe2631888d@rivosinc.com>
 <Y5irn63DQkwumfvW@bruce.bluespec.com>
From:   Vineet Gupta <vineetg@rivosinc.com>
In-Reply-To: <Y5irn63DQkwumfvW@bruce.bluespec.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 12/13/22 08:43, Darius Rad wrote:
> On Fri, Dec 09, 2022 at 11:42:19AM -0800, Vineet Gupta wrote:
>> But keeping the V unit disabled by default and using prctl as a gatekeeper
>> to enable it feels unnecessary and tedious.
>> Here's my reasoning below (I'm collating comments from prior msgs as well).
> Please reference the previous discussion [1] which has covered topics that
> have not been discussed recently.
>
> [1] https://lists.infradead.org/pipermail/linux-riscv/2021-September/thread.html#8361

I sure read thru that thread, and many more :-) to get context.
The highlight is we should something because AVX/AMX do so (or failed to 
do so).
But on the flip side ARM SVE is not disabling this by default.
Your other concerns seems to be potential power implications for leaving 
it on and sharing of V unit across harts (see more on that below)
Maybe leaving it on all the time will be motivation for hw designers to 
be more considerate of the idle power draw.

>
>> 2. People want the prctl gatekeeping for ability to gracefully handle memory
>> allocation failure for the extra V-state within kernel. But that is only
>> additional 4K (for typical 128 wide V regs) per task.
> But vector state scales up to as much as 256k.  Are you suggesting that
> there is no possibility that future systems would support more than
> VLEN=128?

I mentioned "typical". And below also said that memory allocation 
concerns are moot, since fork/execve failures due to failing to allocate 
would take care of those anyways.

>> If that is failing,
>> the system is not doing well anyways. Besides it is not an issue at all
>> since ENOMEM in clone/execve for the additional space should handle the
>> failure anyways. Only very sophisticated apps would downgrade from executing
>> V to Scalar code if the prctl failed.
> This seems unlikely.  As vector support does not exist in any present
> hardware, and the vector extension is only optional in the RISC-V profiles
> that include it, I would think that it is almost certain that any
> application that supports V would have a fallback path for when the V
> extension is not available.

For specialized cases sure we would expect fat binaries with IFUNC based 
dispatches (glibc mem*/str* are obvious examples). But with newer 
compilers autovec is increasing becoming default even at medium 
optimization levels such as -O2. So V code littered all over is just a 
matter of time for the profiles/variants which allow V. For less capable 
systems w/o V this is all but moot discussion since kernel itself need 
not be built with V enabled.


> Another motivation for requiring that user space request use of the vector
> extension is that the vector unit may be shared between multiple harts
> and/or have power or performance implications in the system.  By requiring
> that user space request access, it allows the system to decline that
> access, and user space can handle this gracefully.

But in this specific example won't the prctl cause more pain. So 2 
concurrent processes on 2 different harts with shared V unit. One sends 
prctl to enable and other wants to disable, what would the kernel do. 
Will it just be whoever ends up running later wins. Granted I'm not too 
familiar with how such a cross-hart sharing would work in a Vector 
instructions being part of ISA  (vs. Vector accelerator with job 
push/pull approach)

Honestly I'm sympathetic to your power concerns with keeping V enabled 
all the time. But the mechanics of implementing this prctl makes me 
wary. Assuming this is done from dynamic loader, it implies loader 
itself needs to be built with V disabled. This also leaves bunch of perf 
on table since loader does tons of of string and memory operations which 
could potentially benefit from V enabled code, granted it is not deal 
breaker.


> If we add a mechanism for user space to request access to the vector
> extension, and it turns out that it was unnecessary, the worst that has
> happened is a slight inconvenience.
>
> If we do not add such a mechanism, and later determine that it is
> necessary, we have a much greater problem.  There would be backward
> compatibility issues with the ABI, and such a mechanism could probably not
> be fully implemented at all due to the desire to support potential future
> legacy vector code.

Very true, but this in itself is not sufficient of a reason to warrant 
adding it now.

> This is a similar problem on x86.  According to some, it was handled poorly
> with AVX-512 by missing this type of mechanism, and improved with AMX [2].
> There is opportunity to learn from that experience and do things better on
> RISC-V.
>
> [2] https://lore.kernel.org/lkml/87k0ntazyn.ffs@nanos.tec.linutronix.de/

Right, but then why did ARM SVE guys chose to not take this path.

-Vineet