Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 02 May 2023 07:55:35 -0700 (PDT)
Subject:     Re: [PATCH 0/4] Expose the isa-string via the AT_BASE_PLATFORM aux vector
In-Reply-To: <CAAeLtUADqnwrgBjhDd+iNfwhMU6YBiMUZzeopVMAh0Jq19W9JA@mail.gmail.com>
CC:     jrtc27@jrtc27.com, bjorn@kernel.org, heiko@sntech.de,
        linux-riscv@lists.infradead.org,
        Paul Walmsley <paul.walmsley@sifive.com>,
        kito.cheng@sifive.com, Conor Dooley <conor.dooley@microchip.com>,
        matthias.bgg@gmail.com, heinrich.schuchardt@canonical.com,
        greentime.hu@sifive.com, nick.knight@sifive.com,
        christoph.muellner@vrull.eu,
        Richard Henderson <richard.henderson@linaro.org>,
        Arnd Bergmann <arnd@arndb.de>, linux-kernel@vger.kernel.org,
        heiko.stuebner@vrull.eu
From:   Palmer Dabbelt <palmer@dabbelt.com>
To:     philipp.tomsich@vrull.eu
Message-ID: <mhng-12185417-4256-4106-9bf5-2d14f66612db@palmer-ri-x1c9>
Mime-Version: 1.0 (MHng)
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On Tue, 02 May 2023 00:03:59 PDT (-0700), philipp.tomsich@vrull.eu wrote:
> On Mon, 1 May 2023 at 22:08, Jessica Clarke <jrtc27@jrtc27.com> wrote:
>>
>> On 1 May 2023, at 20:55, Björn Töpel <bjorn@kernel.org> wrote:
>> >
>> > Heiko Stuebner <heiko@sntech.de> writes:
>> >
>> >> From: Heiko Stuebner <heiko.stuebner@vrull.eu>
>> >>
>> >> The hwprobing infrastructure was merged recently [0] and contains a
>> >> mechanism to probe both extensions but also microarchitecural features
>> >> on a per-core level of detail.
>> >>
>> >> While discussing the solution internally we identified some possible issues,
>> >> tried to understand the underlying issue and come up with a solution for it.
>> >> All these deliberations overlapped with hwprobing being merged, but that
>> >> shouldn't really be an issue, as both have their usability - see below.
>> >> Also please see the "Things to consider" at the bottom!
>> >>
>> >>
>> >> Possible issues:
>> >> - very much limited to Linux
>> >> - schedulers run processes on all cores by default, so will need
>> >>  the common set of extensions in most cases
>> >
>> > ...which hwprobe has support for via the CPU mask. no?
>> >
>> >> - each new extensions requires an uapi change, requiring at least
>> >>  two pieces of software to be changed
>> >> - adding another extension requires a review process (only known
>> >>  extensions can be exposed to user-space)
>> >> - vendor extensions have special needs and therefore possibly
>> >>  don’t fit well
>> >>
>> >>
>> >> Limited to Linux:
>> >> -----------------
>> >>
>> >> The syscall and its uapi is Linux-specific and other OSes probably
>> >> will not defer to our review process and requirements just to get
>> >> new bits in. Instead most likely they'll build their own systems,
>> >> leading to fragmentation.
>> >
>> > There are a number of examples where multiple OSs have followed what
>> > Linux does, and vice versa. I'd say the opposite -- today system
>> > builders do not do their own solution, but review what's out there and
>> > mimics existing ones.
>>
>> Where they can. But if the interface is “make architecture-dependent
>> syscall X” that’s not going to fly on other OSes where syscalls are not
>> architecture-dependent. Similarly if it’s “go read auxarg Y” where Y is
>> architecture-dependent and the OS in question has
>> architecture-independent auxargs. Or the system doesn’t even have
>> auxargs. Now, at the end of the day, I couldn’t care less what Linux
>> does to communicate the information to userspace, what matters is what
>> the userspace interface is that random IFUNCs are going to make use of.
>> Something which seems to be woefully lacking from this discussion. Is
>> the interface going to just be syscall(2)? Or is there going to be some
>> higher-level interface that other OSes *do* have a hope of being able
>> to implement?
>>
>> > Personally I think this argument is moot, and will not matter much for
>> > fragmentation.
>> >
>> >> Feature on all cores:
>> >> ---------------------
>> >>
>> >> Arnd previously ([1]) commented in the discussion, that there
>> >> should not be a need for optimization towards hardware with an
>> >> asymmetric set of features. We believe so as well, especially
>> >> when talking about an interface that helps processes to identify
>> >> the optimized routines they can execute.
>> >>
>> >> Of course seeing it with this finality might not take into account
>> >> the somewhat special nature of RISC-V, but nevertheless it describes
>> >> the common case for programs very well.
>> >>
>> >> For starters the scheduler in its default behaviour, will try to use any
>> >> available core, so the standard program behaviour will always need the
>> >> intersection of available extensions over all cores.
>> >>
>> >>
>> >> Limiting program execution to specific cores will likely always be a
>> >> special use case and already requires Linux-specific syscalls to
>> >> select the set of cores.
>> >>
>> >> So while it can come in handy to get per-core information down the road
>> >> via the hwprobing interface, most programs will just want to know if
>> >> they can use a extension on just any core.
>> >>
>> >>
>> >> Review process:
>> >> ---------------
>> >>
>> >> There are so many (multi-letter-)extensions already with even more in
>> >> the pipeline. To expose all of them, each will require a review process
>> >> and uapi change that will make defining all of them slow as the
>> >> kernel needs patching after which userspace needs to sync in the new
>> >> api header.
>> >>
>> >>
>> >> Vendor-extensions:
>> >> ------------------
>> >>
>> >> Vendor extensions are special in their own right.
>> >> Userspace probably will want to know about them, but we as the kernel
>> >> don't want to care about them too much (except as errata), as they're
>> >> not part of the official RISC-V ISA spec.
>> >>
>> >> Getting vendor extensions from the dt to userspace via hwprobe would
>> >> require coordination efforts and as vendors have the tendency to invent
>> >> things during their development process before trying to submit changes
>> >> upstream this likely would result in conflicts with assigned ids down
>> >> the road. Which in turn then may create compatibility-issues with
>> >> userspace builds built on top of the mainline kernel or a pre-
>> >> existing vendor kernel.
>> >>
>> >> The special case also is that vendor A could in theory implement an
>> >> extension from vendor B. So this would require to actually assign
>> >> separate hwprobe keys to vendors (key for xthead extensions, key for
>> >> xventana extensions, etc). This in turn would require vendors to
>> >> come to the mainline kernel to get assigned a key (which in reality
>> >> probably won't happen), which would then make the kernel community
>> >> sort of an id authority.
>> >>
>> >>
>> >>
>> >>
>> >> To address these, the attached patch series adds a second interface
>> >> for the common case and "just" exposes the isa-string via the
>> >> AT_BASE_PLATFORM aux vector.
>> >
>> > *A second interface* introduced the second hwprobe landed. Really?
>> > Start a discussion on how to extend hwprobe instead.
>>
>> I’ve been trying to push for something other than this for months, but
>> RVI took no interest in dealing with it until it got closer to these
>> landing, at which point finally some action was taken. But even then,
>> heels were dragged, and it took hwprobe being landed to force them to
>> finally publish things. But of course too late, so now the ecosystem is
>> forever screwed thanks to their inaction.
>
> I am similarly frustrated as I had been asking for a universally
> acceptable solution since August 2021 (and Kito gave a presentation on
> the issue at LPC21) that was meant to avoid system calls; however, the
> design of hwprobe has happened outside of RVI and without getting RVI
> involved to drive coordination between members.

Linux development happens on LKML, not at RVI.  If you want to 
participate you need to do so on the mailing lists.  We have been 
through this many times over the years.

> The bottleneck (in cases like this one) at RVI is that it is
> volunteer-driven and dependent on the buy-in of its membership: there
> are no technical resources except for us company delegates.  If

Though apparently you have resources to start a pointless flame war 
after everyone else has agreed on a solution.

> members decide to work a gap without involving RVI, then the chances
> of fragmentation are high.  Nothing we can do about that, as RVI is
> not a traffic cop.

This isn't the first time you've complained you have no resources to do 
the work, but then gotten mad when someone else does.  That's not 
constructive behavior, please stop.

>> All I wanted was some simple extension string -> version number function
>> as a standardised userspace interface... because at the end of the day
>> people just want to know “can I use extension Y?”, possibly with a
>> minimum version. But maybe there’s some hope that Linux libcs will
>> translate such queries into poking at this hwprobe thing. Though god
>> knows what they’re going to do about vendor extensions, especially if
>> those vendor extensions only get supported in vendor forks of the
>> kernel (who’s allocating their encodings? Xvendorfoo exists to
>> namespace them and give vendors control...).
>
> The support for vendor extensions without a central registry remains
> the strongest reason for a different interface, as RISC-V has the
> flexibility to add vendor extensions as one of its strongest selling
> points.
>
> It is a pity that the current interface was designed without involving
> RVI (and that I had to ask my team to put together a patch set for

We talked about this at Plumbers, you and Mark (the RISC-V CTO) were both 
there.  Additionally this has been discussion on the mailing list for 
the better part of a year, including proposals similar to this one.  
Pretending nobody from RVI was involved is just blatantly false.

This isn't the first time we've had the RVI leadership making false 
statements in order to drum up controversy on the mailing lists.  Please 
stop, we have real work to do and this just gets in everyone's way.

> further discussion, given that none of the other major vendors in RVI
> stepped forward).  I guarantee that plenty of reviewers would have
> highlighted that a central registry (even if it is just a kernel
> header) should be avoided.

This has been reviewed by engineers from multiple vendors on LKML, 
including Hekio who sent this patch set.

> So what is the best way to support vendor-defined/vendor-specific
> extensions without a central registry?

Stop with the pointless flame wars.