Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Message-ID: <8a9952bf-67ff-4ab0-9e44-e2c2f2a68076@rivosinc.com>
Date:   Thu, 7 Dec 2023 10:23:20 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [External] [RFC PATCH 0/3] riscv: add support for SBI Supervisor
 Software Events
To:     Xu Lu <luxu.kernel@bytedance.com>
Cc:     Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        Atish Patra <atishp@atishpatra.org>,
        Anup Patel <apatel@ventanamicro.com>,
        linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org,
        Himanshu Chauhan <hchauhan@ventanamicro.com>
References: <20231026143122.279437-1-cleger@rivosinc.com>
 <CAPYmKFsFLQnhXRPXpcoNfO-kEyjYLXD2Hm-F-O=Yxe1WJoSL9w@mail.gmail.com>
Content-Language: en-US
From:   =?UTF-8?B?Q2zDqW1lbnQgTMOpZ2Vy?= <cleger@rivosinc.com>
In-Reply-To: <CAPYmKFsFLQnhXRPXpcoNfO-kEyjYLXD2Hm-F-O=Yxe1WJoSL9w@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 07/12/2023 10:09, Xu Lu wrote:
> Pardon. It seems that the code in opensbi[1] is not complete for PMU NMI now.
> For example, the pmu ovf irq is still delegated to supervisor mode and
> thus can not really play a role as NMI. And neither the kernel nor
> opensbi will inject a pmu event.
> 
> To complete the work, we think maybe 'enable_cb' and 'disable_cb'
> functions can be supplied for sbi_sse_cb_ops.
> When sbi_sse_enable is called to enable pmu event, the enable_cb will
> be called to revoke the delegation of pmu ovf irq and enable this irq
> in CSR_MIE.

Hi Xu,

Indeed, this part has been developed but was left out for the RFC. But
your understanding is correct.

> When pmu ovf irq occurs, kernel traps into opensbi and opensbi injects
> the event via sbi_sse_inject_event and eret back to kernel.
> 
> Please point it out if we have any misunderstanding.
> 
> By the way, how is SSE going now? We will appreciate it very much if
> we can participate in some development work in kernel or opensbi and
> be of any help.

The development is almost complete, Anup/Himanchu will send a new
revision of the spec addressing various comments and I'll resend the RFC
following that spec update.

Regards,

Clément

> 
> Regards!
> 
> Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [1]
> 
> On Thu, Oct 26, 2023 at 10:31 PM Clément Léger <cleger@rivosinc.com> wrote:
>>
>> The SBI Supervisor Software Events (SSE) extensions provides a mechanism
>> to inject software events from an SBI implementation to supervisor
>> software such that it preempts all other supervisor level traps and
>> interrupts [1].
>>
>> Various events are defined and can be send asynchronously to supervisor
>> software (RAS, PMU, DEBUG, Asynchronous page fault) from SBI as well
>> as platform specific events. Events can be either local (per-hart) or
>> global. Events can be nested on top of each other based on priority and
>> can interrupt the kernel at any time.
>>
>> First patch adds the SSE definitions. Second one adds support for SSE
>> itself. Implementation is split between arch specific code and generic
>> part (similarly to what is done for ARM SDEI). Finally, the last patch
>> add support fro SSE event in the SBI PMU driver. If the SSE event is
>> available from the SBI then, it will be used instead of the normal
>> interrupt.
>>
>> Amongst the specific points that needs to be handle is the interruption
>> at any point of the kernel execution and more specifically during
>> exception handling. Due to the fact that the exception entry
>> implementation uses the SCRATCH CSR as both the current task struct and
>> as the temporary register to switch the stack and save register, it is
>> difficult to reliably get the current task struct if we get interrupted
>> at this specific moment. A fixup-like mechanism allows to mark the
>> location of the current task struct depending on the entry level
>> (user/kernel) and the location. This is then used in the SSE assembly to
>> determine where is located the current task_struct.
>>
>> Contrary to pseudo NMI [2], SSE does not modifies the way interrupts are
>> handled and does not adds any overhead to existing code. Moreover, it
>> provides "true" NMI-like interrupts which can interrupt the kernel at
>> any time (even in exception handling). This is particularly crucial for
>> RAS errors which needs to be handled as fast as possible to avoid any
>> fault propagation. Additionally, SSE event handling is faster that the
>> standard IRQ handling path with almost half executed instruction (700 vs
>> 1590). Some complementary tests/perf measurements will be done.
>>
>> For testing purpose, one can use the provided SBI implementation at [3].
>> This series also needs patch [4] to fix a bug in the PMU driver.
>>
>> Link: https://lists.riscv.org/g/tech-prs/message/515 [1]
>> Link: https://lore.kernel.org/lkml/20231023082911.23242-10-luxu.kernel@bytedance.com/T/ [2]
>> Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [3]
>> Link: https://lore.kernel.org/linux-arm-kernel/20231026084010.11888-1-alexghiti@rivosinc.com/ [4]
>>
>> ---
>>
>> Clément Léger (3):
>>   riscv: add SBI SSE extension definitions
>>   riscv: add support for SBI Supervisor Software Events extension
>>   perf: RISC-V: add support for SSE event
>>
>>  arch/riscv/include/asm/asm-prototypes.h |   5 +
>>  arch/riscv/include/asm/sbi.h            |  40 ++
>>  arch/riscv/include/asm/sse.h            |  94 +++++
>>  arch/riscv/kernel/Makefile              |   1 +
>>  arch/riscv/kernel/asm-offsets.c         |  17 +
>>  arch/riscv/kernel/entry.S               | 156 ++++++++
>>  arch/riscv/kernel/sbi.c                 |   4 +
>>  arch/riscv/kernel/sse.c                 |  97 +++++
>>  arch/riscv/kernel/stacktrace.c          |  13 +
>>  arch/riscv/kernel/vmlinux.lds.S         |   6 +
>>  drivers/firmware/Kconfig                |  10 +
>>  drivers/firmware/Makefile               |   1 +
>>  drivers/firmware/riscv_sse.c            | 496 ++++++++++++++++++++++++
>>  drivers/perf/riscv_pmu_sbi.c            |  51 ++-
>>  include/linux/riscv_sse.h               |  56 +++
>>  15 files changed, 1038 insertions(+), 9 deletions(-)
>>  create mode 100644 arch/riscv/include/asm/sse.h
>>  create mode 100644 arch/riscv/kernel/sse.c
>>  create mode 100644 drivers/firmware/riscv_sse.c
>>  create mode 100644 include/linux/riscv_sse.h
>>
>> --
>> 2.42.0
>>