Received-SPF: pass (google.com: domain of linux-kernel+bounces-137558-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Tue, 09 Apr 2024 12:43:51 -0700 (PDT)
Subject:     Re: [PATCH] riscv: Do not save the scratch CSR during suspend
In-Reply-To: <3010bb57-c13f-494a-9e8c-0ae6393b1eee@sifive.com>
CC: jeeheng.sia@starfivetech.com, linux-riscv@lists.infradead.org,
  aou@eecs.berkeley.edu, ajones@ventanamicro.com, Conor Dooley <conor.dooley@microchip.com>,
  leyfoon.tan@starfivetech.com, Paul Walmsley <paul.walmsley@sifive.com>, pavel@ucw.cz, rafael@kernel.org,
  linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org
From: Palmer Dabbelt <palmer@dabbelt.com>
To: samuel.holland@sifive.com
Message-ID: <mhng-fa663ef4-be7b-455b-b467-f4d331a269ea@palmer-ri-x1c9a>
Precedence: bulk
Mime-Version: 1.0 (MHng)
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On Thu, 21 Mar 2024 16:51:31 PDT (-0700), samuel.holland@sifive.com wrote:
> On 2024-03-14 11:55 PM, JeeHeng Sia wrote:
>>
>>
>>> -----Original Message-----
>>> From: Samuel Holland <samuel.holland@sifive.com>
>>> Sent: Wednesday, March 13, 2024 3:57 AM
>>> To: Palmer Dabbelt <palmer@dabbelt.com>; linux-riscv@lists.infradead.org
>>> Cc: Samuel Holland <samuel.holland@sifive.com>; Albert Ou <aou@eecs.berkeley.edu>; Andrew Jones <ajones@ventanamicro.com>;
>>> Conor Dooley <conor.dooley@microchip.com>; Leyfoon Tan <leyfoon.tan@starfivetech.com>; Paul Walmsley
>>> <paul.walmsley@sifive.com>; Pavel Machek <pavel@ucw.cz>; Rafael J. Wysocki <rafael@kernel.org>; JeeHeng Sia
>>> <jeeheng.sia@starfivetech.com>; linux-kernel@vger.kernel.org; linux-pm@vger.kernel.org
>>> Subject: [PATCH] riscv: Do not save the scratch CSR during suspend
>>>
>>> While the processor is executing kernel code, the value of the scratch
>>> CSR is always zero, so there is no need to save the value. Continue to
>>> write the CSR during the resume flow, so we do not rely on firmware to
>>> initialize it.
>>>
>>> Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
>>> ---
>>>
>>>  arch/riscv/include/asm/suspend.h | 1 -
>>>  arch/riscv/kernel/suspend.c      | 3 +--
>>>  2 files changed, 1 insertion(+), 3 deletions(-)
>>>
>>> diff --git a/arch/riscv/include/asm/suspend.h b/arch/riscv/include/asm/suspend.h
>>> index 491296a335d0..6569eefacf38 100644
>>> --- a/arch/riscv/include/asm/suspend.h
>>> +++ b/arch/riscv/include/asm/suspend.h
>>> @@ -13,7 +13,6 @@ struct suspend_context {
>>>  	/* Saved and restored by low-level functions */
>>>  	struct pt_regs regs;
>>>  	/* Saved and restored by high-level functions */
>>> -	unsigned long scratch;
>>>  	unsigned long envcfg;
>>>  	unsigned long tvec;
>>>  	unsigned long ie;
>>> diff --git a/arch/riscv/kernel/suspend.c b/arch/riscv/kernel/suspend.c
>>> index 299795341e8a..3d306d8a253d 100644
>>> --- a/arch/riscv/kernel/suspend.c
>>> +++ b/arch/riscv/kernel/suspend.c
>>> @@ -14,7 +14,6 @@
>>>
>>>  void suspend_save_csrs(struct suspend_context *context)
>>>  {
>>> -	context->scratch = csr_read(CSR_SCRATCH);
>>>  	if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_XLINUXENVCFG))
>>>  		context->envcfg = csr_read(CSR_ENVCFG);
>>>  	context->tvec = csr_read(CSR_TVEC);
>>> @@ -37,7 +36,7 @@ void suspend_save_csrs(struct suspend_context *context)
>>>
>>>  void suspend_restore_csrs(struct suspend_context *context)
>>>  {
>>> -	csr_write(CSR_SCRATCH, context->scratch);
>>> +	csr_write(CSR_SCRATCH, 0);
>> If the register is always zero, do we need to explicitly write zero to the register during resume?
>
> The register contains zero while executing in the kernel. While executing in
> userspace, the value is nonzero. The value is checked at the beginning of
> handle_exception(). We must ensure the value is zero before enabling interrupts,
> or we might incorrectly think the interrupt was entered from userspace.
>
> We don't know what the value will be when the hart comes out of non-retentive
> suspend. Per the SBI HSM specification, Table 6: "All other registers remain in
> an undefined state."

We're also not setting it at all in `.macro suspend_restore_csrs`, which 
I think is just a bug?

That said, I'm kind of seeing bugs everywhere I look in this now -- what 
about all the other registers we can poke, like timers/counters or the 
V/F state (or anything from M-mode, though maybe that's just someone 
else's problem)?

I also think we'd break on medlow kernels, as a bunch of this relies on 
medany-as-PIC for the SATP-off transition.

Maybe I'm going crazy here, though...

> Regards,
> Samuel