Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 05 May 2022 08:29:08 +0100
Message-ID: <874k24igjf.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     "Guilherme G. Piccoli" <gpiccoli@igalia.com>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        will Deacon <will@kernel.org>,
        "Michael Kelley (LINUX)" <mikelley@microsoft.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>, mark.rutland@arm.com,
        Russell King <linux@armlinux.org.uk>,
        Ard Biesheuvel <ardb@kernel.org>, broonie@kernel.org,
        "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        "linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>
Subject: Re: Should arm64 have a custom crash shutdown handler?
In-Reply-To: <427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com>
References: <427a8277-49f0-4317-d6c3-4a15d7070e55@igalia.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Wed, 04 May 2022 21:00:42 +0100,
"Guilherme G. Piccoli" <gpiccoli@igalia.com> wrote:
> 
> Hi folks, this email is to ask feedback / trigger a discussion about the
> concept of custom crash shutdown handler, that is "missing" in arm64
> while it's present in many architectures [mips, powerpc, x86, sh (!)].
> 
> Currently, when we kexec in arm64, the function machine_crash_shutdown()
> is called as a handler to disable CPUs and (potentially) do extra
> quiesce work. In the aforementioned architectures, there's a way to
> override this function, if for example an hypervisor wish to have its
> guests running their own custom shutdown machinery.
>
> For powerpc/mips, the approach is a generic shutdown function that might
> call other handler-registered functions, whereas x86/sh relies in the
> "machine_ops" structure, having the crash shutdown as a callback in such
> struct.
> 
> The usage for that is very broad, but heavy users are hypervisors like
> Hyper-V / KVM (CCed Michael and Vitaly here for this reason). The
> discussion about the need for that in arm64 is from another thread [0],
> so before start implementing/playing with that, I'd like to ask ARM64
> community if there is any feedback and in case it's positive, what is
> the best implementation strategy (struct callback vs. handler call), etc.
> 
> I've CCed ARM64/ARM32 maintainers plus extra people I found as really
> involved with ARM architecture - sorry if I added people I shouldn't or
> if I forgot somebody (though the ARM mailing-list is CC).

I have the feeling that you are conflating two different things here:

(1) general shutdown/reboot, whether this because of a crash or not

(2) kexec, for which the whole point is that it is possible to handle
*everything* from within the kernel

On arm64:

(1) is already abstracted via PSCI. The hypervisor can do whatever it
wants there (KVM, not needing anything, just forwards this to
userspace for fun and profit -- if something has to be done, the VMM
is the right spot). I expect other hypervisors to do the same thing
(and that's what the architecture expects anyway).

(2) must, by definition, fit into the architectural envelope. If you
need help from another entity in the system to be able to kexec,
something is broken, because the hypervisor doesn't implement the
architecture correctly (and frankly, we really don't need much to be
able to kexec).

Not having any 'machine_ops' indirection was a conscious decision on
arm64, if only to avoid the nightmare that 32bit was at a time with
every single platform doing their own stuff. Introducing them would
not be an improvement, but simply the admission that hypervisors are
simply too broken for words. And I don't buy the "but x86 has it!"
argument. x86 is a nightmare of PV mess that we can happily ignore,
because we don't do PV for core operations at all.

If something has to be done to quiesce the system, it probably is
related to the system topology, and must be linked to it. We already
have these requirements in order to correctly stop ongoing DMA, shut
down IOMMUs, and other similar stuff. What other requirements does
your favourite hypervisor have?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.