Received-SPF: pass (google.com: domain of linux-kernel+bounces-15881-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
MIME-Version: 1.0
References: <20231222193607.15474-1-graf@amazon.com> <20231222195144.24532-1-graf@amazon.com>
 <20231222195144.24532-2-graf@amazon.com>
In-Reply-To: <20231222195144.24532-2-graf@amazon.com>
From: Rob Herring <robh+dt@kernel.org>
Date: Wed, 3 Jan 2024 11:48:58 -0700
Message-ID: <CAL_JsqJSYgq7BJnSxwKebEX8e9tL-FG74rT+eLJ4e32kKZC30g@mail.gmail.com>
Subject: Re: [PATCH v2 07/17] kexec: Add documentation for KHO
To: Alexander Graf <graf@amazon.com>
Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, 
	linux-mm@kvack.org, devicetree@vger.kernel.org, 
	linux-arm-kernel@lists.infradead.org, kexec@lists.infradead.org, 
	linux-doc@vger.kernel.org, x86@kernel.org, 
	Eric Biederman <ebiederm@xmission.com>, "H. Peter Anvin" <hpa@zytor.com>, Andy Lutomirski <luto@kernel.org>, 
	Peter Zijlstra <peterz@infradead.org>, Steven Rostedt <rostedt@goodmis.org>, 
	Andrew Morton <akpm@linux-foundation.org>, Mark Rutland <mark.rutland@arm.com>, 
	Tom Lendacky <thomas.lendacky@amd.com>, Ashish Kalra <ashish.kalra@amd.com>, 
	James Gowans <jgowans@amazon.com>, 
	Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>, arnd@arndb.de, pbonzini@redhat.com, 
	madvenka@linux.microsoft.com, Anthony Yznaga <anthony.yznaga@oracle.com>, 
	Usama Arif <usama.arif@bytedance.com>, David Woodhouse <dwmw@amazon.co.uk>, 
	Benjamin Herrenschmidt <benh@kernel.crashing.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 22, 2023 at 12:52=E2=80=AFPM Alexander Graf <graf@amazon.com> w=
rote:
>
> With KHO in place, let's add documentation that describes what it is and
> how to use it.
>
> Signed-off-by: Alexander Graf <graf@amazon.com>
> ---
>  Documentation/kho/concepts.rst   | 88 ++++++++++++++++++++++++++++++++
>  Documentation/kho/index.rst      | 19 +++++++
>  Documentation/kho/usage.rst      | 57 +++++++++++++++++++++
>  Documentation/subsystem-apis.rst |  1 +
>  4 files changed, 165 insertions(+)
>  create mode 100644 Documentation/kho/concepts.rst
>  create mode 100644 Documentation/kho/index.rst
>  create mode 100644 Documentation/kho/usage.rst
>
> diff --git a/Documentation/kho/concepts.rst b/Documentation/kho/concepts.=
rst
> new file mode 100644
> index 000000000000..8e4fe8c57865
> --- /dev/null
> +++ b/Documentation/kho/concepts.rst
> @@ -0,0 +1,88 @@
> +.. SPDX-License-Identifier: GPL-2.0-or-later
> +
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> +Kexec Handover Concepts
> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> +
> +Kexec HandOver (KHO) is a mechanism that allows Linux to preserve state =
-
> +arbitrary properties as well as memory locations - across kexec.
> +
> +It introduces multiple concepts:
> +
> +KHO Device Tree
> +---------------
> +
> +Every KHO kexec carries a KHO specific flattened device tree blob that
> +describes the state of the system. Device drivers can register to KHO to
> +serialize their state before kexec. After KHO, device drivers can read
> +the device tree and extract previous state.

How does this work with kexec when there is also the FDT for the h/w?
The h/w FDT has a /chosen property pointing to this FDT blob?

> +
> +KHO only uses the fdt container format and libfdt library, but does not
> +adhere to the same property semantics that normal device trees do: Prope=
rties
> +are passed in native endianness and standardized properties like ``regs`=
` and
> +``ranges`` do not exist, hence there are no ``#...-cells`` properties.

I think native endianness is asking for trouble. libfdt would need
different swap functions here than elsewhere in the kernel for example
which wouldn't even work. So you are just crossing your fingers that
you aren't using any libfdt functions that swap. And when I sync
dtc/libfdt and that changes, I might break you.

Also, if you want to dump the FDT and do a dtc DTB->DTS pass, it is
not going to be too readable given that outputs swapped 32-bit values
for anything that's a 4 byte multiple.

> +
> +KHO introduces a new concept to its device tree: ``mem`` properties. A
> +``mem`` property can inside any subnode in the device tree. When present=
,
> +it contains an array of physical memory ranges that the new kernel must =
mark
> +as reserved on boot. It is recommended, but not required, to make these =
ranges
> +as physically contiguous as possible to reduce the number of array eleme=
nts ::
> +
> +    struct kho_mem {
> +            __u64 addr;
> +            __u64 len;
> +    };
> +
> +After boot, drivers can call the kho subsystem to transfer ownership of =
memory
> +that was reserved via a ``mem`` property to themselves to continue using=
 memory
> +from the previous execution.
> +
> +The KHO device tree follows the in-Linux schema requirements. Any elemen=
t in
> +the device tree is documented via device tree schema yamls that explain =
what
> +data gets transferred.

If this is all separate, then I think the schemas should be too. And
then from my (DT maintainer) perspective, you can do whatever you want
here (like FIT images). The dtschema tools are pretty much only geared
for "normal" DTs. A couple of problems come to mind. You can't exclude
or change standard properties. The decoding of the DTB to run
validation assumes big endian. We could probably split things up a
bit, but you may be better off just using jsonschema directly. I'm not
even sure running validation here would that valuable. You have 1
source of code generating the DT and 1 consumer. Yes, there's
different kernel versions to deal with, but it's not 100s of people
creating 1000s of DTs with 100s of nodes.

You might look at the netlink stuff which is using its own yaml syntax
to generate code and jsonschema is used to validate the yaml.

Rob