Received-SPF: pass (google.com: domain of linux-kernel+bounces-29245-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Precedence: bulk
MIME-Version: 1.0
References: <20231222193607.15474-1-graf@amazon.com> <20231222195144.24532-1-graf@amazon.com>
 <20231222195144.24532-2-graf@amazon.com> <CAL_JsqJSYgq7BJnSxwKebEX8e9tL-FG74rT+eLJ4e32kKZC30g@mail.gmail.com>
 <c96203ba-a5b2-4765-9d30-0f4e66c82cd7@amazon.com>
In-Reply-To: <c96203ba-a5b2-4765-9d30-0f4e66c82cd7@amazon.com>
From: Rob Herring <robh+dt@kernel.org>
Date: Wed, 17 Jan 2024 10:54:43 -0600
Message-ID: <CAL_JsqK2WaH+vWd-hcrCSzycMGTztX4i2p1wpECseD_M3EY0tA@mail.gmail.com>
Subject: Re: [PATCH v2 07/17] kexec: Add documentation for KHO
To: Alexander Graf <graf@amazon.com>
Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, 
	linux-mm@kvack.org, devicetree@vger.kernel.org, 
	linux-arm-kernel@lists.infradead.org, kexec@lists.infradead.org, 
	linux-doc@vger.kernel.org, x86@kernel.org, 
	Eric Biederman <ebiederm@xmission.com>, "H. Peter Anvin" <hpa@zytor.com>, Andy Lutomirski <luto@kernel.org>, 
	Peter Zijlstra <peterz@infradead.org>, Steven Rostedt <rostedt@goodmis.org>, 
	Andrew Morton <akpm@linux-foundation.org>, Mark Rutland <mark.rutland@arm.com>, 
	Tom Lendacky <thomas.lendacky@amd.com>, Ashish Kalra <ashish.kalra@amd.com>, 
	James Gowans <jgowans@amazon.com>, 
	Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>, arnd@arndb.de, pbonzini@redhat.com, 
	madvenka@linux.microsoft.com, Anthony Yznaga <anthony.yznaga@oracle.com>, 
	Usama Arif <usama.arif@bytedance.com>, David Woodhouse <dwmw@amazon.co.uk>, 
	Benjamin Herrenschmidt <benh@kernel.crashing.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, Jan 17, 2024 at 8:02=E2=80=AFAM Alexander Graf <graf@amazon.com> wr=
ote:
>
>
> On 03.01.24 19:48, Rob Herring wrote:
> >
> > On Fri, Dec 22, 2023 at 12:52=E2=80=AFPM Alexander Graf <graf@amazon.co=
m> wrote:
> >> With KHO in place, let's add documentation that describes what it is a=
nd
> >> how to use it.
> >>
> >> Signed-off-by: Alexander Graf <graf@amazon.com>
> >> ---
> >>   Documentation/kho/concepts.rst   | 88 ++++++++++++++++++++++++++++++=
++
> >>   Documentation/kho/index.rst      | 19 +++++++
> >>   Documentation/kho/usage.rst      | 57 +++++++++++++++++++++
> >>   Documentation/subsystem-apis.rst |  1 +
> >>   4 files changed, 165 insertions(+)
> >>   create mode 100644 Documentation/kho/concepts.rst
> >>   create mode 100644 Documentation/kho/index.rst
> >>   create mode 100644 Documentation/kho/usage.rst
> >>
> >> diff --git a/Documentation/kho/concepts.rst b/Documentation/kho/concep=
ts.rst
> >> new file mode 100644
> >> index 000000000000..8e4fe8c57865
> >> --- /dev/null
> >> +++ b/Documentation/kho/concepts.rst
> >> @@ -0,0 +1,88 @@
> >> +.. SPDX-License-Identifier: GPL-2.0-or-later
> >> +
> >> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> +Kexec Handover Concepts
> >> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >> +
> >> +Kexec HandOver (KHO) is a mechanism that allows Linux to preserve sta=
te -
> >> +arbitrary properties as well as memory locations - across kexec.
> >> +
> >> +It introduces multiple concepts:
> >> +
> >> +KHO Device Tree
> >> +---------------
> >> +
> >> +Every KHO kexec carries a KHO specific flattened device tree blob tha=
t
> >> +describes the state of the system. Device drivers can register to KHO=
 to
> >> +serialize their state before kexec. After KHO, device drivers can rea=
d
> >> +the device tree and extract previous state.

Can you avoid calling anything "device tree" as much as possible. We
can't avoid the format is FDT/DTB, but otherwise none of this is
Devicetree as most folks know it. Sure, there can be trees of devices
which are not Devicetree, but this is neither. You could have used
BSON or any hierarchical key-value pair serialization format just as
easily (if we already had a parser in the kernel).

> > How does this work with kexec when there is also the FDT for the h/w?
> > The h/w FDT has a /chosen property pointing to this FDT blob?
>
>
> Yep, exactly.

Those properties need to be documented here[1].

[...]

> >> +KHO introduces a new concept to its device tree: ``mem`` properties. =
A
> >> +``mem`` property can inside any subnode in the device tree. When pres=
ent,
> >> +it contains an array of physical memory ranges that the new kernel mu=
st mark
> >> +as reserved on boot. It is recommended, but not required, to make the=
se ranges
> >> +as physically contiguous as possible to reduce the number of array el=
ements ::
> >> +
> >> +    struct kho_mem {
> >> +            __u64 addr;
> >> +            __u64 len;
> >> +    };
> >> +
> >> +After boot, drivers can call the kho subsystem to transfer ownership =
of memory
> >> +that was reserved via a ``mem`` property to themselves to continue us=
ing memory
> >> +from the previous execution.
> >> +
> >> +The KHO device tree follows the in-Linux schema requirements. Any ele=
ment in
> >> +the device tree is documented via device tree schema yamls that expla=
in what
> >> +data gets transferred.
> > If this is all separate, then I think the schemas should be too. And
> > then from my (DT maintainer) perspective, you can do whatever you want
> > here (like FIT images). The dtschema tools are pretty much only geared
> > for "normal" DTs. A couple of problems come to mind. You can't exclude
> > or change standard properties. The decoding of the DTB to run
> > validation assumes big endian. We could probably split things up a
> > bit, but you may be better off just using jsonschema directly. I'm not
> > even sure running validation here would that valuable. You have 1
> > source of code generating the DT and 1 consumer. Yes, there's
> > different kernel versions to deal with, but it's not 100s of people
> > creating 1000s of DTs with 100s of nodes.
> >
> > You might look at the netlink stuff which is using its own yaml syntax
> > to generate code and jsonschema is used to validate the yaml.
>
>
> I'm currently a lot more interested in the documentation aspect than in
> the validation, yeah. So I think for v3, I'll just throw the schemas
> into the Documentation/kho directory without any validation. We can
> worry about that later :)

I'll regret that when I get patches fixing them, but okay.

Rob

[1] https://github.com/devicetree-org/dt-schema/blob/main/dtschema/schemas/=
chosen.yaml