Received: by 2002:a05:7412:b995:b0:f9:9502:5bb8 with SMTP id it21csp1019677rdb; Fri, 22 Dec 2023 11:52:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IGzqCvoe+1x8TC4rao7WeSVrxHfg05NPaWBWXKSRCFudMUVPneogmuStDFwuPWPmGpCsmfn X-Received: by 2002:a05:6a20:8e15:b0:195:18d3:5996 with SMTP id y21-20020a056a208e1500b0019518d35996mr2187928pzj.72.1703274746604; Fri, 22 Dec 2023 11:52:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703274746; cv=none; d=google.com; s=arc-20160816; b=qPyQS/fOILPor/cxeP+AHw1W0vkb/goMj7Iye2SVrmKY2yn1Sbt7rpyplC74bLnHDD Nto/b+2Dg9GGuzdiGNg3SZDh1CVhpvhJbD+B4pem8ncBussg9nttJtMiS+WWUCDZ6qHB MacXophhBicEy59cBWq0LRnBbnTvnulgd+3+pVEyDvAmNRqa0nLDP4ED8QL+e6+GsNC8 +NWN9fwydtDeBm9wA38qdFLmeOSIzeC2oa9xtTtPFmW5hHNqlYGrXB/lZar64ATCGGYc vtoEf0savF4iW679FH3Lk3fwKdX35T6VXiK3hRlyc4W4D4JJBnx5rt+U2Na+WsEXq2iq MI6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Wy3QrJqbxk6W3Bwc5BKomfQATEZZd0nBtswCXV4R7vM=; fh=xprWcyUKTLi4rhte7gFCvduD673j12awuFkCPnYDQVY=; b=C1LzEw6QoglUYOKE+iWfjy78L//jhvps2XOAkk5zU6IaZdYaW978QRBjyoj05XKctm dqmn3O6XbhpvBj2g/dy4XCwVSJArUADFWum6PNWN9kVc0XkzDGfgxBzbZ2UNsd/otnlb gxTcCHGpCmzABWixwuBXVqf6EmczqnjJ6BeQJRvpOQF7GqPVgFPaNyur0zI2pXjB4Hkh v5f7bqoCKFPl9zd3lCoDyBl2R4w5jR9mwgOrE2f1ASd+PDVSPF9VQwIeLMnPtOJxGCLP R6ME2jLFwci24GzLG2MxKB45DAx6yLG3dVCcE4Q9//P/7YZL6MwpvUrXixFrtoO5AyOn F+JQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=sRTTzQ0R; spf=pass (google.com: domain of linux-kernel+bounces-10032-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10032-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id i14-20020a63540e000000b005c66af62bacsi3787329pgb.174.2023.12.22.11.52.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Dec 2023 11:52:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-10032-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b=sRTTzQ0R; spf=pass (google.com: domain of linux-kernel+bounces-10032-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-10032-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 3E25B28530B for ; Fri, 22 Dec 2023 19:52:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1A55B2E85A; Fri, 22 Dec 2023 19:52:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="sRTTzQ0R" X-Original-To: linux-kernel@vger.kernel.org Received: from smtp-fw-52004.amazon.com (smtp-fw-52004.amazon.com [52.119.213.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A4C02E82C; Fri, 22 Dec 2023 19:52:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1703274723; x=1734810723; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Wy3QrJqbxk6W3Bwc5BKomfQATEZZd0nBtswCXV4R7vM=; b=sRTTzQ0R+cvUWMJHeZ/1z8Ns9v1K1H0nVdac2SEbYRghJ2Dn3pjXm2sL LRxhKiSPA9SfIj0eCgbknn4jgD89gC8gORk3FulOtMJfLFaPCQJLurWcV JDEbDeDHj6vr5PiIjHwz2CZIv5LCZnpF//ggejWl+Fmqlg17sNvGZeEAP M=; X-IronPort-AV: E=Sophos;i="6.04,297,1695686400"; d="scan'208";a="174097101" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-iad-1d-m6i4x-e651a362.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-52004.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Dec 2023 19:52:01 +0000 Received: from smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev (iad7-ws-svc-p70-lb3-vlan3.iad.amazon.com [10.32.235.38]) by email-inbound-relay-iad-1d-m6i4x-e651a362.us-east-1.amazon.com (Postfix) with ESMTPS id 6EDEC8053C; Fri, 22 Dec 2023 19:51:54 +0000 (UTC) Received: from EX19MTAUWA001.ant.amazon.com [10.0.7.35:11667] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.56.23:2525] with esmtp (Farcaster) id 7869ce95-383d-43dc-8bf7-e44da388af62; Fri, 22 Dec 2023 19:51:53 +0000 (UTC) X-Farcaster-Flow-ID: 7869ce95-383d-43dc-8bf7-e44da388af62 Received: from EX19D020UWC004.ant.amazon.com (10.13.138.149) by EX19MTAUWA001.ant.amazon.com (10.250.64.204) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Fri, 22 Dec 2023 19:51:53 +0000 Received: from dev-dsk-graf-1a-5ce218e4.eu-west-1.amazon.com (10.253.83.51) by EX19D020UWC004.ant.amazon.com (10.13.138.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Fri, 22 Dec 2023 19:51:49 +0000 From: Alexander Graf To: CC: , , , , , , , Eric Biederman , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Rob Herring" , Steven Rostedt , "Andrew Morton" , Mark Rutland , "Tom Lendacky" , Ashish Kalra , James Gowans , Stanislav Kinsburskii , , , , Anthony Yznaga , Usama Arif , David Woodhouse , Benjamin Herrenschmidt Subject: [PATCH v2 07/17] kexec: Add documentation for KHO Date: Fri, 22 Dec 2023 19:51:34 +0000 Message-ID: <20231222195144.24532-2-graf@amazon.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231222195144.24532-1-graf@amazon.com> References: <20231222193607.15474-1-graf@amazon.com> <20231222195144.24532-1-graf@amazon.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: EX19D032UWA002.ant.amazon.com (10.13.139.81) To EX19D020UWC004.ant.amazon.com (10.13.138.149) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit With KHO in place, let's add documentation that describes what it is and how to use it. Signed-off-by: Alexander Graf --- Documentation/kho/concepts.rst | 88 ++++++++++++++++++++++++++++++++ Documentation/kho/index.rst | 19 +++++++ Documentation/kho/usage.rst | 57 +++++++++++++++++++++ Documentation/subsystem-apis.rst | 1 + 4 files changed, 165 insertions(+) create mode 100644 Documentation/kho/concepts.rst create mode 100644 Documentation/kho/index.rst create mode 100644 Documentation/kho/usage.rst diff --git a/Documentation/kho/concepts.rst b/Documentation/kho/concepts.rst new file mode 100644 index 000000000000..8e4fe8c57865 --- /dev/null +++ b/Documentation/kho/concepts.rst @@ -0,0 +1,88 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +======================= +Kexec Handover Concepts +======================= + +Kexec HandOver (KHO) is a mechanism that allows Linux to preserve state - +arbitrary properties as well as memory locations - across kexec. + +It introduces multiple concepts: + +KHO Device Tree +--------------- + +Every KHO kexec carries a KHO specific flattened device tree blob that +describes the state of the system. Device drivers can register to KHO to +serialize their state before kexec. After KHO, device drivers can read +the device tree and extract previous state. + +KHO only uses the fdt container format and libfdt library, but does not +adhere to the same property semantics that normal device trees do: Properties +are passed in native endianness and standardized properties like ``regs`` and +``ranges`` do not exist, hence there are no ``#...-cells`` properties. + +KHO introduces a new concept to its device tree: ``mem`` properties. A +``mem`` property can inside any subnode in the device tree. When present, +it contains an array of physical memory ranges that the new kernel must mark +as reserved on boot. It is recommended, but not required, to make these ranges +as physically contiguous as possible to reduce the number of array elements :: + + struct kho_mem { + __u64 addr; + __u64 len; + }; + +After boot, drivers can call the kho subsystem to transfer ownership of memory +that was reserved via a ``mem`` property to themselves to continue using memory +from the previous execution. + +The KHO device tree follows the in-Linux schema requirements. Any element in +the device tree is documented via device tree schema yamls that explain what +data gets transferred. + +Mem cache +--------- + +The new kernel needs to know about all memory reservations, but is unable to +parse the device tree yet in early bootup code because of memory limitations. +To simplify the initial memory reservation flow, the old kernel passes a +preprocessed array of physically contiguous reserved ranges to the new kernel. + +These reservations have to be separate from architectural memory maps and +reservations because they differ on every kexec, while the architectural ones +get passed directly between invocations. + +The less entries this cache contains, the faster the new kernel will boot. + +Scratch Region +-------------- + +To boot into kexec, we need to have a physically contiguous memory range that +contains no handed over memory. Kexec then places the target kernel and initrd +into that region. The new kernel exclusively uses this region for memory +allocations before it ingests the mem cache. + +We guarantee that we always have such a region through the scratch region: On +first boot, you can pass the ``kho_scratch`` kernel command line option. When +it is set, Linux allocates a CMA region of the given size. CMA gives us the +guarantee that no handover pages land in that region, because handover +pages must be at a static physical memory location and CMA enforces that +only movable pages can be located inside. + +After KHO kexec, we ignore the ``kho_scratch`` kernel command line option and +instead reuse the exact same region that was originally allocated. This allows +us to recursively execute any amount of KHO kexecs. Because we used this region +for boot memory allocations and as target memory for kexec blobs, some parts +of that memory region may be reserved. These reservations are irrenevant for +the next KHO, because kexec can overwrite even the original kernel. + +KHO active phase +---------------- + +To enable user space based kexec file loader, the kernel needs to be able to +provide the device tree that describes the previous kernel's state before +performing the actual kexec. The process of generating that device tree is +called serialization. When the device tree is generated, some properties +of the system may become immutable because they are already written down +in the device tree. That state is called the KHO active phase. diff --git a/Documentation/kho/index.rst b/Documentation/kho/index.rst new file mode 100644 index 000000000000..5e7eeeca8520 --- /dev/null +++ b/Documentation/kho/index.rst @@ -0,0 +1,19 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +======================== +Kexec Handover Subsystem +======================== + +.. toctree:: + :maxdepth: 1 + + concepts + usage + +.. only:: subproject and html + + + Indices + ======= + + * :ref:`genindex` diff --git a/Documentation/kho/usage.rst b/Documentation/kho/usage.rst new file mode 100644 index 000000000000..5efa2a58f9c3 --- /dev/null +++ b/Documentation/kho/usage.rst @@ -0,0 +1,57 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +==================== +Kexec Handover Usage +==================== + +Kexec HandOver (KHO) is a mechanism that allows Linux to preserve state - +arbitrary properties as well as memory locations - across kexec. + +This document expects that you are familiar with the base KHO +:ref:`Documentation/kho/concepts.rst `. If you have not read +them yet, please do so now. + +Prerequisites +------------- + +KHO is available when the ``CONFIG_KEXEC_KHO`` config option is set to y +at compile team. Every KHO producer has its own config option that you +need to enable if you would like to preserve their respective state across +kexec. + +To use KHO, please boot the kernel with the ``kho_scratch`` command +line parameter set to allocate a scratch region. For example +``kho_scratch=512M`` will reserve a 512 MiB scratch region on boot. + +Perform a KHO kexec +------------------- + +Before you can perform a KHO kexec, you need to move the system into the +:ref:`Documentation/kho/concepts.rst ` :: + + $ echo 1 > /sys/kernel/kho/active + +After this command, the KHO device tree is available in ``/sys/kernel/kho/dt``. + +Next, load the target payload and kexec into it. It is important that you +use the ``-s`` parameter to use the in-kernel kexec file loader, as user +space kexec tooling currently has no support for KHO with the user space +based file loader :: + + # kexec -l Image --initrd=initrd -s + # kexec -e + +The new kernel will boot up and contain some of the previous kernel's state. + +For example, if you enabled ``CONFIG_FTRACE_KHO``, the new kernel will contain +the old kernel's trace buffers in ``/sys/kernel/debug/tracing/trace``. + +Abort a KHO exec +---------------- + +You can move the system out of KHO active phase again by calling :: + + $ echo 1 > /sys/kernel/kho/active + +After this command, the KHO device tree is no longer available in +``/sys/kernel/kho/dt``. diff --git a/Documentation/subsystem-apis.rst b/Documentation/subsystem-apis.rst index 930dc23998a0..8207b6514d87 100644 --- a/Documentation/subsystem-apis.rst +++ b/Documentation/subsystem-apis.rst @@ -86,3 +86,4 @@ Storage interfaces misc-devices/index peci/index wmi/index + kho/index -- 2.40.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879