LinuxLists.cc - [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

2018-12-10 21:19:26

Subject: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

For certain applications it is desirable to rapidly boot a KVM virtual
machine. In cases where legacy hardware and software support within the
guest is not needed, Qemu should be able to boot directly into the
uncompressed Linux kernel binary without the need to run firmware.

There already exists an ABI to allow this for Xen PVH guests and the ABI
is supported by Linux and FreeBSD:

https://xenbits.xen.org/docs/unstable/misc/pvh.html

This patch series would enable Qemu to use that same entry point for
booting KVM guests.

Changes from v8:

* Removed unused KVM_GUEST_PVH symbol.

Changes from v7:

(No functional changes from v7 other than rebasing to latest upstream)
* Added Review-by tags as provided by Juergen Gross (1,2,3,6,7)
* Rebasing to upstream 4.18 caused a minor conflict in patch 4 that had
to be hand merged due to this patch:
1fe8388 xen: share start flags between PV and PVH
I just had to make sure we were accounting for the xen_start_flags
in the new code path.
* Rebasing to upstream 4.20-rc4 caused a few minor conflicts in patches
2,3,5,7 that needed to be resolved by hand. The conflicts were due to
upstream non-functional code cleanup changes in arch/x86/xen/Makefile
and arch/x86/platform/pvh/enlighten.c due to these patches:
28c11b0 x86/xen: Move pv irq related functions under CONFIG_XEN_PV
umbrella
357d291 x86/xen: Fix boot loader version reported for PVH guests
3cfa210 xen: don't include <xen/xen.h> from <asm/io.h> and
<asm/dma-mapping.h>
* Qemu and qboot RFC patches have been posted to show one example of how
this functionality can be used. Some preliminary numbers are available
in those cover letters showing the KVM guest boot time improvement.
Qemu:
http://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00957.html
qboot:
http://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00953.html

Changes from v6:

* Addressed issues caught by the kbuild test robot:
- Restored an #include line that had been dropped by mistake (patch 4)
- Removed a pair of #include lines that were no longer needed in a
common code file and causing problems for certain 32-bit configs
(patchs 4 and 7)

Changes from v5:

* The interface changes to the x86/HVM start info layout have
now been accepted into the Xen tree.
* Rebase and merge upstream PVH file changes.
* (Patch 6) Synced up to the final version of the header file that was
acked and pulled into the Xen tree.
* (Patch 1) Fixed typo and removed redundant "def_bool n" line.

Changes from v4:

Note: I've withheld Juergen's earlier "Reviewed-by" tags from patches
1 and 7 since there were minor changes (mostly just addition of
CONFIG_KVM_GUEST_PVH as requested) that came afterwards.

* Changed subject prefix from RFC to PATCH
* Added CONFIG_KVM_GUEST_PVH as suggested
* Relocated the PVH common files to
arch/x86/platform/pvh/{enlighten.c,head.S}
* Realized I also needed to move the objtool override for those files
* Updated a few code comments per reviewer feedback
* Sent out a patch of the hvm_start_info struct changes against the Xen
tree since that is the canonical copy of the header. Discussions on
that thread have resulted in some (non-functional) updates to
start_info.h (patch 6/7) and those changes are reflected here as well
in order to keep the files in sync. The header file has since been
ack'ed for the Xen tree by Jan Beulich.

Changes from v3:

* Implemented Juergen's suggestion for refactoring and moving the PVH
code so that CONFIG_XEN is no longer required for booting KVM guests
via the PVH entry point.
Functionally, nothing has changed from V3 really, but the patches
look completely different now because of all the code movement and
refactoring. Some of these patches can be combined, but I've left
them very small in some cases to make the refactoring and code
movement easier to review.
My approach for refactoring has been to create a PVH entry layer that
still has understanding and knowledge about Xen vs non-Xen guest types
so that it can make run time decisions to handle either case, as
opposed to going all the way and re-writing it to be a completely
hypervisor agnostic and architecturally pure layer that is separate
from guest type details. The latter seemed a bit overkill in this
situation. And I've handled the complexity of having to support
Qemu/KVM boot of kernels compiled with or without CONFIG_XEN via a
pair of xen specific __weak routines that can be overridden in kernels
that support Xen guests. Importantly, the __weak routines are for
xen specific code only (not generic "guest type" specific code) so
there is no clashing between xen version of the strong routine and,
say, a KVM version of the same routine. But I'm sure there are many
ways to skin this cat, so I'm open to alternate suggestions if there
is a compelling reason for not using __weak in this situation.

Changes from v2:

* All structures (including memory map table entries) are padded and
aligned to an 8 byte boundary.

* Removed the "packed" attributes and made changes to comments as
suggested by Jan.

Changes from v1:

* Adopted Paolo's suggestion for defining a v2 PVH ABI that includes the
e820 map instead of using the second module entry to pass the table.

* Cleaned things up a bit to reduce the number of xen vs non-xen special
cases.

Maran Wilson (7):
xen/pvh: Split CONFIG_XEN_PVH into CONFIG_PVH and CONFIG_XEN_PVH
xen/pvh: Move PVH entry code out of Xen specific tree
xen/pvh: Create a new file for Xen specific PVH code
xen/pvh: Move Xen specific PVH VM initialization out of common file
xen/pvh: Move Xen code for getting mem map via hcall out of common
file
xen/pvh: Add memory map pointer to hvm_start_info struct
KVM: x86: Allow Qemu/KVM to use PVH entry point

MAINTAINERS | 1 +
arch/x86/Kbuild | 2 +
arch/x86/Kconfig | 6 ++
arch/x86/kernel/head_64.S | 2 +-
arch/x86/platform/pvh/Makefile | 5 +
arch/x86/platform/pvh/enlighten.c | 137 ++++++++++++++++++++++++
arch/x86/{xen/xen-pvh.S => platform/pvh/head.S} | 0
arch/x86/xen/Kconfig | 3 +-
arch/x86/xen/Makefile | 2 -
arch/x86/xen/enlighten_pvh.c | 94 ++++------------
include/xen/interface/hvm/start_info.h | 63 ++++++++++-
include/xen/xen.h | 3 +
12 files changed, 237 insertions(+), 81 deletions(-)
create mode 100644 arch/x86/platform/pvh/Makefile
create mode 100644 arch/x86/platform/pvh/enlighten.c
rename arch/x86/{xen/xen-pvh.S => platform/pvh/head.S} (100%)

--
2.16.1

2018-12-10 20:06:58

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 1/7] xen/pvh: Split CONFIG_XEN_PVH into CONFIG_PVH and CONFIG_XEN_PVH

In order to pave the way for hypervisors other than Xen to use the PVH
entry point for VMs, we need to factor the PVH entry code into Xen specific
and hypervisor agnostic components. The first step in doing that, is to
create a new config option for PVH entry that can be enabled
independently from CONFIG_XEN.

Signed-off-by: Maran Wilson <[email protected]>
Reviewed-by: Juergen Gross <[email protected]>
---
arch/x86/Kconfig | 6 ++++++
arch/x86/kernel/head_64.S | 2 +-
arch/x86/xen/Kconfig | 3 ++-
3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 8689e794a43c..c2a22a74abee 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -796,6 +796,12 @@ config KVM_GUEST
underlying device model, the host provides the guest with
timing infrastructure such as time of day, and system time

+config PVH
+ bool "Support for running PVH guests"
+ ---help---
+ This option enables the PVH entry point for guest virtual machines
+ as specified in the x86/HVM direct boot ABI.
+
config KVM_DEBUG_FS
bool "Enable debug information for KVM Guests in debugfs"
depends on KVM_GUEST && DEBUG_FS
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 747c758f67b7..d1dbe8e4eb82 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -386,7 +386,7 @@ NEXT_PAGE(early_dynamic_pgts)

.data

-#if defined(CONFIG_XEN_PV) || defined(CONFIG_XEN_PVH)
+#if defined(CONFIG_XEN_PV) || defined(CONFIG_PVH)
NEXT_PGD_PAGE(init_top_pgt)
.quad level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC
.org init_top_pgt + L4_PAGE_OFFSET*8, 0
diff --git a/arch/x86/xen/Kconfig b/arch/x86/xen/Kconfig
index 1ef391aa184d..e07abefd3d26 100644
--- a/arch/x86/xen/Kconfig
+++ b/arch/x86/xen/Kconfig
@@ -74,6 +74,7 @@ config XEN_DEBUG_FS
Enabling this option may incur a significant performance overhead.

config XEN_PVH
- bool "Support for running as a PVH guest"
+ bool "Support for running as a Xen PVH guest"
depends on XEN && XEN_PVHVM && ACPI
+ select PVH
def_bool n
--
2.16.1

2018-12-10 20:07:07

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 5/7] xen/pvh: Move Xen code for getting mem map via hcall out of common file

We need to refactor PVH entry code so that support for other hypervisors
like Qemu/KVM can be added more easily.

The original design for PVH entry in Xen guests relies on being able to
obtain the memory map from the hypervisor using a hypercall. When we
extend the PVH entry ABI to support other hypervisors like Qemu/KVM,
a new mechanism will be added that allows the guest to get the memory
map without needing to use hypercalls.

For Xen guests, the hypercall approach will still be supported. In
preparation for adding support for other hypervisors, we can move the
code that uses hypercalls into the Xen specific file. This will allow us
to compile kernels in the future without CONFIG_XEN that are still capable
of being booted as a Qemu/KVM guest via the PVH entry point.

Signed-off-by: Maran Wilson <[email protected]>
Reviewed-by: Juergen Gross <[email protected]>
---
arch/x86/platform/pvh/enlighten.c | 29 ++++++++++++++---------------
arch/x86/xen/enlighten_pvh.c | 20 ++++++++++++++++++++
2 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/arch/x86/platform/pvh/enlighten.c b/arch/x86/platform/pvh/enlighten.c
index 637bd74ba32d..8040b3fbf545 100644
--- a/arch/x86/platform/pvh/enlighten.c
+++ b/arch/x86/platform/pvh/enlighten.c
@@ -8,11 +8,7 @@
#include <asm/e820/api.h>
#include <asm/x86_init.h>

-#include <asm/xen/interface.h>
-#include <asm/xen/hypercall.h>
-
#include <xen/xen.h>
-#include <xen/interface/memory.h>
#include <xen/interface/hvm/start_info.h>

/*
@@ -31,21 +27,24 @@ static u64 pvh_get_root_pointer(void)
return pvh_start_info.rsdp_paddr;
}

+/*
+ * Xen guests are able to obtain the memory map from the hypervisor via the
+ * HYPERVISOR_memory_op hypercall.
+ * If we are trying to boot a Xen PVH guest, it is expected that the kernel
+ * will have been configured to provide an override for this routine to do
+ * just that.
+ */
+void __init __weak mem_map_via_hcall(struct boot_params *ptr __maybe_unused)
+{
+ xen_raw_printk("Error: Could not find memory map\n");
+ BUG();
+}
+
static void __init init_pvh_bootparams(void)
{
- struct xen_memory_map memmap;
- int rc;
-
memset(&pvh_bootparams, 0, sizeof(pvh_bootparams));

- memmap.nr_entries = ARRAY_SIZE(pvh_bootparams.e820_table);
- set_xen_guest_handle(memmap.buffer, pvh_bootparams.e820_table);
- rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
- if (rc) {
- xen_raw_printk("XENMEM_memory_map failed (%d)\n", rc);
- BUG();
- }
- pvh_bootparams.e820_entries = memmap.nr_entries;
+ mem_map_via_hcall(&pvh_bootparams);

if (pvh_bootparams.e820_entries < E820_MAX_ENTRIES_ZEROPAGE - 1) {
pvh_bootparams.e820_table[pvh_bootparams.e820_entries].addr =
diff --git a/arch/x86/xen/enlighten_pvh.c b/arch/x86/xen/enlighten_pvh.c
index 41a7d6ad74e0..35b7599d2d0b 100644
--- a/arch/x86/xen/enlighten_pvh.c
+++ b/arch/x86/xen/enlighten_pvh.c
@@ -1,13 +1,18 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/acpi.h>

+#include <xen/hvc-console.h>
+
#include <asm/io_apic.h>
#include <asm/hypervisor.h>
+#include <asm/e820/api.h>

#include <xen/xen.h>
#include <asm/xen/interface.h>
#include <asm/xen/hypercall.h>

+#include <xen/interface/memory.h>
+
/*
* PVH variables.
*
@@ -28,3 +33,18 @@ void __init xen_pvh_init(void)
pfn = __pa(hypercall_page);
wrmsr_safe(msr, (u32)pfn, (u32)(pfn >> 32));
}
+
+void __init mem_map_via_hcall(struct boot_params *boot_params_p)
+{
+ struct xen_memory_map memmap;
+ int rc;
+
+ memmap.nr_entries = ARRAY_SIZE(boot_params_p->e820_table);
+ set_xen_guest_handle(memmap.buffer, boot_params_p->e820_table);
+ rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
+ if (rc) {
+ xen_raw_printk("XENMEM_memory_map failed (%d)\n", rc);
+ BUG();
+ }
+ boot_params_p->e820_entries = memmap.nr_entries;
+}
--
2.16.1

2018-12-10 20:08:45

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 3/7] xen/pvh: Create a new file for Xen specific PVH code

We need to refactor PVH entry code so that support for other hypervisors
like Qemu/KVM can be added more easily.

The first step in that direction is to create a new file that will
eventually hold the Xen specific routines.

Signed-off-by: Maran Wilson <[email protected]>
Reviewed-by: Juergen Gross <[email protected]>
---
arch/x86/platform/pvh/enlighten.c | 5 ++---
arch/x86/xen/Makefile | 2 ++
arch/x86/xen/enlighten_pvh.c | 10 ++++++++++
3 files changed, 14 insertions(+), 3 deletions(-)
create mode 100644 arch/x86/xen/enlighten_pvh.c

diff --git a/arch/x86/platform/pvh/enlighten.c b/arch/x86/platform/pvh/enlighten.c
index 02e3ab7ff242..491932991202 100644
--- a/arch/x86/platform/pvh/enlighten.c
+++ b/arch/x86/platform/pvh/enlighten.c
@@ -18,10 +18,9 @@
/*
* PVH variables.
*
- * xen_pvh pvh_bootparams and pvh_start_info need to live in data segment
- * since they are used after startup_{32|64}, which clear .bss, are invoked.
+ * pvh_bootparams and pvh_start_info need to live in the data segment since
+ * they are used after startup_{32|64}, which clear .bss, are invoked.
*/
-bool xen_pvh __attribute__((section(".data"))) = 0;
struct boot_params pvh_bootparams __attribute__((section(".data")));
struct hvm_start_info pvh_start_info __attribute__((section(".data")));

diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile
index b239922f6c6c..084de77a109e 100644
--- a/arch/x86/xen/Makefile
+++ b/arch/x86/xen/Makefile
@@ -36,6 +36,8 @@ obj-$(CONFIG_XEN_PV) += multicalls.o
obj-$(CONFIG_XEN_PV) += xen-asm.o
obj-$(CONFIG_XEN_PV) += xen-asm_$(BITS).o

+obj-$(CONFIG_XEN_PVH) += enlighten_pvh.o
+
obj-$(CONFIG_EVENT_TRACING) += trace.o

obj-$(CONFIG_SMP) += smp.o
diff --git a/arch/x86/xen/enlighten_pvh.c b/arch/x86/xen/enlighten_pvh.c
new file mode 100644
index 000000000000..6be7bc719b38
--- /dev/null
+++ b/arch/x86/xen/enlighten_pvh.c
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/types.h>
+
+/*
+ * PVH variables.
+ *
+ * The variable xen_pvh needs to live in the data segment since it is used
+ * after startup_{32|64} is invoked, which will clear the .bss segment.
+ */
+bool xen_pvh __attribute__((section(".data"))) = 0;
--
2.16.1

2018-12-10 20:09:41

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 4/7] xen/pvh: Move Xen specific PVH VM initialization out of common file

We need to refactor PVH entry code so that support for other hypervisors
like Qemu/KVM can be added more easily.

This patch moves the small block of code used for initializing Xen PVH
virtual machines into the Xen specific file. This initialization is not
going to be needed for Qemu/KVM guests. Moving it out of the common file
is going to allow us to compile kernels in the future without CONFIG_XEN
that are still capable of being booted as a Qemu/KVM guest via the PVH
entry point.

Signed-off-by: Maran Wilson <[email protected]>
Reviewed-by: Konrad Rzeszutek Wilk <[email protected]>
Reviewed-by: Juergen Gross <[email protected]>
---
arch/x86/platform/pvh/enlighten.c | 29 ++++++++++++++++++++---------
arch/x86/xen/enlighten_pvh.c | 22 +++++++++++++++++++++-
include/xen/xen.h | 3 +++
3 files changed, 44 insertions(+), 10 deletions(-)

diff --git a/arch/x86/platform/pvh/enlighten.c b/arch/x86/platform/pvh/enlighten.c
index 491932991202..637bd74ba32d 100644
--- a/arch/x86/platform/pvh/enlighten.c
+++ b/arch/x86/platform/pvh/enlighten.c
@@ -81,27 +81,38 @@ static void __init init_pvh_bootparams(void)
x86_init.acpi.get_root_pointer = pvh_get_root_pointer;
}

+/*
+ * If we are trying to boot a Xen PVH guest, it is expected that the kernel
+ * will have been configured to provide the required override for this routine.
+ */
+void __init __weak xen_pvh_init(void)
+{
+ xen_raw_printk("Error: Missing xen PVH initialization\n");
+ BUG();
+}
+
+/*
+ * When we add support for other hypervisors like Qemu/KVM, this routine can
+ * selectively invoke the appropriate initialization based on guest type.
+ */
+static void hypervisor_specific_init(void)
+{
+ xen_pvh_init();
+}
+
/*
* This routine (and those that it might call) should not use
* anything that lives in .bss since that segment will be cleared later.
*/
void __init xen_prepare_pvh(void)
{
- u32 msr;
- u64 pfn;
-
if (pvh_start_info.magic != XEN_HVM_START_MAGIC_VALUE) {
xen_raw_printk("Error: Unexpected magic value (0x%08x)\n",
pvh_start_info.magic);
BUG();
}

- xen_pvh = 1;
- xen_start_flags = pvh_start_info.flags;
-
- msr = cpuid_ebx(xen_cpuid_base() + 2);
- pfn = __pa(hypercall_page);
- wrmsr_safe(msr, (u32)pfn, (u32)(pfn >> 32));
+ hypervisor_specific_init();

init_pvh_bootparams();
}
diff --git a/arch/x86/xen/enlighten_pvh.c b/arch/x86/xen/enlighten_pvh.c
index 6be7bc719b38..41a7d6ad74e0 100644
--- a/arch/x86/xen/enlighten_pvh.c
+++ b/arch/x86/xen/enlighten_pvh.c
@@ -1,5 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
-#include <linux/types.h>
+#include <linux/acpi.h>
+
+#include <asm/io_apic.h>
+#include <asm/hypervisor.h>
+
+#include <xen/xen.h>
+#include <asm/xen/interface.h>
+#include <asm/xen/hypercall.h>

/*
* PVH variables.
@@ -8,3 +15,16 @@
* after startup_{32|64} is invoked, which will clear the .bss segment.
*/
bool xen_pvh __attribute__((section(".data"))) = 0;
+
+void __init xen_pvh_init(void)
+{
+ u32 msr;
+ u64 pfn;
+
+ xen_pvh = 1;
+ xen_start_flags = pvh_start_info.flags;
+
+ msr = cpuid_ebx(xen_cpuid_base() + 2);
+ pfn = __pa(hypercall_page);
+ wrmsr_safe(msr, (u32)pfn, (u32)(pfn >> 32));
+}
diff --git a/include/xen/xen.h b/include/xen/xen.h
index d7a2678da77f..0e2156786ad2 100644
--- a/include/xen/xen.h
+++ b/include/xen/xen.h
@@ -29,6 +29,9 @@ extern bool xen_pvh;

extern uint32_t xen_start_flags;

+#include <xen/interface/hvm/start_info.h>
+extern struct hvm_start_info pvh_start_info;
+
#ifdef CONFIG_XEN_DOM0
#include <xen/interface/xen.h>
#include <asm/xen/hypervisor.h>
--
2.16.1

2018-12-10 20:09:41

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 6/7] xen/pvh: Add memory map pointer to hvm_start_info struct

The start info structure that is defined as part of the x86/HVM direct boot
ABI and used for starting Xen PVH guests would be more versatile if it also
included a way to pass information about the memory map to the guest. This
would allow KVM guests to share the same entry point.

Signed-off-by: Maran Wilson <[email protected]>
Reviewed-by: Juergen Gross <[email protected]>
---
include/xen/interface/hvm/start_info.h | 63 +++++++++++++++++++++++++++++++++-
1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/include/xen/interface/hvm/start_info.h b/include/xen/interface/hvm/start_info.h
index 648415976ead..50af9ea2ff1e 100644
--- a/include/xen/interface/hvm/start_info.h
+++ b/include/xen/interface/hvm/start_info.h
@@ -33,7 +33,7 @@
* | magic | Contains the magic value XEN_HVM_START_MAGIC_VALUE
* | | ("xEn3" with the 0x80 bit of the "E" set).
* 4 +----------------+
- * | version | Version of this structure. Current version is 0. New
+ * | version | Version of this structure. Current version is 1. New
* | | versions are guaranteed to be backwards-compatible.
* 8 +----------------+
* | flags | SIF_xxx flags.
@@ -48,6 +48,15 @@
* 32 +----------------+
* | rsdp_paddr | Physical address of the RSDP ACPI data structure.
* 40 +----------------+
+ * | memmap_paddr | Physical address of the (optional) memory map. Only
+ * | | present in version 1 and newer of the structure.
+ * 48 +----------------+
+ * | memmap_entries | Number of entries in the memory map table. Zero
+ * | | if there is no memory map being provided. Only
+ * | | present in version 1 and newer of the structure.
+ * 52 +----------------+
+ * | reserved | Version 1 and newer only.
+ * 56 +----------------+
*
* The layout of each entry in the module structure is the following:
*
@@ -62,13 +71,51 @@
* | reserved |
* 32 +----------------+
*
+ * The layout of each entry in the memory map table is as follows:
+ *
+ * 0 +----------------+
+ * | addr | Base address
+ * 8 +----------------+
+ * | size | Size of mapping in bytes
+ * 16 +----------------+
+ * | type | Type of mapping as defined between the hypervisor
+ * | | and guest. See XEN_HVM_MEMMAP_TYPE_* values below.
+ * 20 +----------------|
+ * | reserved |
+ * 24 +----------------+
+ *
* The address and sizes are always a 64bit little endian unsigned integer.
*
* NB: Xen on x86 will always try to place all the data below the 4GiB
* boundary.
+ *
+ * Version numbers of the hvm_start_info structure have evolved like this:
+ *
+ * Version 0: Initial implementation.
+ *
+ * Version 1: Added the memmap_paddr/memmap_entries fields (plus 4 bytes of
+ * padding) to the end of the hvm_start_info struct. These new
+ * fields can be used to pass a memory map to the guest. The
+ * memory map is optional and so guests that understand version 1
+ * of the structure must check that memmap_entries is non-zero
+ * before trying to read the memory map.
*/
#define XEN_HVM_START_MAGIC_VALUE 0x336ec578

+/*
+ * The values used in the type field of the memory map table entries are
+ * defined below and match the Address Range Types as defined in the "System
+ * Address Map Interfaces" section of the ACPI Specification. Please refer to
+ * section 15 in version 6.2 of the ACPI spec: http://uefi.org/specifications
+ */
+#define XEN_HVM_MEMMAP_TYPE_RAM 1
+#define XEN_HVM_MEMMAP_TYPE_RESERVED 2
+#define XEN_HVM_MEMMAP_TYPE_ACPI 3
+#define XEN_HVM_MEMMAP_TYPE_NVS 4
+#define XEN_HVM_MEMMAP_TYPE_UNUSABLE 5
+#define XEN_HVM_MEMMAP_TYPE_DISABLED 6
+#define XEN_HVM_MEMMAP_TYPE_PMEM 7
+
/*
* C representation of the x86/HVM start info layout.
*
@@ -86,6 +133,13 @@ struct hvm_start_info {
uint64_t cmdline_paddr; /* Physical address of the command line. */
uint64_t rsdp_paddr; /* Physical address of the RSDP ACPI data */
/* structure. */
+ /* All following fields only present in version 1 and newer */
+ uint64_t memmap_paddr; /* Physical address of an array of */
+ /* hvm_memmap_table_entry. */
+ uint32_t memmap_entries; /* Number of entries in the memmap table. */
+ /* Value will be zero if there is no memory */
+ /* map being provided. */
+ uint32_t reserved; /* Must be zero. */
};

struct hvm_modlist_entry {
@@ -95,4 +149,11 @@ struct hvm_modlist_entry {
uint64_t reserved;
};

+struct hvm_memmap_table_entry {
+ uint64_t addr; /* Base address of the memory region */
+ uint64_t size; /* Size of the memory region in bytes */
+ uint32_t type; /* Mapping type */
+ uint32_t reserved; /* Must be zero for Version 1. */
+};
+
#endif /* __XEN_PUBLIC_ARCH_X86_HVM_START_INFO_H__ */
--
2.16.1

2018-12-10 20:09:55

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 7/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

2018-12-10 23:13:46

by Maran Wilson

[permalink] [raw]

Subject: [PATCH v9 2/7] xen/pvh: Move PVH entry code out of Xen specific tree

Once hypervisors other than Xen start using the PVH entry point for
starting VMs, we would like the option of being able to compile PVH entry
capable kernels without enabling CONFIG_XEN and all the code that comes
along with that. To allow that, we are moving the PVH code out of Xen and
into files sitting at a higher level in the tree.

This patch is not introducing any code or functional changes, just moving
files from one location to another.

Signed-off-by: Maran Wilson <[email protected]>
Reviewed-by: Konrad Rzeszutek Wilk <[email protected]>
Reviewed-by: Juergen Gross <[email protected]>
---
MAINTAINERS | 1 +
arch/x86/Kbuild | 2 ++
arch/x86/platform/pvh/Makefile | 5 +++++
arch/x86/{xen/enlighten_pvh.c => platform/pvh/enlighten.c} | 0
arch/x86/{xen/xen-pvh.S => platform/pvh/head.S} | 0
arch/x86/xen/Makefile | 4 ----
6 files changed, 8 insertions(+), 4 deletions(-)
create mode 100644 arch/x86/platform/pvh/Makefile
rename arch/x86/{xen/enlighten_pvh.c => platform/pvh/enlighten.c} (100%)
rename arch/x86/{xen/xen-pvh.S => platform/pvh/head.S} (100%)

diff --git a/MAINTAINERS b/MAINTAINERS
index d27401df091f..477ab97c2062 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16461,6 +16461,7 @@ L: [email protected] (moderated for non-subscribers)
T: git git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git
S: Supported
F: arch/x86/xen/
+F: arch/x86/platform/pvh/
F: drivers/*/xen-*front.c
F: drivers/xen/
F: arch/x86/include/asm/xen/
diff --git a/arch/x86/Kbuild b/arch/x86/Kbuild
index 0038a2d10a7a..2089e4414300 100644
--- a/arch/x86/Kbuild
+++ b/arch/x86/Kbuild
@@ -7,6 +7,8 @@ obj-$(CONFIG_KVM) += kvm/
# Xen paravirtualization support
obj-$(CONFIG_XEN) += xen/

+obj-$(CONFIG_XEN_PVH) += platform/pvh/
+
# Hyper-V paravirtualization support
obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/

diff --git a/arch/x86/platform/pvh/Makefile b/arch/x86/platform/pvh/Makefile
new file mode 100644
index 000000000000..9fd25efcd2a3
--- /dev/null
+++ b/arch/x86/platform/pvh/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+OBJECT_FILES_NON_STANDARD_head.o := y
+
+obj-$(CONFIG_XEN_PVH) += enlighten.o
+obj-$(CONFIG_XEN_PVH) += head.o
diff --git a/arch/x86/xen/enlighten_pvh.c b/arch/x86/platform/pvh/enlighten.c
similarity index 100%
rename from arch/x86/xen/enlighten_pvh.c
rename to arch/x86/platform/pvh/enlighten.c
diff --git a/arch/x86/xen/xen-pvh.S b/arch/x86/platform/pvh/head.S
similarity index 100%
rename from arch/x86/xen/xen-pvh.S
rename to arch/x86/platform/pvh/head.S
diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile
index dd2550d33b38..b239922f6c6c 100644
--- a/arch/x86/xen/Makefile
+++ b/arch/x86/xen/Makefile
@@ -1,6 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
OBJECT_FILES_NON_STANDARD_xen-asm_$(BITS).o := y
-OBJECT_FILES_NON_STANDARD_xen-pvh.o := y

ifdef CONFIG_FUNCTION_TRACER
# Do not profile debug and lowlevel utilities
@@ -37,9 +36,6 @@ obj-$(CONFIG_XEN_PV) += multicalls.o
obj-$(CONFIG_XEN_PV) += xen-asm.o
obj-$(CONFIG_XEN_PV) += xen-asm_$(BITS).o

-obj-$(CONFIG_XEN_PVH) += enlighten_pvh.o
-obj-$(CONFIG_XEN_PVH) += xen-pvh.o
-
obj-$(CONFIG_EVENT_TRACING) += trace.o

obj-$(CONFIG_SMP) += smp.o
--
2.16.1

2018-12-11 13:20:04

by Borislav Petkov

[permalink] [raw]

Subject: Re: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

On Mon, Dec 10, 2018 at 11:05:34AM -0800, Maran Wilson wrote:
> For certain applications it is desirable to rapidly boot a KVM virtual
> machine. In cases where legacy hardware and software support within the
> guest is not needed, Qemu should be able to boot directly into the
> uncompressed Linux kernel binary without the need to run firmware.
>
> There already exists an ABI to allow this for Xen PVH guests and the ABI
> is supported by Linux and FreeBSD:
>
> https://xenbits.xen.org/docs/unstable/misc/pvh.html
>
> This patch series would enable Qemu to use that same entry point for
> booting KVM guests.

How would I do that, practically?

Looking at those here:

> * Qemu and qboot RFC patches have been posted to show one example of how
> this functionality can be used. Some preliminary numbers are available
> in those cover letters showing the KVM guest boot time improvement.
> Qemu:
> http://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00957.html
> qboot:
> http://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00953.html

I might still need to do some dancing to get stuff going.

Thx.

--
Regards/Gruss,
Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

2018-12-11 13:23:46

by Borislav Petkov

[permalink] [raw]

Subject: Re: [PATCH v9 1/7] xen/pvh: Split CONFIG_XEN_PVH into CONFIG_PVH and CONFIG_XEN_PVH

On Mon, Dec 10, 2018 at 11:07:28AM -0800, Maran Wilson wrote:
> In order to pave the way for hypervisors other than Xen to use the PVH
> entry point for VMs, we need to factor the PVH entry code into Xen specific
> and hypervisor agnostic components. The first step in doing that, is to
> create a new config option for PVH entry that can be enabled
> independently from CONFIG_XEN.
>
> Signed-off-by: Maran Wilson <[email protected]>
> Reviewed-by: Juergen Gross <[email protected]>
> ---
> arch/x86/Kconfig | 6 ++++++
> arch/x86/kernel/head_64.S | 2 +-
> arch/x86/xen/Kconfig | 3 ++-
> 3 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 8689e794a43c..c2a22a74abee 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -796,6 +796,12 @@ config KVM_GUEST
> underlying device model, the host provides the guest with
> timing infrastructure such as time of day, and system time
>
> +config PVH
> + bool "Support for running PVH guests"
> + ---help---
> + This option enables the PVH entry point for guest virtual machines
> + as specified in the x86/HVM direct boot ABI.
> +
> config KVM_DEBUG_FS
> bool "Enable debug information for KVM Guests in debugfs"
> depends on KVM_GUEST && DEBUG_FS
> diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
> index 747c758f67b7..d1dbe8e4eb82 100644
> --- a/arch/x86/kernel/head_64.S
> +++ b/arch/x86/kernel/head_64.S
> @@ -386,7 +386,7 @@ NEXT_PAGE(early_dynamic_pgts)
>
> .data
>
> -#if defined(CONFIG_XEN_PV) || defined(CONFIG_XEN_PVH)
> +#if defined(CONFIG_XEN_PV) || defined(CONFIG_PVH)
> NEXT_PGD_PAGE(init_top_pgt)
> .quad level3_ident_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC
> .org init_top_pgt + L4_PAGE_OFFSET*8, 0
> diff --git a/arch/x86/xen/Kconfig b/arch/x86/xen/Kconfig
> index 1ef391aa184d..e07abefd3d26 100644
> --- a/arch/x86/xen/Kconfig
> +++ b/arch/x86/xen/Kconfig
> @@ -74,6 +74,7 @@ config XEN_DEBUG_FS
> Enabling this option may incur a significant performance overhead.
>
> config XEN_PVH
> - bool "Support for running as a PVH guest"
> + bool "Support for running as a Xen PVH guest"
> depends on XEN && XEN_PVHVM && ACPI
> + select PVH
> def_bool n
> --

LGTM:

Acked-by: Borislav Petkov <[email protected]>

--
Regards/Gruss,
Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

2018-12-11 19:32:46

by Maran Wilson

[permalink] [raw]

Subject: Re: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

On 12/11/2018 5:18 AM, Borislav Petkov wrote:
> On Mon, Dec 10, 2018 at 11:05:34AM -0800, Maran Wilson wrote:
>> For certain applications it is desirable to rapidly boot a KVM virtual
>> machine. In cases where legacy hardware and software support within the
>> guest is not needed, Qemu should be able to boot directly into the
>> uncompressed Linux kernel binary without the need to run firmware.
>>
>> There already exists an ABI to allow this for Xen PVH guests and the ABI
>> is supported by Linux and FreeBSD:
>>
>> https://xenbits.xen.org/docs/unstable/misc/pvh.html
>>
>> This patch series would enable Qemu to use that same entry point for
>> booting KVM guests.
> How would I do that, practically?
>
> Looking at those here:
>
>> * Qemu and qboot RFC patches have been posted to show one example of how
>> this functionality can be used. Some preliminary numbers are available
>> in those cover letters showing the KVM guest boot time improvement.
>> Qemu:
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00957.html
>> qboot:
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-12/msg00953.html
> I might still need to do some dancing to get stuff going.

Is your question about what options you need to provide to Qemu? Or is
your question about the SW implementation choices?

Assuming the former... once you have compiled all 3 new binaries
(kernel, Qemu, and qboot) then you simply invoke qemu the same way you
normally invoke qemu with qboot + kernel binary, except you provide the
vmlinux (uncompressed) kernel binary when specifying the "-kernel"
parameter. Qemu/qboot will automatically detect that you have provided
an ELF binary, find the PVH ELF note to locate the entry point, and
proceed to boot the kernel via that method. On the other hand, if you
leave all the Qemu options as-is, but simply provide the bzImage
(compressed) kernel binary from the same build, Qemu/qboot will boot the
way it has always done and not look for PVH.

To make it more concrete, here's an example of how I had been invoking
PVH boot recently:

   x86_64-softmmu/qemu-system-x86_64 \
     -name testvm01 \
     -machine q35,accel=kvm,nvdimm \
     -cpu host \
     -m 1024,maxmem=20G,slots=2 \
     -smp 1 \
     -nodefaults \
     -kernel binaries/vmlinux \
     -object
memory-backend-file,id=mem0,share,mem-path=binaries/containers.img,size=235929600
\
     -device nvdimm,memdev=mem0,id=nv0 \
     -append 'console=ttyS0,115200,8n1 root=/dev/pmem0p1 panic=1 rw
init=/usr/lib/systemd/systemd rootfstype=ext4' \
     -bios binaries/bios.bin \
     -serial mon:stdio

Thanks,
-Maran

> Thx.
>

2018-12-12 20:40:35

by Borislav Petkov

[permalink] [raw]

Subject: Re: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

On Tue, Dec 11, 2018 at 11:29:21AM -0800, Maran Wilson wrote:
> Is your question about what options you need to provide to Qemu? Or is your
> question about the SW implementation choices?
>
> Assuming the former...

Yeah, that's what I wanted to know. But looking at it, I'm booting
bzImage here just as quickly and as flexible so I don't see the
advantage of this new method for my use case here of booting kernels
in qemu.

But maybe there's a good use case where firmware is slow and one doesn't
really wanna noodle through it or when one does start a gazillion VMs
per second or whatever...

Thx.

--
Regards/Gruss,
Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

2018-12-12 21:59:03

by Maran Wilson

[permalink] [raw]

Subject: Re: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

On 12/12/2018 12:39 PM, Borislav Petkov wrote:
> On Tue, Dec 11, 2018 at 11:29:21AM -0800, Maran Wilson wrote:
>> Is your question about what options you need to provide to Qemu? Or is your
>> question about the SW implementation choices?
>>
>> Assuming the former...
> Yeah, that's what I wanted to know. But looking at it, I'm booting
> bzImage here just as quickly and as flexible so I don't see the
> advantage of this new method for my use case here of booting kernels
> in qemu.
>
> But maybe there's a good use case where firmware is slow and one doesn't
> really wanna noodle through it or when one does start a gazillion VMs
> per second or whatever...

Right, the time saved is not something you would notice while starting a
VM manually. But it does reduce the time to reach startup_64() in Linux
by about 50% (going from around 94ms to around 47ms) when booting a VM
using Qemu+qboot (for example). That time savings becomes pretty
important when you are trying to use VMs as containers (for instance, as
is the case with Kata containers) and trying to get the latency for
launching such a container really low -- to come as close as possible to
match the latency for launching more traditional containers that don't
have the additional security/isolation of running within a separate VM.

Thanks,
-Maran

>
> Thx.
>

2018-12-13 13:18:01

by Paolo Bonzini

[permalink] [raw]

Subject: Re: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

On 12/12/18 21:39, Borislav Petkov wrote:
> On Tue, Dec 11, 2018 at 11:29:21AM -0800, Maran Wilson wrote:
>> Is your question about what options you need to provide to Qemu? Or is your
>> question about the SW implementation choices?
>>
>> Assuming the former...
> Yeah, that's what I wanted to know. But looking at it, I'm booting
> bzImage here just as quickly and as flexible so I don't see the
> advantage of this new method for my use case here of booting kernels
> in qemu.

It's not firmware that is slow, decompression is. Unlike Xen, which is
using PVH with a regular bzImage and decompression in the host, KVM is
using PVH to boot a vmlinux with no decompression at all.

Paolo

> But maybe there's a good use case where firmware is slow and one doesn't
> really wanna noodle through it or when one does start a gazillion VMs
> per second or whatever...

2018-12-14 23:15:23

by Boris Ostrovsky

[permalink] [raw]

Subject: Re: [PATCH v9 0/7] KVM: x86: Allow Qemu/KVM to use PVH entry point

On 12/10/18 2:05 PM, Maran Wilson wrote:
> For certain applications it is desirable to rapidly boot a KVM virtual
> machine. In cases where legacy hardware and software support within the
> guest is not needed, Qemu should be able to boot directly into the
> uncompressed Linux kernel binary without the need to run firmware.
>
> There already exists an ABI to allow this for Xen PVH guests and the ABI
> is supported by Linux and FreeBSD:
>
> https://xenbits.xen.org/docs/unstable/misc/pvh.html
>
> This patch series would enable Qemu to use that same entry point for
> booting KVM guests.
>

Applied to for-linus-4.21

-boris