This series paves the way to run Linux in so-called non-root cells
(guest partitions) of the Jailhouse hypervisor.
Jailhouse [1] was started 4 years ago as an open-source (GPL) leight-
weight hypervisor that statically partitions SMP systems. It's unique in
that it uses one Linux instance, the root cell, as boot loader and
management console. Jailhouse targets use cases for hard real-time and
safety-critical systems that KVM cannot cater due to its inherent
complexity.
Jaihouse can run bare-metal, free and closed-source RTOSes as secondary
guests and, with this series, also x86 Linux instances. While ARM and
ARM64 non-root Linux guests are feasible without extra patches, thanks
to the high configurability via device trees, x86 requires special
platform support, mostly to step away from non-existing resources in a
non-root Jailhouse cell.
This series ensures that Linux can boot in a non-root cell, including
SMP cells, has working timekeeping and can use the platform UARTs and
PCI devices as assigned to it. In follow-up series, we will propose
optimizations and enhancements for the PCI support, a simplistic debug
console, and some improvement for Linux guests on ARM.
What is not yet in upstream-ready state is a driver for inter-cell
communication. The current implementation of virtual peer-to-peer
network [2] uses an enhanced version of the QEMU ivshmem shared memory
device. However we still need to finish the evaluation of virtio /
vhost-pci options prior to settling over the final interface.
This patch series is also available at
git://git.kiszka.org/linux.git d0036688b2da
Jan
[1] http://jailhouse-project.org
[2] http://git.kiszka.org/?p=linux.git;a=shortlog;h=refs/heads/queues/jailhouse
Jan Kiszka (10):
x86/apic: Install an empty physflat_init_apic_ldr
x86: jailhouse: Add infrastructure for running in non-root cell
x86: jailhouse: Enable APIC and SMP support
x86: jailhouse: Enable PMTIMER
x86: jailhouse: Set up timekeeping
x86: jailhouse: Avoid access of unsupported platform resources
x86: jailhouse: Silence ACPI warning
x86: jailhouse: Halt instead of failing to restart
x86: jailhouse: Wire up IOAPIC for legacy UART ports
x86: jailhouse: Initialize PCI support
arch/x86/Kconfig | 13 ++
arch/x86/include/asm/hypervisor.h | 1 +
arch/x86/include/asm/jailhouse_para.h | 27 ++++
arch/x86/include/asm/tsc.h | 3 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/apic/apic_flat_64.c | 12 +-
arch/x86/kernel/cpu/hypervisor.c | 4 +
arch/x86/kernel/jailhouse.c | 252 ++++++++++++++++++++++++++++++++++
arch/x86/kernel/smpboot.c | 7 +-
arch/x86/kernel/tsc.c | 14 +-
drivers/acpi/Kconfig | 32 ++---
11 files changed, 339 insertions(+), 28 deletions(-)
create mode 100644 arch/x86/include/asm/jailhouse_para.h
create mode 100644 arch/x86/kernel/jailhouse.c
--
2.12.3
From 1584566414501008875@xxx Mon Nov 20 06:44:52 +0000 2017
X-GM-THRID: 1584391330074124598
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
As the comment already stated, there is no need for setting up LDR in
physflat mode as it remains unused. flat_init_apic_ldr only served as a
placeholder for a nop operation so far, causing no harm.
That will change when running over the Jailhouse hypervisor. Here we
must not touch LDR in a way that destroys the mapping originally set up
by the Linux root cell. Jailhouse enforces this setting in order to
efficiently validate any IPI requests sent by a cell.
Avoid a needless clash caused by flat_init_apic_ldr by installing a true
nop handler.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/kernel/apic/apic_flat_64.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/apic/apic_flat_64.c b/arch/x86/kernel/apic/apic_flat_64.c
index aa85690e9b64..34bfdfe29a04 100644
--- a/arch/x86/kernel/apic/apic_flat_64.c
+++ b/arch/x86/kernel/apic/apic_flat_64.c
@@ -218,6 +218,11 @@ static int physflat_acpi_madt_oem_check(char *oem_id, char *oem_table_id)
return 0;
}
+void physflat_init_apic_ldr(void)
+{
+ /* LDR is not used in physflat mode. */
+}
+
static void physflat_send_IPI_allbutself(int vector)
{
default_send_IPI_mask_allbutself_phys(cpu_online_mask, vector);
@@ -251,8 +256,7 @@ static struct apic apic_physflat __ro_after_init = {
.dest_logical = 0,
.check_apicid_used = NULL,
- /* not needed, but shouldn't hurt: */
- .init_apic_ldr = flat_init_apic_ldr,
+ .init_apic_ldr = physflat_init_apic_ldr,
.ioapic_phys_id_map = NULL,
.setup_apic_routing = NULL,
--
2.12.3
From 1584415688649466806@xxx Sat Nov 18 14:49:08 +0000 2017
X-GM-THRID: 1584343153541789422
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
We don't have CMOS access, thus we can't set the warm-reset vectors in
do_boot_cpu. There is no RTC, thus also no wall clock. Furthermore,
there are no ISA IRQs and no PIC. So fill the platform callbacks
accordingly.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/kernel/jailhouse.c | 10 ++++++++++
arch/x86/kernel/smpboot.c | 7 +++++--
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index 8e5b2f0c8a34..cfe8ae0c33a2 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -15,6 +15,7 @@
#include <asm/apic.h>
#include <asm/cpu.h>
#include <asm/hypervisor.h>
+#include <asm/i8259.h>
#include <asm/setup.h>
#define SETUP_JAILHOUSE 0x53484c4a /* "JLHS" */
@@ -50,6 +51,11 @@ static uint32_t __init jailhouse_detect(void)
return jailhouse_cpuid_base();
}
+static void jailhouse_get_wallclock(struct timespec *now)
+{
+ memset(now, 0, sizeof(*now));
+}
+
#define MAX_RETRIES 5
#define SMI_TRESHOLD 50000
@@ -139,7 +145,11 @@ static void __init jailhouse_init_platform(void)
unsigned int cpu;
x86_init.timers.timer_init = jailhouse_timer_init;
+ x86_init.irqs.pre_vector_init = x86_init_noop;
+ legacy_pic = &null_legacy_pic;
+ x86_platform.legacy.rtc = 0;
+ x86_platform.get_wallclock = jailhouse_get_wallclock;
x86_platform.calibrate_cpu = jailhouse_calibrate_cpu;
x86_platform.calibrate_tsc = jailhouse_calibrate_tsc;
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index 5f59e6bee123..81339dbafeba 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -78,6 +78,7 @@
#include <asm/realmode.h>
#include <asm/misc.h>
#include <asm/qspinlock.h>
+#include <asm/jailhouse_para.h>
/* Number of siblings per CPU package */
int smp_num_siblings = 1;
@@ -1006,7 +1007,8 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle,
* the targeted processor.
*/
- if (get_uv_system_type() != UV_NON_UNIQUE_APIC) {
+ if (get_uv_system_type() != UV_NON_UNIQUE_APIC &&
+ !jailhouse_paravirt()) {
pr_debug("Setting warm reset code and vector.\n");
@@ -1078,7 +1080,8 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle,
/* mark "stuck" area as not stuck */
*trampoline_status = 0;
- if (get_uv_system_type() != UV_NON_UNIQUE_APIC) {
+ if (get_uv_system_type() != UV_NON_UNIQUE_APIC &&
+ !jailhouse_paravirt()) {
/*
* Cleanup possible dangling ends...
*/
--
2.12.3
From 1584408331781905432@xxx Sat Nov 18 12:52:12 +0000 2017
X-GM-THRID: 1584408331781905432
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
Jailhouse exposes the PMTIMER as only reference clock to all cells. Pick
up its address from the setup data. Allow to enable the Linux support of
it by relaxing its strict dependency on ACPI.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/Kconfig | 1 +
arch/x86/kernel/jailhouse.c | 4 ++++
drivers/acpi/Kconfig | 32 ++++++++++++++++----------------
3 files changed, 21 insertions(+), 16 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index c5f4f4683b51..6976c035ea63 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -785,6 +785,7 @@ config KVM_DEBUG_FS
config JAILHOUSE_GUEST
bool "Jailhouse non-root cell support"
depends on PARAVIRT && X86_64
+ select X86_PM_TIMER
---help---
This option allows to run Linux as guest in a Jailhouse non-root
cell. You can leave this option disabled if you only want to start
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index ce9416c70656..f7e99f7a8873 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -10,6 +10,7 @@
* the COPYING file in the top-level directory.
*/
+#include <linux/acpi_pmtmr.h>
#include <linux/kernel.h>
#include <asm/apic.h>
#include <asm/cpu.h>
@@ -66,6 +67,9 @@ static void __init jailhouse_init_platform(void)
data->compatible_version > SETUP_REQUIRED_VERSION)
panic("Jailhouse: Unsupported setup data structure");
+ pmtmr_ioport = data->pm_timer_address;
+ pr_debug("Jailhouse: PM-Timer IO Port: %#x\n", pmtmr_ioport);
+
#ifdef CONFIG_X86_X2APIC
/*
* Register x2APIC handlers early. We need them when running
diff --git a/drivers/acpi/Kconfig b/drivers/acpi/Kconfig
index 91477d5ab422..c054f9b4f1eb 100644
--- a/drivers/acpi/Kconfig
+++ b/drivers/acpi/Kconfig
@@ -361,22 +361,6 @@ config ACPI_PCI_SLOT
i.e., segment/bus/device/function tuples, with physical slots in
the system. If you are unsure, say N.
-config X86_PM_TIMER
- bool "Power Management Timer Support" if EXPERT
- depends on X86
- default y
- help
- The Power Management Timer is available on all ACPI-capable,
- in most cases even if ACPI is unusable or blacklisted.
-
- This timing source is not affected by power management features
- like aggressive processor idling, throttling, frequency and/or
- voltage scaling, unlike the commonly used Time Stamp Counter
- (TSC) timing source.
-
- You should nearly always say Y here because many modern
- systems require this timer.
-
config ACPI_CONTAINER
bool "Container and Module Devices"
default (ACPI_HOTPLUG_MEMORY || ACPI_HOTPLUG_CPU)
@@ -558,3 +542,19 @@ config TPS68470_PMIC_OPREGION
using this, are probed.
endif # ACPI
+
+config X86_PM_TIMER
+ bool "Power Management Timer Support" if EXPERT
+ depends on X86 && (ACPI || JAILHOUSE_GUEST)
+ default y
+ help
+ The Power Management Timer is available on all ACPI-capable,
+ in most cases even if ACPI is unusable or blacklisted.
+
+ This timing source is not affected by power management features
+ like aggressive processor idling, throttling, frequency and/or
+ voltage scaling, unlike the commonly used Time Stamp Counter
+ (TSC) timing source.
+
+ You should nearly always say Y here because many modern
+ systems require this timer.
--
2.12.3
From 1584244991736406773@xxx Thu Nov 16 17:35:59 +0000 2017
X-GM-THRID: 1584176828413681691
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
With this change, PCI devices can be detected and used inside a non-root
cell.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/kernel/jailhouse.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index 05459ea0ecc7..e573ea8b0a5f 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -18,6 +18,7 @@
#include <asm/hypervisor.h>
#include <asm/i8259.h>
#include <asm/irqdomain.h>
+#include <asm/pci_x86.h>
#include <asm/reboot.h>
#include <asm/setup.h>
@@ -211,6 +212,15 @@ static void __init jailhouse_init_platform(void)
early_memunmap(data, sizeof(*data));
+ pci_probe = 0;
+ pci_direct_init(1);
+
+ /*
+ * There are no bridges on the virtual PCI root bus under Jailhouse,
+ * thus no other way to discover all devices than a full scan.
+ */
+ pcibios_last_bus = 0xff;
+
/*
* Avoid that the kernel complains about missing ACPI tables - there
* are none in a non-root cell.
--
2.12.3
From 1585257392045251109@xxx Mon Nov 27 21:47:39 +0000 2017
X-GM-THRID: 1585257392045251109
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
Jailhouse provides no guest-initiated restart. So, do not even try to.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/kernel/jailhouse.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index c1ed7fcd7bfa..0bcca175c35e 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -12,10 +12,12 @@
#include <linux/acpi_pmtmr.h>
#include <linux/kernel.h>
+#include <linux/reboot.h>
#include <asm/apic.h>
#include <asm/cpu.h>
#include <asm/hypervisor.h>
#include <asm/i8259.h>
+#include <asm/reboot.h>
#include <asm/setup.h>
#define SETUP_JAILHOUSE 0x53484c4a /* "JLHS" */
@@ -133,6 +135,12 @@ static unsigned long jailhouse_calibrate_tsc(void)
return 0;
}
+static void jailhouse_no_restart(void)
+{
+ pr_notice("Jailhouse: Restart not supported, halting\n");
+ machine_halt();
+}
+
static unsigned int x2apic_get_apic_id(unsigned long id)
{
return id;
@@ -153,6 +161,8 @@ static void __init jailhouse_init_platform(void)
x86_platform.calibrate_cpu = jailhouse_calibrate_cpu;
x86_platform.calibrate_tsc = jailhouse_calibrate_tsc;
+ machine_ops.emergency_restart = jailhouse_no_restart;
+
data = early_memremap(pa_data, sizeof(*data));
if (data->header.type != SETUP_JAILHOUSE ||
--
2.12.3
From 1584194694096903505@xxx Thu Nov 16 04:16:31 +0000 2017
X-GM-THRID: 1584194694096903505
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
Jailhouse support does not depend on ACPI, and does not even use it. But
if it should be enabled, avoid warning about its absence in the
platform.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/kernel/jailhouse.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index cfe8ae0c33a2..c1ed7fcd7bfa 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -180,6 +180,12 @@ static void __init jailhouse_init_platform(void)
smp_found_config = 1;
early_memunmap(data, sizeof(*data));
+
+ /*
+ * Avoid that the kernel complains about missing ACPI tables - there
+ * are none in a non-root cell.
+ */
+ disable_acpi();
}
bool jailhouse_paravirt(void)
--
2.12.3
From 1584190889777553464@xxx Thu Nov 16 03:16:03 +0000 2017
X-GM-THRID: 1584190889777553464
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread
From: Jan Kiszka <[email protected]>
The typical I/O interrupts in non-root cells are MSI-based. However, the
platform UARTs do not support MSI. In order to run a non-root cell that
shall be use one of them, we need to register the standard IOAPIC and
set 1:1 routing for IRQ 3 and 4.
If an IOAPIC is not available, the boot loader clears standard_ioapic in
the setup data, and we skip the registration. If we should not be
allowed to use one of those pins, Jailhouse will simply ignore our
accesses.
Signed-off-by: Jan Kiszka <[email protected]>
---
arch/x86/kernel/jailhouse.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
index 0bcca175c35e..05459ea0ecc7 100644
--- a/arch/x86/kernel/jailhouse.c
+++ b/arch/x86/kernel/jailhouse.c
@@ -17,6 +17,7 @@
#include <asm/cpu.h>
#include <asm/hypervisor.h>
#include <asm/i8259.h>
+#include <asm/irqdomain.h>
#include <asm/reboot.h>
#include <asm/setup.h>
@@ -148,6 +149,14 @@ static unsigned int x2apic_get_apic_id(unsigned long id)
static void __init jailhouse_init_platform(void)
{
+ struct ioapic_domain_cfg ioapic_cfg = {
+ .type = IOAPIC_DOMAIN_STRICT,
+ .ops = &mp_ioapic_irqdomain_ops,
+ };
+ struct mpc_intsrc mp_irq = {
+ .type = MP_INTSRC,
+ .irqtype = mp_INT,
+ };
u64 pa_data = boot_params.hdr.setup_data;
struct jailhouse_setup_data *data;
unsigned int cpu;
@@ -189,6 +198,17 @@ static void __init jailhouse_init_platform(void)
boot_cpu_apic_version);
smp_found_config = 1;
+ if (data->standard_ioapic) {
+ mp_register_ioapic(0, 0xfec00000, gsi_top, &ioapic_cfg);
+
+ /* Register 1:1 mapping for legacy UART IRQs 3 and 4 */
+ mp_irq.srcbusirq = mp_irq.dstirq = 3;
+ mp_save_irq(&mp_irq);
+
+ mp_irq.srcbusirq = mp_irq.dstirq = 4;
+ mp_save_irq(&mp_irq);
+ }
+
early_memunmap(data, sizeof(*data));
/*
--
2.12.3
From 1584280105999469318@xxx Fri Nov 17 02:54:07 +0000 2017
X-GM-THRID: 1584280105999469318
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread