On Mon, Nov 27, 2017 at 09:11:54AM +0100, Jan Kiszka wrote:
> From: Jan Kiszka <[email protected]>
>
> With this change, PCI devices can be detected and used inside a non-root
> cell.
>
> Signed-off-by: Jan Kiszka <[email protected]>
> ---
> arch/x86/kernel/jailhouse.c | 17 +++++++++++++++++
> 1 file changed, 17 insertions(+)
>
> diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
> index 8ff21e1534de..70b857d4b1f5 100644
> --- a/arch/x86/kernel/jailhouse.c
> +++ b/arch/x86/kernel/jailhouse.c
> @@ -18,6 +18,7 @@
> #include <asm/hypervisor.h>
> #include <asm/i8259.h>
> #include <asm/irqdomain.h>
> +#include <asm/pci_x86.h>
> #include <asm/reboot.h>
> #include <asm/setup.h>
>
> @@ -108,6 +109,19 @@ static void jailhouse_no_restart(void)
> machine_halt();
> }
>
> +static int __init jailhouse_pci_arch_init(void)
> +{
> + pci_direct_init(1);
> +
> + /*
> + * There are no bridges on the virtual PCI root bus under Jailhouse,
> + * thus no other way to discover all devices than a full scan.
> + */
> + pcibios_last_bus = 0xff;
Can you help me understand the comment here? If the virtual root bus
is bus 00, are you saying the guest might see devices on bus 00 and
bus 01, with no bus 00 bridge that leads to bus 01?
I suspect you mean something different because you say elsewhere that
ARM "just works" because DT provides more configurability. But even
on ARM with DT, we probe the root bus and only probe other buses when
we find bridges leading to them.
So I suspect the purpose of this may be to discover devices that are
below host bridges not exposed by ACPI. For example, my BIOS may
expose one host bridge:
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
but the chipset may implement devices on bus 7f even though the BIOS
did not advertise the host bridge leading to that bus. This is a case
of a missing host bridge, not a missing bridge on the root bus.
Can you show an example "lspci -v" output to make this concrete?
> + return 0;
> +}
> +
> static void __init jailhouse_init_platform(void)
> {
> u64 pa_data = boot_params.hdr.setup_data;
> @@ -117,6 +131,7 @@ static void __init jailhouse_init_platform(void)
> x86_init.irqs.pre_vector_init = x86_init_noop;
> x86_init.timers.timer_init = jailhouse_timer_init;
> x86_init.mpparse.get_smp_config = jailhouse_get_smp_config;
> + x86_init.pci.arch_init = jailhouse_pci_arch_init;
>
> x86_platform.calibrate_cpu = jailhouse_get_tsc;
> x86_platform.calibrate_tsc = jailhouse_get_tsc;
> @@ -159,6 +174,8 @@ static void __init jailhouse_init_platform(void)
>
> precalibrated_tsc_khz = setup_data.tsc_khz;
>
> + pci_probe = 0;
> +
> /*
> * Avoid that the kernel complains about missing ACPI tables - there
> * are none in a non-root cell.
> --
> 2.12.3
>
On 2018-01-22 23:26, Bjorn Helgaas wrote:
> On Mon, Nov 27, 2017 at 09:11:54AM +0100, Jan Kiszka wrote:
>> From: Jan Kiszka <[email protected]>
>>
>> With this change, PCI devices can be detected and used inside a non-root
>> cell.
>>
>> Signed-off-by: Jan Kiszka <[email protected]>
>> ---
>> arch/x86/kernel/jailhouse.c | 17 +++++++++++++++++
>> 1 file changed, 17 insertions(+)
>>
>> diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
>> index 8ff21e1534de..70b857d4b1f5 100644
>> --- a/arch/x86/kernel/jailhouse.c
>> +++ b/arch/x86/kernel/jailhouse.c
>> @@ -18,6 +18,7 @@
>> #include <asm/hypervisor.h>
>> #include <asm/i8259.h>
>> #include <asm/irqdomain.h>
>> +#include <asm/pci_x86.h>
>> #include <asm/reboot.h>
>> #include <asm/setup.h>
>>
>> @@ -108,6 +109,19 @@ static void jailhouse_no_restart(void)
>> machine_halt();
>> }
>>
>> +static int __init jailhouse_pci_arch_init(void)
>> +{
>> + pci_direct_init(1);
>> +
>> + /*
>> + * There are no bridges on the virtual PCI root bus under Jailhouse,
>> + * thus no other way to discover all devices than a full scan.
>> + */
>> + pcibios_last_bus = 0xff;
>
> Can you help me understand the comment here? If the virtual root bus
> is bus 00, are you saying the guest might see devices on bus 00 and
> bus 01, with no bus 00 bridge that leads to bus 01?
Exactly, and there is even no root bridge for bus 0 as well. We just
hand out the devices. We try hard to keep it that simple in order to
avoid emulation bloat in the hypervisor (which < 9K LoC on Intel right now).
>
> I suspect you mean something different because you say elsewhere that
> ARM "just works" because DT provides more configurability. But even
> on ARM with DT, we probe the root bus and only probe other buses when
> we find bridges leading to them.
This statement about ARM probably does not apply to incomplete PCI
hierarchy. I guess we cannot fill that gap with device tree
descriptions, but I also didn't run into that scenario yet as most ARM
systems we tested do not even allow to partition PCI buses (many IOMMU
implementations on SoCs, where available at all, consider the root
complex as single device).
>
> So I suspect the purpose of this may be to discover devices that are
> below host bridges not exposed by ACPI. For example, my BIOS may
> expose one host bridge:
>
> ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
>
> but the chipset may implement devices on bus 7f even though the BIOS
> did not advertise the host bridge leading to that bus. This is a case
> of a missing host bridge, not a missing bridge on the root bus.
>
> Can you show an example "lspci -v" output to make this concrete?
We we go, using a QEMU/KVM setup
(... -device pci-bridge,chassis_nr=1,id=b1
-device e1000e,addr=1.0,netdev=net,bus=b1):
# lspci -v
00:0e.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: bus master, fast devsel, latency 0
Memory at 100000000 (64-bit, non-prefetchable) [size=256]
Memory at 100000100 (64-bit, non-prefetchable) [size=32]
Capabilities: [50] MSI-X: Enable+ Count=1 Masked-
Kernel driver in use: ivshmem-net
01:01.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
Subsystem: Intel Corporation Device 0000
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at fe840000 (32-bit, non-prefetchable) [size=128K]
Memory at fe860000 (32-bit, non-prefetchable) [size=128K]
I/O ports at c000 [disabled] [size=32]
Memory at fe880000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 52-54-00-ff-ff-12-34-56
Kernel driver in use: e1000e
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux