Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH v2 12/12] x86/jailhouse: Initialize PCI support
To:     Bjorn Helgaas <helgaas@kernel.org>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>,
        "H . Peter Anvin" <hpa@zytor.com>, x86@kernel.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        jailhouse-dev@googlegroups.com
References: <cover.1511770314.git.jan.kiszka@siemens.com>
 <e8d19494b96b68a749bcac514795d864ad9c28c3.1511770314.git.jan.kiszka@siemens.com>
 <20180122222610.GA197931@bhelgaas-glaptop.roam.corp.google.com>
From:   Jan Kiszka <jan.kiszka@siemens.com>
Message-ID: <21646cd4-8ea4-2c1c-723d-9734364d0bab@siemens.com>
Date:   Tue, 23 Jan 2018 08:49:50 +0100
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12)
 Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666
MIME-Version: 1.0
In-Reply-To: <20180122222610.GA197931@bhelgaas-glaptop.roam.corp.google.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2018-01-22 23:26, Bjorn Helgaas wrote:
> On Mon, Nov 27, 2017 at 09:11:54AM +0100, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> With this change, PCI devices can be detected and used inside a non-root
>> cell.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  arch/x86/kernel/jailhouse.c | 17 +++++++++++++++++
>>  1 file changed, 17 insertions(+)
>>
>> diff --git a/arch/x86/kernel/jailhouse.c b/arch/x86/kernel/jailhouse.c
>> index 8ff21e1534de..70b857d4b1f5 100644
>> --- a/arch/x86/kernel/jailhouse.c
>> +++ b/arch/x86/kernel/jailhouse.c
>> @@ -18,6 +18,7 @@
>>  #include <asm/hypervisor.h>
>>  #include <asm/i8259.h>
>>  #include <asm/irqdomain.h>
>> +#include <asm/pci_x86.h>
>>  #include <asm/reboot.h>
>>  #include <asm/setup.h>
>>  
>> @@ -108,6 +109,19 @@ static void jailhouse_no_restart(void)
>>  	machine_halt();
>>  }
>>  
>> +static int __init jailhouse_pci_arch_init(void)
>> +{
>> +	pci_direct_init(1);
>> +
>> +	/*
>> +	 * There are no bridges on the virtual PCI root bus under Jailhouse,
>> +	 * thus no other way to discover all devices than a full scan.
>> +	 */
>> +	pcibios_last_bus = 0xff;
> 
> Can you help me understand the comment here?  If the virtual root bus
> is bus 00, are you saying the guest might see devices on bus 00 and
> bus 01, with no bus 00 bridge that leads to bus 01?

Exactly, and there is even no root bridge for bus 0 as well. We just
hand out the devices. We try hard to keep it that simple in order to
avoid emulation bloat in the hypervisor (which < 9K LoC on Intel right now).

> 
> I suspect you mean something different because you say elsewhere that
> ARM "just works" because DT provides more configurability.  But even
> on ARM with DT, we probe the root bus and only probe other buses when
> we find bridges leading to them.

This statement about ARM probably does not apply to incomplete PCI
hierarchy. I guess we cannot fill that gap with device tree
descriptions, but I also didn't run into that scenario yet as most ARM
systems we tested do not even allow to partition PCI buses (many IOMMU
implementations on SoCs, where available at all, consider the root
complex as single device).

> 
> So I suspect the purpose of this may be to discover devices that are
> below host bridges not exposed by ACPI.  For example, my BIOS may
> expose one host bridge:
> 
>   ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
> 
> but the chipset may implement devices on bus 7f even though the BIOS
> did not advertise the host bridge leading to that bus.  This is a case
> of a missing host bridge, not a missing bridge on the root bus.
> 
> Can you show an example "lspci -v" output to make this concrete?

We we go, using a QEMU/KVM setup
(... -device pci-bridge,chassis_nr=1,id=b1
-device e1000e,addr=1.0,netdev=net,bus=b1):

# lspci -v
00:0e.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
        Subsystem: Red Hat, Inc Inter-VM shared memory
        Flags: bus master, fast devsel, latency 0
        Memory at 100000000 (64-bit, non-prefetchable) [size=256]
        Memory at 100000100 (64-bit, non-prefetchable) [size=32]
        Capabilities: [50] MSI-X: Enable+ Count=1 Masked-
        Kernel driver in use: ivshmem-net

01:01.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
        Subsystem: Intel Corporation Device 0000
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at fe840000 (32-bit, non-prefetchable) [size=128K]
        Memory at fe860000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at c000 [disabled] [size=32]
        Memory at fe880000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 52-54-00-ff-ff-12-34-56
        Kernel driver in use: e1000e

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux