Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Message-ID: <1269bf62-d67f-9f61-0139-dc20d23a9b5e@ti.com>
Date:   Tue, 26 Sep 2023 18:10:36 +0530
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.0
Subject: Re: [RFC] Proposal of QEMU PCI Endpoint test environment
To:     Shunsuke Mie <mie@igel.co.jp>,
        Kishon Vijay Abraham I <kvijayab@amd.com>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        "Michael S. Tsirkin" <mst@redhat.com>
CC:     Paolo Bonzini <pbonzini@redhat.com>,
        Marcel Apfelbaum <marcel.apfelbaum@gmail.com>,
        <qemu-devel@nongnu.org>, Rob Herring <robh@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        <linux-pci@vger.kernel.org>,
        =?UTF-8?Q?Krzysztof_Wilczy=c5=84ski?= <kw@linux.com>,
        Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>,
        Kishon Vijay Abraham I <kishon@kernel.org>
References: <CANXvt5oKt=AKdqv24LT079e+6URnfqJcfTJh0ajGA17paJUEKw@mail.gmail.com>
 <d096e88e-aec5-9920-8d5a-bd8200560c2c@amd.com>
 <fe309259-01f0-871f-4620-3a4bdc56a186@igel.co.jp>
Content-Language: en-US
From:   Vaishnav Achath <vaishnav.a@ti.com>
In-Reply-To: <fe309259-01f0-871f-4620-3a4bdc56a186@igel.co.jp>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi Kishon, all,

On 26/09/23 15:17, Shunsuke Mie wrote:
> 
> On 2023/09/21 18:11, Kishon Vijay Abraham I wrote:
>> +Vaishnav
>>
>> Hi Shunsuke,
>>
>> On 8/18/2023 7:16 PM, Shunsuke Mie wrote:
>>> Hi all,
>>>
>>> We are proposing to add a new test syste to Linux for PCIe Endpoint. That
>>> can be run on QEMU without real hardware. At present, partially we have
>>> confirmed that pci-epf-test is working, but it is not yet complete.
>>> However, we would appreciate your comments on the architecture design.
>>>
>>> # Background
>>> The background is as follows.
>>>
>>> PCI Endpoint function driver is implemented using the PCIe Endpoint
>>> framework, but it requires physical boards for testing, and it is difficult
>>> to test sufficiently. In order to find bugs and hardware-dependent
>>> implementations early, continuous testing is required. Since it is
>>> difficult to automate tests that require hardware, this RFC proposes a
>>> virtual environment for testing PCI endpoint function drivers.
>>
>> This would be quite useful and thank you for attempting it! I would like to
>> compare other mechanisms available in-addition to QEMU before going with the
>> QEMU approach.
> 
> I got it. I'll make a table to compare some methods that includes greybus to
> realize this emulation environment.
> 
> 
> Best,
> 
> Shunsuke
> 
>> Though I don't understand this fully, Looking at
>> https://osseu2023.sched.com/event/1OGk8/emulating-devices-in-linux-using-greybus-subsystem-vaishnav-mohandas-achath-texas-instruments, Vaishnav seems to solve the same problem using greybus for multiple type s of devices.
>>
>> Vaishnav, we'd wait for your OSS presentation but do you have any initial
>> thoughts on how greybus could be used to test PCIe endpoint drivers?
>>

Apologies for the delay, I don't think greybus can be used for PCIe testing as
there is no greybus equivalent for PCIe[1], it can only be used for relatively
simpler devices today, I guess roadtest(UML based)[2] could be an alternative in
this case.

1 -
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/greybus
2 - https://lore.kernel.org/lkml/YjN1ksNGujV611Ka@sirena.org.uk/

Thanks and Regards,
Vaishnav

>> Thanks,
>> Kishon
>>
>>>
>>> # Architecture
>>> The overview of the architecture is as follows.
>>>
>>>    Guest 1                        Guest 2
>>> +-------------------------+    +----------------------------+
>>> | Linux kernel            |    | Linux kernel               |
>>> |                         |    |                            |
>>> | PCI EP function driver  |    |                            |
>>> | (e.g. pci-epf-test)     |    |                            |
>>> |-------------------------|    | PCI Device Driver          |
>>> | (2) QEMU EPC Driver     |    | (e.g. pci_endpoint_test)   |
>>> +-------------------------+    +----------------------------+
>>> +-------------------------+    +----------------------------+
>>> | QEMU                    |    | QEMU                       |
>>> |-------------------------|    |----------------------------|
>>> | (1) QEMU PCI EPC Device *----* (3) QEMU EPF Bridge Device |
>>> +-------------------------+    +----------------------------+
>>>
>>> At present, it is designed to work guests only on the same host, and
>>> communication is done through Unix domain sockets.
>>>
>>> The three parts shown in the figure were introduced this time.
>>>
>>> (1) QEMU PCI Endpoint Controller(EPC) Device
>>> PCI Endpoint Controller implemented as QEMU PCI device.
>>> (2) QEMU PCI Endpoint Controller(EPC) Driver
>>> Linux kernel driver that drives the device (1). It registers a epc device
>>> to linux kernel and handling each operations for the epc device.
>>> (3) QEMU PCI Endpoint function(EPF) Bridge Device
>>> QEMU PCI device that cooperates with (1) and performs accesses to pci
>>> configuration space, BAR and memory space to communicate each guests, and
>>> generates interruptions to the guest 1.
>>>
>>> Each projects are:
>>> (1), (3) https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1
>>> <https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1>
>>> files: hw/misc/{qemu-epc.{c,h}, epf-bridge.c}
>>> (2) https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc
>>> <https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc>
>>> files: drivers/pci/controller/pcie-qemu-ep.c
>>>
>>> # Protocol
>>>
>>> PCI, PCIe has a layer structure that includes Physical, Data Lane and
>>> Transaction. The communicates between the bridge(3) and controller (1)
>>> mimic the Transaction. Specifically, a protocol is implemented for
>>> exchanging fd for communication protocol version check and communication,
>>> in addition to the interaction equivalent to PCIe Transaction Layer Packet
>>> (Read and Write of I/O, Memory, Configuration space and Message). In my
>>> mind, we need to discuss the communication mor.
>>>
>>> We also are planning to post the patch set after the code is organized and
>>> the protocol discussion is matured.
>>>
>>> Best regards,
>>> Shunsuke