+Vaishnav
Hi Shunsuke,
On 8/18/2023 7:16 PM, Shunsuke Mie wrote:
> Hi all,
>
> We are proposing to add a new test syste to Linux for PCIe Endpoint. That
> can be run on QEMU without real hardware. At present, partially we have
> confirmed that pci-epf-test is working, but it is not yet complete.
> However, we would appreciate your comments on the architecture design.
>
> # Background
> The background is as follows.
>
> PCI Endpoint function driver is implemented using the PCIe Endpoint
> framework, but it requires physical boards for testing, and it is difficult
> to test sufficiently. In order to find bugs and hardware-dependent
> implementations early, continuous testing is required. Since it is
> difficult to automate tests that require hardware, this RFC proposes a
> virtual environment for testing PCI endpoint function drivers.
This would be quite useful and thank you for attempting it! I would like
to compare other mechanisms available in-addition to QEMU before going
with the QEMU approach.
Though I don't understand this fully, Looking at
https://osseu2023.sched.com/event/1OGk8/emulating-devices-in-linux-using-greybus-subsystem-vaishnav-mohandas-achath-texas-instruments,
Vaishnav seems to solve the same problem using greybus for multiple type
s of devices.
Vaishnav, we'd wait for your OSS presentation but do you have any
initial thoughts on how greybus could be used to test PCIe endpoint drivers?
Thanks,
Kishon
>
> # Architecture
> The overview of the architecture is as follows.
>
> Guest 1 Guest 2
> +-------------------------+ +----------------------------+
> | Linux kernel | | Linux kernel |
> | | | |
> | PCI EP function driver | | |
> | (e.g. pci-epf-test) | | |
> |-------------------------| | PCI Device Driver |
> | (2) QEMU EPC Driver | | (e.g. pci_endpoint_test) |
> +-------------------------+ +----------------------------+
> +-------------------------+ +----------------------------+
> | QEMU | | QEMU |
> |-------------------------| |----------------------------|
> | (1) QEMU PCI EPC Device *----* (3) QEMU EPF Bridge Device |
> +-------------------------+ +----------------------------+
>
> At present, it is designed to work guests only on the same host, and
> communication is done through Unix domain sockets.
>
> The three parts shown in the figure were introduced this time.
>
> (1) QEMU PCI Endpoint Controller(EPC) Device
> PCI Endpoint Controller implemented as QEMU PCI device.
> (2) QEMU PCI Endpoint Controller(EPC) Driver
> Linux kernel driver that drives the device (1). It registers a epc device
> to linux kernel and handling each operations for the epc device.
> (3) QEMU PCI Endpoint function(EPF) Bridge Device
> QEMU PCI device that cooperates with (1) and performs accesses to pci
> configuration space, BAR and memory space to communicate each guests, and
> generates interruptions to the guest 1.
>
> Each projects are:
> (1), (3) https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1
> <https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1>
> files: hw/misc/{qemu-epc.{c,h}, epf-bridge.c}
> (2) https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc
> <https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc>
> files: drivers/pci/controller/pcie-qemu-ep.c
>
> # Protocol
>
> PCI, PCIe has a layer structure that includes Physical, Data Lane and
> Transaction. The communicates between the bridge(3) and controller (1)
> mimic the Transaction. Specifically, a protocol is implemented for
> exchanging fd for communication protocol version check and communication,
> in addition to the interaction equivalent to PCIe Transaction Layer Packet
> (Read and Write of I/O, Memory, Configuration space and Message). In my
> mind, we need to discuss the communication mor.
>
> We also are planning to post the patch set after the code is organized and
> the protocol discussion is matured.
>
> Best regards,
> Shunsuke
On 2023/09/21 18:11, Kishon Vijay Abraham I wrote:
> +Vaishnav
>
> Hi Shunsuke,
>
> On 8/18/2023 7:16 PM, Shunsuke Mie wrote:
>> Hi all,
>>
>> We are proposing to add a new test syste to Linux for PCIe Endpoint.
>> That
>> can be run on QEMU without real hardware. At present, partially we have
>> confirmed that pci-epf-test is working, but it is not yet complete.
>> However, we would appreciate your comments on the architecture design.
>>
>> # Background
>> The background is as follows.
>>
>> PCI Endpoint function driver is implemented using the PCIe Endpoint
>> framework, but it requires physical boards for testing, and it is
>> difficult
>> to test sufficiently. In order to find bugs and hardware-dependent
>> implementations early, continuous testing is required. Since it is
>> difficult to automate tests that require hardware, this RFC proposes a
>> virtual environment for testing PCI endpoint function drivers.
>
> This would be quite useful and thank you for attempting it! I would
> like to compare other mechanisms available in-addition to QEMU before
> going with the QEMU approach.
I got it. I'll make a table to compare some methods that includes
greybus to realize this emulation environment.
Best,
Shunsuke
> Though I don't understand this fully, Looking at
> https://osseu2023.sched.com/event/1OGk8/emulating-devices-in-linux-using-greybus-subsystem-vaishnav-mohandas-achath-texas-instruments,
> Vaishnav seems to solve the same problem using greybus for multiple
> type s of devices.
>
> Vaishnav, we'd wait for your OSS presentation but do you have any
> initial thoughts on how greybus could be used to test PCIe endpoint
> drivers?
>
> Thanks,
> Kishon
>
>>
>> # Architecture
>> The overview of the architecture is as follows.
>>
>> Guest 1 Guest 2
>> +-------------------------+ +----------------------------+
>> | Linux kernel | | Linux kernel |
>> | | | |
>> | PCI EP function driver | | |
>> | (e.g. pci-epf-test) | | |
>> |-------------------------| | PCI Device Driver |
>> | (2) QEMU EPC Driver | | (e.g. pci_endpoint_test) |
>> +-------------------------+ +----------------------------+
>> +-------------------------+ +----------------------------+
>> | QEMU | | QEMU |
>> |-------------------------| |----------------------------|
>> | (1) QEMU PCI EPC Device *----* (3) QEMU EPF Bridge Device |
>> +-------------------------+ +----------------------------+
>>
>> At present, it is designed to work guests only on the same host, and
>> communication is done through Unix domain sockets.
>>
>> The three parts shown in the figure were introduced this time.
>>
>> (1) QEMU PCI Endpoint Controller(EPC) Device
>> PCI Endpoint Controller implemented as QEMU PCI device.
>> (2) QEMU PCI Endpoint Controller(EPC) Driver
>> Linux kernel driver that drives the device (1). It registers a epc
>> device
>> to linux kernel and handling each operations for the epc device.
>> (3) QEMU PCI Endpoint function(EPF) Bridge Device
>> QEMU PCI device that cooperates with (1) and performs accesses to pci
>> configuration space, BAR and memory space to communicate each guests,
>> and
>> generates interruptions to the guest 1.
>>
>> Each projects are:
>> (1), (3) https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1
>> <https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1>
>> files: hw/misc/{qemu-epc.{c,h}, epf-bridge.c}
>> (2) https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc
>> <https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc>
>> files: drivers/pci/controller/pcie-qemu-ep.c
>>
>> # Protocol
>>
>> PCI, PCIe has a layer structure that includes Physical, Data Lane and
>> Transaction. The communicates between the bridge(3) and controller (1)
>> mimic the Transaction. Specifically, a protocol is implemented for
>> exchanging fd for communication protocol version check and
>> communication,
>> in addition to the interaction equivalent to PCIe Transaction Layer
>> Packet
>> (Read and Write of I/O, Memory, Configuration space and Message). In my
>> mind, we need to discuss the communication mor.
>>
>> We also are planning to post the patch set after the code is
>> organized and
>> the protocol discussion is matured.
>>
>> Best regards,
>> Shunsuke
On Thu, Sep 21, 2023 at 02:41:54PM +0530, Kishon Vijay Abraham I wrote:
> > PCI Endpoint function driver is implemented using the PCIe Endpoint
> > framework, but it requires physical boards for testing, and it is difficult
> > to test sufficiently. In order to find bugs and hardware-dependent
> > implementations early, continuous testing is required. Since it is
> > difficult to automate tests that require hardware, this RFC proposes a
> > virtual environment for testing PCI endpoint function drivers.
>
> This would be quite useful and thank you for attempting it! I would like to
> compare other mechanisms available in-addition to QEMU before going with the
> QEMU approach.
Well, the point of PCIe endpoint subsystem in vhost or similar is that
you can use one and the same endpoint implementation. So you can debug
it using qemu and the use it with a physical port, which would be really
amazing.
Hi Kishon, all,
On 26/09/23 15:17, Shunsuke Mie wrote:
>
> On 2023/09/21 18:11, Kishon Vijay Abraham I wrote:
>> +Vaishnav
>>
>> Hi Shunsuke,
>>
>> On 8/18/2023 7:16 PM, Shunsuke Mie wrote:
>>> Hi all,
>>>
>>> We are proposing to add a new test syste to Linux for PCIe Endpoint. That
>>> can be run on QEMU without real hardware. At present, partially we have
>>> confirmed that pci-epf-test is working, but it is not yet complete.
>>> However, we would appreciate your comments on the architecture design.
>>>
>>> # Background
>>> The background is as follows.
>>>
>>> PCI Endpoint function driver is implemented using the PCIe Endpoint
>>> framework, but it requires physical boards for testing, and it is difficult
>>> to test sufficiently. In order to find bugs and hardware-dependent
>>> implementations early, continuous testing is required. Since it is
>>> difficult to automate tests that require hardware, this RFC proposes a
>>> virtual environment for testing PCI endpoint function drivers.
>>
>> This would be quite useful and thank you for attempting it! I would like to
>> compare other mechanisms available in-addition to QEMU before going with the
>> QEMU approach.
>
> I got it. I'll make a table to compare some methods that includes greybus to
> realize this emulation environment.
>
>
> Best,
>
> Shunsuke
>
>> Though I don't understand this fully, Looking at
>> https://osseu2023.sched.com/event/1OGk8/emulating-devices-in-linux-using-greybus-subsystem-vaishnav-mohandas-achath-texas-instruments, Vaishnav seems to solve the same problem using greybus for multiple type s of devices.
>>
>> Vaishnav, we'd wait for your OSS presentation but do you have any initial
>> thoughts on how greybus could be used to test PCIe endpoint drivers?
>>
Apologies for the delay, I don't think greybus can be used for PCIe testing as
there is no greybus equivalent for PCIe[1], it can only be used for relatively
simpler devices today, I guess roadtest(UML based)[2] could be an alternative in
this case.
1 -
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/greybus
2 - https://lore.kernel.org/lkml/[email protected]/
Thanks and Regards,
Vaishnav
>> Thanks,
>> Kishon
>>
>>>
>>> # Architecture
>>> The overview of the architecture is as follows.
>>>
>>> Guest 1 Guest 2
>>> +-------------------------+ +----------------------------+
>>> | Linux kernel | | Linux kernel |
>>> | | | |
>>> | PCI EP function driver | | |
>>> | (e.g. pci-epf-test) | | |
>>> |-------------------------| | PCI Device Driver |
>>> | (2) QEMU EPC Driver | | (e.g. pci_endpoint_test) |
>>> +-------------------------+ +----------------------------+
>>> +-------------------------+ +----------------------------+
>>> | QEMU | | QEMU |
>>> |-------------------------| |----------------------------|
>>> | (1) QEMU PCI EPC Device *----* (3) QEMU EPF Bridge Device |
>>> +-------------------------+ +----------------------------+
>>>
>>> At present, it is designed to work guests only on the same host, and
>>> communication is done through Unix domain sockets.
>>>
>>> The three parts shown in the figure were introduced this time.
>>>
>>> (1) QEMU PCI Endpoint Controller(EPC) Device
>>> PCI Endpoint Controller implemented as QEMU PCI device.
>>> (2) QEMU PCI Endpoint Controller(EPC) Driver
>>> Linux kernel driver that drives the device (1). It registers a epc device
>>> to linux kernel and handling each operations for the epc device.
>>> (3) QEMU PCI Endpoint function(EPF) Bridge Device
>>> QEMU PCI device that cooperates with (1) and performs accesses to pci
>>> configuration space, BAR and memory space to communicate each guests, and
>>> generates interruptions to the guest 1.
>>>
>>> Each projects are:
>>> (1), (3) https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1
>>> <https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1>
>>> files: hw/misc/{qemu-epc.{c,h}, epf-bridge.c}
>>> (2) https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc
>>> <https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc>
>>> files: drivers/pci/controller/pcie-qemu-ep.c
>>>
>>> # Protocol
>>>
>>> PCI, PCIe has a layer structure that includes Physical, Data Lane and
>>> Transaction. The communicates between the bridge(3) and controller (1)
>>> mimic the Transaction. Specifically, a protocol is implemented for
>>> exchanging fd for communication protocol version check and communication,
>>> in addition to the interaction equivalent to PCIe Transaction Layer Packet
>>> (Read and Write of I/O, Memory, Configuration space and Message). In my
>>> mind, we need to discuss the communication mor.
>>>
>>> We also are planning to post the patch set after the code is organized and
>>> the protocol discussion is matured.
>>>
>>> Best regards,
>>> Shunsuke
Hi Vaishnav,
On 2023/09/26 21:40, Vaishnav Achath wrote:
> Hi Kishon, all,
>
> On 26/09/23 15:17, Shunsuke Mie wrote:
>> On 2023/09/21 18:11, Kishon Vijay Abraham I wrote:
>>> +Vaishnav
>>>
>>> Hi Shunsuke,
>>>
>>> On 8/18/2023 7:16 PM, Shunsuke Mie wrote:
>>>> Hi all,
>>>>
>>>> We are proposing to add a new test syste to Linux for PCIe Endpoint. That
>>>> can be run on QEMU without real hardware. At present, partially we have
>>>> confirmed that pci-epf-test is working, but it is not yet complete.
>>>> However, we would appreciate your comments on the architecture design.
>>>>
>>>> # Background
>>>> The background is as follows.
>>>>
>>>> PCI Endpoint function driver is implemented using the PCIe Endpoint
>>>> framework, but it requires physical boards for testing, and it is difficult
>>>> to test sufficiently. In order to find bugs and hardware-dependent
>>>> implementations early, continuous testing is required. Since it is
>>>> difficult to automate tests that require hardware, this RFC proposes a
>>>> virtual environment for testing PCI endpoint function drivers.
>>> This would be quite useful and thank you for attempting it! I would like to
>>> compare other mechanisms available in-addition to QEMU before going with the
>>> QEMU approach.
>> I got it. I'll make a table to compare some methods that includes greybus to
>> realize this emulation environment.
>>
>>
>> Best,
>>
>> Shunsuke
>>
>>> Though I don't understand this fully, Looking at
>>> https://osseu2023.sched.com/event/1OGk8/emulating-devices-in-linux-using-greybus-subsystem-vaishnav-mohandas-achath-texas-instruments, Vaishnav seems to solve the same problem using greybus for multiple type s of devices.
>>>
>>> Vaishnav, we'd wait for your OSS presentation but do you have any initial
>>> thoughts on how greybus could be used to test PCIe endpoint drivers?
>>>
> Apologies for the delay, I don't think greybus can be used for PCIe testing as
> there is no greybus equivalent for PCIe[1], it can only be used for relatively
> simpler devices today, I guess roadtest(UML based)[2] could be an alternative in
> this case.
Thank you for your comment.
To my understanding, the roadtest uses UML and it interact with hardware
model written in python to do testing. This would be grate for automated
testing to test drivers and subsystems.
For this PCIe endpoint, I think we need to hosts, one that works as a
PCIe endpoint and one that is a PCIe Root Complex to it. Is it possible
to realize the system?
like:
UML + PCIe endpoint function driver <-> python HW model (PCI Endpoint
controller) <-> UML + pci driver for the function
As another option, I'm considering the feasibility of dummy PCIe EPC
driver. It works as a PCIe EPC device in kernel and show pci device
according to the PCIe endpoint function driver to the same host. so It
could be realize a single host and test the function driver.
Best,
Shunsuke
> 1 -
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/greybus
> 2 - https://lore.kernel.org/lkml/[email protected]/
>
> Thanks and Regards,
> Vaishnav
>
>>> Thanks,
>>> Kishon
>>>
>>>> # Architecture
>>>> The overview of the architecture is as follows.
>>>>
>>>> Guest 1 Guest 2
>>>> +-------------------------+ +----------------------------+
>>>> | Linux kernel | | Linux kernel |
>>>> | | | |
>>>> | PCI EP function driver | | |
>>>> | (e.g. pci-epf-test) | | |
>>>> |-------------------------| | PCI Device Driver |
>>>> | (2) QEMU EPC Driver | | (e.g. pci_endpoint_test) |
>>>> +-------------------------+ +----------------------------+
>>>> +-------------------------+ +----------------------------+
>>>> | QEMU | | QEMU |
>>>> |-------------------------| |----------------------------|
>>>> | (1) QEMU PCI EPC Device *----* (3) QEMU EPF Bridge Device |
>>>> +-------------------------+ +----------------------------+
>>>>
>>>> At present, it is designed to work guests only on the same host, and
>>>> communication is done through Unix domain sockets.
>>>>
>>>> The three parts shown in the figure were introduced this time.
>>>>
>>>> (1) QEMU PCI Endpoint Controller(EPC) Device
>>>> PCI Endpoint Controller implemented as QEMU PCI device.
>>>> (2) QEMU PCI Endpoint Controller(EPC) Driver
>>>> Linux kernel driver that drives the device (1). It registers a epc device
>>>> to linux kernel and handling each operations for the epc device.
>>>> (3) QEMU PCI Endpoint function(EPF) Bridge Device
>>>> QEMU PCI device that cooperates with (1) and performs accesses to pci
>>>> configuration space, BAR and memory space to communicate each guests, and
>>>> generates interruptions to the guest 1.
>>>>
>>>> Each projects are:
>>>> (1), (3) https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1
>>>> <https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1>
>>>> files: hw/misc/{qemu-epc.{c,h}, epf-bridge.c}
>>>> (2) https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc
>>>> <https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc>
>>>> files: drivers/pci/controller/pcie-qemu-ep.c
>>>>
>>>> # Protocol
>>>>
>>>> PCI, PCIe has a layer structure that includes Physical, Data Lane and
>>>> Transaction. The communicates between the bridge(3) and controller (1)
>>>> mimic the Transaction. Specifically, a protocol is implemented for
>>>> exchanging fd for communication protocol version check and communication,
>>>> in addition to the interaction equivalent to PCIe Transaction Layer Packet
>>>> (Read and Write of I/O, Memory, Configuration space and Message). In my
>>>> mind, we need to discuss the communication mor.
>>>>
>>>> We also are planning to post the patch set after the code is organized and
>>>> the protocol discussion is matured.
>>>>
>>>> Best regards,
>>>> Shunsuke
hi shunsuke, all,
what about vfio-user + qemu?
qemu already has libvfio-user as submodule.
there is ongoing work to add qemu vfio-user client functionality.
adding people involved to loop, not sure if i forgot somebody.
regards
jiri
On Tue, Oct 03, 2023 at 01:56:03PM +0900, Shunsuke Mie wrote:
> Hi Vaishnav,
>
> On 2023/09/26 21:40, Vaishnav Achath wrote:
> > Hi Kishon, all,
> >
> > On 26/09/23 15:17, Shunsuke Mie wrote:
> > > On 2023/09/21 18:11, Kishon Vijay Abraham I wrote:
> > > > +Vaishnav
> > > >
> > > > Hi Shunsuke,
> > > >
> > > > On 8/18/2023 7:16 PM, Shunsuke Mie wrote:
> > > > > Hi all,
> > > > >
> > > > > We are proposing to add a new test syste to Linux for PCIe Endpoint. That
> > > > > can be run on QEMU without real hardware. At present, partially we have
> > > > > confirmed that pci-epf-test is working, but it is not yet complete.
> > > > > However, we would appreciate your comments on the architecture design.
> > > > >
> > > > > # Background
> > > > > The background is as follows.
> > > > >
> > > > > PCI Endpoint function driver is implemented using the PCIe Endpoint
> > > > > framework, but it requires physical boards for testing, and it is difficult
> > > > > to test sufficiently. In order to find bugs and hardware-dependent
> > > > > implementations early, continuous testing is required. Since it is
> > > > > difficult to automate tests that require hardware, this RFC proposes a
> > > > > virtual environment for testing PCI endpoint function drivers.
> > > > This would be quite useful and thank you for attempting it! I would like to
> > > > compare other mechanisms available in-addition to QEMU before going with the
> > > > QEMU approach.
> > > I got it. I'll make a table to compare some methods that includes greybus to
> > > realize this emulation environment.
> > >
> > >
> > > Best,
> > >
> > > Shunsuke
> > >
> > > > Though I don't understand this fully, Looking at
> > > > https://osseu2023.sched.com/event/1OGk8/emulating-devices-in-linux-using-greybus-subsystem-vaishnav-mohandas-achath-texas-instruments, Vaishnav seems to solve the same problem using greybus for multiple type s of devices.
> > > >
> > > > Vaishnav, we'd wait for your OSS presentation but do you have any initial
> > > > thoughts on how greybus could be used to test PCIe endpoint drivers?
> > > >
> > Apologies for the delay, I don't think greybus can be used for PCIe testing as
> > there is no greybus equivalent for PCIe[1], it can only be used for relatively
> > simpler devices today, I guess roadtest(UML based)[2] could be an alternative in
> > this case.
>
> Thank you for your comment.
>
> To my understanding, the roadtest uses UML and it interact with hardware
> model written in python to do testing. This would be grate for automated
> testing to test drivers and subsystems.
>
> For this PCIe endpoint, I think we need to hosts, one that works as a PCIe
> endpoint and one that is a PCIe Root Complex to it. Is it possible to
> realize the system?
> like:
> UML + PCIe endpoint function driver <-> python HW model (PCI Endpoint
> controller) <-> UML + pci driver for the function
>
>
> As another option, I'm considering the feasibility of dummy PCIe EPC driver.
> It works as a PCIe EPC device in kernel and show pci device according to the
> PCIe endpoint function driver to the same host. so It could be realize a
> single host and test the function driver.
>
>
> Best,
>
> Shunsuke
>
> > 1 -
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/greybus
> > 2 - https://lore.kernel.org/lkml/[email protected]/
> >
> > Thanks and Regards,
> > Vaishnav
> >
> > > > Thanks,
> > > > Kishon
> > > >
> > > > > # Architecture
> > > > > The overview of the architecture is as follows.
> > > > >
> > > > > ?? Guest 1 ? ? ? ? ? ? ? ? ? ? ? ?Guest 2
> > > > > +-------------------------+ ? ?+----------------------------+
> > > > > | Linux kernel ? ? ? ? ? ?| ? ?| Linux kernel? ? ? ? ? ? ? ?|
> > > > > | ? ? ? ? ? ? ? ? ? ? ? ? | ? ?| ? ? ? ? ? ? ? ? ? ? ? ? ? ?|
> > > > > | PCI EP function driver ?| ? ?| ? ? ? ? ? ? ? ? ? ? ? ? ? ?|
> > > > > | (e.g. pci-epf-test) ? ? | ? ?| ? ? ? ? ? ? ? ? ? ? ? ? ? ?|
> > > > > |-------------------------| ? ?| PCI Device Driver ? ? ? ? ?|
> > > > > | (2) QEMU EPC Driver ? ? | ? ?| (e.g. pci_endpoint_test) ? |
> > > > > +-------------------------+ ? ?+----------------------------+
> > > > > +-------------------------+ ? ?+----------------------------+
> > > > > | QEMU ? ? ? ? ? ? ? ? ? ?| ? ?| QEMU ? ? ? ? ? ? ? ? ? ? ? |
> > > > > |-------------------------| ? ?|----------------------------|
> > > > > | (1) QEMU PCI EPC Device *----* (3) QEMU EPF Bridge Device |
> > > > > +-------------------------+ ? ?+----------------------------+
> > > > >
> > > > > At present, it is designed to work guests only on the same host, and
> > > > > communication is done through Unix domain sockets.
> > > > >
> > > > > The three parts shown in the figure were introduced this time.
> > > > >
> > > > > (1) QEMU PCI Endpoint Controller(EPC) Device
> > > > > PCI Endpoint Controller implemented as QEMU PCI device.
> > > > > (2) QEMU PCI Endpoint Controller(EPC) Driver
> > > > > Linux kernel driver that drives the device (1). It registers a epc device
> > > > > to linux kernel and handling each operations for the epc device.
> > > > > (3) QEMU PCI Endpoint function(EPF) Bridge Device
> > > > > QEMU PCI device that cooperates with (1) and performs accesses to pci
> > > > > configuration space, BAR and memory space to communicate each guests, and
> > > > > generates interruptions to the guest 1.
> > > > >
> > > > > Each projects are:
> > > > > (1), (3) https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1
> > > > > <https://github.com/ShunsukeMie/qemu/tree/epf-bridge/v1>
> > > > > files: hw/misc/{qemu-epc.{c,h}, epf-bridge.c}
> > > > > (2) https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc
> > > > > <https://github.com/ShunsukeMie/linux-virtio-rdma/tree/qemu-epc>
> > > > > files: drivers/pci/controller/pcie-qemu-ep.c
> > > > >
> > > > > # Protocol
> > > > >
> > > > > PCI, PCIe has a layer structure that includes Physical, Data Lane and
> > > > > Transaction. The communicates between the bridge(3) and controller (1)
> > > > > mimic the Transaction. Specifically, a protocol is implemented for
> > > > > exchanging fd for communication protocol version check and communication,
> > > > > in addition to the interaction equivalent to PCIe Transaction Layer Packet
> > > > > (Read and Write of I/O, Memory, Configuration space and Message). In my
> > > > > mind, we need to discuss the communication mor.
> > > > >
> > > > > We also are planning to post the patch set after the code is organized and
> > > > > the protocol discussion is matured.
> > > > >
> > > > > Best regards,
> > > > > Shunsuke