From: Frank Li <[email protected]>
Recently more and more people are interested in PCI RC and EP connection,
especially network usage cases. I upstreamed a vntb solution last year.
But the transfer speed is not good enough. I initialized a discussion
at https://lore.kernel.org/imx/[email protected]/T/#t
┌─────────────────────────────────┐ ┌──────────────┐
│ │ │ │
│ │ │ │
│ VirtQueue RX │ │ VirtQueue │
│ TX ┌──┐ │ │ TX │
│ ┌─────────┐ │ │ │ │ ┌─────────┐ │
│ │ SRC LEN ├─────┐ ┌──┤ │◄────┼───┼─┤ SRC LEN │ │
│ ├─────────┤ │ │ │ │ │ │ ├─────────┤ │
│ │ │ │ │ │ │ │ │ │ │ │
│ ├─────────┤ │ │ │ │ │ │ ├─────────┤ │
│ │ │ │ │ │ │ │ │ │ │ │
│ └─────────┘ │ │ └──┘ │ │ └─────────┘ │
│ │ │ │ │ │
│ RX ┌───┼──┘ TX │ │ RX │
│ ┌─────────┐ │ │ ┌──┐ │ │ ┌─────────┐ │
│ │ │◄┘ └────►│ ├─────┼───┼─┤ │ │
│ ├─────────┤ │ │ │ │ ├─────────┤ │
│ │ │ │ │ │ │ │ │ │
│ ├─────────┤ │ │ │ │ ├─────────┤ │
│ │ │ │ │ │ │ │ │ │
│ └─────────┘ │ │ │ │ └─────────┘ │
│ virtio_net └──┘ │ │ virtio_net │
│ Virtual PCI BUS EDMA Queue │ │ │
├─────────────────────────────────┤ │ │
│ PCI EP Controller with eDMA │ │ PCI Host │
└─────────────────────────────────┘ └──────────────┘
Basic idea is
1. Both EP and host probe virtio_net driver
2. There are two queues, one is the EP side(EQ), the other is the Host side.
3. EP side epf driver map Host side’s queue into EP’s space. Called HQ.
4. One working thread
5. pick one TX from EQ and RX from HQ, combine and generate EDMA requests,
and put into the DMA TX queue.
6. Pick one RX from EQ and TX from HQ, combine and generate EDMA requests,
and put into the DMA RX queue.
7. EDMA done irq will mark related item in EP and HQ finished.
The whole transfer is zero copied and uses a DMA queue.
The Shunsuke Mie implemented the above idea.
https://lore.kernel.org/linux-pci/CANXvt5q_qgLuAfF7dxxrqUirT_Ld4B=POCq8JcB9uPRvCGDiKg@mail.gmail.com/T/#t
Similar solution posted at 2019, except use memcpy from/to PCI EP map windows.
Using DMA should be simpler because EDMA can access the whole HOST\EP side memory space.
https://lore.kernel.org/linux-pci/[email protected]/T/
Solution 1 (Based on shunsuke):
Both EP and Host side use virtio.
Using EDMA to simplify data transfer and improve transfer speed.
RDMA implement based on RoCE
- proposal: https://lore.kernel.org/all/[email protected]/T/
- presentation on kvm forum: https://youtu.be/Qrhv6hC_YK4
Solution 2(2020, Kishon)
Previous https://lore.kernel.org/linux-pci/[email protected]/
EP side use vhost, RC side use virtio.
I don’t think anyone works on this thread now.
If using eDMA, it needs both sides to have a transfer queue.
I don't know how to easily implement it on the vhost side.
Solution 3(I am working on)
Implement infiniband rdma driver at both EP and RC side.
EP side build EDMA hardware queue based on EP/RC side’s send and receive
queue and when eDMA finished, write status to complete queue for both EP/RC
side. Use ipoib implement network transfer.
The whole upstream effort is quite huge for these. I don’t want to waste
time and efforts because direction is wrong.
I think Solution 1 is an easy path.
Thanks for organizing the discussion.
On 2023/02/08 4:45, Frank Li wrote:
> From: Frank Li<[email protected]>
>
> Recently more and more people are interested in PCI RC and EP connection,
> especially network usage cases. I upstreamed a vntb solution last year.
> But the transfer speed is not good enough. I initialized a discussion
> athttps://lore.kernel.org/imx/[email protected]/T/#t
I've investigated the vntb + ntbnet device that uses ntb transfer. Is it
difficult to adapt the eDMA to the ntb transfer? It is likely one of the
solutions for the performance problem.
>
> ┌─────────────────────────────────┐ ┌──────────────┐
> │ │ │ │
> │ │ │ │
> │ VirtQueue RX │ │ VirtQueue │
> │ TX ┌──┐ │ │ TX │
> │ ┌─────────┐ │ │ │ │ ┌─────────┐ │
> │ │ SRC LEN ├─────┐ ┌──┤ │◄────┼───┼─┤ SRC LEN │ │
> │ ├─────────┤ │ │ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │ │ │
> │ ├─────────┤ │ │ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │ │ │
> │ └─────────┘ │ │ └──┘ │ │ └─────────┘ │
> │ │ │ │ │ │
> │ RX ┌───┼──┘ TX │ │ RX │
> │ ┌─────────┐ │ │ ┌──┐ │ │ ┌─────────┐ │
> │ │ │◄┘ └────►│ ├─────┼───┼─┤ │ │
> │ ├─────────┤ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │
> │ ├─────────┤ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │
> │ └─────────┘ │ │ │ │ └─────────┘ │
> │ virtio_net └──┘ │ │ virtio_net │
> │ Virtual PCI BUS EDMA Queue │ │ │
> ├─────────────────────────────────┤ │ │
> │ PCI EP Controller with eDMA │ │ PCI Host │
> └─────────────────────────────────┘ └──────────────┘
>
> Basic idea is
> 1. Both EP and host probe virtio_net driver
> 2. There are two queues, one is the EP side(EQ), the other is the Host side.
> 3. EP side epf driver map Host side’s queue into EP’s space. Called HQ.
> 4. One working thread
> 5. pick one TX from EQ and RX from HQ, combine and generate EDMA requests,
> and put into the DMA TX queue.
> 6. Pick one RX from EQ and TX from HQ, combine and generate EDMA requests,
> and put into the DMA RX queue.
> 7. EDMA done irq will mark related item in EP and HQ finished.
>
> The whole transfer is zero copied and uses a DMA queue.
>
> The Shunsuke Mie implemented the above idea.
> https://lore.kernel.org/linux-pci/CANXvt5q_qgLuAfF7dxxrqUirT_Ld4B=POCq8JcB9uPRvCGDiKg@mail.gmail.com/T/#t
>
>
> Similar solution posted at 2019, except use memcpy from/to PCI EP map windows.
> Using DMA should be simpler because EDMA can access the whole HOST\EP side memory space.
> https://lore.kernel.org/linux-pci/[email protected]/T/
>
> Solution 1 (Based on shunsuke):
>
> Both EP and Host side use virtio.
> Using EDMA to simplify data transfer and improve transfer speed.
> RDMA implement based on RoCE
> - proposal:https://lore.kernel.org/all/[email protected]/T/
> - presentation on kvm forum:https://youtu.be/Qrhv6hC_YK4
>
> Solution 2(2020, Kishon)
>
> Previoushttps://lore.kernel.org/linux-pci/[email protected]/
> EP side use vhost, RC side use virtio.
> I don’t think anyone works on this thread now.
> If using eDMA, it needs both sides to have a transfer queue.
> I don't know how to easily implement it on the vhost side.
We had implemented this solution at the design stage of our proposal.
This solution has to prepare a network device and register to the kernel
from scratch for the endpoint. There is a lot of duplicated code, so we
think the solution 1 is better, as Frank said.
> Solution 3(I am working on)
>
> Implement infiniband rdma driver at both EP and RC side.
> EP side build EDMA hardware queue based on EP/RC side’s send and receive
> queue and when eDMA finished, write status to complete queue for both EP/RC
> side. Use ipoib implement network transfer.
The new InfiniBand device has to implement an InfiniBand network layer.
I think it is overengineered for this peer-to-peer communication. In
addition, the driver of the InfiniBand device should be implemented or
emulate the existing InfiniBand device to use the upstream driver. We
want to reduce the cost of implementation and maintenance.
> The whole upstream effort is quite huge for these. I don’t want to waste
> time and efforts because direction is wrong.
>
> I think Solution 1 is an easy path.
>
>
>
Best,
Shunsuke.
> EP side use vhost, RC side use virtio.
> > I don’t think anyone works on this thread now.
> > If using eDMA, it needs both sides to have a transfer queue.
> > I don't know how to easily implement it on the vhost side.
> We had implemented this solution at the design stage of our proposal.
> This solution has to prepare a network device and register to the kernel
> from scratch for the endpoint. There is a lot of duplicated code, so we
> think the solution 1 is better, as Frank said.
> > Solution 3(I am working on)
> >
> > Implement infiniband rdma driver at both EP and RC side.
> > EP side build EDMA hardware queue based on EP/RC side’s send and
> receive
> > queue and when eDMA finished, write status to complete queue for both
> EP/RC
> > side. Use ipoib implement network transfer.
> The new InfiniBand device has to implement an InfiniBand network layer.
> I think it is overengineered for this peer-to-peer communication. In
> addition, the driver of the InfiniBand device should be implemented or
> emulate the existing InfiniBand device to use the upstream driver. We
> want to reduce the cost of implementation and maintenance.
Infiniband driver is quite complex. That's reason why progress is slow in my
side. I hope the endpoint maintainer(kw) and PCI maintainer(Bjorn) can
provide comments.
> > The whole upstream effort is quite huge for these. I don’t want to waste
> > time and efforts because direction is wrong.
> >
> > I think Solution 1 is an easy path.
> >
> >
> >
> Best,
>
> Shunsuke.
On Tue, Feb 07, 2023 at 02:45:27PM -0500, Frank Li wrote:
> From: Frank Li <[email protected]>
>
> Recently more and more people are interested in PCI RC and EP connection,
> especially network usage cases. I upstreamed a vntb solution last year.
> But the transfer speed is not good enough. I initialized a discussion
> at https://lore.kernel.org/imx/[email protected]/T/#t
>
> ┌─────────────────────────────────┐ ┌──────────────┐
> │ │ │ │
> │ │ │ │
> │ VirtQueue RX │ │ VirtQueue │
> │ TX ┌──┐ │ │ TX │
> │ ┌─────────┐ │ │ │ │ ┌─────────┐ │
> │ │ SRC LEN ├─────┐ ┌──┤ │◄────┼───┼─┤ SRC LEN │ │
> │ ├─────────┤ │ │ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │ │ │
> │ ├─────────┤ │ │ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │ │ │
> │ └─────────┘ │ │ └──┘ │ │ └─────────┘ │
> │ │ │ │ │ │
> │ RX ┌───┼──┘ TX │ │ RX │
> │ ┌─────────┐ │ │ ┌──┐ │ │ ┌─────────┐ │
> │ │ │◄┘ └────►│ ├─────┼───┼─┤ │ │
> │ ├─────────┤ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │
> │ ├─────────┤ │ │ │ │ ├─────────┤ │
> │ │ │ │ │ │ │ │ │ │
> │ └─────────┘ │ │ │ │ └─────────┘ │
> │ virtio_net └──┘ │ │ virtio_net │
> │ Virtual PCI BUS EDMA Queue │ │ │
> ├─────────────────────────────────┤ │ │
> │ PCI EP Controller with eDMA │ │ PCI Host │
> └─────────────────────────────────┘ └──────────────┘
>
> Basic idea is
> 1. Both EP and host probe virtio_net driver
> 2. There are two queues, one is the EP side(EQ), the other is the Host side.
> 3. EP side epf driver map Host side’s queue into EP’s space. Called HQ.
> 4. One working thread
> 5. pick one TX from EQ and RX from HQ, combine and generate EDMA requests,
> and put into the DMA TX queue.
> 6. Pick one RX from EQ and TX from HQ, combine and generate EDMA requests,
> and put into the DMA RX queue.
> 7. EDMA done irq will mark related item in EP and HQ finished.
>
> The whole transfer is zero copied and uses a DMA queue.
>
> The Shunsuke Mie implemented the above idea.
> https://lore.kernel.org/linux-pci/CANXvt5q_qgLuAfF7dxxrqUirT_Ld4B=POCq8JcB9uPRvCGDiKg@mail.gmail.com/T/#t
>
>
> Similar solution posted at 2019, except use memcpy from/to PCI EP map windows.
> Using DMA should be simpler because EDMA can access the whole HOST\EP side memory space.
> https://lore.kernel.org/linux-pci/[email protected]/T/
>
> Solution 1 (Based on shunsuke):
>
> Both EP and Host side use virtio.
> Using EDMA to simplify data transfer and improve transfer speed.
> RDMA implement based on RoCE
> - proposal: https://lore.kernel.org/all/[email protected]/T/
> - presentation on kvm forum: https://youtu.be/Qrhv6hC_YK4
>
> Solution 2(2020, Kishon)
>
> Previous https://lore.kernel.org/linux-pci/[email protected]/
> EP side use vhost, RC side use virtio.
> I don’t think anyone works on this thread now.
> If using eDMA, it needs both sides to have a transfer queue.
> I don't know how to easily implement it on the vhost side.
>
> Solution 3(I am working on)
>
> Implement infiniband rdma driver at both EP and RC side.
> EP side build EDMA hardware queue based on EP/RC side’s send and receive
> queue and when eDMA finished, write status to complete queue for both EP/RC
> side. Use ipoib implement network transfer.
>
>
> The whole upstream effort is quite huge for these. I don’t want to waste
> time and efforts because direction is wrong.
>
> I think Solution 1 is an easy path.
>
I didn't had time to look into Shunsuke's series, but from the initial look
of the proposed solutions, option 1 seems to be the best for me.
Thanks,
Mani
>
>
--
மணிவண்ணன் சதாசிவம்