From:   Frank Li <frank.li@nxp.com>
To:     Shunsuke Mie <mie@igel.co.jp>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>
CC:     =?iso-8859-2?Q?Krzysztof_Wilczy=F1ski?= <kw@linux.com>,
        Manivannan Sadhasivam <mani@kernel.org>,
        Kishon Vijay Abraham I <kishon@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Michael S. Tsirkin" <mst@redhat.com>,
        Jason Wang <jasowang@redhat.com>, Jon Mason <jdmason@kudzu.us>,
        Ren Zhijie <renzhijie2@huawei.com>,
        Takanari Hayama <taki@igel.co.jp>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "virtualization@lists.linux-foundation.org" 
        <virtualization@lists.linux-foundation.org>
Subject: RE: [EXT] [RFC PATCH 0/4] PCI: endpoint: Introduce a virtio-net EP
 function
Thread-Topic: [EXT] [RFC PATCH 0/4] PCI: endpoint: Introduce a virtio-net EP
 function
Thread-Index: AQHZN7bpBm+5RmnRNEW5Iyx9WPW3za69bfSA
Date:   Fri, 3 Feb 2023 16:45:52 +0000
Message-ID: <HE1PR0401MB2331EAFF5684D60EC565433688D79@HE1PR0401MB2331.eurprd04.prod.outlook.com>
References: <20230203100418.2981144-1-mie@igel.co.jp>
In-Reply-To: <20230203100418.2981144-1-mie@igel.co.jp>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-2?Q?kYbrvmwTOEvi4SfYde+9tDrRyC8xcdiEUg9GGTRZ34dJHMHXhp8QF5W0d2?=
 =?iso-8859-2?Q?zCA55VhhWfSSyonZTTZqXvyxnhkAk1rL8yjnY120q9MgxljffbI3gxvR63?=
 =?iso-8859-2?Q?Z5DFtxeSQ1+6gX89fuiCHXl+Arxo+hXsemH+/pIp3rLbgkkDS6fHjs0lTs?=
 =?iso-8859-2?Q?2AZI3oEmoX2kqzh+esYLbwbwsWehTxTIsOTLVJvgF50Z3JsByzgw6scCD+?=
 =?iso-8859-2?Q?hvOw9Lz+M+Ptn5vD08HSoBzqvcLI8SCyI2BYDWLuaX63C34axq1FpR8nYz?=
 =?iso-8859-2?Q?cnep/njNLQR38x1mQE3eWRFTHW0+Zb9w24BPD0FihBfesDALyvzE0zdvKP?=
 =?iso-8859-2?Q?sDcBZBJxJjT+5HBZ3d5zSjzM9ZgShmzKXaISGzPLy7DdeNjniOxi2Hy9Yp?=
 =?iso-8859-2?Q?OTOo1haSgfIFnKjgvojaca9HcKWeBiRitwm/O6l0MyxBVdd6iBb5bM9zJs?=
 =?iso-8859-2?Q?x1Fwn/2KwmS2D1US54q/fdnwcsnVHXJhTVm21GDwb846CJyW5BrDakqpwp?=
 =?iso-8859-2?Q?Lo/qwajrr07soao7ue0nCT7n1Oy2l6D0AwTp45mV1cLB5LADQNAd6XDWca?=
 =?iso-8859-2?Q?bIG/l3Mv5se0WgJkHmoDfNwks7DUngCkMmiMJExFSgZPiOrZ/jbaEMK3S0?=
 =?iso-8859-2?Q?brbANJRBvgwrDigfebUl+UrqBh6T+ItevyJFinDLQMLJK1Ld28JuTE2ZsE?=
 =?iso-8859-2?Q?NqX6zNJM90zjzyYPRfAh1dgBkcb41BiFX9EfrEILad/rVysqZUqn7E2k00?=
 =?iso-8859-2?Q?1/SFzY9h3POSyQELr3Msuuv2dM8Kj3X9INqOIpGvndNdb/A12ey5MFZ2VK?=
 =?iso-8859-2?Q?ghKWj0MqF9mI+BKnlMOfD48tiJDFhwueACjf4uyJEYwcqRcR0bXEWe3Tol?=
 =?iso-8859-2?Q?s028qjOS3IZ3U+dooHSIcoLFpCP/atN6jYb7zPN7ibr3z8y2z09G2gpdAO?=
 =?iso-8859-2?Q?5w437ovsGenCrQqaD1ZqPwcl9jQf0LHqHTaEXQWyeEJWdJvBbhDIYGtgvQ?=
 =?iso-8859-2?Q?smx9ei/tLAeMFh/bujZS5qwXbxdZ3wM9G8D3X95mbaM3H9kS9D/TOcUmIp?=
 =?iso-8859-2?Q?85x4YIQghj9aywYB/WDN7S2LdHnRdRw6iu4Osr3bpmzamnXcwZ0KpA7DV1?=
 =?iso-8859-2?Q?+RF2OIMd7DQ8s+Rl7jNYX4a8nLjJfGwdOVsZcvL6mpg0rM7ABfjpXWdM3A?=
 =?iso-8859-2?Q?7OV4Sp3Xqj8uOScUBWuRHmEZQG86b1ESjKWB8DNCsB1E8L8GYJqqcpUmEg?=
 =?iso-8859-2?Q?cI/w+JxMmDV3V8I3akDoYKqcBKLHkEtFuPuwMn28HIcO6Fup1vA37yPsaL?=
 =?iso-8859-2?Q?4rs8v3/wnniJYCy4YgkqeT5RoLOpsch9VeP9wBPUGY/tpLRKAdVxFpiBsO?=
 =?iso-8859-2?Q?CPaAJ1Wa9Urwg+w+P2bHSXKI7CME8z2uLH4xDS0QqF9aVec5EsrJVqD2ps?=
 =?iso-8859-2?Q?O2E+CJdiO97MV7hPxye8a3wALKaPFjJAFMwCeR3eEpJSwE51U1JuBYHbDy?=
 =?iso-8859-2?Q?b0PG8ZD7Hr6OIity7YlQkv/gLzBNGSv4DQocuciRYkncmViRudSMOn4pyo?=
 =?iso-8859-2?Q?BjwnSaU2tKM0CcG77CpF/mctRORTLcMzr0FR2WhoNw3+yC/Y5qQRxEHh82?=
 =?iso-8859-2?Q?rLAAE7tCOpUMI=3D?=
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: HE1PR0401MB2331.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78ae892e-453c-449a-b4c1-08db06061cf5
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Feb 2023 16:45:52.8023
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: a43noGIblr6cFx3lYvOrWRpXa87WDRrNLBDAHhUCAVtqSKZ21LPDCeGDlrrUYCTsnVYz1CZnqxFFQlxSBF5DiA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8038
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Shunsuke Mie <mie@igel.co.jp>
> Sent: Friday, February 3, 2023 4:04 AM
> To: Lorenzo Pieralisi <lpieralisi@kernel.org>
> Cc: Krzysztof Wilczy=F1ski <kw@linux.com>; Manivannan Sadhasivam
> <mani@kernel.org>; Kishon Vijay Abraham I <kishon@kernel.org>; Bjorn
> Helgaas <bhelgaas@google.com>; Michael S. Tsirkin <mst@redhat.com>;
> Jason Wang <jasowang@redhat.com>; Shunsuke Mie <mie@igel.co.jp>;
> Frank Li <frank.li@nxp.com>; Jon Mason <jdmason@kudzu.us>; Ren Zhijie
> <renzhijie2@huawei.com>; Takanari Hayama <taki@igel.co.jp>; linux-
> kernel@vger.kernel.org; linux-pci@vger.kernel.org; virtualization@lists.l=
inux-
> foundation.org
> Subject: [EXT] [RFC PATCH 0/4] PCI: endpoint: Introduce a virtio-net EP
> function
>=20
> Caution: EXT Email
>=20
> This patchset introduce a virtio-net EP device function. It provides a
> new option to communiate between PCIe host and endpoint over IP.
> Advantage of this option is that the driver fully uses a PCIe embedded DM=
A.
> It is used to transport data between virtio ring directly each other. It
> can be expected to better throughput.

Thanks, basic that's what I want.  I am trying use RDMA.=20
But I think virtio-net still be good solution.=20

Frank Li=20

>=20
> To realize the function, this patchset has few changes and introduces a
> new APIs to PCI EP framework related to virtio. Furthermore, it device
> depends on the some patchtes that is discussing. Those depended patchset
> are following:
> - [PATCH 1/2] dmaengine: dw-edma: Fix to change for continuous transfer
> link:
> https://eur01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flore.=
k
> ernel.org%2Fdmaengine%2F20221223022608.550697-1-
> mie%40igel.co.jp%2F&data=3D05%7C01%7CFrank.Li%40nxp.com%7Cac57a62d4
> 10b458a5ba408db05ce0a4e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%
> 7C0%7C638110154722945380%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
> C%7C%7C&sdata=3DtIn0MHzEvrdxaC4KKTvTRvYXBzQ6MyrFa2GXpa3ePv0%3D&
> reserved=3D0
> - [RFC PATCH 0/3] Deal with alignment restriction on EP side
> link:
> https://eur01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flore.=
k
> ernel.org%2Flinux-pci%2F20230113090350.1103494-1-
> mie%40igel.co.jp%2F&data=3D05%7C01%7CFrank.Li%40nxp.com%7Cac57a62d4
> 10b458a5ba408db05ce0a4e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%
> 7C0%7C638110154722945380%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
> C%7C%7C&sdata=3DRLpnDiLwfqQd5QMXdiQyPVCkfOj8q2AyVeZOwWHvlsM%3
> D&reserved=3D0
> - [RFC PATCH v2 0/7] Introduce a vringh accessor for IO memory
> link:
> https://eur01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flore.=
k
> ernel.org%2Fvirtualization%2F20230202090934.549556-1-
> mie%40igel.co.jp%2F&data=3D05%7C01%7CFrank.Li%40nxp.com%7Cac57a62d4
> 10b458a5ba408db05ce0a4e%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%
> 7C0%7C638110154722945380%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
> C%7C%7C&sdata=3D6jgY76BMSbvamb%2Fl3Urjt4Gcizeqon%2BZE5nPssc2kDA%
> 3D&reserved=3D0
>=20
> About this patchset has 4 patches. The first of two patch is little chang=
es
> to virtio. The third patch add APIs to easily access virtio data structur=
e
> on PCIe Host side memory. The last one introduce a virtio-net EP device
> function. Details are in commit respectively.
>=20
> Currently those network devices are testd using ping only. I'll add a
> result of performance evaluation using iperf and etc to the future versio=
n
> of this patchset.
>=20
> Shunsuke Mie (4):
>   virtio_pci: add a definition of queue flag in ISR
>   virtio_ring: remove const from vring getter
>   PCI: endpoint: Introduce virtio library for EP functions
>   PCI: endpoint: function: Add EP function driver to provide virtio net
>     device
>=20
>  drivers/pci/endpoint/Kconfig                  |   7 +
>  drivers/pci/endpoint/Makefile                 |   1 +
>  drivers/pci/endpoint/functions/Kconfig        |  12 +
>  drivers/pci/endpoint/functions/Makefile       |   1 +
>  .../pci/endpoint/functions/pci-epf-vnet-ep.c  | 343 ++++++++++
>  .../pci/endpoint/functions/pci-epf-vnet-rc.c  | 635 ++++++++++++++++++
>  drivers/pci/endpoint/functions/pci-epf-vnet.c | 387 +++++++++++
>  drivers/pci/endpoint/functions/pci-epf-vnet.h |  62 ++
>  drivers/pci/endpoint/pci-epf-virtio.c         | 113 ++++
>  drivers/virtio/virtio_ring.c                  |   2 +-
>  include/linux/pci-epf-virtio.h                |  25 +
>  include/linux/virtio.h                        |   2 +-
>  include/uapi/linux/virtio_pci.h               |   2 +
>  13 files changed, 1590 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vnet-ep.c
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vnet-rc.c
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vnet.c
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-vnet.h
>  create mode 100644 drivers/pci/endpoint/pci-epf-virtio.c
>  create mode 100644 include/linux/pci-epf-virtio.h
>=20
> --
> 2.25.1