From:   Frank Li <frank.li@nxp.com>
To:     "Michael S. Tsirkin" <mst@redhat.com>,
        Shunsuke Mie <mie@igel.co.jp>
CC:     Lorenzo Pieralisi <lpieralisi@kernel.org>,
        =?iso-8859-2?Q?Krzysztof_Wilczy=F1ski?= <kw@linux.com>,
        Manivannan Sadhasivam <mani@kernel.org>,
        Kishon Vijay Abraham I <kishon@kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Jason Wang <jasowang@redhat.com>, Jon Mason <jdmason@kudzu.us>,
        Ren Zhijie <renzhijie2@huawei.com>,
        Takanari Hayama <taki@igel.co.jp>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        "virtualization@lists.linux-foundation.org" 
        <virtualization@lists.linux-foundation.org>
Subject: RE: [EXT] Re: [RFC PATCH 4/4] PCI: endpoint: function: Add EP
 function driver to provide virtio net device
Thread-Topic: [EXT] Re: [RFC PATCH 4/4] PCI: endpoint: function: Add EP
 function driver to provide virtio net device
Thread-Index: AQHZN7liQTou1LlKIkegIEQy1N8nRq69xHgw
Date:   Fri, 3 Feb 2023 22:15:50 +0000
Message-ID: <HE1PR0401MB23313FC60955EADE8A00133088D79@HE1PR0401MB2331.eurprd04.prod.outlook.com>
References: <20230203100418.2981144-1-mie@igel.co.jp>
 <20230203100418.2981144-5-mie@igel.co.jp>
 <20230203052114-mutt-send-email-mst@kernel.org>
In-Reply-To: <20230203052114-mutt-send-email-mst@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?iso-8859-2?Q?+cvAjRxd3WVDFawH0Ffd+yizhHHGaM2R7nJeNT4nW4RxtbCjwFoNkwcD32?=
 =?iso-8859-2?Q?1RIZ9ZQdqvUoxcZzrtlhxP+mtX048vpOyTGzxlsX+7RhNIf9qKTLC4sO/6?=
 =?iso-8859-2?Q?NLJDcNbnYGadHFq5By4fS+cetcm1xifwsYPi+E2/Me4F8mIeyjO/prUHxZ?=
 =?iso-8859-2?Q?EV4EUWlimg8Oo8/CiN6IUlbmbLfo1yyeZtY5wsNvryh6wqTs/5P2S7fdgO?=
 =?iso-8859-2?Q?acouSXU4w9l2T0o/bKqp5MF6kIPhCA34/oot6mTh23zaIA4F/y/OX5BVMX?=
 =?iso-8859-2?Q?ZnJcx5r68d3YQihvhIkIgcfB7xp75jdmXRNxx+bpR35nErjqUJ1s3UwJD6?=
 =?iso-8859-2?Q?eclEPJpahaeuy/iN5yFgRffWCYO6wdzrAJz5fg4WhgcXzdN/0OXQyW0s68?=
 =?iso-8859-2?Q?w5S4eTxpkNafSbwu7erB9kqphR9vZRs0hqBSkYRZ3gJV+WeoTxDma/zX9z?=
 =?iso-8859-2?Q?ZuGefeKYrqOKaeG0xpURLQdVyNH8NdzVPqP4P2aTaUPAfajvuL71BJ/fwe?=
 =?iso-8859-2?Q?/IADkze8KkQ6ZNfOXkzjvsNW7OzOXWQcLKmHqiqBWxIfu4Ze+YtLq6bO/v?=
 =?iso-8859-2?Q?A+4ANiJWnotmB2pLvf9zQ9Qaz9Aka1R9iJxg/iy9O5C3a0K12I5+bfdt7P?=
 =?iso-8859-2?Q?T0Qp/iM4Un9fUk2WufG5uRjHbn6cVQd+UEFMbrvvvDFUIsvnrngDfgIhQF?=
 =?iso-8859-2?Q?BzCx/PILIZCgQAkBldtLhJFGzsRzIoeH7WotMkqqiMsLiEHeYLdeYinJR2?=
 =?iso-8859-2?Q?+KlYWukzVyCZZUFHGNpCj1riUVCcp7BOaB7yAlLAz7ToaW3L4WlpxIsdiV?=
 =?iso-8859-2?Q?gfTr3WafmEpon0TFE+3BI+T2mv6Sgh4Z2iS/3YhzB73eSXv+/R7fGtAS2y?=
 =?iso-8859-2?Q?waP9OI7wI+oWp/jZWuCPhaKq0PG4Qlv6/wBaT6gANGyCEkAhVci/w4516Y?=
 =?iso-8859-2?Q?Wh4K8d7Y2MnkrS+K65zTnkljor/DQTMlTNpdkGNZAbagKZirJyfrEMMu2p?=
 =?iso-8859-2?Q?4R97pVXtlI1qCgR5sO4lG6GpGWc2tY4cWkpDRDY8L0TxZHie6l6pYG6/Wo?=
 =?iso-8859-2?Q?9QJc8cBqma9qy6k+D7DOFSjK7sQDj32aU6h1ebPBE03olXyORnx6FuVvwm?=
 =?iso-8859-2?Q?gRUMQfdqObpS7H8IvVJyzCtQTzUhgCone9O+Zf4qH2wBAanxL3zReC0Y5U?=
 =?iso-8859-2?Q?gU+6mWBxqYRpq9EXrWN1S2KYyXEmmvg8Lt7cC4zgqaaJOoYkF8nGweJfoL?=
 =?iso-8859-2?Q?dg251OGr7Y7QJnBSv8KC6GzmdVdjIG/AMXVn7A9FljY7W/dNev//34H8SY?=
 =?iso-8859-2?Q?1sWp/cRCFsJ+2I/G8wOlhLULohJsQBJZUcQ5WtrFhbWk68jMLoECfCAEn3?=
 =?iso-8859-2?Q?kfoylSR4dsqXsNw67YhLlBimXgGNSkmLT2YtNTIYWCLT1DLxAHeSNIf4yo?=
 =?iso-8859-2?Q?NMy0wTt4AYUnL8sWBXL+fean1hswnZYCC96CahPMCxLLfdfDXUUnILrAMJ?=
 =?iso-8859-2?Q?trHkR/LsfAzumKmkj4NckHRBFDZPlTQWCWgkXex1ZOeMSP+P5JM3hZwsC9?=
 =?iso-8859-2?Q?BqvZEh4LKwZzPJDn8ZP0e521RzlaaphUamIFrLjqKKP0PrzdOQG5DaNJqG?=
 =?iso-8859-2?Q?eTkHDm/9YzGO4=3D?=
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: nxp.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: HE1PR0401MB2331.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 89c3c316-d0a6-4820-432b-08db06343583
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Feb 2023 22:15:50.8293
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: vp5sW0cIJemHOCaH/ETMmXF3Nl5cl9SjH0OejWVJr4bGn62H2mHOGCpPAastHYw50gsI0iLE2N3JnEWMh+qt3w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR04MB7385
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

>=20
> Caution: EXT Email
>=20
> On Fri, Feb 03, 2023 at 07:04:18PM +0900, Shunsuke Mie wrote:
> > Add a new endpoint(EP) function driver to provide virtio-net device. Th=
is
> > function not only shows virtio-net device for PCIe host system, but als=
o
> > provides virtio-net device to EP side(local) system. Virtualy those net=
work
> > devices are connected, so we can use to communicate over IP like a simp=
le
> > NIC.
> >
> > Architecture overview is following:
> >
> > to Host       |                       to Endpoint
> > network stack |                 network stack
> >       |       |                       |
> > +-----------+ |       +-----------+   +-----------+
> > |virtio-net | |       |virtio-net |   |virtio-net |
> > |driver     | |       |EP function|---|driver     |
> > +-----------+ |       +-----------+   +-----------+
> >       |       |             |
> > +-----------+ | +-----------+
> > |PCIeC      | | |PCIeC      |
> > |Rootcomplex|-|-|Endpoint   |
> > +-----------+ | +-----------+
> >   Host side   |          Endpoint side
> >
> > This driver uses PCIe EP framework to show virtio-net (pci) device Host
> > side, and generate virtual virtio-net device and register to EP side.
> > A communication date
>=20
> data?
>=20
> > is diractly
>=20
> directly?
>=20
> > transported between virtqueue level
> > with each other using PCIe embedded DMA controller.
> >
> > by a limitation of the hardware and Linux EP framework, this function
> > follows a virtio legacy specification.
>=20
> what exactly is the limitation and why does it force legacy?
>=20
> > This function driver has beed tested on S4 Rcar (r8a779fa-spider) board=
 but
> > just use the PCIe EP framework and depends on the PCIe EDMA.
> >
> > Signed-off-by: Shunsuke Mie <mie@igel.co.jp>
> > Signed-off-by: Takanari Hayama <taki@igel.co.jp>
> > ---
> >  drivers/pci/endpoint/functions/Kconfig        |  12 +
> >  drivers/pci/endpoint/functions/Makefile       |   1 +
> >  .../pci/endpoint/functions/pci-epf-vnet-ep.c  | 343 ++++++++++

It is actually that not related vnet. Just virtio.=20
I think pci-epf-virtio.c is better.=20

> >  .../pci/endpoint/functions/pci-epf-vnet-rc.c  | 635 ++++++++++++++++++

It is epf driver. rc is quite confused. =20
Maybe you can combine pci-epf-vnet-ep.c and pci-epf-vnet-rc.c to one file.

> >  drivers/pci/endpoint/functions/pci-epf-vnet.c | 387 +++++++++++

This file setup dma transfer according virtio-ring.
How about pci-epf-virtio-dma.c ?

> > +
> > +     epf_vnet_rc_set_config16(vnet, VIRTIO_PCI_ISR,
> VIRTIO_PCI_ISR_QUEUE);
> > +     /*
> > +      * Initialize the queue notify and selector to outside of the app=
ropriate
> > +      * virtqueue index. It is used to detect change with polling. The=
re is no
> > +      * other ways to detect host side driver updateing those values
> > +      */

I am try to use gic-its or other msi controller as doorbell.=20
https://lore.kernel.org/imx/20221125192729.1722913-1-Frank.Li@nxp.com/T/#u

but it may need update host side pci virtio driver.=20

> > +     epf_vnet_rc_set_config16(vnet, VIRTIO_PCI_QUEUE_NOTIFY,
> default_qindex);
> > +     epf_vnet_rc_set_config16(vnet, VIRTIO_PCI_QUEUE_SEL,
> default_qindex);
> > +     /* This pfn is also set to 0 for the polling as well */
> > +     epf_vnet_rc_set_config16(vnet, VIRTIO_PCI_QUEUE_PFN, 0);
> > +
> --
> > 2.25.1