Received-SPF: pass (google.com: domain of linux-kernel+bounces-144138-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Vamsi Krishna Attunuru <vattunuru@marvell.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
CC: Arnd Bergmann <arnd@arndb.de>,
        "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>,
        Jerin Jacob <jerinj@marvell.com>
Subject: RE: [EXTERNAL] Re: [PATCH v5 1/1] misc: mrvl-cn10k-dpi: add Octeon
 CN10K DPI administrative driver
Thread-Topic: [EXTERNAL] Re: [PATCH v5 1/1] misc: mrvl-cn10k-dpi: add Octeon
 CN10K DPI administrative driver
Thread-Index: 
 AQHajNJlUzRgj48oqke7eNtbl1jogrFkj7+AgAAOmwCAACXXAIAAAZzAgADsugCAAFXs8IAACFkAgABEXHCAAD3qAIAA7f3AgAAGhoCAACgGIA==
Date: Sun, 14 Apr 2024 12:32:18 +0000
Message-ID: 
 <MW4PR18MB524447305F8F49847D88A6E2A60A2@MW4PR18MB5244.namprd18.prod.outlook.com>
References: <2024041250-nursing-tidy-db7e@gregkh>
 <MW4PR18MB5244FA7231C64F8A7928B83EA6042@MW4PR18MB5244.namprd18.prod.outlook.com>
 <2024041258-reminder-widely-00c0@gregkh>
 <MW4PR18MB52447D1E7F0CD0D1CA6BAACCA6042@MW4PR18MB5244.namprd18.prod.outlook.com>
 <2024041351-endowment-underrate-4b8d@gregkh>
 <MW4PR18MB52449EBFE3862D7C3065D22AA60B2@MW4PR18MB5244.namprd18.prod.outlook.com>
 <2024041310-bundle-patio-8a22@gregkh>
 <MW4PR18MB5244807D8DE91CA01EEB91B5A60B2@MW4PR18MB5244.namprd18.prod.outlook.com>
 <b5d400e6-9e4a-4f5d-b48f-a75c65ec1ae8@app.fastmail.com>
 <MW4PR18MB52449BD843670B347C0383EDA60A2@MW4PR18MB5244.namprd18.prod.outlook.com>
 <2024041458-directory-image-6a0c@gregkh>
In-Reply-To: <2024041458-directory-image-6a0c@gregkh>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: 
 =?us-ascii?Q?SVdTnGHvMBv17s4nL8JkBUnRwfzduHjWT9FFrJtwX+wnPfaq/YYbaotWNEvg?=
 =?us-ascii?Q?ypwHtF95FsxiIoCG7aLgzX/evmaMZwxBSfw2p+65geW+++xwUpU29oCfmk/n?=
 =?us-ascii?Q?UfJI9hAooNzyWqmkPVlraEp6JweCB+M2DMYjqaJZd+/btsyUkqZQinvC7Dko?=
 =?us-ascii?Q?np2h37I5xJS/Xg4M5kfs67wcqMIpmjioa9XApOzQDUT89adetyV4cjwc0UwB?=
 =?us-ascii?Q?cuITb63vJSZ5fNZbkmcX6NBpptACVVTrvl8EKoAoLHRr5vC/WXE8lOe5IOGt?=
 =?us-ascii?Q?PSfbDsVKWDukjWGWayyQMDkqujPFWc5PtGnMfHTVaVdrDntQLmtYd5Aj9dVO?=
 =?us-ascii?Q?dcF+PcGnL9M4AyTYjf4WQl1aq7UdwzdqaGMfl40D8sI20SZ3O9+GS/vHsIPD?=
 =?us-ascii?Q?ccQwLjr9eT0iU8KTH3MpUrrIcSAMOeCCW4OuomxGAEMlXtmUhVJIhABfiPvb?=
 =?us-ascii?Q?rtfP+QdFmtZF9kQVqQC4MdCUtxjkx3ZfRxcLQYwLDe/v92nQ9EqB2EI/9k0R?=
 =?us-ascii?Q?GUGbJcCXhiERyqRPNTvTtYLgeNH/IWMuAP0pfd6iKvaqGDZUe0zMg+eFnRBU?=
 =?us-ascii?Q?9OEq40tNOC+tNMcvW4JtG6xT61k41H2FjZJ1MJvND4vG68iqef2ygM+J0JGw?=
 =?us-ascii?Q?FEqAUMWtt4GaCBD1O9sOtxJrSNaXxjNHUPwyYzPe2lTDOaDuD07EQtMlPJRZ?=
 =?us-ascii?Q?m28Ad79MnKgAEd9yBEqxYtAT30zsoWpaXlmBdXXrAVsxiGMcULImkzoEI8gj?=
 =?us-ascii?Q?fvq14RkvwLJD7N1QU1lOTnWXcjflBq+0QRzAHPrZbKWI6jpufGmtBcce/A2P?=
 =?us-ascii?Q?SYkXAQ5LlaCYbGmP2oUc7AgEGKz8rg3E8vI6N3CZ+i5Zg4Qmh0nGus8mnNYM?=
 =?us-ascii?Q?/T/+RsHdUcjVK19FYJSRdjqFE6TbQtuvXDSK+qXxMtWXSEBZaeZjl/EG603x?=
 =?us-ascii?Q?y2fywJPo8HSoDaDo/AgkrgAoUMaxlJReR2LanjKiuoZ4cfnDQ7KeR5Te60hS?=
 =?us-ascii?Q?9y53nAf3/8Nhd1k+L6/Ys9SGjYRy+CJ3Fs1H+/ubcobYZVIlPUjK4oqp812G?=
 =?us-ascii?Q?uadqLyFmCIkbpTTS2mrqfSbBzjR+f/VkUiF7+PPh0IuDRhUkHNedPVZJgkvv?=
 =?us-ascii?Q?jSG31Kkv8D18UW/BYRgjnqpJrbXEihtj5jrXP5SnRgRrkdFtag6Pe8/tLGuM?=
 =?us-ascii?Q?Ky4m5NZKdoiF9tOPlMzQ46NqAIJDJMc+YVJSjUbbMWRgDSADDKE3RgfFHfwb?=
 =?us-ascii?Q?8XZZh6b47mtAZqn6Hgzgn0sFAHIKQn8Y1OLEtIMmFFS1HiAKfkIjKEFrm4zq?=
 =?us-ascii?Q?BiuomkfG4zV9H6lM02M8tSlPSlBI8Xa3iJBUcCjqilndNfPPY8Ab8PUEHF3+?=
 =?us-ascii?Q?58XJHqvqhSnKPF78fPL6CPLF0LFBBQFO70a8ovUaP/k/Ixztkixvl1XURb8Y?=
 =?us-ascii?Q?WIUMd3ryWqVJakEgmqzeZ2hII2AZdCCZmlX9V1N7URLRbu66Knv8tsC7EnQL?=
 =?us-ascii?Q?XK0lg7JnNp29VaHWP77+fMyiDZQwJkLzW+oRMcg8F0dnLUllt+E8d4jNiFOs?=
 =?us-ascii?Q?oGXKmCmfPBWrIdit+v20cpvbUtRO59j63ts60c1q?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: marvell.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MW4PR18MB5244.namprd18.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4f3b7fe7-80a4-449c-cbbe-08dc5c7eec69
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Apr 2024 12:32:18.1364
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: AJmKMhtGFvNBUYSFblIhfcIcC+oKCIi9ZqmRKPpL88EnO+HIgZ8V7gX+bq7l0CzCFbwYQ3v6cLGpC4UKOA/8LQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR18MB4155
X-Proofpoint-GUID: BCIrAFVwTmKVd8xrpwh_drS--PH5ZprT
X-Proofpoint-ORIG-GUID: BCIrAFVwTmKVd8xrpwh_drS--PH5ZprT
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-14_02,2024-04-09_01,2023-05-22_02


> -----Original Message-----
> From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Sent: Sunday, April 14, 2024 3:16 PM
> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>
> Cc: Arnd Bergmann <arnd@arndb.de>; linux-kernel@vger.kernel.org; Jerin
> Jacob <jerinj@marvell.com>
> Subject: Re: [EXTERNAL] Re: [PATCH v5 1/1] misc: mrvl-cn10k-dpi: add
> Octeon CN10K DPI administrative driver
>=20
> On Sun, Apr 14, 2024 at 09:33:37AM +0000, Vamsi Krishna Attunuru wrote:
> >
> >
> > > -----Original Message-----
> > > From: Arnd Bergmann <arnd@arndb.de>
> > > Sent: Sunday, April 14, 2024 12:41 AM
> > > To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; Greg
> > > Kroah-Hartman <gregkh@linuxfoundation.org>
> > > Cc: linux-kernel@vger.kernel.org; Jerin Jacob <jerinj@marvell.com>
> > > Subject: Re: [EXTERNAL] Re: [PATCH v5 1/1] misc: mrvl-cn10k-dpi: add
> > > Octeon CN10K DPI administrative driver
> > >
> > > On Sat, Apr 13, 2024, at 18:17, Vamsi Krishna Attunuru wrote:
> > > > From: Greg KH <gregkh@linuxfoundation.org>
> > > >> On Sat, Apr 13, 2024 at 10:58:37AM +0000, Vamsi Krishna Attunuru
> wrote:
> > > >> > From: Greg KH <gregkh@linuxfoundation.org>
> > > >> >
> > > >> > No, it's a normal PCIe sriov capability implemented in all
> > > >> > sriov capable PCIe
> > > >> devices.
> > > >> > Our PF device aka this driver in kernel space service mailbox
> > > >> > requests from userspace applications via VF devices. For
> > > >> > instance, DPI VF device from user space writes into mailbox
> > > >> > registers and the DPI hardware
> > > >> triggers an interrupt to DPI PF device.
> > > >> > Upon PF interrupt, this driver services the mailbox requests.
> > > >>
> > > >> Isn't that a "normal" PCI thing?  How is this different from
> > > >> other devices that have VF?
> > > >
> > > > Looks like there is a lot of confusion for this device. Let me
> > > > explain There are two aspects for this DPI PF device.
> > > > a) It's a PCIe device so it is "using" some of the PCI services
> > > > provided PCIe HW or PCI subsystem
> > > > b) It is "providing" non PCIe service(DPI HW administrative
> > > > function) by using (a) Let me enumerate PF device operations with
> above aspects.
> > > > 1) Means to create VF(s) from PF. It's category (a) service and
> > > > driver uses API (pci_sriov_configure_simple()) from PCI subsystem
> > > > to implement it.
> > > > 2) Means to get the interrupt(mailbox or any device specific
> > > > interrupt). It's category (a) service and driver uses API
> > > > (pci_alloc_irq_vectors()) from PCI subsystem to implement it.
> > > > 3) Means to get the mailbox content from VF by using (2). It's
> > > > category
> > > > (b) service. This service is not part of PCI specification.
> > > > DPI PF device has the mailbox registers(DPI_MBOX_PF_VF_DATA
> > > registers)
> > > > in its PCIe BAR space which are device specific.
> > > > 4) Upon receiving DPI HW administrative function mailbox request,
> > > > service it. Its category (b) service. This service is not part of
> > > > PCI specification.
> > > > For instance, dpi_queue_open & close are requests sent from DPI VF
> > > > device to DPI PF device for setting up the DPI VF queue resources.
> > > > Once its setup by DPI PF, then DPI VF device can use these queues.
> > > > These queues are not part of PCIe specification. These queues are
> > > > used for making DMA by DPI VF device/driver.
> > >
> > > It's not directly my area either, but as far as I can tell from
> > > reading the competing sr-iov based device drivers, these seem to
> > > handle all of the above in the network driver that owns the PF
> > > rather than a separate driver, e.g. for the first point:
> > >
> > > $ git grep -w sriov_configure.=3D drivers/net/
> > > drivers/net/ethernet/amazon/ena/ena_netdev.c:   .sriov_configure =3D
> > > pci_sriov_configure_simple,
> > > drivers/net/ethernet/amd/pds_core/main.c:       .sriov_configure =3D
> > > pdsc_sriov_configure,
> > > drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c:
> .sriov_configure =3D
> > > bnx2x_sriov_configure,
> > > drivers/net/ethernet/broadcom/bnxt/bnxt.c:      .sriov_configure =3D
> > > bnxt_sriov_configure,
> > > drivers/net/ethernet/cavium/liquidio/lio_main.c:        .sriov_config=
ure =3D
> > > liquidio_enable_sriov,
> > > drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:        .sriov_config=
ure =3D
> > > cxgb4_iov_configure,
> > > drivers/net/ethernet/emulex/benet/be_main.c:    .sriov_configure =3D
> > > be_pci_sriov_configure,
> > > drivers/net/ethernet/freescale/enetc/enetc_pf.c:        .sriov_config=
ure =3D
> > > enetc_sriov_configure,
> > > drivers/net/ethernet/fungible/funeth/funeth_main.c:     .sriov_config=
ure
> =3D
> > > funeth_sriov_configure,
> > > drivers/net/ethernet/hisilicon/hns3/hns3_enet.c:        .sriov_config=
ure =3D
> > > hns3_pci_sriov_configure,
> > > drivers/net/ethernet/huawei/hinic/hinic_main.c: .sriov_configure =3D
> > > hinic_pci_sriov_configure,
> > > drivers/net/ethernet/intel/fm10k/fm10k_pci.c:   .sriov_configure     =
   =3D
> > > fm10k_iov_configure,
> > > drivers/net/ethernet/intel/i40e/i40e_main.c:    .sriov_configure =3D
> > > i40e_pci_sriov_configure,
> > > drivers/net/ethernet/intel/ice/ice_main.c:      .sriov_configure =3D
> > > ice_sriov_configure,
> > > drivers/net/ethernet/intel/idpf/idpf_main.c:    .sriov_configure     =
   =3D
> > > idpf_sriov_configure,
> > > drivers/net/ethernet/intel/igb/igb_main.c:      .sriov_configure =3D
> > > igb_pci_sriov_configure,
> > > drivers/net/ethernet/intel/ixgbe/ixgbe_main.c:  .sriov_configure =3D
> > > ixgbe_pci_sriov_configure,
> > > drivers/net/ethernet/marvell/octeon_ep/octep_main.c:
> .sriov_configure =3D
> > > octep_sriov_configure,
> > > drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c:   .sriov_config=
ure
> =3D
> > > otx2_sriov_configure
> > > drivers/net/ethernet/netronome/nfp/nfp_main.c:  .sriov_configure
> =3D
> > > nfp_pcie_sriov_configure,
> > > drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c:    .sriov_config=
ure
> =3D
> > > ionic_sriov_configure,
> > > drivers/net/ethernet/qlogic/qede/qede_main.c:   .sriov_configure =3D
> > > qede_sriov_configure,
> > > drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c:       .sriov_config=
ure =3D
> > > qlcnic_pci_sriov_configure,
> > > drivers/net/ethernet/sfc/ef10.c:        .sriov_configure =3D
> > > efx_ef10_sriov_configure,
> > > drivers/net/ethernet/sfc/ef100.c:       .sriov_configure =3D
> > > ef100_pci_sriov_configure,
> > > drivers/net/ethernet/sfc/ef100_nic.c:   .sriov_configure =3D
> > > IS_ENABLED(CONFIG_SFC_SRIOV) ?
> > > drivers/net/ethernet/sfc/efx.c: .sriov_configure =3D
> efx_pci_sriov_configure,
> > > drivers/net/ethernet/sfc/siena/efx.c:   .sriov_configure =3D
> > > efx_pci_sriov_configure,
> > > drivers/net/ethernet/sfc/siena/siena.c: .sriov_configure =3D
> > > efx_siena_sriov_configure,
> > >
> > > In what way is your hardware different from all the others?
> >
> > All of above devices are network devices which implements struct
> net_device_ops.
> > i.e Those PCI devices are networking devices which are capable of
> sending/receiving network packets.
> > This device doesn't have networking functionality to implement struct
> > net_device_ops, It's a simple PCIe PF device enables it's VFs and servi=
ces
> any mailbox requests.
>=20
> What driver handles the "mailbox requests"?  What are these requests for?

I think, I have already mentioned this in the previous reply. Copying the s=
ame here. Please see (4) in [1].
The DPI PF driver(this driver) handles the mailbox request from DPI VF (a u=
serspace driver implemented using vfio uapis).

DPI PF driver(this driver) does not fit to any of the device class(ethernet=
, crypto, dma..) in linux kernel hence making it=20
as misc device driver. Standard device class always will have standard devi=
ce function. For example,
ethernet device to send/receive ethernet frames, crypto device to enable cr=
ypto transformations,
dma device to enable copying data from source to destination memory.

Now, Why HW designers choose to have DPI(DMA Engine) PF device in first pla=
ce?

a) DPI VF(not DPI PF) device has capability to do the DMA (copying data fro=
m source to destination memory).
b) In order to do DMA, DPI VF device needs a DPI queue.=20
c) Now here is the catch, when VF device starts, it does not have its queue=
s configured. So, DPI VF device/driver
asks(via mailbox) DPI PF(this driver) for the queue setup with required con=
figuration. dpi_queue_open() does the same.
d) Now you may ask, why VF device does NOT configure its queues on its own.=
 That is HW resources
provision optimization(introduced by HW designers), where DMA engines are p=
rovisioned across the VF device queues.
So, PF (administrative)  arbitrates the request from different VF devices v=
ia mailbox and allow to configure _global_ resources
which does not belong to VF.

Hope it clarifies.

[1]
------------------------------
Looks like there is a lot of confusion for this device. Let me explain

There are two aspects for this DPI PF device.

a) It's a PCIe device so it is "using" some of the PCI services provided PC=
Ie HW or PCI subsystem
b) It is "providing" non PCIe service(DPI HW administrative function) by us=
ing (a)

Let me enumerate PF device operations with above aspects.

1) Means to create VF(s) from PF. It's category (a) service and driver uses=
 API (pci_sriov_configure_simple()) from PCI subsystem to implement it.
2) Means to get the interrupt(mailbox or any device specific interrupt). It=
's category (a) service and driver uses API (pci_alloc_irq_vectors()) from =
PCI subsystem to implement it.
3) Means to get the mailbox content from VF by using (2). It's category (b)=
 service. This service is not part of PCI specification.
DPI PF device has the mailbox registers(DPI_MBOX_PF_VF_DATA registers) in i=
ts PCIe BAR space which are device specific.
4) Upon receiving DPI HW administrative function mailbox request, service i=
t. Its category (b) service. This service is not part of PCI specification.
For instance, dpi_queue_open & close are requests sent from DPI VF device t=
o DPI PF device for setting up the DPI VF queue resources. Once its setup b=
y DPI PF,
then DPI VF device can use these queues. These queues are not part of PCIe =
specification. These queues are used for making DMA by DPI VF device/driver=
.

-------------------------------
>=20
> thanks,
>=20
> greg k-h