Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Parav Pandit <parav@mellanox.com>
To:     Greg KH <gregkh@linuxfoundation.org>
CC:     "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "michal.lkml@markovi.net" <michal.lkml@markovi.net>,
        "davem@davemloft.net" <davem@davemloft.net>,
        Jiri Pirko <jiri@mellanox.com>,
        Jakub Kicinski <jakub.kicinski@netronome.com>
Subject: RE: [RFC net-next 8/8] net/mlx5: Add subdev driver to bind to subdev
 devices
Thread-Topic: [RFC net-next 8/8] net/mlx5: Add subdev driver to bind to subdev
 devices
Thread-Index: AQHUz/EC2njdeQVegE6Mw8pbS3FanaX2XwcAgACa0LA=
Date:   Fri, 1 Mar 2019 17:21:13 +0000
Message-ID: <AM4PR0501MB22603530A59C625CDF48D978D1760@AM4PR0501MB2260.eurprd05.prod.outlook.com>
References: <1551418672-12822-1-git-send-email-parav@mellanox.com>
 <1551418672-12822-9-git-send-email-parav@mellanox.com>
 <20190301072158.GC8975@kroah.com>
In-Reply-To: <20190301072158.GC8975@kroah.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 1486a780-e11a-4f9a-f087-08d69e6a4df2
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Mar 2019 17:21:13.0312
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR0501MB2610
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Friday, March 1, 2019 1:22 AM
> To: Parav Pandit <parav@mellanox.com>
> Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
> michal.lkml@markovi.net; davem@davemloft.net; Jiri Pirko
> <jiri@mellanox.com>
> Subject: Re: [RFC net-next 8/8] net/mlx5: Add subdev driver to bind to
> subdev devices
>=20
> On Thu, Feb 28, 2019 at 11:37:52PM -0600, Parav Pandit wrote:
> > Add a subdev driver to probe the subdev devices and create fake
> > netdevice for it.
>=20
> So I'm guessing here is the "meat" of the whole goal here?
>=20
> You just want multiple netdevices per PCI device?  Why can't you do that
> today in your PCI driver?
>=20
Yes, but it just not multiple netdevices.
Let me please elaborate in detail.

There is a swichdev mode of a PCI function for netdevices.
In this mode a given netdev has additional control netdev (called represent=
or netdevice =3D rep-ndev).
This rep-ndev is attached to OVS for adding rules, offloads etc using stand=
ard tc, netfilter infra.
Currently this rep-ndev controls switch side of the settings, but not the h=
ost side of netdev.
So there is discussion to create another netdev or devlink port..

Additionally this subdev has optional rdma device too.

And when we are in switchdev mode, this rdma dev has similar rdma rep devic=
e for control.

In some cases we actually don't create netdev when it is in InfiniBand mode=
.
Here there is PCI device->rdma_device.

In other case, a given sub device for rdma is dual port device, having netd=
evice for each that can use existing netdev->dev_port.

Creating 4 devices of two different classes using one iproute2/ip or iprout=
e2/rdma command is horrible thing to do.

In case if this sub device has to be a passthrough device, ip link command =
will fail badly that day, because we are creating some sub device which is =
not even a netdevice.

So iproute2/devlink which works on bus+device, mainly PCI today, seems righ=
t abstraction point to create sub devices.
This also extends to map ports of the device, health, registers debug, etc =
rich infrastructure that is already built.

Additionally, we don't want mlx driver and other drivers to go through its =
child devices (split logic in netdev and rdma) for power management.
Kernel core code does that well today, that we like to leverage through sub=
dev bus or mfd pm callbacks.

So it is lot more than just creating netdevices.

> What problem are you trying to solve that others also are having that
> requires all of this?
>=20
> Adding a new bus type and subsystem is fine, but usually we want more
> than just one user of it, as this does not really show how it is exercise=
d very
> well.
This subdev and devlink infrastructure solves this problem of creating smal=
ler sub devices out of one PCI device.
Someone has to start.. :-)

To my knowledge, currently Netronome, Broadcom and Mellanox are actively us=
ing this devlink and switchdev infra today.
I added Jakub from Netronome, he is in netdev mailing list, but added in CC=
, to listen his feedback.

> Ideally 3 users would be there as that is when it proves itself that it i=
s
> flexible enough.
>=20

We were looking at drivers/visorbus if we can repurpose it, but GUID device=
 naming scheme is just not user friendly.
It has only single s-Par user and whose guest drivers are still in staging =
for more than a year now. So doesn't really fit well.

> Would just using the mfd subsystem work better for you?  That provides
> core support for "multi-function" drivers/devices already.  What is missi=
ng
> from that subsystem that does not work for you here?
>=20
We were not aware of mfd until now. I looked at very high level now. It's a=
 wrapper to platform devices and seems widely use.
Before subdev proposal, Jason suggested an alternative is to create platfor=
m devices and driver attach to it.

When I read kernel documentation [1], it says "platform devices typically a=
ppear as autonomous entities"
Here instead of autonomy, it is in user's control.
Platform devices probably don't disappear a lot in live system as opposed t=
o subdevices which are created and removed dynamically a lot often.

Not sure if platform device is abuse for this purpose or not.
So which direction to go, devlink->mfd(platform wrapper) or devlink->subdev=
 would be obviously a huge blessing.

[1] https://www.kernel.org/doc/Documentation/driver-model/platform.txt