Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Yuval Avnery <yuvalav@mellanox.com>
To:     Jakub Kicinski <jakub.kicinski@netronome.com>
CC:     Jiri Pirko <jiri@mellanox.com>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Andy Gospodarek <andy@greyhouse.net>,
        Daniel Jurgens <danielj@mellanox.com>
Subject: RE: [PATCH net-next] netdevsim: Add max_vfs to bus_dev
Thread-Topic: [PATCH net-next] netdevsim: Add max_vfs to bus_dev
Thread-Index: AQHVr871nDBB5hrvy0ak2C7+9Ay1lae1Oa8AgAAB1gCAABOZgIAACO8QgAArtYCAAAxfkIAAC50AgABVJmCAAOJ/gIAAIpgggABa3ACAABH7IIAA/jMAgAAW14A=
Date:   Fri, 13 Dec 2019 20:05:00 +0000
Message-ID: <AM6PR05MB51422CE9C249DB03F486CB63C5540@AM6PR05MB5142.eurprd05.prod.outlook.com>
References: <1576033133-18845-1-git-send-email-yuvalav@mellanox.com>
        <20191211095854.6cd860f1@cakuba.netronome.com>
        <AM6PR05MB514244DC6D25DDD433C0E238C55A0@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191211111537.416bf078@cakuba.netronome.com>
        <AM6PR05MB5142CCAB9A06DAC199F7100CC55A0@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191211142401.742189cf@cakuba.netronome.com>
        <AM6PR05MB51423D365FB5A8DB22B1DE62C55A0@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191211154952.50109494@cakuba.netronome.com>
        <AM6PR05MB51425B74E736C5D765356DC8C5550@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191212102517.602a8a5d@cakuba.netronome.com>
        <AM6PR05MB5142F0F18EA6B6F16C5888CEC5550@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191212175418.3b07b7a9@cakuba.netronome.com>
        <AM6PR05MB514261CD6F95F104C0353A4BC5540@AM6PR05MB5142.eurprd05.prod.outlook.com>
 <20191213100828.6767de6e@cakuba.netronome.com>
In-Reply-To: <20191213100828.6767de6e@cakuba.netronome.com>
Accept-Language: he-IL, en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 00ebf884-c424-482e-898a-08d78007bbff
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Dec 2019 20:05:00.1838
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: vLvomEFwfvhZ1uZKICI5wBOzsBrEkxpXqDx7WrBlbqF6GxbjplWGdsab59FO55nXzOA5D77yormsv12WSx7yeQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB4872
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> -----Original Message-----
> From: Jakub Kicinski <jakub.kicinski@netronome.com>
> Sent: Friday, December 13, 2019 10:08 AM
> To: Yuval Avnery <yuvalav@mellanox.com>
> Cc: Jiri Pirko <jiri@mellanox.com>; davem@davemloft.net;
> netdev@vger.kernel.org; linux-kernel@vger.kernel.org; Andy Gospodarek
> <andy@greyhouse.net>
> Subject: Re: [PATCH net-next] netdevsim: Add max_vfs to bus_dev
>=20
> On Fri, 13 Dec 2019 03:21:02 +0000, Yuval Avnery wrote:
> > > I see, is this a more fine grained capability or all or nothing for S=
R-IOV
> control?
> > > I'd think that if the SmartNIC's eswitch just encapsulates all the
> > > frames into a
> > > L4 tunnel it shouldn't care about L2 addresses.
> >
> > People keep saying that, but there are customers who wants this
> > capability :)
>=20
> Right, but we should have a plan for both, right? Some form of a switch
> between L4/no checking/ip link changes are okay vs strict checking/L2/
> SmartNIC provisions MAC addrs?

I am not sure I understand
The L2 checks will be on NIC, not on the switch.
Packet decapsulated and forwarded to the NIC, Where the MAC matters..

>=20
> > > > > What happens if the SR-IOV host changes the MAC? Is it used by
> > > > > HW or is the MAC provisioned by the control CPU used for things
> > > > > like spoof check?
> > > >
> > > > Host shouldn't have privileges to do it.
> > > > If it does, then it's under the host ownership (like in non-smartni=
c
> mode).
> > >
> > > I see so the MAC is fixed from bare metal host's PoV? And it has to
> > > be set
> >
> > Yes
> >
> > > through some high level cloud API (for live migration etc)?
> > > Do existing software stacks like libvirt handle not being able to
> > > set the MAC happily?
> >
> > I am not sure what you mean.
> > What we are talking about here is the E-switch manager setting a MAC to
> another VF.
> > When the VF driver loads it will query this MAC from the NIC. This is
> > the way It works today with "ip link set _vf_ mac"
> >
> > Or in other words we are replacing "ip link set _vf_ mac" and not "ip l=
ink set
> address"
> > So that it can work from the SmartNic embedded system.
> > There is nothing really new here, ip link will not work from a
> > SmartNic, this is why need devlink subdev.
>=20
> Ack, but are we targeting the bare metal cloud scenario here or something
> more limited? In a bare metal cloud AFAIU the customers can use SR-IOV on
> the host, but the MACs need to be communicated/ /requested from the
> cloud management system.

Yes, so the cloud management system communicates with the Control CPU, not =
the host,
Not whatever customer decides to run on the hypervisor. The host PF is powe=
rless here (almost like VF).

>=20
> IOW the ip link and the devlink APIs are in different domains of control.
> Customer has access to ip link and provider has access to devlink.

For host VF - Customer has access to ip link exactly like in non-smartnic m=
ode.
For host PF - "ip link set vf" will return error. Everything running on the=
 host is not-trusted.

>=20
> So my question is does libvirt run by the customer handle the fact that i=
t can't
> poke at ip link gracefully, and if live migration is involved how is the =
customer
> supposed to ask the provider to move an address?

I don't understand the question because I don't understand why is it differ=
ent
from non-smartnic where the host hypervisor is in-charge.