Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Fri, 13 Dec 2019 10:08:28 -0800
From:   Jakub Kicinski <jakub.kicinski@netronome.com>
To:     Yuval Avnery <yuvalav@mellanox.com>
Cc:     Jiri Pirko <jiri@mellanox.com>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Andy Gospodarek <andy@greyhouse.net>
Subject: Re: [PATCH net-next] netdevsim: Add max_vfs to bus_dev
Message-ID: <20191213100828.6767de6e@cakuba.netronome.com>
In-Reply-To: <AM6PR05MB514261CD6F95F104C0353A4BC5540@AM6PR05MB5142.eurprd05.prod.outlook.com>
References: <1576033133-18845-1-git-send-email-yuvalav@mellanox.com>
        <20191211095854.6cd860f1@cakuba.netronome.com>
        <AM6PR05MB514244DC6D25DDD433C0E238C55A0@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191211111537.416bf078@cakuba.netronome.com>
        <AM6PR05MB5142CCAB9A06DAC199F7100CC55A0@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191211142401.742189cf@cakuba.netronome.com>
        <AM6PR05MB51423D365FB5A8DB22B1DE62C55A0@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191211154952.50109494@cakuba.netronome.com>
        <AM6PR05MB51425B74E736C5D765356DC8C5550@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191212102517.602a8a5d@cakuba.netronome.com>
        <AM6PR05MB5142F0F18EA6B6F16C5888CEC5550@AM6PR05MB5142.eurprd05.prod.outlook.com>
        <20191212175418.3b07b7a9@cakuba.netronome.com>
        <AM6PR05MB514261CD6F95F104C0353A4BC5540@AM6PR05MB5142.eurprd05.prod.outlook.com>
Organization: Netronome Systems, Ltd.
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Fri, 13 Dec 2019 03:21:02 +0000, Yuval Avnery wrote:
> > I see, is this a more fine grained capability or all or nothing for SR-IOV control?
> > I'd think that if the SmartNIC's eswitch just encapsulates all the frames into a
> > L4 tunnel it shouldn't care about L2 addresses.  
> 
> People keep saying that, but there are customers who wants this capability :)

Right, but we should have a plan for both, right? Some form of a switch
between L4/no checking/ip link changes are okay vs strict checking/L2/
SmartNIC provisions MAC addrs?

> > > > What happens if the SR-IOV host changes the MAC? Is it used by HW or
> > > > is the MAC provisioned by the control CPU used for things like spoof  
> > > > check?  
> > >
> > > Host shouldn't have privileges to do it.
> > > If it does, then it's under the host ownership (like in non-smartnic mode).  
> > 
> > I see so the MAC is fixed from bare metal host's PoV? And it has to be set  
> 
> Yes
> 
> > through some high level cloud API (for live migration etc)?
> > Do existing software stacks like libvirt handle not being able to set the MAC
> > happily?  
> 
> I am not sure what you mean.
> What we are talking about here is the E-switch manager setting a MAC to another VF.
> When the VF driver loads it will query this MAC from the NIC. This is the way
> It works today with "ip link set _vf_ mac"
> 
> Or in other words we are replacing "ip link set _vf_ mac" and not "ip link set address"
> So that it can work from the SmartNic embedded system.
> There is nothing really new here, ip link will not work from a SmartNic,
> this is why need devlink subdev.

Ack, but are we targeting the bare metal cloud scenario here or
something more limited? In a bare metal cloud AFAIU the customers
can use SR-IOV on the host, but the MACs need to be communicated/
/requested from the cloud management system.

IOW the ip link and the devlink APIs are in different domains of
control. Customer has access to ip link and provider has access to
devlink.

So my question is does libvirt run by the customer handle the fact 
that it can't poke at ip link gracefully, and if live migration is
involved how is the customer supposed to ask the provider to move an
address?