Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 3 Jun 2021 21:40:09 -0600
From:   Alex Williamson <alex.williamson@redhat.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>
Cc:     Jason Gunthorpe <jgg@nvidia.com>,
        Cornelia Huck <cohuck@redhat.com>,
        Kirti Wankhede <kwankhede@nvidia.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "vkoul@kernel.org" <vkoul@kernel.org>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Dey, Megha" <megha.dey@intel.com>,
        "Pan, Jacob jun" <jacob.jun.pan@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>, "Lu, Baolu" <baolu.lu@intel.com>,
        "Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "dmaengine@vger.kernel.org" <dmaengine@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>
Subject: Re: [PATCH v6 00/20] Add VFIO mediated device support and DEV-MSI
 support for the idxd driver
Message-ID: <20210603214009.68fac0c4.alex.williamson@redhat.com>
In-Reply-To: <MWHPR11MB1886D613948986530E9B61CB8C3C9@MWHPR11MB1886.namprd11.prod.outlook.com>
References: <162164243591.261970.3439987543338120797.stgit@djiang5-desk3.ch.intel.com>
        <20210523232219.GG1002214@nvidia.com>
        <86cde154-37c7-c00d-b0c6-06b15b50dbf7@intel.com>
        <20210602231747.GK1002214@nvidia.com>
        <MWHPR11MB188664D9E7CA60782B75AC718C3C9@MWHPR11MB1886.namprd11.prod.outlook.com>
        <20210603014932.GN1002214@nvidia.com>
        <MWHPR11MB1886D613948986530E9B61CB8C3C9@MWHPR11MB1886.namprd11.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Thu, 3 Jun 2021 05:52:58 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Thursday, June 3, 2021 9:50 AM
> > 
> > On Thu, Jun 03, 2021 at 01:11:37AM +0000, Tian, Kevin wrote:
> >   
> > > Jason, can you clarify your attitude on mdev guid stuff? Are you
> > > completely against it or case-by-case? If the former, this is a big
> > > decision thus it's better to have consensus with Alex/Kirti. If the
> > > latter, would like to hear your criteria for when it can be used
> > > and when not...  
> > 
> > I dislike it generally, but it exists so <shrug>. I know others feel
> > more strongly about it being un-kernely and the wrong way to use sysfs.
> > 
> > Here I was remarking how the example in the cover letter made the mdev
> > part seem totally pointless. If it is pointless then don't do it.  
> 
> Is your point about that as long as a mdev requires pre-config
> through driver specific sysfs then it doesn't make sense to use
> mdev guid interface anymore?

Can you describe exactly what step 1. is doing in this case from the
original cover letter ("Enable wq with "mdev" wq type")?  That does
sound a bit like configuring something to use mdev then separately
going to the trouble of creating the mdev.  As Jason suggests, if a wq
is tagged for mdev/vfio, it could just register itself as a vfio bus
driver.

But if we want to use mdev, why doesn't available_instances for your
mdev type simply report all unassigned wq and the `echo $UUID > create`
grabs a wq for mdev?  That would remove this pre-config contention,
right?

> The value of mdev guid interface is providing a vendor-agnostic
> interface for mdev life-cycle management which allows one-
> enable-fit-all in upper management stack. Requiring vendor
> specific pre-config does blur the boundary here.

We need to be careful about using work-avoidance in the upper
management stack as a primary use case for an interface though.

> Alex/Kirt/Cornelia, what about your opinion here? It's better 
> we can have an consensus on when and where the existing
> mdev sysfs could be used, as this will affect every new mdev
> implementation from now on.

I have a hard time defining some fixed criteria for using mdev.  It's
essentially always been true that vendors could write their own vfio
"bus driver", like vfio-pci or vfio-platform, specific to their device.
Mdevs were meant to be a way for the (non-vfio) driver of a device to
expose portions of the device through mediation for use with vfio.  It
seems like that's largely being done here.

What I think has changed recently is this desire to make it easier to
create those vendor drivers and some promise of making module binding
work to avoid the messiness around picking a driver for the device.  In
the auxiliary bus case that I think Jason is working on, it sounds like
the main device driver exposes portions of itself on an auxiliary bus
where drivers on that bus can integrate into the vfio subsystem.  It
starts to get pretty fuzzy with what mdev already does, but it's also a
more versatile interface.  Is it right for everyone?  Probably not.

Is the pre-configuration issue here really a push vs pull problem?  I
can see the requirement in step 1. is dedicating some resources to an
mdev use case, so at that point it seems like the argument is whether we
should just create aux bus devices that get automatically bound to a
vendor vfio-pci variant and we avoid the mdev lifecycle, which is both
convenient and ugly.  On the other hand, mdev has more of a pull
interface, ie. here are a bunch of device types and how many of each we
can support, use create to pull what you need.

> > Remember we have stripped away the actual special need to use
> > mdev. You don't *have* to use mdev anymore to use vfio. That is a
> > significant ideology change even from a few months ago.
> >   
> 
> Yes, "don't have to" but if there is value of doing so  it's
> not necessary to blocking it? One point in my mind is that if
> we should minimize vendor-specific contracts for user to
> manage mdev or subdevice...

Again, this in itself is not a great justification for using mdev,
we're creating vendor specific device types with vendor specific
additional features, that could all be done via some sort of netlink
interface too.  The thing that pushes this more towards mdev for me is
that I don't think each of these wqs appear as devices to the host,
they're internal resources of the parent device and we want to compose
them in ways that are slightly more amenable to traditional mdevs... I
think.  Thanks,

Alex