[permalink] [raw]

Subject: Re: [PATCH 09/11] PCI: add matching checks for driver_override binding

On 6/9/2021 4:27 AM, Alex Williamson wrote:
> On Tue, 8 Jun 2021 19:45:17 -0300
> Jason Gunthorpe <[email protected]> wrote:
>
>> On Tue, Jun 08, 2021 at 03:26:43PM -0600, Alex Williamson wrote:
>>>> drivers that specifically opt into this feature and the driver now has
>>>> the opportunity to provide a proper match table that indicates what HW
>>>> it can properly support. vfio-pci continues to support everything.
>>> In doing so, this also breaks the new_id method for vfio-pci.
>> Does it? How? The driver_override flag is per match entry not for the
>> entire device so new_id added things will work the same as before as
>> their new match entry's flags will be zero.
> Hmm, that might have been a testing issue; combining driverctl with
> manual new_id testing might have left a driver_override in place.
>
>>> Sorry, with so many userspace regressions, crippling the
>>> driver_override interface with an assumption of such a narrow focus,
>>> creating a vfio specific match flag, I don't see where this can go.
>>> Thanks,
>> On the other hand it overcomes all the objections from the last go
>> round: how userspace figures out which driver to use with
>> driver_override and integrating the universal driver into the scheme.
>>
>> pci_stub could be delt with by marking it for driver_override like
>> vfio_pci.
> By marking it a "vfio driver override"? :-\
>
>> But driverctl as a general tool working with any module is not really
>> addressable.
>>
>> Is the only issue the blocking of the arbitary binding? That is not a
>> critical peice of this, IIRC
> We can't break userspace, which means new_id and driver_override need
> to work as they do now. There are scads of driver binding scripts in
> the wild, for vfio-pci and other drivers. We can't assume such a
> narrow scope. Thanks,

what about the following code ?

@@ -152,12 +152,28 @@ static const struct pci_device_id
*pci_match_device(struct pci_driver *drv,
        }
        spin_unlock(&drv->dynids.lock);

-       if (!found_id)
-               found_id = pci_match_id(drv->id_table, dev);
+       if (found_id)
+               return found_id;

-       /* driver_override will always match, send a dummy id */
-       if (!found_id && dev->driver_override)
+       found_id = pci_match_id(drv->id_table, dev);
+       if (found_id) {
+               /*
+                * if we found id in the static table, we must fulfill the
+                * matching flags (i.e. if PCI_ID_F_DRIVER_OVERRIDE flag is
+                * set, driver_override should be provided).
+                */
+               bool is_driver_override =
+                       (found_id->flags & PCI_ID_F_DRIVER_OVERRIDE) != 0;
+               if ((is_driver_override && !dev->driver_override) ||
+                   (dev->driver_override && !is_driver_override))
+                       return NULL;
+       } else if (dev->driver_override) {
+               /*
+                * if we didn't find suitable id in the static table,
+                * driver_override will still , send a dummy id
+                */
                found_id = &pci_device_id_any;
+       }

        return found_id;
}

dynamic ids (new_id) works as before.

Old driver_override works as before.

For "new" driver_override we must fulfill the new rules.

>
> Alex
>

2021-06-14 05:46:54

by Christoph Hellwig

[permalink] [raw]

Subject: Re: [PATCH 09/11] PCI: add matching checks for driver_override binding

On Sun, Jun 13, 2021 at 11:19:46AM +0300, Max Gurtovoy wrote:
> what about the following code ?
>
> @@ -152,12 +152,28 @@ static const struct pci_device_id
> *pci_match_device(struct pci_driver *drv,
> ?????????????? }
> ?????????????? spin_unlock(&drv->dynids.lock);
>
> -???????????? if (!found_id)
> -???????????????????????????? found_id = pci_match_id(drv->id_table, dev);
> +???????????? if (found_id)
> +???????????????????????????? return found_id;

Something is broken in your mailer because this does not look like code
at all.

2021-06-14 08:21:22

On 6/17/2021 2:56 AM, Jason Gunthorpe wrote:
> On Thu, Jun 17, 2021 at 02:51:09AM +0300, Max Gurtovoy wrote:
>> On 6/17/2021 2:44 AM, Jason Gunthorpe wrote:
>>> On Thu, Jun 17, 2021 at 02:42:46AM +0300, Max Gurtovoy wrote:
>>>
>>>> Do you see a reason not adding this alias for stub drivers but
>>>> adding it to vfio_pci drivers ?
>>> It creates uABI without a userspace user and that is strongly
>>> discouraged. The 'stub_pci:' prefix becomes fixed ABI.
>> so is it better to have "pci:v*d*sv*sd*bc*sc*i*" for stub drivers ?
> No, we don't want to convey any new information about stub drivers to
> userspace.
>
>> or not adding alias at all if stub flag is set ?
> Yes, just don't change it at all, IMHO.

Ok.

I've attached 2 patches that I would like to agree on before we'll send
the V5.

They include the pci-pf-stub additions and keep searching for matching
static ids as we discussed.

Max.

>
> Jason

Attachments:

0001-PCI-add-flags-field-to-pci_device_id-structure.patch (6.09 kB)
0002-PCI-add-matching-checks-for-driver_override-binding.patch (6.08 kB)
Download all attachments

2021-07-30 07:55:47

by Shameerali Kolothum Thodi

[permalink] [raw]

Subject: RE: [RFC PATCH v4 00/11] Introduce vfio-pci-core subsystem

Hi Max/ Yishai,

(Sorry I picked this thread instead of the [1] here as I don't have that
in my mailbox)

I see that an update to this series has been posted by Yishai [1] and it mentions
about a branch with all relevant patches,

" A preview of all the patches can be seen here:
https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci"

But sorry I couldn't find the patches in the branch above. Could you
please check and let me know.

Thanks,
Shameer

[1] https://lore.kernel.org/kvm/[email protected]/#R

> -----Original Message-----
> From: Max Gurtovoy [mailto:[email protected]]
> Sent: 03 June 2021 17:08
> To: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]; Shameerali
> Kolothum Thodi <[email protected]>; liulongfang
> <[email protected]>; [email protected]; Max Gurtovoy
> <[email protected]>
> Subject: [RFC PATCH v4 00/11] Introduce vfio-pci-core subsystem
>
> Hi Alex, Cornelia, Jason and Co,
>
> This series split the vfio_pci driver into 2 parts: pci drivers and a
> subsystem driver that will also be library of code. The main pci driver,
> vfio_pci.ko will be used as before and it will bind to the subsystem
> driver vfio_pci_core.ko to register to the VFIO subsystem.
>
> This series is coming to solve some of the issues that were raised in
> the previous attempts for extending vfio-pci for vendor specific
> functionality:
> 1. https://lkml.org/lkml/2020/5/17/376 by Yan Zhao.
> 2. https://www.spinics.net/lists/kernel/msg3903996.html by Longfang Liu
>
> This subsystem framework will also ease on adding new vendor specific
> functionality to VFIO devices in the future by allowing another module
> to provide the pci_driver that can setup number of details before
> registering to VFIO subsystem (such as inject its own operations).
>
> This series also extends the "driver_override" mechanism. We added a flag
> for pci drivers that will declare themselves as "driver_override" capable
> and only declared drivers can use this mechanism in the PCI subsystem.
> Other drivers will not be able to bind to devices that use "driver_override".
> Also, the PCI driver matching will always look for ID table and will never
> generate dummy "match_all" ID table in the PCI subsystem layer. In this
> way, we ensure deterministic behaviour with no races with the original
> pci drivers. In order to get the best match for "driver_override" drivers,
> one can create a userspace program (example can be found at
> https://github.com/maxgurtovoy/linux_tools/blob/main/vfio/bind_vfio_pci_dr
> iver.py)
> that find the 'best match' according to simple algorithm: "the driver
> with the fewest '*' matches wins."
> For example, the vfio-pci driver will match to any pci device. So it
> will have the maximal '*' matches (for all matching IDs: vendor, device,
> subvendor, ...).
> In case we are looking for a match to mlx5 based device, we'll have a
> match to vfio-pci.ko and mlx5-vfio-pci.ko. We'll prefer mlx5-vfio-pci.ko
> since it will have less '*' matches (probably vendor and device IDs will
> match). This will work in the future for NVMe/Virtio devices that can
> match according to a class code or other criteria.
>
> The main goal of this series is to agree on the vfio_pci module split and the
> "driver_override" extensions. The follow-up version will include an extended
> mlx5_vfio_pci driver that will support VF suspend/resume as well.
>
> This series applied cleanly on top of vfio reflck re-design (still haven't sent
> for review) and can be found at:
> https://github.com/Mellanox/NVMEoF-P2P/tree/vfio-v4-external.
>
> Max Gurtovoy (11):
> vfio-pci: rename vfio_pci.c to vfio_pci_core.c
> vfio-pci: rename vfio_pci_private.h to vfio_pci_core.h
> vfio-pci: rename vfio_pci_device to vfio_pci_core_device
> vfio-pci: rename ops functions to fit core namings
> vfio-pci: include vfio header in vfio_pci_core.h
> vfio-pci: introduce vfio_pci.c
> vfio-pci: move igd initialization to vfio_pci.c
> PCI: add flags field to pci_device_id structure
> PCI: add matching checks for driver_override binding
> vfio-pci: introduce vfio_pci_core subsystem driver
> mlx5-vfio-pci: add new vfio_pci driver for mlx5 devices
>
> Documentation/ABI/testing/sysfs-bus-pci | 6 +-
> Documentation/PCI/pci.rst | 1 +
> drivers/pci/pci-driver.c | 22 +-
> drivers/vfio/pci/Kconfig | 27 +-
> drivers/vfio/pci/Makefile | 12 +-
> drivers/vfio/pci/mlx5_vfio_pci.c | 130 +
> drivers/vfio/pci/vfio_pci.c | 2329 +----------------
> drivers/vfio/pci/vfio_pci_config.c | 70 +-
> drivers/vfio/pci/vfio_pci_core.c | 2239 ++++++++++++++++
> drivers/vfio/pci/vfio_pci_igd.c | 16 +-
> drivers/vfio/pci/vfio_pci_intrs.c | 42 +-
> drivers/vfio/pci/vfio_pci_rdwr.c | 18 +-
> drivers/vfio/pci/vfio_pci_zdev.c | 4 +-
> include/linux/mod_devicetable.h | 9 +
> include/linux/pci.h | 27 +
> .../linux/vfio_pci_core.h | 93 +-
> scripts/mod/devicetable-offsets.c | 1 +
> scripts/mod/file2alias.c | 8 +-
> 18 files changed, 2695 insertions(+), 2359 deletions(-)
> create mode 100644 drivers/vfio/pci/mlx5_vfio_pci.c
> create mode 100644 drivers/vfio/pci/vfio_pci_core.c
> rename drivers/vfio/pci/vfio_pci_private.h => include/linux/vfio_pci_core.h
> (56%)
>
> --
> 2.21.0

2021-07-30 11:57:13

by Jason Gunthorpe

[permalink] [raw]

Subject: Re: [RFC PATCH v4 00/11] Introduce vfio-pci-core subsystem

On Fri, Jul 30, 2021 at 07:53:00AM +0000, Shameerali Kolothum Thodi wrote:
> Hi Max/ Yishai,
>
> (Sorry I picked this thread instead of the [1] here as I don't have that
> in my mailbox)
>
> I see that an update to this series has been posted by Yishai [1] and it mentions
> about a branch with all relevant patches,
>
> " A preview of all the patches can be seen here:
> https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci"
>
> But sorry I couldn't find the patches in the branch above. Could you
> please check and let me know.

I fixed it

Jason