Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Subject: Re: [PATCH 8/9] vfio/pci: export nvlink2 support into vendor vfio_pci
 drivers
To:     Alexey Kardashevskiy <aik@ozlabs.ru>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     <alex.williamson@redhat.com>, <cohuck@redhat.com>,
        <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <liranl@nvidia.com>, <oren@nvidia.com>, <tzahio@nvidia.com>,
        <leonro@nvidia.com>, <yarong@nvidia.com>, <aviadye@nvidia.com>,
        <shahafs@nvidia.com>, <artemp@nvidia.com>, <kwankhede@nvidia.com>,
        <ACurrid@nvidia.com>, <cjia@nvidia.com>, <yishaih@nvidia.com>,
        <mjrosato@linux.ibm.com>, <hch@lst.de>
References: <20210309083357.65467-1-mgurtovoy@nvidia.com>
 <20210309083357.65467-9-mgurtovoy@nvidia.com>
 <19e73e58-c7a9-03ce-65a7-50f37d52ca15@ozlabs.ru>
 <8941cf42-0c40-776e-6c02-9227146d3d66@nvidia.com>
 <20210310130246.GW2356281@nvidia.com>
 <3b772357-7448-5fa7-9ecc-c13c08df95c3@ozlabs.ru>
 <20210310194002.GD2356281@nvidia.com>
 <7f0310db-a8e3-4045-c83a-11111767a22f@ozlabs.ru>
 <20210311013443.GH2356281@nvidia.com>
 <d862adf9-6fe7-a99e-6c14-8413aae70cd4@ozlabs.ru>
 <20210311020056.GI2356281@nvidia.com>
 <73c99da0-6624-7aa2-2857-ef68092c0d07@ozlabs.ru>
From:   Max Gurtovoy <mgurtovoy@nvidia.com>
Message-ID: <d2c32828-3417-1872-6451-2450e6fa763d@nvidia.com>
Date:   Thu, 11 Mar 2021 11:44:38 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.0
MIME-Version: 1.0
In-Reply-To: <73c99da0-6624-7aa2-2857-ef68092c0d07@ozlabs.ru>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2021 09:44:47.5526
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 4b8966b9-4317-40f6-ff34-08d8e4724f14
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT063.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3470
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 3/11/2021 9:54 AM, Alexey Kardashevskiy wrote:
>
>
> On 11/03/2021 13:00, Jason Gunthorpe wrote:
>> On Thu, Mar 11, 2021 at 12:42:56PM +1100, Alexey Kardashevskiy wrote:
>>>>> btw can the id list have only vendor ids and not have device ids?
>>>>
>>>> The PCI matcher is quite flexable, see the other patch from Max for
>>>> the igd
>>>   ah cool, do this for NVIDIA GPUs then please, I just discovered 
>>> another P9
>>> system sold with NVIDIA T4s which is not in your list.
>>
>> I think it will make things easier down the road if you maintain an
>> exact list <shrug>
>
>
> Then why do not you do the exact list for Intel IGD? The commit log 
> does not explain this detail.

I expect Intel team to review this series and give a more precise list.

I did the best I could in finding a proper configuration for igd.


>
>
>>>> But best practice is to be as narrow as possible as I hope this will
>>>> eventually impact module autoloading and other details.
>>>
>>> The amount of device specific knowledge is too little to tie it up 
>>> to device
>>> ids, it is a generic PCI driver with quirks. We do not have a separate
>>> drivers for the hardware which requires quirks.
>>
>> It provides its own capability structure exposed to userspace, that is
>> absolutely not a "quirk"
>>
>>> And how do you hope this should impact autoloading?
>>
>> I would like to autoload the most specific vfio driver for the target
>> hardware.
>
>
> Is there an idea how it is going to work? For example, the Intel IGD 
> driver and vfio-pci-igd - how should the system pick one? If there is 
> no MODULE_DEVICE_TABLE in vfio-pci-xxx, is the user supposed to try 
> binding all vfio-pci-xxx drivers until some binds?

For example, in my local setup I did a POC patch that convert some 
drivers to be "manual binding only drivers".

So the IGD driver will have the priority, user will unbind the device 
from it, load igd-vfio-pci, bind the device to it, ends with probing.

For now we separated the driver core stuff until we all agree that this 
series is the right way to go + we also make sure it's backward compatible.

>
>
>> If you someday need to support new GPU HW that needs a different VFIO
>> driver then you are really stuck because things become indeterminate
>> if there are two devices claiming the ID. We don't have the concept of
>> "best match", driver core works on exact match.
>
>
>