Received-SPF: pass (google.com: domain of linux-kernel+bounces-25818-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <b3e07591-8ebc-4924-85fe-29a46fc73d78@intel.com>
Date: Mon, 15 Jan 2024 17:49:47 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 3/3] vfio: Report PASID capability via VFIO_DEVICE_FEATURE
 ioctl
Content-Language: en-US
To: Jason Gunthorpe <jgg@nvidia.com>, Alex Williamson
	<alex.williamson@redhat.com>
CC: <joro@8bytes.org>, <kevin.tian@intel.com>, <robin.murphy@arm.com>,
	<baolu.lu@linux.intel.com>, <cohuck@redhat.com>, <eric.auger@redhat.com>,
	<nicolinc@nvidia.com>, <kvm@vger.kernel.org>, <mjrosato@linux.ibm.com>,
	<chao.p.peng@linux.intel.com>, <yi.y.sun@linux.intel.com>,
	<peterx@redhat.com>, <jasowang@redhat.com>,
	<shameerali.kolothum.thodi@huawei.com>, <lulu@redhat.com>,
	<suravee.suthikulpanit@amd.com>, <iommu@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>, <linux-kselftest@vger.kernel.org>,
	<zhenzhong.duan@intel.com>, <joao.m.martins@oracle.com>,
	<xin.zeng@intel.com>, <yan.y.zhao@intel.com>
References: <20231127063909.129153-1-yi.l.liu@intel.com>
 <20231127063909.129153-4-yi.l.liu@intel.com>
 <20231211110345.1b4526c6.alex.williamson@redhat.com>
 <20231211181028.GL2944114@nvidia.com>
 <20231211114949.273b21c0.alex.williamson@redhat.com>
 <20231212153504.GL3014157@nvidia.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <20231212153504.GL3014157@nvidia.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QmkrZSs2aHJuTTBiSzFXUXlJUUc3eERBaU1FRTg4NUE5a2hiZVdhbUdnV05Q?=
 =?utf-8?B?anp2QkVjTWZDcmpJL09iN1lpd242RUxtck82OHRSR2JJNFpnUi9VRFpxaHgw?=
 =?utf-8?B?Vkc5MDNHNWhmVGJhbXAyeEh5bHpJQVhVYVRDSUxaTmdYRnVIRWZ6QXU5bEov?=
 =?utf-8?B?WEcrUm90QjFYTnRZZFhvdngxZDByZWFGRzlUaFNPOW0wSFFPc3NtWGwvVnRS?=
 =?utf-8?B?N05EdHNWTW1oM1pCMWVlU2RCYW4wYzYycHhKekU0ZzdoN29QOUExMjAvUms0?=
 =?utf-8?B?bXZydThsaW5qM3F4azlNRm5IK1pLb0Q2bjN3SGNTRjVZMDZQYnJPaVdIUnZs?=
 =?utf-8?B?Qnd6R0xCTzJONDVoRzRQcDVqVE4zN3BYSFo1SjNwdXpxVU15eGVYeEF5R091?=
 =?utf-8?B?c1NvSnNtRGFtamk1anlHelYyWVdjTGtNeVhRK21tUitabUpCVk1BVXkxSHdo?=
 =?utf-8?B?QUNEaVFJUm9vZDU3cHRjbVI3SXNDL1VQdlY3OHFxYVBnYXgzT1NIWDc4QXo4?=
 =?utf-8?B?WmJ5RzI2N0dHcG5qNjRZaHRsMjNiZTFmckpxUXRGeW9yZG5qcGZIQmpYS3FC?=
 =?utf-8?B?aDBpbm1UdU00M2lXUHA1Y1hyMjZEemVoNjdxdm9XK3RpQWhVb3c0cHpVZmxa?=
 =?utf-8?B?VkpRY1JLOGFWODVGUkVjV003TzJFUnMxOVIrOFloTEtjcXFJaDkxZGhCMHZN?=
 =?utf-8?B?bGh0NXlGUUtqU0ZlTlRSSG5Qd2lvQkZRWnpFcnAvbDFLbzZXVnZtdHh1ajM3?=
 =?utf-8?B?cDVyeWRjSVBXRnVOWHMxMTF0UEhOZW44ekNpUURVam5haXZWd2c1MnFuR1hw?=
 =?utf-8?B?YW05Wmd4RTQySE5hdkptcXlhSnd5YWR0OU1qVEdwVFlEQ3dsVllPd0lwZWdH?=
 =?utf-8?B?dlBtVkdkeEdUVWgvR3lSUStUMkNQU2w0OXJxdlBCZk4zTmEwM0FBeFdRdEtR?=
 =?utf-8?B?b05uVElid3hQR2tRL0R3cmtsSkVOczliMXZGMU1HbHZRRjI0c0c0RVVxMUl1?=
 =?utf-8?B?RHE4eCtSV0dHZE4yOGpSdWZ0Q01pVGpYa0VYMHVrMm8xc1owQXpnc05DNmdE?=
 =?utf-8?B?MEQvNEl6T04rcDgyQUZlYUNnYTZjelgyLzBTOTAyUWRjT1QwM2hIbnM5eUJJ?=
 =?utf-8?B?NnNCNVpoOUJQaURIYko4L1JOYVdHMC82NXlWd005TlNhUzZNQ1ZOU1NMUFUw?=
 =?utf-8?B?WlZhbWpkYitQS2tCQm4xQXZxS2xidjkxVitJb1Vva2lFTU14cEJ1Wk1kWUF1?=
 =?utf-8?B?RU85bFp3SFpuM1VmVGpLYlNpeTlpS2xzaENYRmd3MThhTU5LNUw5Q0RSZWxE?=
 =?utf-8?B?bGJRYVczcjdJbDNCTmtrNGxOOG9hT21qUnZXdVpoVnNhN0FDZDU0TWhsQ2Z2?=
 =?utf-8?B?bmVsL093ZU5xY01ocE1ZM2s2VlM2WkwyTW5pdFhpNTE4OXR1cjhtMW1wWWtV?=
 =?utf-8?B?aFZ4Y1pmVmRKdzdoV0pSc2QxWmdVVXI5WWVXSk1SUVNRZEFReUtMLzhIRlcx?=
 =?utf-8?B?WFBPWjRJdWRJZ0xDQkdpNzNmTTlVUHlRa2wvY0VtSStmbXd0UHg0aHVnTzh0?=
 =?utf-8?B?bjRLckozZG14TlpRVlRaRVhISU1mTUMzQUpVbkN1RUxWRnVHRVlvZWpjdDNs?=
 =?utf-8?B?T1dJL0grazc0Vmh1MUNXbGhXeWI4T2N0NVBXNDI5R2QxWmRuQ3RiRWNzZXlS?=
 =?utf-8?B?V1FMc3dPeHI5VUJzc0ZKTWtLeFFLMTEwNUhpbHFidE9EQVBNeUtIZWs3R0JO?=
 =?utf-8?B?Z3FUVnB0THYrYjF1YnEyUFZHRlhIT2h6TWNnNE5nRmV3QXdXQVBhVWNVdWJH?=
 =?utf-8?B?bklrT0lpRWtxUGE1MjNLZzhLVjgxMzNXTTdzUFd1bFlhUFVMWkd2L0M1aUpE?=
 =?utf-8?B?bXFqNFdIWFRocjVTb2Z0U1JzNHZUc2lHbURENytzOEYxN3dVVXpmV01OdUdF?=
 =?utf-8?B?L0htSHY1U09LT3F5eFc5dmFDajVFcXBPR3hhU0V3V3hsWmxXbnpuZWhpKzJY?=
 =?utf-8?B?eSs0Q1VpN2M0VXQ0cDl3bnRwbDJGTGVqamladzJ6RnNzVVYzck9GRWRzOG16?=
 =?utf-8?B?dXFzWnhYWk5oak1Tc2ZHZVBJTTBGOUExL29iUUFKQSt0NXMyNExwQ2FpY2tZ?=
 =?utf-8?Q?Fzr0D0dQXdr4ds0OtNiKNqTH5?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 223f64db-2721-49f5-44a6-08dc15aee43c
X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7540.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jan 2024 09:46:48.0506
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0dI8/S0qTus1oFR6i+dsH8c0+0vKaHPvO6usuz2Q8YluXFOh3LgcYUrtvAWGM7TwgDDEuWLwLIWzyvG6Z7maYA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5957
X-OriginatorOrg: intel.com

On 2023/12/12 23:35, Jason Gunthorpe wrote:
> On Mon, Dec 11, 2023 at 11:49:49AM -0700, Alex Williamson wrote:
>> On Mon, 11 Dec 2023 14:10:28 -0400
>> Jason Gunthorpe <jgg@nvidia.com> wrote:
>>
>>> On Mon, Dec 11, 2023 at 11:03:45AM -0700, Alex Williamson wrote:
>>>> On Sun, 26 Nov 2023 22:39:09 -0800
>>>> Yi Liu <yi.l.liu@intel.com> wrote:
>>
>>>>>     the PF). Creating a virtual PASID capability in vfio-pci config space needs
>>>>>     to find a hole to place it, but doing so may require device specific
>>>>>     knowledge to avoid potential conflict with device specific registers like
>>>>>     hiden bits in VF config space. It's simpler by moving this burden to the
>>>>>     VMM instead of maintaining a quirk system in the kernel.
>>>>
>>>> This feels a bit like an incomplete solution though and we might
>>>> already posses device specific knowledge in the form of a variant
>>>> driver.  Should this feature structure include a flag + field that
>>>> could serve to generically indicate to the VMM a location for
>>>> implementing the PASID capability?  The default core implementation
>>>> might fill this only for PFs where clearly an emualted PASID capability
>>>> can overlap the physical capability.  Thanks,
>>>
>>> In many ways I would perfer to solve this for good by having a way to
>>> learn a range of available config space - I liked the suggestion to
>>> use a DVSEC to mark empty space.
>>
>> Yes, DVSEC is the most plausible option for the device itself to convey
>> unused config space, but that requires hardware adoption so presumably
>> we're going to need to fill the gaps with device specific code.  That
>> code might live in a variant driver or in the VMM.  If we have faith
>> that DVSEC is the way, it'd make sense for a variant driver to
>> implement a virtual DVSEC to work out the QEMU implementation and set a
>> precedent.
> 
> How hard do you think it would be for the kernel to synthesize the
> dvsec if the varient driver can provide a range for it?
> 
> On the other hand I'm not so keen on having variant drivers that are
> only doing this just to avoid a table in qemu :\ It seems like a
> reasonable thing to add to existing drivers, though none of them
> support PASID yet..
> 
>> I mostly just want us to recognize that this feature structure also has
>> the possibility to fill this gap and we're consciously passing it over
>> and should maybe formally propose the DVSEC solution and reference it
>> in the commit log or comments here to provide a complete picture.
> 
> You mean by passing an explicit empty range or something in a feature
> IOCTL?

Hi Alex,

Any more suggestion on this? It appears to me that you are fine with PF
to implement the virtual PASID capability in the same offset with physical
PASID capability, while other cases need a way to know where to put the
virtual PASID capability. This may be done by a DVSEC or just pass empty
ranges through the VFIO_DEVICE_FEATURE ioctl?

Regards,
Yi Liu