Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
Message-ID: <88353789-6b37-44ca-aa21-23d27fab37ee@intel.com>
Date:   Tue, 28 Nov 2023 11:11:37 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 3/3] vfio: Report PASID capability via VFIO_DEVICE_FEATURE
 ioctl
Content-Language: en-US
To:     "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "jgg@nvidia.com" <jgg@nvidia.com>,
        "Tian, Kevin" <kevin.tian@intel.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
CC:     "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
References: <20231127063909.129153-1-yi.l.liu@intel.com>
 <20231127063909.129153-4-yi.l.liu@intel.com>
 <SJ0PR11MB6744DC9B7C7D0E4122F224FD92BDA@SJ0PR11MB6744.namprd11.prod.outlook.com>
From:   Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <SJ0PR11MB6744DC9B7C7D0E4122F224FD92BDA@SJ0PR11MB6744.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eCs2WUpocC9tS0ZrckxkL29RSlVmZ09YSlZWcmZDWjQ0TFBxOWFTT2lRYWpu?=
 =?utf-8?B?RVN4STg1SWthT3VCVmlzVG1ra1FaWkt5bkg5cUVXU3FPTGFGOFpYNWxUZkxF?=
 =?utf-8?B?MGE4NHBJT0tkMk1zZUVraWhTdTQxb2tqNE9tb0lkRU5sRUdqTUFjbEFQYWZQ?=
 =?utf-8?B?bHB5QjdxczUxWUl2cFErUlVmUU9nazZXU1dEYjJGeFdXY2RtcGVpamFZL2tp?=
 =?utf-8?B?bnQvL2FIbU53NDM0VEY1eHR5cjdpWWpUdXJBQjh5MHhxcnNJVXRkNlM1engx?=
 =?utf-8?B?bGNBM1o3ZjVKaVM1YThaS0hMOWJmYUhPZUZUNEdZTWdVYkhJcVFWalpiQTU2?=
 =?utf-8?B?RGZRRmJUa3IvNkhxK1oyaWh2Nk56RVpScFR1Z05xNHR4QTRrUDRGdnNBSzMv?=
 =?utf-8?B?QlZFdnBOVXYxMGxuaHhpTjIyd0RtVlR0NHd6UXRTQW43a1Z6OWdBY2R2VGJo?=
 =?utf-8?B?bWVxd3Y4cVIzdS9qbnJiekFZNGwwRGsweXZpaGNEN01zNWJzVGhIODNNTU9D?=
 =?utf-8?B?eGVxY2RZSjllY0U5QVhpRTRORm5xTW5pbTVwNHkvc1ZNVGNyVGM2QzRUb2lz?=
 =?utf-8?B?a29wOXI4OHVjS0pHcUhVMmRCaDdHZGh4S0t6M1FodlpnYjBCQ21DMnFsQlNo?=
 =?utf-8?B?bnFBaU0vSlFkSm5lbHhuQ2pGeXZWVFJxZmliSitLUis2aWdwa3dBc0cyNzl2?=
 =?utf-8?B?bytXMzNvTXpQWm0yaitka0FoTkdLREhsYWhVNzE3OXRjOTZjczR3SjF0SC8r?=
 =?utf-8?B?eTg3SlhpWWhNS2NCaFR1WGFFSzQ5WVVCcXpSTWNJK1lxSTZPV2pCekwrbkpv?=
 =?utf-8?B?OGJheldOcWtmcy9wU2lFSmtIZlVWUFN1aGMwZUNkUzFxUzdLUFNtd243NlNG?=
 =?utf-8?B?czlQQjlqZDI4bjRLeTBYZkpjVkhueGd3MWUvUHRUTm9NRWJDZzlxMTBZQVNx?=
 =?utf-8?B?dlNtQ1hHcDZqcmZVMDNLcE1uYkg4UDZ2WnhReHdRb1FHUHlxQzhheTRENVhy?=
 =?utf-8?B?YnlHdUU3dys1SkVUSDBBeVFDV2NNaENEcTNyQVNnVERaMUNQenRWVHdvcTZZ?=
 =?utf-8?B?MVhCNE5zOGZkZGttTnN0YkdVa2dUNmFoMzBDWEVQRXZKUUJOWDgwSzMwQXc2?=
 =?utf-8?B?ZVZKajIvYWtYU3R4em9HZU5WcUdvNWZ6SjFtUU42YUlXTDZMSWxzS2xMa0lz?=
 =?utf-8?B?ZlU0YWdiMDI1alk5M0xWVXNnbUw3cTd4MFBmS2FVcWhOenFYV3I5QnNjQ1Ey?=
 =?utf-8?B?d1dCZ1pSdlFMSU5yZ1o2L2ZvZmUyVGJOQW4zclRLLzZnbExkNXJyRUcvT1k0?=
 =?utf-8?B?UmM4cWVnVGUrMnJvc3RpbU9zWTVjbVl6VlVmRjJrOFJUWWFxWGNBaFdTdE1x?=
 =?utf-8?B?a24zMHFoa281MVVwM1B2VkV3MklZYy9iWWxTc09iRnQyNlJUa2N4ZTg5d1o5?=
 =?utf-8?B?UmN6SlRDeEM2ZGJLLy9aTlVxM2FteDY2YjFwS1Mwd28rdUVuTWRPa25KUEZa?=
 =?utf-8?B?VmdsNW5DUStHYy9UQ2VYZEpSWThvdkQ1SWh1c1YzY2gwVFYvZ1FZWUNwWjJo?=
 =?utf-8?B?K2Y2YVk5ZVNsOHNkK0ROMXBPRmJNTUdiNzhwblQvcitaMWVZT0MzSS94ZFVL?=
 =?utf-8?B?RllBcVhBcDlRQ1hiNjFYVWxpalpzV3pRaUVuK1VpdnR5ZWRHWVZYQkl4Nkg4?=
 =?utf-8?B?cWIyMm1ha3pvVnFUeDB3Y1g5TFU0VHY5anhKTC9TanVJKy9SSVZyTGVSYW5z?=
 =?utf-8?B?MFdReDd3bUdYOExlN3pvNm1xRUhkSllSSGJCMmxaejZNbjhCWDkzOXJEMFdi?=
 =?utf-8?B?ZjNpRzRnaXpoc1VKUzJJUkVUbGZrc3c4YUJlUFc2RlMyVXJ2YlB5S2RGNFRW?=
 =?utf-8?B?cGs4Uk5ZYnhoS3dTK3FneTRtcHNzTFdnT3ZGSkJ6Z0padzdya0liYUhMUWxm?=
 =?utf-8?B?cWJUMzlIMnErTGNBVFBkSkcvbmtVZjlBM3VFYWFiWUs2ektjeHZieDZ5dENn?=
 =?utf-8?B?UVlxVUtQMW1reFRTZGloSkh1LzVub0pVM1BjdlFjKzc0SHNRSUIrRHFoS3BD?=
 =?utf-8?B?aHk4MkRrcC9vZW5EU0ZWWmd3cEhIL0dIVU1MOXEwYjFBcXdvT2FlUUl0ZUlh?=
 =?utf-8?Q?i5WDiE6FmncG0PMU6A/qb1Bbe?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c7047d7f-f32c-498a-1296-08dbefbf6285
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2023 03:09:07.9688
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 4pNCcYJDN/pfbmKOYtRmhWg8jiJD+Qi6Phi019fLNFic9Zcbh8ix0GMMFjuryLKxLO4RJmLTgFIbNPW5i4AqZA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB8587
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 27 Nov 2023 19:09:26 -0800 (PST)

On 2023/11/27 15:28, Duan, Zhenzhong wrote:
> 
> 
>> -----Original Message-----
>> From: Liu, Yi L <yi.l.liu@intel.com>
>> Sent: Monday, November 27, 2023 2:39 PM
>> Subject: [PATCH 3/3] vfio: Report PASID capability via VFIO_DEVICE_FEATURE
>> ioctl
>>
>> This reports the PASID capability data to userspace via VFIO_DEVICE_FEATURE,
>> hence userspace could probe PASID capability by it. This is a bit different
>> with other capabilities which are reported to userspace when the user reads
>> the device's PCI configuration space. There are two reasons for this.
>>
>> - First, Qemu by default exposes all available PCI capabilities in vfio-pci
>>    config space to the guest as read-only, so adding PASID capability in the
>>    vfio-pci config space will make it exposed to the guest automatically while
>>    an old Qemu doesn't really support it.
>>
>> - Second, PASID capability does not exit on VFs (instead shares the cap of
>>    the PF). Creating a virtual PASID capability in vfio-pci config space needs
>>    to find a hole to place it, but doing so may require device specific
>>    knowledge to avoid potential conflict with device specific registers like
>>    hiden bits in VF config space. It's simpler by moving this burden to the
>>    VMM instead of maintaining a quirk system in the kernel.
>>
>> Suggested-by: Alex Williamson <alex.williamson@redhat.com>
>> Signed-off-by: Yi Liu <yi.l.liu@intel.com>
>> ---
>> drivers/vfio/pci/vfio_pci_core.c | 47 ++++++++++++++++++++++++++++++++
>> include/uapi/linux/vfio.h        | 13 +++++++++
>> 2 files changed, 60 insertions(+)
>>
>> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
>> index 1929103ee59a..8038aa45500e 100644
>> --- a/drivers/vfio/pci/vfio_pci_core.c
>> +++ b/drivers/vfio/pci/vfio_pci_core.c
>> @@ -1495,6 +1495,51 @@ static int vfio_pci_core_feature_token(struct
>> vfio_device *device, u32 flags,
>> 	return 0;
>> }
>>
>> +static int vfio_pci_core_feature_pasid(struct vfio_device *device, u32 flags,
>> +				       struct vfio_device_feature_pasid __user
>> *arg,
>> +				       size_t argsz)
>> +{
>> +	struct vfio_pci_core_device *vdev =
>> +		container_of(device, struct vfio_pci_core_device, vdev);
>> +	struct vfio_device_feature_pasid pasid = { 0 };
>> +	struct pci_dev *pdev = vdev->pdev;
>> +	u32 capabilities = 0;
>> +	int ret;
>> +
>> +	/* We do not support SET of the PASID capability */
>> +	ret = vfio_check_feature(flags, argsz, VFIO_DEVICE_FEATURE_GET,
>> +				 sizeof(pasid));
>> +	if (ret != 1)
>> +		return ret;
>> +
>> +	/*
>> +	 * Needs go to PF if the device is VF as VF shares its PF's
>> +	 * PASID Capability.
>> +	 */
>> +	if (pdev->is_virtfn)
>> +		pdev = pci_physfn(pdev);
>> +
>> +	if (!pdev->pasid_enabled)
>> +		goto out;
> 
> Does a PF bound to VFIO have pasid enabled by default?

Today, host iommu driver (at least intel iommu driver) enables it in the
time of device probe and seems not changed afterward. So yes, VFIO should
see it if pasid is enabled.

> Isn't the guest kernel's responsibility to enable pasid cap of an assigned PF?

guest kernel should not have the capability to change host's pasid
configuration. It can only write to its own vconfig emulated by
hypervisor.

> Thanks
> Zhenzhong
> 
>> +
>> +#ifdef CONFIG_PCI_PASID
>> +	pci_read_config_dword(pdev, pdev->pasid_cap + PCI_PASID_CAP,
>> +			      &capabilities);
>> +#endif
>> +
>> +	if (capabilities & PCI_PASID_CAP_EXEC)
>> +		pasid.capabilities |= VFIO_DEVICE_PASID_CAP_EXEC;
>> +	if (capabilities & PCI_PASID_CAP_PRIV)
>> +		pasid.capabilities |= VFIO_DEVICE_PASID_CAP_PRIV;
>> +
>> +	pasid.width = (capabilities >> 8) & 0x1f;
>> +
>> +out:
>> +	if (copy_to_user(arg, &pasid, sizeof(pasid)))
>> +		return -EFAULT;
>> +	return 0;
>> +}
>> +
>> int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,
>> 				void __user *arg, size_t argsz)
>> {
>> @@ -1508,6 +1553,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device
>> *device, u32 flags,
>> 		return vfio_pci_core_pm_exit(device, flags, arg, argsz);
>> 	case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN:
>> 		return vfio_pci_core_feature_token(device, flags, arg, argsz);
>> +	case VFIO_DEVICE_FEATURE_PASID:
>> +		return vfio_pci_core_feature_pasid(device, flags, arg, argsz);
>> 	default:
>> 		return -ENOTTY;
>> 	}
>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>> index 495193629029..8326faf8622b 100644
>> --- a/include/uapi/linux/vfio.h
>> +++ b/include/uapi/linux/vfio.h
>> @@ -1512,6 +1512,19 @@ struct vfio_device_feature_bus_master {
>> };
>> #define VFIO_DEVICE_FEATURE_BUS_MASTER 10
>>
>> +/**
>> + * Upon VFIO_DEVICE_FEATURE_GET, return the PASID capability for the device.
>> + * Zero width means no support for PASID.
>> + */
>> +struct vfio_device_feature_pasid {
>> +	__u16 capabilities;
>> +#define VFIO_DEVICE_PASID_CAP_EXEC	(1 << 0)
>> +#define VFIO_DEVICE_PASID_CAP_PRIV	(1 << 1)
>> +	__u8 width;
>> +	__u8 __reserved;
>> +};
>> +#define VFIO_DEVICE_FEATURE_PASID 11
>> +
>> /* -------- API for Type1 VFIO IOMMU -------- */
>>
>> /**
>> --
>> 2.34.1
> 

-- 
Regards,
Yi Liu