Received-SPF: pass (google.com: domain of linux-kernel+bounces-4955-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <c6d88551-c480-4a89-ad2b-b873951fb181@intel.com>
Date: Tue, 19 Dec 2023 17:26:21 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 0/6] iommufd: Add nesting infrastructure (part 2/2)
Content-Language: en-US
To: Joel Granados <j.granados@samsung.com>
CC: <joro@8bytes.org>, <alex.williamson@redhat.com>, <jgg@nvidia.com>,
	<kevin.tian@intel.com>, <robin.murphy@arm.com>, <baolu.lu@linux.intel.com>,
	<cohuck@redhat.com>, <eric.auger@redhat.com>, <nicolinc@nvidia.com>,
	<kvm@vger.kernel.org>, <mjrosato@linux.ibm.com>,
	<chao.p.peng@linux.intel.com>, <yi.y.sun@linux.intel.com>,
	<peterx@redhat.com>, <jasowang@redhat.com>,
	<shameerali.kolothum.thodi@huawei.com>, <lulu@redhat.com>,
	<suravee.suthikulpanit@amd.com>, <iommu@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>, <linux-kselftest@vger.kernel.org>,
	<zhenzhong.duan@intel.com>, <joao.m.martins@oracle.com>,
	<xin.zeng@intel.com>, <yan.y.zhao@intel.com>
References: <20231117130717.19875-1-yi.l.liu@intel.com>
 <CGME20231217215720eucas1p2a590aca62ce8eb5ba81df6bc8b1a785d@eucas1p2.samsung.com>
 <20231217112101.6mxn42dw62tbj6uw@localhost>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <20231217112101.6mxn42dw62tbj6uw@localhost>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZWZNR3oyQm1mV0t1aFh2ZmMwV2w4MSsyN1phVGZKVXpVa3FqWXNjWktybVND?=
 =?utf-8?B?N05yQ21oYUM4dVlpRmJmK2hQVUN5eXl0TDVOSHNyb1U4TEpVMVpJSXRGYmlv?=
 =?utf-8?B?UEU2TURhMERpWnlXQmZDVjRGTVhYbDVUS3RybEVldXBMQ3BrM2JkdkZqR0R5?=
 =?utf-8?B?Ui9lWW1hZDhGZFV3WWU5TXRxME04RGVjS2cycnowYWllMEVEVFpjOVAzb0VH?=
 =?utf-8?B?YmpsYWU3THhzc21KSzV0WU43Tkt1bTZEa1FlK09UTzVUM1dzNytGNExBUytI?=
 =?utf-8?B?UzJEL1N3S1AwOFh3clVVdmlFWkNocFZJK2NqQ1A5TktwNXg2NTFzYjhnZU5v?=
 =?utf-8?B?bE5HVlhBSlpleTVucTJtRG0yK0szZnNtYTdkdUo0ZFovMWRPaEJKdHJVUXRE?=
 =?utf-8?B?bSsrZVQwMFhqdUE0cHV0MVVxM1o5cEtLNnBodEdxQzloME5QWTRNeitrMUNG?=
 =?utf-8?B?Sm9aWmVMN1FaaXRnRUx4L05QWjA3UUowNzZxRkc4aXVmMU5rT0NIYXZhQmJU?=
 =?utf-8?B?RDZ2WW11Y0RlUm9hN0FaTXBmUmhGYUJVQXJ6SGVNaVpBSnFlQnQxd0tuVFJO?=
 =?utf-8?B?VDE1YVBrQnlKM0hIQkNCZ0hJY3dYUHZhc1NiUzl0WDBDcnpwdEhDQTRGWE03?=
 =?utf-8?B?SDVZbWIrVm4zRXJuQkRQb0w0dmdYNFByZUlaNGF5SXhaMEZRL1h4Wis0ZHpt?=
 =?utf-8?B?U0d0ODdyYVVOWVlwSHJjNzJlbmxySjZnUzBUYXNJN0ZGMUdxK243VHNzYU83?=
 =?utf-8?B?YndtQVpFRnhJRWw5Wk9SMjdQVlFGdzFDQVlUazllVUlRamNIaThCNGlZcXF4?=
 =?utf-8?B?YjN4UDJqTVJ5OGtkdzB6ZWdpUk1QWG1XYmVSeTVVSTNRWVRCNVRpa1A3bkJl?=
 =?utf-8?B?ZVZaR3VOblU2VjMzVWRvWUplNWU2SnVHSlBnVGxSSEI0SjI4cVBNME9pQ2kr?=
 =?utf-8?B?eHdQN1hPSkRuSnFSWllSV01qUVdsbjI1N1VXQWx2QVRsUjZVUVArN3lZdGta?=
 =?utf-8?B?RVMyazZVVWpkdjVFdFRIWWFVWk1obWVFd0NyOTdnVWdXdVBFa2dIT0Y0UVFu?=
 =?utf-8?B?Zm9hdkV0eW43TXY2QTdMZWZEWVJ3ZzJjQTBUeWptT0ZtOFRRMWRsc1d4R3pC?=
 =?utf-8?B?dkZ6Z0cwVFlxYzFYYyt6eFREVjhkTGRXL3YvaE9TYktwNndqZ1daR2xoV3Ex?=
 =?utf-8?B?YXFuNEFtNjRoWC9leDUwN3FKU2JVMU01NTRRaGhPYzlrbmRDTm5UbXArRlE4?=
 =?utf-8?B?RW5hckJoM29wbVAwcFpyT09DZmZJb0YxK05oS3YydUVKUk9qMTNNemhSUUVo?=
 =?utf-8?B?TFpSL0x6Szk5aDRnd0VmU1NYbGdRMVdKL2FXN3F5RkRqQnRTM3FYdFU5RlM0?=
 =?utf-8?B?eDhFQVlQMmluYXcvYjU3aDZtQVl6SmhpNXEwb3dBOTcrY3RTMmZodmtSamhW?=
 =?utf-8?B?VExEbDhzUUgwem1OcnVRWHB2ZVRVTG1lS0VkM0k4OWxzZnBqTk5HK1lkYmNh?=
 =?utf-8?B?d0RVUGhnbWJqekxLVFlMUXFvTmNZZE9oVFYyNXYvR1RGS09PczkrSXQyNnkr?=
 =?utf-8?B?bGYyaTNtakUzSTNkZTJyOW9RMzVFZGF4UzZZYlVxaXlIa21JYUhrNXZqczlz?=
 =?utf-8?B?MklvNWFISFNOQ21iUmREYStiSmh3aVplVHVNeStpazdob3JTU2RYT0U0WWs4?=
 =?utf-8?B?bWpTU09wNzN2Q3lwL3dHSnEwRnB6Qml1L0FGUTFScURsVmViRUxDYTRvS3k4?=
 =?utf-8?B?eTVGM1hjMjhGeksxZjVxd3dPOWxUUGVzUTQydWJORTc0U2d6NXBxbmQzcWNj?=
 =?utf-8?B?emgvL2Fydk9kdWxZZWRJSDBCZnVBMjFkeFFQaW40UXFoQkkvOTZoTUFDVWFO?=
 =?utf-8?B?TkpuNERFdmEzckdudERCSWwwMWpHK29HeTRWa0IrOWhPUDhVdE9xZms3TzZG?=
 =?utf-8?B?RkNtb2lVMFBVSFpJMmRPc0QyaHpiVEU2aU1kemxJejJ2MUVSSnJxWG51OUU4?=
 =?utf-8?B?REpxZXl5THFPcmxQUjlGVG4wcUZnZkMvbGtKRjNXSmVJemhKZzJBRURTSkNl?=
 =?utf-8?B?Wkx2MXhGT0k0emFrSlBpaXdUR0tFTlJPdTNsbyt1LzF2WHhhcWc0cWZmeW4w?=
 =?utf-8?Q?HPBOn5uLNuxhggaot9K8wFF+d?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 3bab56e2-fc1b-4cd1-b68d-08dc00742cbc
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2023 09:23:34.9959
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Zye1skI1ElDZiQFak1JNhKecR1IVPHKIXn0QpIt4gFbmjtaE7g/aLMk/vrnCeY5jHJ6QqQk6mu1TlKs8/d8klQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB6753
X-OriginatorOrg: intel.com

On 2023/12/17 19:21, Joel Granados wrote:
> Hey Yi
> 
> I have been working with https://github.com/yiliu1765/qemu/tree/zhenzhong/wip/iommufd_nesting_rfcv1

good to know about it.

> and have some questions regarding one of the commits in that series.
> I however cannot find it in lore.kernel.org. Can you please direct me to
> where the rfc was posted? If it has not been posted yet, do you have an
> alternate place for discussion?

the qemu series has not been posted yet as kernel side is still changing.
It still needs some time to be ready for public review. Zhenzhong Duan
is going to post it when it's ready. If you have questions to discuss,
you can post your questions to Zhenzhong and me first. I guess it may be
fine to cc Alex Williamson, Eric Auger, Nicolin Chen, Cédric Le Goater,
Kevin Tian, Jason Gunthorpe and qemu mail list as this is discussion 
something that is going to be posted in public.

> 
> Best
> 
> On Fri, Nov 17, 2023 at 05:07:11AM -0800, Yi Liu wrote:
>> Nested translation is a hardware feature that is supported by many modern
>> IOMMU hardwares. It has two stages (stage-1, stage-2) address translation
>> to get access to the physical address. stage-1 translation table is owned
>> by userspace (e.g. by a guest OS), while stage-2 is owned by kernel. Changes
>> to stage-1 translation table should be followed by an IOTLB invalidation.
>>
>> Take Intel VT-d as an example, the stage-1 translation table is I/O page
>> table. As the below diagram shows, guest I/O page table pointer in GPA
>> (guest physical address) is passed to host and be used to perform the stage-1
>> address translation. Along with it, modifications to present mappings in the
>> guest I/O page table should be followed with an IOTLB invalidation.
>>
>>      .-------------.  .---------------------------.
>>      |   vIOMMU    |  | Guest I/O page table      |
>>      |             |  '---------------------------'
>>      .----------------/
>>      | PASID Entry |--- PASID cache flush --+
>>      '-------------'                        |
>>      |             |                        V
>>      |             |           I/O page table pointer in GPA
>>      '-------------'
>> Guest
>> ------| Shadow |---------------------------|--------
>>        v        v                           v
>> Host
>>      .-------------.  .------------------------.
>>      |   pIOMMU    |  |  FS for GIOVA->GPA     |
>>      |             |  '------------------------'
>>      .----------------/  |
>>      | PASID Entry |     V (Nested xlate)
>>      '----------------\.----------------------------------.
>>      |             |   | SS for GPA->HPA, unmanaged domain|
>>      |             |   '----------------------------------'
>>      '-------------'
>> Where:
>>   - FS = First stage page tables
>>   - SS = Second stage page tables
>> <Intel VT-d Nested translation>
>>
>> This series adds the cache invalidation path for the userspace to invalidate
>> cache after modifying the stage-1 page table. This is based on the first part
>> of nesting [1]
>>
>> Complete code can be found in [2], QEMU could can be found in [3].
>>
>> At last, this is a team work together with Nicolin Chen, Lu Baolu. Thanks
>> them for the help. ^_^. Look forward to your feedbacks.
>>
>> [1] https://lore.kernel.org/linux-iommu/20231026044216.64964-1-yi.l.liu@intel.com/ - merged
>> [2] https://github.com/yiliu1765/iommufd/tree/iommufd_nesting
>> [3] https://github.com/yiliu1765/qemu/tree/zhenzhong/wip/iommufd_nesting_rfcv1
>>
>> Change log:
>>
>> v6:
>>   - No much change, just rebase on top of 6.7-rc1 as part 1/2 is merged
>>
>> v5: https://lore.kernel.org/linux-iommu/20231020092426.13907-1-yi.l.liu@intel.com/#t
>>   - Split the iommufd nesting series into two parts of alloc_user and
>>     invalidation (Jason)
>>   - Split IOMMUFD_OBJ_HW_PAGETABLE to IOMMUFD_OBJ_HWPT_PAGING/_NESTED, and
>>     do the same with the structures/alloc()/abort()/destroy(). Reworked the
>>     selftest accordingly too. (Jason)
>>   - Move hwpt/data_type into struct iommu_user_data from standalone op
>>     arguments. (Jason)
>>   - Rename hwpt_type to be data_type, the HWPT_TYPE to be HWPT_ALLOC_DATA,
>>     _TYPE_DEFAULT to be _ALLOC_DATA_NONE (Jason, Kevin)
>>   - Rename iommu_copy_user_data() to iommu_copy_struct_from_user() (Kevin)
>>   - Add macro to the iommu_copy_struct_from_user() to calculate min_size
>>     (Jason)
>>   - Fix two bugs spotted by ZhaoYan
>>
>> v4: https://lore.kernel.org/linux-iommu/20230921075138.124099-1-yi.l.liu@intel.com/
>>   - Separate HWPT alloc/destroy/abort functions between user-managed HWPTs
>>     and kernel-managed HWPTs
>>   - Rework invalidate uAPI to be a multi-request array-based design
>>   - Add a struct iommu_user_data_array and a helper for driver to sanitize
>>     and copy the entry data from user space invalidation array
>>   - Add a patch fixing TEST_LENGTH() in selftest program
>>   - Drop IOMMU_RESV_IOVA_RANGES patches
>>   - Update kdoc and inline comments
>>   - Drop the code to add IOMMU_RESV_SW_MSI to kernel-managed HWPT in nested translation,
>>     this does not change the rule that resv regions should only be added to the
>>     kernel-managed HWPT. The IOMMU_RESV_SW_MSI stuff will be added in later series
>>     as it is needed only by SMMU so far.
>>
>> v3: https://lore.kernel.org/linux-iommu/20230724110406.107212-1-yi.l.liu@intel.com/
>>   - Add new uAPI things in alphabetical order
>>   - Pass in "enum iommu_hwpt_type hwpt_type" to op->domain_alloc_user for
>>     sanity, replacing the previous op->domain_alloc_user_data_len solution
>>   - Return ERR_PTR from domain_alloc_user instead of NULL
>>   - Only add IOMMU_RESV_SW_MSI to kernel-managed HWPT in nested translation (Kevin)
>>   - Add IOMMU_RESV_IOVA_RANGES to report resv iova ranges to userspace hence
>>     userspace is able to exclude the ranges in the stage-1 HWPT (e.g. guest I/O
>>     page table). (Kevin)
>>   - Add selftest coverage for the new IOMMU_RESV_IOVA_RANGES ioctl
>>   - Minor changes per Kevin's inputs
>>
>> v2: https://lore.kernel.org/linux-iommu/20230511143844.22693-1-yi.l.liu@intel.com/
>>   - Add union iommu_domain_user_data to include all user data structures to avoid
>>     passing void * in kernel APIs.
>>   - Add iommu op to return user data length for user domain allocation
>>   - Rename struct iommu_hwpt_alloc::data_type to be hwpt_type
>>   - Store the invalidation data length in iommu_domain_ops::cache_invalidate_user_data_len
>>   - Convert cache_invalidate_user op to be int instead of void
>>   - Remove @data_type in struct iommu_hwpt_invalidate
>>   - Remove out_hwpt_type_bitmap in struct iommu_hw_info hence drop patch 08 of v1
>>
>> v1: https://lore.kernel.org/linux-iommu/20230309080910.607396-1-yi.l.liu@intel.com/
>>
>> Thanks,
>> 	Yi Liu
>>
>> Lu Baolu (1):
>>    iommu: Add cache_invalidate_user op
>>
>> Nicolin Chen (4):
>>    iommu: Add iommu_copy_struct_from_user_array helper
>>    iommufd/selftest: Add mock_domain_cache_invalidate_user support
>>    iommufd/selftest: Add IOMMU_TEST_OP_MD_CHECK_IOTLB test op
>>    iommufd/selftest: Add coverage for IOMMU_HWPT_INVALIDATE ioctl
>>
>> Yi Liu (1):
>>    iommufd: Add IOMMU_HWPT_INVALIDATE
>>
>>   drivers/iommu/iommufd/hw_pagetable.c          | 35 ++++++++
>>   drivers/iommu/iommufd/iommufd_private.h       |  9 ++
>>   drivers/iommu/iommufd/iommufd_test.h          | 22 +++++
>>   drivers/iommu/iommufd/main.c                  |  3 +
>>   drivers/iommu/iommufd/selftest.c              | 69 +++++++++++++++
>>   include/linux/iommu.h                         | 84 +++++++++++++++++++
>>   include/uapi/linux/iommufd.h                  | 35 ++++++++
>>   tools/testing/selftests/iommu/iommufd.c       | 75 +++++++++++++++++
>>   tools/testing/selftests/iommu/iommufd_utils.h | 63 ++++++++++++++
>>   9 files changed, 395 insertions(+)
>>
>> -- 
>> 2.34.1
>>
> 

-- 
Regards,
Yi Liu