Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
Message-ID: <48391b05-ecbb-4053-bed5-2740806ff06e@intel.com>
Date:   Wed, 22 Nov 2023 11:52:37 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d stage-1
 cache invalidation
Content-Language: en-US
To:     Baolu Lu <baolu.lu@linux.intel.com>,
        Jason Gunthorpe <jgg@nvidia.com>,
        "Tian, Kevin" <kevin.tian@intel.com>
CC:     "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <BN9PR11MB52761D7A88BB2F655355A7728CB4A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20231120230451.GD6083@nvidia.com>
 <BN9PR11MB5276EF10FCE9C96F593696E18CBBA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20231121121704.GE6083@nvidia.com>
 <f5b27724-7c86-4823-ae86-76c92a2760b4@linux.intel.com>
From:   Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <f5b27724-7c86-4823-ae86-76c92a2760b4@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bGdLaW02TGxORVZKak1TbTY3TG8xVjhDZHdLQ2p1eXpBallhMzU2VHZQT05y?=
 =?utf-8?B?STNDT2tmN0tOSTFBemttdDZxMUd2d3VUTWNMMWZtMmovMEJ0M0hRUVJaYnhS?=
 =?utf-8?B?WnR4T2pHU0QrQjR0clhDVWhKUHNBZDdqeWp2TEVDcjNBZ1dTT3lkY1hhb01G?=
 =?utf-8?B?a1cvbm01WlNIckRkRHBwc1hrUXViVUdKUnRmTkRMelFzdzNhWkViL3lUeWJa?=
 =?utf-8?B?YzYxbVZLTHdGL0p1NVFhYXhYaWVMQW1tNFUzaGdGajRYSHpSNlVEZk9kSnl2?=
 =?utf-8?B?eUZlbkpFeFlENEF0dXlreG1XQ2UvQmQrKzl2SnlydWR6dURJc2xRc1oycExw?=
 =?utf-8?B?QXRYbENKWlo1VG1QSVV1UVpvN01xUVE5WXBoM3ZlM3NCbElRR2krdytoUmtV?=
 =?utf-8?B?SGRqUVZnVWVmODhQd3k5OTliK0dIQ3RuTmZ0TnJ4WDIwTUVGZU1JbXBTWVhB?=
 =?utf-8?B?eTdYZ0tDNEFwZ0xVVHdxa1RWS3RRWVVmaitZQnk4Nlp3alk5MGVZaDV2MnhT?=
 =?utf-8?B?bStBVFlVVDJ0VDMxbHIzYi96Sk1lbnJQL3FRekZySUNiL1dVOGFVeXNlb1ZQ?=
 =?utf-8?B?aDBWYnNNYWtsZmV0bjZBcitNaHpLNGl0UHpyVkpkQ1RWencwSUdDV3g0d3Fp?=
 =?utf-8?B?TWRXTFBaOXl2bmlCTjFLOGFrbDVhOVlvREFUY214bHlyVTg4Sm00VzhEZjcw?=
 =?utf-8?B?K3NJc2R3L0R4SENWeDZDZnB4Zkx5QURTblJOdzBFUnBZSGJJUXRYR25mbDBB?=
 =?utf-8?B?VUljZ2VZK09kazVETUl2a2xnV1RkOGRBQVJ2ZEFreTFQcHlic01uSzFncXJD?=
 =?utf-8?B?Qkd0L0daWjNIbDVZWW5CL1JzQzZrU1ZlLzNNUThKTWdHRnZvaDdpNVdXREox?=
 =?utf-8?B?ZUE2bnNIQzdYOVBUU1c3NEcvaXlEeHlvS3F0VmhmamZCc3J6SHZuakR5dzFB?=
 =?utf-8?B?Vzd3MEZwejA2dmtsSnoySEt5cDhyRHZWMTVvVW9jZSthVUtqL3ZBUGZrSE94?=
 =?utf-8?B?TjVjZ3VxZDArV3REeUhrdXJLWGFZTVBXMGRzSjFpenhpL2tIR0VpdjdsU1pn?=
 =?utf-8?B?dmI4MDY5eGZIWHhmSmtTc2FTMW9oeXRqWVF3eThNbjJDWXY4T0NRdi8xU2xT?=
 =?utf-8?B?Z1J6Nkl3WE5HMk0wM0laOTYvMmdnZ1E3Um44aGVVcXRNUFM3ZkdRdzZ0NjBy?=
 =?utf-8?B?Z2R0TkZZanE4TEVvVUY5SlNIL0FEeFJ0elZROHVWcnppUnBURHR0Skd2RUZo?=
 =?utf-8?B?TkhZZkpMU0lmelNQSlAyRzJLK2lvT3NYR3ZZSysxUVJuOUE5cmgrdU14Qklw?=
 =?utf-8?B?cE9xLzlhNlVXaFQ1UWpaS04yTDI0MzhKRVhucU1GTDdwY0FoWE9zMGVOT3RN?=
 =?utf-8?B?UjNSOHFtbUVrZTk0WFhlWThDSWkyTUd0MnlvemIrblFtbzl5Q3J6czVnYU5j?=
 =?utf-8?B?L1dqVWFuZkdCM2c1Zjg3MXVDQ2k2ZDJ4Yk5FRW44MXBBYmNJbS9OTU40elA1?=
 =?utf-8?B?TWNZVnd5SU5KYjVCdGk2NnF3cjlvbzBXUWJ6M1c4clJjNDRrditnMW1YNlhU?=
 =?utf-8?B?SkxEcldFWmxQN3M0aW0rV0M5akpNNGgrN2tyZFhmMVR2VWlJVjdyNE5OTkZp?=
 =?utf-8?B?bHhJK2tlM25vc3VuT09IaXFaUzF3QjZwc2NnaTB2aHBORkxZTitrYW9JMk1X?=
 =?utf-8?B?M1RnSjJIdXd5a3pIQXF4Sjh2Vkk4ZmJOSG1scG9CNWtsUVFWajVKL2JTN0FW?=
 =?utf-8?B?anNIeVM5eWR6UC8xWWRGYmNKaFpTeUdxRk5zd0kySEZMdDRKaDNic1didU5T?=
 =?utf-8?B?WUNDMm5lUER4cW5nYi83bldwVmFNVXlrZHRJV3pwWmR2MEtIMFhybnd2ZERQ?=
 =?utf-8?B?ZThYUHBRTW1YS1lkT2NES25CdEJnTlY4cXlyL2owYXlqcDUwOUVIb1ZWNkZp?=
 =?utf-8?B?enBkWFFYT0RUd0duazJlVXlFOUt4Q3FyYVgwVEJaQ3B3MHRMZVJrNXR0OGJ0?=
 =?utf-8?B?TjFrSjlPc2x2Um9ydFdobkdkeGljdk5veWFmQnVOOEFnR0R4YVVTZ2NLV2k4?=
 =?utf-8?B?QnlsTXd1VWpPV0lxMnN1ZXFPVGNBaUhkNGZQdStKZmVmejkyd3lnK3RudS9H?=
 =?utf-8?Q?NosOhgIqyy11H8dTE8JrHWdPB?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a847ad89-e270-4c4f-77bc-08dbeb0e1eff
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 03:50:09.3178
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: w67BP3S9tLotKXCLJW9BweLmXJOeI01pYc+E/gKxNOLUpO1p71wGsasGtxSRHLikFW9mQGgRDskSTiivgc4Mww==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7361
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 21 Nov 2023 19:50:30 -0800 (PST)

On 2023/11/22 10:32, Baolu Lu wrote:
> On 11/21/23 8:17 PM, Jason Gunthorpe wrote:
>> On Tue, Nov 21, 2023 at 02:54:15AM +0000, Tian, Kevin wrote:
>>>> From: Jason Gunthorpe <jgg@nvidia.com>
>>>> Sent: Tuesday, November 21, 2023 7:05 AM
>>>>
>>>> On Mon, Nov 20, 2023 at 08:26:31AM +0000, Tian, Kevin wrote:
>>>>>> From: Liu, Yi L <yi.l.liu@intel.com>
>>>>>> Sent: Friday, November 17, 2023 9:18 PM
>>>>>>
>>>>>> This adds the data structure for flushing iotlb for the nested domain
>>>>>> allocated with IOMMU_HWPT_DATA_VTD_S1 type.
>>>>>>
>>>>>> This only supports invalidating IOTLB, but no for device-TLB as 
>>>>>> device-TLB
>>>>>> invalidation will be covered automatically in the IOTLB invalidation 
>>>>>> if the
>>>>>> underlying IOMMU driver has enabled ATS for the affected device.
>>>>>
>>>>> "no for device-TLB" is misleading. Here just say that cache invalidation
>>>>> request applies to both IOTLB and device TLB (if ATS is enabled ...)
>>>>
>>>> I think we should forward the ATS invalidation from the guest too?
>>>> That is what ARM and AMD will have to do, can we keep them all
>>>> consistent?
>>>>
>>>> I understand Intel keeps track of enough stuff to know what the RIDs
>>>> are, but is it necessary to make it different?
>>>
>>> probably ask the other way. Now intel-iommu driver always flushes
>>> iotlb and device tlb together then is it necessary to separate them
>>> in uAPI for no good (except doubled syscalls)? :)
>>
>> I wish I knew more about Intel CC design to be able to answer that :|
>>
>> Doesn't the VM issue the ATC flush command regardless? How does it
>> know it has a working ATC but does not need to flush it?
>>
> 
> The Intel VT-d spec doesn't require the driver to flush iotlb and device
> tlb together.

Spec has below description. Although it does not say iotlb and device tlb
should be flushed together, but there is indeed requirement that both 
should be flushed when a page is unmapped.

Chapter 6.5.2.5:
"Since translation requests-without-PASID from a device may be serviced by 
hardware from the
IOTLB, software must always request IOTLB invalidation (iotlb_inv_dsc) 
before requesting
corresponding Device-TLB (dev_tlb_inv_dsc) invalidation."

> Therefore, the current approach of relying on caching mode
> to determine whether device TLB invalidation is necessary appears to be
> a performance optimization rather than an architectural requirement.
> 
> The vIOMMU driver assumes that it is running within a VM guest when
> caching mode is enabled. This assumption leads to an omission of device
> TLB invalidation, relying on the hypervisor to perform a combined flush
> of the IOLB and device TLB.

yes, this is what the current intel iommu driver does. However, whether
rely on caching mode or not is orthogonal with whether we need to uapis
here. I think guest iommu driver could submit both iotlb and device tlb
invalidation request. But Qemu could select if it needs to forward the
device tlb invalidation request to kernel if kernel iommu driver has
already covered the device tlb invalidation when get the request to
invalidate iotlb.

> While this optimization aims to reduce VMEXIT overhead, it introduces
> potential issues:
> 
> - When a Linux guest running on a hypervisor other than KVM/QEMU, the
>    assumption of combined IOLB and device TLB flushing by the hypervisor
>    may be incorrect, potentially leading to missed device TLB
>    invalidation.

Hmmm, this appears to be an intel iommu driver bug, it should submit both
iotlb invalidation and device tlb invalidation requests. But as above, I
think this is orthogonal here. KVM/QEMU could define its own uapi based on
the implementation to gain the best suit.

> 
> - The caching mode doesn't apply to first-stage translation. Therefore,
>    if the driver uses first-stage translation and still relies on caching
>    mode to determine device TLB invalidation, the optimization fails.

yes, caching mode does no apply to first-stage translation table. But in
nested translation, guest does not need to notify hypervisor when there is
page unmapped. is it? So whether caching mode applies to first-stage
translation table does not matter. TBH. I didn't see the problem due to
this reason. But I agree that linux guest intel iommu driver needs to
submit both iotlb and device tlb invalidation request to guarantee it can
work on other hypervisors. And there should be other way to do the
performance optimization.

> 
> A more reasonable optimization would be to allocate a bit in the iommu
> capability registers. The vIOMMU driver could then leverage this bit to
> determine whether it could eliminate a device invalidation request.

this may be something spec can be enhanced. But again it is just to make
guest intel iommu driver to gain performance optimization and also can
work on other hypervisors. As of this uapi design, considering it within
the linux ecosystem is enough.

-- 
Regards,
Yi Liu