Received-SPF: pass (google.com: domain of linux-kernel+bounces-147898-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <5e53c574-5059-4c5a-9a0b-59298df370a5@intel.com>
Date: Wed, 17 Apr 2024 11:40:38 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 1/1] iommu/vt-d: Remove caching mode check before
 device TLB flush
Content-Language: en-US
To: Baolu Lu <baolu.lu@linux.intel.com>, "Zhang, Tina" <tina.zhang@intel.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
CC: "Tian, Kevin" <kevin.tian@intel.com>, Jacob Pan
	<jacob.jun.pan@linux.intel.com>, Joerg Roedel <joro@8bytes.org>, Will Deacon
	<will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20240415013835.9527-1-baolu.lu@linux.intel.com>
 <MW5PR11MB58818970CB8E9A37A6AF563889092@MW5PR11MB5881.namprd11.prod.outlook.com>
 <f3502244-5f37-4f44-94a5-39939ba20eec@intel.com>
 <a5c4b57b-c1f8-4f75-95e4-9a462f4673c9@linux.intel.com>
 <66c09ac9-0107-4b86-993b-70ab6907fc21@intel.com>
 <2b8a3145-1f34-48e4-9822-a95c27f60ba1@linux.intel.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <2b8a3145-1f34-48e4-9822-a95c27f60ba1@linux.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OHpYOEVPQndIRjNLUm9BU05XdS9MUXd3Slp4S2ZGNFNMVEk0Y2VXb1JkeWFn?=
 =?utf-8?B?TkNBVmg2dVpOb2JJV0EwZEt6YW95aUwvTXBRNHdHK3ZjeEpLMnZwcmRMZm12?=
 =?utf-8?B?WUUzT2t4VzRaSFR0RDdiN0VWNHJVOGtGYk1zdDRHdUdoRThZR1VLNFBVc0Y5?=
 =?utf-8?B?ZFlwQjNHSG5SbU8rQWVseFh4YlB2d1JHNFFscjdoN2ZFNjVPYjI0ZGlCdDUr?=
 =?utf-8?B?RzhncHorQkQ2d20xQm9WTy9vZTFML2RGZ2huZzZrN1VqQWI5UDdZMEZnekJ1?=
 =?utf-8?B?ZXA5WFFvZFhrK1d3UUNQTnVEUTlrcVdZaU9NTGxaQUZtMnRaUDFpZE9Ca090?=
 =?utf-8?B?RDMrWnFxNkZmN0lPSVhaOTNkMDR1R05NOWVjQ0M4dk11ME5nVXJvdkdTMUVy?=
 =?utf-8?B?U1dESTBBREJjREtSL2k2UTdwMXRoN2ZwQXhmODZ6S1lDdklESkJvdWc0d1U2?=
 =?utf-8?B?ZVBYY0o5WFpWL1pEQVh1TFNjOVJYUlV2a1pvRmpkTWFtVzAxS0NtdjRkQlZ1?=
 =?utf-8?B?a1FGWW1OeUVJczY0TzQ0OG1WdXBzSVc2RDEvM1BvRldwVzdDT21iVEI2YWtx?=
 =?utf-8?B?RGhock9MdVcvTjBJV203UU8xZDE2Zm0va1htSENMOGJ6N1grVmUwOWxwelF1?=
 =?utf-8?B?MkhRc1phZnd2SUw1K2M2eDhJK3AvSTB3NDU1cVM5MHJCUTJDZnVvZDVzczZS?=
 =?utf-8?B?WjNQSGtMSkR0dWtiREM0dDBjUVdDZlVVRVdwK2k4MjhkMUtRYXRYTGZFM3Zn?=
 =?utf-8?B?NzV6NHlDMHMzZGpvdXZDZmFId3R6TkJqa0NteTNtYnVRVEdnZjQ2M1JTby9D?=
 =?utf-8?B?THJ6YjhDSXN5aVhaSVBBR0lzQlV1d2IyUmk4c0FNSUxiaTNsMlpYRFBjR3Zi?=
 =?utf-8?B?VzlmSFE5WjVSTkVjVXA0Y1l2SXIyNkFWbkdaWUJGVVQzN2RyN05nbFhjekpI?=
 =?utf-8?B?ak1JbGErU3VPVnMxdmx1U3hvMFN0Vmc3QzBuc3o5NmllVllqem9za3lLSGdx?=
 =?utf-8?B?V3B3c2l0K3gwTDZLcndiaUpLTGRtWm8wNklocEhpV1VjV2hBd2lTZTNzOVkr?=
 =?utf-8?B?MWFjcXFjS2MrODVGTUNwQXZUb0xCYmFZR3plV0Fqb2hLMDFqRjFlY3pIb050?=
 =?utf-8?B?amJ5dU9lMjNYT2M4RDA3RCtpZHo5bDUxVUgrYTB1QlRiT0taR3NjQzEwTnlK?=
 =?utf-8?B?MytSb3QzVDQvM1FjOHFGdGMrbUVJYWJSMFF6TFJSRy9XWEZNanlwVDZuSTBm?=
 =?utf-8?B?K1haQmJPTWdVN050K3lRZVlVQWRxMWtuZlpRRytMZjdreCt4Q2p6REc0bU1l?=
 =?utf-8?B?VDJLNEEwbklVK3NaOXhCMERFUExLVWE4VXFYSkVjTXU0K3RYL3ZCR0dOWjYw?=
 =?utf-8?B?RWNaY2lMUUVpenk1SlRvdkpYU1VMSktxNGNYdHFZY0MrT05sSXdROWVxRDgv?=
 =?utf-8?B?dEsxRVEzZzZtQWxVYTdqQ3U4NmlnbVVnaFJTQUllNHV0dC9CVjBaeTBiL0JW?=
 =?utf-8?B?THZ4NEpocCtCREdMSmxFK0lGTDc0ME81VFhpblJGdHVDMzhhaW1odDlYbGlJ?=
 =?utf-8?B?QnIzUVlCL1V3emdxLzk2blpVelpIVjFlTE5tVXVtN1N2ZTVtdmxZVERFYXps?=
 =?utf-8?B?aFNhUFhIUk9PMC9oRndWRzVwR00zZDh4cUtpQ3JLdHVrUUMwUkNoa3lpaFAv?=
 =?utf-8?B?MnhZNnAwZGRhaWVsRHV5VDcrZUdteXN5a2xQc3kyKy9KbnVDWFRIbEZzZnhs?=
 =?utf-8?B?UnlqbTdDSTZkY0ZtWUMwQXQ1ZUxLYzNhc2VKRnhidGFjNXhobGltcWdjWFMy?=
 =?utf-8?B?WU5tYVo1M2ZsZHVkditHN2FOdEVMdVRaRnZ2STd4YzVXUUFtMm1iODVJend4?=
 =?utf-8?B?N1Z5OXJRUVlBQzBWQzBXUnhqSjdZcUR3b2Ezanp1NGl3eEdVSVVjUTE1bEVk?=
 =?utf-8?B?RFlNTWcwOVVNYjZJOWJRZjFkMFpoaHZzYkNTbHdIekFkMm9zSWhxYUxRbEYr?=
 =?utf-8?B?K1Z0QkhTcDhOUWp1UkZYUVh4ZWpzZ2E4cmNXR21mbVl6eDdwV1BnbXhJUDNi?=
 =?utf-8?B?a1FMM04vWEpjOWE1N3dzSjlBUTB5QUR2MlVWSlYxK3poRmxiUzRwQ1MyRHhL?=
 =?utf-8?Q?6S3XRe4dT7sYfCK6eS4baJOlc?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0715dd2d-0c19-43cb-51b1-08dc5e8fa72a
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Apr 2024 03:37:06.1018
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: w1p3wbteTXBLFKv8k4Xr6HIVndb76x3PKv6z9rqRYQvDn12jbGZCqQ32/+9mnGwhOjObHedDcu6bf1GBmawKVg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7074
X-OriginatorOrg: intel.com

On 2024/4/16 11:00, Baolu Lu wrote:
> On 4/15/24 2:56 PM, Yi Liu wrote:
>> On 2024/4/15 13:00, Baolu Lu wrote:
>>> On 4/15/24 10:43 AM, Yi Liu wrote:
>>>> On 2024/4/15 10:22, Zhang, Tina wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Lu Baolu <baolu.lu@linux.intel.com>
>>>>>> Sent: Monday, April 15, 2024 9:39 AM
>>>>>> To: iommu@lists.linux.dev
>>>>>> Cc: Tian, Kevin <kevin.tian@intel.com>; Liu, Yi L 
>>>>>> <yi.l.liu@intel.com>; Jacob
>>>>>> Pan <jacob.jun.pan@linux.intel.com>; Joerg Roedel <joro@8bytes.org>; 
>>>>>> Will
>>>>>> Deacon <will@kernel.org>; Robin Murphy <robin.murphy@arm.com>; linux-
>>>>>> kernel@vger.kernel.org; Lu Baolu <baolu.lu@linux.intel.com>
>>>>>> Subject: [PATCH v3 1/1] iommu/vt-d: Remove caching mode check before
>>>>>> device TLB flush
>>>>>>
>>>>>> The Caching Mode (CM) of the Intel IOMMU indicates if the hardware
>>>>>> implementation caches not-present or erroneous translation-structure 
>>>>>> entries
>>>>>> except for the first-stage translation. The caching mode is 
>>>>>> irrelevant to the
>>>>>> device TLB, therefore there is no need to check it before a device TLB
>>>>>> invalidation operation.
>>>>>>
>>>>>> Remove two caching mode checks before device TLB invalidation in the 
>>>>>> driver.
>>>>>> The removal of these checks doesn't change the driver's behavior in 
>>>>>> critical
>>>>>> map/unmap paths. Hence, there is no functionality or performance impact,
>>>>>> especially since commit <29b32839725f> ("iommu/vt-d:
>>>>>> Do not use flush-queue when caching-mode is on") has already disabled
>>>>>> flush-queue for caching mode. Therefore, caching mode will never call
>>>>>> intel_flush_iotlb_all().
>>>>> The current logic is if the caching mode is being used and a domain 
>>>>> isn't using first level I/O page table, then flush-queue won't be 
>>>>> used. Otherwise, the flush-queue can be enabled.
>>>>> See https://github.com/torvalds/linux/commit/257ec29074
>>>>>
>>>>> In other words, if the caching mode is being used and a domain is 
>>>>> using first level I/O page table, the flush-queue can be used for this 
>>>>> domain to flush iotlb. Could the code change in this patch bring any 
>>>>> performance impact to this case?
>>>>
>>>> This seems to have performance deduction in the nested translation case.
>>>> The iommufd nested support bas been merged in 6.8, while the Qemu side
>>>> is wip. So this performance deduction does not happen until Qemu is
>>>> done. Should this also be considered as a performance regression? TBH.
>>>
>>> Caching mode is irrelevant to first-stage and nesting translations. If
>>> the QEMU implementation still relies on caching mode for nesting
>>> support, it's already broken.
>>
>> For first-stage, yes, caching mode is irrelevant. But for nesting, still
>> be relevant. Because it relies on caching mode to capture the mofifications
>> of the pasid entry. Without caching mode, vIOMMU device model within QEMU 
>> will miss the newly setup pasid entries, hence no way to setup nested
>> translation at all.
> 
> Okay, it's reasonable that nesting translation still uses second-stage.

the key point is not nesting uses second-stage, it is because of the guest
pasid entry capturing :)

> But the removed checks in this patch don't impact the pasid table change
> capture, right?

yes. The removal in this patch is irrelevant to it.

So Tina's concern makes sense. But I don't know if it should be considered
as a performance regression since nesting is not really fully ready (QEMU
side is not there yet).

BTW. I'm not sure if you have considered to correct the other caching mode
checks before dev-TLB invalidation. e.g. the CM check in 
drivers/iommu/intel/pasid.c.

-- 
Regards,
Yi Liu