Received-SPF: pass (google.com: domain of linux-kernel+bounces-42428-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <7adec292-9d38-41ab-a982-bd840b24f3ab@intel.com>
Date: Mon, 29 Jan 2024 17:21:40 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v12 5/5] iommu/vt-d: improve ITE fault handling if target
 device isn't present
Content-Language: en-US
To: "Tian, Kevin" <kevin.tian@intel.com>, Ethan Zhao
	<haifeng.zhao@linux.intel.com>, "baolu.lu@linux.intel.com"
	<baolu.lu@linux.intel.com>, "bhelgaas@google.com" <bhelgaas@google.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>, "jgg@ziepe.ca" <jgg@ziepe.ca>
CC: "dwmw2@infradead.org" <dwmw2@infradead.org>, "will@kernel.org"
	<will@kernel.org>, "lukas@wunner.de" <lukas@wunner.de>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>
References: <20240129034924.817005-1-haifeng.zhao@linux.intel.com>
 <20240129034924.817005-6-haifeng.zhao@linux.intel.com>
 <BN9PR11MB52761CC3E5F08D4B7BAD7F918C7E2@BN9PR11MB5276.namprd11.prod.outlook.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <BN9PR11MB52761CC3E5F08D4B7BAD7F918C7E2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M3ZMdDJ4STgweFpoV2pCQjBoMngrcXhHclhER2JLQkRGeWdKdCtoc0JtQnVU?=
 =?utf-8?B?LzFZVi9INXhGa2UzZ2R5dTU0NlA2b2dwTkQrR3AxdzVmYVJIYXJwWXBRZzk3?=
 =?utf-8?B?S2UyMkpkd0o3RWRVVWVzU0orVElNcFFoN25VZDVyenIwRjFDS09GTDdERjhR?=
 =?utf-8?B?cDdEM0VFMnc5NmlwRnlidkFpTU52bnhTcWdxeTIrbjFhSVJtczVBdkN2RGpi?=
 =?utf-8?B?RTk5SmRpenErM2pXd0tSeXVNZ091bjFmd0oxUWtDYlJxOTRaVFVMYk12bXZY?=
 =?utf-8?B?b28wanZMYkc1akpKV2xiT08vYm9JMW9SWXR6Vk4xMUt6TnZ3NFMrSUNiWE53?=
 =?utf-8?B?MG1FRFJ2ek5BSUNZNS82V2oxMkFvRnVUdWNKa2diYXlCek4zaDRDQ2hXOHFF?=
 =?utf-8?B?YVB6dm5jSDdxZVhLWE5HaFBpUi95S1J4d3A0OGhCanVTVE42Zm5KRUI1dkZX?=
 =?utf-8?B?WW9Ydit3UU4rR3c5b0xpQUxTMTlkTFdCa0VuZEFabzBXY05KMlBZeWRZK2cv?=
 =?utf-8?B?b0tXRlBRR3JMMU44bjZNZUxrYzZQeXhvRU5kODV2dWsvNlRibVBPV254akxI?=
 =?utf-8?B?eXpPWk1VWHN3NG1iR2VBNDNjUjVHek9reTdpak51ODAzL3pDV0dPVkxmbzlO?=
 =?utf-8?B?eGY5NEdNQmVnRWZGUWE0bjVzUFVaY2l2d2dFNWc5MHVlcjA1ZjQ0RjJiTGtZ?=
 =?utf-8?B?YUtYQVdHYkoyQ1RLVHFIdDdXM2Q3N2pHNFhGd0NFNUIvZWNvcVl2Z3RCcit5?=
 =?utf-8?B?U3NWSnRSVmt2R3VBOE9ZTTRzUWovQ1B4ZStIc29zUHBJWFMvWnN1VWVIL1JT?=
 =?utf-8?B?WFVueDJJMFBZMnFoZ0phM1NVVHBxdGVsQS9ETGVzWENrWW9HVDY1RE01R3Jv?=
 =?utf-8?B?czJMb0JjWkphWWZQNGZEWXJuVFhYdFZ0bGR2M2duWXpkc0hGaGtGaWFoOUln?=
 =?utf-8?B?eFp5Ni82a2doYys0NVVXbytidTNpTnZudUlVVTVLZFcxejQwSHp2NnVmYzEx?=
 =?utf-8?B?OHlYMHlpUmhDbi9zajFZWm51aFIxb2pxRGVYUkZUOFRTTktiekZWMU1URTNq?=
 =?utf-8?B?YUdXc2E3RXRjekt1MXZ2STlGK280d1A1VjdmSThpWGdXeWxGLzA3MVE5RG0x?=
 =?utf-8?B?UzVtem5HOXUyTlFpajZPOHNzQkw2a1RSUVdQVmtKTXFzNFdHSWF6eDZ6Y3Vl?=
 =?utf-8?B?enNyWWVTZHZLUytta0V6ODU1VmY4eEVOcmRkVVpXN2dYK2JlL2YrTGhabHds?=
 =?utf-8?B?UEJ1V0s3S0c0SDBuSXJzV1N2YW1rL2dqdUcwVStVSWxUZlI0Zkl1RG83STR3?=
 =?utf-8?B?dEltckRkVnBlZS9WNlpEQjI0U2d6NUxOUmhleW1VcWZjUW9OZ0JiY1VleWhU?=
 =?utf-8?B?VFdBYnhYNVB3ZXpUbGZhS3pvQlJua0JERkhsaXV4czA3Zi9hdnBSbHlqbDBC?=
 =?utf-8?B?UG1vYTQvWXFNaTBYc1lQWGFkeEpsS2hRYTdTc0lFYVh0TllMMWhiL25sODln?=
 =?utf-8?B?QzVOZzFyR2tRZHlXUzJPZWN6RHpzMlpjSEFWRm45Z0FFcUhVVGk3YVNkNGJv?=
 =?utf-8?B?NHhtVWlPR1dPQzR2TExoTTBkcE1UUG5TSWpkbVFTa1R1bUZGcEE5eGNpcFR0?=
 =?utf-8?B?alZaQ3UwUjg5eGNQcm5iMEdPaTZJZzhJM3RUSVhUU3hYRVJPcnZBZ0VJNElM?=
 =?utf-8?B?S21MU1NzSHVUT2dBSDVzZUlNNlREbnI4NzJlQ1VBRkd3ZjB2RmtzSGpueEly?=
 =?utf-8?B?ZzFDczdCUCs5NTRqNGNkNmpwcEs3K0RubDJObUIvS1dDMVZQd1VWQW9sNHVS?=
 =?utf-8?B?djQzZ0l1L0F4OXdOdTRyVmJ2dVMwNjNub0tBTFRpZDhzei90QjhLa2xyMEFZ?=
 =?utf-8?B?MFB4a3E4YjNBRjFhSWxGbWc0bEhhcldoTGVjeXBkYStrZUxvSWljSW83Ykly?=
 =?utf-8?B?WVViUng0S1JpcWt1SG84NXV1RHFSUTBacWJEc01KcGNGaFRaVXdIbXhZbnlr?=
 =?utf-8?B?VURSVTB0Y0RxSnd2WFlpK2xvdDJ2YXFyc1hTdTluemxHMk9XRUpQdjVVaDdx?=
 =?utf-8?B?bFZqS0EzdFRPWkJzRStPalFQY2QyTHMyY1QxMTdmek5Td2hEK3hreHA4a3ZF?=
 =?utf-8?Q?Ce3mLgnFPGG4WavYeq0Lssxow?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 99626e04-6cdc-43c6-a436-08dc20ab44cd
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2024 09:18:34.8489
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mylnPZ3kg8NXwAQL92hyiYlVd5R0z10IyC3vCYzhblJUx3WUS1iryk/sR425zSrBUXjo6s9gE/ZVRfCqjT7xqA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4639
X-OriginatorOrg: intel.com

On 2024/1/29 17:06, Tian, Kevin wrote:
>> From: Ethan Zhao <haifeng.zhao@linux.intel.com>
>> Sent: Monday, January 29, 2024 11:49 AM
>>
>> Because surprise removal could happen anytime, e.g. user could request safe
>> removal to EP(endpoint device) via sysfs and brings its link down to do
>> surprise removal cocurrently. such aggressive cases would cause ATS
>> invalidation request issued to non-existence target device, then deadly
>> loop to retry that request after ITE fault triggered in interrupt context.
>> this patch aims to optimize the ITE handling by checking the target device
>> presence state to avoid retrying the timeout request blindly, thus avoid
>> hard lockup or system hang.
>>
>> Signed-off-by: Ethan Zhao <haifeng.zhao@linux.intel.com>
>> ---
>>   drivers/iommu/intel/dmar.c | 18 ++++++++++++++++++
>>   1 file changed, 18 insertions(+)
>>
>> diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c
>> index 814134e9aa5a..2e214b43725c 100644
>> --- a/drivers/iommu/intel/dmar.c
>> +++ b/drivers/iommu/intel/dmar.c
>> @@ -1272,6 +1272,7 @@ static int qi_check_fault(struct intel_iommu
>> *iommu, int index, int wait_index,
>>   {
>>   	u32 fault;
>>   	int head, tail;
>> +	u64 iqe_err, ite_sid;
>>   	struct q_inval *qi = iommu->qi;
>>   	int shift = qi_shift(iommu);
>>
>> @@ -1316,6 +1317,13 @@ static int qi_check_fault(struct intel_iommu
>> *iommu, int index, int wait_index,
>>   		tail = readl(iommu->reg + DMAR_IQT_REG);
>>   		tail = ((tail >> shift) - 1 + QI_LENGTH) % QI_LENGTH;
>>
>> +		/*
>> +		 * SID field is valid only when the ITE field is Set in FSTS_REG
>> +		 * see Intel VT-d spec r4.1, section 11.4.9.9
>> +		 */
>> +		iqe_err = dmar_readq(iommu->reg + DMAR_IQER_REG);
>> +		ite_sid = DMAR_IQER_REG_ITESID(iqe_err);
>> +
>>   		writel(DMA_FSTS_ITE, iommu->reg + DMAR_FSTS_REG);
>>   		pr_info("Invalidation Time-out Error (ITE) cleared\n");
>>
>> @@ -1325,6 +1333,16 @@ static int qi_check_fault(struct intel_iommu
>> *iommu, int index, int wait_index,
>>   			head = (head - 2 + QI_LENGTH) % QI_LENGTH;
>>   		} while (head != tail);
>>
>> +		/*
>> +		 * If got ITE, we need to check if the sid of ITE is the same as
>> +		 * current ATS invalidation target device, if yes, don't try this
>> +		 * request anymore if the target device isn't present.
>> +		 * 0 value of ite_sid means old VT-d device, no ite_sid value.
>> +		 */
>> +		if (pdev && ite_sid && !pci_device_is_present(pdev) &&
>> +			ite_sid == pci_dev_id(pci_physfn(pdev)))
>> +			return -ETIMEDOUT;
>> +
> 
> since the hardware already reports source id leading to timeout, can't we
> just find the pci_dev according to reported ite_sid? this is a slow path (either
> due to device in bad state or removed) hence it's not necessary to add more
> intelligence to pass the pci_dev in, leading to only a partial fix can be backported.
> 
> It's also more future-proof, say if one day the driver allows batching invalidation
> requests for multiple devices then no need to pass in a list of devices.
> 
> Then it's easier to backport a full fix.

May consider pci_get_domain_bus_and_slot() or
pci_find_bus()/pci_get_slot(). But I doubt if the pci_dev is still tracked
in the bus or a kind of dev list in the device hot removal case. So Ethan
may need to check.

Regards,
Yi Liu

-- 
Regards,
Yi Liu