Received-SPF: pass (google.com: domain of linux-kernel+bounces-15038-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <ae271e08-f390-4ce7-914c-63668a46bc4b@intel.com>
Date: Wed, 3 Jan 2024 09:33:48 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v10 10/10] iommu/vt-d: Add iotlb flush for nested domain
Content-Language: en-US
To: Jason Gunthorpe <jgg@nvidia.com>
CC: <joro@8bytes.org>, <alex.williamson@redhat.com>, <kevin.tian@intel.com>,
	<robin.murphy@arm.com>, <baolu.lu@linux.intel.com>, <cohuck@redhat.com>,
	<eric.auger@redhat.com>, <nicolinc@nvidia.com>, <kvm@vger.kernel.org>,
	<mjrosato@linux.ibm.com>, <chao.p.peng@linux.intel.com>,
	<yi.y.sun@linux.intel.com>, <peterx@redhat.com>, <jasowang@redhat.com>,
	<shameerali.kolothum.thodi@huawei.com>, <lulu@redhat.com>,
	<suravee.suthikulpanit@amd.com>, <iommu@lists.linux.dev>,
	<linux-kernel@vger.kernel.org>, <linux-kselftest@vger.kernel.org>,
	<zhenzhong.duan@intel.com>, <joao.m.martins@oracle.com>,
	<xin.zeng@intel.com>, <yan.y.zhao@intel.com>, <j.granados@samsung.com>
References: <20240102143834.146165-1-yi.l.liu@intel.com>
 <20240102143834.146165-11-yi.l.liu@intel.com>
 <20240102184422.GI50406@nvidia.com>
From: Yi Liu <yi.l.liu@intel.com>
In-Reply-To: <20240102184422.GI50406@nvidia.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RDdodVNsQWlYNGIyN3BoOGMvUlBZQ2NKdFdpT29JSmhGTjFkZ0ZabGtxSTlM?=
 =?utf-8?B?Z29TZVJSYzhuakhsT3RSRzVnRlpxcXBROUJpQUxwZTdmNkgvcFBubHI3KzBi?=
 =?utf-8?B?d3RGYnF5K1djclBIUmUxQ2FmWHFzMldXYUxZMHNtM25IMmV5QzJrOFV0ZDI1?=
 =?utf-8?B?eTM0OENGdzZvTjVRbFNGLzA2Q0FEUFd2VUxlbVN2V0NqN0liSjBmTm5KdGFl?=
 =?utf-8?B?QytQaHNMaTFYUFdOdmd0SXhkTGVIbWJaN0VNSzREVXJBaG9wTHN4OVFMSUpv?=
 =?utf-8?B?VnFEL09vVi9wcURNWDc0VzVobDhQM1d1M1RRVk5ZdjhKRnJicHJCcDlJSm1x?=
 =?utf-8?B?WUdtNzVJa0E2RjIrNlQwbGJkWkduc3FibFVPbk1mR2RXdjRsRkdPZ3J4UzVO?=
 =?utf-8?B?ZEN1NVp4M1FxbHQvdEJSci9TNFFRTktsVGF4WVYxUC9qeEJmeittL3RGc3J1?=
 =?utf-8?B?MDE5OGxESktyV3gzWGpleFVmcXZhbEJsR0ljNDJOQmwvZzhCSzZqdTAwUnph?=
 =?utf-8?B?bjBWRGFOeldVQzVrQjRPQkVsNkxVemFuS3B2SEd1ZlgwNTRuTFBsK1ZCcHlM?=
 =?utf-8?B?ODdPczlTUnU3dlRNKzJDb1IwOXZXVFFvVEJIQU1pSjdJTkhJTjVGOWJxckNG?=
 =?utf-8?B?QnBkdGoxL3llWndOVWFtNEJGR2NhZUtWUWtreUdQTk9SRFJUTmZ4VittYzlT?=
 =?utf-8?B?SURuNStyM1dPSmtTWGxmN1lpdnFyQStEQk1LVit5eDZMdHhqbXNJUEZ6Zi9E?=
 =?utf-8?B?ZDRUNHp4YVI3STBJZGxoNnIrLzRQZWdoTG5LMEFwKy9ONml0SUhtYjhQa3Z0?=
 =?utf-8?B?cXNkdnhTTnV2d1RXUXE3cUJKVlNhTEdsZTFwT2EzOCtIRnMwbldDVEpiRUtL?=
 =?utf-8?B?bEEwWW9LZzlsNWJOUDlLMzlQSWJVeFVvZjZVcHFCRldGV21ZVnpGOVpJcThT?=
 =?utf-8?B?dEFVZkFhMGI3VmRRbmpmMmtWWHgzak93S2MweXF5QXk4MnNCM2VrT25OWHY1?=
 =?utf-8?B?ZndRcTYxNGhyUkxudGx1aWdHWkhyQWswb1VJRjBtTXRXZG5rT1d3b2w2Zi9Q?=
 =?utf-8?B?eDJYTkQzek81T0l4WWNnU09MbHYvMHYvNDVSUnVIQllsT1k5MWhWZDlwSFFQ?=
 =?utf-8?B?ZmxMUEdzMnhaaHpXby9yQ3VLck04bGRiYTB6QVRMaEtOcHcxcUdyQlJMT21D?=
 =?utf-8?B?MkMyVXlTUVk0U2NUVFQvN0JOcnBRaGwzMklRN1JqUEgwWHVDakxXSzhkTU5E?=
 =?utf-8?B?ZGtuckRBZndRS1lORzBkcDN4WlVIUm1LUU5sRkkwQmVtbWVFWjVxS3J4bmRB?=
 =?utf-8?B?d2tnR0dDNkgwLzdFUjJsS3FoeU04VmQ2MTU5UFFFOGR5YjlmaFhQN0J2TVBP?=
 =?utf-8?B?QXdzT0dDWmNHQnBWU0tVc1JrbFFtN2R0bGpDL1dEd3VvYWJsVUZHV09ETytL?=
 =?utf-8?B?a29wcGMrVU5KYWgvbXVzRVYwSmhNTXNTcDlKclhRdGxQQkgzZ3RPeXVYV3RI?=
 =?utf-8?B?VmlJQm5oUGJLek5MRE1FRU43VHNEOHI1UGJhR2VQcXpqQkZqbHg0N215cjNK?=
 =?utf-8?B?OWhjQ0lyRFZhelNQR1Qva21VNy9DcHpadlF1elp2Y2t1N0hsK25vdWhXT0Er?=
 =?utf-8?B?YUdLVnd1Z2w4S1N0bFpTTHNteDZUT3JtaUNpZVp5OE8xOUtMQVFRV2l5ajN6?=
 =?utf-8?B?RU9vQnUvRHppcXluU2lHbzVhUWNDTGpTNSs0aXBXZGM5ekVnTHhJN1dzYzdK?=
 =?utf-8?B?SW5WcUI4ay80SFlvaXF1WkxoUGJqNU5vVy8yVk5sVkw0M1NEM2lQenRqVEE4?=
 =?utf-8?B?QllwbEx2emdjRjBLZGRzVkE4M3k0V2Z6bUIrUytkb3E1b3IwcnB6aFZqeVdS?=
 =?utf-8?B?czhDTTBCeUhIRTVTbndJemNJR3kzNkp3RVNoVU0rUkYySWdiZll6VW4yL21w?=
 =?utf-8?B?WFFhOFFjb0h6ajIyV201aEJUTlA5QjJNbmlVTzNZUFdqdytCZ2JKOC9BRHJ3?=
 =?utf-8?B?Z0JEMVFPa2dYa1hPejhGT2hwclpSeXFaeWxiWXgvR1F2UDlIem1UcHc2MDNy?=
 =?utf-8?B?ajQ2cGhLZDIyYVhPNGNlcXV0UGlRKytxYXJmeFFzbWtRT2RCbW1LZUNXWDBT?=
 =?utf-8?Q?PTiRhM7DcVLgxDJa9MirJaVH0?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f03dd52c-3209-4c21-2cfe-08dc0bfba348
X-MS-Exchange-CrossTenant-AuthSource: SN7PR11MB7540.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 01:30:59.1737
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ed5BReSn3cH/UB+W9UOkWIHMxcatDpG878m0CfMkfTB/q4x9dJI6qxJNjqgyOGgU0MBRbbs22s3C3/tIVKqYdQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR11MB6048
X-OriginatorOrg: intel.com

On 2024/1/3 02:44, Jason Gunthorpe wrote:
> On Tue, Jan 02, 2024 at 06:38:34AM -0800, Yi Liu wrote:
> 
>> +static void intel_nested_flush_cache(struct dmar_domain *domain, u64 addr,
>> +				     unsigned long npages, bool ih, u32 *error)
>> +{
>> +	struct iommu_domain_info *info;
>> +	unsigned long i;
>> +	unsigned mask;
>> +	u32 fault;
>> +
>> +	xa_for_each(&domain->iommu_array, i, info)
>> +		qi_flush_piotlb(info->iommu,
>> +				domain_id_iommu(domain, info->iommu),
>> +				IOMMU_NO_PASID, addr, npages, ih, NULL);
> 
> This locking on the xarray is messed up throughout the driver. There
> could be a concurrent detach at this point which will free info and
> UAF this.

hmmm, xa_for_each() takes and releases rcu lock, and according to the
domain_detach_iommu(), info is freed after xa_erase(). For an existing
info stored in xarray, xa_erase() should return after rcu lock is released.
is it? Any idea? @Baolu

void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
{
	struct iommu_domain_info *info;

	spin_lock(&iommu->lock);
	info = xa_load(&domain->iommu_array, iommu->seq_id);
	if (--info->refcnt == 0) {
		clear_bit(info->did, iommu->domain_ids);
		xa_erase(&domain->iommu_array, iommu->seq_id);
		domain->nid = NUMA_NO_NODE;
		domain_update_iommu_cap(domain);
		kfree(info);
	}
	spin_unlock(&iommu->lock);
}

> This seems to be systemic issue, so I'm going to ignore it here, but
> please make a series to fix it completely.

yeah, this writing is the same with other places that reference the 
iommu_array. If there is real problem, may check with Baolu and Kevin.

> xarray is probably a bad data structure to manage attachment, a linked
> list is going to use less memory in most cases and you need a mutex
> lock anyhow.

below is the commit that introduces iommu_array.

commit ba949f4cd4c39c587e9b722ac7eb7f7e8a42dace
Author: Lu Baolu <baolu.lu@linux.intel.com>
Date:   Tue Jul 12 08:09:05 2022 +0800

     iommu/vt-d: Refactor iommu information of each domain

     When a DMA domain is attached to a device, it needs to allocate a domain
     ID from its IOMMU. Currently, the domain ID information is stored in two
     static arrays embedded in the domain structure. This can lead to memory
     waste when the driver is running on a small platform.

     This optimizes these static arrays by replacing them with an xarray and
     consuming memory on demand.

     Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
     Reviewed-by: Kevin Tian <kevin.tian@intel.com>
     Reviewed-by: Steve Wahl <steve.wahl@hpe.com>
     Link: 
https://lore.kernel.org/r/20220702015610.2849494-4-baolu.lu@linux.intel.com
     Signed-off-by: Joerg Roedel <jroedel@suse.de>


-- 
Regards,
Yi Liu