Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Message-ID: <40c82e24-cff3-1f96-b469-fbfd18238d5d@intel.com>
Date:   Fri, 22 Sep 2023 09:07:28 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.15.1
Subject: Re: [PATCH v4 5/6] iommu: Support mm PASID 1:n with sva domains
To:     Jason Gunthorpe <jgg@ziepe.ca>
CC:     Kevin Tian <kevin.tian@intel.com>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        Michael Shavit <mshavit@google.com>,
        Vasant Hegde <vasant.hegde@amd.com>, <iommu@lists.linux.dev>,
        <linux-kernel@vger.kernel.org>
References: <20230912125936.722348-1-tina.zhang@intel.com>
 <20230912125936.722348-6-tina.zhang@intel.com>
 <20230921233133.GB13795@ziepe.ca>
Content-Language: en-US
From:   Tina Zhang <tina.zhang@intel.com>
In-Reply-To: <20230921233133.GB13795@ziepe.ca>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Tm5mMzREZncxVVgxdHFzVk5pb1A1L0hLRGxDaWZlNkpnbjQyc0ZrQmphSGZv?=
 =?utf-8?B?UHp4OC9jMXZHaXFEdmlOU3o1dWt6QnEvWnZuU3o1V3FyMTNab1ZNUHIxUGk2?=
 =?utf-8?B?TW05aElENWRFaGpRQ0VIVGFMWitLQ0hHZDg3Mm9jQjRtK2pOcmdVbmw2Z0dK?=
 =?utf-8?B?YWpMbzB4TEdpQWhnLzBxM0wyamRmMytwTDFFTFBmVDdYbnNQM2JWbDBvUHJE?=
 =?utf-8?B?YXI2TTFZamZFSys3ZEVNSS9KRmlQaVlpa1FVQW5YR0JHTko0d3V2M28wOFlL?=
 =?utf-8?B?d3AvZm1BVWhrdWlmeHdMOFFoeURvMEpocXA0UTNvVXNuZWlVYlI2QXVJSkRa?=
 =?utf-8?B?VWNldWtON2xSUk5lWGRXeDYrMEpzV3B6U3Bob1JFbzgvRm5jSXBBTUhYMjZC?=
 =?utf-8?B?aGRrRVRqZ2doY29WREQ3RVZQa2Jmai9xSStjNFJIRVV6dHF6YWJuTEY0TExJ?=
 =?utf-8?B?WGQrTjk3Y24rMVduTUM4UllPaUJlamJHaG95bmxIUjdhcWVyZzhNTitiYVRp?=
 =?utf-8?B?VHkzT0NYWDBUOERzMkgramN0eEd6c3R3bUhhL3NmUHVrTE9tZHV1b2w3dWRF?=
 =?utf-8?B?djNNWFFzSUJiRUk3YmZhK3lqYVBoQWVZTVd4UUQ5K1g1YWxJN3hSS3ZzNFVQ?=
 =?utf-8?B?Z2s2cVkrNFNNRW9ZblZUV3RLZkhLdGtrSUlDSE1kRzJlaXdvS21hR1A5dE1v?=
 =?utf-8?B?SVZxQS9XZmV3Szc3RmlGbXN2bjdaVVhEbUhwQ1hXTGN3R1l1YXRzSDF6OXly?=
 =?utf-8?B?My9EaVNVU2tSeXFZUXdRSUNlcWpiYVBCTjBYM0l5c2ZLc2hYV1UwOVRQK3Ra?=
 =?utf-8?B?UWVLS0hKWWJJeHNlUGZscTdwQ09mMS9HNCsreTFIcFQ3Ym5pd3ZEaXZKUTh5?=
 =?utf-8?B?QnlrTi91bWxER0JpYllObkFvQ3pRdlBBbHBHYWtHL0JCKyswWVZzVFAzcjJt?=
 =?utf-8?B?RHp6bmxsU2VXQ0U2eStzcjdabjQ5eEpTZFIzODBtSW9HWFJ0YmhyQzhpMHhH?=
 =?utf-8?B?UnRSa0ppTlpRTmtKK2hUQ2RsS3g0TWNzT1htaVovT2g0RHowa0xIRlhOdmF3?=
 =?utf-8?B?dDRzTVovQ3J0R1NPVlUvRHRvcWQzQkxQQ2E2NkpzOG9OcnVkWFk4OXd3WGVP?=
 =?utf-8?B?TGxVL3pLZGlhSm1sZzB0eXgwckJ2VWdueVppeG1Randpc09kb2xxbE11RDFs?=
 =?utf-8?B?QjdpQXJydWliL0d2aG1NR1FOM3gxL2VLdkl6OExmMmRlejZGZXFHbUovMTVO?=
 =?utf-8?B?YUlpYWJKdWc3WnpRYWRSaSszTndsQmJ4eHJiaVlaL0NiS1NMa0h4eHcram51?=
 =?utf-8?B?NGZnQ281WnpsM3EzTXdOdEI3RmFjNjBpNnVMcnY2YTRHUWJrRVlob2VlZmhk?=
 =?utf-8?B?NU5QM0N4UDdSWDNzRXhjek5VbDAyMXk4NjVKaXJmMW9vbmJCS3lWMTRyVnYv?=
 =?utf-8?B?bzNvNXUxdEVSZFlUdUZxS0liLzN4NXV5d2RsZTBkOGswM1JYN3l0WTZScXUx?=
 =?utf-8?B?Nk45MDRadS84R3lWRXJRNms1djNVK3ZSa2RybkNBQUp2My9abjRDVm8zeVB0?=
 =?utf-8?B?V3VWcVJrWUJkN3duVjJ1OW1IVGp0VHptRG5SZEZlOUJzT1dubTNCZW56MkN4?=
 =?utf-8?B?QmFOS2lFczVLSW1oUUwzYjdNbWp0RlB2N0ZEellpZ3lITFpuL1lsOXhvT1hD?=
 =?utf-8?B?RGo5b21tN1FFNnBNQlV6ZFV4SXowWk42a1Y0akp5VlJPa1lzUThiaUZvcFBv?=
 =?utf-8?B?dkp1M2Z4N2ZKZlNxODJkdXZtRE5ZZ2wrNWJieEx2dUx5c3h4Tjh2d2NaZmt2?=
 =?utf-8?B?TlFzSFZkZEpKb2RXNGJ1V1RNcERHaW5SUVNPWkdXdXg4dVZTZjNmeFVaaDVI?=
 =?utf-8?B?Q0cvS2hSbEFTNmNkQWJoWVVVZk00L0cvRzFyeEp4L2FPSEc1NDFacXFaUjBn?=
 =?utf-8?B?ZDdwVDV1bExuQXNtbS9SVFlDRW1GaVNWa0h0S0pBWi9sSkxTY3l3Q1ZnTEdw?=
 =?utf-8?B?RWZ2a0xoaGdpQWhYeE9sUmhZS2tYRWQrT0FqQTdyTnV2TGtId0tOdEJkQjVs?=
 =?utf-8?B?dEUrd3N3Z0ZvVTkwZTMrekNHS3NzQlVCZHhvTHRRM2FxSlAvMHQrUS84Q3li?=
 =?utf-8?Q?8+bJjSPgsPxe/+TUeuwbaMNX9?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e54caf9c-23c1-4751-1052-08dbbb084e3e
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5881.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2023 01:07:35.2545
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QljY4W6NJMiy3p8L4HzyV2Touha8vGWE93NxlwuQfr/Ahbd++hX4sAHnfzLsWFGAOsOQmcVH1E8Y+0rtqrL3bQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR11MB6433
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 21 Sep 2023 18:07:44 -0700 (PDT)

Hi Jason,

On 9/22/23 07:31, Jason Gunthorpe wrote:
> On Tue, Sep 12, 2023 at 08:59:35PM +0800, Tina Zhang wrote:
>> diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c
>> index 0f956ecd0c9b..79779bbef653 100644
>> --- a/drivers/iommu/iommu-sva.c
>> +++ b/drivers/iommu/iommu-sva.c
>> @@ -15,6 +15,7 @@ static DEFINE_MUTEX(iommu_sva_lock);
>>   static int iommu_sva_alloc_pasid(struct mm_struct *mm, struct device *dev)
>>   {
>>   	ioasid_t pasid;
>> +	struct iommu_mm_data *iommu_mm;
>>   	int ret = 0;
>>   
>>   	if (!arch_pgtable_dma_compat(mm))
>> @@ -28,12 +29,22 @@ static int iommu_sva_alloc_pasid(struct mm_struct *mm, struct device *dev)
>>   		goto out;
>>   	}
>>   
>> +	iommu_mm = kzalloc(sizeof(struct iommu_mm_data), GFP_KERNEL);
>> +	if (!iommu_mm) {
>> +		ret = -ENOMEM;
>> +		goto out;
>> +	}
>> +
>>   	pasid = iommu_alloc_global_pasid(dev);
>>   	if (pasid == IOMMU_PASID_INVALID) {
>> +		kfree(iommu_mm);
>>   		ret = -ENOSPC;
>>   		goto out;
>>   	}
>> -	mm->pasid = pasid;
>> +	iommu_mm->pasid = pasid;
>> +	INIT_LIST_HEAD(&iommu_mm->sva_domains);
>> +	mm->iommu_mm = iommu_mm;
>> +
>>   	ret = 0;
>>   out:
>>   	mutex_unlock(&iommu_sva_lock);
>> @@ -73,16 +84,12 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
>>   
>>   	mutex_lock(&iommu_sva_lock);
>>   	/* Search for an existing domain. */
>> -	domain = iommu_get_domain_for_dev_pasid(dev, mm_get_pasid(mm),
>> -						IOMMU_DOMAIN_SVA);
>> -	if (IS_ERR(domain)) {
>> -		ret = PTR_ERR(domain);
>> -		goto out_unlock;
>> -	}
>> -
>> -	if (domain) {
>> -		domain->users++;
>> -		goto out;
>> +	list_for_each_entry(domain, &mm->iommu_mm->sva_domains, next) {
>> +		ret = iommu_attach_device_pasid(domain, dev, mm_get_pasid(mm));
>> +		if (!ret) {
>> +			domain->users++;
>> +			goto out;
>> +		}
>>   	}
>>   
>>   	/* Allocate a new domain and set it on device pasid. */
>> @@ -96,6 +103,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm
>>   	if (ret)
>>   		goto out_free_domain;
>>   	domain->users = 1;
>> +	list_add(&domain->next, &mm->iommu_mm->sva_domains);
>> +
>>   out:
>>   	mutex_unlock(&iommu_sva_lock);
>>   	handle->dev = dev;
>> @@ -128,8 +137,9 @@ void iommu_sva_unbind_device(struct iommu_sva *handle)
>>   	struct device *dev = handle->dev;
>>   
>>   	mutex_lock(&iommu_sva_lock);
>> +	iommu_detach_device_pasid(domain, dev, pasid);
>>   	if (--domain->users == 0) {
>> -		iommu_detach_device_pasid(domain, dev, pasid);
>> +		list_del(&domain->next);
>>   		iommu_domain_free(domain);
>>   	}
>>   	mutex_unlock(&iommu_sva_lock);
>> @@ -209,4 +219,5 @@ void mm_pasid_drop(struct mm_struct *mm)
>>   		return;
>>   
>>   	iommu_free_global_pasid(mm_get_pasid(mm));
>> +	kfree(mm->iommu_mm);
>>   }
>> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
>> index b9c9f14a95cc..c61bc45d5a82 100644
>> --- a/include/linux/iommu.h
>> +++ b/include/linux/iommu.h
>> @@ -109,6 +109,11 @@ struct iommu_domain {
>>   		struct {	/* IOMMU_DOMAIN_SVA */
>>   			struct mm_struct *mm;
>>   			int users;
>> +			/*
>> +			 * Next iommu_domain in mm->iommu_mm->sva-domains list
>> +			 * protected by iommu_sva_lock.
>> +			 */
>> +			struct list_head next;
>>   		};
>>   	};
>>   };
>> @@ -1186,17 +1191,13 @@ static inline bool tegra_dev_iommu_get_stream_id(struct device *dev, u32 *stream
>>   }
>>   
>>   #ifdef CONFIG_IOMMU_SVA
>> -static inline void mm_pasid_init(struct mm_struct *mm)
>> -{
>> -	mm->pasid = IOMMU_PASID_INVALID;
>> -}
>>   static inline bool mm_valid_pasid(struct mm_struct *mm)
>>   {
>> -	return mm->pasid != IOMMU_PASID_INVALID;
>> +	return mm->iommu_mm ? true : false;
>>   }
> 
> HUm this isn't locked very nicely.
> 
> Above do
> 
> smp_store_release(&mm->iommu_mm, iommu_mm);
> 
> And then do
> 
>    return READ_ONCE(mm->iommu_mm)
Good suggestion. Thanks.

> 
> (no need for ternaries with bools, compiler generates it automatically)
> 
> 
>>   static inline u32 mm_get_pasid(struct mm_struct *mm)
>>   {
>> -	return mm->pasid;
>> +	return mm->iommu_mm ? mm->iommu_mm->pasid : IOMMU_PASID_INVALID;
>>   }
> 
> Then this should be
> 
> struct iommu_mm_data *iommu_mm = READ_ONCE(mm->iommu_mm);
> 
> if (!iommu_mm)
>     return IOMMU_PASID_INVALID;
> return iommu_mm->pasid;
> 
> Keeping in mind that the kfree(mm->iommu_mm) being placed in the
> mm_drop is critical to this working safely.
Right. Thanks.


Regards,
-Tina
> 
> Otherwise the logic looks OK.
> 
> Jason
>