Received-SPF: pass (google.com: domain of linux-kernel+bounces-207261-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <996ad6df-c499-4070-b3a9-1cdccfcf5d09@linux.intel.com>
Date: Sun, 9 Jun 2024 15:23:18 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Cc: baolu.lu@linux.intel.com, "iommu@lists.linux.dev"
 <iommu@lists.linux.dev>,
 "virtualization@lists.linux-foundation.org"
 <virtualization@lists.linux-foundation.org>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v6 07/10] iommufd: Fault-capable hwpt
 attach/detach/replace
To: "Tian, Kevin" <kevin.tian@intel.com>, Jason Gunthorpe <jgg@ziepe.ca>,
 Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
 Robin Murphy <robin.murphy@arm.com>,
 Jean-Philippe Brucker <jean-philippe@linaro.org>,
 Nicolin Chen <nicolinc@nvidia.com>, "Liu, Yi L" <yi.l.liu@intel.com>,
 Jacob Pan <jacob.jun.pan@linux.intel.com>,
 Joel Granados <j.granados@samsung.com>
References: <20240527040517.38561-1-baolu.lu@linux.intel.com>
 <20240527040517.38561-8-baolu.lu@linux.intel.com>
 <BN9PR11MB5276FC9E1E7CF82BF51BC1FD8CFB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Language: en-US
From: Baolu Lu <baolu.lu@linux.intel.com>
In-Reply-To: <BN9PR11MB5276FC9E1E7CF82BF51BC1FD8CFB2@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 6/7/24 5:30 PM, Tian, Kevin wrote:
>> From: Lu Baolu <baolu.lu@linux.intel.com>
>> Sent: Monday, May 27, 2024 12:05 PM
>>
>> Add iopf-capable hw page table attach/detach/replace helpers. The pointer
>> to iommufd_device is stored in the domain attachment handle, so that it
>> can be echo'ed back in the iopf_group.
> 
> this message needs an update. now the device pointer is not in the
> attach handle.

The iommufd_device pointer is in the attach handle provided by iommufd 
in attach or replace path.

> and there worths a explanation about VF in the commit msg.
> 
>> @@ -376,7 +377,10 @@ int iommufd_hw_pagetable_attach(struct
>> iommufd_hw_pagetable *hwpt,
>>   	 * attachment.
>>   	 */
>>   	if (list_empty(&idev->igroup->device_list)) {
>> -		rc = iommu_attach_group(hwpt->domain, idev->igroup->group);
>> +		if (hwpt->fault)
>> +			rc = iommufd_fault_domain_attach_dev(hwpt, idev);
>> +		else
>> +			rc = iommu_attach_group(hwpt->domain, idev-
>>> igroup->group);
> 
> Does it read better to have a iommufd_attach_device() wrapper with
> above branches handled internally?

Yes. Will do this in the next version.

> 
>>
>> +static int iommufd_fault_iopf_enable(struct iommufd_device *idev)
>> +{
>> +	struct device *dev = idev->dev;
>> +	int ret;
>> +
>> +	/*
>> +	 * Once we turn on PCI/PRI support for VF, the response failure code
>> +	 * could not be forwarded to the hardware due to PRI being a shared
> 
> you could but just doing so is incorrect. ????
> 
> s/could/should/

Done.

> 
>> +	 * resource between PF and VFs. There is no coordination for this
>> +	 * shared capability. This waits for a vPRI reset to recover.
>> +	 */
> 
> this may go a bit further to talk about supporting it requires an emulation
> in iommufd (i.e. pause any further fault delivery until vPRI reset). It is a
> future work so disable it for VF at this point.

Yes.

> 
>> +void iommufd_fault_domain_detach_dev(struct iommufd_hw_pagetable
>> *hwpt,
>> +				     struct iommufd_device *idev)
>> +{
>> +	struct iommufd_attach_handle *handle;
>> +
>> +	handle = iommufd_device_get_attach_handle(idev);
>> +	iommu_detach_group_handle(hwpt->domain, idev->igroup->group);
>> +	iommufd_auto_response_faults(hwpt, handle);
>> +	iommufd_fault_iopf_disable(idev);
>> +	kfree(handle);
> 
> this assumes that the detach callback of the iommu driver needs to drain
> in-fly page requests so no further reference to handle or queue new req
> to the deliver queue when it returns, otherwise there is a use-after-free
> race or stale requests in the fault queue which auto response doesn't
> cleanly handle.
> 
> iirc previous discussion reveals that only intel-iommu driver guarantees
> that behavior. In any case this should be documented to avoid this being
> used in a non-conforming iommu driver.
> 
> If I misunderstood, some comment why no race in this window is also
> appreciated. ????

Yes. The iommu driver needs to guarantee that there will be no iopf
request for a RID or PASID after the domain has been detached. This
implies that:

- IOMMU hardware should not put further iopf in its hardware queue if
   the domain has been detached.
- Before the domain detachment is complete, the iommu driver should
   flush all iopf targeting the detached domain in the hardware page
   request queue.

> 
>> +}
>> +
>> +static int __fault_domain_replace_dev(struct iommufd_device *idev,
>> +				      struct iommufd_hw_pagetable *hwpt,
>> +				      struct iommufd_hw_pagetable *old)
>> +{
>> +	struct iommufd_attach_handle *handle, *curr = NULL;
>> +	int ret;
>> +
>> +	if (old->fault)
>> +		curr = iommufd_device_get_attach_handle(idev);
>> +
>> +	if (hwpt->fault) {
>> +		handle = kzalloc(sizeof(*handle), GFP_KERNEL);
>> +		if (!handle)
>> +			return -ENOMEM;
>> +
>> +		handle->handle.domain = hwpt->domain;
>> +		handle->idev = idev;
>> +		ret = iommu_replace_group_handle(idev->igroup->group,
>> +						 hwpt->domain, &handle-
>>> handle);
>> +	} else {
>> +		ret = iommu_replace_group_handle(idev->igroup->group,
>> +						 hwpt->domain, NULL);
>> +	}
>> +
>> +	if (!ret && curr) {
>> +		iommufd_auto_response_faults(old, curr);
>> +		kfree(curr);
>> +	}
> 
> In last version you said auto response is required only when switching
> from fault-capable old to fault-incapable new. But above code doesn't
> reflect that description?

What the current code does is auto-respond to all page faults targeting
the old fault-capable hwpt. I'm also okay if we decide to limit this to
flushing page faults only if the new hwpt is *not* fault-capable.

Best regards,
baolu