Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jacob Pan <jacob.jun.pan@linux.intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        Jason Gunthorpe <jgg@nvidia.com>,
        Lu Baolu <baolu.lu@linux.intel.com>,
        Joerg Roedel <joro@8bytes.org>,
        Robin Murphy <robin.murphy@arm.com>,
        Jean-Philippe Brucker <jean-philippe@linaro.com>,
        "dmaengine@vger.kernel.org" <dmaengine@vger.kernel.org>,
        "vkoul@kernel.org" <vkoul@kernel.org>
CC:     Will Deacon <will@kernel.org>,
        David Woodhouse <dwmw2@infradead.org>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "Zanussi, Tom" <tom.zanussi@intel.com>,
        "Ranganathan, Narayan" <narayan.ranganathan@intel.com>
Subject: RE: [PATCH v7 3/4] iommu/vt-d: Add set_dev_pasid callback for dma
 domain
Thread-Topic: [PATCH v7 3/4] iommu/vt-d: Add set_dev_pasid callback for dma
 domain
Thread-Index: AQHZjZxI8RRUCSa7gUG6SDjy6Qi7Gq9qiM7g
Date:   Thu, 25 May 2023 06:56:33 +0000
Message-ID: <BN9PR11MB52760139A4844C8DF0EE1BE98C469@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230523173451.2932113-1-jacob.jun.pan@linux.intel.com>
 <20230523173451.2932113-4-jacob.jun.pan@linux.intel.com>
In-Reply-To: <20230523173451.2932113-4-jacob.jun.pan@linux.intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?KUOR4mZaHituXwYc4Ytpd2qYr8YL5JfHP/ChnzE5xIRxbqQ//UmoOxVjV1P6?=
 =?us-ascii?Q?BMs1BtIBwg+8m9RrLx3ZeZsegBfpwMQszQONQqjBEta6f4Y9WcuqgZhX1pVw?=
 =?us-ascii?Q?OPfBiPTFOf/PLWRt3NP86lzvhe/qpJ6w4QjZuWVhh4O9tc/hipPreNfT+u4P?=
 =?us-ascii?Q?avpLJEqYZCFYAO7IJjjWJlrQN9WMVkwLA8cTEpuGkrvQaOX6Za0MHeguDJyp?=
 =?us-ascii?Q?58WN+2wl2sUL6cud2acfUBYNkhvY+2MgvP2o+1N45O1luYELio4h95hdYwlc?=
 =?us-ascii?Q?bwKAJJT1zYH1IYYard/G0CWzuZDTdboIyl6v6M375y7Z9G1dlUhXFJ8/Ninb?=
 =?us-ascii?Q?u59wmNtgUNtEh3UuAcdFXPYAPASqSdgu/5RuzKNPUVH7+yHO+i9JPdVuz5Gt?=
 =?us-ascii?Q?tTjWNhghV/AJwpFCWd3U8eeavysGgv61907C8FfkD+4Rdu6XrfQDbqduCubV?=
 =?us-ascii?Q?0OsU+6d+25NxOEOeWNXm4WLtF23CWYs0zxfFhhwEu6ojFmkBcQECxgRCArKd?=
 =?us-ascii?Q?cCZ3tfvlI287M6InGGgHQ+lyMaxvW2eNBNnBtFtsSgZrh9LoAZyLTuiWg7WF?=
 =?us-ascii?Q?SW4evB2KNf1S6KhEv/ys2Iy9N+nmsbpxTOdC8Hp6hWpsm3P1LnHI+vMd4vwm?=
 =?us-ascii?Q?TXso0U8Hy3Zf8pRVlAjNp8tGz2n0bYFSXSvddRtHQroL8r507tT68oOjZIX7?=
 =?us-ascii?Q?ZEsWy31BnSJFc0An71vMKXbcBvw9eUWQ387L6kBqgB6Fk6fDGPD+06IGRymM?=
 =?us-ascii?Q?lrpEdPwrWvMZis/zi6wWLHO0e3ntplqhczmYFsv0M+x+fY+iuZthtDwS9Rkr?=
 =?us-ascii?Q?zNJuxr4xogp68EneEbxFDC2jfyY430vRmpmG56TZpFXg7HUsiM7CPXGr/Rgo?=
 =?us-ascii?Q?1yekiHTpIkkZhi5mvR9fayMH0yc4XUqPeVqP/M/bL6BJj5eD/+IuU3xjwE8U?=
 =?us-ascii?Q?3GjHJqIpGPPRiJawHiSWtme5STsYo/kGWH71eNViJpVQE2vwmS1UrbeT+VQr?=
 =?us-ascii?Q?CoYv9Mav++tSB6AIxqCD4HPD8euAvSaj72ag3OxsGBLvu9qI1ZIyAKuhkjVM?=
 =?us-ascii?Q?W1vEc5QLlhxRlJpptrvfLRJtyTPmQzwN9twFVWqMI+IVIt+GOwvdVgdWFsIA?=
 =?us-ascii?Q?eJE5Z7lnMCaqFLq6qUE/c6413gBp/cxLNJZyeKwCbetecKljN+YQCBfFoM8k?=
 =?us-ascii?Q?a4sV+mMTw41GTBh3/y3yhtHO33U5heOtQ7A60Ln8LOJ2el+1OAyv2a7wrtD5?=
 =?us-ascii?Q?Ils8jP93eFypYPLJ0FhXGlOnpdKsUq2kVc/Supmb/IzPK18yVs3lMvEGQULY?=
 =?us-ascii?Q?LJEPFmhXfLnWI/nhX6ydoUj5irR+cAjXMtL7d6XfTtuhe3NXPnmsdSHQjUma?=
 =?us-ascii?Q?hGwPA/Lg83frOKLfJgWoEndEAXT5wzkFILGvVUTbloUtwJbhTLpE2EZ66eUh?=
 =?us-ascii?Q?O44+0VHnv/nVvpD8li717RnR96eJ8TgTff9enXL9/pJ9IhO98R+Z8ZWx2CJk?=
 =?us-ascii?Q?R73mj0FBJ7WrBgJhcSMi9AStgR6JSiLji9TsUroXBrRP0DJjqjvQYqIqUheX?=
 =?us-ascii?Q?4atwYicKl276soQa/EHu+kbQa6OQFbUSA0wuM+eB?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7864e0ac-ded4-46a5-5c00-08db5ced2d49
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 May 2023 06:56:33.9415
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 6NbrVTRt0dvHDuhUuIDw16cGawYriMchOrWh3sYwYPGp071zoExr3KDxmH27HH56uTYGiCtCXTGLGdrHNMPbbQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR11MB6024
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_MSPIKE_H2,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Jacob Pan <jacob.jun.pan@linux.intel.com>
> Sent: Wednesday, May 24, 2023 1:35 AM
>=20
> @@ -1472,6 +1482,37 @@ static void iommu_flush_dev_iotlb(struct
> dmar_domain *domain,
>  	spin_lock_irqsave(&domain->lock, flags);
>  	list_for_each_entry(info, &domain->devices, link)
>  		__iommu_flush_dev_iotlb(info, addr, mask);
> +
> +	list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain)
> {
> +		info =3D dev_iommu_priv_get(dev_pasid->dev);
> +		qi_flush_dev_iotlb_pasid(info->iommu,
> +					 PCI_DEVID(info->bus, info->devfn),
> +					 info->pfsid, dev_pasid->pasid,
> +					 info->ats_qdep, addr,
> +					 mask);
> +	}

Check info->ats_enabled instead of doing it blindly.

> +static void domain_flush_pasid_iotlb(struct intel_iommu *iommu,
> +				     struct dmar_domain *domain, u64 addr,
> +				     unsigned long npages, bool ih)
> +{
> +	u16 did =3D domain_id_iommu(domain, iommu);
> +	struct dev_pasid_info *dev_pasid;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&domain->lock, flags);
> +	list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain)
> +		qi_flush_piotlb(iommu, did, dev_pasid->pasid, addr, npages,
> ih);
> +
> +	if (!list_empty(&domain->devices))
> +		qi_flush_piotlb(iommu, did, IOMMU_NO_PASID, addr,
> npages, ih);

Old code doesn't have this empty list check. I'm not sure whether any
corner case might exist but if you do plan to add it it's better to put it
in a separate patch to allow bisect.

>  	spin_unlock_irqrestore(&domain->lock, flags);
>  }
>=20
> @@ -1492,7 +1533,7 @@ static void iommu_flush_iotlb_psi(struct
> intel_iommu *iommu,
>  		ih =3D 1 << 6;
>=20
>  	if (domain->use_first_level) {
> -		qi_flush_piotlb(iommu, did, IOMMU_NO_PASID, addr, pages,
> ih);
> +		domain_flush_pasid_iotlb(iommu, domain, addr, pages, ih);
>  	} else {
>  		unsigned long bitmask =3D aligned_pages - 1;
>=20

Why cannot this pasid be used with a second level config?

> @@ -4720,25 +4762,99 @@ static void intel_iommu_iotlb_sync_map(struct
> iommu_domain *domain,
>  static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t
> pasid)
>  {
>  	struct intel_iommu *iommu =3D device_to_iommu(dev, NULL, NULL);
> +	struct dev_pasid_info *curr, *dev_pasid =3D NULL;
> +	struct dmar_domain *dmar_domain;
>  	struct iommu_domain *domain;
> +	unsigned long flags;
>=20
> -	/* Domain type specific cleanup: */
>  	domain =3D iommu_get_domain_for_dev_pasid(dev, pasid, 0);
> -	if (domain) {
> -		switch (domain->type) {
> -		case IOMMU_DOMAIN_SVA:
> -			intel_svm_remove_dev_pasid(dev, pasid);
> -			break;
> -		default:
> -			/* should never reach here */
> -			WARN_ON(1);
> +	if (!domain)
> +		goto out_tear_down;
> +
> +	/*
> +	 * The SVA implementation needs to stop mm notification, drain the
> +	 * pending page fault requests before tearing down the pasid entry.
> +	 * The VT-d spec (section 6.2.3.1) also recommends that software
> +	 * could use a reserved domain id for all first-only and pass-through
> +	 * translations. Hence there's no need to call
> domain_detach_iommu()
> +	 * in the sva domain case.
> +	 */

It's probably clearer to say:

/*
 * SVA domain requires special treatment before tearing down the pasid
 * entry:
 *   1) pasid is stored in mm instead of in dev_pasid;
 *   2) all SVA domains share a reserved domain id per recommendation
 *      from VT-d spec (section 6.2.3.1) so domain_detach_iommu() is
 *      not required;
 *   3) additional cleanup is required e.g. stopping mm notification,
 *      draining the pending page fault requests, etc.
 * Better handle it in a separate helper.
 */

>=20
> +static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
> +				     struct device *dev, ioasid_t pasid)
> +{
> +	struct device_domain_info *info =3D dev_iommu_priv_get(dev);
> +	struct dmar_domain *dmar_domain =3D to_dmar_domain(domain);
> +	struct intel_iommu *iommu =3D info->iommu;
> +	struct dev_pasid_info *dev_pasid;
> +	unsigned long flags;
> +	int ret;
> +
> +	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
> +		return -EOPNOTSUPP;
> +
> +	if (context_copied(iommu, info->bus, info->devfn))
> +		return -EBUSY;
> +
> +	ret =3D prepare_domain_attach_device(domain, dev);
> +	if (ret)
> +		return ret;
> +
> +	dev_pasid =3D kzalloc(sizeof(*dev_pasid), GFP_KERNEL);
> +	if (!dev_pasid)
> +		return -ENOMEM;

should it check whether this pasid has been attached?

> +
> +	ret =3D domain_attach_iommu(dmar_domain, iommu);
> +	if (ret)
> +		goto out_free;
> +
> +	if (domain_type_is_si(dmar_domain))
> +		ret =3D intel_pasid_setup_pass_through(iommu, dmar_domain,
> +						     dev, pasid);
> +	else if (dmar_domain->use_first_level)
> +		ret =3D domain_setup_first_level(iommu, dmar_domain,
> +					       dev, pasid);
> +	else
> +		ret =3D intel_pasid_setup_second_level(iommu, dmar_domain,
> +						     dev, pasid);

Here you allow attaching pasid to a domain using second-level but all
prior changes are only for first-level.