From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Lu Baolu <baolu.lu@linux.intel.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "dmaengine@vger.kernel.org" <dmaengine@vger.kernel.org>
CC:     Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        "Robin Murphy" <robin.murphy@arm.com>,
        "Yu, Fenghua" <fenghua.yu@intel.com>,
        "Jiang, Dave" <dave.jiang@intel.com>,
        Vinod Koul <vkoul@kernel.org>,
        Jacob Pan <jacob.jun.pan@linux.intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2 5/5] iommu/vt-d: Move PRI handling to IOPF feature path
Thread-Topic: [PATCH v2 5/5] iommu/vt-d: Move PRI handling to IOPF feature
 path
Thread-Index: AQHZUjL48KR1g1UW3UyUdTfzF/rwfa79CJVA
Date:   Thu, 16 Mar 2023 07:17:06 +0000
Message-ID: <BN9PR11MB52766D4948FA00CB02D3F2188CBC9@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20230309025639.26109-1-baolu.lu@linux.intel.com>
 <20230309025639.26109-6-baolu.lu@linux.intel.com>
In-Reply-To: <20230309025639.26109-6-baolu.lu@linux.intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?J8cG86p9FrIKImf8HghI0HZBmb+flkjj66PCzZSldplWeyFbRIWKdioTCH99?=
 =?us-ascii?Q?u5eFkDRjzOFgnhpeDtMC41534yxXj8ckaaUVffZfRmDtRej12tH1Tg8fHysG?=
 =?us-ascii?Q?Mtm6gpYtErD/7a5B4+S/NQ6q6wDPRdEBIvseYR4+m75O/SrZ5oZEjHJf1gka?=
 =?us-ascii?Q?I3zIx4L7Pui3Wgd3fbYnnkqTG7ZuSEI7jY7pyE1gAA32yqZ764wFxEgZpUTk?=
 =?us-ascii?Q?L17YT3heKJsqI3+MFcHFQK7PfoAgDm8WTMAGv+lR4RhTzGJGDGNQm1ULlK2Q?=
 =?us-ascii?Q?3VCWmjDVawrm1LCfn6JwJy/EMQN4Pv+buEKaZ16VC8m4POYv1jRwZ0nnJMJD?=
 =?us-ascii?Q?VjiJqUNd7NjZSi2iEZtlfeTAtdWGtSit3e/BgwAnNe7AxrOpK7Q07UuGv8zl?=
 =?us-ascii?Q?XumW07D7bt0m60+eENiBXEIXlOU9zGKTQHJQnQB05ZL40Bw6heOCmHUdJ1K3?=
 =?us-ascii?Q?kD1SQDTXFSVTac48j5n5HAeDf7oGPY4tnHor9SUUTYFkrg9/I+ylPPr1mkOt?=
 =?us-ascii?Q?3NX0c2DdE2N9/8UPXeAqkuIBM8uo+4JUo8MArLyI/cT+P4OZMNrIVd7pOfaR?=
 =?us-ascii?Q?ed0oVn9h9V3zYCsi3jGqA79yrrXzM1cxfUmOJ9MX6cYfHuuGVJior3fUIWq/?=
 =?us-ascii?Q?fiTqZUi7T3G7iRmDvMO5AD8b+Z3H4+FUcK2kJPwNnGPXa1+tKw8619d31OwK?=
 =?us-ascii?Q?uvGdzTEzFCHzYS9wnI1XS56CjGhLWXg6/25b2iVoHC4J6EHsxJntAyuZKOqw?=
 =?us-ascii?Q?iGKWZJh1ujGIp73OLX5Bbg4R/eq8eI7/ZGm0CXaPRojfBjsuP0dxGiHqWATC?=
 =?us-ascii?Q?XhjrAQ/CUlMaOCjMaBDe0NG1fbN3+EMLtW1SW65HXbRDx+mr/uUOrCenO+cj?=
 =?us-ascii?Q?II/+0XwzXYq0DZDJiqB4HMvfLQ82GbtYYWyRBYezDlUCkykc/WF7/AvTuh7r?=
 =?us-ascii?Q?TAFbKmwpdwNLi8wo8RnfZWkqv8rX0HbmiX1aQkOdQIc0RZmNpNsoFZF69Tnb?=
 =?us-ascii?Q?FuJnw/X9oFoByCHrF8zWZ0JTTpvvYn/8fx3LApz779DnEAJTa8EjboUKTpbE?=
 =?us-ascii?Q?WqZc7ImRF43bKIGlW6uCiLXuH34bojvfB+rlkLOULla2V+gyBSK43cDSmwlX?=
 =?us-ascii?Q?I0OYpx2FkwYVqQlPEnLlL24y8dc52s//Z3G4uJqYLsKaiaM+YrbRG59QmosL?=
 =?us-ascii?Q?6VQWcmMV22U4+Pm3wiufteIUjVw2BppnnnxpfDEe7IRcECUoo3Wk5aKacQtS?=
 =?us-ascii?Q?hJJC8MgNyEoZWwX8M1n+XrVwimtpLidoACqh+0EDVP2wkyT1k4OoF3JqSAqE?=
 =?us-ascii?Q?FWb92pZlGU6hZbOj7yfCf0X+w/W9W9Lp3eR1tW8GgGSWkP0CuY3TGPnukwIY?=
 =?us-ascii?Q?uvgZY8xzo/XykcPNUFrtKn9JjVN1EijsCuXBgboXB+SWFaWhyJnwh5oIth3o?=
 =?us-ascii?Q?ew7XPp4B+r3MTkbZquzJV19wTuYmcseUWYEK2NRK7X0E+n0uK+7YknIhvv2b?=
 =?us-ascii?Q?0iOGxuMQjNKrw77dfGF0Lg7WUqjfaLg3R6ARZ6ot2M+WqAcbg8+QxHzGgYVB?=
 =?us-ascii?Q?qwtg/WLCS2wKee9vHHrYLOG/gf8LaMCmGmRHE0WC?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 17265fb2-5cf8-4e78-e8ec-08db25ee72cd
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Mar 2023 07:17:06.1129
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: NCe04xtdC8bZOtYukKbFTV/ljVpYG+/IAe4dsRPCJEJ9cQxRWJMQq6aytsvdTRCW2/GasahIk9TI4TwFERBuvw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB8356
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Thursday, March 9, 2023 10:57 AM
>=20
> @@ -4689,17 +4704,21 @@ static int intel_iommu_disable_iopf(struct device
> *dev)
>  {
>  	struct device_domain_info *info =3D dev_iommu_priv_get(dev);
>  	struct intel_iommu *iommu =3D info->iommu;
> -	int ret;
>=20
> -	ret =3D iommu_unregister_device_fault_handler(dev);
> -	if (ret)
> -		return ret;
> +	if (!info->pri_enabled)
> +		return -EINVAL;
>=20
> -	ret =3D iopf_queue_remove_device(iommu->iopf_queue, dev);
> -	if (ret)
> -		iommu_register_device_fault_handler(dev,
> iommu_queue_iopf, dev);
> +	pci_disable_pri(to_pci_dev(dev));
> +	info->pri_enabled =3D 0;
>=20
> -	return ret;
> +	/*
> +	 * With pri_enabled checked, unregistering fault handler and
> +	 * removing device from iopf queue should never fail.
> +	 */
> +	iommu_unregister_device_fault_handler(dev);
> +	iopf_queue_remove_device(iommu->iopf_queue, dev);
> +
> +	return 0;
>  }

PCIe spec says that clearing the enable bit doesn't mean in-fly
page requests are completed:
--
Enable (E) - This field, when set, indicates that the Page Request
Interface is allowed to make page requests. If this field is Clear,
the Page Request Interface is not allowed to issue page requests.
If both this field and the Stopped field are Clear, then the Page
Request Interface will not issue new page requests, but has
outstanding page requests that have been transmitted or are
queued for transmission