Received-SPF: pass (google.com: domain of linux-kernel+bounces-179659-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Lu Baolu <baolu.lu@linux.intel.com>, Jason Gunthorpe <jgg@ziepe.ca>, Joerg
 Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>, Robin Murphy
	<robin.murphy@arm.com>, Jean-Philippe Brucker <jean-philippe@linaro.org>,
	Nicolin Chen <nicolinc@nvidia.com>, "Liu, Yi L" <yi.l.liu@intel.com>, "Jacob
 Pan" <jacob.jun.pan@linux.intel.com>, Joel Granados <j.granados@samsung.com>
CC: "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"virtualization@lists.linux-foundation.org"
	<virtualization@lists.linux-foundation.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v5 5/9] iommufd: Add iommufd fault object
Thread-Topic: [PATCH v5 5/9] iommufd: Add iommufd fault object
Thread-Index: AQHamw8IPv7IV85GSUGxhOe1hMt7NbGYATUQ
Date: Wed, 15 May 2024 08:37:09 +0000
Message-ID: <BN9PR11MB52769AC595B6BDA8FB4639258CEC2@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240430145710.68112-1-baolu.lu@linux.intel.com>
 <20240430145710.68112-6-baolu.lu@linux.intel.com>
In-Reply-To: <20240430145710.68112-6-baolu.lu@linux.intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?p5h8ycVSS3JXPZ8tnSBfiqIVHEVDEumfskriFCHRuobti6c+el2+MNyYMlJt?=
 =?us-ascii?Q?6nD0p89b+b0XZneLY9EEnHPz/X/DdZ3/QGkv6Y2Z4pwvb2rjp2UElVShqn1l?=
 =?us-ascii?Q?IBrrqg7pgWXrPZpdNmVatbCU3bVL7nn3EkMY/bn1eE/qYwB/Z7PQZwQjTJXy?=
 =?us-ascii?Q?ScqVo5K4IMwMEwZ5Ja1UzLndHsbVgJ0RWke9GlQIBTwGQ3IWTfXKo1gjU45u?=
 =?us-ascii?Q?O/Qcf+lVHXsORPsRyYuCbHadikKIRnQzrrLZ5XgEx70g6g6Xeriy7yXABQKY?=
 =?us-ascii?Q?t/6LleAytqQlJ1ePdPWsFV9aOI3c3+10NL6p/v/6UrrZwRkNhSqSb1xzG1oY?=
 =?us-ascii?Q?k/SmJzcL/ost51fI9zrxdmhAMu27goH6ZxFUifks1zskv2Dm0WY49Hp8PeVr?=
 =?us-ascii?Q?kDrQB/Flrxmvo5f9h2HVBGVPN2h39ew7UDDciYbfGHa7egHBy98U6RppMaLr?=
 =?us-ascii?Q?b7Y/wOPIt7pTC5PYq5DPh4sG6bWJFUbT07Yp7AbhYF3nQuFKzFp3AIxeeoe+?=
 =?us-ascii?Q?nyVGfNBKiZo6+wqeJi3ySFce3kLa+t8mhm7hl0vGhwgwV+pq7FV4uKFQ6TUA?=
 =?us-ascii?Q?/m7ZzDNEqMFnQjwTDjmgFF6qk9mjiA+0yB/YZA6Gsrf9DHYRfjCji/ORvIgu?=
 =?us-ascii?Q?OWLpI2d/jH7ZAM57d0wANxfKoKnnd7IsvyFXnLjqU1cXaa9eI09NQMCMCKhl?=
 =?us-ascii?Q?HnhmTTVWo1kbSPX1YV4eKIE/OhZ6wdjaNdv3oo+it1BxfAqhzh5oKCC+fUyQ?=
 =?us-ascii?Q?YRaD2pkKKNPkKaNQ7Grx/1rG57gH56nWntC1EOd6X8Ziao7H3EkmSJCDbd3D?=
 =?us-ascii?Q?80/EwcxRNOU5oRMTe5LLzSF5AYqmspTazzkLAaQLjqVtZHbTMEJ/oqkdgLGN?=
 =?us-ascii?Q?Z5i6kVTFKDFktjOXFBoPVgblupx/vDrdMTfYO46EXv+WlVyPD2ueZtu5qXch?=
 =?us-ascii?Q?m5Lzo8YchNk+Um1zsvn0VlyX/6QSUBel5nZVSxrMZdhzASG8dyBXZ/2QP/BW?=
 =?us-ascii?Q?1soCWufaGlfzsfbnKUsGGp1kwDAX9LFXr4nMP3OuE/+96BSRFki4yH2HGXdW?=
 =?us-ascii?Q?sdB41tkXe4Yi5n2sPQ8/xc0v4IXmmR2D6J/9WiPNMc//VqA+VKmRvRYVW0s3?=
 =?us-ascii?Q?K8Lj0yKPyqrY9OcXOshYIJIK4KNgaVfn29mTdnWYe5BQDcd07eNFK//0XE+q?=
 =?us-ascii?Q?JvZXR/8GvAzT5cA+boC4PLN8TYsPRFBkqv82tB3Sc3eJum055oR3fkfMDNhM?=
 =?us-ascii?Q?koyZdP6SFZ9PQyujCkNw6ZgYokhUq6RLZQ2bJDe1YStKBB8z3ePLX/Ogo1KG?=
 =?us-ascii?Q?/dTGgR20iJmfsgzL2bJdS1lhFlAJ2eyJ88FZMSW2LrT5so6tgdCU1vTcwkWR?=
 =?us-ascii?Q?2yRYFauqP3YJAk1Q8U1cpkPM+eWlQvAAWkglv9/rRP1LmfK0vzgkPLFuy1k9?=
 =?us-ascii?Q?bWeF8sYRfyWKvrePF4PQfrU9N7c06je8zG4z8wSEDvL5MMjC6RjePWQQNzeV?=
 =?us-ascii?Q?kUbAJAUb+5yZSVQkYW0XMr1x7tqLWq6K6ZJIf9/NkuItqOJpVYOvAaxeL6ZS?=
 =?us-ascii?Q?zszd/uKpFyIPknFNlc0Iz+5+La+EQ3RfkAxwB2Yu?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 86f13714-753e-418b-b9d8-08dc74ba3603
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 May 2024 08:37:09.8484
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: m1MmMYVmsvtBEvdJ87Azmqkoh+WJsQgR1NzmFVlAKuPh71mPEa3pRIsPyHDOpXj/vBy6SBUw2WxJQn8Of0ExEw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR11MB8405
X-OriginatorOrg: intel.com

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Tuesday, April 30, 2024 10:57 PM
>=20
> @@ -131,6 +131,9 @@ struct iopf_group {
>  	struct iommu_attach_handle *attach_handle;
>  	/* The device's fault data parameter. */
>  	struct iommu_fault_param *fault_param;
> +	/* Used by handler provider to hook the group on its own lists. */
> +	struct list_head node;
> +	u32 cookie;

better put together with attach_handle.

rename 'node' to 'handle_node'

> @@ -128,6 +128,7 @@ enum iommufd_object_type {
>  	IOMMUFD_OBJ_HWPT_NESTED,
>  	IOMMUFD_OBJ_IOAS,
>  	IOMMUFD_OBJ_ACCESS,
> +	IOMMUFD_OBJ_FAULT,

Agree with Jason that 'FAULT_QUEUE' sounds a clearer object name.

> @@ -395,6 +396,8 @@ struct iommufd_device {
>  	/* always the physical device */
>  	struct device *dev;
>  	bool enforce_cache_coherency;
> +	/* outstanding faults awaiting response indexed by fault group id */
> +	struct xarray faults;

this...

> +struct iommufd_fault {
> +	struct iommufd_object obj;
> +	struct iommufd_ctx *ictx;
> +	struct file *filep;
> +
> +	/* The lists of outstanding faults protected by below mutex. */
> +	struct mutex mutex;
> +	struct list_head deliver;
> +	struct list_head response;

..and here worth a discussion.

First the response list is not used. If continuing the choice of queueing
faults per device it should be removed.

But I wonder whether it makes more sense to keep this response
queue per fault object. sounds simpler to me.

Also it's unclear why we need the response message to carry the
same info as the request while only id/code/cookie are used.

+struct iommu_hwpt_page_response {
+	__u32 size;
+	__u32 flags;
+	__u32 dev_id;
+	__u32 pasid;
+	__u32 grpid;
+	__u32 code;
+	__u32 cookie;
+	__u32 reserved;
+};

If we keep the response queue in the fault object, the response message
only needs to carry size/flags/code/cookie and cookie can identify the
pending message uniquely in the response queue.

> +static ssize_t iommufd_fault_fops_write(struct file *filep, const char _=
_user
> *buf,
> +					size_t count, loff_t *ppos)
> +{
> +	size_t response_size =3D sizeof(struct iommu_hwpt_page_response);
> +	struct iommufd_fault *fault =3D filep->private_data;
> +	struct iommu_hwpt_page_response response;
> +	struct iommufd_device *idev =3D NULL;
> +	struct iopf_group *group;
> +	size_t done =3D 0;
> +	int rc;
> +
> +	if (*ppos || count % response_size)
> +		return -ESPIPE;
> +
> +	mutex_lock(&fault->mutex);
> +	while (count > done) {
> +		rc =3D copy_from_user(&response, buf + done, response_size);
> +		if (rc)
> +			break;
> +
> +		if (!idev || idev->obj.id !=3D response.dev_id)
> +			idev =3D container_of(iommufd_get_object(fault->ictx,
> +							       response.dev_id,
> +
> IOMMUFD_OBJ_DEVICE),
> +					    struct iommufd_device, obj);
> +		if (IS_ERR(idev))
> +			break;
> +
> +		group =3D xa_erase(&idev->faults, response.cookie);
> +		if (!group)
> +			break;

is 'continue' better?

> +
> +		iopf_group_response(group, response.code);

PCIe spec states that a response failure disables the PRI interface. For SR=
-IOV
it'd be dangerous allowing user to trigger such code to VF to close the ent=
ire
shared PRI interface.

Just another example lacking of coordination for shared capabilities betwee=
n
PF/VF. But exposing such gap to userspace makes it worse.

I guess we don't want to make this work depending on that cleanup. The
minimal correct thing is to disallow attaching VF to a fault-capable hwpt
with a note here that once we turn on support for VF the response failure
code should not be forwarded to the hardware. Instead it's an indication
that the user cannot serve more requests and such situation waits for
a vPRI reset to recover.

> +		iopf_free_group(group);
> +		done +=3D response_size;
> +
> +		iommufd_put_object(fault->ictx, &idev->obj);

get/put is unpaired:

		if (!idev || idev->obj.id !=3D response.dev_id)
			idev =3D iommufd_get_object();

		...

		iommufd_put_object(idev);

The intention might be reusing idev if multiple fault responses are
for a same idev. But idev is always put in each iteration then following
messages will access the idev w/o holding the reference.