Received-SPF: pass (google.com: domain of linux-kernel+bounces-52827-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Mon, 5 Feb 2024 10:32:31 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Baolu Lu <baolu.lu@linux.intel.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>, Joerg Roedel <joro@8bytes.org>,
	Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
	Jean-Philippe Brucker <jean-philippe@linaro.org>,
	Nicolin Chen <nicolinc@nvidia.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	Jacob Pan <jacob.jun.pan@linux.intel.com>,
	Longfang Liu <liulongfang@huawei.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	Joel Granados <j.granados@samsung.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v11 13/16] iommu: Improve iopf_queue_remove_device()
Message-ID: <20240205143231.GB10476@nvidia.com>
References: <20240130080835.58921-1-baolu.lu@linux.intel.com>
 <20240130080835.58921-14-baolu.lu@linux.intel.com>
 <BN9PR11MB5276E70CAB272B212977F0C98C472@BN9PR11MB5276.namprd11.prod.outlook.com>
 <416b19fa-bc7a-4ffd-a4c4-9440483fc039@linux.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <416b19fa-bc7a-4ffd-a4c4-9440483fc039@linux.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?T9qROq3e6Dv2KH69k/FPN1u11dGlWdEC8ZtUsuAqmIjkL4bEAa/5B2jH8rQ1?=
 =?us-ascii?Q?2nx31LHCj1wSRJTylN9YWKFA+aeab/Q7XiBNKMi9HVNMWsXqve965oTA6Yfo?=
 =?us-ascii?Q?Lq6J6cuHPCZKLTms5lIzu0SI/8j8xQoavAZAJFtUpHyl+TEYs/FwYArdK0vF?=
 =?us-ascii?Q?Gw2WyqC6whxAyezX5HucTeRbGD9F9YkNfCXCWhfKzuWKgYZmWkp4BiyuPSvj?=
 =?us-ascii?Q?DGXfDK3x+asaxaJamJQ34kPGV/qQsenxDOz+2KMAYuSVmnIsiBr57sc8YhUO?=
 =?us-ascii?Q?3bnZQcttD4rjH7QFyn626orX5497YKWGsiZJuhpyk68MJauH5GYXZ22T6GeU?=
 =?us-ascii?Q?tlqs5ia4a9p5zLlVQoeHqdx4ePqt6mUqc+U86NEpiVF4GWbUHGQnDa7bqPSu?=
 =?us-ascii?Q?jsYDaOcZarKH0iFjBPKdmfSBEpBapjQZU71KW0A4I2HW9AY9nGYkfc5ZFOp/?=
 =?us-ascii?Q?LAm4TnHSdeee9ZQVthXeapmzfK0ZG409sMu7dCXgwl3Ze7ZNtIoBvxsRSHHQ?=
 =?us-ascii?Q?MVolazPzC7rWetqVXz7Hbltb3t3Etp0NeAMnTL4mPiECmEHg4fbukHjf+EDQ?=
 =?us-ascii?Q?wU/aGxDPqEozK6Ow5er3WkjBMveO+3RpLudNWNujoiTjqpRZ1T6ovJwiIzlA?=
 =?us-ascii?Q?diYe+fZJdlqWZE4k1Q4PqJJj3HhQVFRmHq7xye7KIIp9LH+oAmbO70ARpdRN?=
 =?us-ascii?Q?HeVZpPO4nZRwPNBXC5yXGUn9GCRZlzPxXJsUsqr9yrhSL/xDJgE7lSQmzHgs?=
 =?us-ascii?Q?775VZ8JZdDbgazitaoRxSVJH1vOWRxUctUKyPeqwQT+QVJ9ph+MqrjTk6YlR?=
 =?us-ascii?Q?fexb70ZEezmmNmau670brzFGzMnNvcEegrGGHUjxUd2h9LE1+zdeetng1VEH?=
 =?us-ascii?Q?fuXVG/11FP/0LlxtD7Oj1nluKfNjJNlbSVPkZA9c3bAiv8vgw6nt08VbPOyA?=
 =?us-ascii?Q?IklAo6LJvreDeZ7ub60rczyUAL+7PfkX5KM5sQOvKQLRnTGvFWsEG4RYbNFN?=
 =?us-ascii?Q?2/b9OSkQQqVHM5Z4xpFWVweSvo+5bU+kFp/m21jFjg229Iu/uesOeuHgUpJ3?=
 =?us-ascii?Q?+/g48xMK0j+XODh8MAIIZcy5KQR8YvgWZsBjYxUxLiMdsZWGaGIJTVfUA5WB?=
 =?us-ascii?Q?fazuefJ83bzM/o205rUsjuloBjdMiyYKboIBGmdvoMTp9m8CQVSDeTuwgJns?=
 =?us-ascii?Q?Gnf+vrSljS8Sqb19pi/UIIy8InJJJGzZFeDLZd9g0ljN9HwyvfJTnLuVVe56?=
 =?us-ascii?Q?3XKpXTeKuC/KP0/IEvxQfh21MeiEx8P4UaNhRhNGJM/JHDJUBtDE7kltNbLa?=
 =?us-ascii?Q?vpSNiVNhVFFNCIAGyIRPiix9NE/4EjQCnNthPyoCAbxyDs7OBJ0ak/EWp9Hz?=
 =?us-ascii?Q?HAHm19bkIisKuNBOTM8JTabPlwLyUlUqpLLvyPqHhHCeAB9bUAUlcg/Uiazr?=
 =?us-ascii?Q?i8M0ixmdlsrEc1dxsJl+4Fr8eAOPx3k7n1ZUOTT947yiwscXYVxpx2DbwXo+?=
 =?us-ascii?Q?1NKpIy34phMUwIRGCP9iGBWNdXQqX6ErccGo2XNyZ8Z4vyjPKMMSIHHWkAwu?=
 =?us-ascii?Q?KeIUB8QeYi+Pj1yYgN4=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8392a240-220c-4879-af51-08dc265749d9
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2024 14:32:32.3881
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oB2b+tTXf3IAZLrWOz46H1oUf07xB+wn0OxtcazKSGt5GZGiwtp26DILwdjRmeAx
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4440

On Mon, Feb 05, 2024 at 07:55:23PM +0800, Baolu Lu wrote:
> On 2024/2/5 17:00, Tian, Kevin wrote:
> > > From: Lu Baolu <baolu.lu@linux.intel.com>
> > > Sent: Tuesday, January 30, 2024 4:09 PM
> > >    *
> > > - * Caller makes sure that no more faults are reported for this device.
> > > + * Removing a device from an iopf_queue. It's recommended to follow
> > > these
> > > + * steps when removing a device:
> > >    *
> > > - * Return: 0 on success and <0 on error.
> > > + * - Disable new PRI reception: Turn off PRI generation in the IOMMU
> > > hardware
> > > + *   and flush any hardware page request queues. This should be done
> > > before
> > > + *   calling into this helper.
> > 
> > this 1st step is already not followed by intel-iommu driver. The Page
> > Request Enable (PRE) bit is set in the context entry when a device
> > is attached to the default domain and cleared only in
> > intel_iommu_release_device().
> > 
> > but iopf_queue_remove_device() is called when IOMMU_DEV_FEAT_IOPF
> > is disabled e.g. when idxd driver is unbound from the device.
> > 
> > so the order is already violated.
> > 
> > > + * - Acknowledge all outstanding PRQs to the device: Respond to all
> > > outstanding
> > > + *   page requests with IOMMU_PAGE_RESP_INVALID, indicating the device
> > > should
> > > + *   not retry. This helper function handles this.
> > > + * - Disable PRI on the device: After calling this helper, the caller could
> > > + *   then disable PRI on the device.
> > 
> > intel_iommu_disable_iopf() disables PRI cap before calling this helper.
> 
> You are right. The individual drivers should be adjusted accordingly in
> separated patches. Here we just define the expected behaviors of the
> individual iommu driver from the core's perspective.

Yeah, I don't think the driver really works properly before this
documentation was added either :\

We also need to check that the proposed AMD patches (SVA support part
4) are working right before they are merged.

Jason