From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Nicolin Chen <nicolinc@nvidia.com>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     "joro@8bytes.org" <joro@8bytes.org>,
        "will@kernel.org" <will@kernel.org>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
Subject: RE: [PATCH v2 08/10] iommufd/device: Use iommu_group_replace_domain()
Thread-Topic: [PATCH v2 08/10] iommufd/device: Use
 iommu_group_replace_domain()
Thread-Index: AQHZOzoV+UAbwLQwnEuO17sJd5b+0K7F+NzggAEnS4CAAE2/cIABdh8AgANLy2CAAFjcAIAAdaUAgAFQnACAAPZWYA==
Date:   Wed, 15 Feb 2023 01:37:19 +0000
Message-ID: <BN9PR11MB5276CBCF780BD7C61154BF8B8CA39@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <cover.1675802050.git.nicolinc@nvidia.com>
 <4653f009c3dacae8ebf3a4865aaa944aa9c7cc7e.1675802050.git.nicolinc@nvidia.com>
 <BN9PR11MB5276C1807B710CAD3E5820D78CD99@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+Vh479cDD7LX2x/@Asurada-Nvidia>
 <BN9PR11MB5276268D3ED0360913A05C368CDE9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+bc8OrWfw3Fq57n@Asurada-Nvidia>
 <BL1PR11MB5271180A1DE8C739455AD06D8CDD9@BL1PR11MB5271.namprd11.prod.outlook.com>
 <Y+nrTsIz2UHH/1lD@Asurada-Nvidia> <Y+pN/vtO8I/oiMOJ@nvidia.com>
 <Y+toXAUa2Cbwn7I0@Asurada-Nvidia>
In-Reply-To: <Y+toXAUa2Cbwn7I0@Asurada-Nvidia>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?oi9aiQUwtUTD4m9f9Fv+GXYAlveN7kL0iSet0sTy39A8J6cQyd9GiT5c8U1U?=
 =?us-ascii?Q?I4WgQOPsm+QZW0PZr/c8x1BYaFKKakxDPN8YYYpqy83NojNXa1fntlNmgCdy?=
 =?us-ascii?Q?k422kZDw7cQMm+xSyvIgiwACK9dZWYuS2SGw+4B5OpprrbR2JBEHZnS+nGKR?=
 =?us-ascii?Q?tZiIJs+6qhMaOPDrY28Q6baG63Z5/OB0ZUH2ynV4G/aHx2Z/2mX5M0bBj/2p?=
 =?us-ascii?Q?TFk++Br9xrDy6yE7hEqAD6cEotssfTQxrmz6x5Dw0wqcUjWY0rCKj56wcDlu?=
 =?us-ascii?Q?XpPp9o77dOf9JAydAmzY5Hm4SWD4ZbdjX+a6M5x6sGt1vWvkXlMMgbzo5jxY?=
 =?us-ascii?Q?Z+CM3GKzn6dex8du55g6tiy1HQnhkBiaRYBSoWUvNI4twDOrrRCkIePlu1tj?=
 =?us-ascii?Q?i/KSs7aM9y9H+wOT3UIox7ABXwIezfciUUSVoh8cGNUpizObhm1WewpN5NCX?=
 =?us-ascii?Q?nkWXSHCQTvYpsk2WeAmtT+y3j2rxhM4gX+I3aMLA/T6ppn7lu0Pft0a1fRti?=
 =?us-ascii?Q?VScJxacCRxuqPVYbLfguLLcmA9i9Flpir3EJxQhCEaaDr7EAE1QhkfVZPgGl?=
 =?us-ascii?Q?FoRtnL/3AEZpvSNYnao15tqeUDyER8bfZJ5A692Ro5GNf9AUCzlj7+U2LhkJ?=
 =?us-ascii?Q?Y1PJsYR/bWoEXoTCQlLqJEuTvjtOSJxIsm3KLLMB7Z7m0VVpMo9rDMqCWRQZ?=
 =?us-ascii?Q?8x0fJ+lxloVidoKgLvHcPicAlquEHKXMvQcRHUgqvaSObbbzFmho0B9FhliV?=
 =?us-ascii?Q?2njZOyZB0kvi5njt6kNo8bW5+aTXTtr5+A4NJIXH803bHayP36CgfRVP9yUE?=
 =?us-ascii?Q?a2e1letvyHssdC3sF2KgCV5NUh3cFCMNt729dcpYt9IpcMlvArHmTi3Bdldk?=
 =?us-ascii?Q?9jeLfuI2Csr6yck7FFLBg0v7I5bUrTQn2oxR5LW/yecKZpqpDPzr7LZHU1oE?=
 =?us-ascii?Q?AC9xBXitmcF6kwYj1ZrIoIbOj0SEzKOiyXa3EufIFMVNCQIEJvmhNp1+lyIE?=
 =?us-ascii?Q?rnINwbpsDQL5IkqYLi1ssOrUMZ73S0QL0mxvtqqcoDwQADfxVy4tnJ0JDsB9?=
 =?us-ascii?Q?cS3Y3CFCAwptjHuVePb8l2XVkIhp3mrohnzkWtEmeuOGmo0iEC62kmoz8Q5i?=
 =?us-ascii?Q?1/H1jVRx1qZgAqcmE4VX/1nNCZfuUFeLaScOBR9TaPv7sAwt3ktYDqdurw9F?=
 =?us-ascii?Q?JWWhn9dMLPLP3EH1rkss7UoT5+SGrrYhBPH6ILKmyXby8RzncYY6Na3XGZJP?=
 =?us-ascii?Q?1sLI8bPJDWjORojyCkiSEKLO9EYq3mrgn8zoEIy6kQ/0F+KDrsuwnw4R5OeU?=
 =?us-ascii?Q?vMPLOQnAMNzUz8CuPHNE0W/Y8sj28oXpjVn7t/Lip27onFX17APfwTkJK+/b?=
 =?us-ascii?Q?3ipiKGMlDm6SImCYOsboAklaPEqhOZ1PDIdeDwCSTD5Xlptt8DbMLIxvCvmX?=
 =?us-ascii?Q?wd5vYfGyM+DLTUBW3fd7t0lkC/VOKeiYXYdJSCK2+fw41vZkFav+yPqJ8fep?=
 =?us-ascii?Q?hzYSjBmOXpeBHwoHgyGC5Jq3R1qn++FmGSR/VquWnTukiUKQBvPT7vr/JXjb?=
 =?us-ascii?Q?Rf3sjH1T/vRLcWTZtW48K1QLveGJBOyhcD7mJlAS?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 549d0300-130a-48b0-c592-08db0ef52dbb
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Feb 2023 01:37:19.9488
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RN4yCimyKZC0ow59ra2gr9E4eL8xq9eE46ENEiDQEiJv/BscMFQ+Qp/uZbGuMFT97GLOBbNtitnbbIHGvVwF1Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7459
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Tuesday, February 14, 2023 6:54 PM
>=20
> On Mon, Feb 13, 2023 at 10:49:34AM -0400, Jason Gunthorpe wrote:
> > On Sun, Feb 12, 2023 at 11:48:30PM -0800, Nicolin Chen wrote:
> >
> > > What about point 1? If dev2 and dev3 are already replaced when
> > > doing iommu_group_replace_domain() on dev1, their idev objects
> > > still have the old hwpt/iopt until user space does another two
> > > IOCTLs on them, right?
> >
> > We have a complicated model for multi-device groups...
> >
> > The first device in the group to change domains must move all the
> > devices in the group
> >
> > But userspace is still expected to run through and change all the
> > other devices
> >
> > So replace should be a NOP if the group is already linked to the right
> > domain.
> >
> > This patch needs to make sure that incosistency in the view betwen the
> > iommu_group and the iommufd model doesn't cause a functional
> > problem.
>=20
> Yea, I was thinking that we'd need to block any access to the
> idev->hwpt of a pending device's, before the kernel finishes
> the "NOP" IOCTL from userspace, maybe with a helper:
> 	(iommu_get_domain_for_dev(idev->dev) !=3D idev->hwpt->domain)
>=20

I didn't see what would be broken w/o such blocking measure.

Can you elaborate?