From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
CC:     Jason Gunthorpe <jgg@nvidia.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "will@kernel.org" <will@kernel.org>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "shuah@kernel.org" <shuah@kernel.org>,
        "Liu, Yi L" <yi.l.liu@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>
Subject: RE: [PATCH v2 08/10] iommufd/device: Use iommu_group_replace_domain()
Thread-Topic: [PATCH v2 08/10] iommufd/device: Use
 iommu_group_replace_domain()
Thread-Index: AQHZOzoV+UAbwLQwnEuO17sJd5b+0K7F+NzggAEnS4CAAE2/cIABdh8AgANLy2CAAFjcAIAAdaUAgAFQnACAAPZWYIAABlAAgAACwKA=
Date:   Wed, 15 Feb 2023 02:15:59 +0000
Message-ID: <BN9PR11MB5276FA23E61977389D0C1A198CA39@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <4653f009c3dacae8ebf3a4865aaa944aa9c7cc7e.1675802050.git.nicolinc@nvidia.com>
 <BN9PR11MB5276C1807B710CAD3E5820D78CD99@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+Vh479cDD7LX2x/@Asurada-Nvidia>
 <BN9PR11MB5276268D3ED0360913A05C368CDE9@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+bc8OrWfw3Fq57n@Asurada-Nvidia>
 <BL1PR11MB5271180A1DE8C739455AD06D8CDD9@BL1PR11MB5271.namprd11.prod.outlook.com>
 <Y+nrTsIz2UHH/1lD@Asurada-Nvidia> <Y+pN/vtO8I/oiMOJ@nvidia.com>
 <Y+toXAUa2Cbwn7I0@Asurada-Nvidia>
 <BN9PR11MB5276CBCF780BD7C61154BF8B8CA39@BN9PR11MB5276.namprd11.prod.outlook.com>
 <Y+w8TJ2jfDATftbr@Asurada-Nvidia>
In-Reply-To: <Y+w8TJ2jfDATftbr@Asurada-Nvidia>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?qUsGKEbND7rgyZ4p+vRAE7/MV5iGOz/LNWZcLHkL5u+sA7UnC8tlLev1Ar/e?=
 =?us-ascii?Q?QLTQuB5o0ICxdAjsT13a4dUV/Hk9eoQjhLbA7dEAKQD5f013/9VHQZeJft2s?=
 =?us-ascii?Q?Rmlo4YuE6YWoIyOkCtDPjuAXSFfljW2iDZCBxIdQvJhqBF3EmW4wo8DWPtHy?=
 =?us-ascii?Q?BNogETLodYdgjhulUkqx67vBTG5qj9SFV0h4ihwoyxGBdPzGLNiVB12JktR6?=
 =?us-ascii?Q?t+cIx33q74iO8nJUlneQSJ89dZCtJy43jOMOHeXRcAKNb2ZuQnp3+mwN1xLy?=
 =?us-ascii?Q?IwjLNcGSqaEf9SK16uJYHiuWngoUVbNMwNJ/m8WCs97r9WDK++P0e2ibcCEa?=
 =?us-ascii?Q?KHG8hlMjA+Kytp5thejl9E7gZIHttMxNmp9FTF5iYU/JUWVcVXCBEFxbfJI5?=
 =?us-ascii?Q?F0y3bpn5651RZhBy5uJFDPBtvKPS5cRyCi9vYWpRzWlZMhtPBxd8T6aiIuaI?=
 =?us-ascii?Q?Ok8fK33woFH9Sdd0QW7Zu72oTWbYIDyYm6BAqDwG3hxf4P9etHqPgYS50CO5?=
 =?us-ascii?Q?7NxjA/wKG7MxjbbYxBMaWBfaT6wXQFizV1PdYwkOGMTBzE+FunyIekVT0AVv?=
 =?us-ascii?Q?lz/euHmNk1HUv8uGr/kIgcU43cdjfzw06lUy5ZWEnefr14DxSgKKz4Yh4sov?=
 =?us-ascii?Q?fiV2m3d5I7P6S1Zi2AsGGNMCUgP9OZP2QZcBXSXNuFskRic9dFPUzANyktlO?=
 =?us-ascii?Q?0/nYQHK3/lRfixSZOVjWJonkuyun+2WkuEm2ObV2sbaAbYHc38L49tJ0sVk9?=
 =?us-ascii?Q?x6w32v8le/b3F28ESRuLb3lSm8hTg9p6cogrc7NX+ST7szM6bYcQBhdVhTcO?=
 =?us-ascii?Q?NO/q1vx1X/0ygl7Xgh0UX1duzWeQzXCMMnTqJByorhGoym9CtEzphP6mRdyw?=
 =?us-ascii?Q?4nIwCxgojfyxuxBzzBnq85WKXgcl0CAt6M/4/7IRP5Jo7PMO0QeeEEWHdY2N?=
 =?us-ascii?Q?6IedC3PXbbMzkGJ88IGJkjODq/liptWXXf532DFQT9o1pcA7HNlox3fOMEqq?=
 =?us-ascii?Q?k4ClH5iLaGNjChqeHtMiNWXtFK91hP7Q9EDf/MUEs5Y5x309uQdCg+waGDd5?=
 =?us-ascii?Q?c8JnIFelh9kEirMlvgnVP5iJo8iFQfCPne+NVlKOc22O/RGfcHVmNbkX0nuX?=
 =?us-ascii?Q?nJCBNIjI0kgteIUqAxkI/2wKSfeqjZeb+sih15wTFMFGP0ne4ePg4jUnivEk?=
 =?us-ascii?Q?lseh0nPo37DTUy7/C4pqUga5XQNglQLNWwfhsBA7fmpYrh7q5Hh/uRuiDfG7?=
 =?us-ascii?Q?SmunwS1ukmQGPhqiz77qvUQQwwSCrItF0r57rPeQiY7jgVxtVe42y9k89y2d?=
 =?us-ascii?Q?iEjMXIcQ/wSAgPSuMSMJLr61/DG/IxATsUM+ujZbKW87TsBd9ljgv4vheTmK?=
 =?us-ascii?Q?2KG/UuYBtszoboyTmc5ptz4vaVABq+ULr3EeaelMuH2qe1cstAMcRmrrJLaF?=
 =?us-ascii?Q?yLJebSHGL+yaCfJf2tSFlZVkdjAjkk00GwUxLbD8et9sRtIhCoi1VH+wdaJM?=
 =?us-ascii?Q?EsNhKwmVVI4EOQlS6P4WduFxubREizZqAPzIdgS7OojSE5EMnYF5ayatOaIM?=
 =?us-ascii?Q?z076hyoOaUfAgLueG/VT9WdAEOn16lNsOWIk1582?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: eb39a4cd-e5aa-48da-f00d-08db0efa9432
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Feb 2023 02:15:59.3411
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 9UuUILihVQmj96JbMAGoc8CJaZmEGa8mMP70RVHzPkB0/zkHMs7XVj21fgu/C+HHeVIJb6TGQMGF0ne1vLrfug==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR11MB7732
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Wednesday, February 15, 2023 9:59 AM
>=20
> On Wed, Feb 15, 2023 at 01:37:19AM +0000, Tian, Kevin wrote:
>=20
> > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > Sent: Tuesday, February 14, 2023 6:54 PM
> > >
> > > On Mon, Feb 13, 2023 at 10:49:34AM -0400, Jason Gunthorpe wrote:
> > > > On Sun, Feb 12, 2023 at 11:48:30PM -0800, Nicolin Chen wrote:
> > > >
> > > > > What about point 1? If dev2 and dev3 are already replaced when
> > > > > doing iommu_group_replace_domain() on dev1, their idev objects
> > > > > still have the old hwpt/iopt until user space does another two
> > > > > IOCTLs on them, right?
> > > >
> > > > We have a complicated model for multi-device groups...
> > > >
> > > > The first device in the group to change domains must move all the
> > > > devices in the group
> > > >
> > > > But userspace is still expected to run through and change all the
> > > > other devices
> > > >
> > > > So replace should be a NOP if the group is already linked to the ri=
ght
> > > > domain.
> > > >
> > > > This patch needs to make sure that incosistency in the view betwen =
the
> > > > iommu_group and the iommufd model doesn't cause a functional
> > > > problem.
> > >
> > > Yea, I was thinking that we'd need to block any access to the
> > > idev->hwpt of a pending device's, before the kernel finishes
> > > the "NOP" IOCTL from userspace, maybe with a helper:
> > >       (iommu_get_domain_for_dev(idev->dev) !=3D idev->hwpt->domain)
> > >
> >
> > I didn't see what would be broken w/o such blocking measure.
> >
> > Can you elaborate?
>=20
> iommu_group { idev1, idev2 }
>=20
> (1) Attach all devices to domain1 {
> 	group->domain =3D domain1;
> 	idev1->hwpt =3D hwpt1; // domain1
> 	idev2->hwpt =3D hwpt1; // domain1
> }
>=20
> (2) Attach (replace) idev1 only to domain2 {
> 	group->domain =3D domain2
> 	idev1->hwpt =3D hwpt2; // domain2
> 	idev2->hwpt =3D=3D domain1 // !=3D iommu_get_domain_for_dev()
> }
>=20
> Then if user space isn't aware of these and continues to do
> IOMMU_IOAS_MAP for idev2. Though IOVA mappings may be added
> onto the domain2 correctly, yet domain2's iopt itree won't
> reflect that, until idev2->hwpt is updated too, right?

IOMMU_IOAS_MAP is for ioas instead of for device.

even w/o any device attached we still allow ioas map.

also note in case of domain1 still actively attached to idev3 in
another group, banning IOAS_MAP due to idev2 in transition
is also incorrect for idev3.

>=20
> And the tricky thing is that, though we advocate a device-
> centric uAPI, we'd still have to ask user space to align the
> devices in the same iommu_group, which is also not present
> in QEMU's RFC v3 series.

IMHO this requirement has been stated clearly from the start
when designing this device centric interface. QEMU should be
improved to take care of it.