Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   "Liu, Yi L" <yi.l.liu@intel.com>
To:     "Tian, Kevin" <kevin.tian@intel.com>,
        Nicolin Chen <nicolinc@nvidia.com>,
        Jason Gunthorpe <jgg@nvidia.com>
CC:     "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>
Subject: RE: [PATCH v4 09/12] iommu/vt-d: Add iotlb flush for nested domain
Thread-Topic: [PATCH v4 09/12] iommu/vt-d: Add iotlb flush for nested domain
Thread-Index: AQHZvh/uqwr3PK6FO0KhG3jtetteiK/WrrGAgAhNwpCAANOqgIAAnN2AgABV8ACAAPYtAIAABp4QgAADR4CAAAcG8A==
Date:   Wed, 9 Aug 2023 09:30:12 +0000
Message-ID: <DS0PR11MB7529C310FAEA61B6E7988629C312A@DS0PR11MB7529.namprd11.prod.outlook.com>
References: <20230724111335.107427-1-yi.l.liu@intel.com>
 <20230724111335.107427-10-yi.l.liu@intel.com>
 <BN9PR11MB527690EBAA872A16AE8926F88C0BA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <DS0PR11MB7529F4D4DABBE29E9B7BF5C9C30CA@DS0PR11MB7529.namprd11.prod.outlook.com>
 <ZNGypUee/U9d68TZ@Asurada-Nvidia> <ZNI2O4Upyna5AWDA@nvidia.com>
 <ZNJ+Uv/WJwngosjJ@Asurada-Nvidia>
 <BN9PR11MB5276BFFEC7E12EEBD4503BF08C12A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <DS0PR11MB7529C3646E38542457D7B75DC312A@DS0PR11MB7529.namprd11.prod.outlook.com>
 <BN9PR11MB5276912120F662498910A1D48C12A@BN9PR11MB5276.namprd11.prod.outlook.com>
In-Reply-To: <BN9PR11MB5276912120F662498910A1D48C12A@BN9PR11MB5276.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Ocyjz8L9rxkk7hWlAQIm7Hoz49RI1ic1zVtVZsSUozbxbdA3dSuBB6LANT1h?=
 =?us-ascii?Q?iZt8Zb51mjHJBo7kw87WVk7GyneTSz43HpevHPU7GR/qwSQGB2nIdQio64wZ?=
 =?us-ascii?Q?rwNzP2ROozJTtWAmfAoXX+6DhYEi9DxXREgOGEPvELRSGLx/N/PSjBkm6bMZ?=
 =?us-ascii?Q?ff1/eqK5SY5UnII09NJXAfwWuaVfg0olbwPFMakmj8F3C9HjyzVMVbjKJF0X?=
 =?us-ascii?Q?6tgsFctE4mKn4h3Eo3x5dCapek3s6goeCli/nIIEnmgz8y3hl73bQokg2es2?=
 =?us-ascii?Q?zFAncsIfyJfCkyeRQjypp6QIEB54Me4ZejqfRhIg+ndcXGxSNEARlwJ9slbb?=
 =?us-ascii?Q?jhfc/fJixO0L+8Bp6tp6yi7ellrWjQVe56kufUoFv8boejHM+Bh1giQxz659?=
 =?us-ascii?Q?P3LcNB5ZClS1ShkqxRdLFAUwDXP7l1uGbQ0zy49X8aCaucSEKriZfonBq53n?=
 =?us-ascii?Q?icGBuah8q+EFqATH2ZHWT+2gBVlprcNT68W9BqCRypVytBrnb1zarHjiGci/?=
 =?us-ascii?Q?j00HozyHzD9hb+NEny1Zo0VU0iPPr3nwyAZJcsahLHhcYui/WYowW5fTQJTl?=
 =?us-ascii?Q?pFeVgbe4lxZlFVsZM2FPUu0ke5N1Pz0Ui1Oc7eFX+IBrbDm5uoqsz5mc+4bf?=
 =?us-ascii?Q?d66aGMXckbWIM1/RymJGTF5UGZyVfsRxLp77l79dwXKD+8PEiKLypqYh09QN?=
 =?us-ascii?Q?r//Jw2f1E0tTUYsnGs5fH/cba+zLP+JVrJDF7v34zxbguWNTPNhgH9lMsh/h?=
 =?us-ascii?Q?nrAOV2pHJ/t33l9OPhPhg8/IHt01Gtzun1+uPkFqNNOvArHcYzuKnSYYS4yU?=
 =?us-ascii?Q?jswsk5Oq8lQ16yK9YxDZ6wHrFyDdglvnG+O7XTr+IF/8FcxqtLBCdZhTJ3V0?=
 =?us-ascii?Q?jPIQnA47UeFTBQinF3TC9jVjttDrBxJ7HlbgFwMupkDm6kzWirZeARWtr+lq?=
 =?us-ascii?Q?Txv2FfVeAgBD/d2IIsThuUVvLB5PIXqZWzrNnw3FoHuAsvbX2XpRBH8kQ+RL?=
 =?us-ascii?Q?tlTSE1K0FXK7LnsposFVr1GF/7EuUOmcDKOyS1dbXPbaXoZXeWwBKW3v2Qxi?=
 =?us-ascii?Q?vtKVMGqPYPaFBKuLQa2F5VSMbM6C832SbLQLj40kd8R2xJ8eCijtns7sNzJn?=
 =?us-ascii?Q?tV4Oh+yGwpSQpWz6wn8qJ/w7y4aRXICBijoSz14+EIlpbh8ormHY2O+6w18j?=
 =?us-ascii?Q?qqsHoZczCjdG9QiW4fcsdrJJrMuaPVccKOqFHBRCBv5/GR+AX1wPNG/YXDB+?=
 =?us-ascii?Q?rdWr/YZVVzQtmWfXjLuN+YV4gqnwXjovkwbN+vZBAIoLPnkUVO7L3sjrpqys?=
 =?us-ascii?Q?ccvWGYA4fiSjco9oF+jrwj+GpfPOLBz85xwFvx6ZJ4PHoCMCFPHdgmfN5BIK?=
 =?us-ascii?Q?PG/n5BMXzEbvydwveLGOT3H+e6cbeRIi9y1o5zYIwAxIEyUSLU5mhagd5f/u?=
 =?us-ascii?Q?nvZLggs8oCxCQ9vKsNoxvNOZV6FAXbLK1NOreZ0oF+o3lyUpDSvCknWCh2nV?=
 =?us-ascii?Q?hPxa/vvRIykc8xpi9vyobbpaL55skaB0+xk+KZGTVzIzlEBjEBa/8xwdjQ?=
 =?us-ascii?Q?=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7529.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d12e904-143a-465f-3cd5-08db98bb3ba7
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Aug 2023 09:30:12.9648
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: Hrq5Yc3L0LaJpytYoFrMdHBhu/g/Ca6/l2gHvQZEMGISxFz+dhnbxbWDwtQmab0TNPM4J25h//7DuQrT1nzmMg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4533
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Tian, Kevin <kevin.tian@intel.com>
> Sent: Wednesday, August 9, 2023 4:58 PM
>=20
> > From: Liu, Yi L <yi.l.liu@intel.com>
> > Sent: Wednesday, August 9, 2023 4:50 PM
> >
> > > From: Tian, Kevin <kevin.tian@intel.com>
> > > Sent: Wednesday, August 9, 2023 4:23 PM
> > >
> > > > From: Nicolin Chen <nicolinc@nvidia.com>
> > > > Sent: Wednesday, August 9, 2023 1:42 AM
> > > >
> > > > On Tue, Aug 08, 2023 at 09:34:03AM -0300, Jason Gunthorpe wrote:
> > > > > On Mon, Aug 07, 2023 at 08:12:37PM -0700, Nicolin Chen wrote:
> > > > > > On Mon, Aug 07, 2023 at 03:08:29PM +0000, Liu, Yi L wrote:
> > > > > > > > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > > > > > > > Sent: Monday, July 24, 2023 7:14 PM
> > > > > > > > >
> > > > > > > > > +static int intel_nested_cache_invalidate_user(struct
> > > > iommu_domain
> > > > > > > > > *domain,
> > > > > > > > > +                                         void *user_data=
)
> > > > > > > > > +{
> > > > > > > > > +   struct iommu_hwpt_vtd_s1_invalidate_desc *req =3D use=
r_data;
> > > > > > > > > +   struct iommu_hwpt_vtd_s1_invalidate *inv_info =3D use=
r_data;
> > > > > > > > > +   struct dmar_domain *dmar_domain =3D
> > to_dmar_domain(domain);
> > > > > > > > > +   unsigned int entry_size =3D inv_info->entry_size;
> > > > > > > > > +   u64 uptr =3D inv_info->inv_data_uptr;
> > > > > > > > > +   u64 nr_uptr =3D inv_info->entry_nr_uptr;
> > > > > > > > > +   struct device_domain_info *info;
> > > > > > > > > +   u32 entry_nr, index;
> > > > > > > > > +   unsigned long flags;
> > > > > > > > > +   int ret =3D 0;
> > > > > > > > > +
> > > > > > > > > +   if (get_user(entry_nr, (uint32_t __user
> > > > *)u64_to_user_ptr(nr_uptr)))
> > > > > > > > > +           return -EFAULT;
> > > > > > > > > +
> > > > > > > > > +   for (index =3D 0; index < entry_nr; index++) {
> > > > > > > > > +           ret =3D copy_struct_from_user(req, sizeof(*re=
q),
> > > > > > > > > +                                       u64_to_user_ptr(u=
ptr + index *
> > > > > > > > > entry_size),
> > > > > > > > > +                                       entry_size);
> > > > > > > >
> > > > > > > > If continuing this direction then the driver should also ch=
eck minsz
> > etc.
> > > > > > > > for struct iommu_hwpt_vtd_s1_invalidate and
> > > > iommu_hwpt_vtd_s1_invalidate_desc
> > > > > > > > since they are uAPI and subject to change.
> > > > > > >
> > > > > > > Then needs to define size in the uapi data structure, and cop=
y size
> > first
> > > > and
> > > > > > > check minsz before going forward. How about the structures fo=
r
> > hwpt
> > > > alloc
> > > > > > > like struct iommu_hwpt_vtd_s1? Should check minsz for them as
> > well?
> > > > > >
> > > > > > Assuming that every uAPI data structure needs a min_size, we ca=
n
> > > > > > either add a structure holding all min_sizes like iommufd main.=
c
> > > > > > or have another xx_min_len in iommu_/domain_ops.
> > > > >
> > > > > If driver is doing the copy it is OK that driver does the min_siz=
e
> > > > > check too
> > > >
> > > > Ah, just realized my reply above was missing a context..
> > > >
> > > > Yi and I are having a concern that the core iommu_hpwt_alloc()
> > > > and iommu_hwpt_cache_invalidate(), in the nesting series, copy
> > > > data without checking the min_sizes. So, we are trying to add
> > > > the missing piece into the next version but not sure which way
> > > > could be optimal.
> > > >
> > > > It probably makes sense to add cache_invalidate_user_min_len
> > > > next to the existing cache_invalidate_user_data_len. For the
> > > > iommu_hwpt_alloc, we are missing a data_len, as the core just
> > > > uses sizeof(union iommu_domain_user_data) when calling the
> > > > copy_struct_from_user().
> > > >
> > > > Perhaps we could add two pairs of data_len/min_len in the ops
> > > > structs:
> > > > 	// iommu_ops
> > > > 	const size_t domain_alloc_user_data_len; // for sanity&copy
> > > > 	const size_t domain_alloc_user_min_len; // for sanity only
> > > > 	// iommu_domain_ops
> > > > 	const size_t cache_invalidate_user_data_len; // for sanity&copy
> > > > 	const size_t cache_invalidate_user_min_len; // for sanity only
> > > >
> > >
> > > What about creating a simple array to track type specific len in
> > > iommufd instead of adding more fields to iommu/domain_ops?
> > > anyway it's iommufd doing the copy and all the type specific
> > > structures are already defined in the uapi header.
> >
> > Then index the array with type value, is it? Seems like we have defined
> > such array before for the length of hwpt_alloc and invalidate structure=
s.
> > but finally we dropped it the array may grow largely per new types.
>=20
> I'm not sure how many types iommufd will support in reality.
>
> Just my personal feeling that having information contained in iommufd
> is cleaner than expanding iommu core abstraction to assist iommufd
> user buffer copy/verification.
>=20
> >
> > >
> > > and a similar example already exists in union ucmd_buffer which
> > > includes type specific structures to avoid memory copy...
> >
> > Not quite get here. ucmd_buffer is a union used to copy any user
> > data. But here we want to check the minsz of the the user data.
> > Seems not related.
> >
>=20
> that's the example for recording vendor specific structure information
> in iommufd and it will also grow along with new added types.

Yeah, adding new structures to ucmd_buffer may increase the size as
well if the new one is larger. While for an array, if there is new entry,
it is for sure to increase the size. I remember there is one tricky thing
when handling the selftest type. E.g. it is defined as 0xbadbeef, if using
it to index array, it would expire. So we have some special handling on
it. If defining the things in iommu_ops, it is simpler. Selftest may be
not so critical to determining the direction though.

Regards,
Yi Liu