Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Jason Gunthorpe <jgg@nvidia.com>
CC:     "Liu, Yi L" <yi.l.liu@intel.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: RE: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d stage-1
 cache invalidation
Thread-Topic: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Thread-Index: AQHaGViOWXGe59nutU2rEx/0zRDvrrCC4vaggAD2R4CAAD710IAAnmMAgAEUIFA=
Date:   Wed, 22 Nov 2023 04:58:24 +0000
Message-ID: <BN9PR11MB5276BC5DD9824923C8A3ACC18CBAA@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <BN9PR11MB52761D7A88BB2F655355A7728CB4A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20231120230451.GD6083@nvidia.com>
 <BN9PR11MB5276EF10FCE9C96F593696E18CBBA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20231121121704.GE6083@nvidia.com>
In-Reply-To: <20231121121704.GE6083@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6MGkly+EP4mGQ9DuphZcGhU/0aV2a4qidOQG/hTY2xC9beorEaz6CDSwRNfb?=
 =?us-ascii?Q?9t28gN1IGQim39lTcm1Se3aA4avFG778sfD5MmW4uARo1Oiel68EMtBrruxG?=
 =?us-ascii?Q?38P3W4b1boWnc+jLZiYnbT9bP/RSuSyCXKAaPdfEpizaIz5g8OZQ8MTgheLF?=
 =?us-ascii?Q?3JXbhbmwieU95VXB/CX+DRGWWOW9ouynnhK3Zpu1hxaRkNpZclNyOywL86yR?=
 =?us-ascii?Q?/8/BWN6KH1MPxwwhNsX3c87wmfhgZcQX8AhAahfzPkYdg4PtLSynMJv83jIM?=
 =?us-ascii?Q?S3/QwZU1GTyoeEe0K0EVgfoL54JRx1kNz+SmV/c0MaDGtlDiNM9a64OXe6ae?=
 =?us-ascii?Q?VRFRUtbmvahROQO08qUAWgqv0iIAytTfoufeWBpq++LnepeQgT03X2lZL9Ew?=
 =?us-ascii?Q?svEzRcZVVS1BjS6RW9nEB4R1QNLBe40iV2oCMJPhbqMSHGf0EmZGGpL2T/8W?=
 =?us-ascii?Q?MUfIPQXtnjjHR2dFibzXvVzORxfyGGgquHqw0HwTs5JA9bqCqvvXWsWHMH5G?=
 =?us-ascii?Q?RasMPIzkwZh1IlGPy9qhxQssdvD1BvJoqRkTt3RqpVaD2HKaG1feF1vgvkRM?=
 =?us-ascii?Q?ZYaYjGfr6Pg2igLO3FhO2WAas4kGB5i8B6m3HYaYraZuIcI1gAsVEEgQAg4y?=
 =?us-ascii?Q?xLIRbff4Fc2BGg9Zn0vtoPQhKaxCGT7uFFVaUoBTZof8L8LbkD53/O1httie?=
 =?us-ascii?Q?OSFKD20fmL/TrBu+sgeNy/U6Pa9C/ZzXQ9fEcw4HdzzNMUqV6slS4D6BzOiE?=
 =?us-ascii?Q?48y2oHkyNlzslX6cmQLvOpr/SC/0hZ0Bz7J+VtqtR85gFWEENtb17QOh32WM?=
 =?us-ascii?Q?IAHgFLb/gyts/xRM9GAw8NlcdNs3UqWZ4hUMk5RaahNPjrQbyPO8mfXKr1vX?=
 =?us-ascii?Q?xZ7HJ/fOteIg+SKEf2I2k6nFkmO1SOgQ8EVLmfV34ZOIE9nyPmh9+LoH5y9P?=
 =?us-ascii?Q?+uli8Hz/fYmVrPQlxLtIfdyLJkgp2iwRWU8ZMaQ0qFoeLREhnD0/bd7c9nAO?=
 =?us-ascii?Q?qUBZrabwD/yonVtN9Y0f3JNeTtYTayHgaRLG2+8xzXqIy6LmtsgpQFnTKqps?=
 =?us-ascii?Q?wyJszsYTcmD5V3o72/W+B9phbaQ5LbjZhYXDFJRaQ7fqnYmWXQaX1Tx0gJ6Q?=
 =?us-ascii?Q?zYcRIu5v84rXq8fXIzpxF9/4Fh+KFgPZn8m/RDHasOW4exscNiKBdSGm6OzA?=
 =?us-ascii?Q?NGIkCS2mnLVeeSWwO05bkul7P6SQnLElwCGg867QVWZXrQPGgUBqSERI49Et?=
 =?us-ascii?Q?H54vKfg06lBihN7zmVv4tbNuLuBjJ9DnSZoNcBTkAbm3Vdc4WhL2PZ6ZQXl0?=
 =?us-ascii?Q?yt88tq76MIF+B50RlbBWR077wPK/6Y5ggJ8aDgd5CsXwfpUxjpYkfFuqhf/G?=
 =?us-ascii?Q?/Uxt0ySx9SuDfbht+R2qqnCkVktFA7PgWA1Pr9AiNiD2RgEQkzGPQNKI1IZz?=
 =?us-ascii?Q?R2Jmh9bGj6VEO0Gc5rWBd0EwjJ3PbiY0Ptik986vc8GL86AoCFBFzzRq1Abj?=
 =?us-ascii?Q?UycSn+VtXnMthvFUgoiaGg67rE+JEM2S5KhN4JlL3z/VGwWCnGBMSeOTPLtT?=
 =?us-ascii?Q?b2i3z5PZrOcI22X8yNxsw2j1gfi5WWfM8Jpe3a9I?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fd766d77-0992-4812-8b1f-08dbeb17a846
X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Nov 2023 04:58:24.2759
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: r+Q7h7O1PTkLbBhyOKpgIT8iN8HIBLYtgacWDsGobGBLdSvW/nb7ZSoUX8HUZVbhxkH4GHHoPc3YEcYJ+qND0Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4949
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 21 Nov 2023 20:58:54 -0800 (PST)

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Tuesday, November 21, 2023 8:17 PM
>=20
> On Tue, Nov 21, 2023 at 02:54:15AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Tuesday, November 21, 2023 7:05 AM
> > >
> > > On Mon, Nov 20, 2023 at 08:26:31AM +0000, Tian, Kevin wrote:
> > > > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > > > Sent: Friday, November 17, 2023 9:18 PM
> > > > >
> > > > > This adds the data structure for flushing iotlb for the nested do=
main
> > > > > allocated with IOMMU_HWPT_DATA_VTD_S1 type.
> > > > >
> > > > > This only supports invalidating IOTLB, but no for device-TLB as d=
evice-
> TLB
> > > > > invalidation will be covered automatically in the IOTLB invalidat=
ion if
> the
> > > > > underlying IOMMU driver has enabled ATS for the affected device.
> > > >
> > > > "no for device-TLB" is misleading. Here just say that cache invalid=
ation
> > > > request applies to both IOTLB and device TLB (if ATS is enabled ...=
)
> > >
> > > I think we should forward the ATS invalidation from the guest too?
> > > That is what ARM and AMD will have to do, can we keep them all
> > > consistent?
> > >
> > > I understand Intel keeps track of enough stuff to know what the RIDs
> > > are, but is it necessary to make it different?
> >
> > probably ask the other way. Now intel-iommu driver always flushes
> > iotlb and device tlb together then is it necessary to separate them
> > in uAPI for no good (except doubled syscalls)? :)
>=20
> I wish I knew more about Intel CC design to be able to answer that :|
>=20
> Doesn't the VM issue the ATC flush command regardless? How does it
> know it has a working ATC but does not need to flush it?
>=20
> > anyway this is driver specific contract. I don't see a need to keep
> > it consistent for all.
>=20
> Given that ARM and AMD need this and would have serious bugs if it
> didn't work this way I'm mildly concerned that Intel will be missing
> something here..
>=20
> To my mind it seems like this is just a hold over from the prior
> design.
>=20

As Yi/Baolu discussed there is an issue in intel-iommu driver which
incorrectly skips devtlb invalidation in the guest with the assumption
that the host combines iotlb/devtlb invalidation together. This is
incorrect and should be fixed.

But what I was talking about earlier is about the uAPI between
viommu and iommu driver. I don't see a need of having separate
invalidation cmds for each, as I'm not sure what the user can
expect in the window when iotlb and devtlb are out of sync.

then we just define hwpt 'cache' invalidation in vtd always refers to
both iotlb and devtlb. Then viommu just needs to call invalidation
uapi once when emulating virtual iotlb invalidation descriptor
while emulating the following devtlb invalidation descriptor
as a nop.