Received-SPF: pass (google.com: domain of linux-kernel+bounces-136354-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Lu Baolu <baolu.lu@linux.intel.com>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>
CC: "Liu, Yi L" <yi.l.liu@intel.com>, Joerg Roedel <joro@8bytes.org>, "Will
 Deacon" <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH 2/2] iommu/vt-d: Remove caching mode check before devtlb
 flush
Thread-Topic: [PATCH 2/2] iommu/vt-d: Remove caching mode check before devtlb
 flush
Thread-Index: AQHaiPoC7RpD2axGS0qZEPwCpJ1HeLFfi1CA
Date: Tue, 9 Apr 2024 07:30:16 +0000
Message-ID: <BN9PR11MB52764F42C20B6B18DDE7E66B8C072@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20240407144232.190355-1-baolu.lu@linux.intel.com>
 <20240407144232.190355-2-baolu.lu@linux.intel.com>
In-Reply-To: <20240407144232.190355-2-baolu.lu@linux.intel.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?nsDSNzD4tlXJLAS6uUd6suuWvLlmJrRMGsrUAOlIHoChLv6duzFi8ALroA9m?=
 =?us-ascii?Q?ljKP175eFtXAcEE0va7zZZLJ5qBmw+dSP+FfNx+UTWF+vcmx0zgBPQfULcU1?=
 =?us-ascii?Q?s8fPQzuht80S9OlQtW9lhXXx0euIQXO1RUToVipEp7nNWZGFeqSzxVsFLIDz?=
 =?us-ascii?Q?hJz0x45kS8Ko19m7QntgIDgIdWskmlYoN2uldrDOhixvxnWqKV49o/hxRWNs?=
 =?us-ascii?Q?0h6FOpnawhkUB63KHwrdDmXl0/3WVGWgw4TjrOwLejD3IQIva6duBeJlOh/q?=
 =?us-ascii?Q?qIbEcKZHQcQZdxwiXrlPnQhPlYJIK2gOvEq/Vc7JbzwkFUJ9bMf+V8RecN1T?=
 =?us-ascii?Q?0sSaNGmAZQjVSVwiuZ5OXMmGdmDK4Y+dfRHGn7H767v2EP1D1JyGUibfV5Nx?=
 =?us-ascii?Q?My+t3DCPP1ijBG8o7GNn8HABkQFTB3Hdzle66RpMm6kq5MqYBVBDPWY3/uzQ?=
 =?us-ascii?Q?TxD4CRZqzrj2Br/LAQJCdVvkg2qRZc8BNCCOKBy/46/BSysfW0wXu4c7aeJv?=
 =?us-ascii?Q?jPtS8xUzN9+PXMBdBTI8uggIC4xmumrrGistPSyFDnmnBRl0F+4UXk+HKnsA?=
 =?us-ascii?Q?tawYkhxnMy4W0oYTuxXtZ7rBrU08lwYeFyWxxd/UNEJP7nnsWM6q0nbUqghA?=
 =?us-ascii?Q?RHaqoqsiPvRMdaXIzINIu0Hiq56QphxD4+dZzN2WWLKXIFgBd6j5tnLAew7X?=
 =?us-ascii?Q?QXElBD7HyzsaGO3wUFWLHM43DDIXr3Z7D2hftGQoUe3IP+CsSIIZHmAfGXnR?=
 =?us-ascii?Q?AZJrSx+Na/YdQHwxggFUnJFBnBdu6/QXhN3bx+UPODPIcCf5w+D3/IL6/uU7?=
 =?us-ascii?Q?t5UaUKbZ+oGayKNvLazxcET/gipsBwJhKjvnkgEWIPRo3P9NEIYAgu7sv/NS?=
 =?us-ascii?Q?z+/0qhyLTTuPyYmu4UCMms8S4/ipib3wbqYfOn74hZ5bQ824XiSpbdqQok/y?=
 =?us-ascii?Q?cB68r4ow74nrewCQYEZoBWBPMQ/oW1jgTPrv0f1kKxoyeD4mfj32SqmZMMoV?=
 =?us-ascii?Q?UeXpi5sXDMRM/CiNfn/hi277SkzJbdlydBg7kCLYBpEOa+8J1KQDXwJwVhUW?=
 =?us-ascii?Q?j+U1viMwdN0oSbFxmaUFvoUcfac6i9iXmRPXKtI6KvhRtuwfO+q8o1N2zbrO?=
 =?us-ascii?Q?H3OkDE4PciO6XbVBk8Vb+NLJkcLI/FSGX6gc5HdoMhlLyqlbb7HiqR8YP30D?=
 =?us-ascii?Q?YjuIcj6M8y/N4B2Ypo4WklFf9FPcpArsapQHUu/bEaGYfrxrtmETnEx9/4YP?=
 =?us-ascii?Q?sAbhIsYjDM+yxnPNyielWo7eWY3T0bqsf+7moMa3/stxicOt10Ke4TVP/yuM?=
 =?us-ascii?Q?fMjiwIbDN1sKzAinQq8G4q9/+HPubk+82YecWRRvLKrj9F/6QofRbeU2sBmO?=
 =?us-ascii?Q?lqsh6wCcm6aWezG6RwMyXbutl+ZfImWDDaRIUpZnNBrulJCiK+LqgPKqumlh?=
 =?us-ascii?Q?Ebl7er4gLXvbQUdl5/O+Sm0cAykpA5orWagcZp0Za5PEPz7P2dHu0sKMlCFa?=
 =?us-ascii?Q?ODKM6JQ3J+ZnbIBGiRr6EwGDaJeVWrCAajIYRig1n7uc0RdsfHSUjJgJA8r6?=
 =?us-ascii?Q?i8jtEZCo4iRElFikXdGcdhyqCiYjv7uWA2fwSXMt?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7e1c7746-bd5d-4268-a553-08dc5866e6f4
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Apr 2024 07:30:16.4222
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 8p1Dlvettp7FSpGMpC+FkgQXrQo01vjL+NCdsrwbw8is4/vSlKMd8sXf4heWbBoz/Vk74jRgiLGMTcqh1t11PA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR11MB6409
X-OriginatorOrg: intel.com

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Sunday, April 7, 2024 10:43 PM
>=20
> The Caching Mode (CM) of the Intel IOMMU indicates if the hardware
> implementation caches not-present or erroneous translation-structure
> entries except the first-stage translation. The caching mode is
> unrelated to the device TLB , therefore there is no need to check
> it before a device TLB invalidation operation.
>=20
> Before the scalable mode is introduced, caching mode is treated as
> an indication that the driver is running in a VM guest. This is just
> a software contract as shadow page table is the only way to implement
> a virtual IOMMU. But the VT-d spec doesn't state this anywhere. After
> the scalable mode is introduced, this doesn't stand for anymore, as
> caching mode is not relevant for the first-stage translation. A virtual
> IOMMU implementation is free to support first-stage translation only
> with caching mode cleared.

I didn't get the connection to the scalable mode.

if required we can still use caching mode to imply running as a guest.
Just need to make sure its implementation conforming to the VT-d spec.

>=20
> Remove the caching mode check before device TLB invalidation to ensure
> compatibility with the scalable mode use cases.
>=20
> Fixes: 792fb43ce2c9 ("iommu/vt-d: Enable Intel IOMMU scalable mode by
> default")
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> ---
>  drivers/iommu/intel/iommu.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
> index 493b6a600394..681789b1258d 100644
> --- a/drivers/iommu/intel/iommu.c
> +++ b/drivers/iommu/intel/iommu.c
> @@ -1501,7 +1501,7 @@ static void iommu_flush_iotlb_psi(struct
> intel_iommu *iommu,
>  	else
>  		__iommu_flush_iotlb_psi(iommu, did, pfn, pages, ih);
>=20
> -	if (!cap_caching_mode(iommu->cap) && !map)
> +	if (!map)
>  		iommu_flush_dev_iotlb(domain, addr, mask);

as commented earlier better squash this in patch1.

>  }
>=20
> @@ -1575,8 +1575,7 @@ static void intel_flush_iotlb_all(struct
> iommu_domain *domain)
>  			iommu->flush.flush_iotlb(iommu, did, 0, 0,
>  						 DMA_TLB_DSI_FLUSH);
>=20
> -		if (!cap_caching_mode(iommu->cap))
> -			iommu_flush_dev_iotlb(dmar_domain, 0,
> MAX_AGAW_PFN_WIDTH);
> +		iommu_flush_dev_iotlb(dmar_domain, 0,
> MAX_AGAW_PFN_WIDTH);
>  	}
>=20

I'm hesitating to agree with this change. Strictly speaking it's correct.
but w/o supporting batch invalidation this implies performance drop
on viommu due to more VM-exits and there may incur user complaints
when their VMs upgrade to a newer kernel version.

So it'd be better to keep this behavior and fix it together with batch
invalidation support. Anyway none of the viommu implementations
today (either shadow or nested translation) relies on the correct
devtlb behavior from the guest otherwise it's already broken.