Received-SPF: pass (google.com: domain of linux-kernel+bounces-17942-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Fri, 5 Jan 2024 10:37:03 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
	"jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
	"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
	"pandoh@google.com" <pandoh@google.com>,
	"loganodell@google.com" <loganodell@google.com>
Subject: Re: [RFC PATCH 6/6] iommu/amd: Introduce nested translation support
Message-ID: <20240105143703.GA50406@nvidia.com>
References: <20231212160139.174229-1-suravee.suthikulpanit@amd.com>
 <20231212160139.174229-7-suravee.suthikulpanit@amd.com>
 <BN9PR11MB5276BA85118EECCCD89681EB8C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <d2cc19fc-f8c6-4b3a-ba88-5db947fb1604@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <d2cc19fc-f8c6-4b3a-ba88-5db947fb1604@amd.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?x4CUf9EQSBP/tGMw+OyG+UDr1IbIPsi6tIWggjPgYCDEzG6T/vAy6USAHcCv?=
 =?us-ascii?Q?cgEz71ktnCTmZtXCwWd2QC6sj8Zu/1JZBomogfGVNzqwwNF/x20N6rv3mlZe?=
 =?us-ascii?Q?xOUOkkEhl2SeXfOPVIajCNXDThaniHmqYGXfUXzFLNlRyI987wzepDO2vKRY?=
 =?us-ascii?Q?fhqwolnpsvSUgaK2hwdlieI0qwFMqTE5K5I9FwGtrGfEWISA700OSUFycfQJ?=
 =?us-ascii?Q?4SZaLiQf+4J68SNfWWz5aJHXh4PUD7AQjxPB5xv2rYMofnYi1w8/WziXfMZv?=
 =?us-ascii?Q?flHaqfZ4pC3ors3g0rOokEp8PeoHCAR31DsW52weFXW43dlzxoY3s7m6z87n?=
 =?us-ascii?Q?QXRZxzrixnctq93LfSmSN6XWA1Oh/XQip4tfIcKk+c09XxfJQO8wuqT3t4sM?=
 =?us-ascii?Q?Vmgi8bKhzJRDhUkNeCyytf74d/Wx8bPMVygla9kGwfHWEPzq8DpgfMVhv71b?=
 =?us-ascii?Q?UW5oZA2DwoaZoqk/I9KcKwNZmBEaUiqi9J6Pif+B4AW52jrJ2Qy0dKFpavLT?=
 =?us-ascii?Q?JKo7vgJFvlFYSX7T9QCvwrwMRw62nwzLgI6pyd8DVUOVBy40lKgtkHNlMT3M?=
 =?us-ascii?Q?CO91wneXVXL5c6BgzFyNTfz8pUFeeu7coNobuKKp45KjfIGdpsTG7Hb9ExqL?=
 =?us-ascii?Q?Xvh/6umy/N2ZYQ0cxyLy69gbiEb5/YDFissA3Btfzc5chh7SLcvrpCfpEsB7?=
 =?us-ascii?Q?r7IqfGygeJfjZ+7bq869CJcFHq2XsCkDsfBOXmU2seDnBbO39DfVB13nfBQg?=
 =?us-ascii?Q?D1YhJNHN1/GXGqRx+z5uOTKama24QNll1FMuSG1S/vfyiXyV407dqUOpnXnI?=
 =?us-ascii?Q?FhqctgRWUAFIOsKcpwU40pRibBcYYedmlIxdh15avCD0pHhMeOlWsl/Q8yzP?=
 =?us-ascii?Q?/Xsyv6HG91aasCijO9TjfsipotnsSIPqFL865qR55f6IXSb56miKOoyiNIk5?=
 =?us-ascii?Q?MtlhmkAoxJohn2ujiruicjqORhPrTZV+LTT6I3iB/7kr/n+NfT8MDyyULrIm?=
 =?us-ascii?Q?yWo4nI1++6v1Xw1PihM38m51LkNTrU1WhDkKaB5oMjkhJV9Z2tMMoYzJ0U7+?=
 =?us-ascii?Q?kXVWsFE3gB2lCgFRN4hSb48PJKtLdIX1j/gTYK94ddgy0n0dqsdevDe8PF16?=
 =?us-ascii?Q?CCaF39m0fmaFZAYQ6MHhgJccrz+73f/9+u8r517OKBMfXc42Q2QmeU1Ptq7k?=
 =?us-ascii?Q?jDe2DoPiPE6/FpTyV/0PgWONB+RmSYKjdCu9HtDIubJWysg1eRJvXmiVBY4p?=
 =?us-ascii?Q?TBVBUWnTs6kZjjy6ZYI035I2lYizzoV7sZ7HJdX+O+NM8Ihqpp8obGe3phqt?=
 =?us-ascii?Q?ckDfPXGZB3Xf1AjKNNlWE6msV1H8YrMb6BVmV2xZXfChtiSr/GMGgoKsJpre?=
 =?us-ascii?Q?GmpfkgIpKHsubOISqnse5cJW0QXcAEq9RmS/t03SgqfZ2X/Wz/KfRdx+Uv7n?=
 =?us-ascii?Q?3gj3ZfeLlA82oVs749QSuBYA+0hP7SwgL0hoVJX+QcKZKpSwc9vpjbZ/r6Ci?=
 =?us-ascii?Q?SwBXHFiOv4b/EKdGq3xYWALAnH9ADynXOAYTsYP/nK/V0mu/VD9QcYidrToN?=
 =?us-ascii?Q?+0AoO8NWtcakr8sQkio=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 23396138-73b3-4f79-e572-08dc0dfbc906
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 14:37:04.1741
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /81jNGl1MiH/GRgmkZYa7gycLK0LrZdfGkOubaKogRpQep0y6XMDUeBaY8/tAVWK
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5050

On Fri, Jan 05, 2024 at 08:39:52PM +0700, Suthikulpanit, Suravee wrote:
> Hi Kevin,
> 
> On 12/15/2023 2:45 PM, Tian, Kevin wrote:
> > > From: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > > Sent: Wednesday, December 13, 2023 12:02 AM
> > > 
> > > To support nested translation on AMD IOMMU, the driver needs to
> > > program DTE[GCR3 Table Root Pointer] with the address provided by
> > > the guest via struct iommu_hwpt_amd_v2, which is passed as a parameter
> > > of the struct iommu_ops.domain_alloc_user() with the flag
> > > IOMMU_HWPT_ALLOC_NEST_PARENT.
> > > 
> > > Note that current implementation only support GCR3TRPMode for
> > > nested translation, which uses GPA to program GCR3 Table Root Pointer.
> > > 
> > 
> > means there is a plan to support another mode in the future or
> > actually the nested translation requires GCR3TRPMode as a
> > functional requirement? imho the point of GPA is assumed
> > in the nested configuration in concept...
> 
> On (older) system, which does not support GCR3TRPMode, the IOMMU driver
> needs to program the device's DTE[GCR3 Table Root Pointer] field w/ SPA.

Meaning that on older systems the GCR3 Table Root Pointer is not
translated by the parent v1 page table?

> When QEMU presents an AMD vIOMMU device to a guest, the guest programs the
> guest DTE[GCR3 Table Root Pointer] with GPA. Then we need to :
> 1. Traps the DTE write
> 2. Translate the GPA->SPA
> 3. Program DTE with SPA.
> 
> With the GCR3TRPMode, we can skip step 2 above and directly program step 3
> with GPA.

Do you want to support this? It will be hard to do because it is not
just those three steps (which are easy) but that you have to somehow
maintain coherence with any changes to the parent page table, so you
have to hook the iommu_domain unmap as well...

Jason