Received-SPF: pass (google.com: domain of linux-kernel+bounces-17936-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Fri, 5 Jan 2024 10:31:02 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
Cc: linux-kernel@vger.kernel.org, iommu@lists.linux.dev, joro@8bytes.org,
	yi.l.liu@intel.com, kevin.tian@intel.com, nicolinc@nvidia.com,
	eric.auger@redhat.com, vasant.hegde@amd.com, jon.grimm@amd.com,
	santosh.shukla@amd.com, Dhaval.Giani@amd.com, pandoh@google.com,
	loganodell@google.com
Subject: Re: [RFC PATCH 6/6] iommu/amd: Introduce nested translation support
Message-ID: <20240105143102.GZ50406@nvidia.com>
References: <20231212160139.174229-1-suravee.suthikulpanit@amd.com>
 <20231212160139.174229-7-suravee.suthikulpanit@amd.com>
 <20231213135206.GD3259566@nvidia.com>
 <12c5b705-1387-4d50-8706-cf5c35dcb3cc@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <12c5b705-1387-4d50-8706-cf5c35dcb3cc@amd.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?jLNyLYGuDUVdQDsJQZdP2B9z2IIUMjWKDLgEiWr0WmNynAeuxrMM8U7liM2I?=
 =?us-ascii?Q?1hflseFUMxQCFyACtD9V2cyC+/x6eqG/ioOE0VYFpK28GGGIVKBnZ4eEIL2q?=
 =?us-ascii?Q?Ta0qxAQD2SV864OqnDQXwKgsJwQ07Upl/5X3CjT27wmnbu6fe4fJ/BXjyoFv?=
 =?us-ascii?Q?976odFM7d7uth8cOlHQAE5bX7/xPh3G5eOfopPFUheExIr2vmJVdftmrp0jR?=
 =?us-ascii?Q?uueIDMsfM+WbsW9ayJrC7x1tv1DvGrRjt20h4NdnNaMeEp+FhqB6NioMeN1D?=
 =?us-ascii?Q?ILsbfDhRvbIuikQJsZspPtyqCtRMfwxeSqitmGIqetUJ3k0tbrOhML7/3K9Y?=
 =?us-ascii?Q?uySZnrYr6cczNA4HFNoaP8kRu+2Bo0GHSp7QzsifgrFhXWjPyAd9z4/nc7Ap?=
 =?us-ascii?Q?7h6Vgne07IXYqmCw78eGJnr+DIPDDTFn11oz0ydZepyBVYsUl/9FCzc/gEHS?=
 =?us-ascii?Q?c4xJDFtVmj0JwQtZcTY1mGidjKeq/e3i4oqFxb2fL9IyqOi0IZV8cY/uwCZJ?=
 =?us-ascii?Q?sc99RdqqBzSKidLFtaQtAJ4MN/pvJkaDnTJupfzwUaJ56GyRqfm+gCDlqnq+?=
 =?us-ascii?Q?ruNXq0Fp2YlXOYO0IGtRMyjyHyocRm2HvMuP9V8wqXlUvQjPONnI4g4vqB94?=
 =?us-ascii?Q?a4RKY9gosNmXBM7+cQvHG6WP8LGNzc7QPvrovNc9/Tly7eXpSmn/QQUbgPET?=
 =?us-ascii?Q?J15TuyQ+neMyzR5knBPr75tUwFSy/xvtCc+uWOieCCvI8mNCDQqK/OC9N0Oz?=
 =?us-ascii?Q?GoSRdZBpkXnNGf42Jg/V5T0ChWZ7miEDecfZR/d0r2H5fEgQVAlD6X++Ebx+?=
 =?us-ascii?Q?yMEO2oUzU3G7oKBYly8eCZk/NjZ84ZZcBTp/ZZt+etdZHeiq53Sr3PDhV0Rp?=
 =?us-ascii?Q?l/aO6QqyCmw7+Ur5xCRY6x+dPzxhVbgNWAiaIJGFrdnD8Yrp57O/6ZjUvS1n?=
 =?us-ascii?Q?8ZQ4cR2uSjzaFTuvwfcsfqrY8qz0+6ugoVuecieqEJeEXZXQ7xzGEQPzgw0/?=
 =?us-ascii?Q?Pk9ZcxFMeSDrtivY70jPGhuMggvqH96m4YdCw3S7pWoePbhMgfZ1NEJUDKBD?=
 =?us-ascii?Q?wWT59gPbttF6ktMGf9vbP9WIGqkqkF4TNtnzB6D/Lk1KxqWGqmPhpg9abriY?=
 =?us-ascii?Q?Hu95re/b9VC4dueVSK+yDWVrvELyAfA1KS8XuRlAZoq1pQHcsPqPBAZa51a0?=
 =?us-ascii?Q?mxno8xMjRlmn3fUh63umoRAxwfS8ccJ7RVpy8NkHmtF7gu6QbUxuV9soaEwP?=
 =?us-ascii?Q?41hztZo0LIThhXxrgEr2Ow+7xit/vhpoTbrtYr3ygd7proqC9ajEE40AG7ra?=
 =?us-ascii?Q?S8akzFYj8D3srgK5uSWX/eNVmtAoPsLvkaFdKN2loFrm8vhApoxRHGepUACU?=
 =?us-ascii?Q?a6ZEGHrbwS+zqHp0irEBtGSrJ1yl6MQ+5PPRQcNV0qSKAuVsKLo3yGsk9ShB?=
 =?us-ascii?Q?o4ryIaHhgcXd481kIlZKObQC6ndolXuAsxhj2tOLyZR+WqsJwGoRvS0ZtXRq?=
 =?us-ascii?Q?jSkLYcTCHOuh4tqkxxzLaJNFKEnI/d/USEqeu0ZXPOCxkoZJG2EPqIfJqKuC?=
 =?us-ascii?Q?SQwTml78sZgf5MzwpuU=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 53409002-5a9e-4a48-aa2a-08dc0dfaf1e8
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 14:31:03.2899
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +kq90Pwbp/Inze7F9MjqtVsOZlGWkhhZdahMt/FF7ghnAI4/BbMiQbRDIQVIBNxx
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7910

On Fri, Jan 05, 2024 at 08:38:47PM +0700, Suthikulpanit, Suravee wrote:
> > > +	if (!pdev)
> > > +		return -EINVAL;
> > > +
> > > +	/* Note: Currently only support GCR3TRPMode with nested translation */
> > > +	if (!check_feature2(FEATURE_GCR3TRPMODE))
> > > +		return -EOPNOTSUPP;
> > > +
> > > +	ret = amd_iommu_set_gcr3tbl_trp(iommu, pdev, hwpt->gcr3, hwpt->glx,
> > > +					hwpt->guest_paging_mode);
> > 
> > Waah?
> > 
> > This is touching the dev table? That is not right, allocation is only
> > *ALLOCATION*. The dev table can't be changed until you do attachment.
> 
> My understanding is QEMU should call:
> 
> 1. iommufd_backend_get_ioas()
>    -> ioctl(IOMMU_IOAS_ALLOC)
> 
> 2. For parent domain
> iommufd_backend_alloc_hwpt(IOMMU_HWPT_ALLOC_NEST_PARENT)
>   -> ioctl( IOMMU_HWPT_ALLOC)
>   --- in Linux ---
>   ....
>   -> iommufd_hwpt_paging_alloc()
>     -> struct iommu_ops.domain_alloc_user(IOMMU_HWPT_ALLOC_NEST_PARENT)
> 
> 3. For parent domain
> iommufd_device_attach_hwpt()
>   -> vfio_device_attach_container()
>     -> ioctl(VFIO_DEVICE_ATTACH_IOMMUFD_PT)
>     --- in Linux ---
>     vfio_iommufd_physical_attach_ioas()
>     -> iommufd_device_attach()
>       -> iommufd_device_do_attach()
>         -> iommufd_hw_pagetable_attach()
> 
> 4. Same as (2) but for child domain w/ stage 1 table.
> 
> 5. Same as (3) but for child domain w/ stage 1 table.

Yes, but understand it is not API that the driver can depend on that
order. #3 can be skipped and there can be multiple parents and
everything must still work.

> You want the gcr3 table root point in the DTE to be update at step 5 only,
> right? If so, this should be okay.

#3 sets the DTE to point to just the parent domain as a V1 page table

#4/5 sets the DTE to point to both the V1 page table and the GCR3 table

The DTE is calculated based only on the *currently* attached
iommu_domain's for the struct device.

> > Please look at the smmuv3 patches and try to be structurally
> > similar. AMD and SMMUv3 are *very similar* in how their HW works
> > excluding the viommu stuff.
> 
> Could you please point me to the series? I found one but it was really old.
> I might have missed the latest stuff.

https://lore.kernel.org/linux-iommu/0-v1-e289ca9121be+2be-smmuv3_newapi_p1_jgg@nvidia.com/
https://lore.kernel.org/linux-iommu/0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com/

(and the github link has the other parts)

> > You also can't assume your parent is currently attached to anything.
> > 
> > The construction of the DTE has to be from-scratch based on the parent
> > domain and the provided values in the "hwpt". Again see how smmuv3
> > does this where there is one function that builds the entire DTE
> > (called STE)
> 
> Ok. I'll program fields of the DTE, which are related to DMA-remapping with
> v1 and v2 table using the information in the parent and child domains only.

The smmu code looks like:

static void arm_smmu_make_nested_domain_ste(
	struct arm_smmu_ste *target, struct arm_smmu_master *master,
	struct arm_smmu_nested_domain *nested_domain, bool ats_enabled)
{
        [..]
        // Incorporate the "v1 fields" into the "DTE"
	arm_smmu_make_s2_domain_ste(target, master, nested_domain->s2_parent,
				    ats_enabled);

        // Incorporate the "GCR3 fields" into the "DTE"
	target->data[0] |= cpu_to_le64(FIELD_PREP(STRTAB_STE_0_CFG,
						  STRTAB_STE_0_CFG_NESTED)) |
			   nested_domain->ste[0];

Where "arm_smmu_ste *" is stack memory that holds the "DTE" being
constructed. Once the make* family of function figure out the exact
STE that the struct device needs, another function writes the STE to
the HW visible location(s).

The goal is to be able to calculate the required DTE for the #3 and
#4/5 cases you list above directly without making assumptions about
the current state of the DTE or anything else.

> A device can be attached to a domain, which could be either one-level or
> nested domain. In case of nesting, the parent domain contains information
> for the stage2 (v1) table, while the child domain contains information for
> the stage1 (v2) table. For each domain, we need to keep track of the parent
> domain.

The nesting domain, and only the nesting domain, stores a pointer to
it's parent domain - they are a unit together.

Jason