Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Date:   Wed, 13 Dec 2023 09:52:06 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc:     linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
        joro@8bytes.org, yi.l.liu@intel.com, kevin.tian@intel.com,
        nicolinc@nvidia.com, eric.auger@redhat.com, vasant.hegde@amd.com,
        jon.grimm@amd.com, santosh.shukla@amd.com, Dhaval.Giani@amd.com,
        pandoh@google.com, loganodell@google.com
Subject: Re: [RFC PATCH 6/6] iommu/amd: Introduce nested translation support
Message-ID: <20231213135206.GD3259566@nvidia.com>
References: <20231212160139.174229-1-suravee.suthikulpanit@amd.com>
 <20231212160139.174229-7-suravee.suthikulpanit@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20231212160139.174229-7-suravee.suthikulpanit@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+a6CLWS3o4kdtt7MKkfv8yYwYl2txDoah9BBVtL9e7lOS7kqDCCNxSRD15zL?=
 =?us-ascii?Q?RWrr7yCOVNQt1ao4xZ779I3ICb55Izcl4RdsQoA+z+eCOMdGVHG3BfNk1O/y?=
 =?us-ascii?Q?WbyvSOzWz3iV395irDo5P3Ja3UJ8AhpXanb0GQgPx4NcrxmerXLO3wHHF9qr?=
 =?us-ascii?Q?BC7VE1a90ct30RkCeV0TMOz75waRaPdZP7HZ6TqTLntcnP6lKiFx5hw6+3eX?=
 =?us-ascii?Q?8UK0g9HMYZXrGccgfWBUeRW49PcImFtDbF8DUTVSkUZ09eMts2p+YchnHnIG?=
 =?us-ascii?Q?OivAtYudMtY8EZeuLo0vlpHqmXadNgeU7dE3e+KTkKDnzRUwZRHPxoiastDl?=
 =?us-ascii?Q?lK9/l/xpC+bZqd4msGfuegcuvBH0fThvtZOkCJvO/vVel1bW/yWWc3HFdIpR?=
 =?us-ascii?Q?L9r007UNH5+SlXhPKVyR4x8rQGgIXikGav0CeXoUZo7wgBaKhaC4sxCqNhwY?=
 =?us-ascii?Q?Y7I90NHNQuegXDdNZqNSO5BYJjUy2R2YHIsSUYU0nlS0d4Qyg5pWNLfGtkSR?=
 =?us-ascii?Q?ZZ8Ldw3hBOz6JBjJZRxIxhomYBmkUluyg8k50WY1DmaHyr0QN5/rntfvQVBN?=
 =?us-ascii?Q?8riC2F6k46u55wDoD8pg6DpIoviJiSle9zoKc3dNkn2tGa6Nd6Uh9ILBnPeQ?=
 =?us-ascii?Q?WxaUrk0xBc/gOiFR5A5GIkbuG0iBEKUR8Pp6g2wpXJGSI/tJdA6K2R8d+wvt?=
 =?us-ascii?Q?lwHrP7zx8Yklm4kX3KWbt8HyzFUupZm6jnrkZFovCGxknJ4apaZTStQZ4A2F?=
 =?us-ascii?Q?pE2EL3aqgp9R1iYSU6wSTAcc7Vla2s1Ut4xcsWRSo1OPhKCD8/zhXm3J5sEi?=
 =?us-ascii?Q?IOvqcMAHNjeD63p8rc/Hv+VFL854pMjOwLtCIN0i6OhK/1TsB9oToE5K8fTn?=
 =?us-ascii?Q?T9XS+3ksjnP/IMUlvv9XopxNRXDX6QTdzRTvO6SPFijjbCStDeBU/cGLEHg5?=
 =?us-ascii?Q?ukguzUJfCTjsDov4NsNc0DtUrSWHgDS7vAgDpaSmbZARw5zft5EreCA3ZJhL?=
 =?us-ascii?Q?Rjn+69piOn84sMAz3zdWxzqd0+JFACl7Dwt/qAxeqnDH2PmLXiFhUA+XoRTt?=
 =?us-ascii?Q?21exA/UaJb7KrwnQ4ndhzCdZ765qeJiml/37H+Y6KE1cwdMrSGIbkLuo1Wao?=
 =?us-ascii?Q?Necb8I6CzWWEkvUadEg9j+GUGg9fr2xe7tAwydKbLkq15t64FndKOEhI8PpW?=
 =?us-ascii?Q?sIKyuBVHDe4+fJEO90GXsb19fznrNKcGzS7lKcK6flFDZkuu5wXAS0hQXWKe?=
 =?us-ascii?Q?bRWZRRTgAyGpCXrUex/DzEd/Vn9TlzK8aP2M1Sev6wpHgJgbxrRcghQmxrqu?=
 =?us-ascii?Q?jSgJXJKGKKcOW0XycuoDMwhWZkHLlVMqhOGtI/bquiggrr5YgTxTzhkDvzM9?=
 =?us-ascii?Q?CkjFeln3xz2QVnPwtEIH6UNLCnZUgxgM8dVHC6zAxucu2KlRBZ7eNqdTVBiF?=
 =?us-ascii?Q?XCeXdguLofKShMe6OZ7WitItZ9YY/GbIO+f1GUOPJeIUHVCiCEd7ee2hvzHH?=
 =?us-ascii?Q?Xw8RG112T9se2z/6VtkSW3YbM0krL7yeG+7dbw+Z/BCLmHSSnkRbq2eLto51?=
 =?us-ascii?Q?rfS1HT+gan4POkj7VnSnlCqIfjpuObvxWPXs3U+a?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ba22807f-3a96-4e56-a759-08dbfbe2b211
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2023 13:52:07.2981
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mw6AAeJ8hM8ZiRpEjwDoKBKWP9AY7atJK5ns048m8Ykn414SNjezTA5SnzvV+xLW
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8815
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Dec 2023 05:56:57 -0800 (PST)

On Tue, Dec 12, 2023 at 10:01:39AM -0600, Suravee Suthikulpanit wrote:

> -	if ((flags & ~IOMMU_HWPT_ALLOC_DIRTY_TRACKING) || parent || user_data)
> +		ret = udata_to_iommu_hwpt_amd_v2(user_data, &hwpt);
> +		if (ret)
> +			return ERR_PTR(ret);
> +
> +		return amd_iommu_nested_domain_alloc(dev, &hwpt);
> +	}
> +
> +	/* Check supported flags */
> +	if (flags & (~(IOMMU_HWPT_ALLOC_NEST_PARENT |
> +		       IOMMU_HWPT_ALLOC_DIRTY_TRACKING)))
> +		return ERR_PTR(-EOPNOTSUPP);
> +
> +	if (!check_nested_support(flags))
>  		return ERR_PTR(-EOPNOTSUPP);
>  
> -	return do_iommu_domain_alloc(type, dev, flags);
> +	dom = iommu_domain_alloc(dev->bus);

Please don't call iommu_domain_alloc, call your internal function and
force it to allocate the v1 domain..

> +static int nested_gcr3_update(struct iommu_hwpt_amd_v2 *hwpt, struct iommu_domain *udom)
> +{
> +	int ret;
> +	u16 hdev_id;
> +	struct pci_dev *pdev;
> +	struct amd_iommu *iommu;
> +
> +	iommu = get_amd_iommu_from_devid(hwpt->iommu_id);
> +	hdev_id = get_hdev_id(iommu, hwpt->gid, hwpt->gdev_id);
> +
> +	pr_debug("%s: gid=%u, hdev_id=%#x, gcr3=%#llx\n",
> +		 __func__, hwpt->gid, hdev_id,
> +		 (unsigned long long) hwpt->gcr3);
> +
> +	pdev = pci_get_domain_bus_and_slot(0, PCI_BUS_NUM(hdev_id),
> +					   hdev_id & 0xff);

Huh? "hdev_id"? This is not OK..

The device you are allowed to look at is the "struct device *dev" passed
to alloc. You cannot pass in a struct device and then override it with
another value.

> +	if (!pdev)
> +		return -EINVAL;
> +
> +	/* Note: Currently only support GCR3TRPMode with nested translation */
> +	if (!check_feature2(FEATURE_GCR3TRPMODE))
> +		return -EOPNOTSUPP;
> +
> +	ret = amd_iommu_set_gcr3tbl_trp(iommu, pdev, hwpt->gcr3, hwpt->glx,
> +					hwpt->guest_paging_mode);

Waah?

This is touching the dev table? That is not right, allocation is only
*ALLOCATION*. The dev table can't be changed until you do attachment.

Please look at the smmuv3 patches and try to be structurally
similar. AMD and SMMUv3 are *very similar* in how their HW works
excluding the viommu stuff.

You also can't assume your parent is currently attached to anything.

The construction of the DTE has to be from-scratch based on the parent
domain and the provided values in the "hwpt". Again see how smmuv3
does this where there is one function that builds the entire DTE
(called STE)

I'm skeptical you can do this properly without also restructuring the
DTE logic like I've mentioned before, there is a reason I did that for
SMMUv3. :)

> +struct iommu_domain *amd_iommu_nested_domain_alloc(struct device *dev,
> +						   struct iommu_hwpt_amd_v2 *hwpt)
> +{
> +	int ret;
> +	struct iommu_domain *dom;
> +	struct protection_domain *pdom;
> +
> +	dom = iommu_domain_alloc(dev->bus);
> +	if (!dom)
> +		return ERR_PTR(-ENOMEM);

Also no, do not allocate a normal domain and then 'wreck'
it into a nesting domain. Refactor the allocation code to be in
smaller chucks so you can alloc and init the memory directly here.

Jason