Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Wed, 13 Dec 2023 09:27:21 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc:     linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
        joro@8bytes.org, yi.l.liu@intel.com, kevin.tian@intel.com,
        nicolinc@nvidia.com, eric.auger@redhat.com, vasant.hegde@amd.com,
        jon.grimm@amd.com, santosh.shukla@amd.com, Dhaval.Giani@amd.com,
        pandoh@google.com, loganodell@google.com
Subject: Re: [RFC PATCH 2/6] iommu/amd: Add support for hw_info for iommu
 capability query
Message-ID: <20231213132721.GT3014157@nvidia.com>
References: <20231212160139.174229-1-suravee.suthikulpanit@amd.com>
 <20231212160139.174229-3-suravee.suthikulpanit@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20231212160139.174229-3-suravee.suthikulpanit@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?R322ozOdouwtaWsZGmnuNIMGbg006OzAYcr4GBxhZvTsWJPnTHNWJUs6yuNf?=
 =?us-ascii?Q?O/MPnNSJllVHTt1SauZQ+Cy4LA7OA1ZnjwJFqIWjJoq8TFpeYLVYZZYlWG/5?=
 =?us-ascii?Q?DGMCiDW/esdP77jzjo8lNAFivFSRTuj8KRYJ/aS0WaKKkwE8ev6DK6mhBtqe?=
 =?us-ascii?Q?sDSWS8TlZRJmCDCqJUGXYfHu7QP7+4bTd/WEy9QtHrMs+WSCm73tfdR3b5Lu?=
 =?us-ascii?Q?psnoBQKLgnAuW31ZB+J8YlVNh06MmqV7kWUzB0FcW0YGA1yVDiDxwoY9Da0m?=
 =?us-ascii?Q?oCFIHO/F9hbNznnxjKwAFH20Qe65xn44JbRS0JUws98P+r4LeBj/MrzE4Waz?=
 =?us-ascii?Q?iO8R2//0NdHlb0Yyrtv9BXjuczrt275KdpJ6Z1BwjwxYb3TXBOnGR3/8SfzO?=
 =?us-ascii?Q?dj0acPgtJU64sMEbxbZ5yS2sMWRu8j1KY0cbNrXgRZs05PmBBtYRvH03VS8V?=
 =?us-ascii?Q?71wW2ISnOJOjO8nVfIYV77vJS4Z0mPBJqmpYLwOYm0N7WlDzLtKiA7XQgbPm?=
 =?us-ascii?Q?WPbwVEtiJK+j4AtUFDC6styRwclfalwubkwz1tLU05kAxk7ffuZyV6v/sFQ8?=
 =?us-ascii?Q?YRpQbEY5sB5zKVNUEs0uvPezOozkD5R4HzfI80Xh7MODbWGQ8HehHOgZlJ23?=
 =?us-ascii?Q?iwoFPk+xgvzd8Jfof5CiO2lNGwRs8p63XkkiEHz9hSfNKFCm3FdK03FwnjRB?=
 =?us-ascii?Q?iqaIMdIK+nHYXw5rd+7ZDFqL+hEf5Zk4ze7thk35+ZsOA/L9GexFeRsiM+fW?=
 =?us-ascii?Q?1qVR0S0k39JMkFThR02LFhdmuy+d0TE8hK00t99HvSLpO0LIQP9epE/5AzkT?=
 =?us-ascii?Q?7fOOERns+RGoKQqqQhq0dCgteSb79AtUzyyCowqcu3dj+iepnViar3VhDvie?=
 =?us-ascii?Q?U1hPqCq27vWlPlnUR3emSVR4DvvfQezqkw4jbedPUc9wL+AJ2XMQOonqR9qC?=
 =?us-ascii?Q?HVMdzqpJS3r0z9N70Z1LUODGN7zfoQlHZqZKa6T0SpnSqGGpN+Eikdep8F8+?=
 =?us-ascii?Q?LmwE49AAzOJaxel8wLupjn1uhsBSgkp7Vv6/8S3MRN5c1D6r9ZidI8YJOjAe?=
 =?us-ascii?Q?K+kNN9qyJzg4FQMjktcPqkbVcVVv36srTtwQXe1ArrIY4pJYD0JzGqvmkfVw?=
 =?us-ascii?Q?fHIvUhQ5LJJ0udDdEPjyCO6IQ7bSwndtIoyQx5UrHp2TXAQCl5GJppzqgW3Y?=
 =?us-ascii?Q?MB9BT+iiTBQ4qX29VIGD+27prSTC8sC9JVPszJYDqvZAjBIBm2CHdIuZIRCi?=
 =?us-ascii?Q?P4exahrbO2U6tnKZ+fhO+2NMyiBwpMGxJ+IthfJR/NRJdslGje+uHIkP1c5k?=
 =?us-ascii?Q?VPFSP0/jQ7YA2Qtn8Uo2XAdZsPzMQhOJfKM3Lfdc36y84IK6YO8i75pKkHlF?=
 =?us-ascii?Q?MUcVa8h/mfzQ80zpiDWgY9XKLpWrnLAopOxQufd1ggCPVfL/5lLnNYlyio5u?=
 =?us-ascii?Q?6YMfTIPFUaYuVr3LG6rPszSDYJHtTx7ueBwYVLgl5l/UJhp2WihpEsU986yO?=
 =?us-ascii?Q?lF7aXeNH89ytFTmF6C/WEzNIhuKn95CD4bXVAojbEL8U6Co482aaJXg4tJah?=
 =?us-ascii?Q?MJs/XbUCA9D35hb+QJcyhaTE8LoJaQZqb4pY7Uyf?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 58aa15ad-53db-4f50-2123-08dbfbdf3d34
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2023 13:27:22.7431
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: clFVRkSJRa+2ZPs6IFhTAZekxaCGVrkI78qHdj3T6E/DnpEvQubKuwPwIpt2icDa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB9064
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Dec 2023 05:28:12 -0800 (PST)

On Tue, Dec 12, 2023 at 10:01:35AM -0600, Suravee Suthikulpanit wrote:
> AMD IOMMU Extended Feature (EFR) and Extended Feature 2 (EFR2) registers
> specify features supported by each IOMMU hardware instance.
> The IOMMU driver checks each feature-specific bits before enabling
> each feature at run time.
> 
> For IOMMUFD, the hypervisor determines which IOMMU features to support
> in the guest, and communicates this information to user-space (e.g. QEMU)
> via iommufd IOMMU_DEVICE_GET_HW_INFO ioctl.
> 
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> ---
>  drivers/iommu/amd/amd_iommu.h       |  2 ++
>  drivers/iommu/amd/amd_iommu_types.h |  3 +++
>  drivers/iommu/amd/iommu.c           | 38 +++++++++++++++++++++++++++++
>  include/uapi/linux/iommufd.h        | 13 ++++++++++
>  4 files changed, 56 insertions(+)
> 
> diff --git a/drivers/iommu/amd/amd_iommu.h b/drivers/iommu/amd/amd_iommu.h
> index 108253edbeb0..4118129f4a24 100644
> --- a/drivers/iommu/amd/amd_iommu.h
> +++ b/drivers/iommu/amd/amd_iommu.h
> @@ -72,6 +72,8 @@ void amd_iommu_dev_flush_pasid_pages(struct iommu_dev_data *dev_data,
>  void amd_iommu_dev_flush_pasid_all(struct iommu_dev_data *dev_data,
>  				   ioasid_t pasid);
>  
> +void amd_iommu_build_efr(u64 *efr, u64 *efr2);
> +
>  #ifdef CONFIG_IRQ_REMAP
>  int amd_iommu_create_irq_domain(struct amd_iommu *iommu);
>  #else
> diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
> index 14f67a8cf755..956fd4658a4a 100644
> --- a/drivers/iommu/amd/amd_iommu_types.h
> +++ b/drivers/iommu/amd/amd_iommu_types.h
> @@ -100,12 +100,15 @@
>  #define FEATURE_HDSUP		BIT_ULL(52)
>  #define FEATURE_SNP		BIT_ULL(63)
>  
> +#define FEATURE_GATS_5LEVEL	1ULL
>  #define FEATURE_GATS_SHIFT	12
>  #define FEATURE_GATS_MASK	(0x03ULL << FEATURE_GATS_SHIFT)
>  
> +#define FEATURE_GLX_3LEVEL	0ULL
>  #define FEATURE_GLX_SHIFT	14
>  #define FEATURE_GLX_MASK	(0x03ULL << FEATURE_GLX_SHIFT)
>  
> +#define FEATURE_PASMAX_16	0xFULL
>  #define FEATURE_PASMAX_SHIFT	32
>  #define FEATURE_PASMAX_MASK	(0x1FULL << FEATURE_PASMAX_SHIFT)
>  
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index 4e4ff1550cf3..c41932e9f16a 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -2822,8 +2822,46 @@ static const struct iommu_dirty_ops amd_dirty_ops = {
>  	.read_and_clear_dirty = amd_iommu_read_and_clear_dirty,
>  };
>  
> +void amd_iommu_build_efr(u64 *efr, u64 *efr2)
> +{
> +	if (efr) {
> +		*efr = (FEATURE_GT | FEATURE_GIOSUP | FEATURE_PPR);
> +
> +		/* 5-level v2 page table support */
> +		*efr |= ((FEATURE_GATS_5LEVEL << FEATURE_GATS_SHIFT) &
> +			 FEATURE_GATS_MASK);
> +
> +		/* 3-level GCR3 table support */
> +		*efr |= ((FEATURE_GLX_3LEVEL << FEATURE_GLX_SHIFT) &
> +			 FEATURE_GLX_MASK);
> +
> +		/* 16-bit PASMAX support */
> +		*efr |= ((FEATURE_PASMAX_16 << FEATURE_PASMAX_SHIFT) &
> +			 FEATURE_PASMAX_MASK);
> +	}
> +
> +	if (efr2)
> +		*efr2 = 0;

Why are you checking for null here? It is never called with null

> +/**
> + * struct iommu_hw_info_amd - AMD IOMMU device info
> + *
> + * @efr : Value of AMD IOMMU Extended Feature Register (EFR)
> + * @efr2: Value of AMD IOMMU Extended Feature 2 Register (EFR2)

Please reference a section in the spec for what these are just for
clarity

> + */
> +struct iommu_hw_info_amd {
> +	__u64 efr;
> +	__u64 efr2;
> +};

__aligned_u64

Jason