Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 22 Jun 2023 10:46:25 -0300
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc:     linux-kernel@vger.kernel.org, iommu@lists.linux.dev,
        kvm@vger.kernel.org, joro@8bytes.org, robin.murphy@arm.com,
        yi.l.liu@intel.com, alex.williamson@redhat.com,
        nicolinc@nvidia.com, baolu.lu@linux.intel.com,
        eric.auger@redhat.com, pandoh@google.com, kumaranand@google.com,
        jon.grimm@amd.com, santosh.shukla@amd.com, vasant.hegde@amd.com,
        jay.chen@amd.com, joseph.chung@amd.com
Subject: Re: [RFC PATCH 00/21] iommu/amd: Introduce support for HW
 accelerated vIOMMU w/ nested page table
Message-ID: <ZJRQseaYkTJGHNsk@nvidia.com>
References: <20230621235508.113949-1-suravee.suthikulpanit@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230621235508.113949-1-suravee.suthikulpanit@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?SHPcMxsKZRnAi3LRfvmIi1lGBkgLijW98bSFZk/Bd40oBhshWVhcLBXW92CN?=
 =?us-ascii?Q?4sd1Vv2mep+rfhB7OSYGKLX7w4C8IJdIIV4077bKpU+mHMb+D1qBo81TAuxn?=
 =?us-ascii?Q?XuEfHe6jQYEQ2oriuLAwoW4tyOUz8B7fRSEssF3AW5Z53eWp2xi0PiFnCuSA?=
 =?us-ascii?Q?2s6S9w0ysJhOg5lCB8LBeUypcMo3oF1YkcM3+oWOwJn/TzYWId27pADvP/YH?=
 =?us-ascii?Q?LPpNDjd6+Z9VYir9XEcTTyBvrIgT5Ao3btXfzGpp6khHAWx9FX/RLfNuO+7t?=
 =?us-ascii?Q?NMKM1v2E6MhA0iE/7m6FxygCXii/7gfcZYGuhfwXANNdysUA60gbu4IC4Kic?=
 =?us-ascii?Q?lxZjM7z0XxiYRWYH2AyHcjrPCfIob/ja6s5Wpnm1g8U8i0OVm+bWiBTCV6i/?=
 =?us-ascii?Q?eTtF4tTnKQKJGqFNaUPfQPKE0lZBgKA/PcLZy7skh5fWwwXs9Kan1tloUjmZ?=
 =?us-ascii?Q?clKjsGw9r3jT7uKJf9rtlH8iFu3x+J4gPMl5rPpe/O1VnneNvYs5tv3A9KsC?=
 =?us-ascii?Q?00mwjpZjR+Jw0dbJPA5377D4XLFlp3gaxXJiO1Q8vMtBiMq58kqT8tg7UWzC?=
 =?us-ascii?Q?B0VzlFujbQj0nXzLAsc/LpxX9Y5ji88uJFdv2+v7Oy4f09FiI0uqddDO7gUe?=
 =?us-ascii?Q?y3Afpzn+Vi9MjZ6fx4LruVd+tHlbOj/h7VhfVUtXgRYp4rOSCROu/fd6305d?=
 =?us-ascii?Q?doSnwcpA1XCQqAe/DSwdA3cRPzFxOdzKXFTvqOAgBV8J898JruuU2UhRhYOl?=
 =?us-ascii?Q?ecK6kBZH8JKRY4WvpHOWrEBBs/uzZJWp/N9J/kJq1Jesq//wdPOz5SdN8cvD?=
 =?us-ascii?Q?Fz5ZElXSsAGMQXDTjux1MqjqHDRrQjF86W21heHse6yz9WkojckJV9xFg9QN?=
 =?us-ascii?Q?S6b97JdxK11C1Dm4tq4BrwsXgH8wfl0YEvY6tF9kiJ4tvsDaAkpsJQgRgpDG?=
 =?us-ascii?Q?7mw7eM1lBL5rgAhPGEg4OZXXlG420RnMoczyRqHp6iFkos3U4qonUq4kLBbX?=
 =?us-ascii?Q?a1LB3FMM1hp3DPTtqPA5szu637s1dibuxrFHkB7oSQF57xc0JfPXuRBFKCnt?=
 =?us-ascii?Q?dkPmE2omVJjljLsSJshRTIglEdNk93tdRfxZiKVEDXK9mm61wXZ5G/nusg/Z?=
 =?us-ascii?Q?TSO7dJpycmCrVrnSg3TzQlreLL48QZ4SRZcnBdYN4l85h918e95s3GdYzezG?=
 =?us-ascii?Q?XmwqyPwgPCWVMDgzXMw/KOTr/nAjAxePfSIze+zL4yDUqcJ6bWILNTw6rEGG?=
 =?us-ascii?Q?pJODGhOFJzQmpjKhFEIUnDp+1tPsbJTl0RWjvdztVXE2FClkah9CngWem86H?=
 =?us-ascii?Q?oMo3+kd5c8nYKZ6bHXTCIRglSeqHKh6mLSgT7X4HswaP4uUr5Oh9KiVecGrD?=
 =?us-ascii?Q?o06NE6NwUpGcUP/wIgQra6iV7D7gUCHPoCRwl9EzOapY5dQ7XFx+DkgiX5wV?=
 =?us-ascii?Q?aBUODXcOu0Cg5z49xHmcQEUrj8JxwvmLNS5WE/KjAtGS1CCOpYlwMdY3dSv5?=
 =?us-ascii?Q?QjzIUV/dstoks1CVgR/Yp32tKF6BTG5E78xBP2f7zo1YIzCwHbaMfxSbcD6Y?=
 =?us-ascii?Q?D53kUVLqsSZ99xxg9MzAIqdg/5R5/Fn2TY65ILVH?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4fc184bf-acb1-4e6e-43e2-08db732713f9
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2023 13:46:28.4062
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LIWiboctYfS3E3Zte1fcWAZXCVg1xVDmnCXMyoMR7ZbSmATzR9moNuAtXyOseh/+
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6450
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jun 21, 2023 at 06:54:47PM -0500, Suravee Suthikulpanit wrote:

> Since the IOMMU hardware virtualizes the guest command buffer, this allows
> IOMMU operations to be accelerated such as invalidation of guest pages
> (i.e. stage1) when the command is issued by the guest kernel without
> intervention from the hypervisor.

This is similar to what we are doing on ARM as well.
 
> This series is implemented on top of the IOMMUFD framework. It leverages
> the exisiting APIs and ioctls for providing guest iommu information
> (i.e. struct iommu_hw_info_amd), and allowing guest to provide guest page
> table information (i.e. struct iommu_hwpt_amd_v2) for setting up user
> domain.
> 
> Please see the [4],[5], and [6] for more detail on the AMD HW-vIOMMU.
> 
> NOTES
> -----
> This series is organized into two parts:
>   * Part1: Preparing IOMMU driver for HW-vIOMMU support (Patch 1-8).
> 
>   * Part2: Introducing HW-vIOMMU support (Patch 9-21).
> 
>   * Patch 12 and 21 extends the existing IOMMUFD ioctls to support
>     additional opterations, which can be categorized into:
>     - Ioctls to init/destroy AMD HW-vIOMMU instance
>     - Ioctls to attach/detach guest devices to the AMD HW-vIOMMU instance.
>     - Ioctls to attach/detach guest domains to the AMD HW-vIOMMU instance.
>     - Ioctls to trap certain AMD HW-vIOMMU MMIO register accesses.
>     - Ioctls to trap AMD HW-vIOMMU command buffer initialization.

No one else seems to need this kind of stuff, why is AMD different?

Emulation and mediation to create the vIOMMU is supposed to be in the
VMM side, not in the kernel. I don't want to see different models by
vendor.

Even stuff like setting up the gcr3 should not be it's own ioctls,
that is now how we are modeling things at all.

I think you need to take smaller steps in line with the other
drivers so we can all progress through this step by step together.

To start focus only on user space page tables and kernel mediated
invalidation and fit into the same model as everyone else. This is
approx the same patches and uAPI you see for ARM and Intel. AFAICT
AMD's HW is very similar to ARM's, so you should be aligning to the
ARM design.

Then maybe we can argue if a kernel vIOMMU emulation/mediation is
appropriate or not, but this series is just too much as is.

I also want to see the AMD driver align with the new APIs for
PASID/etc before we start shovling more stuff into it. This is going
to be part of the iommufd contract as well, I'm very unhappy to see
drivers pick and choosing what part of the contract they implement.

Regards,
Jason