Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Date:   Mon, 4 Dec 2023 11:08:00 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Yan Zhao <yan.y.zhao@intel.com>
Cc:     iommu@lists.linux.dev, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, alex.williamson@redhat.com,
        pbonzini@redhat.com, seanjc@google.com, joro@8bytes.org,
        will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com,
        baolu.lu@linux.intel.com, dwmw2@infradead.org, yi.l.liu@intel.com
Subject: Re: [RFC PATCH 00/42] Sharing KVM TDP to IOMMU
Message-ID: <20231204150800.GD1493156@nvidia.com>
References: <20231202091211.13376-1-yan.y.zhao@intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20231202091211.13376-1-yan.y.zhao@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YhDo7g9kPL2EPN9p30ot7jr5gvhjuYsrKogU7ux26Yx92M5iqT8PlPpeX5qJ?=
 =?us-ascii?Q?OMQAyPSTCk2yNVV4MSxJlB1Cy1CksLnnfquHYW61Rko3SVclbuoztKeuSN3i?=
 =?us-ascii?Q?YCB7Ejc+rS++Fudg4mgAR+YZyUCoRSIgusuRKNKVlazdP+QrMXqJ5r+IjWsx?=
 =?us-ascii?Q?p5GvzonyQXVQ7molnKnQbJhs014LMT0AACxpM+yVK97krbM8SMopxz3O1o3E?=
 =?us-ascii?Q?fmHA8cuyvp8d1/IK2HJpdb82JUtz2iKw7jXQMFoReu60wos33rgxYBTBIGBl?=
 =?us-ascii?Q?SHPjm9ZpujokDNwNtSd0JDA+40aWNtgkHqvWr+AMuwkOEbk+0DcFIc9DpCcJ?=
 =?us-ascii?Q?R+DqCD/Bi97tEFMQFPxTFfT6mKMgbz338Goq0xaEiOvHZEw0AvKo9MhxJt3a?=
 =?us-ascii?Q?XyX0fNdMLFA8IPSJZw040TViH9yCxglNv6OkO4Kz8mapJvV7YAehwZYG8pYr?=
 =?us-ascii?Q?AHQmTkbMbR4y9SkHbwjKQUCYfChrTucUju+6i3ceNaXNb0fyfNuZvWeR8oCT?=
 =?us-ascii?Q?2UxMb4vwgglKCceKYRmbc6Z0XHPhQRhqmLs+R+3m+gfzxHrh2OrpUdpOiqhk?=
 =?us-ascii?Q?/T2kYSsUz56hoZTP3sdQqEaLHS9E2wH03a/ne78EbYDuFNhk0t4OgKIsUZnj?=
 =?us-ascii?Q?DSVK6dJ+yF8xzo9mmgwfV5QkzB44R8TG+NJvXGeM0oV0xWc4WdyxEtQ3/Y3a?=
 =?us-ascii?Q?0Zh/ZRcJOWCngR7H0hCJ4w7IMDSuk+BuSPr4cxx0eSU9bpCis8BIaSoLHUcP?=
 =?us-ascii?Q?rcZN3WaqxzlJBzEYP85TmW4G2DPrs7uhd+2dB3NijDkCUTwDtuAgqBX0Rpeq?=
 =?us-ascii?Q?91veIDnr7oGTePixjia7crwLo+OndlOISQH01kpkP8KjfgfevPdKk5EMx9FL?=
 =?us-ascii?Q?p/2L5OFdqbouLEXo6C4k2VT0EE1EWQHu/TZAPe98J0AsTDJvDHK798IL4GdB?=
 =?us-ascii?Q?egFD5WgL6rFKncuhvaECzg/nXLLM8VTWKMRJVbawUjQaxbLCQydlMJnYUUR6?=
 =?us-ascii?Q?Hs8WWTXQ4rTEzD3vIQRBJj03UsaaNu7wxBLHSbTPi12lBXWx1XPjHKdHXsX8?=
 =?us-ascii?Q?orh+QeYwoH7JpNWnBqiw0bBcZIyjKVpzg/1GNm8YECEdsD02FWWcCwV+JzDO?=
 =?us-ascii?Q?+wp0x8nwIbgMUI3JlVEKC9/tWRwj3RMInKeS21os9/sDEKZAJDXnW5g7EsZY?=
 =?us-ascii?Q?v7YZhQCtT53Z7syl3S9j9wgerXPdYYl0cRR7cazW1xkm41dlmdTFiAruLbo1?=
 =?us-ascii?Q?hHEJAZg7Qg6YAak+FKgnBqLXyLvDvHzdUpqyJbScbEh+l8p6vsM2T8JHSKBH?=
 =?us-ascii?Q?Th6RIMgriZaPa/XICU/qMRnW0omnR1MMQOwJMsuFprJq9aKr78GfI9mX4Xpg?=
 =?us-ascii?Q?usX/C+GtBd9ofIF5f/NwVkxuWel4e34rYewW8k6hjbmTEhhcrNP+JPqsumOk?=
 =?us-ascii?Q?W6s2fuIa4w1h7kNsao82E2AXlxnA6jZ1FxRcpQrrNLn1533crxh92W17MH2L?=
 =?us-ascii?Q?ofFv/O+Z2lOZWUG9axmXFE1UNEfDVk8ckD+AzEq5oO8V+COlm+DWy5jj9hj9?=
 =?us-ascii?Q?XQ9/BfnM5+YayZr3/wJDtzy1Wz+7VjX8WO2t6EVe?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8f40d9c6-b43f-4399-fd22-08dbf4dacec7
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 15:08:01.3351
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: q0cN8zLQQeAX0OLfS9CbfEjtp09fOqXO6RV/0LyX3JeZEmMBPoo9YrxvP5BENLo3
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4517
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 07:08:24 -0800 (PST)

On Sat, Dec 02, 2023 at 05:12:11PM +0800, Yan Zhao wrote:
> In this series, term "exported" is used in place of "shared" to avoid
> confusion with terminology "shared EPT" in TDX.
> 
> The framework contains 3 main objects:
> 
> "KVM TDP FD" object - The interface of KVM to export TDP page tables.
>                       With this object, KVM allows external components to
>                       access a TDP page table exported by KVM.

I don't know much about the internals of kvm, but why have this extra
user visible piece? Isn't there only one "TDP" per kvm fd? Why not
just use the KVM FD as a handle for the TDP?

> "IOMMUFD KVM HWPT" object - A proxy connecting KVM TDP FD to IOMMU driver.
>                             This HWPT has no IOAS associated.
> 
> "KVM domain" in IOMMU driver - Stage 2 domain in IOMMU driver whose paging
>                                structures are managed by KVM.
>                                Its hardware TLB invalidation requests are
>                                notified from KVM via IOMMUFD KVM HWPT
>                                object.

This seems broadly the right direction

> - About device which partially supports IOPF
> 
>   Many devices claiming PCIe PRS capability actually only tolerate IOPF in
>   certain paths (e.g. DMA paths for SVM applications, but not for non-SVM
>   applications or driver data such as ring descriptors). But the PRS
>   capability doesn't include a bit to tell whether a device 100% tolerates
>   IOPF in all DMA paths.

The lack of tolerance for truely DMA pinned guest memory is a
significant problem for any real deployment, IMHO. I am aware of no
device that can handle PRI on every single DMA path. :(

>   A simple way is to track an allowed list of devices which are known 100%
>   IOPF-friendly in VFIO. Another option is to extend PCIe spec to allow
>   device reporting whether it fully or partially supports IOPF in the PRS
>   capability.

I think we need something like this.

> - How to map MSI page on arm platform demands discussions.

Yes, the recurring problem :(

Probably the same approach as nesting would work for a hack - map the
ITS page into the fixed reserved slot and tell the guest not to touch
it and to identity map it.

Jason