Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3;
Date:   Tue, 5 Dec 2023 11:51:49 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     Sean Christopherson <seanjc@google.com>
CC:     <iommu@lists.linux.dev>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <alex.williamson@redhat.com>,
        <jgg@nvidia.com>, <pbonzini@redhat.com>, <joro@8bytes.org>,
        <will@kernel.org>, <robin.murphy@arm.com>, <kevin.tian@intel.com>,
        <baolu.lu@linux.intel.com>, <dwmw2@infradead.org>,
        <yi.l.liu@intel.com>
Subject: Re: [RFC PATCH 00/42] Sharing KVM TDP to IOMMU
Message-ID: <ZW6eVexQNIqtwDaZ@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20231202091211.13376-1-yan.y.zhao@intel.com>
 <ZW4Fx2U80L1PJKlh@google.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZW4Fx2U80L1PJKlh@google.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gLsNvpFMxqVmBzuJnp5bPRNqd287UJeFKTQhd6FHDNR6NqoMvb+yzg7b9DtF?=
 =?us-ascii?Q?Dr+7M7rDVVcgdGtY9KQ65X2dO5/UeM5tgfpIEgPtxP09pYrTcmpBBaW4ZRSn?=
 =?us-ascii?Q?HMSI3dWVyx96Oz0ZeJijZivR0zK7XvllMHl+mNGktNvLAeUuflRsdBlHCpmb?=
 =?us-ascii?Q?zDWu5gBgjCGdD+t977voYyqY/U1Ui1m93073mK81IlQHgljdyf0y36PvvbF1?=
 =?us-ascii?Q?sQseVEF9VN6UQElRyh5qERKl9g2yS5N04iy6gr+ELiAJuCa6PcPO96Q0GftT?=
 =?us-ascii?Q?TJKqdjlyjAdV9NlY1xzU7D9fgsEQEIP7yUqKhZSuG+r8xF176O3HMp6cZNIi?=
 =?us-ascii?Q?xg7KXmcINYu6OJY1F9qcqcp/5ShMimdbeb8dtThV5oYZ9a5jG3o2fK3uu1eW?=
 =?us-ascii?Q?hm3wICst557AURw87KLcUTU8SaHh8D6+V9uB6/Z9zQ12JILmDPOtK35r990L?=
 =?us-ascii?Q?+OmsDnqk//k4T87cdaupKiZGmWF8hCcvcA99Vlpp5eGCggZomvsQwmmvXndO?=
 =?us-ascii?Q?ICUXVShq3WBuptFYn/q2acmDIdEQKT/R7svU0LJrhDo2GQxwYkSsa3gwVzTP?=
 =?us-ascii?Q?KbKAfaogBdPeShNhIUngiK3LUHSzoO7I2G3OjH/F4quJB9ohxXhXeaOetglM?=
 =?us-ascii?Q?3C3NrMBZO0WlDIy9CYPIG37zSgh8+/8H+cvHdwy9AQk33bZgch1afgNafzFJ?=
 =?us-ascii?Q?1JCv6PKOFzEqEr0VgAE2QxjIqPtxHbN4I5/dQi+pefDmNYyThFj96HiuAQf/?=
 =?us-ascii?Q?Uk5mbfYMkw9Y2FLpFiwbtNMvJiNrilxHH7HwCDuFUJrWdAs56nFvbjDLAhja?=
 =?us-ascii?Q?oG9ZJcIMHzBbbBscxMbPYs1TeNvORKkVxR7c2ADQLWoyo6s5sjfbEUFOohrQ?=
 =?us-ascii?Q?pVOktW1yXoo7PrK/Nto9iQAQNk0Pzr8+03Y13GIMG9+8V8PoKNGdoT3YuGwf?=
 =?us-ascii?Q?mmggnlLduJyuxFK9GjHYKuQUXlL49yACMzQO79jyJUjVCCVxUK3yEIdVdTui?=
 =?us-ascii?Q?gYitxY7paFUEJjOoD3Ycx7Lyzdvq/RYxxZKcyw2VtwZME4ODyABgCbbi3UoS?=
 =?us-ascii?Q?R1/0UXecItF2/j/xBEgbHBoG/IcXGPBi6vuEG08o3PS+lYMQv2uQ2A0xy8Fg?=
 =?us-ascii?Q?FyMjYTp2uFG7Nxa3BTvd6IaTvtYAnU5wj82mBE1kEIRpk8M/Ar68GSu0YeUX?=
 =?us-ascii?Q?vLfIjL6ip6p5uJEwqTPl2xto3y+8BWIwa516ldefjJ//c/uX0rXx6qN6nmQo?=
 =?us-ascii?Q?KbC6HskU/cU1cnhyXAIBavyeMrp8+WwQAds1M+RHh1gDJctL6mk8ZczjnN5F?=
 =?us-ascii?Q?SeChg7OHIVYX5OvhMi2CW6MbucbrWoIEG4TwFWv3Z4OkK/eP8sGAolhob5pI?=
 =?us-ascii?Q?KkCzfdyTkCX23l2NghJnNU6LiQIuQ8W2EmmZgZxLeCrAY2rRDOF++s6YtyzD?=
 =?us-ascii?Q?gpYzJ+llc3x7KVp61s7MHaEZobauQGzfO25b1//FTZEd54+UtADcS9W9ukp3?=
 =?us-ascii?Q?c0foB8TMl+ntvBdl9AAwl1bkWHKMfcbBAAD5hABSkNpv2nwDbjzKS3S7uMKP?=
 =?us-ascii?Q?7r1/7yKOagATxJL7+Cn/eDVvpf4PFm7duKOyArrF?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b5b01ee0-95d4-4670-50b0-08dbf549919d
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 04:20:53.1597
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rLDW5+NU5YEHYW88yEcocr7CNP/4d0EWhcqT06hHtswKpbrxQ1WlbboWqD+0HDeohVWs5c90OS5wz3AzHjddfQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6737
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 20:21:35 -0800 (PST)

On Mon, Dec 04, 2023 at 09:00:55AM -0800, Sean Christopherson wrote:
> On Sat, Dec 02, 2023, Yan Zhao wrote:
> Please list out the pros and cons for each.  In the cons column for piggybacking
> KVM's page tables:
> 
>  - *Significantly* increases the complexity in KVM
The complexity to KVM (up to now) are
a. fault in non-vCPU context
b. keep exported root always "active"
c. disallow non-coherent DMAs
d. movement of SPTE_MMU_PRESENT

for a, I think it's accepted, and we can see eager page split allocates
       non-leaf pages in non-vCPU context already.
for b, it requires exported TDP root to keep "active" in KVM's "fast zap" (which
       invalidates all active TDP roots). And instead, the exported TDP's leaf
       entries are all zapped.
       Though it looks not "fast" enough, it avoids an unnecessary root page
       zap, and it's actually not frequent --
       - one for memslot removal (IO page fault is unlikey to happen during VM
                                  boot-up)
       - one for MMIO gen wraparound (which is rare)
       - one for nx huge page mode change (which is rare too)
for c, maybe we can work out a way to remove the MTRR stuffs.
for d, I added a config to turn on/off this movement. But right, KVM side will
       have to sacrifice a bit for software usage and take care of it when the
       config is on.

>  - Puts constraints on what KVM can/can't do in the future (see the movement
>    of SPTE_MMU_PRESENT).
>  - Subjects IOMMUFD to all of KVM's historical baggage, e.g. the memslot deletion
>    mess, the truly nasty MTRR emulation (which I still hope to delete), the NX
>    hugepage mitigation, etc.
NX hugepage mitigation only exists on certain CPUs. I don't see it in recent
Intel platforms, e.g. SPR and GNR...
We can disallow sharing approach if NX huge page mitigation is enabled.
But if pinning or partial pinning are not involved, nx huge page will only cause
unnecessary zap to reduce performance, but functionally it still works well.

Besides, for the extra IO invalidation involved in TDP zap, I think SVM has the
same issue. i.e. each zap in primary MMU is also accompanied by a IO invalidation.

> 
> Please also explain the intended/expected/targeted use cases.  E.g. if the main
> use case is for device passthrough to slice-of-hardware VMs that aren't memory
> oversubscribed, 
>
The main use case is for device passthrough with all devices supporting full
IOPF.
Opportunistically, we hope it can be used in trusted IO, where TDP are shared
to IO side. So, there's only one page table audit required and out-of-sync
window for mappings between CPU and IO side can also be eliminated.

> > - Unified page table management
> >   The complexity of allocating guest pages per GPAs, registering to MMU
> >   notifier on host primary MMU, sub-page unmapping, atomic page merge/split
> 
> Please find different terminology than "sub-page".  With Sub-Page Protection, Intel
> has more or less established "sub-page" to mean "less than 4KiB granularity".  But
> that can't possibly what you mean here because KVM doesn't support (un)mapping
> memory at <4KiB granularity.  Based on context above, I assume you mean "unmapping
> arbitrary pages within a given range".
>
Ok, sorry for this confusion.
By "sub-page unmapping", I mean atomic huge page splitting and unmapping smaller
range in the previous huge page.