Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Date:   Tue, 5 Dec 2023 09:31:56 +0800
From:   Yan Zhao <yan.y.zhao@intel.com>
To:     Sean Christopherson <seanjc@google.com>
CC:     Jason Gunthorpe <jgg@nvidia.com>, <iommu@lists.linux.dev>,
        <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <alex.williamson@redhat.com>, <pbonzini@redhat.com>,
        <joro@8bytes.org>, <will@kernel.org>, <robin.murphy@arm.com>,
        <kevin.tian@intel.com>, <baolu.lu@linux.intel.com>,
        <dwmw2@infradead.org>, <yi.l.liu@intel.com>
Subject: Re: [RFC PATCH 00/42] Sharing KVM TDP to IOMMU
Message-ID: <ZW59jO6FIVHI7Y6J@yzhao56-desk.sh.intel.com>
Reply-To: Yan Zhao <yan.y.zhao@intel.com>
References: <20231202091211.13376-1-yan.y.zhao@intel.com>
 <20231204150800.GD1493156@nvidia.com>
 <ZW4AeZfCYgv6zcy4@google.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZW4AeZfCYgv6zcy4@google.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?js1UT1W9207aq5p7Ilej0sYFfov8iYRDotSMnzVuPmilyNRsGOcX5MRYFEOa?=
 =?us-ascii?Q?MAmvUMBNoFTl/5OWkFvaDNnPXuI8omHWmiyDbWy+I1aTm/AyyWvKazTmwr0s?=
 =?us-ascii?Q?Crp5w21aj92FyBRCo17eGEVk0A+x537vFaeGWxLAmtc7ZN0FNGheb5muwqzx?=
 =?us-ascii?Q?JMExMs0JOyFfcM9jq/jrHk2WcypaD0HA9EWG60ud82sN+LTBtzZXPpqjWfG0?=
 =?us-ascii?Q?QTHKlCCA/Eo/93p31/W2d2raYs8VzA6jYIc8RwwTKyYwJbHk+BrEABksycp5?=
 =?us-ascii?Q?JfXKAiij68mogc3Lm8VEHOzLDyIa8VHWB/OMsLsEAZNecGX7EJdo3uZLIwHD?=
 =?us-ascii?Q?5F4X25rwf6soycxtiL4fdqLGVjNNgSK4CNjid9RMXyT0+7fzaJUr43ScgGHA?=
 =?us-ascii?Q?Od+oosmdHPxZU+YyPrrMmu12iNX5ihz6e5bVWU8T55SttEjYrFzi4WEHtQL8?=
 =?us-ascii?Q?IUonXlLDAZQVvRJ82AzuGvuCA4WM4o7hkCng/xW89v4Tmy/rCeTKVGLR/Zm2?=
 =?us-ascii?Q?W2vFWhDfg/+6lNHHY4WYG/Ex5FALfdY1mLAjkkJNBFIv/Hn5OdWvW/wZoL6Z?=
 =?us-ascii?Q?W1lserqMkmaVA4Qar+mNO/rr044B5M8hUnbxMMWWRvDZDCz6hSftHCLAjcYw?=
 =?us-ascii?Q?w1By+bpPPzMQyjoRHFDttsI1zp7GqRO/1EhtgzWObuQRUHanwmSWz3JLUSiq?=
 =?us-ascii?Q?d2nLXWbnAYmbfBRxJYZWLhwv1T4jPEqLYGddGuuLEZDpUpXeCbpDAHEd8NFE?=
 =?us-ascii?Q?MW3fIcdHXAK3dnEn9K5lk/Qs633HwkVQNRZ0snkz9zRokKf9+XSBVXe61NWW?=
 =?us-ascii?Q?Y17roP7PfnALsYqZqQevN3DWuUAmuXDLfm22JHUelo8dnZQeyBtbSRZGgMN2?=
 =?us-ascii?Q?usVE1UuCUkH98BdaqIziZsPjvSxVabDV/St9RHBXv0alzmGgtS4k6FpnSvKX?=
 =?us-ascii?Q?iHWGzDuWuYrcgc+kCl1jloussN/KmQvkOQqpjRakjgdyJzdnjV+g1IwZuolt?=
 =?us-ascii?Q?H1OaKahmA8lytIh+WFjh33cscsy76pcQ9Q1lTplHOmBHMuZW/UDaRcTeqqRS?=
 =?us-ascii?Q?fXs1nN0URKf3cWztkjavQ9CkT0lWZXVELsrd2WiX1jPRgvuIdPpcsnp0DXUh?=
 =?us-ascii?Q?66jpI+flb3FlVYAeF3u0AgN7ksNSUD5360WmZkettnYxJ3oEmu8Zky7mr/jS?=
 =?us-ascii?Q?lGvYGPzVn9RkCXO14n18cKmxM1y9j4wIPsNQ70/DGuO6rk5iHKVGF+4pqJrV?=
 =?us-ascii?Q?yKjcDGueCC1xLQH0g5z76bJmWMLhvPtNxP4LFas17M9sbWCPr3m930MiMTRc?=
 =?us-ascii?Q?odrqR8MXl8CrOiksgvgsxv0dehEw5RDBMG5Cd/3rxFqRe5TCFkhwkD4dQixT?=
 =?us-ascii?Q?7XudfnneLwiajct4rDaHIOi4P9BHdp9xNkOumyt4LIBC3Z9ss5RHgdqFcyOi?=
 =?us-ascii?Q?lAvXbxWBWw2RD60r1zhtAZqb/8c0Rrhaz6n8+XYzZktd6fQqQl/hdBe+PTGt?=
 =?us-ascii?Q?sXTaDGnzwyHtXQEarxI2RMAYCyg8nveVl6mUhopBiYTccfr73hKC5IxsTKz3?=
 =?us-ascii?Q?SdVgZNbbzfuCeKPSZI6CzKix4ATrFTjT0Hpl0/bW?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bdf8b831-e485-4147-3dd0-08dbf536064c
X-MS-Exchange-CrossTenant-AuthSource: DS7PR11MB5966.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 02:00:58.7420
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 92YaK3YDB1uq1/Kr7gBeqgqMYr2ebMFzNiLcFTlVeeJxjNAU55nP8raKtPKhN9AqMmcye5oSxnLDg9GiQ3QZgQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5339
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 18:01:16 -0800 (PST)

On Mon, Dec 04, 2023 at 08:38:17AM -0800, Sean Christopherson wrote:
> On Mon, Dec 04, 2023, Jason Gunthorpe wrote:
> > On Sat, Dec 02, 2023 at 05:12:11PM +0800, Yan Zhao wrote:
> > > In this series, term "exported" is used in place of "shared" to avoid
> > > confusion with terminology "shared EPT" in TDX.
> > > 
> > > The framework contains 3 main objects:
> > > 
> > > "KVM TDP FD" object - The interface of KVM to export TDP page tables.
> > >                       With this object, KVM allows external components to
> > >                       access a TDP page table exported by KVM.
> > 
> > I don't know much about the internals of kvm, but why have this extra
> > user visible piece?
> 
> That I don't know, I haven't looked at the gory details of this RFC.
> 
> > Isn't there only one "TDP" per kvm fd?
> 
> No.  In steady state, with TDP (EPT) enabled and assuming homogeneous capabilities
> across all vCPUs, KVM will have 3+ sets of TDP page tables *active* at any given time:
> 
>   1. "Normal"
>   2. SMM
>   3-N. Guest (for L2, i.e. nested, VMs)
Yes, the reason to introduce KVM TDP FD is to let KVM know which TDP the user
wants to export(share).

For as_id=0 (which is currently the only supported as_id to share), a TDP with
smm=0, guest_mode=0 will be chosen.

Upon receiving the KVM_CREATE_TDP_FD ioctl, KVM will try to find an existing
TDP root with role specified by as_id 0. If there's existing TDP with the target
role found, KVM will just export this one; if no existing one found, KVM will
create a new TDP root in non-vCPU context.
Then, KVM will mark the exported TDP as "exported".


                                         tdp_mmu_roots                           
                                             |                                   
 role | smm | guest_mode              +------+-----------+----------+            
------|-----------------              |      |           |          |            
  0   |  0  |  0 ==> address space 0  |      v           v          v            
  1   |  1  |  0                      |  .--------.  .--------. .--------.       
  2   |  0  |  1                      |  |  root  |  |  root  | |  root  |       
  3   |  1  |  1                      |  |(role 1)|  |(role 2)| |(role 3)|       
                                      |  '--------'  '--------' '--------'       
                                      |      ^                                   
                                      |      |    create or get   .------.       
                                      |      +--------------------| vCPU |       
                                      |              fault        '------'       
                                      |                            smm=1         
                                      |                       guest_mode=0       
                                      |                                          
          (set root as exported)      v                                          
.--------.    create or get   .---------------.  create or get   .------.        
| TDP FD |------------------->| root (role 0) |<-----------------| vCPU |        
'--------'        fault       '---------------'     fault        '------'        
                                      .                            smm=0         
                                      .                       guest_mode=0       
                                      .                                          
                 non-vCPU context <---|---> vCPU context                         
                                      .                                          
                                      .                       

No matter the TDP is exported or not, vCPUs just load TDP root according to its
vCPU modes.
In this way, KVM is able to share the TDP in KVM address space 0 to IOMMU side.

> The number of possible TDP page tables used for nested VMs is well bounded, but
> since devices obviously can't be nested VMs, I won't bother trying to explain the
> the various possibilities (nested NPT on AMD is downright ridiculous).
In future, if possible, I wonder if we can export an TDP for nested VM too.
E.g. in scenarios where TDP is partitioned, and one piece is for L2 VM.
Maybe we can specify that and tell KVM the very piece of TDP to export.

> Nested virtualization aside, devices are obviously not capable of running in SMM
> and so they all need to use the "normal" page tables.
>
> I highlighted "active" above because if _any_ memslot is deleted, KVM will invalidate
> *all* existing page tables and rebuild new page tables as needed.  So over the
> lifetime of a VM, KVM could theoretically use an infinite number of page tables.
Right. In patch 36, the TDP root which is marked as "exported" will be exempted
from "invalidate". Instead, an "exported" TDP just zaps all leaf entries upon
memory slot removal.
That is to say, for an exported TDP, it can be "active" until it's unmarked as
exported.