Received-SPF: pass (google.com: domain of linux-kernel+bounces-193638-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Wed, 29 May 2024 15:56:38 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: Isaku Yamahata <isaku.yamahata@intel.com>
CC: Chao Gao <chao.gao@intel.com>, <kvm@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <isaku.yamahata@gmail.com>, Paolo Bonzini
	<pbonzini@redhat.com>, <erdemaktas@google.com>, Sean Christopherson
	<seanjc@google.com>, Sagi Shahar <sagis@google.com>, Kai Huang
	<kai.huang@intel.com>, <chen.bo@intel.com>, <hang.yuan@intel.com>,
	<tina.zhang@intel.com>, Fengwei Yin <fengwei.yin@intel.com>,
	<isaku.yamahata@linux.intel.com>
Subject: Re: [PATCH v19 070/130] KVM: TDX: TDP MMU TDX support
Message-ID: <ZlbftscsqGQjWZqj@chenyu5-mobl2>
References: <cover.1708933498.git.isaku.yamahata@intel.com>
 <56cdb0da8bbf17dc293a2a6b4ff74f6e3e034bbd.1708933498.git.isaku.yamahata@intel.com>
 <ZgTgOVNmyVVgfvFM@chao-email>
 <ZlL2m3PKnYqc3uHr@chenyu5-mobl2>
 <20240529005519.GA386318@ls.amr.corp.intel.com>
 <20240529005859.GB386318@ls.amr.corp.intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240529005859.GB386318@ls.amr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?z7uE4OyVMPBWSKVobr7IVkCt8rBedtP/XzqPUCZgM/iqCrIx3YfPqPM8KBJh?=
 =?us-ascii?Q?f8W22RPlgOsSQBWpt2ZEGpjhTvf2HRqEhSE/3LSlneqqeM+87Y0/f2iriXkV?=
 =?us-ascii?Q?1E6k1opJY1SuZWYBwek0+F2F5W0GXO8kozChKWaU1ZfcxJK4G3UzLSsQMhYj?=
 =?us-ascii?Q?ccIOg3yJWlS0EtsKIxaC+qZM+SvUU3UuUP0l0Dzhk53UFKpJAzM6N4FAEgki?=
 =?us-ascii?Q?yi8ptnpZl5pMdG2BFcaT5QVmb1gDFcZ2XaLuJ7tBrApGm0gHxgK0eXY+Wunn?=
 =?us-ascii?Q?UnuQaQWRrc8IzOMuTzC2hFSfMZjsJ5tchXgdGLFRVDVUfqVTZuReJvm2MTRA?=
 =?us-ascii?Q?zycfPnBEoOTpeI6T3o8Qa1mzDXivTLeWGKUWvpsNw5fJB8c9Rp4IFXWhModK?=
 =?us-ascii?Q?59viiq4oPgI4p5Bwdoh6HQh4T4/ZWR1MnpmGnf2AXzYQ93rIgY5phqVIsKN+?=
 =?us-ascii?Q?yplqPYI3LWKw63wS2BQprnkwT0wrUpnGUpqGH6BZeyqCbopt6zXGb+cn2IfB?=
 =?us-ascii?Q?pSLuLx/QDnLw9p5nlYm4IGgGRXQE5Iz5te0Qmy4/btFxza9fNbKt9VwrhkOI?=
 =?us-ascii?Q?k2KvysMf+97flI9p30z8hBRW0muW/CJnV+oTONZwe+ItO2J1qh29oi4/WFA9?=
 =?us-ascii?Q?kDNuE0ujxWtNgsH0V63nwtHDxzpvzpzMqtrrbOIz1EJ7i46BGXS46uGVLn0r?=
 =?us-ascii?Q?kECvsL7k4AdSiGVOxSDVCLNQ1fC5FbfGU0WTJI66WSyn6uP8GYb9N7uyIEJp?=
 =?us-ascii?Q?XaSvG+zzJTAtGNBVtXwRRhrLnQKaWFBnFVXloQarCoE+OZSbexqtY1OF3b0c?=
 =?us-ascii?Q?9oTjqOq8BaqX1kUO9Do9M6SFOLO0XCLtcED+mJbIkG6g5Hhed6hFCgyGmjKO?=
 =?us-ascii?Q?NqeLv6nqgOb/mmdMfeaGNX/EonOPKJ6BijIaTasXEQhIz/b+IUOhpQGLxFSX?=
 =?us-ascii?Q?MV+DSA9yO8TU2nGvvzEFx1cBd0SBZLo7G6z1isFIY13Y5+GnXqRmL703vLoo?=
 =?us-ascii?Q?NuW56+WIyObhqQzoBPM0r6+sMSUwVHOtIw3ZGLTKJWeg69WpJyIizQmjnth+?=
 =?us-ascii?Q?UVp5//L5i6xpLKjhV25Na20+agt0Z0qLz0jm9tWaGag0To4oUkBH3wNotDPZ?=
 =?us-ascii?Q?KMf/rJhoB7qDlItpHu8IB+a47NilviwlYX3NwmO3t6Joifn66SVPXUIkFyud?=
 =?us-ascii?Q?uUcYyk752pr/mD3EySDgKmGC6ozjN3+alqv6+3L0O16+F8aRc8QnRjsgv+CV?=
 =?us-ascii?Q?DqbAwl/+bxFuHRhlcKwFGpPFfe5aFKE+AD9HmFkjc/BtnfO4P8nkExFsIF84?=
 =?us-ascii?Q?cAV0Pe5r9lXB1i6oZv+GEhnqn87d4BYmjhPjmHVpCb22Z7Lednmzg95CTTAw?=
 =?us-ascii?Q?xQgNNELW3apfv/0Kffuna1GOC54pINgY9/IZLM2imj+syq8mcWSsAN1jRVmV?=
 =?us-ascii?Q?pIoI7VYcRfkOKgGtrW4+GyK0ICVg1R86ZisbIUW3Pr62OWn2dq+4CuRntQ4C?=
 =?us-ascii?Q?PW1ki2BiY7fpQB4eVaKaaAncT5st60aD8Vhk+YuagtQxu6BTUGF3CMMs+ozB?=
 =?us-ascii?Q?Il2qYlEB82kEqQUuS1eAZ33VbojjQao+FNPCFCua?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8c275a27-73dd-4cad-7809-08dc7fb4e80f
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 May 2024 07:56:54.5363
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LfX/+WDM66XmwKUvv8Szz4sjfnjVl0uNSEKIesn6WUDnJl9zM+uSBFxD7/pA5NSv+kReZdC9Z5qrpDMI27nraw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB8042
X-OriginatorOrg: intel.com

On 2024-05-28 at 17:58:59 -0700, Isaku Yamahata wrote:
> On Tue, May 28, 2024 at 05:55:19PM -0700,
> Isaku Yamahata <isaku.yamahata@intel.com> wrote:
> 
> > On Sun, May 26, 2024 at 04:45:15PM +0800,
> > Chen Yu <yu.c.chen@intel.com> wrote:
> > 
> > > On 2024-03-28 at 11:12:57 +0800, Chao Gao wrote:
> > > > >+#if IS_ENABLED(CONFIG_HYPERV)
> > > > >+static int vt_flush_remote_tlbs(struct kvm *kvm);
> > > > >+#endif
> > > > >+
> > > > > static __init int vt_hardware_setup(void)
> > > > > {
> > > > > 	int ret;
> > > > >@@ -49,11 +53,29 @@ static __init int vt_hardware_setup(void)
> > > > > 		pr_warn_ratelimited("TDX requires mmio caching.  Please enable mmio caching for TDX.\n");
> > > > > 	}
> > > > > 
> > > > >+#if IS_ENABLED(CONFIG_HYPERV)
> > > > >+	/*
> > > > >+	 * TDX KVM overrides flush_remote_tlbs method and assumes
> > > > >+	 * flush_remote_tlbs_range = NULL that falls back to
> > > > >+	 * flush_remote_tlbs.  Disable TDX if there are conflicts.
> > > > >+	 */
> > > > >+	if (vt_x86_ops.flush_remote_tlbs ||
> > > > >+	    vt_x86_ops.flush_remote_tlbs_range) {
> > > > >+		enable_tdx = false;
> > > > >+		pr_warn_ratelimited("TDX requires baremetal. Not Supported on VMM guest.\n");
> > > > >+	}
> > > > >+#endif
> > > > >+
> > > > > 	enable_tdx = enable_tdx && !tdx_hardware_setup(&vt_x86_ops);
> > > > > 	if (enable_tdx)
> > > > > 		vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size,
> > > > > 					   sizeof(struct kvm_tdx));
> > > > > 
> > > > >+#if IS_ENABLED(CONFIG_HYPERV)
> > > > >+	if (enable_tdx)
> > > > >+		vt_x86_ops.flush_remote_tlbs = vt_flush_remote_tlbs;
> > > > 
> > > > Is this hook necessary/beneficial to TDX?
> > > >
> > > 
> > > I think so.
> > > 
> > > We happended to encounter the following error and breaks the boot up:
> > > "SEAMCALL (0x000000000000000f) failed: 0xc0000b0800000001"
> > > 0xc0000b0800000001 indicates the TDX_TLB_TRACKING_NOT_DONE, and it is caused
> > > by page demotion but not yet doing a tlb shotdown by tlb track.
> > > 
> > > 
> > > It was found on my system the CONFIG_HYPERV is not set, and it makes
> > > kvm_arch_flush_remote_tlbs() not invoking tdx_track() before the
> > > tdh_mem_page_demote(), which caused the problem.
> > > 
> > > > if no, we can leave .flush_remote_tlbs as NULL. if yes, we should do:
> > > > 
> > > > struct kvm_x86_ops {
> > > > ...
> > > > #if IS_ENABLED(CONFIG_HYPERV) || IS_ENABLED(TDX...)
> > > > 	int  (*flush_remote_tlbs)(struct kvm *kvm);
> > > > 	int  (*flush_remote_tlbs_range)(struct kvm *kvm, gfn_t gfn,
> > > > 					gfn_t nr_pages);
> > > > #endif
> > > 
> > > If the flush_remote_tlbs implementation are both available in HYPERV and TDX,
> > > does it make sense to remove the config checks? I thought when commit 0277022a77a5
> > > was introduced, the only user of flush_remote_tlbs() is hyperv, and now
> > > there is TDX.
> > 
> > You don't like IS_ENABLED(CONFIG_HYPERV) || IS_ENABLED(CONFIG_TDX_HOST) in many
> > places?  Then, we can do something like the followings.  Although It would be
> > a bit ugly than the commit of 0277022a77a5, it's better to keep the intention
> > of it.
> > 
> 
> Ah, we have already __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE. We can use it.

Yes, and there is also __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS so we can use
ifdef __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE when needed.

thanks,
Chenyu
> -- 
> Isaku Yamahata <isaku.yamahata@intel.com>