Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 5 Apr 2023 13:07:33 -0700
From:   Isaku Yamahata <isaku.yamahata@gmail.com>
To:     "Huang, Kai" <kai.huang@intel.com>
Cc:     "isaku.yamahata@gmail.com" <isaku.yamahata@gmail.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "Shahar, Sagi" <sagis@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Aktas, Erdem" <erdemaktas@google.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "zhi.wang.linux@gmail.com" <zhi.wang.linux@gmail.com>,
        "dmatlack@google.com" <dmatlack@google.com>,
        "Yamahata, Isaku" <isaku.yamahata@intel.com>
Subject: Re: [PATCH v13 003/113] KVM: TDX: Initialize the TDX module when
 loading the KVM intel kernel module
Message-ID: <20230405200733.GJ1112017@ls.amr.corp.intel.com>
References: <20ebae70fd625f8a0fe87f98c25613a2d4dc5792.camel@intel.com>
 <20230315072711.GF3922605@ls.amr.corp.intel.com>
 <8ee89a1376babf0a5dbc2feb614890b7e2ccf2f8.camel@intel.com>
 <20230316002702.GA197448@ls.amr.corp.intel.com>
 <3ebe8d34ecf199b924f4892ce911077005526628.camel@intel.com>
 <20230328235839.GA1069687@ls.amr.corp.intel.com>
 <5896fb851d20de4aab55307a73e2b4a4243ca155.camel@intel.com>
 <20230329215609.GB1069687@ls.amr.corp.intel.com>
 <f02637a56848cff1a39a2c7e2350c11b180a87ea.camel@intel.com>
 <20230330010438.GC1112017@ls.amr.corp.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20230330010438.GC1112017@ls.amr.corp.intel.com>
Precedence: bulk

On Wed, Mar 29, 2023 at 06:04:38PM -0700,
Isaku Yamahata <isaku.yamahata@gmail.com> wrote:

> On Wed, Mar 29, 2023 at 11:17:31PM +0000,
> "Huang, Kai" <kai.huang@intel.com> wrote:
> 
> > On Wed, 2023-03-29 at 14:56 -0700, Isaku Yamahata wrote:
> > > On Wed, Mar 29, 2023 at 01:13:45AM +0000,
> > > "Huang, Kai" <kai.huang@intel.com> wrote:
> > > 
> > > > > 
> > > > > > >  
> > > > > > > +	/*
> > > > > > > +	 * TDX requires those methods to enable VMXON by
> > > > > > > +	 * kvm_hardware_enable/disable_all()
> > > > > > > +	 */
> > > > > > > +	static_call_update(kvm_x86_check_processor_compatibility,
> > > > > > > +			   ops->runtime_ops->check_processor_compatibility);
> > > > > > > +	static_call_update(kvm_x86_hardware_enable,
> > > > > > > +			   ops->runtime_ops->hardware_enable);
> > > > > > > +	static_call_update(kvm_x86_hardware_disable,
> > > > > > > +			   ops->runtime_ops->hardware_disable);
> > > > > > >  	r = ops->hardware_setup();
> > > > > > >  	if (r != 0)
> > > > > > >  		goto out_mmu_exit;
> > > > > > 
> > > > > > Hmm.. I think this is ugly.  Perhaps we should never do any
> > > > > > static_call(kvm_x86_xxx)() in hardware_setup(), because hardware_setup() is
> > > > > > called before kvm_ops_update() and may update vendor's kvm_x86_ops.
> > > > > > 
> > > > > > So probably use hardware_enable_all() in hardware_setup() is a bad idea.
> > > > > > 
> > > > > > I think we have below options on how to handle:
> > > > > > 
> > > > > > 1) Use VMX's kvm_x86_ops directly in tdx_hardware_setup().  For instance,
> > > > > > something like below:
> > > > > > 
> > > > > > int __init tdx_hardware_setup(struct kvm_x86_ops *x86_ops)
> > > > > > {
> > > > > > 	...
> > > > > > 
> > > > > > 	cpus_read_lock();
> > > > > > 	r = on_each_cpu(vt_x86_ops.hardware_enable, ...);
> > > > > > 	if (!r)
> > > > > > 		r = tdx_module_setup();
> > > > > > 	on_each_cpu(vt_x86_ops.hardware_disable, ...);
> > > > > > 	cpus_read_unlock();
> > > > > > 
> > > > > > 	...
> > > > > > }
> > > > > > 
> > > > > > But this doesn't clean up nicely when there's some particular cpus fail to do
> > > > > > hardware_enable().  To clean up nicely, we do need additional things similar to
> > > > > > the hardware_enable_all() code path: a per-cpu variable or a cpumask_t + a
> > > > > > wrapper of vt_x86_ops->hardware_enable() to track which cpus have done
> > > > > > hardware_enable() successfully.
> > > > > > 
> > > > > > 2) Move those static_call_update() into tdx_hardware_setup() so they are TDX
> > > > > > code self-contained.  But this would require exposing kvm_x86_ops as symbol,
> > > > > > which isn't nice either.
> > > > > > 
> > > > > > 3) Introduce another kvm_x86_init_ops->hardware_post_setup(), which is called
> > > > > > after kvm_ops_update().
> > > > > > 
> > > > > > Personally, I think 3) perhaps is the most elegant one, but not sure whether
> > > > > > Sean/Paolo has any opinion.
> > > > > 
> > > > > I think we can simply update the ops before calling hardware_enable() and
> > > > > clean up ops on failure.
> > > > > 
> > > > > 
> > > > 
> > > > This doesn't work because hardware_setup() may update vendor's kvm_x86_ops.
> > > > 
> > > > If you do kvm_ops_update() before hardware_setup(), you need to manually update
> > > > those updated (in hardware_setup()) callbacks again after. 
> > > 
> > > We can call kvm_ops_update() twice before and after hardware_setup().
> > > 
> > 
> > Personally I think it's too ugly.  
> 
> So you prefer the option 3 to calling kvm_ops_update() twice. Okay, I'll update
> the patch.

After playing with hardware_post_setup(), it's inevitable to call
kvm_ops_update() twice.
When VMX initialization succeeded with hardware_setup(), but TDX initialization
with hardware_post_setup() failed, we'd like to support only VMX with warning
message.  In such case, we need to revert x86_ops to VMX only.
It doesn't make sense to introduce hardware_post_setup() to avoid calling
kvm_update_ops twice.
-- 
Isaku Yamahata <isaku.yamahata@gmail.com>