Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 5 Jul 2023 13:56:50 -0700
From:   Isaku Yamahata <isaku.yamahata@gmail.com>
To:     "Huang, Kai" <kai.huang@intel.com>
Cc:     "peterz@infradead.org" <peterz@infradead.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Raj, Ashok" <ashok.raj@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "david@redhat.com" <david@redhat.com>,
        "bagasdotme@gmail.com" <bagasdotme@gmail.com>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "ak@linux.intel.com" <ak@linux.intel.com>,
        "Wysocki, Rafael J" <rafael.j.wysocki@intel.com>,
        "kirill.shutemov@linux.intel.com" <kirill.shutemov@linux.intel.com>,
        "Chatre, Reinette" <reinette.chatre@intel.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "Yamahata, Isaku" <isaku.yamahata@intel.com>,
        "nik.borisov@suse.com" <nik.borisov@suse.com>,
        "hpa@zytor.com" <hpa@zytor.com>, "Shahar, Sagi" <sagis@google.com>,
        "imammedo@redhat.com" <imammedo@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>, "Gao, Chao" <chao.gao@intel.com>,
        "Brown, Len" <len.brown@intel.com>,
        "sathyanarayanan.kuppuswamy@linux.intel.com" 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>,
        "x86@kernel.org" <x86@kernel.org>, isaku.yamahata@gmail.com
Subject: Re: [PATCH v12 20/22] x86/virt/tdx: Allow SEAMCALL to handle #UD and
 #GP
Message-ID: <20230705205650.GJ3436214@ls.amr.corp.intel.com>
References: <20230628203823.GR38236@hirez.programming.kicks-ass.net>
 <42e13ccf7f27a68c0dd64640eed378c38ef40967.camel@intel.com>
 <20230630100659.GF2533791@hirez.programming.kicks-ass.net>
 <20230630102141.GA2534364@hirez.programming.kicks-ass.net>
 <20230630120650.GB2534364@hirez.programming.kicks-ass.net>
 <fdd81fbd2424d8da04f98d156668cad5e73c740b.camel@intel.com>
 <20230705102137.GX4253@hirez.programming.kicks-ass.net>
 <ab3dea02892920cd6640a31a9c846afd6c6a9d54.camel@intel.com>
 <20230705121921.GZ4253@hirez.programming.kicks-ass.net>
 <3bef89e1bc5935625483cc5bf339c13f643c2c29.camel@intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <3bef89e1bc5935625483cc5bf339c13f643c2c29.camel@intel.com>
Precedence: bulk

On Wed, Jul 05, 2023 at 12:53:58PM +0000,
"Huang, Kai" <kai.huang@intel.com> wrote:

> On Wed, 2023-07-05 at 14:19 +0200, Peter Zijlstra wrote:
> > On Wed, Jul 05, 2023 at 11:34:53AM +0000, Huang, Kai wrote:
> > 
> > > Yeah I think from long-term's view, since SEAMCALLs to support live migration
> > > pretty much uses all RCX/RDX/R8-R15 as input/output, it seems reasonable to
> > > unify all of them, although I guess there might be some special handling to
> > > VP.VMCALL and/or VP.ENTER, e.g., below:
> > > 
> > >         /* TDVMCALL leaf return code is in R10 */                              
> > >         movq %r10, %rax
> > > 
> > > So long-termly, I don't have objection to that.  But my thinking is for the
> > > first version of TDX host support, we don't have to support all SEAMCALLs but
> > > only those involved in basic TDX support. 
> > 
> > Since those calls are out now, we should look at them now, there is no
> > point in delaying the pain. That then gives us two options:
> > 
> >  - we accept them and their wonky calling convention and our code should
> >    be ready for it.
> > 
> >  - we reject them and send the TDX team a message to please try again
> >    but with a saner calling convention.
> > 
> > Sticking our head in the sand and pretending like they don't exist isn't
> > really a viable option at this point.
> 
> OK.  I'll work on this.
> 
> But I think even we want to unify __tdx_module_call() and __tdx_hypercall(), the
> first step should be making __tdx_module_call() look like __tdx_hypercall()?  I
> mean from organizing patchset's point of view, we cannot just do in one big
> patch but need to split into small patches with each doing one thing.
> 
> By thinking is perhaps we can organize this way:
> 
>  1) Patch(es) to make TDX_MODULE_CALL macro / __tdx_module_call() look like
> __tdx_hypercall().
>  2) Add SEAMCALL support based on TDX_MODULE_CALL, e.g., implement __seamcall().
>  3) Unify __tdx_module_call()/__seamcall() with __tdx_hypercall().
> 
> Does this look good?
> 
> Btw, I've already part 1) based on your code, and sent the patches to Kirill for
> review.  Should I sent them out first?
> 
> > 
> > > Also, the new SEAMCALLs to handle live migration all seem to have below
> > > statement:
> > > 
> > > 	AVX, AVX2	May be reset to the architectural INIT state
> > > 	and
> > > 	AVX512
> > > 	state
> > > 
> > > Which means those SEAMCALLs need to preserve AVX* states too?
> > 
> > Yes, we need to ensure the userspace 'FPU' state is saved before
> > we call them. But I _think_ that KVM already does much of that.
> 
> Let me look into this.

KVM VCPU_RUN ioctl saves/restores FPU state by kvm_load_guest_fpu() and
kvm_put_guest_fpu() which calls fpu_swap_kvm_fpstate().
Other KVM ioctls doesn't modify FPU.  Because some SEAMCALLs related for live
migration don't preserve FPU state, we need explicit save/restore of FPU state.

-- 
Isaku Yamahata <isaku.yamahata@gmail.com>