Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Message-ID: <039eaa7c35020774b74dc5e2d03bb0ecfa7c6d60.camel@redhat.com>
Subject: Re: [PATCH v7 02/26] x86/fpu/xstate: Refine CET user xstate bit
 enabling
From:   Maxim Levitsky <mlevitsk@redhat.com>
To:     "Yang, Weijiang" <weijiang.yang@intel.com>
Cc:     linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        dave.hansen@intel.com, pbonzini@redhat.com, seanjc@google.com,
        peterz@infradead.org, chao.gao@intel.com,
        rick.p.edgecombe@intel.com, john.allen@amd.com
Date:   Wed, 06 Dec 2023 17:57:21 +0200
In-Reply-To: <a3a14562-db72-4c19-9f40-7778f14fc516@intel.com>
References: <20231124055330.138870-1-weijiang.yang@intel.com>
         <20231124055330.138870-3-weijiang.yang@intel.com>
         <c22d17ab04bf5f27409518e3e79477d579b55071.camel@redhat.com>
         <cdf53e44-62d0-452d-9c06-5c2d2ce3ce66@intel.com>
         <20d45cb6adaa4a8203822535e069cdbbf3b8ba2d.camel@redhat.com>
         <a3a14562-db72-4c19-9f40-7778f14fc516@intel.com>
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) 
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Wed, 2023-12-06 at 09:03 +0800, Yang, Weijiang wrote:
> On 12/5/2023 5:53 PM, Maxim Levitsky wrote:
> > On Fri, 2023-12-01 at 14:51 +0800, Yang, Weijiang wrote:
> > > On 12/1/2023 1:26 AM, Maxim Levitsky wrote:
> > > > On Fri, 2023-11-24 at 00:53 -0500, Yang Weijiang wrote:
> > > > > Remove XFEATURE_CET_USER entry from dependency array as the entry doesn't
> > > > > reflect true dependency between CET features and the user xstate bit.
> > > > > Enable the bit in fpu_kernel_cfg.max_features when either SHSTK or IBT is
> > > > > available.
> > > > > 
> > > > > Both user mode shadow stack and indirect branch tracking features depend
> > > > > on XFEATURE_CET_USER bit in XSS to automatically save/restore user mode
> > > > > xstate registers, i.e., IA32_U_CET and IA32_PL3_SSP whenever necessary.
> > > > > 
> > > > > Note, the issue, i.e., CPUID only enumerates IBT but no SHSTK is resulted
> > > > > from CET KVM series which synthesizes guest CPUIDs based on userspace
> > > > > settings,in real world the case is rare. In other words, the exitings
> > > > > dependency check is correct when only user mode SHSTK is available.
> > > > > 
> > > > > Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
> > > > > Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
> > > > > Tested-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
> > > > > ---
> > > > >    arch/x86/kernel/fpu/xstate.c | 9 ++++++++-
> > > > >    1 file changed, 8 insertions(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
> > > > > index 73f6bc00d178..6e50a4251e2b 100644
> > > > > --- a/arch/x86/kernel/fpu/xstate.c
> > > > > +++ b/arch/x86/kernel/fpu/xstate.c
> > > > > @@ -73,7 +73,6 @@ static unsigned short xsave_cpuid_features[] __initdata = {
> > > > >    	[XFEATURE_PT_UNIMPLEMENTED_SO_FAR]	= X86_FEATURE_INTEL_PT,
> > > > >    	[XFEATURE_PKRU]				= X86_FEATURE_OSPKE,
> > > > >    	[XFEATURE_PASID]			= X86_FEATURE_ENQCMD,
> > > > > -	[XFEATURE_CET_USER]			= X86_FEATURE_SHSTK,
> > > > >    	[XFEATURE_XTILE_CFG]			= X86_FEATURE_AMX_TILE,
> > > > >    	[XFEATURE_XTILE_DATA]			= X86_FEATURE_AMX_TILE,
> > > > >    };
> > > > > @@ -798,6 +797,14 @@ void __init fpu__init_system_xstate(unsigned int legacy_size)
> > > > >    			fpu_kernel_cfg.max_features &= ~BIT_ULL(i);
> > > > >    	}
> > > > >    
> > > > > +	/*
> > > > > +	 * CET user mode xstate bit has been cleared by above sanity check.
> > > > > +	 * Now pick it up if either SHSTK or IBT is available. Either feature
> > > > > +	 * depends on the xstate bit to save/restore user mode states.
> > > > > +	 */
> > > > > +	if (boot_cpu_has(X86_FEATURE_SHSTK) || boot_cpu_has(X86_FEATURE_IBT))
> > > > > +		fpu_kernel_cfg.max_features |= BIT_ULL(XFEATURE_CET_USER);
> > > > > +
> > > > >    	if (!cpu_feature_enabled(X86_FEATURE_XFD))
> > > > >    		fpu_kernel_cfg.max_features &= ~XFEATURE_MASK_USER_DYNAMIC;
> > > > >    
> > > > I am curious:
> > > > 
> > > > Any reason why my review feedback was not applied even though you did agree
> > > > that it is reasonable?
> > > My apology! I changed the patch per you feedback but found XFEATURE_CET_USER didn't
> > > work before sending out v7 version, after a close look at the existing code:
> > > 
> > >           for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) {
> > >                   unsigned short cid = xsave_cpuid_features[i];
> > > 
> > >                   /* Careful: X86_FEATURE_FPU is 0! */
> > >                   if ((i != XFEATURE_FP && !cid) || !boot_cpu_has(cid))
> > >                           fpu_kernel_cfg.max_features &= ~BIT_ULL(i);
> > >           }
> > > 
> > > With removal of XFEATURE_CET_USER entry from xsave_cpuid_features, actually
> > > above check will clear the bit from fpu_kernel_cfg.max_features.
> > Are you sure about this? If we remove the XFEATURE_CET_USER from the xsave_cpuid_features,
> > then the above loop will not touch it - it loops only over the items in the xsave_cpuid_features
> > array.
> 
> No,  the code is a bit tricky, the actual array size is XFEATURE_XTILE_DATA( ie, 18) + 1, those xfeature bits not listed in init code leave a blank entry with xsave_cpuid_features[i] == 0, so for the blank elements, the loop hits (i != XFEATURE_FP && !cid) then the relevant xfeature bit for i is cleared in fpu_kernel_cfg.max_features. I had the same illusion at first when I replied your comments in v6, and modified the code as you suggested but found the issue during tests. Please double check it.

Oh I see now. IMHO the current code is broken, or at least it violates the 
'Clear XSAVE features that are disabled in the normal CPUID' comment, because
it also clears all xfeatures which have no CPUID bit in the table (except FPU,
for which we have a workaround).


How about we do this instead:

	for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) {
		unsigned short cid = xsave_cpuid_features[i];

		if (cid && !boot_cpu_has(cid))
			fpu_kernel_cfg.max_features &= ~BIT_ULL(i);
	}


The only side effect is that this code will not clear features bits for which kernel knows
no CPUID bit but kernel anyway clears _all_ features that it knows nothing about so there
is no net change in behavior.

(That is kernel only allows XFEATURE_MASK_USER_SUPPORTED | XFEATURE_MASK_SUPERVISOR_SUPPORTED)

Best regards,
	Maxim Levitsky


> > What I suggested was that we remove the XFEATURE_CET_USER from the xsave_cpuid_features
> > and instead do this after the above loop.
> > 
> > if (!boot_cpu_has(X86_FEATURE_SHSTK) && !boot_cpu_has(X86_FEATURE_IBT))
> >     fpu_kernel_cfg.max_features &= ~BIT_ULL(XFEATURE_CET_USER);
> > 
> > Which is pretty much just a manual iteration of the loop, just instead of checking
> > for absence of single feature, it checks that both features are absent.
> > 
> > Best regards,
> > 	Maxim Levitsky
> > 
> > 
> > > So now I need
> > > to add it back conditionally.
> > > Your sample code is more consistent with existing code in style, but I don't want to
> > > hack into the loop and handle XFEATURE_CET_USER specifically.  Just keep the handling
> > > and rewording the comments which is also straightforward.
> > > 
> > > > https://lore.kernel.org/lkml/c72dfaac-1622-94cf-a81d-9d7ed81b2f55@intel.com/
> > > > 
> > > > Best regard
> > > > 	Maxim Levitsky
> > > > 
> > 
> >