Received-SPF: pass (google.com: domain of linux-kernel+bounces-172882-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Wed, 8 May 2024 16:08:18 +0800
From: Oliver Sang <oliver.sang@intel.com>
To: Borislav Petkov <bp@alien8.de>
CC: Sean Christopherson <seanjc@google.com>, <oe-lkp@lists.linux.dev>,
	<lkp@intel.com>, <linux-kernel@vger.kernel.org>, <x86@kernel.org>, "Ingo
 Molnar" <mingo@kernel.org>, Srikanth Aithal <sraithal@amd.com>,
	<fengwei.yin@intel.com>, <yujie.liu@intel.com>, <oliver.sang@intel.com>
Subject: Re: [tip:x86/alternatives] [x86/alternatives] ee8962082a:
 WARNING:at_arch/x86/kernel/cpu/cpuid-deps.c:#do_clear_cpu_cap
Message-ID: <Zjsy8v0pZHPMkjPY@xsang-OptiPlex-9020>
References: <ZjE7DkTBSbPlBN8k@google.com>
 <20240430193211.GEZjFHO0ayDXtgvbE7@fat_crate.local>
 <ZjFLpkgI3Zl4dsXs@google.com>
 <20240430223305.GFZjFxoSha7S5BYbIu@fat_crate.local>
 <20240504124822.GAZjYulrGPPX_4w4zK@fat_crate.local>
 <ZjiCJz4myN2DLnZ5@xsang-OptiPlex-9020>
 <Zjj3Lrv2NNHLEzzk@google.com>
 <20240506155759.GOZjj-B_Qrz4DCXwmb@fat_crate.local>
 <ZjnTW4XQwVHEiSaW@xsang-OptiPlex-9020>
 <20240507114814.GBZjoU_u5kYBhLips3@fat_crate.local>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240507114814.GBZjoU_u5kYBhLips3@fat_crate.local>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?D7l4VqsarEaUrtwPIu5JYBsoab/F/PT3xnzalkvlV94Fm/bixyc6Tcg8lD3C?=
 =?us-ascii?Q?2UyWMN7961YjwVUTpQeihuq3Z4ueS4NEkkkIsKGgxM9M3pVcSgyRW2OjmytQ?=
 =?us-ascii?Q?0v/cEFizPSHNz73LLN4Tz0Gqf5dAhpLHNuxJiZs5eb6Wfo+lcxrNmn7o2TKB?=
 =?us-ascii?Q?YTLM0vTfyTgTAADVH+gI2s/LKD5CGRbukyJ+bFDyoIe3xNz52sIWcs+qhhJZ?=
 =?us-ascii?Q?cMsTa9K39dUMdA8kkVXeo/WH74MYJPTWS4mgypPLSEeHGzZ2QcOSIMzbOoE3?=
 =?us-ascii?Q?fJr9aa8XMrd4PTx08FEZLYqXkEPHhrT97FCzkpZiIFQPYWJ01llEk0+ZBozh?=
 =?us-ascii?Q?jfdAtmXinNIyZUi7iFy4A638YrKVBz7RsycVGZc42Lt0daLQzYWqMuHKdFdS?=
 =?us-ascii?Q?4w61Fy/lnflqZzs3g9jfhCdZ23sdZGDomXwddrcSUSS002B2KKyaMYDD1JJA?=
 =?us-ascii?Q?fVpKGxhqKEy0oEuqOgHRrdtmYsCkyboO+SFeK6wDhFyPvcf9KmYvljF+iWer?=
 =?us-ascii?Q?2GLNRdzPZLUVFz8B/WfLx4ns3L90eXxQTR/x6tDw6w46F5APCZIswJjmlyTd?=
 =?us-ascii?Q?ZPyr/w4yZDWH3tmkbfp+zQa0AYFe/spB6BH4Iahqow+WP8IDV+87h2Po4Mkw?=
 =?us-ascii?Q?k1ZLyGx9t548fWe8lsSap3a+mgI/m8Z+jbiiVSjFigyK4C8HVcQp90djvTuB?=
 =?us-ascii?Q?HpElo7KGkq5uNzPmL1rdNjMcxpC+s1wGWjPvswTaIUK4V8yrNM15lEedBQPD?=
 =?us-ascii?Q?MXGQ3FAXDgTZeBr5+ZBBfQT/2i+YZIJa+J4n4NuLLes8W/+k4sDJ6pepxTW7?=
 =?us-ascii?Q?eyXsD8spu4wkFm/p3M32BlTO3rYJ1fto31wB/H/RzXFC64EQt46nWmdhTRG1?=
 =?us-ascii?Q?kb6AHLdHGvIGlLEktSEd8kFlJ6XN3xO0d0ls64DUznf/iBvdzLZTh5Muo1+V?=
 =?us-ascii?Q?PE4zJTW3BhSd8I16vNM7tL67hqM8t+eDfaoNiHh5eHKGn0hueUGyQOF6LA5L?=
 =?us-ascii?Q?j4lsu4an25eMFCpgX+anwOvJAlN3DWCdhm7f7S9lLtZ2hZJ2RcQCx6J9ayzF?=
 =?us-ascii?Q?31Rg+j4r2ZZnC4Dy2e97dw7cvSE21n+r5nJJ56RN4ua927ApVbuQnpbwxbu8?=
 =?us-ascii?Q?QTK1D2w1aPx7/qSppMdj5OpS18QzyXqYmHrPHdQdSJF7SNiM2QiuWE1N+366?=
 =?us-ascii?Q?NqXsLhe/7QBM7u+5M3nxjmA8NzE778D1jFP9YvuEH9w6H8RZVI/H61eITj5Q?=
 =?us-ascii?Q?RVh79trZAf/MdumF+m/xRADm+7ylkibfMfgFuUjpmLAPTswkOKMziwXHDtRi?=
 =?us-ascii?Q?zaBBrhq6y3stBSneqI9eBeXTKyrvdh7SmvvxfG8CcZLdhTEoD+X7tjCqZmTL?=
 =?us-ascii?Q?OlkS5Pzpj5uyztkz64Kn46yQQDnr7ncCO0eEFYZfngQJBiDzQ/v53UH/fbci?=
 =?us-ascii?Q?N13T4fY26bS6a3g9Ejr/FVihav/7K/F4ygwv/o8MPmNuxTkos9dPuudVbslY?=
 =?us-ascii?Q?cwG6bva9G/SsqmjxtKj857py839EVCAJgBK7Qz5b9m8GCjDoRdvljnly6BSL?=
 =?us-ascii?Q?TNDYNmAXQXK58Sa7jDnUJW8ntbK1L5l/J7KjgUF9nXreeY2yWWMp94r3iTp3?=
 =?us-ascii?Q?5Q=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8c41509a-ff29-4dcf-71d2-08dc6f360ae9
X-MS-Exchange-CrossTenant-AuthSource: LV3PR11MB8603.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2024 08:08:28.5064
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: BXJOs1f3s1C9KiWiHqonNw5h1PcvNoqSjGzhHsVux1j5bh3aXDiey6qBWEZdSJT0CFuYQ4GmXl+eJmjSgQfOow==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7382
X-OriginatorOrg: intel.com


hi, Boris,


very sorry that this could be a wrong report due to BIOS issues.
please just ignore it.


thanks a lot for reminding us below:

> BUT(!), look what the original dmesg said:
> 
> [    0.055225][    T0] x86/cpu: VMX (outside TXT) disabled by BIOS
> 
> So that FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX bit was not set back then. Why?
> 
> Oliver, have you done any BIOS config changes in the meantime?

we really don't do any manual change recently, however, per your reminding,
we rerun tests on both ee8962082a and its parent v6.9-rc3 again.

while we made original report, we always saw below for both commits.
"x86/cpu: VMX (outside TXT) disabled by BIOS"

by rerun today, we cannot see it again, on both commits.

then for ee8962082a, the reported
  WARNING:at_arch/x86/kernel/cpu/cpuid-deps.c:#do_clear_cpu_cap
disappeared also while no "x86/cpu: VMX (outside TXT) disabled by BIOS"

we are using kexec to boot into kernel, and the kernel images keep same,
so we are sure this strage phenomenon is not caused by kernel build.

now we doubt it's really a bios issue, we found another similar machine,
which also show "x86/cpu: VMX (outside TXT) disabled by BIOS", but after
several rounds of cold reboot, the message disappeared, too.

we will investigate this BIOS further and avoid this kind of misleading
report in the future.

sorry again :(


On Tue, May 07, 2024 at 01:48:14PM +0200, Borislav Petkov wrote:
> From: Borislav Petkov <bp@alien8.de>
> To: Oliver Sang <oliver.sang@intel.com>
> Cc: Sean Christopherson <seanjc@google.com>, oe-lkp@lists.linux.dev,
> 	lkp@intel.com, linux-kernel@vger.kernel.org, x86@kernel.org,
> 	Ingo Molnar <mingo@kernel.org>, Srikanth Aithal <sraithal@amd.com>
> Bcc: bp@alien8.de
> Subject: Re: [tip:x86/alternatives] [x86/alternatives] ee8962082a:
>  WARNING:at_arch/x86/kernel/cpu/cpuid-deps.c:#do_clear_cpu_cap
> Reply-To: 
> In-Reply-To: <ZjnTW4XQwVHEiSaW@xsang-OptiPlex-9020>
> 
> On Tue, May 07, 2024 at 03:08:11PM +0800, Oliver Sang wrote:
> > I applied the debug pach ontop of lastest Linus master:
> > 
> > 1621a826233a7 debug patch from Boris for ee8962082a
> > dccb07f2914cd (HEAD, linus/master) Merge tag 'for-6.9-rc7-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
> > 
> > attached dmesg and cpuinfo (a little diff, so I attached it again)
> 
> Thanks, now what are we seeing here:
> 
> [    0.763720][    T0] x86/cpu: init_ia32_feat_ctl: CPU0: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU1: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU2: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU3: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU4: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU5: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU6: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU7: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU8: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU9: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU10: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU11: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU12: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU13: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU14: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU15: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU16: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU17: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU18: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU19: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU20: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU21: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU22: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU23: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU24: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU25: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU26: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU27: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU28: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU29: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU30: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU31: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU32: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU33: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU34: FEAT_CTL: 0x5, tboot: 0
> [    0.055231][    T0] x86/cpu: init_ia32_feat_ctl: CPU35: FEAT_CTL: 0x5, tboot: 0
> 
> So following the code in init_ia32_feat_ctl(), the BSP'll get to
> 
> 	if (msr & FEAT_CTL_LOCKED)
> 		goto update_caps;
> 
> and that is the case - FEAT_CTL_LOCKED, bit 0, is set.
> 
> It'll go to the update_caps label and there it'll do:
> 
> 	if (!cpu_has(c, X86_FEATURE_VMX))
> 		goto update_sgx;
> 
> VMX is set if I judge by the attached cpuinfo-2 so the next check comes:
> 
>         if ( (tboot && !(msr & FEAT_CTL_VMX_ENABLED_INSIDE_SMX)) ||
>             (!tboot && !(msr & FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX))) {
>                 if (IS_ENABLED(CONFIG_KVM_INTEL))
>                         pr_err_once("VMX (%s TXT) disabled by BIOS\n",
>                                     tboot ? "inside" : "outside");
>                 clear_cpu_cap(c, X86_FEATURE_VMX);
> 
> tboot is 0, so the second conditional:
> 
> 	(!tboot && !(msr & FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX))
> 
> FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX, bit 2 is set. So that conditional is
> not true either. And the pr_err_once() doesn't appear in dmesg.
> 
> BUT(!), look what the original dmesg said:
> 
> [    0.055225][    T0] x86/cpu: VMX (outside TXT) disabled by BIOS
> 
> So that FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX bit was not set back then. Why?
> 
> Oliver, have you done any BIOS config changes in the meantime?
> 
> This all looks really weird.
> 
> The other possibility would be if something changed between -rc3
> (the branch x86/alternatives is based on) and -rc7. Unlikely but by now
> everything's possible.
> 
> What could also be the case is, the BSP's FEAT_CTL is 0x0 (unconfigured,
> whatever), we'd go in, set FEAT_CTL_LOCKED and that'll lock the bit in
> all FEAT_CTLs on all cores, then it'll set
> FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX. I'm presuming microcode forces this
> and am obviously grasping at straws...
> 
> Then CPU1 will come, FEAT_CTL_LOCKED will be set but
> FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX won't be set, leading to the warn.
> 
> But then again, SDM says that that MSR's scope is "Thread" which means,
> locking that MSR on the BSP won't have effect on the same MSR on the
> other HT thread.
> 
> Weird.
> 
> Ok, here's a bit modified debug patch ontop of the alternatives branch:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/log/?h=feat_ctl
> 
> please run it and send me dmesg again.
> 
> Thx.
> 
> -- 
> Regards/Gruss,
>     Boris.
> 
> https://people.kernel.org/tglx/notes-about-netiquette