From:   "Tian, Kevin" <kevin.tian@intel.com>
To:     Paolo Bonzini <pbonzini@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Yamahata, Isaku" <isaku.yamahata@intel.com>,
        "Huang, Kai" <kai.huang@intel.com>,
        "Nakajima, Jun" <jun.nakajima@intel.com>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "Gao, Chao" <chao.gao@intel.com>
Subject: Q. about KVM and CPU hotplug
Thread-Topic: Q. about KVM and CPU hotplug
Thread-Index: Adflvg/SIgoQKcU9QlaunmfcAiKUJg==
Date:   Tue, 30 Nov 2021 08:27:02 +0000
Message-ID: <BL1PR11MB54295ADE4D7A81523EA50B2D8C679@BL1PR11MB5429.namprd11.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
dlp-version: 11.6.200.16
dlp-product: dlpe-windows
dlp-reaction: no-action
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Z4gTzkahk1aIfAhqHXflRoNFc3Mt3TL3UokDOeXAoxiEmITnnQIPjRUnu3s0?=
 =?us-ascii?Q?XjlbwVUbvbZYCJYxgvYxan7+QpOrphv0VdkEFt0Zc6qYoB6sFmc48xOWw85u?=
 =?us-ascii?Q?Cb+IpWG56jy2utgsZQR7d1BMnf9J1KCqgD9D/kxALRAjj4wFPPcM/ScGh3Un?=
 =?us-ascii?Q?IGD2Y87Z8P2o0xpQKpt6Lx4C0WBRkXjS220vH96H15gSxLm5EpVjztIMRhny?=
 =?us-ascii?Q?wyd937ACyZogd0jJCgLXS4nCWyGiabkyxdqooaBHIIgJVfkHLGUiLl97Amx3?=
 =?us-ascii?Q?JGO6s0KQ14fe+3BiyanMt4TIupcsDY6/ZS3XPo7g9OkxoUBhsnkvUTUZGaC4?=
 =?us-ascii?Q?lR6nq9WaLI9L5bRKZW+6MJzJ2VKA1eFi8KBVEJ+1XUtserNktVHQ9cAC2avG?=
 =?us-ascii?Q?YO1QgilBooD4s9ePPt7SXacgTgQ9JOJuvQKk33ZAHcHwYpg5MbLSaK9zzJKf?=
 =?us-ascii?Q?dkcCR4CYD9y1vgGluCLlgzO2oICjyhEqDp2m9Y1RTM9gybiubFvU0BSSt16R?=
 =?us-ascii?Q?vdEPdfiSCuPy5yYshiJ8uHxfBLiFnn+X4oN2eSDJABnfbBJyjBaz0GgmLYkJ?=
 =?us-ascii?Q?AWNdrYIdXMj8KFwkjvGnMw4RcnTLW/60CzCHrJel0yYoN8uHzUx3ZL1uTW1F?=
 =?us-ascii?Q?/IEGhA1A/t1Cm/Ybw0x8+TSZcUVHISAhNoc7lnR5wxe8v6flmrxbbXCPrNTy?=
 =?us-ascii?Q?YfPBYTkITs+ZK7gxYRkf9ai4ahPxgOnUARP5cOTlgAOMuYz2HV0lNWWFZa6L?=
 =?us-ascii?Q?8BNYhk/thDR7ykp9i7gCIq1SHtMOwICrjqYb4q3hJhH03WuCdPDDZysmNym7?=
 =?us-ascii?Q?iLk+ZArayue/oBH8kSF+qMWs54tCaCMmvElkFHlm4zg9GN4wJJULN0ANpJRy?=
 =?us-ascii?Q?7v8U+9TyeTDD5BfiUqWSMrGSLeV2i9Y4mbJ/KSfx5C2MkVsULfvxo5k1dX4x?=
 =?us-ascii?Q?uQqC+IVfB4BTJ6JxffucjO4teh5bV84HQZckH9H6UCVkJUxO35MPk85Me2rT?=
 =?us-ascii?Q?hW2ZKJR6CP5EDt5bRRObgX4umpurFF5u6owFV/kznEBQYXQScxgDy1EnvJ/0?=
 =?us-ascii?Q?fhi5b0Ea4tDPS5ee3zLD7XFX8DFgdOIs1A24Zc4A+7IGKnCS/meqMFk162eq?=
 =?us-ascii?Q?8jpjgeCbv4lyrnW8MV9p7GP0eZ70k2uATg+giCq0p9xGjhy0tm1sR3UaYuEt?=
 =?us-ascii?Q?0mQxnFnBGcUHpqU5Wih/DETrR8PfyCGc9PU8yxSR52f1Bw9KOc5o/MfZKLhs?=
 =?us-ascii?Q?fPKRUjHy8FGegAnpXXqrffl1pjIEE4tgu8vkKqu8XUcnOVVPbMJlJdbbRB5R?=
 =?us-ascii?Q?n96Wimwow0MUbpKxoMGVGzlvb1uFk0dJ3lKX53EpUc1ooWyfPmshQXxL6uuB?=
 =?us-ascii?Q?gVBwfDgq4zeFiKxWTqMzW28K3e9VpcY10W2KB6NRhs+E+UGKNtyBUhzaPpWU?=
 =?us-ascii?Q?+fzdjjLDXD3iKji6cGSH4Wi08lJr9V3OGV3PnoePx5UthESMHsCLCmX7Ius0?=
 =?us-ascii?Q?SBOEvHqR6siGpli5+BTA86dmBYXIxNp3HBmSAHgns2+0PvyS74aSXYvyU1aO?=
 =?us-ascii?Q?E3YXnFI7s+NtIclojy+Cg/3OmaW6ADqEgYunrnpzkLBKpIOyJVRQEmclrzTV?=
 =?us-ascii?Q?8PSEvLr4vI/XQcmQxurLrc4=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5429.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9759637a-46bb-40a0-560e-08d9b3db2fa9
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Nov 2021 08:27:02.6959
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: uZ0CP+qJZQT+4Fi0TaRJzMuaU+/RjQKLK9ki89TurRRwBqlCMm+bp1sAXM26LwLk1b8gVixMZh6FfF7AFhGzAw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3380
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi, Paolo/Thomas,

I'm curious about the consequence if KVM fails to initialize a=20
hotplugged CPU.

Looking at the code KVM has been added to the CPU hotplug state=20
machine:

	r =3D cpuhp_setup_state_nocalls(CPUHP_AP_KVM_STARTING, "kvm/cpu:starting",
		kvm_starting_cpu, kvm_dying_cpu);

	static int kvm_starting_cpu(unsigned int cpu)
	{
		raw_spin_lock(&kvm_count_lock);
		if (kvm_usage_count)
			hardware_enable_nolock(NULL);
		raw_spin_unlock(&kvm_count_lock);
		return 0;
	}

kvm_starting_cpu() always return success as the callbacks in the=20
STARTING section are not allowed to fail.

However hardware_enable_nolock() may fail for various reasons:

	static void hardware_enable_nolock(void *junk)
	{
		int cpu =3D raw_smp_processor_id();
		int r;

		if (cpumask_test_cpu(cpu, cpus_hardware_enabled))
			return;

		cpumask_set_cpu(cpu, cpus_hardware_enabled);

		r =3D kvm_arch_hardware_enable();

		if (r) {
			cpumask_clear_cpu(cpu, cpus_hardware_enabled);
			atomic_inc(&hardware_enable_failed);
			pr_info("kvm: enabling virtualization on CPU%d failed\n", cpu);
		}
	}

Upon error hardware_enable_failed is incremented. However this variable
is checked only in hardware_enable_all() called when the 1st VM is called.

This implies that KVM may be left in a state where it doesn't know a CPU=20
not ready to host VMX operations.

Then I'm curious what will happen if a vCPU is scheduled to this CPU. Does
KVM indirectly catch it (e.g. vmenter fail) and return a deterministic erro=
r=20
to Qemu at some point or may it lead to undefined behavior? And is there
any method to prevent vCPU thread from being scheduled to the CPU?

We found this open when considering TDX and CPU hotplug.=20

By design the current generation of TDX doesn't support CPU hotplug.=20
Only boot-time CPUs can be initialized for TDX (and must be done en=20
masse in one breath). Attempting to do seamcalls on a hotplugged CPU
simply fails, thus it potentially affects any trusted domain in case its
vCPUs are scheduled to the plugged CPU.=20

There is a puzzle whether we should just document such restriction or=20
need more proactive measure (e.g. to prevent such case happen). Since=20
it's similar to above situation where KVM fails to init on a hotplugged=20
CPU, we'd like to seek your suggestion first.

Thanks
Kevin