Received-SPF: pass (google.com: domain of linux-kernel+bounces-188235-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Fri, 24 May 2024 10:39:00 +0800
From: Chao Gao <chao.gao@intel.com>
To: "Huang, Kai" <kai.huang@intel.com>
CC: Sean Christopherson <seanjc@google.com>, Paolo Bonzini
	<pbonzini@redhat.com>, <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 3/6] KVM: Add a module param to allow enabling
 virtualization when KVM is loaded
Message-ID: <Zk/9xMepBqAXEItK@chao-email>
References: <20240522022827.1690416-1-seanjc@google.com>
 <20240522022827.1690416-4-seanjc@google.com>
 <8b344a16-b28a-4f75-9c1a-a4edf2aa4a11@intel.com>
 <Zk7Eu0WS0j6/mmZT@chao-email>
 <c4fa17ca-d361-4cb7-a897-9812571aa75f@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <c4fa17ca-d361-4cb7-a897-9812571aa75f@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ftMkiGnTENVqcGyiZJU3fp6Tp3xt05oJ9mWBmuU42LuOqBktrbEDbRWoCorf?=
 =?us-ascii?Q?yPdBANisCxQfyU0fDAjuG7K/5wqdgBOmXA+LmNUgvi6MXpqUHhxRUZFQuvA5?=
 =?us-ascii?Q?AJzq3QbRqa4DV7uqButrA3ui1/MhG4fhz5eM22Q2cfLaRpyhfieGewYdBaGI?=
 =?us-ascii?Q?8EvVR9zpHBh++dNAHdficOI0V9gaW1oNNvcfRpi4z3MAIEWJG7KPQ9lJ9brA?=
 =?us-ascii?Q?NrLA9YPben5aui9xQp/K5igrFjvGeN6AYmFg0w1t/GzshGtUBZmMSYKKoVqF?=
 =?us-ascii?Q?p22JlUauj1FVfg6LnFWME4oVDL/u2TyvSvCkqZ1JZKzVdtguvQJFpI6gPTqW?=
 =?us-ascii?Q?qDut36/5QHPi0TJ7GijEdYkgMhKka9+ZtQfPQ2lcUSZaf4losaNRxv+h6bC3?=
 =?us-ascii?Q?6gEWSw2miLEXBGmiv544cFqbNtlXiicWOXP71EgHxsnopwGvosxiPMyYvMCF?=
 =?us-ascii?Q?ba1hk23paNuzITo2rosE1uT9lpOmhIVtBRLdEBu11CaqCl+np5OLH/KayFC1?=
 =?us-ascii?Q?HkyGhMDZ7B/WWKBKNoFhISvYDX4vONGv2V34KpcBkVR/CXbnPH2ZWqH59E+9?=
 =?us-ascii?Q?7MCvCqMa4DD8V9nuTuUzK5GK15nZsa6/gBEth2gEvLognIFsnGVUPq5BGwkJ?=
 =?us-ascii?Q?5wxFFmyVdwqJH07bGRz8XksRvDrchc3LDcdlJ+mILDQ2f3CImFzTOZyJ+xcG?=
 =?us-ascii?Q?chh+s5Z4FQCTt+E5h3ylq8sU1pTmqfpmv3Axbb2KJ7Yh7VnkOjCfxs1zU3qx?=
 =?us-ascii?Q?+0G0dK4lmddS4k7gWFqCapbrY+bop2BxTc6aEwcsqvFOpz79ntnfH5IcK6ZW?=
 =?us-ascii?Q?CN38triR91jArwJ+QZs41fZSDyp6bOu+BYLTX+DSOztH6O9GMWwIoGWyXpIa?=
 =?us-ascii?Q?jJUqe0vTiwAD1TcOgouLIAfF1RIqwyGax17o8LW2TXbSG2WSPrw8iRq5is1A?=
 =?us-ascii?Q?6neshkV71Tf80Mb/WZu/9UzKT5Ce+VvwM3TD61RJ+mBOMJyHXqWvkiEbz2xW?=
 =?us-ascii?Q?Svtm4bHrFT+0A436aVF5+GfftvvZvPAvPrK2dVNCFim1Ocm3U2nV+Ua2USqC?=
 =?us-ascii?Q?ToYEJ8QFsLrZf/j7E74o8NmKXjnPxLvxeVOXvO6pKMAW/ZJoaU+eJwbJsDQk?=
 =?us-ascii?Q?nDROnIac+L/lD2rOQOZxcHfrJ6btb3kngHmKNXxWrjFHkrJmrBsg/SKXV/9L?=
 =?us-ascii?Q?m86THca/mAaNVs3XcUBET5yjPe64RIaXhjdp+q/a1O2P0gA4RMky0Q4/AjZz?=
 =?us-ascii?Q?leRY1WMTdgSkvOcehlAOv+SVCqxWNoacDwoHNe+1hNs6YT7zwLkUaSdbkeIV?=
 =?us-ascii?Q?gQ2FvYdFZ23QFuPYCPeGr67iUAkH9NvZGHBFqXhxGF4IWpA7idasBj6rqqQZ?=
 =?us-ascii?Q?wyMtIMH8wvA0+L1lwg3C6eQ2/MZVc8Acp4QCVXx2pOG//21fEFTtrV/gZmPX?=
 =?us-ascii?Q?D4QdwH5g1pH1pP65wJ1yqRs8iHlWpKdOGhebH362zO9RwAtrnRHn+ZeSZDQj?=
 =?us-ascii?Q?QxAGtPYSofsjlq0oCHRDU+CrHyo60HWdoVDJftdnfC9k4YBuDILNkD/rxTWp?=
 =?us-ascii?Q?F/QFOCIHKyHXpL9Pq+yymVxE/J8mREsDMVeC4uui?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f1c53587-7bc5-4914-6ae1-08dc7b9aafc7
X-MS-Exchange-CrossTenant-AuthSource: CH3PR11MB8660.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2024 02:39:08.7619
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: n+pYbtP3vzzH8G7pNB7k8AEh3Iz12rcBItr55mUxZ0qLrlrcfSkpNxgeRZOT4i9igxGtKwV1tyC31qjnHYSdtg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7124
X-OriginatorOrg: intel.com

On Fri, May 24, 2024 at 11:11:37AM +1200, Huang, Kai wrote:
>
>
>On 23/05/2024 4:23 pm, Chao Gao wrote:
>> On Thu, May 23, 2024 at 10:27:53AM +1200, Huang, Kai wrote:
>> > 
>> > 
>> > On 22/05/2024 2:28 pm, Sean Christopherson wrote:
>> > > Add an off-by-default module param, enable_virt_at_load, to let userspace
>> > > force virtualization to be enabled in hardware when KVM is initialized,
>> > > i.e. just before /dev/kvm is exposed to userspace.  Enabling virtualization
>> > > during KVM initialization allows userspace to avoid the additional latency
>> > > when creating/destroying the first/last VM.  Now that KVM uses the cpuhp
>> > > framework to do per-CPU enabling, the latency could be non-trivial as the
>> > > cpuhup bringup/teardown is serialized across CPUs, e.g. the latency could
>> > > be problematic for use case that need to spin up VMs quickly.
>> > 
>> > How about we defer this until there's a real complain that this isn't
>> > acceptable?  To me it doesn't sound "latency of creating the first VM"
>> > matters a lot in the real CSP deployments.
>> 
>> I suspect kselftest and kvm-unit-tests will be impacted a lot because
>> hundreds of tests are run serially. And it looks clumsy to reload KVM
>> module to set enable_virt_at_load to make tests run faster. I think the
>> test slowdown is a more realistic problem than running an off-tree
>> hypervisor, so I vote to make enabling virtualization at load time the
>> default behavior and if we really want to support an off-tree hypervisor,
>> we can add a new module param to opt in enabling virtualization at runtime.
>
>I am not following why off-tree hypervisor is ever related to this.

Enabling virtualization at runtime was added to support an off-tree hypervisor
(see the commit below).  To me, supporting an off-tree hypervisor while KVM is
autoloaded is a niche usage. so, my preference is to make enabling
virtualization at runtime opt-in rather than the default.

commit 10474ae8945ce08622fd1f3464e55bd817bf2376
Author: Alexander Graf <agraf@suse.de>
Date:   Tue Sep 15 11:37:46 2009 +0200

    KVM: Activate Virtualization On Demand

    X86 CPUs need to have some magic happening to enable the virtualization
    extensions on them. This magic can result in unpleasant results for
    users, like blocking other VMMs from working (vmx) or using invalid TLB
    entries (svm).

    Currently KVM activates virtualization when the respective kernel module
    is loaded. This blocks us from autoloading KVM modules without breaking
    other VMMs.

    To circumvent this problem at least a bit, this patch introduces on
    demand activation of virtualization. This means, that instead
    virtualization is enabled on creation of the first virtual machine
    and disabled on destruction of the last one.

    So using this, KVM can be easily autoloaded, while keeping other
    hypervisors usable.

>
>Could you elaborate?
>
>The problem of enabling virt during module loading by default is it impacts
>all ARCHs. Given this performance downgrade (if we care) can be resolved by
>explicitly doing on_each_cpu() below, I am not sure why we want to choose
>this radical approach.

IIUC, we plan to set up TDX module at KVM load time; we need to enable virt
at load time at least for TDX. Definitely, on_each_cpu() can solve the perf
concern. But a solution which can also satisfy TDX's need is better to me.

>
>
>> > Or we just still do:
>> > 
>> > 	cpus_read_lock();
>> > 	on_each_cpu(hardware_enable_nolock, ...);
>> > 	cpuhp_setup_state_nocalls_cpuslocked(...);
>> > 	cpus_read_unlock();
>> > 
>> > I think the main benefit of series is to put all virtualization enabling
>> > related things into one single function.  Whether using cpuhp_setup_state()
>> > or using on_each_cpu() shouldn't be the main point.
>> >