Received: by 2002:a05:7412:8d10:b0:f3:1519:9f41 with SMTP id bj16csp6818186rdb; Fri, 15 Dec 2023 09:05:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IGRvsZpmVdGOoBf5y6YI3W/Dzze2ri9oFP7gJYwrRtWQpFWNChyA1jrAh51K8v0y37qT024 X-Received: by 2002:a05:6a21:3810:b0:190:10aa:f4f0 with SMTP id yi16-20020a056a21381000b0019010aaf4f0mr11825933pzb.57.1702659901999; Fri, 15 Dec 2023 09:05:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702659901; cv=none; d=google.com; s=arc-20160816; b=Hre+c+AVvnRpv3sTFd5aUlsr7L02M8u7tzLRsAJ1TH7GF6Eo7UynP5UBJLeoDw2cxP gYK7DTrSw4klZ4hQhPXrQ9u9SH59x1V3HHyT++nQ84xARwNqI4EhFTQJXNcZRr6BR4vb 7YWbp3xI5Y0c0A7K8eTq5nEdCLuvwIghMF8ehCCiF5mXO3ODHx78XePdiPmMUwYlHZof nsoONAWi63wAOjOR++S2AI+bcmvygrwXzYjll239cdODPdGg9utcdm66YOyUby/YwzZm MPDGcVPogMItkeO5JTSJWIa6qh5IwiQfUbccWLAW36bmjxpx3r0qNY87wT9CmOFYfvt/ PK4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:organization:references :in-reply-to:message-id:subject:cc:to:from:date; bh=WNTD9ihIvcdyUmAijUbtr/rr3zTYE6iZWqgPlDM/dLE=; fh=4iHbZ733mqZc4Bx83AV0konbWC9aXbRvmPAEmCiuGYk=; b=A47Fn/qZBK/qodSlTvar2SUlbMy5YGsHnkHlbGzYMPR2tPEVQI/fctPuTXDq44Padd ZvIaTfV9/n2zxygAVN2a37KrxMplAcI8jlfcweXYgs4EovRxQJj5+PWxrcGxzyDuNlH2 SU/aug2C07CFFXZ4U7qCGlnJ/YHmcWkz0JCnINoQvSbMyZi7u4m/+u1C9Jzu7ykVD1Iz Vhx16mXcJOJ1kPEn+V4ROFOtAzZFr7luBAxLzjpZwa98U4FrN/sRBObgXmlSFo8/w6yn rqhp7+++odrCb4KmzrsMbn/+yWCcPNAiOSV5eTTz2PWQVnAoZIyp7gO3YtHqGDcQzrA3 W00Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-1351-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-1351-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id q5-20020a655245000000b005c602ff4068si12781693pgp.438.2023.12.15.09.05.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 09:05:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-1351-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-1351-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-1351-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 7BBDBB22465 for ; Fri, 15 Dec 2023 17:04:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 556AF3FE2F; Fri, 15 Dec 2023 17:04:36 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD6203EA82; Fri, 15 Dec 2023 17:04:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=Huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4SsFsm2gd0z6JB0q; Sat, 16 Dec 2023 01:03:24 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (unknown [7.191.163.240]) by mail.maildlp.com (Postfix) with ESMTPS id 2B695140390; Sat, 16 Dec 2023 01:04:30 +0800 (CST) Received: from localhost (10.202.227.76) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Fri, 15 Dec 2023 17:04:29 +0000 Date: Fri, 15 Dec 2023 17:04:28 +0000 From: Jonathan Cameron To: "Russell King (Oracle)" CC: , , , , , , , , , , , , , , Salil Mehta , Jean-Philippe Brucker , , , James Morse Subject: Re: [PATCH RFC v3 19/21] arm64: document virtual CPU hotplug's expectations Message-ID: <20231215170428.00000d81@Huawei.com> In-Reply-To: References: Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-ClientProxiedBy: lhrpeml100004.china.huawei.com (7.191.162.219) To lhrpeml500005.china.huawei.com (7.191.163.240) On Wed, 13 Dec 2023 12:50:49 +0000 Russell King (Oracle) wrote: > From: James Morse > > Add a description of physical and virtual CPU hotplug, explain the > differences and elaborate on what is required in ACPI for a working > virtual hotplug system. > > Signed-off-by: James Morse > --- > Outstanding comment: > https://lore.kernel.org/r/20230914174137.00000a62@Huawei.com Hmm. This one is the comment that if we allow for a legacy unware guest, we have no way of indicating that CPUS that were enabled at boot can ever be removed. Effectively that means that without the cloud being aware of the VM capabilities before it is booted (and can maybe use the proposed OSC) there is no way of knowing if a CPU can be removed. Sounds profitable :) I'm fine with that. So as long a people grasp the concern and we make sure that the QEMU side doesn't change it's legacy behavior (I think we are fine in Salil's latest set). Reviewed-by: Jonathan Cameron Jonathan > --- > Documentation/arch/arm64/cpu-hotplug.rst | 79 ++++++++++++++++++++++++ > Documentation/arch/arm64/index.rst | 1 + > 2 files changed, 80 insertions(+) > create mode 100644 Documentation/arch/arm64/cpu-hotplug.rst > > diff --git a/Documentation/arch/arm64/cpu-hotplug.rst b/Documentation/arch/arm64/cpu-hotplug.rst > new file mode 100644 > index 000000000000..76ba8d932c72 > --- /dev/null > +++ b/Documentation/arch/arm64/cpu-hotplug.rst > @@ -0,0 +1,79 @@ > +.. SPDX-License-Identifier: GPL-2.0 > +.. _cpuhp_index: > + > +==================== > +CPU Hotplug and ACPI > +==================== > + > +CPU hotplug in the arm64 world is commonly used to describe the kernel taking > +CPUs online/offline using PSCI. This document is about ACPI firmware allowing > +CPUs that were not available during boot to be added to the system later. > + > +``possible`` and ``present`` refer to the state of the CPU as seen by linux. > + > + > +CPU Hotplug on physical systems - CPUs not present at boot > +---------------------------------------------------------- > + > +Physical systems need to mark a CPU that is ``possible`` but not ``present`` as > +being ``present``. An example would be a dual socket machine, where the package > +in one of the sockets can be replaced while the system is running. > + > +This is not supported. > + > +In the arm64 world CPUs are not a single device but a slice of the system. > +There are no systems that support the physical addition (or removal) of CPUs > +while the system is running, and ACPI is not able to sufficiently describe > +them. > + > +e.g. New CPUs come with new caches, but the platform's cache toplogy is > +described in a static table, the PPTT. How caches are shared between CPUs is > +not discoverable, and must be described by firmware. > + > +e.g. The GIC redistributor for each CPU must be accessed by the driver during > +boot to discover the system wide supported features. ACPI's MADT GICC > +structures can describe a redistributor associated with a disabled CPU, but > +can't describe whether the redistributor is accessible, only that it is not > +'always on'. > + > +arm64's ACPI tables assume that everything described is ``present``. > + > + > +CPU Hotplug on virtual systems - CPUs not enabled at boot > +--------------------------------------------------------- > + > +Virtual systems have the advantage that all the properties the system will > +ever have can be described at boot. There are no power-domain considerations > +as such devices are emulated. > + > +CPU Hotplug on virtual systems is supported. It is distinct from physical > +CPU Hotplug as all resources are described as ``present``, but CPUs may be > +marked as disabled by firmware. Only the CPU's online/offline behaviour is > +influenced by firmware. An example is where a virtual machine boots with a > +single CPU, and additional CPUs are added once a cloud orchestrator deploys > +the workload. > + > +For a virtual machine, the VMM (e.g. Qemu) plays the part of firmware. > + > +Virtual hotplug is implemented as a firmware policy affecting which CPUs can be > +brought online. Firmware can enforce its policy via PSCI's return codes. e.g. > +``DENIED``. > + > +The ACPI tables must describe all the resources of the virtual machine. CPUs > +that firmware wishes to disable either from boot (or later) should not be > +``enabled`` in the MADT GICC structures, but should have the ``online capable`` > +bit set, to indicate they can be enabled later. The boot CPU must be marked as > +``enabled``. The 'always on' GICR structure must be used to describe the > +redistributors. > + > +CPUs described as ``online capable`` but not ``enabled`` can be set to enabled > +by the DSDT's Processor object's _STA method. On virtual systems the _STA method > +must always report the CPU as ``present``. Changes to the firmware policy can > +be notified to the OS via device-check or eject-request. > + > +CPUs described as ``enabled`` in the static table, should not have their _STA > +modified dynamically by firmware. Soft-restart features such as kexec will > +re-read the static properties of the system from these static tables, and > +may malfunction if these no longer describe the running system. Linux will > +re-discover the dynamic properties of the system from the _STA method later > +during boot. > diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst > index d08e924204bf..78544de0a8a9 100644 > --- a/Documentation/arch/arm64/index.rst > +++ b/Documentation/arch/arm64/index.rst > @@ -13,6 +13,7 @@ ARM64 Architecture > asymmetric-32bit > booting > cpu-feature-registers > + cpu-hotplug > elf_hwcaps > hugetlbpage > kdump