Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Dongli Zhang <dongli.zhang@oracle.com>
To:     xen-devel@lists.xenproject.org, x86@kernel.org
Cc:     boris.ostrovsky@oracle.com, jgross@suse.com,
        sstabellini@kernel.org, tglx@linutronix.de, mingo@redhat.com,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        joe.jin@oracle.com, linux-kernel@vger.kernel.org
Subject: [PATCH v2 1/1] xen: delay xen_hvm_init_time_ops() if kdump is boot on vcpu>=32
Date:   Wed, 27 Oct 2021 18:25:43 -0700
Message-Id: <20211028012543.8776-1-dongli.zhang@oracle.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?90jjWBT5DMTT5TE7WJ7uCYuMl3UkX10N8b6g7zr8ZQSpf2xNqKPKxKvFmsCl?=
 =?us-ascii?Q?xUHYB9TJyyOAWgD8peO0JLJBMgsaKC1Ryjqo6AEcbU55xZF/DA3L/c3yQmnk?=
 =?us-ascii?Q?dAr76I1pb3TGQGIxggg7PVYWStyZErJw3v3QXqgwi7ncbfuaSpJetsvCYFQp?=
 =?us-ascii?Q?bntFohuSaPU12yHfptFAdtwJn16mZIyifPEWUjwepe69A4nwZicC5H03Yu7N?=
 =?us-ascii?Q?mV08XCfjiqlFJm1X4MPWM7FhNjUWZV4NjS82sP8UJlztSnArPjHEiUW1pGli?=
 =?us-ascii?Q?Ahr2RMCOUcjIQeDZ/ZzOVHPeJTpzoNzirQDqK5p95AaNt0olQ07CZhZyYiWx?=
 =?us-ascii?Q?+TP2ODfCpm0JVM2pXByNU2Xus28q5Z3Y61+G39tgianMAlvHbcbEE0hI+8us?=
 =?us-ascii?Q?wo3539OIgjsSJNAdqzYJTWI/+bG0z0pcDXPaFIsOpCFbYx81Ea2bREbXIM3A?=
 =?us-ascii?Q?1BPNAkDSRyvSI3x0XcYZ6GRir7APZ6R8E4yn2sfL7wujuEPIOzuVEUSGRPaO?=
 =?us-ascii?Q?iHUJFQhNLqtOYSw5FZtdEHNVj5y5Sxt7WYHGkCfL8/ifZRSNCCpWTves6VLb?=
 =?us-ascii?Q?78PV+X+cI54COPvn4bgbTNDsQxNmwzMFyfZgP4jVGof63/roUFzp28THAUk+?=
 =?us-ascii?Q?tXmud+uQZu00LjFrj4Otd1SSEm/cXHRzwy5zeqcnkL9aSNjF+iZKYSrtYhpC?=
 =?us-ascii?Q?kLGjSfHxJHiSSWyk2s9pICsL977TMh6V6VJb/tkKlyzkLnCANuFG1T3z1PQM?=
 =?us-ascii?Q?98AObkglseytl9zuHE+oY6g7a3XGjfBQE7Nc7arAGUsecwL7D6xdSZtKcyOp?=
 =?us-ascii?Q?GdXLLwoVHuQXmgRO8b8MolF3mgOQOxDG9ABiyq2w7CjOFPDn9E/EUx9Pm0+W?=
 =?us-ascii?Q?NYo9BI53cCD6bIeVfgYczP56fi/4+S727c/q6aiuHkBuj2LH9ViUOEWC5wJY?=
 =?us-ascii?Q?c1h3dpYTKBF5ZDp4vxdVNAEqy7AW4bMaMjFYGVc55SgIQ5qzOlsFU62a40wz?=
 =?us-ascii?Q?eoDGz6Xj2EgY/8OUKOz/UHG+ZvYItwGf9YieJKuHoeAMvaYBXoIAL2cns2Y4?=
 =?us-ascii?Q?5J97rxZ5r95rjq2yXJqUsGezX/s6ZHD6LasMGNKnHtkyAEdZ3k0yWjLUpLFj?=
 =?us-ascii?Q?+NrjjC6PbEwiTLXl1HtExYiqL3nP4JSJ2YBVxafKAwFAz0Trlq5zj2FTa2Qw?=
 =?us-ascii?Q?UNZTRmpoKvsCPGmcxyJBuCUGwQOhstSuixCNePjviHUTic+TeXhLzflsTxHt?=
 =?us-ascii?Q?Dwn5SEEuk8B6KHsORQ8Gq7OnOJ5ZnmEO0birBxVVm620eBFvtjTxWSs3QzLs?=
 =?us-ascii?Q?ncRgjeqp0dP1JQSv/ZZLcoSahU96gCINlHolJmhhJ74ilRwWmgvWVX7bxtuA?=
 =?us-ascii?Q?e3IJ5BcpNguE3b/gjFVYiJcITiAlOwlQKJsG929+IhCShaLHqvifIJAmT/NR?=
 =?us-ascii?Q?7UClibjGpV4yTEnRioKcK6gW5PjhAIeetmzPHaVaQWrusQEmYjVannfxtPOX?=
 =?us-ascii?Q?OcQQVSMAIPtzRf+NPsk5hegEVhwbLFjY9OLmBWArkBk+BP9odJUiYYnJoX0v?=
 =?us-ascii?Q?fYtt9Mf/zsEGfQDWIfMG1cFBhlqVUBnsR8OJPc0K+mTuQ30ckvDE0quA2yIV?=
 =?us-ascii?Q?0fCCTiWyvhSKP3OPqcbXO18=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ee9e348d-fa09-4a9f-f005-08d999b1f0e0
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Oct 2021 01:26:18.1638
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Jki7SmEMuVURk2MiuuiD6+QLaRR6Q9Gcq0RGgOTkxdbW/AWGjUu80ZY9PqJgoY0dzunokPZLGHrfkqL+T8OWPA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3761
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10150 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 suspectscore=0
 mlxscore=0 adultscore=0 malwarescore=0 phishscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2110280006
X-Proofpoint-GUID: HVIeaRs42jHKtvGrgD95e0-5YT6Jr_vo
X-Proofpoint-ORIG-GUID: HVIeaRs42jHKtvGrgD95e0-5YT6Jr_vo
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

The sched_clock() can be used very early since
commit 857baa87b642 ("sched/clock: Enable sched clock early"). In addition,
with commit 38669ba205d1 ("x86/xen/time: Output xen sched_clock time from
0"), kdump kernel in Xen HVM guest may panic at very early stage when
accessing &__this_cpu_read(xen_vcpu)->time as in below:

setup_arch()
 -> init_hypervisor_platform()
     -> x86_init.hyper.init_platform = xen_hvm_guest_init()
         -> xen_hvm_init_time_ops()
             -> xen_clocksource_read()
                 -> src = &__this_cpu_read(xen_vcpu)->time;

This is because Xen HVM supports at most MAX_VIRT_CPUS=32 'vcpu_info'
embedded inside 'shared_info' during early stage until xen_vcpu_setup() is
used to allocate/relocate 'vcpu_info' for boot cpu at arbitrary address.

However, when Xen HVM guest panic on vcpu >= 32, since
xen_vcpu_info_reset(0) would set per_cpu(xen_vcpu, cpu) = NULL when
vcpu >= 32, xen_clocksource_read() on vcpu >= 32 would panic.

This patch delays xen_hvm_init_time_ops() to later in
xen_hvm_smp_prepare_boot_cpu() after the 'vcpu_info' for boot vcpu is
registered when the boot vcpu is >= 32.

Another option is to always delay xen_hvm_init_time_ops() for any vcpus
(including vcpu=0). Since to delay xen_hvm_init_time_ops() may lead to
clock backward issue, it is preferred to avoid that for regular boot (The
pv_sched_clock=native_sched_clock() is used at the very beginning until
xen_sched_clock() is registered). That requires to adjust
xen_sched_clock_offset. That's why we only delay xen_hvm_init_time_ops()
for vcpu>=32.

This issue can be reproduced on purpose via below command at the guest
side when kdump/kexec is enabled:

"taskset -c 33 echo c > /proc/sysrq-trigger"

Reference:
https://lists.xenproject.org/archives/html/xen-devel/2021-10/msg00571.html
Cc: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
Changed since v1:
  - Add commit message to explain why xen_hvm_init_time_ops() is delayed
    for any vcpus. (Suggested by Boris Ostrovsky)
  - Add a comment in xen_hvm_smp_prepare_boot_cpu() referencing the related
    code in xen_hvm_guest_init(). (suggested by Juergen Gross)

 arch/x86/xen/enlighten_hvm.c | 20 +++++++++++++++++++-
 arch/x86/xen/smp_hvm.c       |  8 ++++++++
 2 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
index e68ea5f4ad1c..7734dec52794 100644
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -216,7 +216,25 @@ static void __init xen_hvm_guest_init(void)
 	WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm, xen_cpu_dead_hvm));
 	xen_unplug_emulated_devices();
 	x86_init.irqs.intr_init = xen_init_IRQ;
-	xen_hvm_init_time_ops();
+
+	/*
+	 * Only MAX_VIRT_CPUS 'vcpu_info' are embedded inside 'shared_info'
+	 * and the VM would use them until xen_vcpu_setup() is used to
+	 * allocate/relocate them at arbitrary address.
+	 *
+	 * However, when Xen HVM guest panic on vcpu >= MAX_VIRT_CPUS,
+	 * per_cpu(xen_vcpu, cpu) is still NULL at this stage. To access
+	 * per_cpu(xen_vcpu, cpu) via xen_clocksource_read() would panic.
+	 *
+	 * Therefore we delay xen_hvm_init_time_ops() to
+	 * xen_hvm_smp_prepare_boot_cpu() when boot vcpu is >= MAX_VIRT_CPUS.
+	 */
+	if (xen_vcpu_nr(0) >= MAX_VIRT_CPUS)
+		pr_info("Delay xen_hvm_init_time_ops() as kernel is running on vcpu=%d\n",
+			xen_vcpu_nr(0));
+	else
+		xen_hvm_init_time_ops();
+
 	xen_hvm_init_mmu_ops();
 
 #ifdef CONFIG_KEXEC_CORE
diff --git a/arch/x86/xen/smp_hvm.c b/arch/x86/xen/smp_hvm.c
index 6ff3c887e0b9..f99043df8bb5 100644
--- a/arch/x86/xen/smp_hvm.c
+++ b/arch/x86/xen/smp_hvm.c
@@ -19,6 +19,14 @@ static void __init xen_hvm_smp_prepare_boot_cpu(void)
 	 */
 	xen_vcpu_setup(0);
 
+	/*
+	 * The xen_hvm_init_time_ops() is delayed from
+	 * xen_hvm_guest_init() to here to avoid panic when the kernel
+	 * boots from vcpu>=MAX_VIRT_CPUS (32).
+	 */
+	if (xen_vcpu_nr(0) >= MAX_VIRT_CPUS)
+		xen_hvm_init_time_ops();
+
 	/*
 	 * The alternative logic (which patches the unlock/lock) runs before
 	 * the smp bootup up code is activated. Hence we need to set this up
-- 
2.17.1