Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Dongli Zhang <dongli.zhang@oracle.com>
To:     xen-devel@lists.xenproject.org, x86@kernel.org
Cc:     linux-kernel@vger.kernel.org, boris.ostrovsky@oracle.com,
        jgross@suse.com, sstabellini@kernel.org, tglx@linutronix.de,
        mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com,
        joe.jin@oracle.com
Subject: [PATCH v4 2/2] xen: delay xen_hvm_init_time_ops() if kdump is boot on vcpu>=32
Date:   Wed,  2 Mar 2022 08:40:32 -0800
Message-Id: <20220302164032.14569-3-dongli.zhang@oracle.com>
In-Reply-To: <20220302164032.14569-1-dongli.zhang@oracle.com>
References: <20220302164032.14569-1-dongli.zhang@oracle.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dOP///kj3fAaOgnhY7LTb31y6qH/A3m0+YO1aKOod7OIRuieNNMitZODQ1yt?=
 =?us-ascii?Q?YO3IIzFOb610lRl3MBrE0UOBPCldK70iQVIDa5Bv46yQLnHflde/6n/3zwsR?=
 =?us-ascii?Q?DKD3xO40OsUfFPh+AWWakQHqS31RB/pmn/cFAfGHQFPhglofbq+GUzqIHpXc?=
 =?us-ascii?Q?pFmRbWSN+0ngplFtuTqnEAymp3X0Sp2IhC+xksSJ4xHLRMTOiJDsdIUKlNWg?=
 =?us-ascii?Q?yY/qopm4Dafr8ODhvCl8iQKk/Cgh6QvQz2StFSRTps4vzwoRtQNg4RSKetQM?=
 =?us-ascii?Q?7U2bRK3RnTlOHmBQG9OxdHgZVBPA6If9HjDotySzIy/JZkugchpwDr261hqI?=
 =?us-ascii?Q?KYz7OnWQ2L63Qo3ybV0hL5MKqXv/0ZbwT7un8hdKRdkyMEoDiMgUwbZdzsw6?=
 =?us-ascii?Q?5IRrhrUqYYKq5FvGrFFC9tpcVy5xiKX8ROTxE5WTG56uUQjEsjtzl4oZhynx?=
 =?us-ascii?Q?LYcg0G57VAzxF8WNzA0tOvg7i49pzOWI6ox7X21JiIyieJnoOtZGI5Vkiv/R?=
 =?us-ascii?Q?Exrju77OGuXPLRlqaGD7qlAmNdVHk9OmhDHur2FuXYnM8uIE1iawhsCSHAXO?=
 =?us-ascii?Q?+e2V/88i1VrFm5A0z747bqRTIkHqCBJis8mbLdhg+UejCSmcR8hW3o0b+6qz?=
 =?us-ascii?Q?8TOyI9caH/qLZA3yZneykNz9vo82v89ts7XOFMy5yk2jVcB6lXFQL0+NUCNW?=
 =?us-ascii?Q?4zUIbXAKcsgDIl5cE/hMVm0EHNL90pypnigBJ+klUBI/ZreoCJkM6w1lWxc3?=
 =?us-ascii?Q?UvTYTWiIPzgkLyVqp0T3teVzzTiKFMr5xnS6RPX4JBRzBM0+v/oHxrU4KG68?=
 =?us-ascii?Q?fg3zzW9PWkMOn4te9uEIuIKhwAfVqT2UyjYpj3Pt0y9BoqfWkIH5thmoNURl?=
 =?us-ascii?Q?dC6e46xLYMQ2jkYnsicYSBBWdw5J59zmyYyDjI1HynLCDrxb155Nm54qInr2?=
 =?us-ascii?Q?qYSkisxfHLueFRFswiOH1e902e/cIO7cup1YmzmQNJ/CC1yO+QuJDnKV+4/r?=
 =?us-ascii?Q?pbyjBAUtlOV15kfFKE35OPCJHc8km2GGQTlnxftoXXTPa/3iwVsq6aJSVnC4?=
 =?us-ascii?Q?QVg32erY6n6dR11RTp6yGoP2hkLe0hT7EZQHQ+qiI0Fj46VJOc+81J5HUD4x?=
 =?us-ascii?Q?UTRIi/Z2+lClX/xaWy+xkZk5otKRmi4S29+JZNXMLCrt2SL66TzbtwcHl3qS?=
 =?us-ascii?Q?kyDxoS5+RlSnpYmz4JmBQveJDas4QvtFIO3rQY9UDKlpSskNm24PtTRSn1Un?=
 =?us-ascii?Q?aTXV7MB48g0yRCw2aNctyn+zVEuz+PrMH/sjTTY7FKbb8XuZD7Rd8Qc8mOoC?=
 =?us-ascii?Q?oOswq4T3o675mv+iU34C1RBvlpbTuNYD0vtW1ONpdvYZQ/KP96qLrnCaKWp6?=
 =?us-ascii?Q?TqWNK6Js9PfvyoEyqYMIUrqeJW9ta3Ukg3i/UHCBjLPT7KULX/7b7sHNUmg4?=
 =?us-ascii?Q?/RYVV1rg44rqpk9pctMvJqBbGPgiqKFdAv3/GNTnPf1kAflxq3wZjDz7juLL?=
 =?us-ascii?Q?GZ3EAbUGzg29ya/IeZfmOzIdONq+2VewlDaQKinHgKKrecVIzOP/3C1CaqUe?=
 =?us-ascii?Q?X06WRAy0MdPrOgWFla2wuR/l72yQE2zvjktcCdZWYSAxWrkyVf/CZI0E0jQi?=
 =?us-ascii?Q?Quxs3Zl5CmZZaQRkXe9gZ80=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e6c7a851-a288-4101-9d60-08d9fc6b82d2
X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2663.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Mar 2022 16:41:33.7114
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +1TgDRK4ve3mXHKXq/401zlsOIroDpALX/dfar9kXhL/HxH9/6/IK3CgJgUpVd9areO02m/3TYxAvtB4nlsExg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1001MB2144
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10274 signatures=686787
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 malwarescore=0
 mlxscore=0 phishscore=0 suspectscore=0 adultscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2203020073
X-Proofpoint-GUID: qkEPK_YnsbroMSihPPrq00FLhgwsaZjL
X-Proofpoint-ORIG-GUID: qkEPK_YnsbroMSihPPrq00FLhgwsaZjL
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

The sched_clock() can be used very early since commit 857baa87b642
("sched/clock: Enable sched clock early"). In addition, with commit
38669ba205d1 ("x86/xen/time: Output xen sched_clock time from 0"), kdump
kernel in Xen HVM guest may panic at very early stage when accessing
&__this_cpu_read(xen_vcpu)->time as in below:

setup_arch()
 -> init_hypervisor_platform()
     -> x86_init.hyper.init_platform = xen_hvm_guest_init()
         -> xen_hvm_init_time_ops()
             -> xen_clocksource_read()
                 -> src = &__this_cpu_read(xen_vcpu)->time;

This is because Xen HVM supports at most MAX_VIRT_CPUS=32 'vcpu_info'
embedded inside 'shared_info' during early stage until xen_vcpu_setup() is
used to allocate/relocate 'vcpu_info' for boot cpu at arbitrary address.

However, when Xen HVM guest panic on vcpu >= 32, since
xen_vcpu_info_reset(0) would set per_cpu(xen_vcpu, cpu) = NULL when
vcpu >= 32, xen_clocksource_read() on vcpu >= 32 would panic.

This patch calls xen_hvm_init_time_ops() again later in
xen_hvm_smp_prepare_boot_cpu() after the 'vcpu_info' for boot vcpu is
registered when the boot vcpu is >= 32.

This issue can be reproduced on purpose via below command at the guest
side when kdump/kexec is enabled:

"taskset -c 33 echo c > /proc/sysrq-trigger"

The bugfix for PVM is not implemented due to the lack of testing
environment.

Cc: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
Changed since v1:
  - Add commit message to explain why xen_hvm_init_time_ops() is delayed
    for any vcpus. (Suggested by Boris Ostrovsky)
  - Add a comment in xen_hvm_smp_prepare_boot_cpu() referencing the related
    code in xen_hvm_guest_init(). (suggested by Juergen Gross)
Changed since v2:
  - Delay for all VCPUs. (Suggested by Boris Ostrovsky)
  - Add commit message that why PVM is not supported by this patch
  - Test if kexec/kdump works with mainline xen (HVM and PVM)
Changed since v3:
  - Re-use v2 but move the login into xen_hvm_init_time_ops() (Suggested
    by Boris Ostrovsky)

 arch/x86/xen/smp_hvm.c |  6 ++++++
 arch/x86/xen/time.c    | 25 ++++++++++++++++++++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/arch/x86/xen/smp_hvm.c b/arch/x86/xen/smp_hvm.c
index 6ff3c887e0b9..b70afdff419c 100644
--- a/arch/x86/xen/smp_hvm.c
+++ b/arch/x86/xen/smp_hvm.c
@@ -19,6 +19,12 @@ static void __init xen_hvm_smp_prepare_boot_cpu(void)
 	 */
 	xen_vcpu_setup(0);
 
+	/*
+	 * Called again in case the kernel boots on vcpu >= MAX_VIRT_CPUS.
+	 * Refer to comments in xen_hvm_init_time_ops().
+	 */
+	xen_hvm_init_time_ops();
+
 	/*
 	 * The alternative logic (which patches the unlock/lock) runs before
 	 * the smp bootup up code is activated. Hence we need to set this up
diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 55b3407358a9..dcf292cc859e 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -558,16 +558,36 @@ static void xen_hvm_setup_cpu_clockevents(void)
 
 void __init xen_hvm_init_time_ops(void)
 {
+	static bool hvm_time_initialized;
+
+	if (hvm_time_initialized)
+		return;
+
 	/*
 	 * vector callback is needed otherwise we cannot receive interrupts
 	 * on cpu > 0 and at this point we don't know how many cpus are
 	 * available.
 	 */
 	if (!xen_have_vector_callback)
-		return;
+		goto exit;
 
 	if (!xen_feature(XENFEAT_hvm_safe_pvclock)) {
 		pr_info("Xen doesn't support pvclock on HVM, disable pv timer");
+		goto exit;
+	}
+
+	/*
+	 * Only MAX_VIRT_CPUS 'vcpu_info' are embedded inside 'shared_info'.
+	 * The __this_cpu_read(xen_vcpu) is still NULL when Xen HVM guest
+	 * boots on vcpu >= MAX_VIRT_CPUS (e.g., kexec), To access
+	 * __this_cpu_read(xen_vcpu) via xen_clocksource_read() will panic.
+	 *
+	 * The xen_hvm_init_time_ops() should be called again later after
+	 * __this_cpu_read(xen_vcpu) is available.
+	 */
+	if (!__this_cpu_read(xen_vcpu)) {
+		pr_info("Delay xen_init_time_common() as kernel is running on vcpu=%d\n",
+			xen_vcpu_nr(0));
 		return;
 	}
 
@@ -577,6 +597,9 @@ void __init xen_hvm_init_time_ops(void)
 	x86_cpuinit.setup_percpu_clockev = xen_hvm_setup_cpu_clockevents;
 
 	x86_platform.set_wallclock = xen_set_wallclock;
+
+exit:
+	hvm_time_initialized = true;
 }
 #endif
 
-- 
2.17.1