Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Kairui Song <ryncsn@gmail.com>
To:     linux-kernel@vger.kernel.org
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Eric Biederman <ebiederm@xmission.com>, x86@kernel.org,
        kexec@lists.infradead.org, Kairui Song <kasong@tencent.com>
Subject: [PATCH 2/2] x86/mpparse, kexec: probe apic driver early for x2apic
Date:   Wed,  8 Jun 2022 14:43:48 +0800
Message-Id: <20220608064348.66402-3-ryncsn@gmail.com>
In-Reply-To: <20220608064348.66402-1-ryncsn@gmail.com>
References: <20220608064348.66402-1-ryncsn@gmail.com>
Reply-To: Kairui Song <kasong@tencent.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

From: Kairui Song <kasong@tencent.com>

Following kernel panic is observed when doing kdump/kexec on
virtual machines that uses MPTABLE, not ACPI MADT, and supports x2apic:

  Intel MultiProcessor Specification v1.4
  MPTABLE: OEM ID: BOCHSCPU
  MPTABLE: Product ID: 0.1
  MPTABLE: APIC at: 0xFEE00000
  BUG: unable to handle page fault for address: ffffffffff5fc020
  #PF: supervisor read access in kernel mode
  #PF: error_code(0x0000) - not-present page
  PGD 25e15067 P4D 25e15067 PUD 25e17067 PMD 25e18067 PTE 0
  Oops: 0000 [#1] SMP NOPTI
  CPU: 0 PID: 0 Comm: swapper Not tainted 5.14.10-300.fc35.x86_64 #1
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1.fc35 04/01/2014
  RIP: 0010:native_apic_mem_read+0x2/0x10
  Code: 14 25 20 cd e3 82 c3 90 bf 30 08 00 00 ff 14 25 18 cd e3 82 c3 cc cc cc 89 ff 89 b7 00 c0 5f ff c3 0f 1f 80 00 00 00 00 89 ff <8b> 87 00 c0 5f ff c3 0f 1f 80 00 00 00 0
  RSP: 0000:ffffffff82e03e18 EFLAGS: 00010046
  RAX: ffffffff81064840 RBX: ffffffffff240b6c RCX: ffffffff82f17428
  RDX: c0000000ffffdfff RSI: 00000000ffffdfff RDI: 0000000000000020
  RBP: ffff888023200000 R08: 0000000000000000 R09: ffffffff82e03c50
  R10: ffffffff82e03c48 R11: ffffffff82f47468 R12: ffffffffff240b40
  R13: ffffffffff200b30 R14: 0000000000000000 R15: 00000000000000d4
  FS:  0000000000000000(0000) GS:ffffffff8365b000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: ffffffffff5fc020 CR3: 0000000025e10000 CR4: 00000000000006b0
  Call Trace:
   ? read_apic_id+0x15/0x30
   ? register_lapic_address+0x76/0x97
   ? default_get_smp_config+0x28b/0x42d
   ? dmi_check_system+0x1c/0x60
   ? acpi_boot_init+0x1d/0x4c3
   ? setup_arch+0xb37/0xc2a
   ? slab_is_available+0x5/0x10
   ? start_kernel+0x61/0x980
   ? load_ucode_bsp+0x4c/0xcd
   ? secondary_startup_64_no_verify+0xc2/0xcb
  Modules linked in:
  CR2: ffffffffff5fc020
  random: get_random_bytes called from oops_exit+0x35/0x60 with crng_init=0
  ---[ end trace c9e569df3bdbefd3 ]---

The panic happens within following init code:
setup_arch()
  ....
  check_x2apic()     <-- x2apic is enabled by first kernel before kexec,
                         this set x2apic_mode = 1, make sure later probes
                         will recognize pre-enabled x2apic.
  ....
  acpi_boot_init();  <-- If ACPI MADT is in use, this will switch apic driver
                         to x2apic, but it will do nothing with MPTABLE.
  x86_dtb_init();
  get_smp_config();
    default_get_smp_config();  <-- MPTABLE setup.
      check_physptr();
        smp_read_mpc();
          register_lapic_address(); <-- * panic here *
  init_apic_mappings();
  ....

The problem here is MPTABLE setup calls register_lapic_address(), which
is still using apic_flat driver, and access the apic MMIO interface. But
the address is never mapped for pre-enabled x2apic, since commit
0450193bffed6 ("x86, x2apic: Don't map lapic addr for preenabled x2apic systems"),
then it panics.

Simply map it won't work either, in x2apic mode the MMIO interface is
not usable (Intel SDM Volume 3A 10.12.2), later setups will still fail with
other errors. So it needs do a proper apic driver probe and switch to
x2apic driver to perform MSR operation instead.

Such issue is currently only seen with kdump/kexec, kernel enabled the
x2apic in first kernel and kept it enabled to 2nd kernel.

This can be easily reproduced with qemu-kvm, use -no-acpi and enable
x2apic, so x2apic with MPTABLE will be in use, then trigger kdump/kexec.

Signed-off-by: Kairui Song <kasong@tencent.com>
---
 arch/x86/kernel/mpparse.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/mpparse.c b/arch/x86/kernel/mpparse.c
index fed721f90116..7658c8184e8c 100644
--- a/arch/x86/kernel/mpparse.c
+++ b/arch/x86/kernel/mpparse.c
@@ -202,8 +202,10 @@ static int __init smp_read_mpc(struct mpc_table *mpc, unsigned early)
 		return 0;
 
 	/* Initialize the lapic mapping */
-	if (!acpi_lapic)
+	if (!acpi_lapic) {
+		apic_early_probe();
 		register_lapic_address(mpc->lapic);
+	}
 
 	if (early)
 		return 1;
-- 
2.35.2