Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 29 Jul 2022 18:28:09 +0800
From:   Tao Liu <ltao@redhat.com>
To:     Tom Lendacky <thomas.lendacky@amd.com>
Cc:     Joerg Roedel <joro@8bytes.org>, x86@kernel.org,
        kvm@vger.kernel.org, Peter Zijlstra <peterz@infradead.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        virtualization@lists.linux-foundation.org,
        Arvind Sankar <nivedita@alum.mit.edu>, hpa@zytor.com,
        Jiri Slaby <jslaby@suse.cz>,
        David Rientjes <rientjes@google.com>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Martin Radev <martin.b.radev@gmail.com>,
        Joerg Roedel <jroedel@suse.de>,
        Kees Cook <keescook@chromium.org>,
        Cfir Cohen <cfir@google.com>, linux-coco@lists.linux.dev,
        Andy Lutomirski <luto@kernel.org>,
        Dan Williams <dan.j.williams@intel.com>,
        Juergen Gross <jgross@suse.com>,
        Mike Stunes <mstunes@vmware.com>,
        Sean Christopherson <seanjc@google.com>,
        kexec@lists.infradead.org, linux-kernel@vger.kernel.org,
        Eric Biederman <ebiederm@xmission.com>,
        Erdem Aktas <erdemaktas@google.com>
Subject: Re: [PATCH v3 00/10] x86/sev: KEXEC/KDUMP support for SEV-ES guests
Message-ID: <YuO2OUp4OmnXoqUa@localhost.localdomain>
References: <20220127101044.13803-1-joro@8bytes.org>
 <YmuqifsJltdh7rpv@localhost.localdomain>
 <46b27b72-aabf-f37d-7304-29debeefd8ae@amd.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <46b27b72-aabf-f37d-7304-29debeefd8ae@amd.com>
Precedence: bulk

Hi Tom,

On Fri, Apr 29, 2022 at 08:08:28AM -0500, Tom Lendacky wrote:
> On 4/29/22 04:06, Tao Liu wrote:
> > On Thu, Jan 27, 2022 at 11:10:34AM +0100, Joerg Roedel wrote:
> 
> > 
> > Hi Joerg,
> > 
> > I tried the patch set with 5.17.0-rc1 kernel, and I have a few questions:
> > 
> > 1) Is it a bug or should qemu-kvm 6.2.0 be patched with specific patch? Because
> >     I found it will exit with 0 when I tried to reboot the VM with sev-es enabled.
> >     However with only sev enabled, the VM can do reboot with no problem:
> 
> Qemu was specifically patched to exit on reboot with SEV-ES guests. Qemu
> performs a reboot by resetting the vCPU state, which can't be done with an
> SEV-ES guest because the vCPU state is encrypted.
> 

Sorry for the late response, and thank you for the explanation!

> > 
> > [root@dell-per7525-03 ~]# virsh start TW-SEV-ES --console
> > ....
> > Fedora Linux 35 (Server Edition)
> > Kernel 5.17.0-rc1 on an x86_64 (ttyS0)
> > ....
> > [root@fedora ~]# reboot
> > .....
> > [   48.077682] reboot: Restarting system
> > [   48.078109] reboot: machine restart
> >                         ^^^^^^^^^^^^^^^ guest vm reached restart
> > [root@dell-per7525-03 ~]# echo $?
> > 0
> > ^^^ qemu-kvm exit with 0, no reboot back to normal VM kernel
> > [root@dell-per7525-03 ~]#
> > 
> > 2) With sev-es enabled and the 2 patch sets applied: A) [PATCH v3 00/10] x86/sev:
> > KEXEC/KDUMP support for SEV-ES guests, and B) [PATCH v6 0/7] KVM: SVM: Add initial
> > GHCB protocol version 2 support. I can enable kdump and have vmcore generated:
> > 
> > [root@fedora ~]# dmesg|grep -i sev
> > [    0.030600] SEV: Hypervisor GHCB protocol version support: min=1 max=2
> > [    0.030602] SEV: Using GHCB protocol version 2
> > [    0.296144] AMD Memory Encryption Features active: SEV SEV-ES
> > [    0.450991] SEV: AP jump table Blob successfully set up
> > [root@fedora ~]# kdumpctl status
> > kdump: Kdump is operational
> > 
> > However without the 2 patch sets, I can also enable kdump and have vmcore generated:
> > 
> > [root@fedora ~]# dmesg|grep -i sev
> > [    0.295754] AMD Memory Encryption Features active: SEV SEV-ES
> >                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ patch set A & B
> > 	       not applied, so only have this string.
> > [root@fedora ~]# echo c > /proc/sysrq-trigger
> > ...
> > [    2.759403] kdump[549]: saving vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2022-04-18-05:58:50/
> > [    2.804355] kdump[555]: saving vmcore-dmesg.txt complete
> > [    2.806915] kdump[557]: saving vmcore
> >                             ^^^^^^^^^^^^^ vmcore can still be generated
> > ...
> > [    7.068981] reboot: Restarting system
> > [    7.069340] reboot: machine restart
> > 
> > [root@dell-per7525-03 ~]# echo $?
> > 0
> > ^^^ same exit issue as question 1.
> > 
> > I doesn't have a complete technical background of the patch set, but isn't
> > it the issue which this patch set is trying to solve? Or I missed something?
> 
> The main goal of this patch set is to really to solve the ability to perform
> a kexec. I would expect kdump to work since kdump shuts down all but the
> executing vCPU and performs its operations before "rebooting" (which will
> exit Qemu as I mentioned above). But kexec requires the need to restart the
> APs from within the guest after they have been stopped. That requires
> specific support and actions on the part of the guest kernel in how the APs
> are stopped and restarted.

Recently I got one sev-es flaged machine borrowed and retested the patch, which
worked fine for kexec when sev-es enabled. With the patchset applied in 5.17.0-rc1, 
kexec'ed kernel can bring up all APs with no problem.

However as for kdump, I find one issue. Although kdump kernel can work well on one
cpu, but we can still enable multi-cpus by removing the "nr_cpus=1" kernel parameter
in kdump sysconfig. I was expecting kdump kernel can bring up all APs as kexec did,
however:

[    0.000000] Command line: elfcorehdr=0x5b000000 BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.0-rc1+ ro resume=/dev/mapper/rhel-swap biosdevname=0 net.ifnames=0 console=ttyS0 irqpoll reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable disable_cpu_apicid=0 iTCO_wdt.pretimeout=0
...
[    0.376663] smp: Bringing up secondary CPUs ...
[    0.377599] x86: Booting SMP configuration:
[    0.378342] .... node  #0, CPUs:      #1
[   10.377698] smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
[   10.379882]  #2
[   20.379645] smpboot: do_boot_cpu failed(-1) to wakeup CPU#2
[   20.380648] smp: Brought up 1 node, 1 CPU
[   20.381600] smpboot: Max logical packages: 4
[   20.382597] smpboot: Total of 1 processors activated (4192.00 BogoMIPS)

Turns out for kdump, the APs were not stopped properly, so I modified the following code:

--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -26,6 +26,7 @@
 #include <asm/cpu.h>
 #include <asm/nmi.h>
 #include <asm/smp.h>
+#include <asm/sev.h>
 
 #include <linux/ctype.h>
 #include <linux/mc146818rtc.h>
@@ -821,6 +822,7 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs)
 
        atomic_dec(&waiting_for_crash_ipi);
        /* Assume hlt works */
+       sev_es_stop_this_cpu();
        halt();
        for (;;)
                cpu_relax();

[    0.000000] Command line: elfcorehdr=0x5b000000 BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.0-rc1-hack+ ro resume=/dev/mapper/rhel-swap biosdevname=0 net.ifnames=0 console=ttyS0 irqpoll reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable disable_cpu_apicid=0 iTCO_wdt.pretimeout=0
...
[    0.402618] smp: Bringing up secondary CPUs ...
[    0.403308] x86: Booting SMP configuration:
[    0.404171] .... node  #0, CPUs:      #1 #2 #3
[    0.407362] smp: Brought up 1 node, 4 CPUs
[    0.408907] smpboot: Max logical packages: 4
[    0.409172] smpboot: Total of 4 processors activated (16768.01 BogoMIPS)

Now all APs can work in kdump kernel.

Thanks,
Tao Liu

> 
> Thanks,
> Tom
> 
> > 
> > Thanks,
> > Tao Liu
> > > _______________________________________________
> > > Virtualization mailing list
> > > Virtualization@lists.linux-foundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
> > 
>