Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Joan Bruguera <joanbrugueram@gmail.com>
To:     Borislav Petkov <bp@alien8.de>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        linux-kernel@vger.kernel.org, x86@kernel.org,
        Thomas Gleixner <tglx@linutronix.de>
Subject: Re: Wake-up from suspend to RAM broken under `retbleed=stuff`
Date:   Sun,  8 Jan 2023 17:11:59 +0000
Message-Id: <20230108171159.1311342-1-joanbrugueram@gmail.com>
In-Reply-To: <Y7qp80cK+r247eCU@zn.tnic>
References: <Y7qp80cK+r247eCU@zn.tnic>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi, sorry that my mail didn't have proper reproduction instructions.

Something important I missed is that on non-affected hardware / QEMU,
the kernel parameter must be `retbleed=stuff,force`, as otherwise
`retbleed=stuff` is a no-op.
In any case, lscpu (or /sys/devices/system/cpu/vulnerabilities/retbleed)
should contain "Mitigation: Stuffing".

That's most likely the missing piece as otherwise, all combinations I've
tested exhibit this behavior, so I assumed that the problem always
reproduces regardless of the config.

More details follow in case it still doesn't reproduce:

The real HW where I found the problem is an HP 250 G6 (Intel i5-7200U),
with a desktop environment (Arch Linux, systemd, Sway, etc.), with both:
* A recent mainline kernel: commit 0a71553536d270e988580a3daa9fc87535908221
* The latest x86-tip: commit 9c4fb147c3492fd4be1b89c22a4c333308f6f44a
The config I use is the following one, which is based on the
Arch Linux kernel config (and I leave any new options at their defaults):
https://aur.archlinux.org/cgit/aur.git/plain/config?h=linux-mainline&id=20ffc62e08f6b0d48a088bccb6e0c3606b88083a
I need to either disable the BPF LSM or apply this patch to boot it:
https://lore.kernel.org/lkml/20230107051456.8800-1-joanbrugueram@gmail.com/
I suspend with `systemctl suspend`. Waking up hangs with `retbleed=stuff`
and works without it.

For verification I booted Fedora-Cloud-Base-Rawhide-20230107.n.0.x86_64.qcow2
(from https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Cloud):
    qemu-system-x86_64 -nodefaults -enable-kvm -smp 1 -m 1024 -vga std \
        -hda Fedora-Cloud-Base-Rawhide-20230107.n.0.x86_64.qcow2 \
        -serial stdio -monitor tcp:127.0.0.1:55555,server,nowait
I appended `retbleed=stuff,force init=/bin/sh` to the kernel command line,
suspended through sysfs and then wrote `system_wakeup` on the QEMU monitor.

For debugging/tracing, I booted the Arch kernel with a busybox initrd:
    qemu-system-x86_64 -nodefaults -no-reboot -initrd initramfs.gz \
        -kernel /boot/vmlinuz-linux-x86tip \
        -append "console=ttyS0 loglevel=4 retbleed=stuff,force" \
        -serial stdio -monitor tcp:127.0.0.1:55555,server,nowait
Suspended through sysfs, enabled singlestep and logs in the QEMU monitor,
then did a system_wakeup.

Regards,
- Joan