Greeting,
FYI, we noticed the following commit (built with clang-14):
commit: f154f290855b070cc94dd44ad253c0ef8a9337bb ("x86/mm/64: Flush global TLB on boot and AP bringup")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git x86/mm
in testcase: boot
on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+-------------------------------------------------+------------+------------+
| | 9de4999050 | f154f29085 |
+-------------------------------------------------+------------+------------+
| BUG:kernel_reboot-without-warning_in_boot_stage | 0 | 13 |
+-------------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
Physical KASLR using RDTSC...
Virtual KASLR using RDTSC...
Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel.
BUG: kernel reboot-without-warning in boot stage
Linux version 5.16.0-rc3-00003-gf154f290855b #1
Command line: ip=::::vm-snb-192::dhcp root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_SERVER=internal-lkp-server selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw rcuperf.shutdown=0 watchdog_thresh=240
Kboot worker: lkp-worker30
Elapsed time: 60
To reproduce:
# build kernel
cd linux
cp config-5.16.0-rc3-00003-gf154f290855b .config
make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
cd <mod-install-dir>
find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email
# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang
On Thu, Dec 09, 2021 at 10:41:41PM +0800, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed the following commit (built with clang-14):
>
> commit: f154f290855b070cc94dd44ad253c0ef8a9337bb ("x86/mm/64: Flush global TLB on boot and AP bringup")
> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git x86/mm
>
> in testcase: boot
>
> on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
>
> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>
>
> +-------------------------------------------------+------------+------------+
> | | 9de4999050 | f154f29085 |
> +-------------------------------------------------+------------+------------+
> | BUG:kernel_reboot-without-warning_in_boot_stage | 0 | 13 |
> +-------------------------------------------------+------------+------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <[email protected]>
>
>
> Physical KASLR using RDTSC...
> Virtual KASLR using RDTSC...
>
> Decompressing Linux... Parsing ELF... Performing relocations... done.
> Booting the kernel.
> BUG: kernel reboot-without-warning in boot stage
> Linux version 5.16.0-rc3-00003-gf154f290855b #1
> Command line: ip=::::vm-snb-192::dhcp root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_SERVER=internal-lkp-server selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw rcuperf.shutdown=0 watchdog_thresh=240
>
> Kboot worker: lkp-worker30
> Elapsed time: 60
My guest boots fine here on an AMD host, albeit it takes a while until
"Booting the kernel" appears.
You probably should verify by hand whether your guest simply needs
longer before it starts booting the kernel...
early console in setup code
early console in extract_kernel
input_data: 0x0000000005ce82e0
input_len: 0x0000000001d21920
output: 0x0000000001000000
output_len: 0x000000000650d2ec
kernel_total_size: 0x0000000006e2c000
needed_size: 0x0000000007000000
trampoline_32bit: 0x000000000009d000
Physical KASLR using RDTSC...
Virtual KASLR using RDTSC...
Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel.
[ 0.000000][ T0] Linux version 5.16.0-rc3-00005-g35fa745286ac (boris@zn) (ClangBuiltLinux clang version 14.0.0 (https://github.com/llvm/llvm-project 08a770c378ca5d8975ef305d15f297c3f980d186), GNU ld (GNU Binutils for Debian) 2.37) #3 PREEMPT Tue Dec 14 17:29:41 CET 2021
[ 0.000000][ T0] Command line: root=/dev/sda1 resume=/dev/sda2 debug ignore_loglevel log_buf_len=16M earlyprintk=ttyS0,115200 console=ttyS0,115200 console=tty0 no_console_suspend net.ifnames=0
[ 0.000000][ T0] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000][ T0] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000][ T0] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000][ T0] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000][ T0] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000][ T0] signal: max sigframe size: 1360
[ 0.000000][ T0] printk: debug: ignoring loglevel setting.
[ 0.000000][ T0] printk: bootconsole [earlyser0] enabled
[ 0.000000][ T0] BIOS-provided physical RAM map:
...
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
Hi Boris,
On Tue, Dec 14, 2021 at 05:38:57PM +0100, Borislav Petkov wrote:
> On Thu, Dec 09, 2021 at 10:41:41PM +0800, kernel test robot wrote:
> >
> >
> > Greeting,
> >
> > FYI, we noticed the following commit (built with clang-14):
> >
> > commit: f154f290855b070cc94dd44ad253c0ef8a9337bb ("x86/mm/64: Flush global TLB on boot and AP bringup")
> > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git x86/mm
> >
> > in testcase: boot
> >
> > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
> >
> > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
> >
> >
> > +-------------------------------------------------+------------+------------+
> > | | 9de4999050 | f154f29085 |
> > +-------------------------------------------------+------------+------------+
> > | BUG:kernel_reboot-without-warning_in_boot_stage | 0 | 13 |
> > +-------------------------------------------------+------------+------------+
> >
> >
> > If you fix the issue, kindly add following tag
> > Reported-by: kernel test robot <[email protected]>
> >
> >
> > Physical KASLR using RDTSC...
> > Virtual KASLR using RDTSC...
> >
> > Decompressing Linux... Parsing ELF... Performing relocations... done.
> > Booting the kernel.
> > BUG: kernel reboot-without-warning in boot stage
> > Linux version 5.16.0-rc3-00003-gf154f290855b #1
> > Command line: ip=::::vm-snb-192::dhcp root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_SERVER=internal-lkp-server selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw rcuperf.shutdown=0 watchdog_thresh=240
> >
> > Kboot worker: lkp-worker30
> > Elapsed time: 60
>
> My guest boots fine here on an AMD host, albeit it takes a while until
> "Booting the kernel" appears.
>
> You probably should verify by hand whether your guest simply needs
> longer before it starts booting the kernel...
We have verified by hand, still can reproduce this issue.
We also double checked, this issue still exists in 35fa745286 ("x86/mm: Include
spinlock_t definition in pgtable") which is the head of tip/x86/mm, thanks.
=========================================================================================
compiler/kconfig/rootfs/sleep/tbox_group/testcase:
clang-14/x86_64-randconfig-a013-20211207/debian-10.4-x86_64-20200603.cgz/1/vm-snb/boot
commit:
9de4999050 ("x86/realmode: Add comment for Global bit usage in trampoline_pgd")
f154f29085 ("x86/mm/64: Flush global TLB on boot and AP bringup")
35fa745286 ("x86/mm: Include spinlock_t definition in pgtable.")
9de4999050b5f2e8 f154f290855b070cc94dd44ad25 35fa745286ac44ee26ed100c2bd
---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
:27 100% 27:27 100% 27:27 dmesg.BUG:kernel_reboot-without-warning_in_boot_stage
>
> early console in setup code
> early console in extract_kernel
> input_data: 0x0000000005ce82e0
> input_len: 0x0000000001d21920
> output: 0x0000000001000000
> output_len: 0x000000000650d2ec
> kernel_total_size: 0x0000000006e2c000
> needed_size: 0x0000000007000000
> trampoline_32bit: 0x000000000009d000
> Physical KASLR using RDTSC...
> Virtual KASLR using RDTSC...
>
> Decompressing Linux... Parsing ELF... Performing relocations... done.
> Booting the kernel.
> [ 0.000000][ T0] Linux version 5.16.0-rc3-00005-g35fa745286ac (boris@zn) (ClangBuiltLinux clang version 14.0.0 (https://github.com/llvm/llvm-project 08a770c378ca5d8975ef305d15f297c3f980d186), GNU ld (GNU Binutils for Debian) 2.37) #3 PREEMPT Tue Dec 14 17:29:41 CET 2021
> [ 0.000000][ T0] Command line: root=/dev/sda1 resume=/dev/sda2 debug ignore_loglevel log_buf_len=16M earlyprintk=ttyS0,115200 console=ttyS0,115200 console=tty0 no_console_suspend net.ifnames=0
> [ 0.000000][ T0] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
> [ 0.000000][ T0] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
> [ 0.000000][ T0] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
> [ 0.000000][ T0] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
> [ 0.000000][ T0] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
> [ 0.000000][ T0] signal: max sigframe size: 1360
> [ 0.000000][ T0] printk: debug: ignoring loglevel setting.
> [ 0.000000][ T0] printk: bootconsole [earlyser0] enabled
> [ 0.000000][ T0] BIOS-provided physical RAM map:
> ...
>
> --
> Regards/Gruss,
> Boris.
>
> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG N?rnberg
> _______________________________________________
> LKP mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
On Wed, Dec 15, 2021 at 03:00:13PM +0800, Carel Si wrote:
> We have verified by hand, still can reproduce this issue.
Ok, please give details how exactly you reproduce: host, guest, kernel
versions, configs, machine types, i.e., /proc/cpuinfo, dmesg, etc. I'd
like to see if I can find a similar machine here.
Also, would it be possible to upload your vmlinuz somewhere so that I
can download it for testing?
Thx.
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
Hi Boris,
On 12/15/2021 6:05 PM, Borislav Petkov wrote:
> On Wed, Dec 15, 2021 at 03:00:13PM +0800, Carel Si wrote:
>> We have verified by hand, still can reproduce this issue.
>
> Ok, please give details how exactly you reproduce: host, guest, kernel
> versions, configs, machine types, i.e., /proc/cpuinfo, dmesg, etc. I'd
> like to see if I can find a similar machine here.
>
> Also, would it be possible to upload your vmlinuz somewhere so that I
> can download it for testing?
The testing was with Qemu. And we found that the hang is related with
clang-14.
The original report showed the kernel is built with clang-14:
# build kernel
cd linux
cp config-5.16.0-rc3-00003-gf154f290855b .config
make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
And the clang-14 generate different code comparing to clang-11. I pasted
the native_write_cr4 assembly code generated with clang-14 and clang-11 to:
https://zerobin.net/?ced930258536c677#U6et+H97oxbpdYclFvAX0F3ha0rCJctLE53mJjDKrgo=
The extra code generated by clang-14 is like:
ffffffff810b8784: 48 89 d8 mov %rbx,%rax
ffffffff810b8787: 48 c1 e8 03 shr $0x3,%rax
ffffffff810b878b: 48 b9 00 00 00 00 00 movabs $0xdffffc0000000000,%rcx
ffffffff810b8792: fc ff df
ffffffff810b8795: 80 3c 08 00 cmpb $0x0,(%rax,%rcx,1)
--> Qemu reboot after this instruction from x86_64_start_kernel
ffffffff810b8799: 74 08 je ffffffff810b87a3 <native_write_cr4+0x84>
ffffffff810b879b: 48 89 df mov %rbx,%rdi
ffffffff810b879e: e8 cc 7c 64 00 callq ffffffff8170046f <__asan_report_load8_noabort>
ffffffff810b87a3: 48 ff 03 incq (%rbx)
ffffffff810b87a6: 5b pop %rbx
Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
This function is early function called before kasan_init.
Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
be happy with this kind of early TLB flush? Thanks.
Regards
Yin, Fengwei
>
> Thx.
>
On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
> The testing was with Qemu.
This is hardly what I asked for.
> And we found that the hang is related with clang-14.
I saw that already.
> The original report showed the kernel is built with clang-14:
> # build kernel
> cd linux
> cp config-5.16.0-rc3-00003-gf154f290855b .config
> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
I saw that too.
> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
> This function is early function called before kasan_init.
>
> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
> be happy with this kind of early TLB flush? Thanks.
Ok, I don't understand: I asked for how exactly to reproduce and whether
you can send me your vmlinux you built with your clang-14. What I get is
some possible explanation about what might be happening.
So what do you expect me to do? Say, "oh, sure, you're right, send me a
patch" without even being able to see for myself what the root cause is?
What if it is not the kernel's fault but clang-14 is miscompiling crap
as in so many other cases?
I built clang-14 and built with your .config and it works here fine. So
why does yours fail?
Or what's the point of all this?
I mean, if you cannot send me what I ask for, you can say so. Then I can
ignore this whole report altogether and waste my time somewhere else.
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
Hi Boris,
On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
> > The testing was with Qemu.
>
> This is hardly what I asked for.
>
> > And we found that the hang is related with clang-14.
>
> I saw that already.
>
> > The original report showed the kernel is built with clang-14:
> > # build kernel
> > cd linux
> > cp config-5.16.0-rc3-00003-gf154f290855b .config
> > make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
> > make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
>
> I saw that too.
>
> > Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
> > This function is early function called before kasan_init.
> >
> > Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
> > be happy with this kind of early TLB flush? Thanks.
>
> Ok, I don't understand: I asked for how exactly to reproduce and whether
> you can send me your vmlinux you built with your clang-14. What I get is
> some possible explanation about what might be happening.
>
> So what do you expect me to do? Say, "oh, sure, you're right, send me a
> patch" without even being able to see for myself what the root cause is?
>
> What if it is not the kernel's fault but clang-14 is miscompiling crap
> as in so many other cases?
>
> I built clang-14 and built with your .config and it works here fine. So
> why does yours fail?
>
> Or what's the point of all this?
>
> I mean, if you cannot send me what I ask for, you can say so. Then I can
> ignore this whole report altogether and waste my time somewhere else.
We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
Machine types can refer to:
https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
If there's any other msg needed, pls feel free to propose, thanks.
Below are our full steps to reproduce the issue:
# download lkp-tests
$ git clone https://github.com/intel/lkp-tests.git
$ cd lkp-tests/
# download vmlinuz
$ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
# dowmload modules.cgz
$ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
# download job-script which is attached
# run lkp qemu
lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
~/lkp-tests/pkg/lkp-src ~/lkp-tests
x86_64
==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Removing existing $pkgdir/ directory...
==> Starting build()...
make: Entering directory '/home/carel/lkp-tests/bin/event'
klcc -D_FORTIFY_SOURCE=2 -c -o wakeup.o wakeup.c
klcc -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
rm -f wakeup.o
strip wakeup
make: Leaving directory '/home/carel/lkp-tests/bin/event'
==> Entering fakeroot environment...
x86_64
==> Starting package()...
==> Creating package "lkp-src"...
103987 blocks
renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
==> Leaving fakeroot environment.
==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
~/lkp-tests
12 blocks
result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
downloading initrds ...
use local modules: /home/carel/.lkp/cache/modules.cgz
/usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
440459 blocks
/usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
1773 blocks
/usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
2321 blocks
/usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
6856 blocks
exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
early console in setup code
early console in extract_kernel
input_data: 0x0000000006ffc2e0
input_len: 0x000000000260cb2b
output: 0x0000000001000000
output_len: 0x00000000079e7da4
kernel_total_size: 0x0000000008a2c000
needed_size: 0x0000000008c00000
trampoline_32bit: 0x000000000009d000
Physical KASLR using RDTSC...
Virtual KASLR using RDTSC...
Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel.
>
> --
> Regards/Gruss,
> Boris.
>
> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG N?rnberg
Hi Boris,
On 12/16/2021 7:58 PM, Carel Si wrote:
> Hi Boris,
>
> On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
>> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
>>> The testing was with Qemu.
>>
>> This is hardly what I asked for.
>>
>>> And we found that the hang is related with clang-14.
>>
>> I saw that already.
>>
>>> The original report showed the kernel is built with clang-14:
>>> # build kernel
>>> cd linux
>>> cp config-5.16.0-rc3-00003-gf154f290855b .config
>>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
>>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
>>
>> I saw that too.
>>
>>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
>>> This function is early function called before kasan_init.
>>>
>>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
>>> be happy with this kind of early TLB flush? Thanks.
>>
>> Ok, I don't understand: I asked for how exactly to reproduce and whether
>> you can send me your vmlinux you built with your clang-14. What I get is
>> some possible explanation about what might be happening.
>>
>> So what do you expect me to do? Say, "oh, sure, you're right, send me a
>> patch" without even being able to see for myself what the root cause is?
>>
>> What if it is not the kernel's fault but clang-14 is miscompiling crap
>> as in so many other cases?
>>
>> I built clang-14 and built with your .config and it works here fine. So
>> why does yours fail?
>>
>> Or what's the point of all this?
I had concern that our report is an invalid report because you can't reproduce
it in your side. If that's the case, it could waste more your time. That's why
I did check and shared what I got. I am very sorry if I did it wrong.
If you don't want to use lkp tool to reproduce the issue, following command
could be used as well:
Use Qemu command so only kernel image need be downloaded:
qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200"
to reproduce it.
Regards
Yin, Fengwei
>>
>> I mean, if you cannot send me what I ask for, you can say so. Then I can
>> ignore this whole report altogether and waste my time somewhere else.
>
> We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
> https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
>
> Machine types can refer to:
> https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
>
> If there's any other msg needed, pls feel free to propose, thanks.
>
> Below are our full steps to reproduce the issue:
>
> # download lkp-tests
> $ git clone https://github.com/intel/lkp-tests.git
>
> $ cd lkp-tests/
>
> # download vmlinuz
> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
>
> # dowmload modules.cgz
> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
>
> # download job-script which is attached
>
> # run lkp qemu
> lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
>
> ~/lkp-tests/pkg/lkp-src ~/lkp-tests
> x86_64
> ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
> ==> Checking runtime dependencies...
> ==> Checking buildtime dependencies...
> ==> WARNING: Using existing $srcdir/ tree
> ==> Removing existing $pkgdir/ directory...
> ==> Starting build()...
> make: Entering directory '/home/carel/lkp-tests/bin/event'
> klcc -D_FORTIFY_SOURCE=2 -c -o wakeup.o wakeup.c
> klcc -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
> rm -f wakeup.o
> strip wakeup
> make: Leaving directory '/home/carel/lkp-tests/bin/event'
> ==> Entering fakeroot environment...
> x86_64
> ==> Starting package()...
> ==> Creating package "lkp-src"...
> 103987 blocks
> renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
> ==> Leaving fakeroot environment.
> ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
> ~/lkp-tests
> 12 blocks
> result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
> downloading initrds ...
> use local modules: /home/carel/.lkp/cache/modules.cgz
> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
> 440459 blocks
> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> 1773 blocks
> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> 2321 blocks
> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> 6856 blocks
> exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
> early console in setup code
> early console in extract_kernel
> input_data: 0x0000000006ffc2e0
> input_len: 0x000000000260cb2b
> output: 0x0000000001000000
> output_len: 0x00000000079e7da4
> kernel_total_size: 0x0000000008a2c000
> needed_size: 0x0000000008c00000
> trampoline_32bit: 0x000000000009d000
> Physical KASLR using RDTSC...
> Virtual KASLR using RDTSC...
>
> Decompressing Linux... Parsing ELF... Performing relocations... done.
> Booting the kernel.
>
>>
>> --
>> Regards/Gruss,
>> Boris.
>>
>> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
Add Bruce and clang folks to the party.
On Thu, Dec 16, 2021 at 08:21:15PM +0800, Yin Fengwei wrote:
> Hi Boris,
>
> On 12/16/2021 7:58 PM, Carel Si wrote:
> > Hi Boris,
> >
> > On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
> >> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
> >>> The testing was with Qemu.
> >>
> >> This is hardly what I asked for.
> >>
> >>> And we found that the hang is related with clang-14.
> >>
> >> I saw that already.
> >>
> >>> The original report showed the kernel is built with clang-14:
> >>> # build kernel
> >>> cd linux
> >>> cp config-5.16.0-rc3-00003-gf154f290855b .config
> >>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
> >>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
> >>
> >> I saw that too.
> >>
> >>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
> >>> This function is early function called before kasan_init.
> >>>
> >>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
> >>> be happy with this kind of early TLB flush? Thanks.
> >>
> >> Ok, I don't understand: I asked for how exactly to reproduce and whether
> >> you can send me your vmlinux you built with your clang-14. What I get is
> >> some possible explanation about what might be happening.
> >>
> >> So what do you expect me to do? Say, "oh, sure, you're right, send me a
> >> patch" without even being able to see for myself what the root cause is?
> >>
> >> What if it is not the kernel's fault but clang-14 is miscompiling crap
> >> as in so many other cases?
> >>
> >> I built clang-14 and built with your .config and it works here fine. So
> >> why does yours fail?
> >>
> >> Or what's the point of all this?
>
> I had concern that our report is an invalid report because you can't reproduce
> it in your side. If that's the case, it could waste more your time. That's why
> I did check and shared what I got. I am very sorry if I did it wrong.
Sure, you can always add your analysis but I'd like to reproduce myself
too. So, in the future, please answer the questions and then feel free
to add your analysis - I'll gladly have a look.
Which wasn't that far from the truth, btw.
But it isn't KASAN but GCOV profiling. Or is it KCOV profiling which
clang does.
That thing adds some counting glue to native_write_cr4():
(my comments from the actual singlestepping in qemu start with '##' below)
movq $__llvm_gcov_ctr.48+8, %rbx ## mov $0xffffffff8837d3c0,%rbx
.LBB8_1: # %set_register
# =>This Inner Loop Header: Depth=1
jmp .Ltmp42
...
.Ltmp42: # Block address taken
.LBB8_7: # %if.end79
movq %rbx, %rax ## 0xffffffff8837d3c0
shrq $3, %rax ## 0x1ffffffff106fa78
movabsq $-2305847407260205056, %rcx # imm = 0xDFFFFC0000000000 ## 0xdffffc0000000000
cmpb $0, (%rax,%rcx)
je .LBB8_9
so the memory address CMP accesses is something as nonsensical as
0xfffffbfff106fa78
so I'm guessing we need to setup something for that __llvm_gcov_ctr to
deref properly but I haven't dug deeper.
The important thing is that this triggers with clang-13 and -14. gcc is
fine with the same config but that probably is because gcc does other
profiling - gcov - I guess. Looking at the resulting asm, it has a bunch
of those counter increments:
incq __gcov0.native_write_cr4+88(%rip) # __gcov0.native_write_cr4[11]
but no weird memory references.
So, clang folks, what's up?
The fix is simple but I'd like to understand first why does this fail
only with clang, 13 and newer.
(I mean, melver pointed me to
380d53c45ff2 ("compiler_attributes.h: define __no_profile, add to noinstr")
which explains why 13 and newer).
Btw, joro, that second hunk is I think needed too because a couple of
lines earlier we set up the cr4 shadow so I think you should use it
instead of touching the hw CR4.
---
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464de5e3..79b3d67addcc 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
}
EXPORT_SYMBOL(native_write_cr0);
-void native_write_cr4(unsigned long val)
+void __no_profile native_write_cr4(unsigned long val)
{
unsigned long bits_changed = 0;
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 75acb6027a87..68d2b7f9a913 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
/* Kill off the identity-map trampoline */
reset_early_page_tables();
- __native_tlb_flush_global(native_read_cr4());
+ __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
clear_bss();
Leaving in the rest for the newly added folks.
> If you don't want to use lkp tool to reproduce the issue, following command
> could be used as well:
>
> Use Qemu command so only kernel image need be downloaded:
> qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200"
> to reproduce it.
>
>
>
> Regards
> Yin, Fengwei
>
>
>
> >>
> >> I mean, if you cannot send me what I ask for, you can say so. Then I can
> >> ignore this whole report altogether and waste my time somewhere else.
> >
> > We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
> > https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
> >
> > Machine types can refer to:
> > https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
> >
> > If there's any other msg needed, pls feel free to propose, thanks.
> >
> > Below are our full steps to reproduce the issue:
> >
> > # download lkp-tests
> > $ git clone https://github.com/intel/lkp-tests.git
> >
> > $ cd lkp-tests/
> >
> > # download vmlinuz
> > $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
> >
> > # dowmload modules.cgz
> > $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
> >
> > # download job-script which is attached
> >
> > # run lkp qemu
> > lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
> >
> > ~/lkp-tests/pkg/lkp-src ~/lkp-tests
> > x86_64
> > ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
> > ==> Checking runtime dependencies...
> > ==> Checking buildtime dependencies...
> > ==> WARNING: Using existing $srcdir/ tree
> > ==> Removing existing $pkgdir/ directory...
> > ==> Starting build()...
> > make: Entering directory '/home/carel/lkp-tests/bin/event'
> > klcc -D_FORTIFY_SOURCE=2 -c -o wakeup.o wakeup.c
> > klcc -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
> > rm -f wakeup.o
> > strip wakeup
> > make: Leaving directory '/home/carel/lkp-tests/bin/event'
> > ==> Entering fakeroot environment...
> > x86_64
> > ==> Starting package()...
> > ==> Creating package "lkp-src"...
> > 103987 blocks
> > renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
> > ==> Leaving fakeroot environment.
> > ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
> > ~/lkp-tests
> > 12 blocks
> > result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
> > downloading initrds ...
> > use local modules: /home/carel/.lkp/cache/modules.cgz
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
> > 440459 blocks
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > 1773 blocks
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > 2321 blocks
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > 6856 blocks
> > exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
> > early console in setup code
> > early console in extract_kernel
> > input_data: 0x0000000006ffc2e0
> > input_len: 0x000000000260cb2b
> > output: 0x0000000001000000
> > output_len: 0x00000000079e7da4
> > kernel_total_size: 0x0000000008a2c000
> > needed_size: 0x0000000008c00000
> > trampoline_32bit: 0x000000000009d000
> > Physical KASLR using RDTSC...
> > Virtual KASLR using RDTSC...
> >
> > Decompressing Linux... Parsing ELF... Performing relocations... done.
> > Booting the kernel.
> >
> >>
> >> --
> >> Regards/Gruss,
> >> Boris.
> >>
> >> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
Hi Boris,
On Fri, Dec 17, 2021 at 01:52:53PM +0100, Borislav Petkov wrote:
> Add Bruce and clang folks to the party.
>
> On Thu, Dec 16, 2021 at 08:21:15PM +0800, Yin Fengwei wrote:
> > Hi Boris,
> >
> > On 12/16/2021 7:58 PM, Carel Si wrote:
> > > Hi Boris,
> > >
> > > On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
> > >> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
> > >>> The testing was with Qemu.
> > >>
> > >> This is hardly what I asked for.
> > >>
> > >>> And we found that the hang is related with clang-14.
> > >>
> > >> I saw that already.
> > >>
> > >>> The original report showed the kernel is built with clang-14:
> > >>> # build kernel
> > >>> cd linux
> > >>> cp config-5.16.0-rc3-00003-gf154f290855b .config
> > >>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
> > >>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
> > >>
> > >> I saw that too.
> > >>
> > >>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
> > >>> This function is early function called before kasan_init.
> > >>>
> > >>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
> > >>> be happy with this kind of early TLB flush? Thanks.
> > >>
> > >> Ok, I don't understand: I asked for how exactly to reproduce and whether
> > >> you can send me your vmlinux you built with your clang-14. What I get is
> > >> some possible explanation about what might be happening.
> > >>
> > >> So what do you expect me to do? Say, "oh, sure, you're right, send me a
> > >> patch" without even being able to see for myself what the root cause is?
> > >>
> > >> What if it is not the kernel's fault but clang-14 is miscompiling crap
> > >> as in so many other cases?
> > >>
> > >> I built clang-14 and built with your .config and it works here fine. So
> > >> why does yours fail?
> > >>
> > >> Or what's the point of all this?
> >
> > I had concern that our report is an invalid report because you can't reproduce
> > it in your side. If that's the case, it could waste more your time. That's why
> > I did check and shared what I got. I am very sorry if I did it wrong.
>
> Sure, you can always add your analysis but I'd like to reproduce myself
> too. So, in the future, please answer the questions and then feel free
> to add your analysis - I'll gladly have a look.
>
> Which wasn't that far from the truth, btw.
>
> But it isn't KASAN but GCOV profiling. Or is it KCOV profiling which
> clang does.
This is GCOV, -fprofile-arcs.
> That thing adds some counting glue to native_write_cr4():
>
> (my comments from the actual singlestepping in qemu start with '##' below)
>
> movq $__llvm_gcov_ctr.48+8, %rbx ## mov $0xffffffff8837d3c0,%rbx
> .LBB8_1: # %set_register
> # =>This Inner Loop Header: Depth=1
>
> jmp .Ltmp42
> ...
>
> .Ltmp42: # Block address taken
> .LBB8_7: # %if.end79
> movq %rbx, %rax ## 0xffffffff8837d3c0
> shrq $3, %rax ## 0x1ffffffff106fa78
> movabsq $-2305847407260205056, %rcx # imm = 0xDFFFFC0000000000 ## 0xdffffc0000000000
> cmpb $0, (%rax,%rcx)
> je .LBB8_9
>
> so the memory address CMP accesses is something as nonsensical as
>
> 0xfffffbfff106fa78
>
> so I'm guessing we need to setup something for that __llvm_gcov_ctr to
> deref properly but I haven't dug deeper.
I am not reallys ure how exactly GCOV works under the hood so I cannot
really comment on it (Nick might); it seems like llvm_gcov_init needs to
get called for __llvm_gcov_ctr to get set up properly and maybe that
hasn't happened at the point.
> The important thing is that this triggers with clang-13 and -14. gcc is
> fine with the same config but that probably is because gcc does other
> profiling - gcov - I guess. Looking at the resulting asm, it has a bunch
> of those counter increments:
>
> incq __gcov0.native_write_cr4+88(%rip) # __gcov0.native_write_cr4[11]
>
> but no weird memory references.
>
> So, clang folks, what's up?
>
> The fix is simple but I'd like to understand first why does this fail
> only with clang, 13 and newer.
>
> (I mean, melver pointed me to
>
> 380d53c45ff2 ("compiler_attributes.h: define __no_profile, add to noinstr")
>
> which explains why 13 and newer).
This is a bit of a brain dump, apologies for not offering much upfront
analysis, I am not as familiar with LLVM internals as Nick but this
might help others look into the problem.
I ended up seeing this thread yesterday through a lore filter that I
have and I bisected LLVM based on the fact that it happened with
clang-13 but not clang-12; that bisect pointed out Nick's commit in LLVM
that added the no profile attribute, which means that GCOV and KASAN
need to be enabled to see this bug. I was not able to reproduce it with
just one of them enabled at a time.
With that, I removed the no profile attribute dependency on GCOV_KERNEL
and bisected again, landing on the commit in LLVM 13 that enables the
new pass manager, which fundamentally changes how LLVM transforms its
IR. Whenever that has happened in the past, it usually points to a
pre-existing issue; if I go back to clang-11 (the current minimum of
-next) and enable the NPM there with -fexperimental-new-pass-manager, I
see this hang so it seems like there might be some issue how GCOV and
KASAN are manipulated together in the context of the NPM that was not
present with the legacy pass manager. I do see tests in LLVM that test
to make sure __llvm_gcov_ctr does not get instrumented with KASAN, maybe
there is another interaction that should not be happening between those
two?
> Btw, joro, that second hunk is I think needed too because a couple of
> lines earlier we set up the cr4 shadow so I think you should use it
> instead of touching the hw CR4.
>
> ---
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 0083464de5e3..79b3d67addcc 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
> }
> EXPORT_SYMBOL(native_write_cr0);
>
> -void native_write_cr4(unsigned long val)
> +void __no_profile native_write_cr4(unsigned long val)
> {
> unsigned long bits_changed = 0;
>
> diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
> index 75acb6027a87..68d2b7f9a913 100644
> --- a/arch/x86/kernel/head64.c
> +++ b/arch/x86/kernel/head64.c
> @@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
> /* Kill off the identity-map trampoline */
> reset_early_page_tables();
>
> - __native_tlb_flush_global(native_read_cr4());
> + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
>
> clear_bss();
>
>
>
> Leaving in the rest for the newly added folks.
>
> > If you don't want to use lkp tool to reproduce the issue, following command
> > could be used as well:
> >
> > Use Qemu command so only kernel image need be downloaded:
> > qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200"
> > to reproduce it.
> >
> >
> >
> > Regards
> > Yin, Fengwei
> >
> >
> >
> > >>
> > >> I mean, if you cannot send me what I ask for, you can say so. Then I can
> > >> ignore this whole report altogether and waste my time somewhere else.
> > >
> > > We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
> > > https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
> > >
> > > Machine types can refer to:
> > > https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
> > >
> > > If there's any other msg needed, pls feel free to propose, thanks.
> > >
> > > Below are our full steps to reproduce the issue:
> > >
> > > # download lkp-tests
> > > $ git clone https://github.com/intel/lkp-tests.git
> > >
> > > $ cd lkp-tests/
> > >
> > > # download vmlinuz
> > > $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
> > >
> > > # dowmload modules.cgz
> > > $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
> > >
> > > # download job-script which is attached
> > >
> > > # run lkp qemu
> > > lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
> > >
> > > ~/lkp-tests/pkg/lkp-src ~/lkp-tests
> > > x86_64
> > > ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
> > > ==> Checking runtime dependencies...
> > > ==> Checking buildtime dependencies...
> > > ==> WARNING: Using existing $srcdir/ tree
> > > ==> Removing existing $pkgdir/ directory...
> > > ==> Starting build()...
> > > make: Entering directory '/home/carel/lkp-tests/bin/event'
> > > klcc -D_FORTIFY_SOURCE=2 -c -o wakeup.o wakeup.c
> > > klcc -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
> > > rm -f wakeup.o
> > > strip wakeup
> > > make: Leaving directory '/home/carel/lkp-tests/bin/event'
> > > ==> Entering fakeroot environment...
> > > x86_64
> > > ==> Starting package()...
> > > ==> Creating package "lkp-src"...
> > > 103987 blocks
> > > renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
> > > ==> Leaving fakeroot environment.
> > > ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
> > > ~/lkp-tests
> > > 12 blocks
> > > result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
> > > downloading initrds ...
> > > use local modules: /home/carel/.lkp/cache/modules.cgz
> > > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
> > > 440459 blocks
> > > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > > 1773 blocks
> > > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > > 2321 blocks
> > > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > > 6856 blocks
> > > exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
> > > early console in setup code
> > > early console in extract_kernel
> > > input_data: 0x0000000006ffc2e0
> > > input_len: 0x000000000260cb2b
> > > output: 0x0000000001000000
> > > output_len: 0x00000000079e7da4
> > > kernel_total_size: 0x0000000008a2c000
> > > needed_size: 0x0000000008c00000
> > > trampoline_32bit: 0x000000000009d000
> > > Physical KASLR using RDTSC...
> > > Virtual KASLR using RDTSC...
> > >
> > > Decompressing Linux... Parsing ELF... Performing relocations... done.
> > > Booting the kernel.
Cheers,
Nathan
Hi Boris,
On 12/17/2021 8:52 PM, Borislav Petkov wrote:
> Add Bruce and clang folks to the party.
>
> On Thu, Dec 16, 2021 at 08:21:15PM +0800, Yin Fengwei wrote:
>> Hi Boris,
>>
>> On 12/16/2021 7:58 PM, Carel Si wrote:
>>> Hi Boris,
>>>
>>> On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
>>>> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
>>>>> The testing was with Qemu.
>>>>
>>>> This is hardly what I asked for.
>>>>
>>>>> And we found that the hang is related with clang-14.
>>>>
>>>> I saw that already.
>>>>
>>>>> The original report showed the kernel is built with clang-14:
>>>>> # build kernel
>>>>> cd linux
>>>>> cp config-5.16.0-rc3-00003-gf154f290855b .config
>>>>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
>>>>> make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
>>>>
>>>> I saw that too.
>>>>
>>>>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
>>>>> This function is early function called before kasan_init.
>>>>>
>>>>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
>>>>> be happy with this kind of early TLB flush? Thanks.
>>>>
>>>> Ok, I don't understand: I asked for how exactly to reproduce and whether
>>>> you can send me your vmlinux you built with your clang-14. What I get is
>>>> some possible explanation about what might be happening.
>>>>
>>>> So what do you expect me to do? Say, "oh, sure, you're right, send me a
>>>> patch" without even being able to see for myself what the root cause is?
>>>>
>>>> What if it is not the kernel's fault but clang-14 is miscompiling crap
>>>> as in so many other cases?
>>>>
>>>> I built clang-14 and built with your .config and it works here fine. So
>>>> why does yours fail?
>>>>
>>>> Or what's the point of all this?
>>
>> I had concern that our report is an invalid report because you can't reproduce
>> it in your side. If that's the case, it could waste more your time. That's why
>> I did check and shared what I got. I am very sorry if I did it wrong.
>
> Sure, you can always add your analysis but I'd like to reproduce myself
> too. So, in the future, please answer the questions and then feel free
> to add your analysis - I'll gladly have a look.
Thanks a lot for sharing this. Lessons learnt here. Will follow this
rule exactly in the future.
>
> Which wasn't that far from the truth, btw.
>
> But it isn't KASAN but GCOV profiling. Or is it KCOV profiling which
> clang does.
>
> That thing adds some counting glue to native_write_cr4():
>
> (my comments from the actual singlestepping in qemu start with '##' below)
>
> movq $__llvm_gcov_ctr.48+8, %rbx ## mov $0xffffffff8837d3c0,%rbx
> .LBB8_1: # %set_register
> # =>This Inner Loop Header: Depth=1
>
> jmp .Ltmp42
> ...
>
> .Ltmp42: # Block address taken
> .LBB8_7: # %if.end79
> movq %rbx, %rax ## 0xffffffff8837d3c0
> shrq $3, %rax ## 0x1ffffffff106fa78
> movabsq $-2305847407260205056, %rcx # imm = 0xDFFFFC0000000000 ## 0xdffffc0000000000
> cmpb $0, (%rax,%rcx)
> je .LBB8_9
>
> so the memory address CMP accesses is something as nonsensical as
>
> 0xfffffbfff106fa78
>
> so I'm guessing we need to setup something for that __llvm_gcov_ctr to
> deref properly but I haven't dug deeper.
>
> The important thing is that this triggers with clang-13 and -14. gcc is
> fine with the same config but that probably is because gcc does other
> profiling - gcov - I guess. Looking at the resulting asm, it has a bunch
> of those counter increments:
>
> incq __gcov0.native_write_cr4+88(%rip) # __gcov0.native_write_cr4[11]
>
> but no weird memory references.
>
> So, clang folks, what's up?
>
> The fix is simple but I'd like to understand first why does this fail
> only with clang, 13 and newer.
>
> (I mean, melver pointed me to
>
> 380d53c45ff2 ("compiler_attributes.h: define __no_profile, add to noinstr")
>
> which explains why 13 and newer).
>
> Btw, joro, that second hunk is I think needed too because a couple of
> lines earlier we set up the cr4 shadow so I think you should use it
> instead of touching the hw CR4.
>
> ---
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 0083464de5e3..79b3d67addcc 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
> }
> EXPORT_SYMBOL(native_write_cr0);
>
> -void native_write_cr4(unsigned long val)
> +void __no_profile native_write_cr4(unsigned long val)
> {
> unsigned long bits_changed = 0;
>
> diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
> index 75acb6027a87..68d2b7f9a913 100644
> --- a/arch/x86/kernel/head64.c
> +++ b/arch/x86/kernel/head64.c
> @@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
> /* Kill off the identity-map trampoline */
> reset_early_page_tables();
>
> - __native_tlb_flush_global(native_read_cr4());
> + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
>
> clear_bss();
>
>
>
> Leaving in the rest for the newly added folks.
I tried this fix and it works fine in my local env. Will test it also in our
test box once we back to office. Thanks.
Regards
Yin, Fengwei
>
>> If you don't want to use lkp tool to reproduce the issue, following command
>> could be used as well:
>>
>> Use Qemu command so only kernel image need be downloaded:
>> qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200"
>> to reproduce it.
>>
>>
>>
>> Regards
>> Yin, Fengwei
>>
>>
>>
>>>>
>>>> I mean, if you cannot send me what I ask for, you can say so. Then I can
>>>> ignore this whole report altogether and waste my time somewhere else.
>>>
>>> We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
>>> https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
>>>
>>> Machine types can refer to:
>>> https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
>>>
>>> If there's any other msg needed, pls feel free to propose, thanks.
>>>
>>> Below are our full steps to reproduce the issue:
>>>
>>> # download lkp-tests
>>> $ git clone https://github.com/intel/lkp-tests.git
>>>
>>> $ cd lkp-tests/
>>>
>>> # download vmlinuz
>>> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
>>>
>>> # dowmload modules.cgz
>>> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
>>>
>>> # download job-script which is attached
>>>
>>> # run lkp qemu
>>> lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
>>>
>>> ~/lkp-tests/pkg/lkp-src ~/lkp-tests
>>> x86_64
>>> ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
>>> ==> Checking runtime dependencies...
>>> ==> Checking buildtime dependencies...
>>> ==> WARNING: Using existing $srcdir/ tree
>>> ==> Removing existing $pkgdir/ directory...
>>> ==> Starting build()...
>>> make: Entering directory '/home/carel/lkp-tests/bin/event'
>>> klcc -D_FORTIFY_SOURCE=2 -c -o wakeup.o wakeup.c
>>> klcc -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
>>> rm -f wakeup.o
>>> strip wakeup
>>> make: Leaving directory '/home/carel/lkp-tests/bin/event'
>>> ==> Entering fakeroot environment...
>>> x86_64
>>> ==> Starting package()...
>>> ==> Creating package "lkp-src"...
>>> 103987 blocks
>>> renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
>>> ==> Leaving fakeroot environment.
>>> ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
>>> ~/lkp-tests
>>> 12 blocks
>>> result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
>>> downloading initrds ...
>>> use local modules: /home/carel/.lkp/cache/modules.cgz
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
>>> 440459 blocks
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
>>> 1773 blocks
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
>>> 2321 blocks
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
>>> 6856 blocks
>>> exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
>>> early console in setup code
>>> early console in extract_kernel
>>> input_data: 0x0000000006ffc2e0
>>> input_len: 0x000000000260cb2b
>>> output: 0x0000000001000000
>>> output_len: 0x00000000079e7da4
>>> kernel_total_size: 0x0000000008a2c000
>>> needed_size: 0x0000000008c00000
>>> trampoline_32bit: 0x000000000009d000
>>> Physical KASLR using RDTSC...
>>> Virtual KASLR using RDTSC...
>>>
>>> Decompressing Linux... Parsing ELF... Performing relocations... done.
>>> Booting the kernel.
>>>
>>>>
>>>> --
>>>> Regards/Gruss,
>>>> Boris.
>>>>
>>>> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
>
On Fri, Dec 17, 2021 at 11:04:13AM -0700, Nathan Chancellor wrote:
> This is GCOV, -fprofile-arcs.
Aha, and no_profile_instrument_function disables that.
> I am not reallys ure how exactly GCOV works under the hood so I cannot
> really comment on it (Nick might); it seems like llvm_gcov_init needs to
> get called for __llvm_gcov_ctr to get set up properly and maybe that
> hasn't happened at the point.
Hmm, right, there is a
.p2align 4, 0x90 # -- Begin function __llvm_gcov_init
.type __llvm_gcov_init,@function
__llvm_gcov_init: # @__llvm_gcov_init
in arch/x86/kernel/cpu/common.s which contains native_write_cr4() which does
jmp llvm_gcov_init@PLT
and that function is part of initcalls:
.section .init_array.0,"aw",@init_array
.p2align 3
.quad __llvm_gcov_init
.section .init_array.1,"aw",@init_array
.p2align 3
.quad asan.module_ctor
.section .fini_array.1,"aw",@fini_array
.p2align 3
.quad asan.module_dtor
.type .L___asan_gen_.313,@object # @___asan_gen_.313
.section .rodata.str1.1,"aMS",@progbits,1
which, AFAICT, gets called by kernel_init_freeable->do_basic_setup->do_ctors()
which is a lot later than x86_64_start_kernel() so I guess the
__no_profile tag for that function probably makes sense as a fix.
> This is a bit of a brain dump, apologies for not offering much upfront
> analysis, I am not as familiar with LLVM internals as Nick but this
> might help others look into the problem.
No, this is still highly appreciated - thanks for taking the time!
> I ended up seeing this thread yesterday through a lore filter that I
> have
Nice filtering. :-)
I did Cc [email protected] in the hope that you guys would see it.
> and I bisected LLVM based on the fact that it happened with
> clang-13 but not clang-12; that bisect pointed out Nick's commit in LLVM
> that added the no profile attribute, which means that GCOV and KASAN
> need to be enabled to see this bug. I was not able to reproduce it with
> just one of them enabled at a time.
Ah, that's an interesting point:
$ grep -E "(GCOV|KASAN)" /tmp/config-5.16.0-rc3-00003-gf154f290855b | grep -v "^#"
CONFIG_KASAN_SHADOW_OFFSET=0xdffffc0000000000
CONFIG_GCOV_KERNEL=y
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
CONFIG_GCOV_PROFILE_ALL=y
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_KASAN_SW_TAGS=y
CONFIG_KASAN=y
CONFIG_KASAN_GENERIC=y
CONFIG_KASAN_INLINE=y
CONFIG_KASAN_VMALLOC=y
> With that, I removed the no profile attribute dependency on GCOV_KERNEL
> and bisected again, landing on the commit in LLVM 13 that enables the
> new pass manager, which fundamentally changes how LLVM transforms its
> IR. Whenever that has happened in the past, it usually points to a
> pre-existing issue; if I go back to clang-11 (the current minimum of
> -next) and enable the NPM there with -fexperimental-new-pass-manager, I
> see this hang so it seems like there might be some issue how GCOV and
> KASAN are manipulated together in the context of the NPM that was not
> present with the legacy pass manager.
Aha, so it used to work, apparently. Or the NPM is adding additional
code which needs to be initialized because it works ok after the
constructors have run.
> I do see tests in LLVM that test to make sure __llvm_gcov_ctr does not
> get instrumented with KASAN, maybe there is another interaction that
> should not be happening between those two?
Right.
Ok, thanks for the insights, let's see what Nick figures out here.
Thx.
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
On Sat, Dec 18, 2021 at 06:39:46PM +0800, Yin Fengwei wrote:
> Thanks a lot for sharing this. Lessons learnt here. Will follow this
> rule exactly in the future.
Thanks. And for the future, please trim your reply like I just did. :)
> I tried this fix and it works fine in my local env. Will test it also
> in our test box once we back to office. Thanks.
Good, much appreciated.
Thx.
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
On 12/18/21 7:01 PM, Borislav Petkov wrote:
> On Sat, Dec 18, 2021 at 06:39:46PM +0800, Yin Fengwei wrote:
>> Thanks a lot for sharing this. Lessons learnt here. Will follow this
>> rule exactly in the future.
>
> Thanks. And for the future, please trim your reply like I just did. :)
Sure. Thanks.
>
>> I tried this fix and it works fine in my local env. Will test it also
>> in our test box once we back to office. Thanks.
>
> Good, much appreciated.
It's our pleasure. Carel will send out the official test report on our
test box.
Regards
Yin, Fengwei
>
> Thx.
>
From: Borislav Petkov <[email protected]>
Commit in Fixes added a global TLB flush on the early boot path, after
the kernel switches off of the trampoline page table.
Compiler profiling options add additional measurement code
which needs to be initialized prior to use. The global flush in
x86_64_start_kernel() happens before those initializations can happen,
leading to accessing invalid memory.
The second issue this fixes is with KASAN: for a similar reason,
kasan_early_init() needs to have happened before KASAN-instrumented
functions are called.
Therefore, reorder the flush to happen after the KASAN early init
and prevent the compilers from adding profiling instrumentation to
native_write_cr4().
Fixes: f154f290855b ("x86/mm/64: Flush global TLB on boot and AP bringup")
Reported-by: "J. Bruce Fields" <[email protected]>
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Borislav Petkov <[email protected]>
Link: https://lore.kernel.org/r/20211209144141.GC25654@xsang-OptiPlex-9020
---
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/head64.c | 16 ++++++++++++++--
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464de5e3..79b3d67addcc 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
}
EXPORT_SYMBOL(native_write_cr0);
-void native_write_cr4(unsigned long val)
+void __no_profile native_write_cr4(unsigned long val)
{
unsigned long bits_changed = 0;
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 75acb6027a87..f5e80a8377ad 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -483,10 +483,12 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
/* Kill off the identity-map trampoline */
reset_early_page_tables();
- __native_tlb_flush_global(native_read_cr4());
-
clear_bss();
+ /*
+ * This needs to happen *before* kasan_early_init() because latter maps stuff
+ * into that page.
+ */
clear_page(init_top_pgt);
/*
@@ -498,6 +500,16 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
kasan_early_init();
+ /*
+ * Flush global TLB entries which could be left over from the trampoline page
+ * table.
+ *
+ * This needs to happen *after* kasan_early_init() as KASAN-enabled .configs
+ * instrument native_write_cr4() so KASAN must be initialized for that
+ * instrumentation to work.
+ */
+ __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
+
idt_setup_early_handler();
copy_bootdata(__va(real_mode_data));
--
2.29.2
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
Hi Boris,
On Fri, Dec 17, 2021 at 08:52:53PM +0800, Borislav Petkov wrote:
> ---
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 0083464de5e3..79b3d67addcc 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
> }
> EXPORT_SYMBOL(native_write_cr0);
>
> -void native_write_cr4(unsigned long val)
> +void __no_profile native_write_cr4(unsigned long val)
> {
> unsigned long bits_changed = 0;
>
> diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
> index 75acb6027a87..68d2b7f9a913 100644
> --- a/arch/x86/kernel/head64.c
> +++ b/arch/x86/kernel/head64.c
> @@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
> /* Kill off the identity-map trampoline */
> reset_early_page_tables();
>
> - __native_tlb_flush_global(native_read_cr4());
> + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
>
> clear_bss();
>
We tested your patch, it can fix the issue, thanks.
=========================================================================================
compiler/kconfig/rootfs/sleep/tbox_group/testcase:
clang-14/x86_64-randconfig-a013-20211207/debian-10.4-x86_64-20200603.cgz/1/vm-snb/boot
commit:
9de4999050 ("x86/realmode: Add comment for Global bit usage in trampoline_pgd")
f154f29085 ("x86/mm/64: Flush global TLB on boot and AP bringup")
d12ddfe498 ("fixup-for-f154f29085")
9de4999050b5f2e8 f154f290855b070cc94dd44ad25 d12ddfe498329a0967c008d8183
---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
:27 100% 27:27 0% :27 dmesg.BUG:kernel_reboot-without-warning_in_boot_stage
On Tue, 21 Dec 2021 at 15:34, Carel Si <[email protected]> wrote:
>
> Hi Boris,
>
> On Fri, Dec 17, 2021 at 08:52:53PM +0800, Borislav Petkov wrote:
> > ---
> > diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> > index 0083464de5e3..79b3d67addcc 100644
> > --- a/arch/x86/kernel/cpu/common.c
> > +++ b/arch/x86/kernel/cpu/common.c
> > @@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
> > }
> > EXPORT_SYMBOL(native_write_cr0);
> >
> > -void native_write_cr4(unsigned long val)
> > +void __no_profile native_write_cr4(unsigned long val)
> > {
> > unsigned long bits_changed = 0;
> >
> > diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
> > index 75acb6027a87..68d2b7f9a913 100644
> > --- a/arch/x86/kernel/head64.c
> > +++ b/arch/x86/kernel/head64.c
> > @@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
> > /* Kill off the identity-map trampoline */
> > reset_early_page_tables();
> >
> > - __native_tlb_flush_global(native_read_cr4());
> > + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
> >
> > clear_bss();
Also double-check the Makefile-based solution: the Makefiles in both
directories already contain various KCOV_INSTRUMENT/K*SAN_SANITIZE :=
n, so perhaps adding GCOV_PROFILE or GCOV_PROFILE_whicheverfile.o := n
may be more appropriate should the functions that should not be
instrumented be a moving target, and prone to breakage again in
future.
On Tue, Dec 21, 2021 at 10:31:54PM +0800, Carel Si wrote:
> We tested your patch, it can fix the issue, thanks.
You mean "it does fix the issue". :-)
Thanks for testing, I'll add your Tested-by.
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
On Tue, Dec 21, 2021 at 04:10:31PM +0100, Marco Elver wrote:
> Also double-check the Makefile-based solution: the Makefiles in both
> directories already contain various KCOV_INSTRUMENT/K*SAN_SANITIZE :=
> n, so perhaps adding GCOV_PROFILE or GCOV_PROFILE_whicheverfile.o := n
> may be more appropriate should the functions that should not be
> instrumented be a moving target, and prone to breakage again in
> future.
Yeah, I asked whether we should exclude the whole ...cpu/common.c from
profiling - it has mostly init code so it probably doesn't matter
for coverage but no one said anything so I left it to the function
annotation.
And also, I'm still waiting on clang folks to chime in on the New Pass
Manager and whether there's a bug in clang there so that we won't need
the __no_profile annotation at all. I mean, gcc is fine with that config
so unless clang is doing more profiling gunk than gcc and requires that
__llvm_gcov_init constructor... see Nathan's mail upthread.
Thx.
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
The following commit has been merged into the x86/mm branch of tip:
Commit-ID: b64dfcde1ca9cb82e38e573753f0c0db8fb841c2
Gitweb: https://git.kernel.org/tip/b64dfcde1ca9cb82e38e573753f0c0db8fb841c2
Author: Borislav Petkov <[email protected]>
AuthorDate: Fri, 17 Dec 2021 16:48:29 +01:00
Committer: Borislav Petkov <[email protected]>
CommitterDate: Wed, 22 Dec 2021 11:51:20 +01:00
x86/mm: Prevent early boot triple-faults with instrumentation
Commit in Fixes added a global TLB flush on the early boot path, after
the kernel switches off of the trampoline page table.
Compiler profiling options enabled with GCOV_PROFILE add additional
measurement code on clang which needs to be initialized prior to
use. The global flush in x86_64_start_kernel() happens before those
initializations can happen, leading to accessing invalid memory.
GCOV_PROFILE builds with gcc are still ok so this is clang-specific.
The second issue this fixes is with KASAN: for a similar reason,
kasan_early_init() needs to have happened before KASAN-instrumented
functions are called.
Therefore, reorder the flush to happen after the KASAN early init
and prevent the compilers from adding profiling instrumentation to
native_write_cr4().
Fixes: f154f290855b ("x86/mm/64: Flush global TLB on boot and AP bringup")
Reported-by: "J. Bruce Fields" <[email protected]>
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Borislav Petkov <[email protected]>
Tested-by: Carel Si <[email protected]>
Tested-by: "J. Bruce Fields" <[email protected]>
Link: https://lore.kernel.org/r/20211209144141.GC25654@xsang-OptiPlex-9020
---
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/head64.c | 16 ++++++++++++++--
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464..79b3d67 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -384,7 +384,7 @@ set_register:
}
EXPORT_SYMBOL(native_write_cr0);
-void native_write_cr4(unsigned long val)
+void __no_profile native_write_cr4(unsigned long val)
{
unsigned long bits_changed = 0;
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 75acb60..f5e80a8 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -483,10 +483,12 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
/* Kill off the identity-map trampoline */
reset_early_page_tables();
- __native_tlb_flush_global(native_read_cr4());
-
clear_bss();
+ /*
+ * This needs to happen *before* kasan_early_init() because latter maps stuff
+ * into that page.
+ */
clear_page(init_top_pgt);
/*
@@ -498,6 +500,16 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
kasan_early_init();
+ /*
+ * Flush global TLB entries which could be left over from the trampoline page
+ * table.
+ *
+ * This needs to happen *after* kasan_early_init() as KASAN-enabled .configs
+ * instrument native_write_cr4() so KASAN must be initialized for that
+ * instrumentation to work.
+ */
+ __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
+
idt_setup_early_handler();
copy_bootdata(__va(real_mode_data));
Hi Boris,
On 12/21/21 11:22 PM, Borislav Petkov wrote:
> And also, I'm still waiting on clang folks to chime in on the New Pass
> Manager and whether there's a bug in clang there so that we won't need
> the __no_profile annotation at all. I mean, gcc is fine with that config
> so unless clang is doing more profiling gunk than gcc and requires that
> __llvm_gcov_init constructor... see Nathan's mail upthread.
Did you get update from clang folks for this behavior? Thanks.
Regards
Yin, Fengwei
On Wed, Jan 05, 2022 at 10:35:20AM +0800, Yin Fengwei wrote:
> Did you get update from clang folks for this behavior? Thanks.
No. Why?
--
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
On 1/5/2022 7:36 PM, Borislav Petkov wrote:
> On Wed, Jan 05, 2022 at 10:35:20AM +0800, Yin Fengwei wrote:
>> Did you get update from clang folks for this behavior? Thanks.
>
> No. Why?
My understanding:
it's better that clang restore its behavior to clang-12. Or people
need to use __no_profile annotation if want to add function before
GCOV/KASAN is initialized. Thanks.
Regards
Yin, Fengwei
>
On Wed, Jan 05, 2022 at 08:47:28PM +0800, Yin Fengwei wrote:
> it's better that clang restore its behavior to clang-12.
Probably, but I'm not gonna hold my breath - folks are likely very
busy...
> Or people need to use __no_profile annotation if want to add function
> before GCOV/KASAN is initialized.
... yap, and because fixing it this way is easy, I've already queued the
__no_profile solution:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/mm&id=b64dfcde1ca9cb82e38e573753f0c0db8fb841c2
Thanks.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
Hi Boris,
On 1/5/22 11:21 PM, Borislav Petkov wrote:
> ... yap, and because fixing it this way is easy, I've already queued the
> __no_profile solution:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/mm&id=b64dfcde1ca9cb82e38e573753f0c0db8fb841c2
This fixing is good to me.
Just a little concern: if people need to add function in the future,
if he/she is not using clang-14, it's possible that he/she misses the
__no_profile and triggers this kind of issue again.
It may not be a big problem (we will capture it in our test. :)). Thanks.
Regards
Yin, Fengwei
>
> Thanks.