Hi all,
We met BUG_ON in do_device_not_available(fpu exception handler) when run redhat7 in kvm guest,
and there is no special test on this guest, only some network packet receipt and transmission.
I checked the new kernel version, found this commit 4ecd16ec7059390b430af34bd8bc3ca2b5dcef9a
Author: Andy Lutomirski <[email protected]>
Date: Sun Jan 24 14:38:06 2016 -0800
x86/fpu: Fix math emulation in eager fpu mode
Systems without an FPU are generally old and therefore use lazy FPU
switching. Unsurprisingly, math emulation in eager FPU mode is a
bit buggy. Fix it.
There were two bugs involving kernel code trying to use the FPU
registers in eager mode even if they didn't exist and one BUG_ON()
that was incorrect.
The BUG_ON() is incorrect, but I have no idea about eager fpu, why the BUG_ON is incorrect?
Should we backport the patch to v3.10, or is there some bugs in the qemu-kvm?
Any reply will be appreciated.
Thanks,
Kefeng
[1] BUG_ON
----------------------------------------------------
[347134.486436] ------------[ cut here ]------------
[347134.487310] kernel BUG at arch/x86/kernel/traps.c:643!
[347134.487398] invalid opcode: 0000 [#1] SMP
[347134.500532] Modules linked in:loop binfmt_misc nf_log_ipv4 nf_log_common xt_LOG softdog ipmi_devintf ipmi_msghandler xfs libcrc32c tipc squashfs ipt_REJECT iptable_filter ip_tables dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw ppdev gf128mul i2c_piix4 glue_helper ablk_helper i2c_core cryptd serio_raw parport_pc parport pcspkr ext3 mbcache jbd ata_generic pata_acpi virtio_console(OVE) virtio_blk(OVE) virtio_balloon(OVE) virtio_net(OVE) ata_piix libata floppy virtio_pci(OVE) virtio_ring(OVE) virtio(OVE)
[347134.525182] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O E ----V------- 3.10.0-229.20.1.x86_64 #1 SMP Mon Apr 18 11:26:55 UTC 2016
[347134.525182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20160318_175052-HGH1000008214 04/01/2014
[347134.525182] task: ffff8803fa2c5c00 ti: ffff8803fa2ec000 task.ti: ffff8803fa2ec000
[347134.525182] RIP: 0010:[<ffffffff8160fb13>] [<ffffffff8160fb13>] do_device_not_available+0x13/0x60
[347134.525182] RSP: 0018:ffff8803fa2abc80 EFLAGS: 00010046
[347134.525182] RAX: 000000008160ecec RBX: 0000000000000000 RCX: ffffffff8160ecec
[347134.525182] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff8803fa2abc98
[347134.525182] RBP: ffff8803fa2abc88 R08: 0000000000000000 R09: 0000000000000000
[347134.525182] R10: 0000000000000001 R11: 0000000000000005 R12: ffff8803fa2c5c00
[347134.525182] R13: ffff88040f550a40 R14: ffff8803fa2c6298 R15: 0000000000000005
[347134.544262] FS: 0000000000000000(0000) GS:ffff88040f540000(0000) knlGS:0000000000000000
[347134.544262] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[347134.544262] CR2: 00007f15a4ac0e30 CR3: 00000003f5081000 CR4: 00000000000407e0
[347134.544262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[347134.544262] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[347134.544262] Stack:
[347134.544262] 0000000000000001 ffff8803fa2abd88 ffffffff81618d8e 0000000000000005
[347134.544262] ffff8803fa2c6298 ffff88040f550a40 ffff8803fa2c5c00 ffff8803fa2abd88
[347134.544262] ffff8803fa284500 0000000000000005 0000000000000001 0000000000000000
[347134.544262] Call Trace:
[347134.544262] Code: 81 a4 24 90 00 00 00 ff fe ff ff e9 df fe ff ff e8 c3 f5 a5 ff 0f 1f 00 66 66 66 66 90 55 48 89 e5 53 66 66 66 66 90 31 db 66 90 <0f> 0b 0f 1f 00 65 8b 1c 25 74 f1 00 00 e8 fb 86 b4 ff eb ea 66
[347134.544262] RIP [<ffffffff8160fb13>] do_device_not_available+0x13/0x60
[347134.544262] RSP <ffff8803fa2abc80>
[347134.580457] ---[ end trace 7c0ed2be7ded5c73 ]---
[347134.580457] Kernel panic - not syncing: Fatal exception
[347134.580457] Shutting down cpus with NMI
[2] The /proc/cpuinfo shows below(show only the first cpu0),
--------------------------------
localhost:~ # cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
stepping : 7
microcode : 0x1
cpu MHz : 2899.992
cache size : 4096 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm xsaveopt
bogomips : 5799.98
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
On Fri, 2016-10-14 at 14:14 +0800, Kefeng Wang wrote:
> Hi all,
>
> We met BUG_ON in do_device_not_available(fpu exception handler) when
> run redhat7 in kvm guest,
> and there is no special test on this guest, only some network packet
> receipt and transmission.
>
> I checked the new kernel version, found this commit
> 4ecd16ec7059390b430af34bd8bc3ca2b5dcef9a
> Author: Andy Lutomirski <[email protected]>
> Date: Sun Jan 24 14:38:06 2016 -0800
>
> x86/fpu: Fix math emulation in eager fpu mode
>
> Systems without an FPU are generally old and therefore use lazy
> FPU
> switching. Unsurprisingly, math emulation in eager FPU mode is a
> bit buggy. Fix it.
This patch is for "systems without an FPU"
Before we go on, I would like to know what kind of CPU
you tell your KVM virtual machine to present to the guest,
and what kind of CPU your host system has.
Are you by any chance configuring the CPU inside your
virtual machine without an FPU? It is possible to mask
out bits presented in the CPUID result inside a KVM
guest, so I suspect this is possible...
> There were two bugs involving kernel code trying to use the FPU
> registers in eager mode even if they didn't exist and one
> BUG_ON()
> that was incorrect.
>
>
> The BUG_ON() is incorrect, but I have no idea about eager fpu, why
> the BUG_ON is incorrect?
> Should we backport the patch to v3.10, or is there some bugs in the
> qemu-kvm?
> Any reply will be appreciated.
>
> Thanks,
> Kefeng
>
> [1] BUG_ON
> ----------------------------------------------------
> [347134.486436] ------------[ cut here ]------------
> [347134.487310] kernel BUG at arch/x86/kernel/traps.c:643!
> [347134.487398] invalid opcode: 0000 [#1] SMP
> [347134.500532] Modules linked in:loop binfmt_misc nf_log_ipv4
> nf_log_common xt_LOG softdog ipmi_devintf ipmi_msghandler xfs
> libcrc32c tipc squashfs ipt_REJECT iptable_filter ip_tables dm_mod
> crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel
> ghash_clmulni_intel aesni_intel lrw ppdev gf128mul i2c_piix4
> glue_helper ablk_helper i2c_core cryptd serio_raw parport_pc parport
> pcspkr ext3 mbcache jbd ata_generic pata_acpi virtio_console(OVE)
> virtio_blk(OVE) virtio_balloon(OVE) virtio_net(OVE) ata_piix libata
> floppy virtio_pci(OVE) virtio_ring(OVE) virtio(OVE)
> [347134.525182] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O
> E ----V------- 3.10.0-229.20.1.x86_64 #1 SMP Mon Apr 18 11:26:55
> UTC 2016
> [347134.525182] Hardware name: QEMU Standard PC (i440FX + PIIX,
> 1996), BIOS rel-1.8.1-0-g4adadbd-20160318_175052-HGH1000008214
> 04/01/2014
> [347134.525182] task: ffff8803fa2c5c00 ti: ffff8803fa2ec000 task.ti:
> ffff8803fa2ec000
> [347134.525182] RIP: 0010:[<ffffffff8160fb13>] [<ffffffff8160fb13>]
> do_device_not_available+0x13/0x60
> [347134.525182] RSP: 0018:ffff8803fa2abc80 EFLAGS: 00010046
> [347134.525182] RAX: 000000008160ecec RBX: 0000000000000000 RCX:
> ffffffff8160ecec
> [347134.525182] RDX: 00000000ffffffff RSI: 0000000000000000 RDI:
> ffff8803fa2abc98
> [347134.525182] RBP: ffff8803fa2abc88 R08: 0000000000000000 R09:
> 0000000000000000
> [347134.525182] R10: 0000000000000001 R11: 0000000000000005 R12:
> ffff8803fa2c5c00
> [347134.525182] R13: ffff88040f550a40 R14: ffff8803fa2c6298 R15:
> 0000000000000005
> [347134.544262] FS: 0000000000000000(0000) GS:ffff88040f540000(0000)
> knlGS:0000000000000000
> [347134.544262] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [347134.544262] CR2: 00007f15a4ac0e30 CR3: 00000003f5081000 CR4:
> 00000000000407e0
> [347134.544262] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [347134.544262] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [347134.544262] Stack:
> [347134.544262] 0000000000000001 ffff8803fa2abd88 ffffffff81618d8e
> 0000000000000005
> [347134.544262] ffff8803fa2c6298 ffff88040f550a40 ffff8803fa2c5c00
> ffff8803fa2abd88
> [347134.544262] ffff8803fa284500 0000000000000005 0000000000000001
> 0000000000000000
> [347134.544262] Call Trace:
> [347134.544262] Code: 81 a4 24 90 00 00 00 ff fe ff ff e9 df fe ff ff
> e8 c3 f5 a5 ff 0f 1f 00 66 66 66 66 90 55 48 89 e5 53 66 66 66 66 90
> 31 db 66 90 <0f> 0b 0f 1f 00 65 8b 1c 25 74 f1 00 00 e8 fb 86 b4 ff
> eb ea 66
> [347134.544262] RIP [<ffffffff8160fb13>]
> do_device_not_available+0x13/0x60
> [347134.544262] RSP <ffff8803fa2abc80>
> [347134.580457] ---[ end trace 7c0ed2be7ded5c73 ]---
> [347134.580457] Kernel panic - not syncing: Fatal exception
> [347134.580457] Shutting down cpus with NMI
>
>
>
> [2] The /proc/cpuinfo shows below(show only the first cpu0),
> --------------------------------
> localhost:~ # cat /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 45
> model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
> stepping : 7
> microcode : 0x1
> cpu MHz : 2899.992
> cache size : 4096 KB
> physical id : 0
> siblings : 8
> core id : 0
> cpu cores : 8
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep
> mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall
> nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu
> pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt
> tsc_deadline_timer aes xsave avx hypervisor lahf_lm xsaveopt
> bogomips : 5799.98
> clflush size : 64
> cache_alignment : 64
> address sizes : 42 bits physical, 48 bits virtual
> power management:
>
--
All Rights Reversed.
On Thu, Oct 13, 2016 at 11:14 PM, Kefeng Wang
<[email protected]> wrote:
> Hi all,
>
> We met BUG_ON in do_device_not_available(fpu exception handler) when run redhat7 in kvm guest,
> and there is no special test on this guest, only some network packet receipt and transmission.
>
> I checked the new kernel version, found this commit 4ecd16ec7059390b430af34bd8bc3ca2b5dcef9a
> Author: Andy Lutomirski <[email protected]>
> Date: Sun Jan 24 14:38:06 2016 -0800
>
> x86/fpu: Fix math emulation in eager fpu mode
>
> Systems without an FPU are generally old and therefore use lazy FPU
> switching. Unsurprisingly, math emulation in eager FPU mode is a
> bit buggy. Fix it.
>
> There were two bugs involving kernel code trying to use the FPU
> registers in eager mode even if they didn't exist and one BUG_ON()
> that was incorrect.
>
>
> The BUG_ON() is incorrect, but I have no idea about eager fpu, why the BUG_ON is incorrect?
> Should we backport the patch to v3.10, or is there some bugs in the qemu-kvm?
> Any reply will be appreciated.
The BUG_ON was incorrect because you could hit it if FPU emulation was
enabled. But, unless you explicitly set the "eagerfpu=" option or you
have some really weird set of cpu flags, old kernels shouldn't have
hit it. Is the cpuinfo you pasted below from the guest? Also, could
you attach whatever dmesg has to say about FPU in a crashing guest?
--Andy
>
> Thanks,
> Kefeng
>
> [1] BUG_ON
> ----------------------------------------------------
> [347134.486436] ------------[ cut here ]------------
> [347134.487310] kernel BUG at arch/x86/kernel/traps.c:643!
> [347134.487398] invalid opcode: 0000 [#1] SMP
> [347134.500532] Modules linked in:loop binfmt_misc nf_log_ipv4 nf_log_common xt_LOG softdog ipmi_devintf ipmi_msghandler xfs libcrc32c tipc squashfs ipt_REJECT iptable_filter ip_tables dm_mod crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw ppdev gf128mul i2c_piix4 glue_helper ablk_helper i2c_core cryptd serio_raw parport_pc parport pcspkr ext3 mbcache jbd ata_generic pata_acpi virtio_console(OVE) virtio_blk(OVE) virtio_balloon(OVE) virtio_net(OVE) ata_piix libata floppy virtio_pci(OVE) virtio_ring(OVE) virtio(OVE)
> [347134.525182] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G O E ----V------- 3.10.0-229.20.1.x86_64 #1 SMP Mon Apr 18 11:26:55 UTC 2016
> [347134.525182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.1-0-g4adadbd-20160318_175052-HGH1000008214 04/01/2014
> [347134.525182] task: ffff8803fa2c5c00 ti: ffff8803fa2ec000 task.ti: ffff8803fa2ec000
> [347134.525182] RIP: 0010:[<ffffffff8160fb13>] [<ffffffff8160fb13>] do_device_not_available+0x13/0x60
> [347134.525182] RSP: 0018:ffff8803fa2abc80 EFLAGS: 00010046
> [347134.525182] RAX: 000000008160ecec RBX: 0000000000000000 RCX: ffffffff8160ecec
> [347134.525182] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff8803fa2abc98
> [347134.525182] RBP: ffff8803fa2abc88 R08: 0000000000000000 R09: 0000000000000000
> [347134.525182] R10: 0000000000000001 R11: 0000000000000005 R12: ffff8803fa2c5c00
> [347134.525182] R13: ffff88040f550a40 R14: ffff8803fa2c6298 R15: 0000000000000005
> [347134.544262] FS: 0000000000000000(0000) GS:ffff88040f540000(0000) knlGS:0000000000000000
> [347134.544262] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [347134.544262] CR2: 00007f15a4ac0e30 CR3: 00000003f5081000 CR4: 00000000000407e0
> [347134.544262] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [347134.544262] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [347134.544262] Stack:
> [347134.544262] 0000000000000001 ffff8803fa2abd88 ffffffff81618d8e 0000000000000005
> [347134.544262] ffff8803fa2c6298 ffff88040f550a40 ffff8803fa2c5c00 ffff8803fa2abd88
> [347134.544262] ffff8803fa284500 0000000000000005 0000000000000001 0000000000000000
> [347134.544262] Call Trace:
> [347134.544262] Code: 81 a4 24 90 00 00 00 ff fe ff ff e9 df fe ff ff e8 c3 f5 a5 ff 0f 1f 00 66 66 66 66 90 55 48 89 e5 53 66 66 66 66 90 31 db 66 90 <0f> 0b 0f 1f 00 65 8b 1c 25 74 f1 00 00 e8 fb 86 b4 ff eb ea 66
> [347134.544262] RIP [<ffffffff8160fb13>] do_device_not_available+0x13/0x60
> [347134.544262] RSP <ffff8803fa2abc80>
> [347134.580457] ---[ end trace 7c0ed2be7ded5c73 ]---
> [347134.580457] Kernel panic - not syncing: Fatal exception
> [347134.580457] Shutting down cpus with NMI
>
>
>
> [2] The /proc/cpuinfo shows below(show only the first cpu0),
> --------------------------------
> localhost:~ # cat /proc/cpuinfo
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 45
> model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
> stepping : 7
> microcode : 0x1
> cpu MHz : 2899.992
> cache size : 4096 KB
> physical id : 0
> siblings : 8
> core id : 0
> cpu cores : 8
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm xsaveopt
> bogomips : 5799.98
> clflush size : 64
> cache_alignment : 64
> address sizes : 42 bits physical, 48 bits virtual
> power management:
>
--
Andy Lutomirski
AMA Capital Management, LLC
On 2016/10/15 2:36, Andy Lutomirski wrote:
> On Thu, Oct 13, 2016 at 11:14 PM, Kefeng Wang
> <[email protected]> wrote:
>> Hi all,
>>
>> We met BUG_ON in do_device_not_available(fpu exception handler) when run redhat7 in kvm guest,
>> and there is no special test on this guest, only some network packet receipt and transmission.
>>
>> I checked the new kernel version, found this commit 4ecd16ec7059390b430af34bd8bc3ca2b5dcef9a
>> Author: Andy Lutomirski <[email protected]>
>> Date: Sun Jan 24 14:38:06 2016 -0800
>>
>> x86/fpu: Fix math emulation in eager fpu mode
>>
>> Systems without an FPU are generally old and therefore use lazy FPU
>> switching. Unsurprisingly, math emulation in eager FPU mode is a
>> bit buggy. Fix it.
>>
>> There were two bugs involving kernel code trying to use the FPU
>> registers in eager mode even if they didn't exist and one BUG_ON()
>> that was incorrect.
>>
>>
>> The BUG_ON() is incorrect, but I have no idea about eager fpu, why the BUG_ON is incorrect?
>> Should we backport the patch to v3.10, or is there some bugs in the qemu-kvm?
>> Any reply will be appreciated.
>
> The BUG_ON was incorrect because you could hit it if FPU emulation was
> enabled. But, unless you explicitly set the "eagerfpu=" option or you
> have some really weird set of cpu flags, old kernels shouldn't have
> hit it. Is the cpuinfo you pasted below from the guest? Also, could
> you attach whatever dmesg has to say about FPU in a crashing guest?
>
> --Andy
>
Hi Andy and Rik, thanks for your quick response.
Attach more information, and we have no special configuration for fpu, and only met this issue once(can't reproduce).
1) cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-229.20.1.x86_64 root=/dev/vda2 oops=panic softlockup_panic=1 net.ifnames=0 biosdevname=0 nmi_watchdog=1 selinux=0 console=tty0 panic=3
2) virsh dumpxml 3 (only cpu parts)
-----------------------
<memory unit='KiB'>16000000</memory>
<currentMemory unit='KiB'>16000000</currentMemory>
<vcpu placement='static'>8</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
<boot dev='cdrom'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='host-passthrough'>
<topology sockets='1' cores='8' threads='1'/>
</cpu>
-----------------------
3) The host os cpuinfo
-----------------------
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
stepping : 7
microcode : 1808
cpu MHz : 2899.894
cache size : 20480 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm arat epb xsaveopt pl
n pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5799.78
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
4) The guest os cpuinfo
>> [2] The /proc/cpuinfo shows below(show only the first cpu0),
>> --------------------------------
>> localhost:~ # cat /proc/cpuinfo
>> processor : 0
>> vendor_id : GenuineIntel
>> cpu family : 6
>> model : 45
>> model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
>> stepping : 7
>> microcode : 0x1
>> cpu MHz : 2899.992
>> cache size : 4096 KB
>> physical id : 0
>> siblings : 8
>> core id : 0
>> cpu cores : 8
>> apicid : 0
>> initial apicid : 0
>> fpu : yes
>> fpu_exception : yes
>> cpuid level : 13
>> wp : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm xsaveopt
>> bogomips : 5799.98
>> clflush size : 64
>> cache_alignment : 64
>> address sizes : 42 bits physical, 48 bits virtual
>> power management:
>>
>
5) parts of bootmsg
[ 0.000000] Booting paravirtualized kernel on KVM
[ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88040f400000 s82816 r8192 d23680 u262144
[ 0.000000] KVM setup async PF for cpu 0
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 3933317
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-229.20.1.x86_64 root=/dev/vda2 oops=panic softlockup_panic=1 net.ifnames=0 biosdevname=0 nmi_watchdog=1 selinux=0 console=tty0 panic=3
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[ 0.000000] Checking aperture...
[ 0.000000] No AGP bridge found
[ 0.000000] Memory: 15375712k/17032192k available (6255k kernel code, 1049096k absent, 607384k reserved, 4184k data, 1604k init)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=8.
[ 0.000000] Experimental no-CBs for all CPUs
[ 0.000000] Experimental no-CBs CPUs: 0-7.
[ 0.000000] NR_IRQS:327936 nr_irqs:744 16
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] allocated 63963136 bytes of page_cgroup
[ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[ 0.000000] tsc: Detected 2899.992 MHz processor
[ 0.002000] Calibrating delay loop (skipped) preset value.. 5799.98 BogoMIPS (lpj=2899992)
[ 0.002006] pid_max: default: 32768 minimum: 301
[ 0.003029] Security Framework initialized
[ 0.003584] SELinux: Disabled at boot.
[ 0.005623] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[ 0.009697] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[ 0.011961] Mount-cache hash table entries: 4096
[ 0.012178] Initializing cgroup subsys memory
[ 0.013011] Initializing cgroup subsys devices
[ 0.013573] Initializing cgroup subsys freezer
[ 0.014004] Initializing cgroup subsys net_cls
[ 0.015003] Initializing cgroup subsys blkio
[ 0.015558] Initializing cgroup subsys perf_event
[ 0.016007] Initializing cgroup subsys hugetlb
[ 0.016637] CPU: Physical Processor ID: 0
[ 0.017003] CPU: Processor Core ID: 0
[ 0.018004] mce: CPU supports 10 MCE banks
[ 0.018597] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0
tlb_flushall_shift: 6
[ 0.019169] Freeing SMP alternatives: 24k freed
[ 0.023150] ACPI: Core revision 20130517
[ 0.024384] ACPI: All ACPI Tables successfully acquired
[ 0.025016] ftrace: allocating 23959 entries in 94 pages
[ 0.031284] Enabling x2apic
[ 0.031884] Enabled x2apic
[ 0.032004] Switched APIC routing to physical x2apic.
[ 0.033629] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.034002] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz (fam: 06, model: 2d, step
[ 0.035052] Performance Events: 16-deep LBR, SandyBridge events, Intel PMU driver.
[ 0.037935] perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode
[ 0.038005] ... version: 2
[ 0.038544] ... bit width: 48
[ 0.039003] ... generic registers: 4
[ 0.039546] ... value mask: 0000ffffffffffff
[ 0.040003] ... max period: 000000007fffffff
[ 0.040565] ... fixed-purpose events: 3
[ 0.041002] ... event mask: 000000070000000f
[ 0.042819] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[ 0.042791] kvm-clock: cpu 1, msr 4:f889041, secondary cpu clock
[ 0.055016] KVM setup async PF for cpu 1
[ 0.055025] kvm-clock: cpu 2, msr 4:f889081, secondary cpu clock
[ 0.067015] KVM setup async PF for cpu 2
[ 0.067022] kvm-clock: cpu 3, msr 4:f8890c1, secondary cpu clock
[ 0.079015] KVM setup async PF for cpu 3
[ 0.079023] kvm-clock: cpu 4, msr 4:f889101, secondary cpu clock
[ 0.091013] KVM setup async PF for cpu 4
[ 0.091019] kvm-clock: cpu 5, msr 4:f889141, secondary cpu clock
[ 0.103014] KVM setup async PF for cpu 5
[ 0.103021] kvm-clock: cpu 6, msr 4:f889181, secondary cpu clock
[ 0.115014] KVM setup async PF for cpu 6
[ 0.043097] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK
[ 0.115021] kvm-clock: cpu 7, msr 4:f8891c1, secondary cpu clock
[ 0.127022] Brought up 8 CPUs
[ 0.127015] KVM setup async PF for cpu 7
[ 0.128005] smpboot: Total of 8 processors activated (46399.87 BogoMIPS)
[ 0.129202] devtmpfs: initialized
Hi all, any ideas, thanks.
+ gonglei, haibin
On 2016/10/17 15:51, Kefeng Wang wrote:
>
>
> On 2016/10/15 2:36, Andy Lutomirski wrote:
>> On Thu, Oct 13, 2016 at 11:14 PM, Kefeng Wang
>> <[email protected]> wrote:
>>> Hi all,
>>>
>>> We met BUG_ON in do_device_not_available(fpu exception handler) when run redhat7 in kvm guest,
>>> and there is no special test on this guest, only some network packet receipt and transmission.
>>>
>>> I checked the new kernel version, found this commit 4ecd16ec7059390b430af34bd8bc3ca2b5dcef9a
>>> Author: Andy Lutomirski <[email protected]>
>>> Date: Sun Jan 24 14:38:06 2016 -0800
>>>
>>> x86/fpu: Fix math emulation in eager fpu mode
>>>
>>> Systems without an FPU are generally old and therefore use lazy FPU
>>> switching. Unsurprisingly, math emulation in eager FPU mode is a
>>> bit buggy. Fix it.
>>>
>>> There were two bugs involving kernel code trying to use the FPU
>>> registers in eager mode even if they didn't exist and one BUG_ON()
>>> that was incorrect.
>>>
>>>
>>> The BUG_ON() is incorrect, but I have no idea about eager fpu, why the BUG_ON is incorrect?
>>> Should we backport the patch to v3.10, or is there some bugs in the qemu-kvm?
>>> Any reply will be appreciated.
>>
>> The BUG_ON was incorrect because you could hit it if FPU emulation was
>> enabled. But, unless you explicitly set the "eagerfpu=" option or you
>> have some really weird set of cpu flags, old kernels shouldn't have
>> hit it. Is the cpuinfo you pasted below from the guest? Also, could
>> you attach whatever dmesg has to say about FPU in a crashing guest?
>>
>> --Andy
>>
>
> Hi Andy and Rik, thanks for your quick response.
>
> Attach more information, and we have no special configuration for fpu, and only met this issue once(can't reproduce).
>
>
> 1) cmdline
>
> BOOT_IMAGE=/vmlinuz-3.10.0-229.20.1.x86_64 root=/dev/vda2 oops=panic softlockup_panic=1 net.ifnames=0 biosdevname=0 nmi_watchdog=1 selinux=0 console=tty0 panic=3
>
>
> 2) virsh dumpxml 3 (only cpu parts)
> -----------------------
> <memory unit='KiB'>16000000</memory>
> <currentMemory unit='KiB'>16000000</currentMemory>
> <vcpu placement='static'>8</vcpu>
> <resource>
> <partition>/machine</partition>
> </resource>
> <os>
> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
> <boot dev='cdrom'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> </features>
> <cpu mode='host-passthrough'>
> <topology sockets='1' cores='8' threads='1'/>
> </cpu>
> -----------------------
>
> 3) The host os cpuinfo
> -----------------------
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 45
> model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
> stepping : 7
> microcode : 1808
> cpu MHz : 2899.894
> cache size : 20480 KB
> physical id : 0
> siblings : 16
> core id : 0
> cpu cores : 8
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
> nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm arat epb xsaveopt pl
> n pts dtherm tpr_shadow vnmi flexpriority ept vpid
> bogomips : 5799.78
> clflush size : 64
> cache_alignment : 64
> address sizes : 46 bits physical, 48 bits virtual
> power management:
>
>
>
>
> 4) The guest os cpuinfo
>
>>> [2] The /proc/cpuinfo shows below(show only the first cpu0),
>>> --------------------------------
>>> localhost:~ # cat /proc/cpuinfo
>>> processor : 0
>>> vendor_id : GenuineIntel
>>> cpu family : 6
>>> model : 45
>>> model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
>>> stepping : 7
>>> microcode : 0x1
>>> cpu MHz : 2899.992
>>> cache size : 4096 KB
>>> physical id : 0
>>> siblings : 8
>>> core id : 0
>>> cpu cores : 8
>>> apicid : 0
>>> initial apicid : 0
>>> fpu : yes
>>> fpu_exception : yes
>>> cpuid level : 13
>>> wp : yes
>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm xsaveopt
>>> bogomips : 5799.98
>>> clflush size : 64
>>> cache_alignment : 64
>>> address sizes : 42 bits physical, 48 bits virtual
>>> power management:
>>>
>>
>
> 5) parts of bootmsg
>
>
> [ 0.000000] Booting paravirtualized kernel on KVM
> [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
> [ 0.000000] PERCPU: Embedded 28 pages/cpu @ffff88040f400000 s82816 r8192 d23680 u262144
> [ 0.000000] KVM setup async PF for cpu 0
> [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 3933317
> [ 0.000000] Policy zone: Normal
> [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-229.20.1.x86_64 root=/dev/vda2 oops=panic softlockup_panic=1 net.ifnames=0 biosdevname=0 nmi_watchdog=1 selinux=0 console=tty0 panic=3
> [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
> [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
> [ 0.000000] Checking aperture...
> [ 0.000000] No AGP bridge found
> [ 0.000000] Memory: 15375712k/17032192k available (6255k kernel code, 1049096k absent, 607384k reserved, 4184k data, 1604k init)
> [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
> [ 0.000000] Hierarchical RCU implementation.
> [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=8.
> [ 0.000000] Experimental no-CBs for all CPUs
> [ 0.000000] Experimental no-CBs CPUs: 0-7.
> [ 0.000000] NR_IRQS:327936 nr_irqs:744 16
> [ 0.000000] Console: colour VGA+ 80x25
> [ 0.000000] console [tty0] enabled
> [ 0.000000] allocated 63963136 bytes of page_cgroup
> [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
> [ 0.000000] tsc: Detected 2899.992 MHz processor
> [ 0.002000] Calibrating delay loop (skipped) preset value.. 5799.98 BogoMIPS (lpj=2899992)
> [ 0.002006] pid_max: default: 32768 minimum: 301
> [ 0.003029] Security Framework initialized
> [ 0.003584] SELinux: Disabled at boot.
> [ 0.005623] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
> [ 0.009697] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
> [ 0.011961] Mount-cache hash table entries: 4096
> [ 0.012178] Initializing cgroup subsys memory
> [ 0.013011] Initializing cgroup subsys devices
> [ 0.013573] Initializing cgroup subsys freezer
> [ 0.014004] Initializing cgroup subsys net_cls
> [ 0.015003] Initializing cgroup subsys blkio
> [ 0.015558] Initializing cgroup subsys perf_event
> [ 0.016007] Initializing cgroup subsys hugetlb
> [ 0.016637] CPU: Physical Processor ID: 0
> [ 0.017003] CPU: Processor Core ID: 0
> [ 0.018004] mce: CPU supports 10 MCE banks
> [ 0.018597] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0
> Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0
> tlb_flushall_shift: 6
> [ 0.019169] Freeing SMP alternatives: 24k freed
> [ 0.023150] ACPI: Core revision 20130517
> [ 0.024384] ACPI: All ACPI Tables successfully acquired
> [ 0.025016] ftrace: allocating 23959 entries in 94 pages
> [ 0.031284] Enabling x2apic
> [ 0.031884] Enabled x2apic
> [ 0.032004] Switched APIC routing to physical x2apic.
> [ 0.033629] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> [ 0.034002] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz (fam: 06, model: 2d, step
> [ 0.035052] Performance Events: 16-deep LBR, SandyBridge events, Intel PMU driver.
> [ 0.037935] perf_event_intel: PEBS disabled due to CPU errata, please upgrade microcode
> [ 0.038005] ... version: 2
> [ 0.038544] ... bit width: 48
> [ 0.039003] ... generic registers: 4
> [ 0.039546] ... value mask: 0000ffffffffffff
> [ 0.040003] ... max period: 000000007fffffff
> [ 0.040565] ... fixed-purpose events: 3
> [ 0.041002] ... event mask: 000000070000000f
> [ 0.042819] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
> [ 0.042791] kvm-clock: cpu 1, msr 4:f889041, secondary cpu clock
> [ 0.055016] KVM setup async PF for cpu 1
> [ 0.055025] kvm-clock: cpu 2, msr 4:f889081, secondary cpu clock
> [ 0.067015] KVM setup async PF for cpu 2
> [ 0.067022] kvm-clock: cpu 3, msr 4:f8890c1, secondary cpu clock
> [ 0.079015] KVM setup async PF for cpu 3
> [ 0.079023] kvm-clock: cpu 4, msr 4:f889101, secondary cpu clock
> [ 0.091013] KVM setup async PF for cpu 4
> [ 0.091019] kvm-clock: cpu 5, msr 4:f889141, secondary cpu clock
> [ 0.103014] KVM setup async PF for cpu 5
> [ 0.103021] kvm-clock: cpu 6, msr 4:f889181, secondary cpu clock
> [ 0.115014] KVM setup async PF for cpu 6
> [ 0.043097] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK
> [ 0.115021] kvm-clock: cpu 7, msr 4:f8891c1, secondary cpu clock
> [ 0.127022] Brought up 8 CPUs
> [ 0.127015] KVM setup async PF for cpu 7
> [ 0.128005] smpboot: Total of 8 processors activated (46399.87 BogoMIPS)
> [ 0.129202] devtmpfs: initialized
>
>
>
> .
>