Hello John,
The real-time team at Red Hat is discussing backporting the rt patchset in 6.6
to RHEL 9/Stream 9. I decided to test v6.6.20-rt25 from stable-rt with
console_blast.sh. It reported similar NMIs from my testing of 6.7.0-rt6 with
that high cpu count server over uart 8250; which is expected since the patchset
is similar.
One interesting thing is that 6.7.0-rt6 fully preemptive + realtime tuned profile
did not return any NMIs while 6.6 did with that same configuration.
Another aspect I noticed during my testing. I did not set grub to
start with the realtime profile at boot for this machine. When I did set
it the second (and latter) NMI did not show for fully preemptive (the
3rd set at the bottom of this email).
Caller info was enabled. No modifications to the source code were made.
I have not tested previous versions before 6.7.0-rt6 or 6.6.20-rt25;
with the exception of accidently testing 6.6.10-rt19. 6.6.10 also
reported NMIs during this test. If you wish to see these reports please
let me know.
Also, please let me know if I can help in any other way or if you want anything else
tested. I will be moving to 6.8 and using BUG_ON(system_state == SYSTEM_RUNNING);
in the scheduler next for my tests to see if the messages do not show.
Link to lscpu and previous NMI reports:
https://lore.kernel.org/linux-rt-users/ZdUmpblrzX0pcPjz@thinkpad2021/T/#mec258264be95dcba23970def7b248af0c6e942b2
-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
-----------------------------
[ T2614] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
All code
========
0: 0f 1f 00 nopl (%rax)
3: 90 nop
4: 90 nop
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
18: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx
1f: 89 d0 mov %edx,%eax
21: 0f b7 57 08 movzwl 0x8(%rdi),%edx
25: d3 e6 shl %cl,%esi
27: 01 f2 add %esi,%edx
29: ee out %al,(%dx)
2a:* c3 ret <-- trapping instruction
2b: cc int3
2c: cc int3
2d: cc int3
2e: cc int3
2f: 0f 1f 40 00 nopl 0x0(%rax)
33: 90 nop
34: 90 nop
35: 90 nop
36: 90 nop
37: 90 nop
38: 90 nop
39: 90 nop
3a: 90 nop
3b: 90 nop
3c: 90 nop
3d: 90 nop
3e: 90 nop
3f: 90 nop
Code starting with the faulting instruction
===========================================
0: c3 ret
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: 0f 1f 40 00 nopl 0x0(%rax)
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
[ C56] RSP: 0018:ffa0000030a9fb28 EFLAGS: 00000002
[ C56] RAX: 000000000000005f RBX: ffffffffa6ac68c0 RCX: 0000000000000000
[ C56] RDX: 00000000000003f8 RSI: 0000000000000000 RDI: ffffffffa6ac68c0
[ C56] RBP: 000000000000005f R08: 303378302f623178 R09: 302b6d73615f6b72
[ C56] R10: 615f6b726f665f6d R11: 6f72665f74657220 R12: ffffffffa6ac68c0
[ C56] R13: ffffffffa69b9434 R14: 0000000000000000 R15: 0000000000000000
[ C56] FS: 00007f4a81d54740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ C56] CR2: 00007f4a81f30650 CR3: 000000011ea24004 CR4: 0000000000771ee0
[ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ C56] PKRU: 55555554
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] </NMI>
[ C56] <TASK>
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
[ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? do_dup2 (fs/file.c:1142)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4a81e5fc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ C56] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[ C56] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[ C56] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ C56] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[ C56] </TASK>
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
All code
========
0: cc int3
1: 0f 1f 40 00 nopl 0x0(%rax)
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
1a: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx
21: 0f b7 57 08 movzwl 0x8(%rdi),%edx
25: d3 e6 shl %cl,%esi
27: 01 f2 add %esi,%edx
29: ec in (%dx),%al
2a:* 0f b6 c0 movzbl %al,%eax <-- trapping instruction
2d: c3 ret
2e: cc int3
2f: cc int3
30: cc int3
31: cc int3
32: 0f 1f 00 nopl (%rax)
35: 90 nop
36: 90 nop
37: 90 nop
38: 90 nop
39: 90 nop
3a: 90 nop
3b: 90 nop
3c: 90 nop
3d: 90 nop
3e: 90 nop
3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f b6 c0 movzbl %al,%eax
3: c3 ret
4: cc int3
5: cc int3
6: cc int3
7: cc int3
8: 0f 1f 00 nopl (%rax)
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
[ C56] RSP: 0018:ffa0000030a9fb00 EFLAGS: 00000002
[ C56] RAX: ffffffffa45cf400 RBX: 0000000000000000 RCX: 0000000000000000
[ C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffffa6ac68c0
[ C56] RBP: 0000000000002710 R08: 303578302f643278 R09: 302b6b726f665f6d
[ C56] R10: 302b6b726f665f6d R11: 6f72665f74657220 R12: 0000000000000020
[ C56] R13: ffffffffa6ac68c0 R14: 0000000000000000 R15: 0000000000000000
[ C56] FS: 00007f4a81d54740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ C56] CR2: 00007f4a81f30650 CR3: 000000011ea24004 CR4: 0000000000771ee0
[ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ C56] PKRU: 55555554
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] </NMI>
[ C56] <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
[ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? do_dup2 (fs/file.c:1142)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4a81e5fc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ C56] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[ C56] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[ C56] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ C56] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[ C56] </TASK>
[ T2614] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T2614] Call Trace:
[ T2614] <TASK>
[ T2614] dump_stack_lvl (lib/dump_stack.c:107)
[ T2614] panic (kernel/panic.c:340)
[ T2614] ? _printk (kernel/printk/printk.c:2474)
[ T2614] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ T2614] __handle_sysrq (drivers/tty/sysrq.c:601)
[ T2614] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ T2614] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ T2614] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ T2614] vfs_write (fs/read_write.c:582)
[ T2614] ksys_write (fs/read_write.c:637)
[ T2614] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ T2614] ? do_dup2 (fs/file.c:1142)
[ T2614] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ T2614] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T2614] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ T2614] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ T2614] RIP: 0033:0x7f4a81e5fc94
[ T2614] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ T2614] RSP: 002b:00007ffc0e13b318 EFLAGS: 00000202
[ T2614] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4a81e5fc94
[ T2614] RDX: 0000000000000002 RSI: 000055a865695490 RDI: 0000000000000001
[ T2614] RBP: 00007ffc0e13b340 R08: 0000000000000410 R09: 0000000000000001
[ T2614] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T2614] R13: 000055a865695490 R14: 00007f4a81f305c0 R15: 00007f4a81f2df20
[ T2614] </TASK>
[ T2614] Shutting down cpus with NMI
[ T2614] Kernel Offset: disabled
[ T2614] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
[disconnect]
]777;notify;Command completed;KRB5CCNAME=/dev/null console -M conserver-02.hosts.prod.psi.bos.redhat.com intel-eaglestream-spr-15.khw3.lab.eng.bos.redhat.com]777;precmd]0;j@thinkpad2021:~/BTWork/records/2023/12Dec-testprintkupstream/6.6.20-rt25]7;file://thinkpad2021/home/j/BTWork/records/2023/12Dec-testprintkupstream/6.6.20-rt25[?2004h[38;5;166mj[38;5;228m6.6.20-rt25[38;5;43m[38;5;71m
>$ (B[mexit
[?2004l
]777;preexecexit
Script done on 2024-03-08 12:25:38-05:00 [COMMAND_EXIT_CODE="0"]
-----------------------------
NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
-----------------------------
[ T2546] sysrq: Trigger a crash
[ T2546] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
All code
========
0: cc int3
1: 0f 1f 40 00 nopl 0x0(%rax)
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
1a: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx
21: 0f b7 57 08 movzwl 0x8(%rdi),%edx
25: d3 e6 shl %cl,%esi
27: 01 f2 add %esi,%edx
29: ec in (%dx),%al
2a:* 0f b6 c0 movzbl %al,%eax <-- trapping instruction
2d: c3 ret
2e: cc int3
2f: cc int3
30: cc int3
31: cc int3
32: 0f 1f 00 nopl (%rax)
35: 90 nop
36: 90 nop
37: 90 nop
38: 90 nop
39: 90 nop
3a: 90 nop
3b: 90 nop
3c: 90 nop
3d: 90 nop
3e: 90 nop
3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f b6 c0 movzbl %al,%eax
3: c3 ret
4: cc int3
5: cc int3
6: cc int3
7: cc int3
8: 0f 1f 00 nopl (%rax)
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
[ C56] RSP: 0018:ffa00000308e7ae8 EFLAGS: 00000002
[ C56] RAX: ffffffff8d7cf400 RBX: 0000000000000000 RCX: 0000000000000000
[ C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8fcc68c0
[ C56] RBP: 000000000000270f R08: 6469703e2d727275 R09: 2020202020202020
[ C56] R10: 00000000632e2020 R11: 00000000632e2020 R12: 0000000000000020
[ C56] R13: ffffffff8fcc68c0 R14: 0000000000000000 R15: 0000000000000000
[ C56] FS: 00007f4c17301740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ C56] CR2: 00007f4c174dd650 CR3: 000000010994c004 CR4: 0000000000771ee0
[ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ C56] PKRU: 55555554
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] </NMI>
[ C56] <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
[ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ C56] vfs_write (fs/read_write.c:582)
[ C56] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1))
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622)
[ C56] ? do_user_addr_fault (arch/x86/mm/fault.c:1368)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4c1740cc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ C56] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94
[ C56] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001
[ C56] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001
[ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ C56] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20
[ C56] </TASK>
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] Code: cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 0f b7 57 08 d3 e6 01 f2 ec <0f> b6 c0 c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90
All code
========
0: cc int3
1: 0f 1f 40 00 nopl 0x0(%rax)
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
1a: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx
21: 0f b7 57 08 movzwl 0x8(%rdi),%edx
25: d3 e6 shl %cl,%esi
27: 01 f2 add %esi,%edx
29: ec in (%dx),%al
2a:* 0f b6 c0 movzbl %al,%eax <-- trapping instruction
2d: c3 ret
2e: cc int3
2f: cc int3
30: cc int3
31: cc int3
32: 0f 1f 00 nopl (%rax)
35: 90 nop
36: 90 nop
37: 90 nop
38: 90 nop
39: 90 nop
3a: 90 nop
3b: 90 nop
3c: 90 nop
3d: 90 nop
3e: 90 nop
3f: 90 nop
Code starting with the faulting instruction
===========================================
0: 0f b6 c0 movzbl %al,%eax
3: c3 ret
4: cc int3
5: cc int3
6: cc int3
7: cc int3
8: 0f 1f 00 nopl (%rax)
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
[ C56] RSP: 0018:ffa00000308e7ae8 EFLAGS: 00000002
[ C56] RAX: ffffffff8d7cf400 RBX: 0000000000000000 RCX: 0000000000000000
[ C56] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff8fcc68c0
[ C56] RBP: 000000000000270f R08: 64656c74746f7268 R09: 2020202020202020
[ C56] R10: 00000000742e2020 R11: 00000000742e2020 R12: 0000000000000020
[ C56] R13: ffffffff8fcc68c0 R14: 0000000000000000 R15: 0000000000000000
[ C56] FS: 00007f4c17301740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ C56] CR2: 00007f4c174dd650 CR3: 000000010994c004 CR4: 0000000000771ee0
[ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ C56] PKRU: 55555554
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? mem16_serial_out (drivers/tty/serial/8250/8250_port.c:382)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] ? io_serial_in (arch/x86/kernel/early_printk.c:100)
[ C56] </NMI>
[ C56] <TASK>
[ C56] wait_for_xmitr (drivers/tty/serial/8250/8250.h:117 drivers/tty/serial/8250/8250.h:139 drivers/tty/serial/8250/8250_port.c:2106 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
[ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ C56] vfs_write (fs/read_write.c:582)
[ C56] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1))
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622)
[ C56] ? do_user_addr_fault (arch/x86/mm/fault.c:1368)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f4c1740cc94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ C56] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94
[ C56] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001
[ C56] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001
[ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ C56] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20
[ C56] </TASK>
[ T2546] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T2546] Call Trace:
[ T2546] <TASK>
[ T2546] dump_stack_lvl (lib/dump_stack.c:107)
[ T2546] panic (kernel/panic.c:340)
[ T2546] ? _printk (kernel/printk/printk.c:2474)
[ T2546] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ T2546] __handle_sysrq (drivers/tty/sysrq.c:601)
[ T2546] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ T2546] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ T2546] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ T2546] vfs_write (fs/read_write.c:582)
[ T2546] ? __do_sys_newfstatat (fs/stat.c:466 (discriminator 1))
[ T2546] ksys_write (fs/read_write.c:637)
[ T2546] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ T2546] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ T2546] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T2546] ? up_read (./arch/x86/include/asm/preempt.h:104 kernel/locking/rwsem.c:1354 kernel/locking/rwsem.c:1622)
[ T2546] ? do_user_addr_fault (arch/x86/mm/fault.c:1368)
[ T2546] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ T2546] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ T2546] RIP: 0033:0x7f4c1740cc94
[ T2546] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ T2546] RSP: 002b:00007ffe88b9b298 EFLAGS: 00000202
[ T2546] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f4c1740cc94
[ T2546] RDX: 0000000000000002 RSI: 00005634f890b490 RDI: 0000000000000001
[ T2546] RBP: 00007ffe88b9b2c0 R08: 0000000000000410 R09: 0000000000000001
[ T2546] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T2546] R13: 00005634f890b490 R14: 00007f4c174dd5c0 R15: 00007f4c174daf20
[ T2546] </TASK>
[ T2546] Shutting down cpus with NMI
[ T2546] Kernel Offset: disabled
[ T2546] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
-----------------------------
NMI Backtrace for 6.6.20-rt25 fully preemptive with tuned realtime profile
-----------------------------
[ T3402] sysrq: Trigger a crash
[ T3406]
[ T3402] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] Code: cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8d 04 37 31 c9 48 89 c2 48 c1 ea 20 66 0f ae f1 <c3> cc cc cc cc 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90
All code
========
0: cc int3
1: 0f 1f 00 nopl (%rax)
4: 90 nop
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
19: 48 8d 04 37 lea (%rdi,%rsi,1),%rax
1d: 31 c9 xor %ecx,%ecx
1f: 48 89 c2 mov %rax,%rdx
22: 48 c1 ea 20 shr $0x20,%rdx
26: 66 0f ae f1 tpause %ecx
2a:* c3 ret <-- trapping instruction
2b: cc int3
2c: cc int3
2d: cc int3
2e: cc int3
2f: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
34: 90 nop
35: 90 nop
36: 90 nop
37: 90 nop
38: 90 nop
39: 90 nop
3a: 90 nop
3b: 90 nop
3c: 90 nop
3d: 90 nop
3e: 90 nop
3f: 90 nop
Code starting with the faulting instruction
===========================================
0: c3 ret
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 90 nop
10: 90 nop
11: 90 nop
12: 90 nop
13: 90 nop
14: 90 nop
15: 90 nop
[ C56] RSP: 0018:ffa0000039cfba80 EFLAGS: 00000002
[ C56] RAX: 00000099a649b2ae RBX: 00000000000006d5 RCX: 0000000000000000
[ C56] RDX: 0000000000000099 RSI: 00000000000006d5 RDI: 00000099a649abd9
[ C56] RBP: 00000099a649abd9 R08: 303178302f303178 R09: 302b646165726874
[ C56] R10: 6461657268746b5f R11: 7866705f5f203f20 R12: 0000000000000020
[ C56] R13: ffffffff9b8fd320 R14: 0000000000000000 R15: 0000000000000000
[ C56] FS: 00007f12ca52d740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ C56] CR2: 00007f12ca709650 CR3: 000000010fbe6003 CR4: 0000000000771ee0
[ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ C56] PKRU: 55555554
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] ? delay_halt_tpause (arch/x86/lib/delay.c:118)
[ C56] </NMI>
[ C56] <TASK>
[ C56] delay_halt.part.0 (arch/x86/lib/delay.c:163)
[ C56] wait_for_xmitr (./include/linux/nmi.h:147 drivers/tty/serial/8250/8250_port.c:2113 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
[ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f12ca638c94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ C56] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94
[ C56] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001
[ C56] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001
[ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ C56] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20
[ C56] </TASK>
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 53 48 89 fb 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 48 89 d5 eb 09 <48> 01 eb 48 89 c5 48 29 c3 48 8b 05 15 bd a0 00 48 89 de 48 89 ef
All code
========
0: 90 nop
1: 90 nop
2: 90 nop
3: 90 nop
4: 90 nop
5: 90 nop
6: 90 nop
7: 90 nop
8: 90 nop
9: 90 nop
a: 90 nop
b: 90 nop
c: 90 nop
d: 90 nop
e: 90 nop
f: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
14: 55 push %rbp
15: 53 push %rbx
16: 48 89 fb mov %rdi,%rbx
19: 0f 01 f9 rdtscp
1c: 66 90 xchg %ax,%ax
1e: 48 c1 e2 20 shl $0x20,%rdx
22: 48 09 c2 or %rax,%rdx
25: 48 89 d5 mov %rdx,%rbp
28: eb 09 jmp 0x33
2a:* 48 01 eb add %rbp,%rbx <-- trapping instruction
2d: 48 89 c5 mov %rax,%rbp
30: 48 29 c3 sub %rax,%rbx
33: 48 8b 05 15 bd a0 00 mov 0xa0bd15(%rip),%rax # 0xa0bd4f
3a: 48 89 de mov %rbx,%rsi
3d: 48 89 ef mov %rbp,%rdi
Code starting with the faulting instruction
===========================================
0: 48 01 eb add %rbp,%rbx
3: 48 89 c5 mov %rax,%rbp
6: 48 29 c3 sub %rax,%rbx
9: 48 8b 05 15 bd a0 00 mov 0xa0bd15(%rip),%rax # 0xa0bd25
10: 48 89 de mov %rbx,%rsi
13: 48 89 ef mov %rbp,%rdi
[ C56] RSP: 0018:ffa0000039cfba88 EFLAGS: 00000093
[ C56] RAX: 000000ec13deb29f RBX: 000000000000076d RCX: 0000000000004038
[ C56] RDX: 0000000000000050 RSI: 000000000000076d RDI: 000000ec13deb24f
[ C56] RBP: 000000ec13deb24f R08: 20203138312f7068 R09: 7570632020202020
[ C56] R10: 38312f7068757063 R11: 2020202020205320 R12: 0000000000000020
[ C56] R13: ffffffff9b8fd320 R14: 0000000000000000 R15: 0000000000000000
[ C56] FS: 00007f12ca52d740(0000) GS:ff11003fff600000(0000) knlGS:0000000000000000
[ C56] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ C56] CR2: 00007f12ca709650 CR3: 000000010fbe6003 CR4: 0000000000771ee0
[ C56] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ C56] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[ C56] PKRU: 55555554
[ C56] Call Trace:
[ C56] <NMI>
[ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
[ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
[ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
[ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
[ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] ? delay_halt.part.0 (arch/x86/lib/delay.c:169)
[ C56] </NMI>
[ C56] <TASK>
[ C56] wait_for_xmitr (./include/linux/nmi.h:147 drivers/tty/serial/8250/8250_port.c:2113 drivers/tty/serial/8250/8250_port.c:2124)
[ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
[ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
[ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
[ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
[ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
[ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
[ C56] vprintk_emit (kernel/printk/printk.c:2414)
[ C56] _printk (kernel/printk/printk.c:2474)
[ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
[ C56] ? _printk (kernel/printk/printk.c:2474)
[ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
[ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ C56] vfs_write (fs/read_write.c:582)
[ C56] ksys_write (fs/read_write.c:637)
[ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ C56] RIP: 0033:0x7f12ca638c94
[ C56] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ C56] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ C56] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94
[ C56] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001
[ C56] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001
[ C56] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ C56] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20
[ C56] </TASK>
[ T3402] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ T3402] Call Trace:
[ T3402] <TASK>
[ T3402] dump_stack_lvl (lib/dump_stack.c:107)
[ T3402] panic (kernel/panic.c:340)
[ T3402] ? _printk (kernel/printk/printk.c:2474)
[ T3402] sysrq_handle_crash (drivers/tty/sysrq.c:154)
[ T3402] __handle_sysrq (drivers/tty/sysrq.c:601)
[ T3402] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
[ T3402] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
[ T3402] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
[ T3402] vfs_write (fs/read_write.c:582)
[ T3402] ksys_write (fs/read_write.c:637)
[ T3402] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
[ T3402] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
[ T3402] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T3402] ? do_syscall_64 (arch/x86/entry/common.c:88)
[ T3402] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
[ T3402] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
[ T3402] RIP: 0033:0x7f12ca638c94
[ T3402] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d d5 76 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
All code
========
0: c7 00 16 00 00 00 movl $0x16,(%rax)
6: b8 ff ff ff ff mov $0xffffffff,%eax
b: c3 ret
c: 66 2e 0f 1f 84 00 00 cs nopw 0x0(%rax,%rax,1)
13: 00 00 00
16: f3 0f 1e fa endbr64
1a: 80 3d d5 76 0d 00 00 cmpb $0x0,0xd76d5(%rip) # 0xd76f6
21: 74 13 je 0x36
23: b8 01 00 00 00 mov $0x1,%eax
28: 0f 05 syscall
2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction
30: 77 54 ja 0x86
32: c3 ret
33: 0f 1f 00 nopl (%rax)
36: 55 push %rbp
37: 48 89 e5 mov %rsp,%rbp
3a: 48 83 ec 20 sub $0x20,%rsp
3e: 48 rex.W
3f: 89 .byte 0x89
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax
6: 77 54 ja 0x5c
8: c3 ret
9: 0f 1f 00 nopl (%rax)
c: 55 push %rbp
d: 48 89 e5 mov %rsp,%rbp
10: 48 83 ec 20 sub $0x20,%rsp
14: 48 rex.W
15: 89 .byte 0x89
[ T3402] RSP: 002b:00007ffce58c2ef8 EFLAGS: 00000202
[ T3402] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f12ca638c94
[ T3402] RDX: 0000000000000002 RSI: 000055b5493b8490 RDI: 0000000000000001
[ T3402] RBP: 00007ffce58c2f20 R08: 0000000000000410 R09: 0000000000000001
[ T3402] R10: 0000000000000004 R11: 0000000000000202 R12: 0000000000000002
[ T3402] R13: 000055b5493b8490 R14: 00007f12ca7095c0 R15: 00007f12ca706f20
[ T3402] </TASK>
[ T3402] Kernel Offset: disabled
[ T3402] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
--
Sincerly,
John Wyatt
Software Engineer, Core Kernel
On 2024-03-15 15:21:19 [-0400], John B. Wyatt IV wrote:
> Hello John,
Hi John B.,
> The real-time team at Red Hat is discussing backporting the rt patchset in 6.6
> to RHEL 9/Stream 9. I decided to test v6.6.20-rt25 from stable-rt with
> console_blast.sh. It reported similar NMIs from my testing of 6.7.0-rt6 with
> that high cpu count server over uart 8250; which is expected since the patchset
> is similar.
>
> One interesting thing is that 6.7.0-rt6 fully preemptive + realtime tuned profile
> did not return any NMIs while 6.6 did with that same configuration.
The thing is that console_blast.sh does this "show a backtrace on all
CPUs, please" which triggers NMIs on all CPUs for backtrace. I can't
imagine how you did obtain the backtraces without an NMI. Unless the
tuned profile disables this somehow.
> Another aspect I noticed during my testing. I did not set grub to
> start with the realtime profile at boot for this machine. When I did set
> it the second (and latter) NMI did not show for fully preemptive (the
> 3rd set at the bottom of this email).
>
> Caller info was enabled. No modifications to the source code were made.
>
> I have not tested previous versions before 6.7.0-rt6 or 6.6.20-rt25;
> with the exception of accidently testing 6.6.10-rt19. 6.6.10 also
> reported NMIs during this test. If you wish to see these reports please
> let me know.
This NMI part has nothing todo with printk. If you need this clarified,
I would need a reproducer.
…
> -----------------------------
> NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
> -----------------------------
>
> [ T2614] Kernel panic - not syncing: sysrq triggered crash
> [ C56] NMI backtrace for cpu 56
> [ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
> [ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
> [ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
> All code
> =======
…
> 12: 90 nop
> 13: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
> 18: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx
> 1f: 89 d0 mov %edx,%eax
> 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx
> 25: d3 e6 shl %cl,%esi
> 27: 01 f2 add %esi,%edx
> 29: ee out %al,(%dx)
> 2a:* c3 ret <-- trapping instruction
where is this output from? The `ret' opcode usually does not cause a
trap. My guess is that the machine has been interrupted by an external
user at this position.
Side note: This is using early_printk, correct?
…
> [ C56] Call Trace:
> [ C56] <NMI>
> [ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
> [ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
> [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> [ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
> [ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
> [ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> [ C56] </NMI>
This looks okay. The NMI did the backtrace as expected.
> [ C56] <TASK>
> [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
> [ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
> [ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
> [ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
> [ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
> [ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
> [ C56] vprintk_emit (kernel/printk/printk.c:2414)
> [ C56] _printk (kernel/printk/printk.c:2474)
> [ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
> [ C56] ? _printk (kernel/printk/printk.c:2474)
> [ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
> [ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
> [ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
> [ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
> [ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
> [ C56] vfs_write (fs/read_write.c:582)
> [ C56] ksys_write (fs/read_write.c:637)
> [ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
> [ C56] ? do_dup2 (fs/file.c:1142)
> [ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
> [ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
> [ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
> [ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
According to this, someone issued a `crash' via sysrq. Why?
…
> [ T2614] ---[ end Kernel panic - not syncing: sysrq triggered crash ]---
yes exactly.
…
> NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> -----------------------------
This and the following backtrace shows the same picture: The CPU is
crashing due to proc/sysrq request and does CPU-backtraces via NMI and
polls in early_printk, waiting for the UART to become idle (probably).
I don't see an issue here so far.
…
> --
> Sincerly,
> John Wyatt
> Software Engineer, Core Kernel
Sebastian
On Fri, Mar 22, 2024 at 01:29:21PM +0100, Sebastian Andrzej Siewior wrote:
> On 2024-03-15 15:21:19 [-0400], John B. Wyatt IV wrote:
> The thing is that console_blast.sh does this "show a backtrace on all
> CPUs, please" which triggers NMIs on all CPUs for backtrace. I can't
> imagine how you did obtain the backtraces without an NMI. Unless the
> tuned profile disables this somehow.
I may have misapplied it. This is a very manual process right now due to
conserver.
> > -----------------------------
> > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned realtime profile
> > -----------------------------
> >
> > [ T2614] Kernel panic - not syncing: sysrq triggered crash
> > [ C56] NMI backtrace for cpu 56
> > [ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
> > [ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
> > [ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
> > All code
> > =======
> …
> > 12: 90 nop
> > 13: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
> > 18: 0f b6 8f c1 00 00 00 movzbl 0xc1(%rdi),%ecx
> > 1f: 89 d0 mov %edx,%eax
> > 21: 0f b7 57 08 movzwl 0x8(%rdi),%edx
> > 25: d3 e6 shl %cl,%esi
> > 27: 01 f2 add %esi,%edx
> > 29: ee out %al,(%dx)
> > 2a:* c3 ret <-- trapping instruction
>
> where is this output from? The `ret' opcode usually does not cause a
> trap. My guess is that the machine has been interrupted by an external
> user at this position.
Just before the sysrq that crashes the system.
[ T2618] ret_from_fork_asm (arch/x86/entry/entry_64.S:312)
[ T2618] </TASK>
[ T2618] task:kworker/185:0 state:I stack:0 pid:1131 ppid:2 flags:0x00004000
[ T2618] Workqueue: 0x0 (events)
[ T2618] Call Trace:
[ T2618] <TASK>
[ T2618] __schedule (kernel/sched/core.c:5404 kernel/sched/core.c:6718)
[ T2618] schedule (kernel/sched/core.c:6796 kernel/sched/core.c:6810)
[ T2618] worker_thread (kernel/workqueue.c:2734)
[ T2618] ? __pfx_worker_thread (kernel/workqueue.c:2727)
[ T2618] kthread (kernel/kthread.c:388)
[ T2618] ? __pfx_kthread (kernel/kthread.c:341)
[ T2618] ret_from_fork (arch/x86/kernel/process.c:147)
[ T2618] ? __pfx_kthread (kernel/kthread.c:341)
[ T2618] ret_from_fork_asm (arch/x86/entry/entry_64.S:312)
[ T2618] </TASK>
[ T2614] sysrq: Trigger a crash
[ T2618] task:kworker/185:0H state:I
[ T2618] stack:0 pid:1132 ppid:2 flags:0x00004000
[ T2614] Kernel panic - not syncing: sysrq triggered crash
[ C56] NMI backtrace for cpu 56
[ C56] Hardware name: Intel Corporation D50DNP1SBB/D50DNP1SBB, BIOS SE5C7411.86B.9409.D04.2212261349 12/26/2022
[ C56] RIP: 0010:io_serial_out (arch/x86/kernel/early_printk.c:105)
[ C56] Code: 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f b6 8f c1 00 00 00 89 d0 0f b7 57 08 d3 e6 01 f2 ee <c3> cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90
> Side note: This is using early_printk, correct?
I believe so, but it might be preempted? This is the part it stopped in.
static void io_serial_out(unsigned long addr, int offset, int value)
{
outb(value, addr + offset);
}
> …
>
> > [ C56] Call Trace:
> > [ C56] <NMI>
> > [ C56] ? nmi_cpu_backtrace (lib/nmi_backtrace.c:115)
> > [ C56] ? nmi_cpu_backtrace_handler (arch/x86/kernel/apic/hw_nmi.c:47 (discriminator 1))
> > [ C56] ? nmi_handle (arch/x86/kernel/nmi.c:149)
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> > [ C56] ? default_do_nmi (arch/x86/kernel/nmi.c:347)
> > [ C56] ? exc_nmi (arch/x86/kernel/nmi.c:538)
> > [ C56] ? end_repeat_nmi (arch/x86/entry/entry_64.S:1458)
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> > [ C56] ? io_serial_out (arch/x86/kernel/early_printk.c:105)
> > [ C56] </NMI>
>
> This looks okay. The NMI did the backtrace as expected.
>
> > [ C56] <TASK>
> > [ C56] serial8250_console_putchar (./include/linux/serial_core.h:704 drivers/tty/serial/8250/8250_port.c:3347)
> > [ C56] ? __pfx_serial8250_console_putchar (drivers/tty/serial/8250/8250_port.c:3343)
> > [ C56] uart_console_write (drivers/tty/serial/serial_core.c:2134)
> > [ C56] serial8250_console_write_atomic (drivers/tty/serial/8250/8250_port.c:3628)
> > [ C56] nbcon_emit_next_record (kernel/printk/nbcon.c:940)
> > [ C56] __nbcon_atomic_flush_all (kernel/printk/nbcon.c:1192 (discriminator 1) kernel/printk/nbcon.c:1326 (discriminator 1))
> > [ C56] vprintk_emit (kernel/printk/printk.c:2414)
> > [ C56] _printk (kernel/printk/printk.c:2474)
> > [ C56] panic (./arch/x86/include/asm/bitops.h:207 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 kernel/panic.c:528 kernel/panic.c:339)
> > [ C56] ? _printk (kernel/printk/printk.c:2474)
> > [ C56] sysrq_handle_crash (drivers/tty/sysrq.c:154)
> > [ C56] __handle_sysrq (drivers/tty/sysrq.c:601)
> > [ C56] write_sysrq_trigger (drivers/tty/sysrq.c:1165)
> > [ C56] proc_reg_write (fs/proc/inode.c:340 fs/proc/inode.c:352)
> > [ C56] ? preempt_count_add (./include/linux/ftrace.h:974 (discriminator 1) kernel/sched/core.c:5847 (discriminator 1) kernel/sched/core.c:5844 (discriminator 1) kernel/sched/core.c:5872 (discriminator 1))
> > [ C56] vfs_write (fs/read_write.c:582)
> > [ C56] ksys_write (fs/read_write.c:637)
> > [ C56] do_syscall_64 (arch/x86/entry/common.c:51 arch/x86/entry/common.c:81)
> > [ C56] ? do_dup2 (fs/file.c:1142)
> > [ C56] ? syscall_exit_to_user_mode (kernel/entry/common.c:299)
> > [ C56] ? do_syscall_64 (arch/x86/entry/common.c:88)
> > [ C56] ? exc_page_fault (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:72 arch/x86/mm/fault.c:1513 arch/x86/mm/fault.c:1561)
> > [ C56] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120)
>
> According to this, someone issued a `crash' via sysrq. Why?
>
This is part of the console_blast.sh script that John Ogness sent me.
Please see below:
#!/bin/bash
set -e
# This script is intended to be started with no arguments.
#
# It then calls itself to create a pinned process for each CPU. Those
# child processes will run in infinite loops of show-task-states via
# /proc/sysrq-trigger. This generates lots of contention on the console.
#
# Finally, it waits 10 seconds before crashing the system.
if [ $# -eq 0 ]; then
cpus=$(($(nproc) - 1))
for i in $(seq 0 $cpus); do
$0 $i &
done
sleep 15
echo c > /proc/sysrq-trigger
exit 0
fi
# Setup the CPU mask for the provided CPU number. The real taskset(1)
# could simply use -c, but the busybox(1) variant does not support this.
mask=$(printf "0x%x\n" $((1 << $1)))
taskset -p $mask $$
# Wait a moment before beginning blast.
sleep 2
while true; do
echo t > /proc/sysrq-trigger
done
> > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> > -----------------------------
>
> This and the following backtrace shows the same picture: The CPU is
> crashing due to proc/sysrq request and does CPU-backtraces via NMI and
> polls in early_printk, waiting for the UART to become idle (probably).
>
> I don't see an issue here so far.
Luis Goncalves discussed it with me after reading your response. Thank
you for your help. The NMI was needed to flush the buffers upon the
system crashing itself. Does this part about NMI watchdog need to be
documented?
--
Sincerly,
John Wyatt
Software Engineer, Core Kernel
On 2024-03-27 19:44:20 [-0400], John B. Wyatt IV wrote:
> > where is this output from? The `ret' opcode usually does not cause a
> > trap. My guess is that the machine has been interrupted by an external
> > user at this position.
>
> Just before the sysrq that crashes the system.
so this is intentional.
…
> > Side note: This is using early_printk, correct?
>
> I believe so, but it might be preempted? This is the part it stopped in.
>
> static void io_serial_out(unsigned long addr, int offset, int value)
> {
> outb(value, addr + offset);
> }
The function is invoked in NMU context so it can't be preempted.
> > According to this, someone issued a `crash' via sysrq. Why?
> >
>
> This is part of the console_blast.sh script that John Ogness sent me.
>
> Please see below:
…
Okay. Then everything works as it should…
> > > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> > > -----------------------------
> >
> > This and the following backtrace shows the same picture: The CPU is
> > crashing due to proc/sysrq request and does CPU-backtraces via NMI and
> > polls in early_printk, waiting for the UART to become idle (probably).
> >
> > I don't see an issue here so far.
>
> Luis Goncalves discussed it with me after reading your response. Thank
> you for your help. The NMI was needed to flush the buffers upon the
> system crashing itself. Does this part about NMI watchdog need to be
> documented?
Not sure about that one. There is an _a_ _lot_ to be printed from NMI
and the NMI watchdog might trigger if nothing is triggering the
NMI-watchdog during the print job. Also, the crash was requested.
Sebastian
On Tue, Apr 02, 2024 at 12:34:14PM +0200, Sebastian Andrzej Siewior wrote:
> > Just before the sysrq that crashes the system.
>
> so this is intentional.
Yes.
> >
> > This is part of the console_blast.sh script that John Ogness sent me.
> >
> > Please see below:
> …
>
> Okay. Then everything works as it should…
Correct.
>
> > > > NMI Backtrace for 6.6.20-rt25 no forced preemption with tuned throughput-performance profile
> > > > -----------------------------
> > >
> > > This and the following backtrace shows the same picture: The CPU is
> > > crashing due to proc/sysrq request and does CPU-backtraces via NMI and
> > > polls in early_printk, waiting for the UART to become idle (probably).
> > >
> > > I don't see an issue here so far.
> >
> > Luis Goncalves discussed it with me after reading your response. Thank
> > you for your help. The NMI was needed to flush the buffers upon the
> > system crashing itself. Does this part about NMI watchdog need to be
> > documented?
>
> Not sure about that one. There is an _a_ _lot_ to be printed from NMI
> and the NMI watchdog might trigger if nothing is triggering the
> NMI-watchdog during the print job. Also, the crash was requested.
I reran the 6.6 test and no NMI was reported with fully preemptive and
the realtime tuned profile. It was my error; my apologies for that.
I did include more of logs if you want to see here for my 6.8 testing.
https://lore.kernel.org/linux-rt-users/ZgWL2UyknaE2T70C@thinkpad2021/T/#u
Sincerely,
John Wyatt
Software Engineer, Core Kernel
Red Hat
On 2024-04-04 20:10:47 [-0400], John B. Wyatt IV wrote:
> I reran the 6.6 test and no NMI was reported with fully preemptive and
> the realtime tuned profile. It was my error; my apologies for that.
no worries.
> I did include more of logs if you want to see here for my 6.8 testing.
>
> https://lore.kernel.org/linux-rt-users/ZgWL2UyknaE2T70C@thinkpad2021/T/#u
I looked over it and it seems we are good.
Thanks for testing.
> Sincerely,
> John Wyatt
> Software Engineer, Core Kernel
> Red Hat
Sebastian