The skd driver has never handled discards reliably.
The kernel will BUG as a result of issuing discards to the skd device.
Disable the skd driver's discard support until it is proven reliable.
The device-mapper-test-suite test that exposed this bug just issues a
discard that covers a portion of the skd device that was previously
written through a dm-thin device. The discard spans the entire 1GB thin
device (logical sector 0 through 2097152).
dmtest run --profile stec --suite thin-provisioning -n /discard_fully_provisioned_device/
associated device-mapper-test-suite ruby test code follows:
def test_discard_fully_provisioned_device
with_standard_pool(@size) do |pool|
with_new_thins(pool, @volume_size, 0, 1) do |thin, thin2|
wipe_device(thin)
wipe_device(thin2)
assert_used_blocks(pool, 2 * @blocks_per_dev)
thin.discard(0, @volume_size)
assert_used_blocks(pool, @blocks_per_dev)
end
end
...
Signed-off-by: Mike Snitzer <[email protected]>
---
drivers/block/skd_main.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c
index eb6e1e0..5dadecc 100644
--- a/drivers/block/skd_main.c
+++ b/drivers/block/skd_main.c
@@ -4441,12 +4441,15 @@ static int skd_cons_disk(struct skd_device *skdev)
/* set sysfs ptimal_io_size to 8K */
blk_queue_io_opt(q, 8192);
+#if 0
+ /* FIXME: Disable discard support until it no longer BUGs */
/* DISCARD Flag initialization. */
q->limits.discard_granularity = 8192;
q->limits.discard_alignment = 0;
q->limits.max_discard_sectors = UINT_MAX >> 9;
q->limits.discard_zeroes_data = 1;
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, q);
+#endif
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, q);
spin_lock_irqsave(&skdev->lock, flags);
On Wed, Feb 12 2014 at 5:18pm -0500,
Mike Snitzer <[email protected]> wrote:
> The skd driver has never handled discards reliably.
>
> The kernel will BUG as a result of issuing discards to the skd device.
> Disable the skd driver's discard support until it is proven reliable.
Here is the first BUG I recently saw:
------------[ cut here ]------------
Uhhuh. NMI received for unknown reason 21 on CPU 0.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
kernel BUG at include/linux/scatterlist.h:65!
invalid opcode: 0000 [#1] SMP
Modules linked in: dm_thin_pool(O) dm_bio_prison(O) dm_persistent_data(O) dm_bufio(O) dm_mod(O) libcrc32c ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc fcoe libfcoe libfc 8021q scsi_transport_fc garp scsi_tgt stp llc sunrpc cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core i7core_edac edac_core ixgbe dca ptp pps_core mdio ses enclosure sg acpi_cpufreq ext4 jbd2 mbcache sr_mod cdrom pata_acpi ata_generic ata_piix skd sd_mod crc_t10dif crct10dif_common megaraid_sas [last unloaded: dm_mod]
CPU: 5 PID: 0 Comm: swapper/5 Tainted: G W O 3.14.0-rc1.snitm+ #4
Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011
task: ffff8803329aef20 ti: ffff8803329b0000 task.ti: ffff8803329b0000
RIP: 0010:[<ffffffff8125a481>] [<ffffffff8125a481>] blk_rq_map_sg+0x241/0x3f0
RSP: 0018:ffff88033fca3c38 EFLAGS: 00010002
RAX: ffffea000b2984f0 RBX: 0000000000010000 RCX: ffff8803286f6020
RDX: ffffea000b2984f0 RSI: 0000000000000000 RDI: ffff8803286f6000
RBP: ffff88033fca3cc8 R08: ffff8803290233c0 R09: 53538ec752528dc6
R10: ffff88032826f8e0 R11: 00000000555590c9 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000010000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff88033fca0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003bfd2f5170 CR3: 0000000001a0b000 CR4: 00000000000007e0
Stack:
0000000000000002 0000000000000000 ffff88033fca3cf8 ffffffff8112f67e
ffff88033ffd7d80 ffff88032826f8e0 ffff8802555590c9 53538ec752528dc6
ffff8802b66bbb28 ffff8803286f6000 ffff8803290233c0 0000000101fd8b08
Call Trace:
<IRQ>
[<ffffffff8112f67e>] ? __alloc_pages_nodemask+0x12e/0x250
[<ffffffffa0071146>] skd_preop_sg_list+0x46/0x270 [skd]
[<ffffffff811703f2>] ? alloc_pages_current+0xb2/0x170
[<ffffffffa0072997>] skd_request_fn+0x287/0x900 [skd]
[<ffffffffa007551e>] ? skd_isr_completion_posted+0x1ee/0x5d0 [skd]
[<ffffffffa0076233>] skd_isr+0x1a3/0x280 [skd]
[<ffffffff810a73ed>] handle_irq_event_percpu+0x6d/0x200
[<ffffffff810a75c2>] handle_irq_event+0x42/0x70
[<ffffffff810aad19>] handle_edge_irq+0x69/0x120
[<ffffffff81005aec>] handle_irq+0x5c/0x150
[<ffffffff815471f2>] ? __atomic_notifier_call_chain+0x12/0x20
[<ffffffff81547216>] ? atomic_notifier_call_chain+0x16/0x20
[<ffffffff8154da1e>] do_IRQ+0x5e/0x110
[<ffffffff8154376a>] common_interrupt+0x6a/0x6a
<EOI>
[<ffffffff8144d6c3>] ? cpuidle_enter_state+0x53/0xd0
[<ffffffff8144d6bf>] ? cpuidle_enter_state+0x4f/0xd0
[<ffffffff8144d887>] cpuidle_idle_call+0xc7/0x160
[<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30
[<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240
[<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130
[<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30
[<ffffffff81032d5a>] start_secondary+0x7a/0x80
Code: 41 5f c9 c3 66 0f 1f 44 00 00 44 29 f3 44 89 f2 44 89 de 4c 89 c8 eb 93 66 90 48 8b 4d b8 41 f6 c1 03 48 8b 01 0f 84 03 ff ff ff <0f> 0b eb fe 0f 1f 00 48 8b 45 c0 4c 8b 85 78 ff ff ff 48 8b b0
RIP [<ffffffff8125a481>] blk_rq_map_sg+0x241/0x3f0
RSP <ffff88033fca3c38>
---[ end trace 61da6cb864bf7eb8 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Shutting down cpus with NMI
Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
On Wed, Feb 12 2014 at 5:19pm -0500,
Mike Snitzer <[email protected]> wrote:
> On Wed, Feb 12 2014 at 5:18pm -0500,
> Mike Snitzer <[email protected]> wrote:
>
> > The skd driver has never handled discards reliably.
> >
> > The kernel will BUG as a result of issuing discards to the skd device.
> > Disable the skd driver's discard support until it is proven reliable.
>
> Here is the first BUG I recently saw:
And a 2nd:
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 10
CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W O 3.14.0-rc1.snitm+ #4
Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011
0000000000000000 ffff88033fd47bb8 ffffffff8153f180 000000000000fffa
ffffffff817d8778 ffff88033fd47c38 ffffffff8153ef0d 0000000000000010
ffff88033fd47c48 ffff88033fd47be8 0000000000000000 0000000000000000
Call Trace:
<NMI> [<ffffffff8153f180>] dump_stack+0x49/0x61
[<ffffffff8153ef0d>] panic+0xbb/0x1d5
[<ffffffff810e8761>] watchdog_overflow_callback+0xb1/0xc0
[<ffffffff8111e9b8>] __perf_event_overflow+0x98/0x220
[<ffffffff8111f2a4>] perf_event_overflow+0x14/0x20
[<ffffffff8102012e>] intel_pmu_handle_irq+0x1de/0x3c0
[<ffffffff8115f931>] ? unmap_kernel_range_noflush+0x11/0x20
[<ffffffff8131a5c5>] ? ghes_copy_tofrom_phys+0xe5/0x200
[<ffffffff81544e84>] perf_event_nmi_handler+0x34/0x60
[<ffffffff8154464a>] nmi_handle+0x8a/0x170
[<ffffffff81544848>] default_do_nmi+0x68/0x210
[<ffffffff81544a80>] do_nmi+0x90/0xe0
[<ffffffff81543ca7>] end_repeat_nmi+0x1e/0x2e
[<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd]
[<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30
[<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30
[<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30
<<EOE>> <IRQ> [<ffffffffa06ef7d9>] skd_timer_tick+0x39/0x1e0 [skd]
[<ffffffff81069480>] ? __queue_work+0x360/0x360
[<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd]
[<ffffffff8105a318>] call_timer_fn+0x48/0x120
[<ffffffff8105aef5>] run_timer_softirq+0x225/0x290
[<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd]
[<ffffffff8105365c>] __do_softirq+0xfc/0x2b0
[<ffffffff810bc09f>] ? tick_do_update_jiffies64+0x9f/0xd0
[<ffffffff8105390d>] irq_exit+0xbd/0xd0
[<ffffffff8154dbea>] smp_apic_timer_interrupt+0x4a/0x5a
[<ffffffff8154c8ca>] apic_timer_interrupt+0x6a/0x70
<EOI> [<ffffffff8144d710>] ? cpuidle_enter_state+0xa0/0xd0
[<ffffffff8144d6cb>] ? cpuidle_enter_state+0x5b/0xd0
[<ffffffff8144d887>] cpuidle_idle_call+0xc7/0x160
[<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30
[<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240
[<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130
[<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30
[<ffffffff81032d5a>] start_secondary+0x7a/0x80
Shutting down cpus with NMI
Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
------------[ cut here ]------------
WARNING: CPU: 10 PID: 0 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5f/0x70()
Modules linked in: skd(O) dm_thin_pool(O) dm_bio_prison(O) dm_persistent_data(O) dm_bufio(O) libcrc32c ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc fcoe libfcoe 8021q libfc garp stp scsi_transport_fc llc scsi_tgt sunrpc cpufreq_ondemand ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan vhost tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core i7core_edac edac_core ixgbe dca ptp pps_core mdio ses enclosure sg acpi_cpufreq ext4 jbd2 mbcache sr_mod cdrom pata_acpi ata_generic ata_piix sd_mod crc_t10dif crct10dif_common dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: skd]
CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W O 3.14.0-rc1.snitm+ #4
Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011
000000000000007c ffff88033fd478c0 ffffffff8153f180 000000000000007c
0000000000000000 ffff88033fd47900 ffffffff8104e9bc ffff88033fd52c40
ffff88033fc52c40 0000000000000002 ffff88033fd52c40 ffff8803329be250
Call Trace:
<NMI> [<ffffffff8153f180>] dump_stack+0x49/0x61
[<ffffffff8104e9bc>] warn_slowpath_common+0x8c/0xc0
[<ffffffff8104ea0a>] warn_slowpath_null+0x1a/0x20
[<ffffffff8103141f>] native_smp_send_reschedule+0x5f/0x70
[<ffffffff81087e3e>] trigger_load_balance+0x15e/0x200
[<ffffffff8107ccf7>] scheduler_tick+0xa7/0xe0
[<ffffffff8105a031>] update_process_times+0x61/0x80
[<ffffffff8131863c>] ? apei_exec_write_register_value+0x1c/0x20
[<ffffffff810bbfb9>] tick_sched_handle+0x39/0x80
[<ffffffff810bc1e4>] tick_sched_timer+0x54/0x90
[<ffffffff810743be>] __run_hrtimer+0x7e/0x1c0
[<ffffffff810bc190>] ? tick_nohz_handler+0xc0/0xc0
[<ffffffff810747ae>] hrtimer_interrupt+0x10e/0x260
[<ffffffff8103489b>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffff8154dbe5>] smp_apic_timer_interrupt+0x45/0x5a
[<ffffffff8154c8ca>] apic_timer_interrupt+0x6a/0x70
[<ffffffff8153efe4>] ? panic+0x192/0x1d5
[<ffffffff8153ef42>] ? panic+0xf0/0x1d5
[<ffffffff810e8761>] watchdog_overflow_callback+0xb1/0xc0
[<ffffffff8111e9b8>] __perf_event_overflow+0x98/0x220
[<ffffffff8111f2a4>] perf_event_overflow+0x14/0x20
[<ffffffff8102012e>] intel_pmu_handle_irq+0x1de/0x3c0
[<ffffffff8115f931>] ? unmap_kernel_range_noflush+0x11/0x20
[<ffffffff8131a5c5>] ? ghes_copy_tofrom_phys+0xe5/0x200
[<ffffffff81544e84>] perf_event_nmi_handler+0x34/0x60
[<ffffffff8154464a>] nmi_handle+0x8a/0x170
[<ffffffff81544848>] default_do_nmi+0x68/0x210
[<ffffffff81544a80>] do_nmi+0x90/0xe0
[<ffffffff81543ca7>] end_repeat_nmi+0x1e/0x2e
[<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd]
[<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30
[<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30
[<ffffffff815432a1>] ? _raw_spin_lock_irqsave+0x21/0x30
<<EOE>> <IRQ> [<ffffffffa06ef7d9>] skd_timer_tick+0x39/0x1e0 [skd]
[<ffffffff81069480>] ? __queue_work+0x360/0x360
[<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd]
[<ffffffff8105a318>] call_timer_fn+0x48/0x120
[<ffffffff8105aef5>] run_timer_softirq+0x225/0x290
[<ffffffffa06ef7a0>] ? skd_timer_tick_not_online+0x330/0x330 [skd]
[<ffffffff8105365c>] __do_softirq+0xfc/0x2b0
[<ffffffff810bc09f>] ? tick_do_update_jiffies64+0x9f/0xd0
[<ffffffff8105390d>] irq_exit+0xbd/0xd0
[<ffffffff8154dbea>] smp_apic_timer_interrupt+0x4a/0x5a
[<ffffffff8154c8ca>] apic_timer_interrupt+0x6a/0x70
<EOI> [<ffffffff8144d710>] ? cpuidle_enter_state+0xa0/0xd0
[<ffffffff8144d6cb>] ? cpuidle_enter_state+0x5b/0xd0
[<ffffffff8144d887>] cpuidle_idle_call+0xc7/0x160
[<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30
[<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240
[<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130
[<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30
[<ffffffff81032d5a>] start_secondary+0x7a/0x80
---[ end trace 72a22a0dddd989d3 ]---
On Wed, Feb 12 2014 at 5:18pm -0500,
Mike Snitzer <[email protected]> wrote:
> The skd driver has never handled discards reliably.
>
> The kernel will BUG as a result of issuing discards to the skd device.
> Disable the skd driver's discard support until it is proven reliable.
>
> The device-mapper-test-suite test that exposed this bug just issues a
> discard that covers a portion of the skd device that was previously
> written through a dm-thin device. The discard spans the entire 1GB thin
> device (logical sector 0 through 2097152).
>
> dmtest run --profile stec --suite thin-provisioning -n /discard_fully_provisioned_device/
I retested after applying these linux-block.git commits ontop of
3.14-rc1:
5cb8850c9c4a block: Explicitly handle discard/write same segments
8423ae3d7a3c block: Fix cloning of discard/write same bios
And got this:
request botched: dev skd0: type=1, flags=12248081
sector 8390784, nr/cnr 0/128
bio ffff88033169cba0, biotail ffff88032e42bb60, buffer (null), len 0
------------[ cut here ]------------
kernel BUG at block/blk-core.c:2693!
invalid opcode: 0000 [#1] SMP
Modules linked in: dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio libcrc32c ebtable_nat ebtables xt_CHECKSUM iptable_mangle bridge autofs4 target_core_iblock t
arget_core_file target_core_pscsi target_core_mod configfs bnx2fc fcoe libfcoe 8021q libfc garp stp llc scsi_transport_fc scsi_tgt sunrpc cpufreq_ondemand ipt_REJECT n
f_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables bnx2i cnic uio i
pv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_mirror dm_region_hash dm_log vhost_net macvtap macvlan vhost tun kvm_int
el kvm iTCO_wdt iTCO_vendor_support microcode i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core i7core_edac edac_core ixgbe dca ptp pps_core mdio ses enclosure sg ac
pi_cpufreq dm_mod ext4 jbd2 mbcache sr_mod cdrom pata_acpi ata_generic ata_piix skd sd_mod crc_t10dif crct10dif_common megaraid_sas
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W 3.14.0-rc1.snitm+ #5
Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.10.2619.N1 05/24/2011
task: ffff88033299e150 ti: ffff8803329a4000 task.ti: ffff8803329a4000
RIP: 0010:[<ffffffff81252f1a>] [<ffffffff81252f1a>] __blk_end_request_all+0x2a/0x40
RSP: 0018:ffff88033fc43cf8 EFLAGS: 00010002
RAX: 0000000000000001 RBX: ffff88032e636ac8 RCX: 0000000000000006
RDX: 0000000000000001 RSI: ffff88033169cba0 RDI: ffff88032ec755c0
RBP: ffff88033fc43cf8 R08: 0000000000000002 R09: 0000000000000000
R10: 00000000000006f3 R11: 0000000000000001 R12: 0000000000000000
R13: ffff88033195faa8 R14: ffff8800ba396000 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff88033fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003bfea13000 CR3: 000000032fbdc000 CR4: 00000000000007e0
Stack:
ffff88033fc43d58 ffffffffa0037b85 ffff88033fc43d48 ffffffff8129ca09
ffff88033fc43d28 ffff88032e636ac8 ffff8800ba396000 ffff88032e650080
ffff8800ba396000 ffff88032e650080 ffff88032e636ac8 0000000000003c17
Call Trace:
<IRQ>
[<ffffffffa0037b85>] skd_end_request+0x55/0x160 [skd]
[<ffffffff8129ca09>] ? swiotlb_unmap_sg_attrs+0x69/0x80
[<ffffffffa003c513>] skd_isr_completion_posted+0x1e3/0x5d0 [skd]
[<ffffffff810930a3>] ? __wake_up+0x53/0x70
[<ffffffffa003d1b2>] skd_isr+0x122/0x280 [skd]
[<ffffffff810a73ed>] handle_irq_event_percpu+0x6d/0x200
[<ffffffff810a75c2>] handle_irq_event+0x42/0x70
[<ffffffff810aad19>] handle_edge_irq+0x69/0x120
[<ffffffff81005aec>] handle_irq+0x5c/0x150
[<ffffffff815470f2>] ? __atomic_notifier_call_chain+0x12/0x20
[<ffffffff81547116>] ? atomic_notifier_call_chain+0x16/0x20
[<ffffffff8154d91e>] do_IRQ+0x5e/0x110
[<ffffffff8154366a>] common_interrupt+0x6a/0x6a
<EOI>
[<ffffffff8144d5e3>] ? cpuidle_enter_state+0x53/0xd0
[<ffffffff8144d5df>] ? cpuidle_enter_state+0x4f/0xd0
[<ffffffff8144d7a7>] cpuidle_idle_call+0xc7/0x160
[<ffffffff8100cf5e>] arch_cpu_idle+0xe/0x30
[<ffffffff810a696a>] cpu_idle_loop+0x9a/0x240
[<ffffffff810b9e64>] ? clockevents_register_device+0xc4/0x130
[<ffffffff810a6b33>] cpu_startup_entry+0x23/0x30
[<ffffffff81032d5a>] start_secondary+0x7a/0x80
Code: 00 55 48 89 e5 66 66 66 66 90 48 8b 87 78 01 00 00 48 85 c0 75 10 31 c9 8b 57 64 e8 91 ff ff ff 84 c0 75 07 c9 c3 8b 48 64 eb ed <0f> 0b 0f 1f 40 00 eb fa 66 66
66 66 66 2e 0f 1f 84 00 00 00 00
RIP [<ffffffff81252f1a>] __blk_end_request_all+0x2a/0x40
RSP <ffff88033fc43cf8>
---[ end trace 494de22d0f0be0f8 ]---
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.394 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.402 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.405 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.410 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.414 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.417 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.421 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.424 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.428 msecs
INFO: NMI handler (ghes_notify_nmi) took too long to run: 2.431 msecs
Kernel panic - not syncing: Fatal exception in interrupt
Shutting down cpus with NMI
Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)