LinuxLists.cc - perf: fuzzer KASAN unwind_get_return

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
>
> Running on my haswell machine with the imc/uncore patch applied, the
> perf_fuzzer next tripped over this issue.
>
> [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
> [ 202.034496] ==================================================================
> [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
> [ 202.058826] Read of size 8 by task perf_fuzzer/16254
> [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
> [ 202.073068] flags: 0x1ffff8000000400(reserved)
> [ 202.077885] page dumped because: kasan: bad access detected
> [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
> [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
> [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
> [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
> [ 202.125339] Call Trace:
> [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
> [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
> [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
> [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
> [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
> [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
> [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80

Josh, any ideas?

> [ 202.172817] [<ffffffff8100b08d>] perf_callchain_kernel+0x22d/0x270
> [ 202.179590] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80
> [ 202.185548] [<ffffffff8100ae60>] ? arch_perf_update_userpage+0x130/0x130
> [ 202.192849] [<ffffffff81252aaa>] get_perf_callchain+0x24a/0x3e0
> [ 202.199339] [<ffffffff81252860>] ? put_callchain_buffers+0x50/0x50
> [ 202.206092] [<ffffffff81095b17>] ? perf_get_regs_user+0x327/0x380
> [ 202.212751] [<ffffffff81135fd0>] ? lock_release+0x30/0x540
> [ 202.218803] [<ffffffff81252d05>] perf_callchain+0xc5/0xe0
> [ 202.224767] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80
> [ 202.230696] [<ffffffff8124dbf9>] perf_prepare_sample+0x489/0x630
> [ 202.237275] [<ffffffff81135fd0>] ? lock_release+0x30/0x540
> [ 202.243266] [<ffffffff8124de9c>] ? perf_event_output_forward+0xfc/0x130
> [ 202.250472] [<ffffffff8124dda0>] ? perf_prepare_sample+0x630/0x630
> [ 202.257251] [<ffffffff8124e0ae>] perf_event_output+0xae/0x130
> [ 202.263564] [<ffffffff8124e000>] ? perf_event_output_backward+0x130/0x130
> [ 202.270964] [<ffffffff8124e000>] ? perf_event_output_backward+0x130/0x130
> [ 202.278373] [<ffffffff81247cc2>] ? perf_event_update_userpage+0x212/0x2b0
> [ 202.285772] [<ffffffff81247ab0>] ? perf_event_task_disable+0xc0/0xc0
> [ 202.292744] [<ffffffff812fac4f>] ? __asan_loadN+0xf/0x20
> [ 202.298581] [<ffffffff8101757d>] ? setup_pebs_sample_data+0x68d/0x830
> [ 202.305622] [<ffffffff81017a91>] __intel_pmu_pebs_event+0x221/0x3a0
> [ 202.312469] [<ffffffff81135e4d>] ? lock_acquire+0x3d/0x190
> [ 202.318523] [<ffffffff81017870>] ? pebs_update_state+0x150/0x150
> [ 202.325060] [<ffffffff8104c6ec>] ? get_stack_info+0x3c/0x150
> [ 202.331259] [<ffffffff810106b7>] ? __intel_pmu_enable_all+0x77/0xf0
> [ 202.338128] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80
> [ 202.344059] [<ffffffff81018b50>] ? intel_pmu_disable_bts+0x60/0x60
> [ 202.350823] [<ffffffff812fa7c4>] ? __asan_load4+0x24/0x80
> [ 202.356740] [<ffffffff81252d05>] ? perf_callchain+0xc5/0xe0
> [ 202.362855] [<ffffffff81135fd0>] ? lock_release+0x30/0x540
> [ 202.368855] [<ffffffff8124dc31>] ? perf_prepare_sample+0x4c1/0x630
> [ 202.375619] [<ffffffff8124de84>] ? perf_event_output_forward+0xe4/0x130
> [ 202.382849] [<ffffffff81017ffc>] intel_pmu_drain_pebs_nhm+0x3ec/0x530
> [ 202.389899] [<ffffffff81017c10>] ? __intel_pmu_pebs_event+0x3a0/0x3a0
> [ 202.396959] [<ffffffff81247caa>] ? perf_event_update_userpage+0x1fa/0x2b0
> [ 202.406800] [<ffffffff81247cc2>] ? perf_event_update_userpage+0x212/0x2b0
> [ 202.416486] [<ffffffff81247ab0>] ? perf_event_task_disable+0xc0/0xc0
> [ 202.425720] [<ffffffff8101a832>] ? intel_pmu_lbr_read+0x32/0x790
> [ 202.434566] [<ffffffff8123ba26>] ? __perf_event_overflow+0x116/0x280
> [ 202.443735] [<ffffffff810144d8>] ? intel_bts_interrupt+0x88/0x1b0
> [ 202.452538] [<ffffffff81012c7e>] intel_pmu_handle_irq+0x3ae/0x690
> [ 202.461407] [<ffffffff810128d0>] ? intel_pmu_save_and_restart+0x80/0x80
> [ 202.470877] [<ffffffff81135fd0>] ? lock_release+0x30/0x540
> [ 202.479131] [<ffffffff81088eeb>] ? native_apic_msr_write+0x2b/0x30
> [ 202.488181] [<ffffffff8108899c>] ? x2apic_send_IPI_self+0x3c/0x50
> [ 202.497066] [<ffffffff81055d72>] ? native_sched_clock+0x62/0x140
> [ 202.505919] [<ffffffff810081fd>] perf_event_nmi_handler+0x2d/0x50
> [ 202.514832] [<ffffffff8104da91>] nmi_handle+0xb1/0x1d0
> [ 202.522697] [<ffffffff8104d9e5>] ? nmi_handle+0x5/0x1d0
> [ 202.530610] [<ffffffff8104e185>] default_do_nmi+0xe5/0x140
> [ 202.538765] [<ffffffff8104e332>] do_nmi+0x152/0x1b0
> [ 202.546254] [<ffffffff81b8f171>] end_repeat_nmi+0x1a/0x1e
> [ 202.554257] [<ffffffff810106b7>] ? __intel_pmu_enable_all+0x77/0xf0
> [ 202.563167] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0
> [ 202.572060] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0
> [ 202.580864] [<ffffffff812475eb>] ? perf_event_task_tick+0x48b/0x5f0
> [ 202.589703] <EOE> <IRQ> [<ffffffff81101571>] scheduler_tick+0xb1/0x150
> [ 202.598985] [<ffffffff8116e7e7>] update_process_times+0x47/0x60
> [ 202.607433] [<ffffffff81185e53>] tick_sched_handle.isra.14+0x33/0x80
> [ 202.616314] [<ffffffff811869cb>] tick_sched_timer+0x4b/0x90
> [ 202.624322] [<ffffffff8116fbfe>] __hrtimer_run_queues+0x21e/0x540
> [ 202.632864] [<ffffffff81186980>] ? tick_sched_do_timer+0x50/0x50
> [ 202.641337] [<ffffffff8116f9e0>] ? retrigger_next_event+0xa0/0xa0
> [ 202.649947] [<ffffffff8117b8f6>] ? ktime_get_update_offsets_now+0xe6/0x190
> [ 202.659411] [<ffffffff811707f0>] ? hrtimer_interrupt+0xb0/0x220
> [ 202.667864] [<ffffffff8117082f>] hrtimer_interrupt+0xef/0x220
> [ 202.676069] [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0
> [ 202.684444] [<ffffffff8107ec2f>] local_apic_timer_interrupt+0x4f/0x80
> [ 202.693422] [<ffffffff81b903d7>] smp_apic_timer_interrupt+0x57/0x70
> [ 202.702203] [<ffffffff81b8f6a2>] apic_timer_interrupt+0x82/0x90
> [ 202.710591] <EOI> [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0
> [ 202.719609] [<ffffffff8118dc3a>] ? smp_call_function_single+0x14a/0x1b0
> [ 202.728811] [<ffffffff8118dc30>] ? smp_call_function_single+0x140/0x1b0
> [ 202.738039] [<ffffffff8118daf0>] ? generic_exec_single+0x170/0x170
> [ 202.746727] [<ffffffff8123b020>] ? perf_cgroup_attach+0xb0/0xb0
> [ 202.755181] [<ffffffff81238e48>] event_function_call+0x268/0x270
> [ 202.763687] [<ffffffff812426d0>] ? task_ctx_sched_out+0x60/0x60
> [ 202.772057] [<ffffffff81238be0>] ? task_function_call+0xc0/0xc0
> [ 202.780404] [<ffffffff812426d0>] ? task_ctx_sched_out+0x60/0x60
> [ 202.788768] [<ffffffff81238e79>] ? _perf_event_disable+0x29/0x70
> [ 202.797258] [<ffffffff812383d0>] ? update_group_times+0x50/0x50
> [ 202.805667] [<ffffffff81238e97>] ? _perf_event_disable+0x47/0x70
> [ 202.814188] [<ffffffff8113a4d7>] ? do_raw_spin_unlock+0x97/0x130
> [ 202.822733] [<ffffffff81238e50>] ? event_function_call+0x270/0x270
> [ 202.831462] [<ffffffff81238ea8>] _perf_event_disable+0x58/0x70
> [ 202.839778] [<ffffffff812386a3>] perf_event_for_each_child+0x53/0xd0
> [ 202.848576] [<ffffffff81247a51>] perf_event_task_disable+0x61/0xc0
> [ 202.857303] [<ffffffff810daee2>] SyS_prctl+0x3f2/0x690
> [ 202.864853] [<ffffffff810daaf0>] ? SyS_umask+0x40/0x40
> [ 202.872375] [<ffffffff81136c6a>] ? lockdep_sys_exit+0x1a/0xa0
> [ 202.880517] [<ffffffff81004016>] ? lockdep_sys_exit_thunk+0x16/0x30
> [ 202.889310] [<ffffffff81b8dabb>] entry_SYSCALL_64_fastpath+0x1e/0xb2
> [ 202.898177] Memory state around the buggy address:
> [ 202.905288] ffff8800cff0bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [ 202.915044] ffff8800cff0bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [ 202.924697] >ffff8800cff0bd80: f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00
> [ 202.934420] ^
> [ 202.940352] ffff8800cff0be00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [ 202.950141] ffff8800cff0be80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> [ 202.959835] ==================================================================
>

2016-11-15 19:05:09

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Tue, Nov 15, 2016 at 10:57 AM, Peter Zijlstra <[email protected]> wrote:
> On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
>>
>> Running on my haswell machine with the imc/uncore patch applied, the
>> perf_fuzzer next tripped over this issue.
>>
>> [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
>> [ 202.034496] ==================================================================
>> [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
>> [ 202.058826] Read of size 8 by task perf_fuzzer/16254
>> [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
>> [ 202.073068] flags: 0x1ffff8000000400(reserved)
>> [ 202.077885] page dumped because: kasan: bad access detected
>> [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
>> [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
>> [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
>> [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
>> [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
>> [ 202.125339] Call Trace:
>> [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
>> [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
>> [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
>> [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
>> [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
>> [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
>> [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80
>
> Josh, any ideas?

I think this is a false positive due to imprecise unwind that hits a
stack redzone.
We probably need to use READ_ONCE_NOCHECK as in get_wchan.

2016-11-15 19:05:54

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Tue, 15 Nov 2016, Peter Zijlstra wrote:

> On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
> >
> > Running on my haswell machine with the imc/uncore patch applied, the
> > perf_fuzzer next tripped over this issue.
> >
> > [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
> > [ 202.034496] ==================================================================
> > [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
> > [ 202.058826] Read of size 8 by task perf_fuzzer/16254
> > [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
> > [ 202.073068] flags: 0x1ffff8000000400(reserved)
> > [ 202.077885] page dumped because: kasan: bad access detected
> > [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
> > [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> > [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
> > [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
> > [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
> > [ 202.125339] Call Trace:
> > [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
> > [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
> > [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
> > [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
> > [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
> > [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
> > [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80
>
> Josh, any ideas?

>From what I can tell this maps to:

unsigned long unwind_get_return_address(struct unwind_state *state)
{
unsigned long addr;
unsigned long *addr_p = unwind_get_return_address_ptr(state);

if (unwind_done(state))
return 0;

>> addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, *addr_p,
addr_p);

return __kernel_text_address(addr) ? addr : 0;
}

2016-11-15 20:56:17

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Tue, Nov 15, 2016 at 11:04:45AM -0800, Dmitry Vyukov wrote:
> On Tue, Nov 15, 2016 at 10:57 AM, Peter Zijlstra <[email protected]> wrote:
> > On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
> >>
> >> Running on my haswell machine with the imc/uncore patch applied, the
> >> perf_fuzzer next tripped over this issue.
> >>
> >> [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
> >> [ 202.034496] ==================================================================
> >> [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
> >> [ 202.058826] Read of size 8 by task perf_fuzzer/16254
> >> [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
> >> [ 202.073068] flags: 0x1ffff8000000400(reserved)
> >> [ 202.077885] page dumped because: kasan: bad access detected
> >> [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
> >> [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> >> [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
> >> [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
> >> [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
> >> [ 202.125339] Call Trace:
> >> [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
> >> [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
> >> [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
> >> [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
> >> [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
> >> [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
> >> [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80
> >
> > Josh, any ideas?
>
> I think this is a false positive due to imprecise unwind that hits a
> stack redzone.
> We probably need to use READ_ONCE_NOCHECK as in get_wchan.

I'm not so sure about that. The unwind should be precise here: it
should only be looking at the frame pointers and return addresses on the
current task's stack. I can't see any reason why it would be reading
into the KASAN stack redzone.

--
Josh

2016-11-15 20:57:52

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Tue, Nov 15, 2016 at 02:05:50PM -0500, Vince Weaver wrote:
> On Tue, 15 Nov 2016, Peter Zijlstra wrote:
>
> > On Tue, Nov 15, 2016 at 12:43:56PM -0500, Vince Weaver wrote:
> > >
> > > Running on my haswell machine with the imc/uncore patch applied, the
> > > perf_fuzzer next tripped over this issue.
> > >
> > > [ 202.034495] BAD LUCK: lost 371 message(s) from NMI context!
> > > [ 202.034496] ==================================================================
> > > [ 202.048327] BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x35/0x80 at addr ffff8800cff0bd90
> > > [ 202.058826] Read of size 8 by task perf_fuzzer/16254
> > > [ 202.064186] page:ffffea00033fc2c0 count:1 mapcount:0 mapping: (null) index:0x0^Ac
> > > [ 202.073068] flags: 0x1ffff8000000400(reserved)
> > > [ 202.077885] page dumped because: kasan: bad access detected
> > > [ 202.083880] CPU: 4 PID: 16254 Comm: perf_fuzzer Not tainted 4.9.0-rc5+ #5
> > > [ 202.091204] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> > > [ 202.099181] ffff8800cff0b1d8^Ac ffffffff816bb796^Ac ffff8800cff0b270^Ac ffff8800cff0bd90^Ac
> > > [ 202.107896] ffff8800cff0b260^Ac ffffffff812fbe95^Ac 00007ffc9d1ab480^Ac 0000000000000000^Ac
> > > [ 202.116638] ffffffff8125117d^Ac 0000000000000092^Ac 0000000000000000^Ac ffff8800cff0b7c0^Ac
> > > [ 202.125339] Call Trace:
> > > [ 202.127994] <NMI> [<ffffffff816bb796>] dump_stack+0x63/0x8d
> > > [ 202.134184] [<ffffffff812fbe95>] kasan_report_error+0x495/0x4c0
> > > [ 202.140680] [<ffffffff8125117d>] ? perf_output_begin+0x28d/0x4c0
> > > [ 202.147228] [<ffffffff812fc319>] kasan_report+0x39/0x40
> > > [ 202.152987] [<ffffffff81095ce5>] ? unwind_get_return_address+0x35/0x80
> > > [ 202.160094] [<ffffffff812fa8fe>] __asan_load8+0x5e/0x70
> > > [ 202.165859] [<ffffffff81095ce5>] unwind_get_return_address+0x35/0x80
> >
> > Josh, any ideas?
>
> From what I can tell this maps to:
>
> unsigned long unwind_get_return_address(struct unwind_state *state)
> {
> unsigned long addr;
> unsigned long *addr_p = unwind_get_return_address_ptr(state);
>
> if (unwind_done(state))
> return 0;
>
> >> addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, *addr_p,
> addr_p);
>
> return __kernel_text_address(addr) ? addr : 0;
> }

Hi Vince,

Would you mind posting a disassembly of unwind_get_return_address()?

Any idea how recreatable it is? (In particular I'd be interested in
seeing this dump with the latest unwinder improvements in the -tip tree,
which dump the pt_regs associated with an interrupt.)

--
Josh

2016-11-16 13:03:42

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
> Would you mind posting a disassembly of unwind_get_return_address()?

$ objdump -D ivb-dbg/vmlinux | awk '/<[^>]*>:/ { p=0; } /<unwind_get_return_address>:/ { p=1; } { if (p) print $0; }'

ffffffff811afd10 <unwind_get_return_address>:
ffffffff811afd10: e8 eb cc f4 01 callq ffffffff830fca00 <__fentry__>
ffffffff811afd15: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd1c: fc ff df
ffffffff811afd1f: 48 89 fa mov %rdi,%rdx
ffffffff811afd22: 53 push %rbx
ffffffff811afd23: 48 89 fb mov %rdi,%rbx
ffffffff811afd26: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811afd2a: 48 83 ec 18 sub $0x18,%rsp
ffffffff811afd2e: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
ffffffff811afd32: 48 89 f8 mov %rdi,%rax
ffffffff811afd35: 83 e0 07 and $0x7,%eax
ffffffff811afd38: 83 c0 03 add $0x3,%eax
ffffffff811afd3b: 38 d0 cmp %dl,%al
ffffffff811afd3d: 7c 04 jl ffffffff811afd43 <unwind_get_return_address+0x33>
ffffffff811afd3f: 84 d2 test %dl,%dl
ffffffff811afd41: 75 75 jne ffffffff811afdb8 <unwind_get_return_address+0xa8>
ffffffff811afd43: 8b 03 mov (%rbx),%eax
ffffffff811afd45: 85 c0 test %eax,%eax
ffffffff811afd47: 75 08 jne ffffffff811afd51 <unwind_get_return_address+0x41>
ffffffff811afd49: 48 83 c4 18 add $0x18,%rsp
ffffffff811afd4d: 31 c0 xor %eax,%eax
ffffffff811afd4f: 5b pop %rbx
ffffffff811afd50: c3 retq
ffffffff811afd51: 48 8d 7b 38 lea 0x38(%rbx),%rdi
ffffffff811afd55: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd5c: fc ff df
ffffffff811afd5f: 48 89 fa mov %rdi,%rdx
ffffffff811afd62: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811afd66: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811afd6a: 75 53 jne ffffffff811afdbf <unwind_get_return_address+0xaf>
ffffffff811afd6c: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd73: fc ff df
ffffffff811afd76: 48 8b 4b 38 mov 0x38(%rbx),%rcx
ffffffff811afd7a: 48 89 ca mov %rcx,%rdx
ffffffff811afd7d: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811afd81: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811afd85: 75 3f jne ffffffff811afdc6 <unwind_get_return_address+0xb6>
ffffffff811afd87: 48 8d 7b 28 lea 0x28(%rbx),%rdi
ffffffff811afd8b: 48 8b 11 mov (%rcx),%rdx
ffffffff811afd8e: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811afd95: fc ff df
ffffffff811afd98: 48 8d 73 30 lea 0x30(%rbx),%rsi
ffffffff811afd9c: 49 89 f8 mov %rdi,%r8
ffffffff811afd9f: 49 c1 e8 03 shr $0x3,%r8
ffffffff811afda3: 41 80 3c 00 00 cmpb $0x0,(%r8,%rax,1)
ffffffff811afda8: 75 2e jne ffffffff811afdd8 <unwind_get_return_address+0xc8>
ffffffff811afdaa: 48 8b 7b 28 mov 0x28(%rbx),%rdi
ffffffff811afdae: 48 83 c4 18 add $0x18,%rsp
ffffffff811afdb2: 5b pop %rbx
ffffffff811afdb3: e9 08 98 2a 00 jmpq ffffffff814595c0 <ftrace_graph_ret_addr>
ffffffff811afdb8: e8 53 7d 42 00 callq ffffffff815d7b10 <__asan_report_load4_noabort>
ffffffff811afdbd: eb 84 jmp ffffffff811afd43 <unwind_get_return_address+0x33>
ffffffff811afdbf: e8 9c 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
ffffffff811afdc4: eb a6 jmp ffffffff811afd6c <unwind_get_return_address+0x5c>
ffffffff811afdc6: 48 89 cf mov %rcx,%rdi
ffffffff811afdc9: 48 89 0c 24 mov %rcx,(%rsp)
ffffffff811afdcd: e8 8e 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
ffffffff811afdd2: 48 8b 0c 24 mov (%rsp),%rcx
ffffffff811afdd6: eb af jmp ffffffff811afd87 <unwind_get_return_address+0x77>
ffffffff811afdd8: 48 89 74 24 10 mov %rsi,0x10(%rsp)
ffffffff811afddd: 48 89 54 24 08 mov %rdx,0x8(%rsp)
ffffffff811afde2: 48 89 0c 24 mov %rcx,(%rsp)
ffffffff811afde6: e8 75 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
ffffffff811afdeb: 48 8b 74 24 10 mov 0x10(%rsp),%rsi
ffffffff811afdf0: 48 8b 54 24 08 mov 0x8(%rsp),%rdx
ffffffff811afdf5: 48 8b 0c 24 mov (%rsp),%rcx
ffffffff811afdf9: eb af jmp ffffffff811afdaa <unwind_get_return_address+0x9a>
ffffffff811afdfb: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

> Any idea how recreatable it is? (In particular I'd be interested in
> seeing this dump with the latest unwinder improvements in the -tip tree,
> which dump the pt_regs associated with an interrupt.)

Fairly reproducable it seems, doesn't seem to include pt_regs dumps
though :/

tip/master as of this morning.

3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff88042fc87be0
3Read of size 8 by task swapper/28/0
0page:ffffea0010bf21c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d ? dump_stack+0x5e/0x89
d ? kasan_report_error+0x4a5/0x4d0
d ? __asan_report_load8_noabort+0x45/0x50
d ? __kernel_text_address+0x20/0xa0
d ? unwind_next_frame+0x1ba/0x1f0
d ? unwind_next_frame+0x1ba/0x1f0
d ? perf_callchain_kernel+0x33c/0x540
d ? arch_perf_update_userpage+0x340/0x340
d ? get_perf_callchain+0x24d/0x610
d ? put_callchain_buffers+0x50/0x50
d ? number+0x653/0x830
d ? perf_callchain+0x126/0x190
d ? perf_prepare_sample+0x720/0x1010
d ? perf_event_output_forward+0x81/0xf0
d ? perf_prepare_sample+0x1010/0x1010
d ? pointer+0x880/0x880
d ? perf_event_update_userpage+0x16/0x730
d ? __perf_event_overflow+0x1a0/0x510
d ? intel_pmu_handle_irq+0x34b/0xa90
d ? intel_pmu_save_and_restart+0xd0/0xd0
d ? acpi_os_read_memory+0x205/0x23c
d ? format_decode+0xc5/0x7a0
d ? vunmap_page_range+0x26a/0x400
d ? ghes_copy_tofrom_phys+0x141/0x270
d ? ghes_read_estatus+0x112/0x5a0
d ? ghes_copy_tofrom_phys+0x270/0x270
d ? early_printk+0xa4/0xd0
d ? devkmsg_sysctl_set_loglvl+0x160/0x160
d ? perf_event_nmi_handler+0x28/0x40
d ? nmi_handle+0xa1/0x250
d ? default_do_nmi+0x61/0x170
d ? do_nmi+0x191/0x200
d ? end_repeat_nmi+0x1a/0x1e
d ? format_decode+0xc5/0x7a0
d ? format_decode+0xc5/0x7a0
d ? format_decode+0xc5/0x7a0
d <EOE>
d <IRQ>
d ? vsnprintf+0xfc/0x15e0
d ? pointer+0x880/0x880
d ? x86_pmu_enable_all+0x1c0/0x1c0
d ? vscnprintf+0x9/0x30
d ? early_vprintk+0xb0/0x130
d ? trace_raw_output_console+0x160/0x160
d ? memcpy+0x34/0x50
d ? x86_pmu_commit_txn+0x180/0x260
d ? events_sysfs_show+0xb0/0xb0
d ? save_stack+0x33/0xb0
d ? hrtimer_init+0x120/0x120
d ? timerqueue_del+0x62/0x140
d ? perf_event_update_userpage+0x16/0x730
d ? perf_event_update_userpage+0x16/0x730
d ? x86_perf_event_set_period+0x239/0x450
d ? perf_event_update_userpage+0x16/0x730
d ? x86_pmu_enable+0x5f7/0xaa0
d ? printk+0xb6/0xef
d ? printk_emit+0xa0/0xa0
d ? _raw_spin_unlock_irqrestore+0x42/0x70
d ? ___ratelimit+0x1e4/0x3f0
d ? irq_work_run_list+0xa1/0xf0
d ? irq_work_run+0x14/0x40
d ? smp_call_function_single_interrupt+0x60/0x80
d ? call_function_single_interrupt+0x89/0x90
d <EOI>
d ? cpuidle_enter_state+0x113/0x780
d ? cpuidle_enter_state+0x10e/0x780
d ? cpu_load_update_nohz_stop+0x155/0x1b0
d ? cpu_startup_entry+0x19a/0x2c0
d ? start_cpu+0x5/0x14
3Memory state around the buggy address:
3 ffff88042fc87a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042fc87b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88042fc87b80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
3 ^
3 ffff88042fc87c00: 00 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00
3 ffff88042fc87c80: 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00
3==================================================================

3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff880424a97878
3Read of size 8 by task perf_fuzzer/3451
0page:ffffea001092a5c0 count:0 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000000()
1page dumped because: kasan: bad access detected
dCPU: 28 PID: 3451 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d ? dump_stack+0x5e/0x89
d ? kasan_report_error+0x4a5/0x4d0
d ? __asan_report_load8_noabort+0x45/0x50
d ? __kernel_text_address+0x20/0xa0
d ? unwind_next_frame+0x1ba/0x1f0
d ? unwind_next_frame+0x1ba/0x1f0
d ? perf_callchain_kernel+0x33c/0x540
d ? arch_perf_update_userpage+0x340/0x340
d ? get_perf_callchain+0x24d/0x610
d ? put_callchain_buffers+0x50/0x50
d ? ipv6_flowlabel_opt+0x1111/0x17d0
d ? perf_log_itrace_start+0x3a0/0x3a0
d ? cpumask_next_and+0x5a/0xa0
d ? ktime_get_raw_fast_ns+0xd3/0x1e0
d ? perf_callchain+0x126/0x190
d ? perf_prepare_sample+0x720/0x1010
d ? perf_event_output_forward+0x81/0xf0
d ? perf_prepare_sample+0x1010/0x1010
d ? perf_event_update_userpage+0x16/0x730
d ? kasan_unpoison_shadow+0x31/0x40
d ? get_page_from_freelist+0x52e/0x2310
d ? perf_output_begin+0x3a1/0x9b0
d ? cpu_clock_event_add+0x17/0x20
d ? __perf_event_overflow+0x1a0/0x510
d ? perf_swevent_overflow+0x156/0x1f0
d ? perf_tp_event+0x3e8/0x5c0
d ? perf_output_begin_backward+0x960/0x960
d ? perf_tp_event_match.isra.85.part.86+0x140/0x140
d ? __mark_inode_dirty+0x459/0xa50
d ? legitimize_path.isra.28+0x6b/0x150
d ? unlazy_walk+0x456/0x790
d ? memset+0x1f/0x40
d ? perf_trace_writeback_dirty_inode_template+0x3af/0x610
d ? save_stack+0x33/0xb0
d ? inode_congested+0x450/0x450
d ? dput+0x1de/0x530
d ? walk_component+0x2cc/0xdc0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? pick_link+0xbe0/0xbe0
d ? inode_congested+0x450/0x450
d ? __mark_inode_dirty+0x459/0xa50
d ? proc_sys_setattr+0x84/0xb0
d ? notify_change+0x4d6/0xc40
d ? security_inode_need_killpriv+0x58/0x80
d ? do_truncate+0xd7/0x160
d ? file_open_root+0x1a0/0x1a0
d ? path_openat+0x97f/0x3b30
d ? vfs_rename+0x14a0/0x14a0
d ? getname_flags+0xba/0x500
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? do_filp_open+0x175/0x230
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? may_open_dev+0xc0/0xc0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? save_stack+0x33/0xb0
d ? do_sys_open+0x16d/0x310
d ? SyS_write+0xab/0x160
d ? filp_open+0x50/0x50
d ? task_stopped_code+0xf0/0xf0
d ? trace_hardirqs_on_thunk+0x1a/0x1c
d ? entry_SYSCALL_64_fastpath+0x18/0xa8
3Memory state around the buggy address:
3 ffff880424a97700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff880424a97780: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3
3>ffff880424a97800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
3 ^
3 ffff880424a97880: f1 f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f3
3 ffff880424a97900: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================

2016-11-16 13:18:24

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 2:03 PM, Peter Zijlstra <[email protected]> wrote:
> On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
>> Would you mind posting a disassembly of unwind_get_return_address()?
>
> $ objdump -D ivb-dbg/vmlinux | awk '/<[^>]*>:/ { p=0; } /<unwind_get_return_address>:/ { p=1; } { if (p) print $0; }'
>
> ffffffff811afd10 <unwind_get_return_address>:
> ffffffff811afd10: e8 eb cc f4 01 callq ffffffff830fca00 <__fentry__>
> ffffffff811afd15: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd1c: fc ff df
> ffffffff811afd1f: 48 89 fa mov %rdi,%rdx
> ffffffff811afd22: 53 push %rbx
> ffffffff811afd23: 48 89 fb mov %rdi,%rbx
> ffffffff811afd26: 48 c1 ea 03 shr $0x3,%rdx
> ffffffff811afd2a: 48 83 ec 18 sub $0x18,%rsp
> ffffffff811afd2e: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
> ffffffff811afd32: 48 89 f8 mov %rdi,%rax
> ffffffff811afd35: 83 e0 07 and $0x7,%eax
> ffffffff811afd38: 83 c0 03 add $0x3,%eax
> ffffffff811afd3b: 38 d0 cmp %dl,%al
> ffffffff811afd3d: 7c 04 jl ffffffff811afd43 <unwind_get_return_address+0x33>
> ffffffff811afd3f: 84 d2 test %dl,%dl
> ffffffff811afd41: 75 75 jne ffffffff811afdb8 <unwind_get_return_address+0xa8>
> ffffffff811afd43: 8b 03 mov (%rbx),%eax
> ffffffff811afd45: 85 c0 test %eax,%eax
> ffffffff811afd47: 75 08 jne ffffffff811afd51 <unwind_get_return_address+0x41>
> ffffffff811afd49: 48 83 c4 18 add $0x18,%rsp
> ffffffff811afd4d: 31 c0 xor %eax,%eax
> ffffffff811afd4f: 5b pop %rbx
> ffffffff811afd50: c3 retq
> ffffffff811afd51: 48 8d 7b 38 lea 0x38(%rbx),%rdi
> ffffffff811afd55: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd5c: fc ff df
> ffffffff811afd5f: 48 89 fa mov %rdi,%rdx
> ffffffff811afd62: 48 c1 ea 03 shr $0x3,%rdx
> ffffffff811afd66: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
> ffffffff811afd6a: 75 53 jne ffffffff811afdbf <unwind_get_return_address+0xaf>
> ffffffff811afd6c: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd73: fc ff df
> ffffffff811afd76: 48 8b 4b 38 mov 0x38(%rbx),%rcx
> ffffffff811afd7a: 48 89 ca mov %rcx,%rdx
> ffffffff811afd7d: 48 c1 ea 03 shr $0x3,%rdx
> ffffffff811afd81: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
> ffffffff811afd85: 75 3f jne ffffffff811afdc6 <unwind_get_return_address+0xb6>
> ffffffff811afd87: 48 8d 7b 28 lea 0x28(%rbx),%rdi
> ffffffff811afd8b: 48 8b 11 mov (%rcx),%rdx
> ffffffff811afd8e: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
> ffffffff811afd95: fc ff df
> ffffffff811afd98: 48 8d 73 30 lea 0x30(%rbx),%rsi
> ffffffff811afd9c: 49 89 f8 mov %rdi,%r8
> ffffffff811afd9f: 49 c1 e8 03 shr $0x3,%r8
> ffffffff811afda3: 41 80 3c 00 00 cmpb $0x0,(%r8,%rax,1)
> ffffffff811afda8: 75 2e jne ffffffff811afdd8 <unwind_get_return_address+0xc8>
> ffffffff811afdaa: 48 8b 7b 28 mov 0x28(%rbx),%rdi
> ffffffff811afdae: 48 83 c4 18 add $0x18,%rsp
> ffffffff811afdb2: 5b pop %rbx
> ffffffff811afdb3: e9 08 98 2a 00 jmpq ffffffff814595c0 <ftrace_graph_ret_addr>
> ffffffff811afdb8: e8 53 7d 42 00 callq ffffffff815d7b10 <__asan_report_load4_noabort>
> ffffffff811afdbd: eb 84 jmp ffffffff811afd43 <unwind_get_return_address+0x33>
> ffffffff811afdbf: e8 9c 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
> ffffffff811afdc4: eb a6 jmp ffffffff811afd6c <unwind_get_return_address+0x5c>
> ffffffff811afdc6: 48 89 cf mov %rcx,%rdi
> ffffffff811afdc9: 48 89 0c 24 mov %rcx,(%rsp)
> ffffffff811afdcd: e8 8e 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
> ffffffff811afdd2: 48 8b 0c 24 mov (%rsp),%rcx
> ffffffff811afdd6: eb af jmp ffffffff811afd87 <unwind_get_return_address+0x77>
> ffffffff811afdd8: 48 89 74 24 10 mov %rsi,0x10(%rsp)
> ffffffff811afddd: 48 89 54 24 08 mov %rdx,0x8(%rsp)
> ffffffff811afde2: 48 89 0c 24 mov %rcx,(%rsp)
> ffffffff811afde6: e8 75 7d 42 00 callq ffffffff815d7b60 <__asan_report_load8_noabort>
> ffffffff811afdeb: 48 8b 74 24 10 mov 0x10(%rsp),%rsi
> ffffffff811afdf0: 48 8b 54 24 08 mov 0x8(%rsp),%rdx
> ffffffff811afdf5: 48 8b 0c 24 mov (%rsp),%rcx
> ffffffff811afdf9: eb af jmp ffffffff811afdaa <unwind_get_return_address+0x9a>
> ffffffff811afdfb: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
>
>> Any idea how recreatable it is? (In particular I'd be interested in
>> seeing this dump with the latest unwinder improvements in the -tip tree,
>> which dump the pt_regs associated with an interrupt.)
>
> Fairly reproducable it seems, doesn't seem to include pt_regs dumps
> though :/
>
> tip/master as of this morning.

Can you print the stack that it gets after unwinding? If we will see
some garbage there, then it will confirm that it reads from redzones.
You can check taint before/after unwind and dump the stack iff kernel
become tainted during unwind.

> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff88042fc87be0
> 3Read of size 8 by task swapper/28/0
> 0page:ffffea0010bf21c0 count:1 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000400(reserved)
> 1page dumped because: kasan: bad access detected
> dCPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d <NMI>
> d ? dump_stack+0x5e/0x89
> d ? kasan_report_error+0x4a5/0x4d0
> d ? __asan_report_load8_noabort+0x45/0x50
> d ? __kernel_text_address+0x20/0xa0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? perf_callchain_kernel+0x33c/0x540
> d ? arch_perf_update_userpage+0x340/0x340
> d ? get_perf_callchain+0x24d/0x610
> d ? put_callchain_buffers+0x50/0x50
> d ? number+0x653/0x830
> d ? perf_callchain+0x126/0x190
> d ? perf_prepare_sample+0x720/0x1010
> d ? perf_event_output_forward+0x81/0xf0
> d ? perf_prepare_sample+0x1010/0x1010
> d ? pointer+0x880/0x880
> d ? perf_event_update_userpage+0x16/0x730
> d ? __perf_event_overflow+0x1a0/0x510
> d ? intel_pmu_handle_irq+0x34b/0xa90
> d ? intel_pmu_save_and_restart+0xd0/0xd0
> d ? acpi_os_read_memory+0x205/0x23c
> d ? format_decode+0xc5/0x7a0
> d ? vunmap_page_range+0x26a/0x400
> d ? ghes_copy_tofrom_phys+0x141/0x270
> d ? ghes_read_estatus+0x112/0x5a0
> d ? ghes_copy_tofrom_phys+0x270/0x270
> d ? early_printk+0xa4/0xd0
> d ? devkmsg_sysctl_set_loglvl+0x160/0x160
> d ? perf_event_nmi_handler+0x28/0x40
> d ? nmi_handle+0xa1/0x250
> d ? default_do_nmi+0x61/0x170
> d ? do_nmi+0x191/0x200
> d ? end_repeat_nmi+0x1a/0x1e
> d ? format_decode+0xc5/0x7a0
> d ? format_decode+0xc5/0x7a0
> d ? format_decode+0xc5/0x7a0
> d <EOE>
> d <IRQ>
> d ? vsnprintf+0xfc/0x15e0
> d ? pointer+0x880/0x880
> d ? x86_pmu_enable_all+0x1c0/0x1c0
> d ? vscnprintf+0x9/0x30
> d ? early_vprintk+0xb0/0x130
> d ? trace_raw_output_console+0x160/0x160
> d ? memcpy+0x34/0x50
> d ? x86_pmu_commit_txn+0x180/0x260
> d ? events_sysfs_show+0xb0/0xb0
> d ? save_stack+0x33/0xb0
> d ? hrtimer_init+0x120/0x120
> d ? timerqueue_del+0x62/0x140
> d ? perf_event_update_userpage+0x16/0x730
> d ? perf_event_update_userpage+0x16/0x730
> d ? x86_perf_event_set_period+0x239/0x450
> d ? perf_event_update_userpage+0x16/0x730
> d ? x86_pmu_enable+0x5f7/0xaa0
> d ? printk+0xb6/0xef
> d ? printk_emit+0xa0/0xa0
> d ? _raw_spin_unlock_irqrestore+0x42/0x70
> d ? ___ratelimit+0x1e4/0x3f0
> d ? irq_work_run_list+0xa1/0xf0
> d ? irq_work_run+0x14/0x40
> d ? smp_call_function_single_interrupt+0x60/0x80
> d ? call_function_single_interrupt+0x89/0x90
> d <EOI>
> d ? cpuidle_enter_state+0x113/0x780
> d ? cpuidle_enter_state+0x10e/0x780
> d ? cpu_load_update_nohz_stop+0x155/0x1b0
> d ? cpu_startup_entry+0x19a/0x2c0
> d ? start_cpu+0x5/0x14
> 3Memory state around the buggy address:
> 3 ffff88042fc87a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff88042fc87b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3>ffff88042fc87b80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
> 3 ^
> 3 ffff88042fc87c00: 00 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00
> 3 ffff88042fc87c80: 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00
> 3==================================================================
>
>
> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff880424a97878
> 3Read of size 8 by task perf_fuzzer/3451
> 0page:ffffea001092a5c0 count:0 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000000()
> 1page dumped because: kasan: bad access detected
> dCPU: 28 PID: 3451 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d ? dump_stack+0x5e/0x89
> d ? kasan_report_error+0x4a5/0x4d0
> d ? __asan_report_load8_noabort+0x45/0x50
> d ? __kernel_text_address+0x20/0xa0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? perf_callchain_kernel+0x33c/0x540
> d ? arch_perf_update_userpage+0x340/0x340
> d ? get_perf_callchain+0x24d/0x610
> d ? put_callchain_buffers+0x50/0x50
> d ? ipv6_flowlabel_opt+0x1111/0x17d0
> d ? perf_log_itrace_start+0x3a0/0x3a0
> d ? cpumask_next_and+0x5a/0xa0
> d ? ktime_get_raw_fast_ns+0xd3/0x1e0
> d ? perf_callchain+0x126/0x190
> d ? perf_prepare_sample+0x720/0x1010
> d ? perf_event_output_forward+0x81/0xf0
> d ? perf_prepare_sample+0x1010/0x1010
> d ? perf_event_update_userpage+0x16/0x730
> d ? kasan_unpoison_shadow+0x31/0x40
> d ? get_page_from_freelist+0x52e/0x2310
> d ? perf_output_begin+0x3a1/0x9b0
> d ? cpu_clock_event_add+0x17/0x20
> d ? __perf_event_overflow+0x1a0/0x510
> d ? perf_swevent_overflow+0x156/0x1f0
> d ? perf_tp_event+0x3e8/0x5c0
> d ? perf_output_begin_backward+0x960/0x960
> d ? perf_tp_event_match.isra.85.part.86+0x140/0x140
> d ? __mark_inode_dirty+0x459/0xa50
> d ? legitimize_path.isra.28+0x6b/0x150
> d ? unlazy_walk+0x456/0x790
> d ? memset+0x1f/0x40
> d ? perf_trace_writeback_dirty_inode_template+0x3af/0x610
> d ? save_stack+0x33/0xb0
> d ? inode_congested+0x450/0x450
> d ? dput+0x1de/0x530
> d ? walk_component+0x2cc/0xdc0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? pick_link+0xbe0/0xbe0
> d ? inode_congested+0x450/0x450
> d ? __mark_inode_dirty+0x459/0xa50
> d ? proc_sys_setattr+0x84/0xb0
> d ? notify_change+0x4d6/0xc40
> d ? security_inode_need_killpriv+0x58/0x80
> d ? do_truncate+0xd7/0x160
> d ? file_open_root+0x1a0/0x1a0
> d ? path_openat+0x97f/0x3b30
> d ? vfs_rename+0x14a0/0x14a0
> d ? getname_flags+0xba/0x500
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? do_filp_open+0x175/0x230
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? may_open_dev+0xc0/0xc0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? do_sys_open+0x16d/0x310
> d ? SyS_write+0xab/0x160
> d ? filp_open+0x50/0x50
> d ? task_stopped_code+0xf0/0xf0
> d ? trace_hardirqs_on_thunk+0x1a/0x1c
> d ? entry_SYSCALL_64_fastpath+0x18/0xa8
> 3Memory state around the buggy address:
> 3 ffff880424a97700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff880424a97780: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3
> 3>ffff880424a97800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
> 3 ^
> 3 ffff880424a97880: f1 f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f3
> 3 ffff880424a97900: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3==================================================================
>
>
>

2016-11-16 14:37:55

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 02:03:37PM +0100, Peter Zijlstra wrote:
> On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
> > Would you mind posting a disassembly of unwind_get_return_address()?
> > Any idea how recreatable it is? (In particular I'd be interested in
> > seeing this dump with the latest unwinder improvements in the -tip tree,
> > which dump the pt_regs associated with an interrupt.)
>
> Fairly reproducable it seems, doesn't seem to include pt_regs dumps
> though :/
>
> tip/master as of this morning.

Thanks. This is actually a different issue than the one reported by
Vince. In this case FRAME_POINTER is disabled, so it uses the "guess"
unwinder which scans every address on the stack, looking for text
addresses. So the kasan errors are expected.

(The missing pt_regs are also expected: the guess unwinder doesn't show
them.)

I'll work up a patch to fix this. I still have no idea what's causing
Vince's bug in the frame pointer unwinder.

> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff88042fc87be0
> 3Read of size 8 by task swapper/28/0
> 0page:ffffea0010bf21c0 count:1 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000400(reserved)
> 1page dumped because: kasan: bad access detected
> dCPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d <NMI>
> d ? dump_stack+0x5e/0x89
> d ? kasan_report_error+0x4a5/0x4d0
> d ? __asan_report_load8_noabort+0x45/0x50
> d ? __kernel_text_address+0x20/0xa0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? perf_callchain_kernel+0x33c/0x540
> d ? arch_perf_update_userpage+0x340/0x340
> d ? get_perf_callchain+0x24d/0x610
> d ? put_callchain_buffers+0x50/0x50
> d ? number+0x653/0x830
> d ? perf_callchain+0x126/0x190
> d ? perf_prepare_sample+0x720/0x1010
> d ? perf_event_output_forward+0x81/0xf0
> d ? perf_prepare_sample+0x1010/0x1010
> d ? pointer+0x880/0x880
> d ? perf_event_update_userpage+0x16/0x730
> d ? __perf_event_overflow+0x1a0/0x510
> d ? intel_pmu_handle_irq+0x34b/0xa90
> d ? intel_pmu_save_and_restart+0xd0/0xd0
> d ? acpi_os_read_memory+0x205/0x23c
> d ? format_decode+0xc5/0x7a0
> d ? vunmap_page_range+0x26a/0x400
> d ? ghes_copy_tofrom_phys+0x141/0x270
> d ? ghes_read_estatus+0x112/0x5a0
> d ? ghes_copy_tofrom_phys+0x270/0x270
> d ? early_printk+0xa4/0xd0
> d ? devkmsg_sysctl_set_loglvl+0x160/0x160
> d ? perf_event_nmi_handler+0x28/0x40
> d ? nmi_handle+0xa1/0x250
> d ? default_do_nmi+0x61/0x170
> d ? do_nmi+0x191/0x200
> d ? end_repeat_nmi+0x1a/0x1e
> d ? format_decode+0xc5/0x7a0
> d ? format_decode+0xc5/0x7a0
> d ? format_decode+0xc5/0x7a0
> d <EOE>
> d <IRQ>
> d ? vsnprintf+0xfc/0x15e0
> d ? pointer+0x880/0x880
> d ? x86_pmu_enable_all+0x1c0/0x1c0
> d ? vscnprintf+0x9/0x30
> d ? early_vprintk+0xb0/0x130
> d ? trace_raw_output_console+0x160/0x160
> d ? memcpy+0x34/0x50
> d ? x86_pmu_commit_txn+0x180/0x260
> d ? events_sysfs_show+0xb0/0xb0
> d ? save_stack+0x33/0xb0
> d ? hrtimer_init+0x120/0x120
> d ? timerqueue_del+0x62/0x140
> d ? perf_event_update_userpage+0x16/0x730
> d ? perf_event_update_userpage+0x16/0x730
> d ? x86_perf_event_set_period+0x239/0x450
> d ? perf_event_update_userpage+0x16/0x730
> d ? x86_pmu_enable+0x5f7/0xaa0
> d ? printk+0xb6/0xef
> d ? printk_emit+0xa0/0xa0
> d ? _raw_spin_unlock_irqrestore+0x42/0x70
> d ? ___ratelimit+0x1e4/0x3f0
> d ? irq_work_run_list+0xa1/0xf0
> d ? irq_work_run+0x14/0x40
> d ? smp_call_function_single_interrupt+0x60/0x80
> d ? call_function_single_interrupt+0x89/0x90
> d <EOI>
> d ? cpuidle_enter_state+0x113/0x780
> d ? cpuidle_enter_state+0x10e/0x780
> d ? cpu_load_update_nohz_stop+0x155/0x1b0
> d ? cpu_startup_entry+0x19a/0x2c0
> d ? start_cpu+0x5/0x14
> 3Memory state around the buggy address:
> 3 ffff88042fc87a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff88042fc87b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3>ffff88042fc87b80: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
> 3 ^
> 3 ffff88042fc87c00: 00 f4 f4 f4 f3 f3 f3 f3 00 00 00 00 00 00 00 00
> 3 ffff88042fc87c80: 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00
> 3==================================================================
>
>
> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1ba/0x1f0 at addr ffff880424a97878
> 3Read of size 8 by task perf_fuzzer/3451
> 0page:ffffea001092a5c0 count:0 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000000()
> 1page dumped because: kasan: bad access detected
> dCPU: 28 PID: 3451 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #2
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d ? dump_stack+0x5e/0x89
> d ? kasan_report_error+0x4a5/0x4d0
> d ? __asan_report_load8_noabort+0x45/0x50
> d ? __kernel_text_address+0x20/0xa0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? unwind_next_frame+0x1ba/0x1f0
> d ? perf_callchain_kernel+0x33c/0x540
> d ? arch_perf_update_userpage+0x340/0x340
> d ? get_perf_callchain+0x24d/0x610
> d ? put_callchain_buffers+0x50/0x50
> d ? ipv6_flowlabel_opt+0x1111/0x17d0
> d ? perf_log_itrace_start+0x3a0/0x3a0
> d ? cpumask_next_and+0x5a/0xa0
> d ? ktime_get_raw_fast_ns+0xd3/0x1e0
> d ? perf_callchain+0x126/0x190
> d ? perf_prepare_sample+0x720/0x1010
> d ? perf_event_output_forward+0x81/0xf0
> d ? perf_prepare_sample+0x1010/0x1010
> d ? perf_event_update_userpage+0x16/0x730
> d ? kasan_unpoison_shadow+0x31/0x40
> d ? get_page_from_freelist+0x52e/0x2310
> d ? perf_output_begin+0x3a1/0x9b0
> d ? cpu_clock_event_add+0x17/0x20
> d ? __perf_event_overflow+0x1a0/0x510
> d ? perf_swevent_overflow+0x156/0x1f0
> d ? perf_tp_event+0x3e8/0x5c0
> d ? perf_output_begin_backward+0x960/0x960
> d ? perf_tp_event_match.isra.85.part.86+0x140/0x140
> d ? __mark_inode_dirty+0x459/0xa50
> d ? legitimize_path.isra.28+0x6b/0x150
> d ? unlazy_walk+0x456/0x790
> d ? memset+0x1f/0x40
> d ? perf_trace_writeback_dirty_inode_template+0x3af/0x610
> d ? save_stack+0x33/0xb0
> d ? inode_congested+0x450/0x450
> d ? dput+0x1de/0x530
> d ? walk_component+0x2cc/0xdc0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? pick_link+0xbe0/0xbe0
> d ? inode_congested+0x450/0x450
> d ? __mark_inode_dirty+0x459/0xa50
> d ? proc_sys_setattr+0x84/0xb0
> d ? notify_change+0x4d6/0xc40
> d ? security_inode_need_killpriv+0x58/0x80
> d ? do_truncate+0xd7/0x160
> d ? file_open_root+0x1a0/0x1a0
> d ? path_openat+0x97f/0x3b30
> d ? vfs_rename+0x14a0/0x14a0
> d ? getname_flags+0xba/0x500
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? do_filp_open+0x175/0x230
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? may_open_dev+0xc0/0xc0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? save_stack+0x33/0xb0
> d ? do_sys_open+0x16d/0x310
> d ? SyS_write+0xab/0x160
> d ? filp_open+0x50/0x50
> d ? task_stopped_code+0xf0/0xf0
> d ? trace_hardirqs_on_thunk+0x1a/0x1c
> d ? entry_SYSCALL_64_fastpath+0x18/0xa8
> 3Memory state around the buggy address:
> 3 ffff880424a97700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff880424a97780: 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3
> 3>ffff880424a97800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
> 3 ^
> 3 ffff880424a97880: f1 f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 f4 f4 f4 f3
> 3 ffff880424a97900: f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3==================================================================

--
Josh

2016-11-16 14:49:50

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 08:37:46AM -0600, Josh Poimboeuf wrote:
> On Wed, Nov 16, 2016 at 02:03:37PM +0100, Peter Zijlstra wrote:
> > On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
> > > Would you mind posting a disassembly of unwind_get_return_address()?
> > > Any idea how recreatable it is? (In particular I'd be interested in
> > > seeing this dump with the latest unwinder improvements in the -tip tree,
> > > which dump the pt_regs associated with an interrupt.)
> >
> > Fairly reproducable it seems, doesn't seem to include pt_regs dumps
> > though :/
> >
> > tip/master as of this morning.
>
> Thanks. This is actually a different issue than the one reported by
> Vince. In this case FRAME_POINTER is disabled, so it uses the "guess"
> unwinder which scans every address on the stack, looking for text
> addresses. So the kasan errors are expected.
>
> (The missing pt_regs are also expected: the guess unwinder doesn't show
> them.)
>
> I'll work up a patch to fix this. I still have no idea what's causing
> Vince's bug in the frame pointer unwinder.

Hurm,.. by the number of '?' entries in Vince's backtrace I was assuming
it was without frame pointers.

Let me enable those and run again, it didn't insta-trigger like it does
without.

2016-11-16 14:58:44

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 03:49:43PM +0100, Peter Zijlstra wrote:
> On Wed, Nov 16, 2016 at 08:37:46AM -0600, Josh Poimboeuf wrote:
> > On Wed, Nov 16, 2016 at 02:03:37PM +0100, Peter Zijlstra wrote:
> > > On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
> > > > Would you mind posting a disassembly of unwind_get_return_address()?
> > > > Any idea how recreatable it is? (In particular I'd be interested in
> > > > seeing this dump with the latest unwinder improvements in the -tip tree,
> > > > which dump the pt_regs associated with an interrupt.)
> > >
> > > Fairly reproducable it seems, doesn't seem to include pt_regs dumps
> > > though :/
> > >
> > > tip/master as of this morning.
> >
> > Thanks. This is actually a different issue than the one reported by
> > Vince. In this case FRAME_POINTER is disabled, so it uses the "guess"
> > unwinder which scans every address on the stack, looking for text
> > addresses. So the kasan errors are expected.
> >
> > (The missing pt_regs are also expected: the guess unwinder doesn't show
> > them.)
> >
> > I'll work up a patch to fix this. I still have no idea what's causing
> > Vince's bug in the frame pointer unwinder.
>
> Hurm,.. by the number of '?' entries in Vince's backtrace I was assuming
> it was without frame pointers.

When frame pointers are disabled, *all* the addresses are prefixed with
'?'.

When frame pointers are enabled, and there are a lot of '?' addresses,
it usually means the containing functions reserved a lot of stack space
and the printed addresses are mostly leftovers from previous runs.

> Let me enable those and run again, it didn't insta-trigger like it does
> without.

Thanks!

--
Josh

2016-11-16 14:58:56

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 03:49:43PM +0100, Peter Zijlstra wrote:
> Let me enable those and run again, it didn't insta-trigger like it does
> without.

Tada!

$ objdump -D ivb-dbg/vmlinux | awk '/<[^>]*>:/ { p = 0; } /<unwind_get_return_address>:/ { p = 1; } { if (p) print $0; }'

ffffffff811c70d0 <unwind_get_return_address>:
ffffffff811c70d0: e8 8b 61 0e 02 callq ffffffff832ad260 <__fentry__>
ffffffff811c70d5: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c70dc: fc ff df
ffffffff811c70df: 55 push %rbp
ffffffff811c70e0: 48 89 fa mov %rdi,%rdx
ffffffff811c70e3: 48 89 e5 mov %rsp,%rbp
ffffffff811c70e6: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c70ea: 41 56 push %r14
ffffffff811c70ec: 41 55 push %r13
ffffffff811c70ee: 41 54 push %r12
ffffffff811c70f0: 53 push %rbx
ffffffff811c70f1: 48 89 fb mov %rdi,%rbx
ffffffff811c70f4: 48 83 ec 10 sub $0x10,%rsp
ffffffff811c70f8: 0f b6 14 02 movzbl (%rdx,%rax,1),%edx
ffffffff811c70fc: 48 89 f8 mov %rdi,%rax
ffffffff811c70ff: 83 e0 07 and $0x7,%eax
ffffffff811c7102: 83 c0 03 add $0x3,%eax
ffffffff811c7105: 38 d0 cmp %dl,%al
ffffffff811c7107: 7c 08 jl ffffffff811c7111 <unwind_get_return_address+0x41>
ffffffff811c7109: 84 d2 test %dl,%dl
ffffffff811c710b: 0f 85 0e 01 00 00 jne ffffffff811c721f <unwind_get_return_address+0x14f>
ffffffff811c7111: 8b 03 mov (%rbx),%eax
ffffffff811c7113: 85 c0 test %eax,%eax
ffffffff811c7115: 0f 84 c9 00 00 00 je ffffffff811c71e4 <unwind_get_return_address+0x114>
ffffffff811c711b: 48 8d 7b 40 lea 0x40(%rbx),%rdi
ffffffff811c711f: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7126: fc ff df
ffffffff811c7129: 48 89 fa mov %rdi,%rdx
ffffffff811c712c: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c7130: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c7134: 0f 85 ef 00 00 00 jne ffffffff811c7229 <unwind_get_return_address+0x159>
ffffffff811c713a: 4c 8b 63 40 mov 0x40(%rbx),%r12
ffffffff811c713e: 4d 85 e4 test %r12,%r12
ffffffff811c7141: 0f 84 ac 00 00 00 je ffffffff811c71f3 <unwind_get_return_address+0x123>
ffffffff811c7147: 49 8d bc 24 88 00 00 lea 0x88(%r12),%rdi
ffffffff811c714e: 00
ffffffff811c714f: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7156: fc ff df
ffffffff811c7159: 48 89 f9 mov %rdi,%rcx
ffffffff811c715c: 48 c1 e9 03 shr $0x3,%rcx
ffffffff811c7160: 80 3c 01 00 cmpb $0x0,(%rcx,%rax,1)
ffffffff811c7164: 0f 85 4f 01 00 00 jne ffffffff811c72b9 <unwind_get_return_address+0x1e9>
ffffffff811c716a: 41 f6 84 24 88 00 00 testb $0x3,0x88(%r12)
ffffffff811c7171: 00 03
ffffffff811c7173: 75 6f jne ffffffff811c71e4 <unwind_get_return_address+0x114>
ffffffff811c7175: 49 83 ec 80 sub $0xffffffffffffff80,%r12
ffffffff811c7179: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7180: fc ff df
ffffffff811c7183: 4c 89 e2 mov %r12,%rdx
ffffffff811c7186: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c718a: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c718e: 0f 85 2f 01 00 00 jne ffffffff811c72c3 <unwind_get_return_address+0x1f3>
ffffffff811c7194: 4c 8d 73 28 lea 0x28(%rbx),%r14
ffffffff811c7198: 49 8b 14 24 mov (%r12),%rdx
ffffffff811c719c: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c71a3: fc ff df
ffffffff811c71a6: 48 8d 73 30 lea 0x30(%rbx),%rsi
ffffffff811c71aa: 4c 89 f1 mov %r14,%rcx
ffffffff811c71ad: 48 c1 e9 03 shr $0x3,%rcx
ffffffff811c71b1: 80 3c 01 00 cmpb $0x0,(%rcx,%rax,1)
ffffffff811c71b5: 0f 85 15 01 00 00 jne ffffffff811c72d0 <unwind_get_return_address+0x200>
ffffffff811c71bb: 48 8b 7b 28 mov 0x28(%rbx),%rdi
ffffffff811c71bf: 4c 89 e1 mov %r12,%rcx
ffffffff811c71c2: e8 59 7a 2c 00 callq ffffffff8148ec20 <ftrace_graph_ret_addr>
ffffffff811c71c7: 48 89 c7 mov %rax,%rdi
ffffffff811c71ca: 49 89 c5 mov %rax,%r13
ffffffff811c71cd: e8 9e 30 0c 00 callq ffffffff8128a270 <__kernel_text_address>
ffffffff811c71d2: 89 c2 mov %eax,%edx
ffffffff811c71d4: 4c 89 e8 mov %r13,%rax
ffffffff811c71d7: 85 d2 test %edx,%edx
ffffffff811c71d9: 75 0b jne ffffffff811c71e6 <unwind_get_return_address+0x116>
ffffffff811c71db: 80 3d 18 29 f9 02 00 cmpb $0x0,0x2f92918(%rip) # ffffffff84159afa <__print_once.27085>
ffffffff811c71e2: 74 4f je ffffffff811c7233 <unwind_get_return_address+0x163>
ffffffff811c71e4: 31 c0 xor %eax,%eax
ffffffff811c71e6: 48 83 c4 10 add $0x10,%rsp
ffffffff811c71ea: 5b pop %rbx
ffffffff811c71eb: 41 5c pop %r12
ffffffff811c71ed: 41 5d pop %r13
ffffffff811c71ef: 41 5e pop %r14
ffffffff811c71f1: 5d pop %rbp
ffffffff811c71f2: c3 retq
ffffffff811c71f3: 48 8d 7b 38 lea 0x38(%rbx),%rdi
ffffffff811c71f7: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c71fe: fc ff df
ffffffff811c7201: 48 89 fa mov %rdi,%rdx
ffffffff811c7204: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c7208: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c720c: 0f 85 9d 00 00 00 jne ffffffff811c72af <unwind_get_return_address+0x1df>
ffffffff811c7212: 48 8b 43 38 mov 0x38(%rbx),%rax
ffffffff811c7216: 4c 8d 60 08 lea 0x8(%rax),%r12
ffffffff811c721a: e9 5a ff ff ff jmpq ffffffff811c7179 <unwind_get_return_address+0xa9>
ffffffff811c721f: e8 6c b0 45 00 callq ffffffff81622290 <__asan_report_load4_noabort>
ffffffff811c7224: e9 e8 fe ff ff jmpq ffffffff811c7111 <unwind_get_return_address+0x41>
ffffffff811c7229: e8 b2 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c722e: e9 07 ff ff ff jmpq ffffffff811c713a <unwind_get_return_address+0x6a>
ffffffff811c7233: 4c 89 f2 mov %r14,%rdx
ffffffff811c7236: c6 05 bd 28 f9 02 01 movb $0x1,0x2f928bd(%rip) # ffffffff84159afa <__print_once.27085>
ffffffff811c723d: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7244: fc ff df
ffffffff811c7247: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c724b: 80 3c 02 00 cmpb $0x0,(%rdx,%rax,1)
ffffffff811c724f: 75 4d jne ffffffff811c729e <unwind_get_return_address+0x1ce>
ffffffff811c7251: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
ffffffff811c7258: fc ff df
ffffffff811c725b: 48 8b 5b 28 mov 0x28(%rbx),%rbx
ffffffff811c725f: 48 8d bb c0 04 00 00 lea 0x4c0(%rbx),%rdi
ffffffff811c7266: 48 89 fa mov %rdi,%rdx
ffffffff811c7269: 48 c1 ea 03 shr $0x3,%rdx
ffffffff811c726d: 0f b6 04 02 movzbl (%rdx,%rax,1),%eax
ffffffff811c7271: 84 c0 test %al,%al
ffffffff811c7273: 74 04 je ffffffff811c7279 <unwind_get_return_address+0x1a9>
ffffffff811c7275: 3c 03 cmp $0x3,%al
ffffffff811c7277: 7e 2f jle ffffffff811c72a8 <unwind_get_return_address+0x1d8>
ffffffff811c7279: 44 8b 83 c0 04 00 00 mov 0x4c0(%rbx),%r8d
ffffffff811c7280: 48 8d 8b 58 06 00 00 lea 0x658(%rbx),%rcx
ffffffff811c7287: 4c 89 e2 mov %r12,%rdx
ffffffff811c728a: 4c 89 ee mov %r13,%rsi
ffffffff811c728d: 48 c7 c7 e0 1d 45 83 mov $0xffffffff83451de0,%rdi
ffffffff811c7294: e8 49 8c 35 00 callq ffffffff8151fee2 <printk_deferred>
ffffffff811c7299: e9 46 ff ff ff jmpq ffffffff811c71e4 <unwind_get_return_address+0x114>
ffffffff811c729e: 4c 89 f7 mov %r14,%rdi
ffffffff811c72a1: e8 3a b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72a6: eb a9 jmp ffffffff811c7251 <unwind_get_return_address+0x181>
ffffffff811c72a8: e8 e3 af 45 00 callq ffffffff81622290 <__asan_report_load4_noabort>
ffffffff811c72ad: eb ca jmp ffffffff811c7279 <unwind_get_return_address+0x1a9>
ffffffff811c72af: e8 2c b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72b4: e9 59 ff ff ff jmpq ffffffff811c7212 <unwind_get_return_address+0x142>
ffffffff811c72b9: e8 22 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72be: e9 a7 fe ff ff jmpq ffffffff811c716a <unwind_get_return_address+0x9a>
ffffffff811c72c3: 4c 89 e7 mov %r12,%rdi
ffffffff811c72c6: e8 15 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72cb: e9 c4 fe ff ff jmpq ffffffff811c7194 <unwind_get_return_address+0xc4>
ffffffff811c72d0: 4c 89 f7 mov %r14,%rdi
ffffffff811c72d3: 48 89 75 d0 mov %rsi,-0x30(%rbp)
ffffffff811c72d7: 48 89 55 d8 mov %rdx,-0x28(%rbp)
ffffffff811c72db: e8 00 b0 45 00 callq ffffffff816222e0 <__asan_report_load8_noabort>
ffffffff811c72e0: 48 8b 75 d0 mov -0x30(%rbp),%rsi
ffffffff811c72e4: 48 8b 55 d8 mov -0x28(%rbp),%rdx
ffffffff811c72e8: e9 ce fe ff ff jmpq ffffffff811c71bb <unwind_get_return_address+0xeb>
ffffffff811c72ed: 0f 1f 00 nopl (%rax)

---
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042f88bba0
3Read of size 8 by task swapper/2/0
0page:ffffea0010be22c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #3
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d ? printk+0xef/0xef
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_get_return_address+0x1fb/0x220
d unwind_get_return_address+0x1fb/0x220
d perf_callchain_kernel+0x356/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? __perf_event_header__init_id+0x500/0x500
d get_perf_callchain+0x276/0x670
d ? put_callchain_buffers+0x50/0x50
d ? sched_clock_cpu+0x11c/0x1a0
d perf_callchain+0x128/0x1a0
d perf_prepare_sample+0x70e/0xfb0
d perf_event_output_forward+0x93/0x110
d ? perf_prepare_sample+0xfb0/0xfb0
d ? arch_perf_update_userpage+0x26c/0x350
d ? sched_clock_cpu+0x11c/0x1a0
d __perf_event_overflow+0x1a3/0x570
d perf_event_overflow+0x14/0x20
d __intel_pmu_pebs_event+0x3ca/0x610
d ? pebs_update_state+0x310/0x310
d ? acpi_map_lookup+0x40/0xad
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? acpi_map_lookup+0x40/0xad
d ? put_dec+0x1c/0xb0
d ? number+0x71c/0xa70
d ? put_dec+0xb0/0xb0
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? early_serial_putc+0x41/0x70
d ? early_serial_write+0x7c/0xf0
d ? trace_raw_output_console+0x160/0x160
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? acpi_os_read_memory+0x228/0x262
d ? acpi_os_get_timer+0x1a/0x1a
d ? vunmap_page_range+0x269/0x400
d ? ghes_copy_tofrom_phys+0x149/0x270
d ? ghes_read_estatus+0x11e/0x6b0
d ? ghes_copy_tofrom_phys+0x270/0x270
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d default_do_nmi+0x111/0x180
d do_nmi+0x1a2/0x210
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:irq_exit+0x10/0x1d0
dRSP: 0000:ffff88042f887fc8 EFLAGS: 00000046c
dRAX: 0000000000000000 RBX: ffffffff83a77980 RCX: 1ffff10080965faf
dRDX: 1ffff10085f13747 RSI: 0000000000000000 RDI: ffff88042f89ba38
dRBP: ffff88042f887fd0 R08: ffff8804060b1a08 R09: 1ffff10085f1276e
dR10: ffffed0080c16369 R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d ? irq_exit+0x10/0x1d0
d ? irq_exit+0x10/0x1d0
d <EOE>
d <IRQ>
d smp_call_function_single_interrupt+0x70/0x90
d call_function_single_interrupt+0x90/0xa0
dRIP: 0010:cpuidle_enter_state+0x121/0x7a0
dRSP: 0000:ffff88042caffe28 EFLAGS: 00000246c ORIG_RAX: ffffffffffffff04
dRAX: 0000000000000000 RBX: ffff88042f8ab720 RCX: 000000000000001f
dRDX: 1ffff10085f142f9 RSI: 000000002dd33691 RDI: ffff88042f8a17c8
dRBP: ffff88042caffe88 R08: 0000000000000018 R09: ffffffff83f3f320
dR10: 071c71c71c71c71c R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d <EOI>
d ? cpuidle_enter_state+0x11c/0x7a0
d cpuidle_enter+0x17/0x20
d call_cpuidle+0x47/0xc0
d ? cpuidle_select+0x59/0x80
d cpu_startup_entry+0x1a6/0x2d0
d start_secondary+0x245/0x2d0
d start_cpu+0x5/0x14
3Memory state around the buggy address:
3 ffff88042f88ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
3 ffff88042f88bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88042f88bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
3 ^
3 ffff88042f88bc00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042f88bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================
4Disabling lock debugging due to kernel taint
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x5fc/0x780 at addr ffff88042f88bb98
3Read of size 8 by task swapper/2/0
0page:ffffea0010be22c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 2 PID: 0 Comm: swapper/2 Tainted: G B 4.9.0-rc5-00530-gd8866fc-dirty #3
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d ? kasan_report_error+0x420/0x4d0
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_next_frame+0x5fc/0x780
d unwind_next_frame+0x5fc/0x780
d perf_callchain_kernel+0x341/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? __perf_event_header__init_id+0x500/0x500
d get_perf_callchain+0x276/0x670
d ? put_callchain_buffers+0x50/0x50
d ? sched_clock_cpu+0x11c/0x1a0
d perf_callchain+0x128/0x1a0
d perf_prepare_sample+0x70e/0xfb0
d perf_event_output_forward+0x93/0x110
d ? perf_prepare_sample+0xfb0/0xfb0
d ? arch_perf_update_userpage+0x26c/0x350
d ? sched_clock_cpu+0x11c/0x1a0
d __perf_event_overflow+0x1a3/0x570
d perf_event_overflow+0x14/0x20
d __intel_pmu_pebs_event+0x3ca/0x610
d ? pebs_update_state+0x310/0x310
d ? acpi_map_lookup+0x40/0xad
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? acpi_map_lookup+0x40/0xad
d ? put_dec+0x1c/0xb0
d ? number+0x71c/0xa70
d ? put_dec+0xb0/0xb0
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? early_serial_putc+0x41/0x70
d ? early_serial_write+0x7c/0xf0
d ? trace_raw_output_console+0x160/0x160
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? acpi_os_read_memory+0x228/0x262
d ? acpi_os_get_timer+0x1a/0x1a
d ? vunmap_page_range+0x269/0x400
d ? ghes_copy_tofrom_phys+0x149/0x270
d ? ghes_read_estatus+0x11e/0x6b0
d ? ghes_copy_tofrom_phys+0x270/0x270
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d default_do_nmi+0x111/0x180
d do_nmi+0x1a2/0x210
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:irq_exit+0x10/0x1d0
dRSP: 0000:ffff88042f887fc8 EFLAGS: 00000046c
dRAX: 0000000000000000 RBX: ffffffff83a77980 RCX: 1ffff10080965faf
dRDX: 1ffff10085f13747 RSI: 0000000000000000 RDI: ffff88042f89ba38
dRBP: ffff88042f887fd0 R08: ffff8804060b1a08 R09: 1ffff10085f1276e
dR10: ffffed0080c16369 R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d ? irq_exit+0x10/0x1d0
d ? irq_exit+0x10/0x1d0
d <EOE>
d <IRQ>
d smp_call_function_single_interrupt+0x70/0x90
d call_function_single_interrupt+0x90/0xa0
dRIP: 0010:cpuidle_enter_state+0x121/0x7a0
dRSP: 0000:ffff88042caffe28 EFLAGS: 00000246c ORIG_RAX: ffffffffffffff04
dRAX: 0000000000000000 RBX: ffff88042f8ab720 RCX: 000000000000001f
dRDX: 1ffff10085f142f9 RSI: 000000002dd33691 RDI: ffff88042f8a17c8
dRBP: ffff88042caffe88 R08: 0000000000000018 R09: ffffffff83f3f320
dR10: 071c71c71c71c71c R11: ffff88042f89dd04 R12: 00000023af3410aa
dR13: 0000000000000004 R14: 0000000000000004 R15: 0000000000000180
d <EOI>
d ? cpuidle_enter_state+0x11c/0x7a0
d cpuidle_enter+0x17/0x20
d call_cpuidle+0x47/0xc0
d ? cpuidle_select+0x59/0x80
d cpu_startup_entry+0x1a6/0x2d0
d start_secondary+0x245/0x2d0
d start_cpu+0x5/0x14
3Memory state around the buggy address:
3 ffff88042f88ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
3 ffff88042f88bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88042f88bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
3 ^
3 ffff88042f88bc00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042f88bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================

2016-11-16 15:06:23

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, 16 Nov 2016, Peter Zijlstra wrote:

> On Wed, Nov 16, 2016 at 08:37:46AM -0600, Josh Poimboeuf wrote:
> > On Wed, Nov 16, 2016 at 02:03:37PM +0100, Peter Zijlstra wrote:
> > > On Tue, Nov 15, 2016 at 02:57:48PM -0600, Josh Poimboeuf wrote:
> > > > Would you mind posting a disassembly of unwind_get_return_address()?
> > > > Any idea how recreatable it is? (In particular I'd be interested in
> > > > seeing this dump with the latest unwinder improvements in the -tip tree,
> > > > which dump the pt_regs associated with an interrupt.)
> > >
> > > Fairly reproducable it seems, doesn't seem to include pt_regs dumps
> > > though :/
> > >
> > > tip/master as of this morning.
> >
> > Thanks. This is actually a different issue than the one reported by
> > Vince. In this case FRAME_POINTER is disabled, so it uses the "guess"
> > unwinder which scans every address on the stack, looking for text
> > addresses. So the kasan errors are expected.
> >
> > (The missing pt_regs are also expected: the guess unwinder doesn't show
> > them.)
> >
> > I'll work up a patch to fix this. I still have no idea what's causing
> > Vince's bug in the frame pointer unwinder.
>
> Hurm,.. by the number of '?' entries in Vince's backtrace I was assuming
> it was without frame pointers.
>
> Let me enable those and run again, it didn't insta-trigger like it does
> without.

Yes, that machine does have frame pointers enabled.

Sorry for the delay responding, the machine crashed right after I had left
last night and so I wasn't able to take a look at what was going on until
right now.

It triggers fairly quickly on the Haswell machine but not my other
machines (although they possibly don't have quite as many debug options
turned on).

Vince

2016-11-17 04:48:31

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 03:58:49PM +0100, Peter Zijlstra wrote:
> 3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042f88bba0

So I dug through the disassembly (thanks for the vmlinux), and I'm
pretty sure the stack-out-of-bounds address is on the NMI stack, in the
kasan redzone in the stack frame of intel_pmu_handle_irq().

What's weird though is that perf_callchain_kernel() passes the pt_regs
from the IRQ, not from the NMI. The unwinder should have started from
the IRQ stack. But somehow it ended up unwinding to the middle of the
NMI stack.

So it seems like stack corruption in the IRQ or task stack, with a frame
pointer that points back to the middle of the NMI stack for some reason.
But then again, the kasan error report dumped the stack fine. So that
would seem to rule out stack corruption... So I have no idea what's
going on.

I got perf_fuzzer running and tried to recreate, but no luck.

Peter or Vince, can you try to recreate with this patch? It dumps the
raw stack contents during a stack dump. Hopefully that would give a
clue about what's going wrong.

diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 499aa6f..67ff3ac 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -48,6 +48,30 @@ static void printk_stack_address(unsigned long address, int reliable,
printk("%s %s%pB\n", log_lvl, reliable ? "" : "? ", (void *)address);
}

+static void raw_stack_dump(struct stack_info *info)
+{
+ unsigned long *s, word[4];
+ int skip = 0;
+
+ for (s = info->begin; s < info->end; s += 4) {
+ word[0] = READ_ONCE_NOCHECK(s[0]);
+ word[1] = READ_ONCE_NOCHECK(s[1]);
+ word[2] = READ_ONCE_NOCHECK(s[2]);
+ word[3] = READ_ONCE_NOCHECK(s[3]);
+
+ if (!word[0] && !word[1] && !word[2] && !word[3]) {
+ if (!skip)
+ printk("%p: %016x ...\n", s, 0);
+ skip = 1;
+ continue;
+ }
+
+ skip = 0;
+ printk("%p: %016lx %016lx %016lx %016lx\n",
+ s, word[0], word[1], word[2], word[3]);
+ }
+}
+
void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
unsigned long *stack, char *log_lvl)
{
@@ -156,6 +180,8 @@ void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,

if (str_end)
printk("%s <%s>\n", log_lvl, str_end);
+
+ raw_stack_dump(&stack_info);
}
}

2016-11-17 09:05:06

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote:
> Peter or Vince, can you try to recreate with this patch? It dumps the
> raw stack contents during a stack dump. Hopefully that would give a
> clue about what's going wrong.

Here goes... I'll do another run and get you the results of that as
well.

3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042fc0baa0
3Read of size 8 by task perf_fuzzer/11689
0page:ffffea0010bf02c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 26 PID: 11689 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #4
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_get_return_address+0x1fb/0x220
d unwind_get_return_address+0x1fb/0x220
d perf_callchain_kernel+0x356/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? insn_get_opcode.part.4+0x3ac/0x910
d ? format_decode+0xf8/0x790
d ? format_decode+0xfa/0x790
d ? insn_get_modrm.part.5+0x52/0x5a0
d get_perf_callchain+0x276/0x670
d ? insn_get_sib.part.6+0x4a/0x250
d ? put_callchain_buffers+0x50/0x50
d ? sched_clock_cpu+0x11c/0x1a0
d ? format_decode+0xf8/0x790
d perf_callchain+0x128/0x1a0
d perf_prepare_sample+0x70e/0xfb0
d perf_event_output+0x93/0x110
d ? perf_event_output_backward+0x110/0x110
d ? setup_pebs_sample_data+0xd48/0x18e0
d __intel_pmu_pebs_event+0x2b1/0x610
d ? pebs_update_state+0x310/0x310
d ? early_vga_write+0x180/0x180
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? early_vga_write+0x180/0x180
d ? insn_get_prefixes.part.2+0x36d/0xc70
d ? insn_get_opcode.part.4+0x3ac/0x910
d ? number+0x71c/0xa70
d ? insn_get_opcode+0x42/0x50
d ? branch_type+0x122/0x3a0
d ? put_dec+0xb0/0xb0
d ? knc_pmu_handle_irq+0x3a0/0x3a0
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? early_serial_write+0x7c/0xf0
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? acpi_os_read_memory+0x228/0x262
d ? acpi_os_get_timer+0x1a/0x1a
d ? vunmap_page_range+0x269/0x400
d ? ghes_copy_tofrom_phys+0x149/0x270
d ? ghes_read_estatus+0x11e/0x6b0
d ? ghes_copy_tofrom_phys+0x270/0x270
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d default_do_nmi+0x111/0x180
d do_nmi+0x1a2/0x210
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:ctx_sched_in+0x44e/0x15a0
dRSP: 0000:ffff88042e7d7a58 EFLAGS: 00000046c
dRAX: 0000000000024660 RBX: dffffc0000000000 RCX: ffff88041b9a3d00
dRDX: 1ffff100e1dc6908 RSI: 1ffff10085f848e8 RDI: ffff88070ee34840
dRBP: ffff88042e7d7ac0 R08: 0000000000000000 R09: ffff88041b9a4598
dR10: ffffffff83d3e060 R11: 1ffff10085f826ec R12: ffff88070ee34400
dR13: 0000000000000000 R14: ffff88070ee34664 R15: ffff88042fc24660
d ? ctx_sched_in+0x44e/0x15a0
d ? ctx_sched_in+0x44e/0x15a0
d <EOE>
ffff88042fc0a000: 0000000000000000 ...
ffff88042fc0a5a0: 0000000000000000 0000000000000000 0000000000000000 ffff8804afc0a9a3
ffff88042fc0a5c0: ffff88042fc0a9ab 0000000000000060 1ffff10085f814c3 00000000ffffffff
ffff88042fc0a5e0: 0020000000000000 0000000000000000 00000000fffffffd 0000000000000001
ffff88042fc0a600: 00000000000015a0 0000000000000010 ffff1060ffffff09 0000000041b58ab3
ffff88042fc0a620: ffff8804afc0aa0b ffff88042fc0aa13 0000000000000060 1ffff10085f814d0
ffff88042fc0a640: 00000000ffffffff 0020000000000000 0000000000000000 00000000fffffffd
ffff88042fc0a660: 0000000000000001 00000000000015a0 ffffffff8348be10 ffff1060ffffff09
ffff88042fc0a680: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffff88042fc0a758
ffff88042fc0a6a0: ffff880431356130 ffffffff823b7c54 ffff88042fc0a7b0 ffff88042fc0a7b8
ffff88042fc0a6c0: ffff88042fc0a7b0 ffff88042fc0a9a4 0000000000000001 ffffffff8348bec9
ffff88042fc0a6e0: ffff88042fc0aa13 0000000000000018 dffffc0000000000 ffff88042fc0a870
ffff88042fc0a700: ffff88042fc0a7c0 ffff88042fc0a7e8 ffffffff823b7c54 ffff88042fc0a818
ffff88042fc0a720: ffff88042fc0a820 ffff88042fc0a818 ffff88042fc0aa0c 1ffff10085f814ec
ffff88042fc0a740: ffffed0085f81504 000000007fffffff ffff8804afc0aa0b ffffffff8348beca
ffff88042fc0a760: 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0 0000000000000000
ffff88042fc0a780: ffff1060ffffff09 ffffffff823b8d95 0000000041b58ab3 ffffffff839ff69c
ffff88042fc0a7a0: ffffffff823b8d00 0000000000000000 0000000000000020 ffff88042fc0a838
ffff88042fc0a7c0: ffffed0085f814ff 1ffff10085f81515 ffff88042fc0a988 ffffffff814f6efe
ffff88042fc0a7e0: ffffffffffffffff ffff88042fc0a890 ffffffff823b8d95 0000000041b58ab3
ffff88042fc0a800: ffffffff839ff69c ffffffff823b8d00 ffffffff838ad9d9 ffffffff00000020
ffff88042fc0a820: ffff88042fc0a8a0 ffff88042fc0a858 ffff88042fc0a8c8 ffff88042fc0aa00
ffff88042fc0a840: dffffc0000000000 ffff88042fc0a890 ffffffff813c861f ffffffff8348beca
ffff88042fc0a860: 0000000000000000 000000000000044e 00000000000015a0 ffffffff838ad9d9
ffff88042fc0a880: ffffffff838ad9d9 ffff88042fc0aa00 ffff88042fc0a9b0 ffffffff813c883f
ffff88042fc0a8a0: 000000000000000c 0000000041b58ab3 ffffffff83a0c758 ffffffff813c86d0
ffff88042fc0a8c0: ffffed0085f8151f 0000000000000000 ffff88042fc0aa88 ffff0a00ffffff05
ffff88042fc0a8e0: ffff88042fc0a9b1 ffff88042fc0aced ffff88042fc0a9b2 0000000000000000
ffff88042fc0a900: ffff88042fc0a958 000000000000044e 0000000000000000 0000000000000020
ffff88042fc0a920: 1ffff10085f8152f ffff0a00ffffff05 1ffff10085f8152f ffff88042fc0a998
ffff88042fc0a940: ffff0a00ffffff05 ffff88042fc0aa19 ffff88042fc0ad4e ffff88042fc0aa1a
ffff88042fc0a960: 0000000000000000 ffff88042fc0a9c0 ffffffff823adbcd 0000000041b58ab3
ffff88042fc0a980: ffffffff83a40559 1ffff10085f8153c ffff0a00ffffff05 1ffff10085f8153c
ffff88042fc0a9a0: ffff88042fc0aa00 ffff88042fc0ad35 ffff88042fc0af30 ffff0a00ffffff05
ffff88042fc0a9c0: ffff88042fc0ab48 ffffffff823af620 0000000000000000 0000000000000000
ffff88042fc0a9e0: 0000000041b58ab3 ffffffff83a40559 ffffffff823af530 0000000000000000
ffff88042fc0aa00: 656863735f787463 3478302b6e695f64 61353178302f6534 303578302f000030
ffff88042fc0aa20: 0000300030650000 0000000000000000 0000000000000000 0000000000000000
ffff88042fc0aa40: ffff0a00ffffff05 ffff88042fc0ab25 ffff88042fc0ae7a ffff88042fc0ab26
ffff88042fc0aa60: ffff88042fc0af30 ffff88042fc0ad30 ffffffff823adb30 1ffff10085f81558
ffff88042fc0aa80: 00000000ffffffff 0020f10085f8155c ffff0a00ffffff00 1ffff10000000010
ffff88042fc0aaa0: ffff88042fc0ab00 ffff88042fc0aa80 ffff88042fc0b010 ffff103000001005
ffff88042fc0aac0: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffffffffffffffff
ffff88042fc0aae0: 3266633061616330 6666666638383034 ffffffff00000010 0000000000000000
ffff88042fc0ab00: ffff88042fc24660 ffff88042fc0af10 ffff103000001009 0000000041b58ab3
ffff88042fc0ab20: ffff88042fc0af30 ffff88042fc0ad75 ffff88042fc0ad30 1ffff10085f81570
ffff88042fc0ab40: ffffffffffffffff 002088042fc0ac08 ffffffff823b6600 ffffffff00000010
ffff88042fc0ab60: ffff88042fc0ad00 ffffffff00000010 ffff88042fc0ad10 ffff103000001009
ffff88042fc0ab80: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffffffff839f498b
ffff88042fc0aba0: 3833396634393862 6666666666666666 ffff88042fc0ac87 ffff88042fc0ac08
ffff88042fc0abc0: ffffffff823ad785 ffff88042fc0acc8 0000000000000001 ffff88042fc0ad85
ffff88042fc0abe0: ffff88042fc0ac08 ffffffff816213e5 ffff88042fc0ad85 ffffffff839f498f
ffff88042fc0ac00: dffffc0000000000 ffff88042fc0ace8 ffffffff823b72eb ffff88042fc0b028
ffff88042fc0ac20: ffff88042fc0b030 ffffffff839f4990 ffff88042fc0ad30 1ffff10085f8158c
ffff88042fc0ac40: ffffed0085f81606 0000000000000200 ffff88042fc0af30 0000000000000001
ffff88042fc0ac60: 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0 ffff88042fc0acc8
ffff88042fc0ac80: ffff103000001000 1ffff10085f81595 ffffffff83c88b40 ffff88042fc0af10
ffff88042fc0aca0: ffffffff81338af4 0000000041b58ab3 ffff88042fc0acd0 ffffffff811c0a51
ffff88042fc0acc0: ffff88042fc0ad86 dffffc0000000000 ffff88042fc0ad00 ffffffff811c0afc
ffff88042fc0ace0: ffff88042fc0ad30 00000000ffffffff 1ffff10085f815a2 ffffffff83c88b40
ffff88042fc0ad00: ffff88042fc0af78 ffffffff81338af4 0000000041b58ab3 ffffffff83a03c41
ffff88042fc0ad20: ffffffff81338a00 ffff0a00ffffff00 3430383866666666 3030646130636632
ffff88042fc0ad40: 666666666666203a 6138333331386666 3066666666203030 6666666666303061
ffff88042fc0ad60: 3636303320363636 3032363636363636 3636362030333033 3336363636363636
ffff88042fc0ad80: 1f000a3633363336 ffffffff83c88b40 ffff88042fc0b008 ffffffff81338af4
ffff88042fc0ada0: 0000000041b58ab3 ffffffff83a03c41 ffffffff81338a00 0000000041b58ab3
ffff88042fc0adc0: 6177647261486401 3a656d616e206572 43206c65746e4920 697461726f70726f
ffff88042fc0ade0: 3030363253206e6f 30303632532f5a47 534f4942202c5a47 3030364335455320
ffff88042fc0ae00: 2e32302e4236382e 2e323030302e3230 3331303233323231 2f32312030313231
ffff88042fc0ae20: 0a333130322f3332 3d3d3d3d3d3d3d00 6f20646165523301 3820657a69732066
ffff88042fc0ae40: 6b73617420796220 75665f6672657020 3631312f72657a7a 5f746567000a3938
ffff88042fc0ae60: 615f6e7275746572 302b737365726464 3278302f62663178 6461207461203032
ffff88042fc0ae80: 3866666666207264 6230636632343038 000000000a306161 1ffff10085f815d7
ffff88042fc0aea0: ffffffff83c88b40 ffff88042fc0b120 ffffffff81338af4 0000000041b58ab3
ffff88042fc0aec0: ffffffff83a03c41 ffffffff81338a00 0000000000000000 4e494e5241573401
ffff88042fc0aee0: 6365726e75203a47 00000000ffffffff 0000000000000000 ffff88042fc0bef8
ffff88042fc0af00: ffffffff8342d740 ffff88042fc0afc0 ffff88042fc0af90 ffffffff8133c807
ffff88042fc0af20: 3430383866666666 3861636230636632 66726570206e6920 3a72657a7a75665f
ffff88042fc0af40: 00000a3938363131 0000000000000000 00000000ffffffff 0000000000000000
ffff88042fc0af60: ffff88042fc0bef8 ffffffff839f4970 ffff88042fc0b028 ffff88042fc0aff8
ffff88042fc0af80: ffffffff8133c807 0000000000000000 ffff88042fc0b040 ffffffff8151fea9
ffff88042fc0afa0: 0000000041b58ab3 ffffffff839ff69c ffffffff8151fdf3 0000000000000000
ffff88042fc0afc0: 0000000000000000 000000002fc0b050 1ffff10085f81601 0000000000000000
ffff88042fc0afe0: ffff88042fc0bef8 ffff88041b9a3d00 ffff88042fc0c000 ffff88042fc0b0a8
ffff88042fc0b000: ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c ffffffff8151fdf3
ffff88042fc0b020: ffff88042fc0b110 ffff880400000030 ffff88042fc0b0b8 ffff88042fc0b068
ffff88042fc0b040: ffff88042fc0bef8 ffff88041b9a3d00 ffff88042fc0bff8 ffff88042fc0b080
ffff88042fc0b060: ffffffff813b9d58 0000000000000000 ffff88042fc0b040 ffff88042fc0bef8
ffff88042fc0b080: 0000000000000000 ffff88042fc0b040 ffff88042fc0bef8 ffff88042fc0b060
ffff88042fc0b0a0: 0000000000000009 ffff88042fc0b178 ffffffff811504e6 ffffffff814f6efe
ffff88042fc0b0c0: ffffffff839fbd23 000000002fc0b0f8 0000000000000020 ffffffff839fbd5b
ffff88042fc0b0e0: ffffffff839fbd1f 0000000000000005 ffff88042fc0a000 ffff88042fc0c000
ffff88042fc0b100: ffff88042e7d7a58 0000000000000001 ffff88042e7d0000 ffff88042e7d8000
ffff88042fc0b120: 0000000000000000 0000000000000022 ffff88041b9a3d00 0000000000000000
ffff88042fc0b140: ffff88042e7d7ac0 0000000000000000 0000000000000086 0000000000000000
ffff88042fc0b160: ffff88042fc0b420 ffffed0085f81685 dffffc0000000000 ffff88042fc0b188
ffff88042fc0b180: ffffffff811505c4 ffff88042fc0b1a8 ffffffff8238dfb3 ffff88042fc0b238
ffff88042fc0b1a0: ffff88042fc0baa0 ffff88042fc0b228 ffffffff816221c1 ffffffff839ff69c
ffff88042fc0b1c0: ffff88042fc0b2c0 ffff88042fc0b2b8 0000000000000097 0000000000000001
ffff88042fc0b1e0: 00000000000000c0 ffff88042fc0bef8 ffff88042fc0b9d8 ffff88042fc0bef8
ffff88042fc0b200: ffff88042fc0b310 ffff88042fc0baa0 ffff88042fc0b420 ffffed0085f81685
ffff88042fc0b220: dffffc0000000000 ffff88042fc0b260 ffffffff81622323 ffff88042fc0baa0
ffff88042fc0b240: ffff88042fc0baa0 0000000000000008 ffff88042fc0b200 ffffffff811c72cb
ffff88042fc0b260: ffff88042fc0b2a0 ffffffff811c72cb ffff88042fc0b320 ffff88042fc0b310
ffff88042fc0b280: ffff88042fc0b310 ffff88042fc0b390 ffff88042fc0b420 ffffed0085f81685
ffff88042fc0b2a0: ffff88042fc0b3b8 ffffffff8100f356 ffff880710753b80 ffff88042fc0b5b0
ffff88042fc0b2c0: 0000000041b58ab3 ffff88042fc0b42c 1ffff10085f8165e ffff88042fc0b432
ffff88042fc0b2e0: 0000000000000004 ffff88042fc0b428 0000000041b58ab3 ffffffff839f613c
ffff88042fc0b300: ffffffff8100f000 ffffffff823bbf3c 0000000000000005 ffff88042fc0a000
ffff88042fc0b320: ffff88042fc0c000 ffff88042e7d7a58 0000000000000020 ffff88041b9a3d00
ffff88042fc0b340: 0000000000000000 ffff88042fc0ba98 0000000000000000 ffff88042fc0b4e8
ffff88042fc0b360: ffff88042fc0b536 ffffffff823ad258 0000000000000002 ffffffff823ad25a
ffff88042fc0b380: ffff88042fc0b3c0 ffffffff823bc4f2 000000000001bdc0 ffff88042fc0b460
ffff88042fc0b3a0: ffff880407f9adc0 1ffff10085f81680 ffff880407f9adc0 ffff88042fc0b488
ffff88042fc0b3c0: ffffffff81513f16 ffffffff823bca8a ffff88042fc0b4e8 ffff880400000000
ffff88042fc0b3e0: ffff88042fc0b6e0 ffff880400000000 ffffffff00000001 000000000000000c
ffff88042fc0b400: 0000000041b58ab3 ffffffff83a0a369 ffffffff81513ca0 ffff880710753b80
ffff88042fc0b420: ffff880407f9adc0 000000010000007f ffff88042f000001 ffffffff812c58ec
ffff88042fc0b440: ffff88042fc0b536 ffffffff823ad258 0000000000000002 ffff880710753b80
ffff88042fc0b460: ffff880710753b80 0000000000000040 ffff88042fc0b6e0 ffff880708c8f700
ffff88042fc0b480: 0000000000000000 ffff88042fc0b4d0 ffffffff81514438 ffffffff00000001
ffff88042fc0b4a0: 1ffff10085f81699 ffff88042fc0b574 ffff880710753b80 ffff88042fc0b570
ffff88042fc0b4c0: ffff880710753b80 ffff88042fc0b7c0 ffff88042fc0b540 ffffffff8150ac7e
ffff88042fc0b4e0: ffff880710753b80 0000000100000000 0000000100000000 0000000100000000
ffff88042fc0b500: 000000000002832d ffff88042fc0b576 ffff88042fc0b6e0 1ffff10085f816aa
ffff88042fc0b520: ffff880710753b80 ffff88042fc0b7c0 ffff88042fc0b6e0 dffffc0000000000
ffff88042fc0b540: ffff88042fc0b630 ffffffff8150b7d3 0000000041b58ab3 ffffffff83a0d7b0
ffff88042fc0b560: ffffffff8150b740 ffff880710753b80 0040000100000009 00000000000000a8
ffff88042fc0b580: ffff88042fc0b6e0 ffff88042fc0b6e0 ffff88042c6502c0 ffff88042fc0b7c0
ffff88042fc0b5a0: ffff880710753b80 000000000002832d ffff88042fc0b630 ffffffff81027da8
ffff88042fc0b5c0: ffff880710753d48 0000000000000000 ffffffffffffff99 ffff88042fc0b770
ffff88042fc0b5e0: ffffffff84156db0 ffff88042fc0b7d0 ffff88042fc13440 ffff880710753c68
ffff88042fc0b600: ffff880400000000 ffff88042c6502c0 ffff88042c6502c0 ffff88042fc0b7c0
ffff88042fc0b620: 0000000000000001 ffff880710753b80 ffff88042fc0b9b0 ffffffff81028f01
ffff88042fc0b640: 0000000000000000 ...
ffff88042fc0b660: 0000000000000000 1ffff10085f816d8 1ffffffff082adb6 0000000000000007
ffff88042fc0b680: fffffbfff082adb7 0000000000000000 ffff88042fc0b7c0 ffff88042fc0bef8
ffff88042fc0b6a0: ffff880710753b80 000000000000007b ffff88042c655750 ffff88042fc0b6e0
ffff88042fc0b6c0: 0000000041b58ab3 ffffffff839f20b0 ffffffff81028c50 0000000000000000
ffff88042fc0b6e0: ffff88042fc24660 ffff88070ee34664 0000000000000000 ffff88070ee34400
ffff88042fc0b700: ffff88042fc0ba98 dffffc0000000000 1ffff10085f826ec ffffffff83d3e060
ffff88042fc0b720: ffff88041b9a4598 0000000000000000 0000000000024660 ffff88041b9a3d00
ffff88042fc0b740: 1ffff100e1dc6908 1ffff10085f848e8 ffff88070ee34840 ffffffffffffffff
ffff88042fc0b760: ffffffff811c09f0 0000000000000010 0000000000000002 ffff88042fc0ba80
ffff88042fc0b780: 0000000000000000 ffffffff839f2108 ffffffff8102b200 0000000000000000
ffff88042fc0b7a0: 0000000000000000 ...
ffff88042fc0b7c0: 0000000000000000 0000000000000000 0000000000000000 00000000000000bc
ffff88042fc0b7e0: 0000000000000000 0000000000000000 0000000005080021 000000000002832d
ffff88042fc0b800: ffffffff811c09f0 0000000000000000 0000002a5c7c1bbd 0000000000000000
ffff88042fc0b820: 000000000004f8a6 ffff88042fc0b9c0 ffff880407f9adc0 ffffffff823bb28d
ffff88042fc0b840: 0000000000000000 ...
ffff88042fc0b860: ffff88042fc0b9c0 ffff88042fc0ba18 ffff88042fc0ba20 ffff88042fc0b975
ffff88042fc0b880: ffff88042fc0b9c4 ffff88042fc0b9c0 ffff88042fc0ba60 0000000000000002
ffff88042fc0b8a0: ffff88042fc0b9c4 ffff88042fc0b928 ffffffff823bbf3c ffffffff823ac07c
ffff88042fc0b8c0: ffff88042fc0b9c0 ffff88042fc0b9c0 ffff88042fc0ba20 ffff88042fc0ba08
ffff88042fc0b8e0: 0000000000000068 ffff88042fc0b910 ffff88042fc0b9d0 ffff88042fc0ba18
ffff88042fc0b900: ffff88042fc0b9c0 ffff88042fc0b9c0 ffff88042fc0ba60 0000000000000002
ffff88042fc0b920: 0000000000000002 ffff88042fc0b940 ffffffff823beb42 1ffff10085f8172c
ffff88042fc0b940: ffff88042fc0ba88 ffffffff8102cf02 ffffffff83a40503 ffffffff823ab960
ffff88042fc0b960: 0000000041b58ab3 ffffffff839f711e ffffffff8102cde0 0000000000000000
ffff88042fc0b980: 0000000000000000 0000000000000000 dffffc0000000000 ffff88042fc0bb08
ffff88042fc0b9a0: ffff880710753b80 ffff88042fc13440 ffff88042fc0bb30 ffffffff81029856
ffff88042fc0b9c0: 0000000100000000 0000000100000000 ffff88042fc13d78 ffff88042fc0bef8
ffff88042fc0b9e0: 1ffff10085f81745 ffff88042c650000 ffff88042fc0ba88 0000000000000007
ffff88042fc0ba00: ffffed0085f827af ffff88042c655750 fffffbfff082adbc ffff88042fc0bac8
ffff88042fc0ba20: ffff88042fc0ba88 0000000041b58ab3 ffffffff839f20d0 ffffffff81029260
ffff88042fc0ba40: 0000000000000000 0000000000000001 0000000000000000 0000000000000000
ffff88042fc0ba60: ffff88042fc13f00 000000000000000f ffff88042fc13440 dffffc0000000000
ffff88042fc0ba80: 000000000000000f 000000000000007f 0000000000000000 ffff88042fc0bac8
ffff88042fc0baa0: ffffffff811c0afc fffffbfff082adc4 00000003ffffffff 1ffff10085f8175b
ffff88042fc0bac0: ffffffff83c88b40 0000000000000000 0000000000000000 ffff88042fc13d90
ffff88042fc0bae0: ffffffff00000000 ffffffff00029ffe 0000000000000000 ffffed0085f827b2
ffff88042fc0bb00: ffff88042fc13f10 0000000000000001 0000002a5c7bda16 ffffffff83c16120
ffff88042fc0bb20: 0000002a5c7bda04 ffff88042fc13440 ffff88042fc0bdf8 ffffffff8101e752
ffff88042fc0bb40: ffff88042fc13d78 1ffff10085f81770 ffffed0085f827af 0000006400000000
ffff88042fc0bb60: 0000000000000000 ffff88042fc0bef8 0000000000000000 ffff88042fc0bd60
ffff88042fc0bb80: 0000000041b58ab3 ffffffff839f1dd0 ffffffff8101e2a0 ffff88042fc0bc40
ffff88042fc0bba0: ffffffff8256fd3e ffffc900000c5000 1ffff10085f81777 0000000041b58ab3
ffff88042fc0bbc0: 0000000000000000 ffffffff8256fb16 0000000000000000 0000000000000000
ffff88042fc0bbe0: 0000000000000000 ...
ffff88042fc0bc00: 0000000000000000 0000000000000000 0000000000000000 1ffff10085f8178a
ffff88042fc0bc20: ffff88042fc0bcf0 ffffc900000cff94 ffff88042fc0bd60 dffffc0000000000
ffff88042fc0bc40: ffff88072ec02000 ffff88042fc0bcb8 ffffffff815e0639 ffff88072ec02000
ffff88042fc0bc60: ffff88042f62d000 ffffc900000e9fff ffffc900000ea000 ffffffff83c0bc98
ffff88042fc0bc80: ffffc900000e9fff ffffc900000ea000 ffffc900000e9000 0000000000000000
ffff88042fc0bca0: ffff88072eabc4e0 00000000bdceb03c ffff880723dda214 ffff88042fc0bcc8
ffff88042fc0bcc0: ffffc900000e9000 ffff88042fc0bd18 ffffffff82609379 0000000000000014
ffff88042fc0bce0: 0000000000000000 0010000000000001 ffff88072eaca300 1ffff10085f817a8
ffff88042fc0bd00: ffffc900000cff80 ffff88042fc0bda0 0000000000000001 ffff88042fc0bdc8
ffff88042fc0bd20: ffffffff826095be ffffffff83c16120 0000002a5b4aac17 ffff88072eaca308
ffff88042fc0bd40: 0000000041b58ab3 ffffffff83a49886 ffffffff826094a0 ffffffff83a03c64
ffff88042fc0bd60: 00000000bdceb028 0000000000000000 0000000000000020 ffff88042fc0bdf8
ffff88042fc0bd80: ffff88042fc0bdb0 ffff88042fc0bdb0 ffffffff84c270c0 dffffc0000000000
ffff88042fc0bda0: ffff88072eaca300 dffffc0000000000 ffffffff83f6e7c0 0000000000000000
ffff88042fc0bdc0: ffffffff83f6ea20 ffff88042fc0be18 ffff88042fc0bef8 0000002a5c7bda16
ffff88042fc0bde0: ffffffff83c16120 0000002a5c7bda04 ffffffff83c16130 ffff88042fc0be18
ffff88042fc0be00: ffffffff8100a25d dffffc0000000000 0000000000000001 ffff88042fc0be80
ffff88042fc0be20: ffffffff81150f5e 0000000000000000 0000000000000000 ffff88042fc0bef8
ffff88042fc0be40: 0000000000000000 ffffffff83c64ce8 0000000000000587 ffff88042fc0bef8
ffff88042fc0be60: 0000000000000001 0000000000000007 ffff88041b9a447c ffff88042fc0bf90
ffff88042fc0be80: ffff88042fc0bea8 ffffffff81151b41 ffff88041b9a3d00 ffff88042fc0bef8
ffff88042fc0bea0: ffff88041b9a3d00 ffff88042fc0bee8 ffffffff81151d52 0000000000000000
ffff88042fc0bec0: 0000000000000001 ffff88070ee34400 0000000000000000 ffff88070ee34664
ffff88042fc0bee0: ffff88042fc24660 ffff88042fc0bef9 ffffffff832ad08e ffff88042fc24660
ffff88042fc0bf00: ffff88070ee34664 0000000000000000 ffff88070ee34400 ffff88042e7d7ac0
ffff88042fc0bf20: dffffc0000000000 1ffff10085f826ec ffffffff83d3e060 ffff88041b9a4598
ffff88042fc0bf40: 0000000000000000 0000000000024660 ffff88041b9a3d00 1ffff100e1dc6908
ffff88042fc0bf60: 1ffff10085f848e8 ffff88070ee34840 ffffffffffffffff ffffffff814f6efe
ffff88042fc0bf80: 0000000000000010 0000000000000046 ffff88042e7d7a58 0000000000000000
ffff88042fc0bfa0: ffffffff814f6efe 0000000000000010 0000000000000046 ffff88042e7d7a58
ffff88042fc0bfc0: 0000000000000000 0000000000000001 1ffff100e1dc6908 ffffffff814f6efe
ffff88042fc0bfe0: 0000000000000010 0000000000000046 ffff88042e7d7a58 0000000000000000
d ? sched_clock_cpu+0x11c/0x1a0
d perf_event_sched_in+0x60/0x80
d __perf_event_task_sched_in+0x381/0x6e0
d ? perf_sched_cb_inc+0x190/0x190
d finish_task_switch+0x347/0x570
d __schedule+0x69c/0x1760
d ? __send_signal+0x7ff/0xea0
d ? io_schedule_timeout+0x3a0/0x3a0
d ? release_task+0x1230/0x1230
d ? send_signal+0x5f/0xb0
d ? do_send_sig_info+0xc9/0x120
d schedule+0x94/0x1b0
d do_wait+0x5b2/0x710
d ? wait_consider_task+0x3580/0x3580
d ? syscall_exit_register+0x4f0/0x4f0
d ? SYSC_kill+0x1a3/0x450
d SyS_wait4+0xd3/0x170
d ? SyS_waitid+0x1d0/0x1d0
d ? task_stopped_code+0x100/0x100
d ? syscall_slow_exit_work+0x138/0x1e0
d ? SyS_waitid+0x1d0/0x1d0
d do_syscall_64+0x17f/0x380
d entry_SYSCALL64_slow_path+0x25/0x25
dRIP: 0033:0x7f614ef57afa
dRSP: 002b:00007fff4ff89d58 EFLAGS: 00000246c ORIG_RAX: 000000000000003d
dRAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f614ef57afa
dRDX: 0000000000000000 RSI: 00007fff4ff89d6c RDI: 0000000000003176
dRBP: 00007fff4ff89d70 R08: 00007f614f23c0a4 R09: 00007f614f23c120
dR10: 0000000000000000 R11: 0000000000000246 R12: 0000000000401810
dR13: 00007fff4ff8c0f0 R14: 0000000000000000 R15: 0000000000000000
ffff88042e7d0000: 0000000057ac6e9d 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d0020: 0000000000000000 ...
ffff88042e7d0820: 0000000000000000 0000000000000000 e280002c00000001 e460002c00000001
ffff88042e7d0840: ffff88042e7d07e8 00000000000c010d ffffc9000013103a ffff88042e7d0898
ffff88042e7d0860: 0000000000000000 0000000010001002 0000000000000000 9c80000300000000
ffff88042e7d0880: a480000300000000 0000000000000000 0000000000000000 ffff88042e7d0840
ffff88042e7d08a0: 00000000002d010d ffffc9000013103f ffff88042e7d0000 0000000000000000
ffff88042e7d08c0: 0000000000000000 ...
ffff88042e7d10e0: e280002c00000001 e460002c00000001 0000000000000000 ffff88042e7d0d68
ffff88042e7d1100: ffffffffffffffff ffffc90000131073 00000000000001ae 0000000000000000
ffff88042e7d1120: 0000000000000000 0000000000000000 9e20000300000000 a140000300000000
ffff88042e7d1140: 0000000000000000 0000000000000000 ffff88042e7d10e0 0000000000000007
ffff88042e7d1160: ffff88042e7d08b0 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d1180: 0000000000000000 ...
ffff88042e7d1980: 0000000000000000 0000000000000000 e280002c00000001 e460002c00000001
ffff88042e7d19a0: ffff88042e7d1690 0000000000000007 0000000000000000 ffff88042e7d2738
ffff88042e7d19c0: ffffffffffffffff ffffc900001310b3 00000000000001ae 0000000000000000
ffff88042e7d19e0: 0000000000000000 0000000000000000 9e20000300000000 a140000300000000
ffff88042e7d1a00: 0000000000000000 0000000000000000 ffff88042e7d1160 0000000000000000
ffff88042e7d1a20: 0000000000000000 ...
ffff88042e7d2240: e280002c00000001 e460002c00000001 a480000300000000 0000000000000000
ffff88042e7d2260: 0000000000000000 ffff88042e7d2370 00000000007b010d ffffc90000131076
ffff88042e7d2280: 0000000000000000 0000000000000000 ffff88042e7d22c0 0000000000000003
ffff88042e7d22a0: 9c80000300000000 a480000300000000 0000000000000000 0000000000000000
ffff88042e7d22c0: ffff88042e7d1a10 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d22e0: 0000000000000000 ...
ffff88042e7d2ae0: 0000000000000000 0000000000000000 e9c0003800000001 eba0003800000001
ffff88042e7d2b00: ffff88042e7d2948 00000000002d010d ffffc900001310af 0000000000000000
ffff88042e7d2b20: 0000000000000000 ffffc900001310af 0000000000000000 9c80000300000000
ffff88042e7d2b40: a480000300000000 0000000000000000 0000000000000000 ffff88042e7d2f78
ffff88042e7d2b60: 0000000000a0010d ffffc900001310b3 ffff88042e7d22c0 0000000000000000
ffff88042e7d2b80: 0000000000000000 ...
ffff88042e7d33a0: e9c0003800000001 eba0003800000001 0000000000000000 0000000000000000
ffff88042e7d33c0: 0000000000000000 0000000000000000 9e20000300000000 a140000300000000
ffff88042e7d33e0: 0000000000000000 0000000000000000 ffff88042e7d35b0 0000000000000007
ffff88042e7d3400: 00000000fffffffd ffff88042e7d4058 ffffc900001310f6 ffffc900001310f6
ffff88042e7d3420: ffff88042e7d2b70 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d3440: 0000000000000000 ...
ffff88042e7d3c40: 0000000000000000 0000000000000000 e9c0003800000001 eba0003800000001
ffff88042e7d3c60: ffffffffffffffff ffffc90000131143 000000000000000f 0000000000000000
ffff88042e7d3c80: 0000000000000000 0000000000000000 9e20000300000000 a140000300000000
ffff88042e7d3ca0: 0000000000000000 0000000000000000 ffff88042e7d3d90 0000000000000007
ffff88042e7d3cc0: 0000000000000000 ffff88042e7d4c60 ffff88042e7d3420 0000000000000000
ffff88042e7d3ce0: 0000000000000000 ...
ffff88042e7d4500: e9c0003800000001 eba0003800000001 a480000300000000 0000000000000000
ffff88042e7d4520: 0000000000000000 ffff88042e7d47e8 0000000000a0010d ffffc900001310f6
ffff88042e7d4540: 0000000000000000 0000000000000000 0000000000000000 0000000000000002
ffff88042e7d4560: 9c80000300000000 a480000300000000 0000000000000000 0000000000000000
ffff88042e7d4580: ffff88042e7d3cd0 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d45a0: 0000000000000000 ...
ffff88042e7d4da0: 0000000000000000 0000000000000000 e9c0003800000001 eba0003800000001
ffff88042e7d4dc0: ffff88042e7d4d68 00000000002d010d ffffc90000131130 0000000000000000
ffff88042e7d4de0: 0000000000000000 0000000000000000 0000000000000000 aee0000300000000
ffff88042e7d4e00: a480000300000000 0000000000000000 0000000000000000 ffff88042e7d4cb8
ffff88042e7d4e20: 000000000036010d ffffc90000131126 ffff88042e7d4580 0000000000000000
ffff88042e7d4e40: 0000000000000000 ...
ffff88042e7d5660: e9c0003800000001 eba0003800000001 9e20000300000000 a140000300000000
ffff88042e7d5680: 0000000000000000 0000000000000000 ffff88042e7d53f0 0000000000000007
ffff88042e7d56a0: 0000000000000000 ffff88042e7d6738 ffffffffffffffff ffffc90000131170
ffff88042e7d56c0: 0000000000000000 ...
ffff88042e7d56e0: ffff88042e7d4e30 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d5700: 0000000000000000 ...
ffff88042e7d5f00: 0000000000000000 0000000000000000 e9c0003800000001 eba0003800000001
ffff88042e7d5f20: 0000000000000000 0000000000000000 9e20000300000000 a140000300000000
ffff88042e7d5f40: 0000000000000000 0000000000000000 ffff88042e7d5ee0 0000000000000007
ffff88042e7d5f60: 0000000000000000 ffff88042e7d6f78 ffffffffffffffff ffffc900001311ea
ffff88042e7d5f80: 000000000000000d 0000000000000000 ffff88042e7d56e0 0000000000000000
ffff88042e7d5fa0: 0000000000000000 ...
ffff88042e7d60a0: 0000000000000000 ffff88042e7d6198 ffff88042e7d6190 ffff88042e7d61f8
ffff88042e7d60c0: 0000000000000001 ffff88042e7d61f8 ffff88042e7d61d0 ffff88042e7d6130
ffff88042e7d60e0: ffffffff811c75da ffff88042e7d6228 0000000000000000 ffff88042e7d6198
ffff88042e7d6100: ffff88042e7d6190 ffff88042e7d7f48 0000000000000001 ffff88042e7d7f48
ffff88042e7d6120: ffff88042e7d61d0 ffff88042e7d6180 ffffffff811c75da ffff88042e7d6190
ffff88042e7d6140: ffff88042e7d6180 ffffffff811c71d2 ffff88042e7d61b8 ffff88042e7d6218
ffff88042e7d6160: ffff88042e7d6218 0000000000000000 0000000000000000 ffff88041e8a2dc0
ffff88042e7d6180: ffff88042e7d61f8 ffffffff8116dfd3 0000000000000001 ffffffff832ab679
ffff88042e7d61a0: 000000002c780c3a 000000000000002e ffff88042e7d6208 ffffffff8242feaa
ffff88042e7d61c0: 0000000000000190 ffff88042e7d6230 0000000000080c3a 0000000000000000
ffff88042e7d61e0: ffff88042e7d62d0 ffff88042e7d62c8 ffff88042e7d6330 0000000000000001
ffff88042e7d6200: ffff88042e7d6330 ffff88042e7d6308 ffff88042e7d6268 ffffffff811c75da
ffff88042e7d6220: ffffffff811c75da 0000000000000000 ffff88042e7d62d0 ffff88042e7d62c8
ffff88042e7d6240: ffff88042e7d7f48 0000000000000001 ffff88042e7d7f48 ffff88042e7d6308
ffff88042e7d6260: ffff88042e7d62b8 ffffffff811c75da ffff88042e7d62c8 ffff88042e7d62b8
ffff88042e7d6280: ffffffff811c71d2 ffff88042e7d62f0 ffff88042e7d6350 ffff88042e7d6350
ffff88042e7d62a0: 0000000000000000 0000000000000000 ffff88041b9a3d00 ffff88042e7d6330
ffff88042e7d62c0: ffffffff8116dfd3 00000000a897a839 0000000000000120 ffff88042e7d6368
ffff88042e7d62e0: ffff8807248d6bc0 ffff88042e7d6340 ffffffff8242fc8f 0000000002408240
ffff88042e7d6300: ffff88042e7d6350 000000000007a839 0000000000000000 0000000002408240
ffff88042e7d6320: ffff88072865fbf0 0000000002408240 ffff88072865fbf0 ffff880427cb6e80
ffff88042e7d6340: ffff88042e7d6570 ffffffff81621065 0000004000000024 ffff88042e7d6368
ffff88042e7d6360: 0000000000000000 ffffffff8116e03b ffffffff81620ff6 ffffffff8162126d
ffff88042e7d6380: ffffffff816217d2 ffffffff8161d1e0 ffffffff81ed2aa1 ffffffff81ed00cd
ffff88042e7d63a0: ffffffff81f13e2b ffffffff81f14af4 ffffffff81da4df2 ffffffff81da51dc
ffff88042e7d63c0: ffffffff81dbe1a2 ffffffff81d425f2 ffffffff81d43145 ffffffff81d4ce2d
ffff88042e7d63e0: ffffffff81d4dfeb ffffffff81d50c7f ffffffff81d8f93d ffffffff81d91111
ffff88042e7d6400: ffffffff81d93171 ffffffff81e9fb75 ffffffff81e560ba ffffffff81e5a578
ffff88042e7d6420: ffffffff8154c563 ffffffff81e59732 ffffffff8154e6ea ffffffff81527f8a
ffff88042e7d6440: ffffffff8152814c ffffffff81eafe8d ffffffff81e7eee5 ffffffff8165fe8a
ffff88042e7d6460: ffffffff8166030e ffffffff8128a10a ffffffff81002a09 ffffffff81005265
ffff88042e7d6480: ffffffff832abcb8 ffff88042e7f42f0 ffffed0085cfe85e ffff880414af35a0
ffff88042e7d64a0: ffff88072865fbd0 1ffff10085cfaca7 ffff88070a606188 ffff88042e7f4260
ffff88042e7d64c0: ffff880414af3480 ffff88042e7d6518 ffffffff81f15847 ffff88042e7f4390
ffff88042e7d64e0: 0000000600000007 ffff880414af34ac ffff88070acf95f8 1ffff10085cfaca7
ffff88042e7d6500: ffff88042e7d65d8 ffff880414af3480 ffff8804270170b0 ffff88042e7d65f8
ffff88042e7d6520: ffffffff81dab421 0000000000000286 ffff88042e7f42f0 0000000041b58ab3
ffff88042e7d6540: ffffffff83a33fb7 ffffffff81dab2f0 00000000000009af 1ffff10085cfacaf
ffff88042e7d6560: ffff88042e7d65f8 ffffffff81d9fb81 ffff88042e7d65c0 0000000041b58ab3
ffff88042e7d6580: ffffffff83a34296 ffffffff81d9fa90 ffff880414af34ac ffff8807875c0000
ffff88042e7d65a0: 00000000000009af 0000000000000015 ffff880414af3480 ffff8804270170b0
ffff88042e7d65c0: ffff88042e7d65f8 ffffffff81dab2ca ffffffff81da10b9 ffff88042e7d6850
ffff88042e7d65e0: ffff88070acf95f8 ffff880427017000 0000000000000015 ffff88042e7d6878
ffff88042e7d6600: ffffffff81dc3893 ffffffff81ed2aa1 ffff88041b9a41c0 ffff88041b9a3d00
ffff88042e7d6620: 1ffff10083734838 ffff880427017006 ffff88070acf9600 ffff88042e7d6770
ffff88042e7d6640: ffff88042e7d69d0 ffff88042e7d6990 000000000359dc90 ffff88042e7d6a10
ffff88042e7d6660: ffff88070acf9608 1ffff10085cfacd2 ffff880700000133 ffff88042e7d6950
ffff88042e7d6680: ffff88070acf9610 ffff88042e7d6778 ffff88042e7d6770 ffff88042e7d67d8
ffff88042e7d66a0: 0000000000000001 ffff88042e7d67d8 ffff88042e7d67b0 ffff88042e7d6710
ffff88042e7d66c0: ffffffff811c75da ffff88042e7d6720 0000000000000000 ffff88042e7d6778
ffff88042e7d66e0: ffff88042e7d6770 ffff88042e7d7f48 0000000000000001 ffff88042e7d7f48
ffff88042e7d6700: ffff88042e7d67b0 ffff88042e7d6760 ffffffff811c75da ffff88042e7d6770
ffff88042e7d6720: ffff88042e7d6810 ffff88042e7d6808 ffff88042e7d6870 0000000000000001
ffff88042e7d6740: ffff88042e7d6870 ffff88042e7d6848 ffff88042e7d67a8 ffffffff811c75da
ffff88042e7d6760: ffff88042e7d67d8 0000000000000000 ffff88042e7d6810 ffff88042e7d6808
ffff88042e7d6780: ffff88042e7d7f48 0000000000000001 ffff88042e7d7f48 ffff88042e7d6848
ffff88042e7d67a0: ffff88042e7d67f8 ffffffff811c75da ffff88042e7d6808 ffff88042e7d67f8
ffff88042e7d67c0: ffffffff811c71d2 ffff88042e7d6830 ffff88042e7d6890 ffff88042e7d6890
ffff88042e7d67e0: 0000000000000000 0000000000000000 ffff88041b9a3d00 ffff88042e7d6870
ffff88042e7d6800: ffffffff8116dfd3 00000000931ae748 00000000000000d8 ffff88042e7d68a8
ffff88042e7d6820: ffff880716332570 ffff88042e7d6880 ffffffff8242fc8f 0000000002000000
ffff88042e7d6840: ffff88042e7d6890 00000000000ae748 0000000000000000 0000000002000000
ffff88042e7d6860: ffff88070acf9848 ffff88070acf9930 ffffea001c2b3e00 ffff88070acf9848
ffff88042e7d6880: ffff88042e7d6ab0 ffffffff81621065 000000400000001b ffff88042e7d68a8
ffff88042e7d68a0: ffffffff00000000 ffffffff8116e03b ffffffff81620ff6 ffffffff81621851
ffff88042e7d68c0: ffffffff8161d85e ffffffff81da8a5a ffffffff81d4ce60 ffffffff81d4dfeb
ffff88042e7d68e0: ffffffff81d50c7f ffffffff81d8f93d ffffffff81d91111 ffffffff81d93171
ffff88042e7d6900: ffffffff81e9fb75 ffffffff81e560ba ffffffff81e5a578 ffffffff8154c563
ffff88042e7d6920: ffffffff81e59732 ffffffff8154e6ea ffffffff81527f8a ffffffff8152814c
ffff88042e7d6940: ffffffff81eafe8d ffffffff81e7eee5 ffffffff8165fe8a ffffffff8166030e
ffff88042e7d6960: ffffffff8128a10a ffffffff81002a09 ffffffff81005265 ffffffff832abcb8
ffff88042e7d6980: ffffffffffffffff 0000000000000000 ffff88042e7d6a50 0000000041b58ab3
ffff88042e7d69a0: ffffffff83a33ff3 ffffffff81dbfe60 ffff880427cb6880 ffff880414af32c0
ffff88042e7d69c0: ffffffff8161d85e 0000000000000000 01000000e9500700 ffff88042e7d6a78
ffff88042e7d69e0: 0000000000000296 0000000000000000 0000000000000000 0000000000000000
ffff88042e7d6a00: ffff880427cb7000 ffff8807053fe010 ffff8807053fe100 ffff88042e7d6a40
ffff88042e7d6a20: ffffffff81621867 ffff8807053fe010 ffffffff81ee9d85 ffff880427cb7000
ffff88042e7d6a40: ffff88042e7d6a70 ffffffff8161d85e ffff8807053fe010 0000000000000282
ffff88042e7d6a60: ffff880715838ab0 ffff880715838ab8 ffff88042e7d6a98 0000000000000282
ffff88042e7d6a80: 0000000000000000 0000000000000000 0000000000000000 ffff880427cb6880
ffff88042e7d6aa0: ffff88070acf9848 ffff88070acf9930 ffff88042e7d6ad8 ffffffff81621867
ffff88042e7d6ac0: ffff88070acf9848 ffffffff81da8a5a ffff880427cb6880 ffff88042e7d6b08
ffff88042e7d6ae0: ffffffff8161d85e 0000000000000002 ffff88070acf98fd 1ffff100e159f31f
ffff88042e7d6b00: dffffc0000000000 ffff88042e7d6b58 ffffffff81da8a5a ffffffff81d53833
ffff88042e7d6b20: 000000000000001c ffffed00e159f309 ffff88042e7d71f8 0000000000000001
ffff88042e7d6b40: 0000000000000000 ffff88042e7d6fc8 ffff88070acf95f8 ffff88042e7d6ff0
ffff88042e7d6b60: ffffffff81d4ce60 ffff880400000001 ffff88042e7d6d08 ffffffff00000000
ffff88042e7d6b80: ffffffff81d8f93d ffffffff81d91111 ffffffff81d93171 ffffffff81e9fb75
ffff88042e7d6ba0: ffffffff81e560ba ffffffff81e5a578 000000028154c563 ffff88042e7d722c
ffff88042e7d6bc0: ffffed0085cfae4d ffff88042e7d7269 1ffff10085cfad85 ffff88042e7d7200
ffff88042e7d6be0: ffff88042e7d7228 ffff88042e7d7208 ffff88042e7d7254 ffff88042e7d7234
ffff88042e7d6c00: ffff88042e7d7250 ffffffff81005265 ffff88042e7d6d88 ffff88040000000a
ffff88042e7d6c20: ffff880700000002 0000000041b58ab3 ffffffff83a1d770 ffffffff81d4b4f0
ffff88042e7d6c40: 1ffff10083734838 ffff8807000750e8 ffff88070a606188 0000000000000001
ffff88042e7d6c60: ffff8807053fdee0 ffff880715f6c8c0 ffff88042e7d6cb0 ffffffff81ed00cd
ffff88042e7d6c80: ffff880715f6c9b8 ffff8807000750e8 ffff88070a606188 0000000000000001
ffff88042e7d6ca0: 1ffff10085cfadad ffff880715f6c8c0 ffff88042e7d6ce0 ffffffff81f13e2b
ffff88042e7d6cc0: ffff88070a606188 ffff880400000002 ffff88070a606188 0000000000000000
ffff88042e7d6ce0: ffff88042e7d6d28 ffffffff81f14af4 0000000100000000 ffff880400000000
ffff88042e7d6d00: ffff8804165fe600 0000000000000002 0000000000000000 0000000000000000
ffff88042e7d6d20: 0000000000000000 ffff880427cb7000 ffff8807053fdee0 ffff8807053fdfd0
ffff88042e7d6d40: ffff88042e7d6d68 ffffffff81621867 ffff8807053fdee0 ffffffff81ee9d85
ffff88042e7d6d60: ffff880427cb7000 ffff88042e7d6d98 ffffffff8161d85e ffff8807053fdee0
ffff88042e7d6d80: 0000000000000292 ffff880700000001 ffff880715f6c8f8 ffff88042e7d6dc0
ffff88042e7d6da0: ffffffff8131f879 ffff880715f6c908 ffff88042e7d6df8 ffffffff81e6e5f1
ffff88042e7d6dc0: ffff880400000000 ffff880400000001 ffff8807053fdf60 ffff88070a606188
ffff88042e7d6de0: ffff8807053fdee0 ffff88042e7d7030 ffff880715f6c8c0 ffff88042e7d6e30
ffff88042e7d6e00: ffffffff81f15244 ffff8804000750e8 ffff88070a606188 ffff8807053fc2f0
ffff88042e7d6e20: ffffed00e0a7f85e ffff88071567c820 ffff88072865fdb0 ffff88042e7d6f88
ffff88042e7d6e40: ffff88070a606188 ffff8807053fc260 ffff88071567c700 ffff88042e7d6ea8
ffff88042e7d6e60: ffffffff81f15847 ffff88042e7d6f70 0000003400000037 ffff88071567c72c
ffff88042e7d6e80: 0000000000000200 ffff88042e7d6f88 ffffffff81d47503 ffff88070a606188
ffff88042e7d6ea0: ffff88071567c700 ffff88042e7d6fb0 ffffffff81d4722c ffff88042e7d6fa8
ffff88042e7d6ec0: ffff88042e7d6fa0 ffff88042e7d7008 0000000000000001 ffff88042e7d7008
ffff88042e7d6ee0: ffff88042e7d6fe0 ffff88042e7d6f40 ffff88042e7d6fe0 ffff88042e7d6fd8
ffff88042e7d6f00: ffff88042e7d7040 0000000000000001 ffff88042e7d7040 ffff88042e7d7018
ffff88042e7d6f20: 0000000000000008 0000000000000000 ffff88042e7d7230 ffffffff81510c31
ffff88042e7d6f40: 0000000041b58ab3 ffffffff83a0d680 ffffffff814e3680 0000000000000001
ffff88042e7d6f60: 0000000000000008 0000000000000000 ffff88042e7d7270 ffffffff81510c31
ffff88042e7d6f80: ffff88042e7d6fd8 ffff88072bdf9314 0000000000000041 dffffc0000000000
ffff88042e7d6fa0: ffff88042b6dc428 ffff88072bdf9200 ffff88042e7d6fc0 ffffffff823d7aa5
ffff88042e7d6fc0: ffff88042e7d7010 ffffffff8101c610 ffffffff81510890 ffff880400000001
ffff88042e7d6fe0: ffff880400000000 0000000000000000 ffff88042e7d7010 ffffffff810191fa
ffff88042e7d7000: dffffc0000000000 ffff88042e7d7138 ffff88042e7d70f8 ffffffff8100cac2
ffff88042e7d7020: 0000000000000000 0000000002000000 ffff88042e7d7120 ffff88042e7d7118
ffff88042e7d7040: 0000000041b58ab3 ffffffff83a0d6b0 ffffffff814e3a20 ffff880700000000
ffff88042e7d7060: 0000000000000008 0000000000000000 ffff88042e7d7370 ffffffff81510c31
ffff88042e7d7080: ffffffff8100c720 ffff88072f313440 0000000200000002 ffffffff81009e18
ffff88042e7d70a0: 0000000000000008 0000000000000000 ffff88042e7d73b0 ffffffff81510c31
ffff88042e7d70c0: 0000000000000008 ffff88072f313660 ffff88042e7d70f8 ffffffff816213e5
ffff88042e7d70e0: 0000000000000008 0000000000000000 ffff88042e7d73f0 ffffffff81510c31
ffff88042e7d7100: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffff88071f2ac690
ffff88042e7d7120: 1ffff10085cfae28 ffffffe426b0e000 ffff8804148d8880 ffff88042e7d7420
ffff88042e7d7140: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 000000000094f188
ffff88042e7d7160: 1ffff10085cfae30 ffffffe426b0e000 ffff8804148d8880 ffff88042e7d7460
ffff88042e7d7180: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffff88042ca53600
ffff88042e7d71a0: ffff88042e7d71b0 ffffffff823d7aa5 ffff88042e7d7200 ffffffff8101c610
ffff88042e7d71c0: ffff88071f2ac400 0000000000000007 fffffbfff082b618 ffff88042a192c18
ffff88042e7d71e0: ffff88042f8d5ca0 0000000000000027 ffff88042e7d7200 ffffffff823d7aa5
ffff88042e7d7200: ffff88042e7d7240 ffffffff8238dbde 0000000000000050 ffff88042e7d7328
ffff88042e7d7220: ffff88042e7d7648 dffffc0000000000 ffff88042b742278 dffffc0000000000
ffff88042e7d7240: ffffffff00000000 ffff88042e7d74a8 ffff88042e7d7360 ffff88042e7d7268
ffff88042e7d7260: ffffffff823d7aa5 ffff88042e7d7388 ffffffff81009474 ffff88042e7d7288
ffff88042e7d7280: ffffffff00000004 ffff880400000002 dffffc0000000000 1ffff10085cfae54
ffff88042e7d72a0: 0000000041b58ab3 ffffffff839f60d6 ffffffff81009250 1ffff10085cfae58
ffff88042e7d72c0: 0000000041b58ab3 ffffffff83a0d680 ffffffff814e3680 0000000300000003
ffff88042e7d72e0: 0000000300000000 0000000000000003 0000000000000001 1ffff10085cfae60
ffff88042e7d7300: 0000000041b58ab3 ffffffff83a0dbdc ffffffff815397d0 ffff88042b746650
ffff88042e7d7320: ffff88042ca51f80 ffff88042e7d7338 ffffffff823d7aa5 0000000000000001
ffff88042e7d7340: ffffffffffffffff 0000000000000000 0000000000000246 ffff88042ca50480
ffff88042e7d7360: ffffffff81621106 0000000000000000 dffffc0000000000 ffff88042e7d7388
ffff88042e7d7380: ffffffff81621428 ffff88042e7d7470 ffffffff8153ccb5 ffff88043fffcb80
ffff88042e7d73a0: 0000000000000002 ffff880400000000 0000000000000001 dffffc0000000000
ffff88042e7d73c0: ffffea0010579e00 0000000000000000 ffff88042e7d7498 ffff88042e7d7508
ffff88042e7d73e0: ffff88042e7d7430 ffffffff8153bf9c 0000000000000040 fffffbfff0826e1e
ffff88042e7d7400: ffff88043fffcb80 ffff88042e7d74f8 ffffea00104202c0 0000000000000246
ffff88042e7d7420: 0000000000000040 0000000000000000 ffff88042e7d7730 ffffffff81510c31
ffff88042e7d7440: ffff88043fffd700 0000000000000000 00000000024200ca ffff88042e7d74e8
ffff88042e7d7460: 00000000024200ca ffff88042e7d7548 ffff88042e7d7570 ffffffff815411d9
ffff88042e7d7480: ffffea0000000002 ffff88042e62ce68 0000000000000000 ffff88042e62cdc0
ffff88042e7d74a0: 1ffff10085cfae98 ffffffe4024000c0 ffff880717de6e80 ffff88042e7d77a0
ffff88042e7d74c0: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffffffff81540fc0
ffff88042e7d74e0: 1ffff10085cfaea0 ffff88043fffd700 0000000000000000 ffff88043fffd700
ffff88042e7d7500: ffff88042e7d75f0 ffff88042e7d75e8 ffff88042e7d7650 0000000000000000
ffff88042e7d7520: 1ffff10085cfaea5 0000000000000020 ffff88042e7d7830 ffffffff8151095b
ffff88042e7d7540: ffff88042e7d7580 ffff880400000000 ffff88042e7d7640 ffff88042e7d7638
ffff88042e7d7560: ffff88042e7d7650 ffff88042e7d7648 ffff88042e7d76b0 0000000000000001
ffff88042e7d7580: ffff88042e7d76b0 ffff88042e62ce68 0000000000000000 ffff88042e62cdc0
ffff88042e7d75a0: 1ffff10085cfaeb8 0000000000000000 ffff88070fc86e80 ffff88042e7d78a0
ffff88042e7d75c0: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 0000000041b58ab3
ffff88042e7d75e0: ffffffff83a0dbdc ffffffff815397d0 ffff88042e7d76e0 ffff88042e7d76d8
ffff88042e7d7600: ffff88042e7d7740 0000000000000001 ffff88042e7d7740 ffff88042aed2200
ffff88042e7d7620: dffffc0000000000 ffffffff00000000 ffff88042e7d7890 ffff88042e7d7748
ffff88042e7d7640: ffff88042e7d7650 ffffffff823d7aa5 ffff88042e7d7770 ffffffff8100954c
ffff88042e7d7660: ffff88042e7d7790 ffff880400000005 ffff880400000004 dffffc0000000000
ffff88042e7d7680: 1ffff10085cfaed1 0000000041b58ab3 ffffffff839f60d6 ffffffff81009250
ffff88042e7d76a0: ffffffff8101a831 0000000300000005 0000000000000004 ffff88042fc13b60
ffff88042e7d76c0: 0000000000000005 0000000000000021 0000000000000002 0000000200000003
ffff88042e7d76e0: 0000000000000000 ...
ffff88042e7d7720: ffff88042e7d7770 ffffffff8101c610 0000000000000001 ffff880700000002
ffff88042e7d7740: ffff880700000000 0000000000000001 ffff88042e7d7770 ffffffff810191fa
ffff88042e7d7760: dffffc0000000000 ffff88042e7d789c ffff88042e7d7858 ffffffff8100cac2
ffff88042e7d7780: ffff880710753b80 ffff880708c8f700 1ffffffff082adc1 0000000000029ffe
ffff88042e7d77a0: 1ffff10085cfaefa ffff88042fc13b60 ffff880700000003 ffff880400000000
ffff88042e7d77c0: ffff88042e7d7890 ffff88042fc13440 0000000041b58ab3 ffffffff839f6109
ffff88042e7d77e0: ffffffff8100c720 ffff88042fc13440 0000000200000000 ffffffff81009e18
ffff88042e7d7800: 0000000241b58ab3 0000000041b58ab3 ffffffff839f60e8 ffffffff81009c50
ffff88042e7d7820: 000000000000000c ffff88042fc13660 ffff88042e7d7858 ffffffff816213e5
ffff88042e7d7840: ffff88042fc13440 ffff88042fc13440 1ffff10085cfaf0e ffff88042e7d79d8
ffff88042e7d7860: ffffffff81009b68 ffffffff83c167c0 0000000041b58ab3 ffffffff839f60e8
ffff88042e7d7880: ffffffff810099d0 0000000000000000 0000000000000021 ffff880400000001
ffff88042e7d78a0: ffffffff8116dfd3 00000000e587263b 0000000000000078 ffff88042e7d7948
ffff88042e7d78c0: ffff880422a6e860 ffff88042e7d7920 ffffffff8242fc8f 0000000002280020
ffff88042e7d78e0: ffff88042e7d7930 000000000007263b 0000000000000000 ffff8807238e5500
ffff88042e7d7900: ffff880704d2d000 ffff8807238e56c8 ffff88042e7d7960 ffff8804274b0000
ffff88042e7d7920: ffff88042e7d7960 ffffffff8100ef1c ffff88071eebe768 ffff8807238e5500
ffff88042e7d7940: ffff880704d2d000 ffff8807238e56c0 ffff88042e7d79a0 ffff8804274b0000
ffff88042e7d7960: ffff88042e7d79a0 ffffffff8100ef1c ffff88042e7d79d8 ffff8807238e5500
ffff88042e7d7980: ffff8804274b0000 00000000023a9a15 ffff8807238e5500 ffff8804274b0000
ffff88042e7d79a0: ffff88042e7d7a00 ffffffff814ffc80 0000000000000000 ffff8807238e5578
ffff88042e7d79c0: 00000000b446421f 00000000b20bdf59 0000002a5b4a1cfa ffff88042fc13648
ffff88042e7d79e0: 0000000000000004 ffff880701d95900 0000000000000006 ffff88042e7d7a48
ffff88042e7d7a00: ffffffff812c58ec ffff88042e7d7a28 ffffffff8101d12c ffff8807238e5500
ffff88042e7d7a20: 0000000000000004 ffff880701d95900 0000000000000006 ffff88042fc24660
ffff88042e7d7a40: ffff880701d95978 ffff88042e7d7ac0 ffffffff814f798b ffffffff812c58ec
ffff88042e7d7a60: ffff88072f324660 ffff880701d95900 ffffed0085f848e9 ffff88042fc2474c
ffff88042e7d7a80: ffff880400000001 ffff88042fc24660 ffff880701d95950 ffff88042fc24660
ffff88042e7d7aa0: ffff88041b9a3d00 ffff880701d95900 ffff88041b9a3d00 1ffff10085cfaf67
ffff88042e7d7ac0: ffff88042e7d7ae8 ffffffff814f80b0 ffff88041b9a4590 ffff880701d95900
ffff88042e7d7ae0: ffff88042e7d7b98 ffff88042e7d7bc0 ffffffff814fe321 ffff88041b9a3d00
ffff88042e7d7b00: ffff880701d95900 ffff88041b9a4598 dffffc0000000000 ffff88042fc24660
ffff88042e7d7b20: ffff880701d95908 ffff88042fc24668 ffff88042cb9bd00 0000000041b58ab3
ffff88042e7d7b40: ffffffff83a0a0e1 ffffffff814fdfa0 ffff88041b9a3d00 ffff88072f3216d8
ffff88042e7d7b60: ffff88041b9a4598 ffff88072c684590 dffffc0000000000 ffff88041b9a4840
ffff88042e7d7b80: 1ffff10085cfaf71 0000000041b58ab3 ffffffff839fba92 ffff88041b9a3d00
ffff88042e7d7ba0: ffff88042fc20d80 ffff88042cb9bd00 ffff880700b1cc80 ffff88041b9a4118
ffff88042e7d7bc0: ffff88042e7d7c10 ffffffff812a5e77 0000000000000000 ffff88072f320d80
ffff88042e7d7be0: 0000000000000000 ffff88072f320d80 ffff88041b9a3d00 ffff88072c683d00
ffff88042e7d7c00: ffff88071570c800 ffff88041b9a4118 ffff88042e7d7d28 ffffffff8329e69c
ffff88042e7d7c20: ffff88042e7d7c98 ffffffff8125a75f ffff88071d4cb408 ffff88072f321608
ffff88042e7d7c40: 1ffff10085cfaf8c 000000001b26d110 ffff88072f321630 ffff88041b9a42d0
ffff88042e7d7c60: 0000000041b58ab3 ffffffff83a0af50 ffffffff8329e000 1ffff10085cfaf90
ffff88042e7d7c80: 0000000041b58ab3 ffffffff83a0a700 ffffffff81239600 ffff88042e7d7cc8
ffff88042e7d7ca0: ffffffff8125ae5f 000000011d4cadc0 0000000000000286 1ffff10085cfaf9c
ffff88042e7d7cc0: ffff88071d4cadc0 ffff88042e7d7d68 ffffffff8125ca69 ffff880700000001
ffff88042e7d7ce0: 0000000041b58ab3 ffffffff83a001ed 0000000000000246 ffff88042e7d7e78
ffff88042e7d7d00: ffffed00837347a0 ffff88041b9a3d00 ffff88041b9a4200 ffff88041b9a3d00
ffff88042e7d7d20: ffff88041b9a3cf0 ffff88042e7d7d48 ffffffff8329f7f4 dffffc0000000000
ffff88042e7d7d40: ffff88042e7d7e50 ffff88042e7d7e18 ffffffff8123d132 1ffff10085cfafb2
ffff88042e7d7d60: ffffed00837347a0 ffff88041b9a3d00 ffff88042e7d7e78 ffff88042e7d7ea0
ffff88042e7d7d80: ffffed0085cfafca ffff88042e7d7e54 0000000041b58ab3 ffffffff839ff9ee
ffff88042e7d7da0: ffffffff8123cb80 ffffffff814a7730 0000000000000000 ffff88042e7d7ec0
ffff88042e7d7dc0: 0000000000000000 ffff88042e7d7ee8 ffffffff8125d8c3 ffff88041fbef780
ffff88042e7d7de0: ffff880727608c80 0000000000000000 ffff88042e7d7ed0 1ffff10085cfafc6
ffff88042e7d7e00: 0000000000000000 0000000000000000 ffff880727608c80 ffff88042e7d7ef8
ffff88042e7d7e20: ffffffff812409d3 00007fff4ff89d6c 0000000041b58ab3 ffffffff839ffa10
ffff88042e7d7e40: ffffffff81240900 000000000000003d 0000000400000000 ffff880727608c80
ffff88042e7d7e60: 0000000000000000 00007fff4ff89d6c 0000000000000000 ffffffff00000000
ffff88042e7d7e80: ffff88041b9a3d00 ffffffff81237f10 ffff880429da68a8 ffff880429da68a8
ffff88042e7d7ea0: ffffffff00000000 ffff88042e7d7f58 dffffc0000000000 ffff88042e7d7ef8
ffff88042e7d7ec0: ffffffff81004388 0000000000000000 ffff88042e7d7f58 ffffffff81240900
ffff88042e7d7ee0: ffff88041b9a3d00 0000000000000000 0000000000000000 ffff88042e7d7f48
ffff88042e7d7f00: ffffffff8100540f 0000000000000006 ffffed00e4f1da71 ffff88071b26d100
ffff88042e7d7f20: 0000000000000000 1ffff10085cfaff5 0000000000000000 0000000000401810
ffff88042e7d7f40: 00007fff4ff8c0f0 00007fff4ff89d70 ffffffff832abcdf 0000000000000000
ffff88042e7d7f60: 0000000000000000 00007fff4ff8c0f0 0000000000401810 00007fff4ff89d70
ffff88042e7d7f80: 0000000000000000 0000000000000246 0000000000000000 00007f614f23c120
ffff88042e7d7fa0: 00007f614f23c0a4 ffffffffffffffda 00007f614ef57afa 0000000000000000
ffff88042e7d7fc0: 00007fff4ff89d6c 0000000000003176 000000000000003d 00007f614ef57afa
ffff88042e7d7fe0: 0000000000000033 0000000000000246 00007fff4ff89d58 000000000000002b
3Memory state around the buggy address:
3 ffff88042fc0b980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042fc0ba00: 00 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2
3>ffff88042fc0ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
3 ^
3 ffff88042fc0bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042fc0bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
3==================================================================

2016-11-17 09:13:53

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 10:04:46AM +0100, Peter Zijlstra wrote:
> On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote:
> > Peter or Vince, can you try to recreate with this patch? It dumps the
> > raw stack contents during a stack dump. Hopefully that would give a
> > clue about what's going wrong.
>
>
> Here goes... I'll do another run and get you the results of that as
> well.

This one is funny, I've not seen that WARNING before. Let me do a third
run.

4WARNING: unrecognized kernel stack return address ffff88072f20bef8 at ffff88072f20bdd0 in swapper/10:0
3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88072f20bda8
3Read of size 8 by task swapper/10/0
0page:ffffea001cbc82c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x6ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #4
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d ? unwind_get_return_address+0x1fb/0x220
d ? kasan_report_error+0x5/0x4d0
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_get_return_address+0x130/0x220
d ? unwind_get_return_address+0x1fb/0x220
d unwind_get_return_address+0x1fb/0x220
d perf_callchain_kernel+0x356/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? ftrace_ops_list_func+0x252/0x370
d ? perf_callchain_kernel+0x5/0x550
d ? perf_callchain+0x128/0x1a0
d get_perf_callchain+0x276/0x670
d ? put_callchain_buffers+0x50/0x50
d ? get_perf_callchain+0x5/0x670
d ? rcu_is_watching+0x30/0x70
d perf_callchain+0x128/0x1a0
d ? setup_pebs_sample_data+0xd48/0x18e0
d perf_prepare_sample+0x70e/0xfb0
d ? __rcu_read_lock+0x5/0x50
d perf_event_output+0x93/0x110
d ? perf_event_output_backward+0x110/0x110
d ? setup_pebs_sample_data+0xd48/0x18e0
d __intel_pmu_pebs_event+0x2b1/0x610
d ? pebs_update_state+0x310/0x310
d ? perf_cgroup_attach+0xb0/0xb0
d ? ctx_resched+0x1a0/0x1a0
d ? rcu_is_watching+0x30/0x70
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? rcu_is_watching+0x30/0x70
d ? ftrace_ops_list_func+0x252/0x370
d ? intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? ftrace_call+0x5/0x34
d ? __intel_pmu_pebs_event+0x5/0x610
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? ftrace_call+0x5/0x34
d ? intel_pmu_drain_pebs_nhm+0x5/0xbf0
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? ftrace_ops_list_func+0x252/0x370
d ? perf_event_nmi_handler+0x2d/0x50
d ? ftrace_call+0x5/0x34
d ? ftrace_call+0x5/0x34
d ? ftrace_ops_list_func+0x252/0x370
d ? intel_pmu_handle_irq+0x5/0xa90
d ? perf_event_nmi_handler+0x5/0x50
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d ? nmi_handle+0x5/0x250
d default_do_nmi+0x67/0x180
d do_nmi+0x1a2/0x210
d ? ctx_resched+0x1a0/0x1a0
d ? perf_cgroup_attach+0xb0/0xb0
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:remote_function+0x113/0x180
dRSP: 0018:ffff88072f207f60 EFLAGS: 00000806c
dRAX: 0000000000000000 RBX: ffff880421affd60 RCX: 1ffff1008435ffaf
dRDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88072f224f10
dRBP: ffff88072f207f80 R08: 00000000000004c1 R09: ffffed00e5e426ca
dR10: 00000000ffffffff R11: 1ffff100e5e426ec R12: ffffffff814f8270
dR13: ffff880421affd78 R14: ffff880421affcc0 R15: ffffffff814e6a70
d ? perf_cgroup_attach+0xb0/0xb0
d ? ctx_resched+0x1a0/0x1a0
d ? remote_function+0x113/0x180
d ? remote_function+0x113/0x180
d <EOE>
ffff88072f20a000: 0000000000000000 ...
ffff88072f20a5a0: 0000000000000000 0000000000000000 0000000000000000 ffff8807af20a9a6
ffff88072f20a5c0: ffff88072f20a9ae 0000000000000060 1ffff100e5e414c3 00000000ffffffff
ffff88072f20a5e0: 0020000000000000 0000000000000000 00000000fffffffd 0000000000000001
ffff88072f20a600: 0000000000000180 0000000000000010 ffff1060ffffff09 0000000041b58ab3
ffff88072f20a620: ffff8807af20aa0e ffff88072f20aa16 0000000000000060 1ffff100e5e414d0
ffff88072f20a640: 00000000ffffffff 0020000000000000 0000000000000000 00000000fffffffd
ffff88072f20a660: 0000000000000001 0000000000000180 ffffffff8348be10 ffff1060ffffff09
ffff88072f20a680: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffff88072f20a758
ffff88072f20a6a0: ffff88072f313830 ffffffff823b7c54 ffff88072f20a7b0 ffff88072f20a7b8
ffff88072f20a6c0: ffff88072f20a7b0 ffff88072f20a9a7 0000000000000001 ffffffff8348bec9
ffff88072f20a6e0: ffff88072f20aa16 0000000000000018 dffffc0000000000 ffff88072f20a870
ffff88072f20a700: ffff88072f20a7c0 ffff88072f20a7e8 ffffffff823b7c54 ffff88072f20a818
ffff88072f20a720: ffff88072f20a820 ffff88072f20a818 ffff88072f20aa0f 1ffff100e5e414ec
ffff88072f20a740: ffffed00e5e41504 000000007fffffff ffff8807af20aa0e ffffffff8348beca
ffff88072f20a760: 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0 ffff880402cb94a8
ffff88072f20a780: ffff1060ffffff09 ffff88072f20aa00 1ffff100e5e41515 ffff88072f20a890
ffff88072f20a7a0: ffffffff814e6b83 ffffffffffffffff ffff88072f20a868 fffffbfff07121c8
ffff88072f20a7c0: ffffed00e5e414ff 1ffff100e5e41515 ffff88072f20a988 ffffffff814e6b83
ffff88072f20a7e0: ffffffffffffffff ffff88072f20a890 ffffffff823b8d95 0000000041b58ab3
ffff88072f20a800: ffffffff839ff69c ffffffff823b8d00 ffffed00e5e41515 ffff880700000020
ffff88072f20a820: ffff88072f20a8a0 ffff88072f20a858 ffff88072f20a8c8 ffff88072f20aa00
ffff88072f20a840: dffffc0000000000 ffff88072f20a890 ffffffff813c861f ffff88072f20aec8
ffff88072f20a860: 0000000000000000 0000000000000113 0000000000000180 ffffffff838ad5e6
ffff88072f20a880: ffffffff838ad5e6 ffff88072f20aa00 ffff88072f20a9b0 ffffffff813c883f
ffff88072f20a8a0: 000000000000000f 0000000041b58ab3 ffffffff83a0c758 ffffffff813c86d0
ffff88072f20a8c0: ffff88072f20af30 0000000000000000 ffff88072f20a988 ffffffff832ad2e7
ffff88072f20a8e0: ffff88072f20a998 ffffffff832ad2e7 ffff88072f20a9b4 0000000000000000
ffff88072f20a900: ffff88072f20a9c0 0000000000000113 ffff88072f20ab48 ffff880700000020
ffff88072f20a920: 0000000000000000 ffffffff839fbe88 0000000000000042 0000000000000001
ffff88072f20a940: ffff0a00ffffff05 ffff88072f20aa1b ffff88072f20ad50 ffff88072f20aa1c
ffff88072f20a960: 0000000000000000 ffff88072f20a9c0 ffffffff823adbcd 0000000041b58ab3
ffff88072f20a980: ffffffff83a40559 1ffff100e5e4153c ffff0a00ffffff05 1ffff100e5e4153c
ffff88072f20a9a0: ffff88072f20aa00 ffff88072f20ad35 ffff88072f20af30 ffff0a00ffffff05
ffff88072f20a9c0: ffff88072f20ab48 ffffffff823af620 ffff88072f20aa88 ffffffff832ad2e7
ffff88072f20a9e0: 0000000041b58ab3 ffffffff83a40559 ffffffff823af530 0000000000000000
ffff88072f20aa00: 665f65746f6d6572 2b6e6f6974636e75 78302f3331317830 3035780000303831
ffff88072f20aa20: 0000300030657800 ffffffff83a13036 0000000000000053 0000000000000001
ffff88072f20aa40: ffff0a00ffffff05 ffff88072f20ab25 ffff88072f20ae7a ffff88072f20ab26
ffff88072f20aa60: ffff88072f20af30 ffff88072f20ad30 ffffffff823adb30 1ffff100e5e41558
ffff88072f20aa80: 00000000ffffffff 0020f100e5e4155c ffff0a00ffffff00 1ffff10000000010
ffff88072f20aaa0: ffff88072f20ab00 ffff88072f20aa80 ffff88072f20b010 ffff103000001005
ffff88072f20aac0: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffffffffffffffff
ffff88072f20aae0: 3266323061616330 6666666638383037 ffffffff00000010 0000000000000000
ffff88072f20ab00: ffffffff814e6a70 ffff88072f20af10 ffff103000001009 0000000041b58ab3
ffff88072f20ab20: ffff88072f20af30 ffff88072f20ad75 ffff88072f20ad30 1ffff100e5e41570
ffff88072f20ab40: ffffffffffffffff 002088072f20ac08 ffffffff823b6600 ffffffff00000010
ffff88072f20ab60: ffff88072f20ad00 ffffffff00000010 ffff88072f20ad10 ffff103000001009
ffff88072f20ab80: 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960 ffffffff839f498b
ffff88072f20aba0: 3833396634393862 6666666666666666 ffff88072f20ac87 ffff88072f20ac08
ffff88072f20abc0: ffffffff823ad785 ffff88072f20acc8 0000000000000001 ffff88072f20ad85
ffff88072f20abe0: ffff88072f20ac08 ffffffff816213e5 ffff88072f20ad85 ffffffff839f498f
ffff88072f20ac00: dffffc0000000000 ffff88072f20ace8 ffffffff823b72eb ffff88072f20b028
ffff88072f20ac20: ffff88072f20b030 ffffffff839f4990 ffff88072f20ad30 1ffff100e5e4158c
ffff88072f20ac40: ffffed00e5e41606 0000000000000200 ffff88072f20af30 0000000000000001
ffff88072f20ac60: 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0 ffff88072f20acc8
ffff88072f20ac80: ffff103000001000 1ffff100e5e41595 ffffffff83c88b40 ffff88072f20af10
ffff88072f20aca0: ffffffff81338af4 0000000041b58ab3 ffff88072f20acd0 ffffffff811c0a51
ffff88072f20acc0: ffff88072f20ad86 dffffc0000000000 ffff88072f20ad00 ffffffff811c0afc
ffff88072f20ace0: ffff88072f20ad30 00000000ffffffff 1ffff100e5e415a2 ffffffff83c88b40
ffff88072f20ad00: ffff88072f20af78 ffffffff81338af4 0000000041b58ab3 ffffffff83a03c41
ffff88072f20ad20: ffffffff81338a00 ffff0a00ffffff00 3730383866666666 3030646130326632
ffff88072f20ad40: 666666666666203a 6138333331386666 3066666666203030 6666666666303061
ffff88072f20ad60: 3636303320363636 3032363636363636 3636362030333033 3336363636363636
ffff88072f20ad80: 1f000a3633363336 ffffffff83c88b40 ffff88072f20b008 ffffffff81338af4
ffff88072f20ada0: 0000000041b58ab3 ffffffff83a03c41 ffffffff81338a00 0000000041b58ab3
ffff88072f20adc0: 6177647261486401 3a656d616e206572 43206c65746e4920 ffff880402cb9498
ffff88072f20ade0: ffff88072f20ae30 ffffffff81448fa2 534f4942202c5a47 ffff880402cb94a8
ffff88072f20ae00: ffffffff8133c807 00000000ffffffff 0000000000000000 ffff88072f20bef8
ffff88072f20ae20: ffffffff8342d740 ffff88072f20afc0 ffff88072f20aee8 ffffffff832ad2e7
ffff88072f20ae40: ffff880402cb9498 ffff88072f20ae98 ffffffff81448fa2 ffff880402cb9498
ffff88072f20ae60: ffff880402cb94a8 ffffffff8133c807 00000000ffffffff 0000000000000018
ffff88072f20ae80: ffff88072f20bef8 ffffffff839f4970 ffff88072f20b028 ffff88072f20af50
ffff88072f20aea0: ffffffff832ad2e7 ffffffff81448fa2 ffffffff8342d740 ffff880402cb94a8
ffff88072f20aec0: ffff880402cb9498 ffff88072f20aff8 ffffffff81448fa2 ffff88072c4c8000
ffff88072f20aee0: ffff880402cb94a8 ffff88072f20b028 ffffffff839f4970 ffff88072f20b068
ffff88072f20af00: 0000000000000000 0000000000000000 ffff88072f20b028 ffffffff839f4970
ffff88072f20af20: ffffffff832ad2e7 ffffffff81338a05 ffffffff81448fa2 ffff880402cb94a8
ffff88072f20af40: ffff880402cb94a8 ffff88072f20b0a8 00000000ffffffff 0000000000000018
ffff88072f20af60: ffff88072f20bef8 ffffffff839f4970 ffff88072f20b028 ffff88072f20aff8
ffff88072f20af80: ffffffff8133c807 0000000000000000 00000000ffffffff 0000000000000000
ffff88072f20afa0: ffff88072f20af80 ffffffff8133c7c5 ffff88072f20bef8 0000000000000000
ffff88072f20afc0: 0000000000000000 000000002f20b080 1ffff100e5e41601 0000000000000018
ffff88072f20afe0: ffff88072f20bef8 ffff88072c4c8000 ffff88072f20c000 ffff88072f20b0a8
ffff88072f20b000: ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c ffffffff8151fdf3
ffff88072f20b020: ffffffff839ff69c ffffffff00000030 ffff88072f20b0b8 ffff88072f20b068
ffff88072f20b040: ffff88072f20b020 ffffffff839f4970 ffff88072f20bff8 ffffffff8151fdf8
ffff88072f20b060: ffffffff813b9d58 ffff88072f20b090 ffff88072f20b040 ffff88072f20b020
ffff88072f20b080: ffff88072f20b090 ffff88072f20b040 ffff88072f20b020 ffff88072f20b060
ffff88072f20b0a0: 0000000000000009 ffff88072f20b178 ffffffff811504e6 ffffffff814e6b83
ffff88072f20b0c0: ffffffff839fbd23 000000002f20b188 0000000000000020 ffffffff839fbd5b
ffff88072f20b0e0: ffffffff839fbd1f 0000000000000005 ffff88072f20a000 ffff88072f20c000
ffff88072f20b100: ffff88072f207f60 0000000000000002 ffff88072f200000 ffff88072f208000
ffff88072f20b120: ffff88072c4d7d78 0000000000000024 ffff88072c4c8000 0000000000000000
ffff88072f20b140: ffff88072f207f80 0000000000000000 0000000000000086 0000000000000000
ffff88072f20b160: ffff88072f20b420 ffffed00e5e41685 dffffc0000000000 ffff88072f20b188
ffff88072f20b180: ffffffff811505c4 ffff88072f20b1a8 ffffffff8238dfb3 ffff88072f20b238
ffff88072f20b1a0: ffff88072f20bda8 ffff88072f20b228 ffffffff816221c1 ffffffff811c72cb
ffff88072f20b1c0: 1ffff100e5e41669 ffff88072f20b238 0000000000000086 ffffffff81621d25
ffff88072f20b1e0: dffffc0000000000 1ffff100e5e4166a 1ffff100e5e417b5 1ffff100e5e41669
ffff88072f20b200: ffff88072f20b310 ffff88072f20bda8 ffff88072f20b420 ffffed00e5e41685
ffff88072f20b220: dffffc0000000000 ffff88072f20b260 ffffffff81622323 ffff88072f20bda8
ffff88072f20b240: ffff88072f20bda8 0000000000000008 ffffffff811c7200 ffffffff811c72cb
ffff88072f20b260: ffff88072f20b2a0 ffffffff811c72cb ffff88072f20b338 ffff88072f20b288
ffff88072f20b280: ffff88072f20b310 ffff88072f20b390 ffff88072f20b420 ffffed00e5e41685
ffff88072f20b2a0: ffff88072f20b3b8 ffffffff8100f356 000000000001bdc0 ffff88072f20b460
ffff88072f20b2c0: ffff88070b9fcec0 ffff88072f20b42c 1ffff100e5e4165e ffff88072f20b432
ffff88072f20b2e0: 0000000000000004 ffff88072f20b428 0000000041b58ab3 ffffffff839f613c
ffff88072f20b300: ffffffff8100f000 ffff88072f20b488 0000000000000005 ffff88072f20a000
ffff88072f20b320: ffff88072f20c000 ffff88072f207f60 0000000000000020 ffff88072c4c8000
ffff88072f20b340: 0000000000000000 ffff88072f20bda0 0000000000000000 ffff88072f20b420
ffff88072f20b360: ffffffff81448fa2 ffffffff8100f005 ffff880402cb94a8 ffffffff81514438
ffff88072f20b380: ffff880405b11980 0000000000000040 000000000001bdc0 ffff88072f20b460
ffff88072f20b3a0: ffff88070b9fcec0 1ffff100e5e41680 ffff88070b9fcec0 ffff88072f20b488
ffff88072f20b3c0: ffffffff81513f16 ffff88072f20b574 ffff880405b11980 ffff880700000000
ffff88072f20b3e0: ffff88072f20b6e0 ffff880700000000 ffff880700000001 000000000000000c
ffff88072f20b400: 0000000041b58ab3 ffffffff83a0a369 ffffffff81513ca0 0000000000000001
ffff88072f20b420: ffff88070b9fcec0 000000020000007f ffff88072f000001 ffffffff81513ca5
ffff88072f20b440: 1ffff100e5e416ef ffff88072f20b588 ffffffff8135d050 00000000000000ff
ffff88072f20b460: ffff880405b11980 0000000000000040 ffff88072f20b6e0 ffff88072f224f08
ffff88072f20b480: 0000000000000000 ffff88072f20b4d0 ffffffff81514438 ffffffff00000001
ffff88072f20b4a0: ffffffff81027da8 ffff88072f20b574 ffff880405b11980 ffff88072f20b570
ffff88072f20b4c0: ffff880405b11980 ffff88072f20b7c0 ffff88072f20b540 ffffffff8150ac7e
ffff88072f20b4e0: ffff880405b11980 0000000100000000 ffffffff81359595 ffff880402cb9498
ffff88072f20b500: 00000000000000ff ffff88072f20b576 ffff88072f20b6e0 1ffff100e5e416aa
ffff88072f20b520: ffff880405b11980 ffff88072f20b7c0 ffff88072f20b6e0 dffffc0000000000
ffff88072f20b540: ffff88072f20b630 ffffffff8150b7d3 0000000041b58ab3 ffffffff83a0d7b0
ffff88072f20b560: ffffffff8150b740 ffff880405b11980 0048000100000009 00000000000000a8
ffff88072f20b580: ffff88072f20b6e0 ffff88072f20b6e0 ffff88072c610210 ffff88072f20b7c0
ffff88072f20b5a0: ffff880405b11980 00000000000000ff ffff88072f20b630 ffffffff81027da8
ffff88072f20b5c0: ffff88072c610210 ffff88072f20bef8 ffff880405b11980 ffff88072f20b770
ffff88072f20b5e0: ffffffff84156db0 ffff88072f20b7d0 ffff88072f213440 ffff880405b11a68
ffff88072f20b600: ffff880700000000 ffff88072c610210 ffff88072c610210 ffff88072f20b7c0
ffff88072f20b620: 0000000000000001 ffff880405b11980 ffff88072f20b9b0 ffffffff81028f01
ffff88072f20b640: 0000000000000000 ...
ffff88072f20b660: 0000000000000000 1ffff100e5e416d8 1ffffffff082adb6 0000000000000007
ffff88072f20b680: fffffbfff082adb7 0000000000000000 ffff88072f20b7c0 ffff88072f20bef8
ffff88072f20b6a0: ffff880405b11980 0000000000000005 ffff88072c610580 ffff88072f20b6e0
ffff88072f20b6c0: 0000000041b58ab3 ffffffff839f20b0 ffffffff81028c50 0000000000000000
ffff88072f20b6e0: ffffffff814e6a70 ffff880421affcc0 ffff880421affd78 ffffffff814f8270
ffff88072f20b700: ffff88072f20bd50 ffff880421affd60 1ffff100e5e426ec 00000000ffffffff
ffff88072f20b720: ffffed00e5e426ca 00000000000004c1 0000000000000000 1ffff1008435ffaf
ffff88072f20b740: 0000000000000000 0000000000000000 ffff88072f224f10 ffffffffffffffff
ffff88072f20b760: ffffffff8135d050 0000000000000010 0000000000000803 ffff88072f20bd48
ffff88072f20b780: 0000000000000018 ffffffff839f2108 ffffffff8102b200 0000000000000000
ffff88072f20b7a0: 0000000000000000 ...
ffff88072f20b7c0: 0000000000000000 0000000000000000 0000000000000000 0000000000000100
ffff88072f20b7e0: 0000000000000000 0000000000000000 0000000005080021 00000000000000ff
ffff88072f20b800: ffffffff8135d050 0000000000000000 0000004c8c598365 0000000000013955
ffff88072f20b820: 0000000000000000 000000000000000a ffff88070b9fcec0 0000000000000000
ffff88072f20b840: 0000000000000000 ...
ffff88072f20b860: 0000000000000000 0000000000000000 0000000000000000 ffff880402cb9498
ffff88072f20b880: ffff88072f20b8d0 ffffffff81448fa2 0000000000000000 ffff880402cb94a8
ffff88072f20b8a0: ffffffff81029856 0000000000000000 dffffc0000000000 ffff88072f20bb08
ffff88072f20b8c0: ffff880405b11980 ffff88072f213440 ffff88072f20b988 ffffffff832ad2e7
ffff88072f20b8e0: 0000000000000000 ...
ffff88072f20b900: ffff88072f20bb30 0000000000000000 0000000000000000 0000000000000000
ffff88072f20b920: 0000000000000008 0000000000000000 fffffbfff082adbc ffff88072c610580
ffff88072f20b940: ffff88072c610000 ffff88072f20bef8 ffff880405b11980 0000000000000000
ffff88072f20b960: ffffffff81028c55 0000000000000000 0000000000000000 0000000000000000
ffff88072f20b980: 0000000000000000 0000000000000000 dffffc0000000000 ffff88072f20bb08
ffff88072f20b9a0: ffff880405b11980 ffff88072f213440 ffff88072f20bb30 ffffffff81029856
ffff88072f20b9c0: 0000000000000000 0000000000000000 ffff88072f213d78 ffff88072f20bef8
ffff88072f20b9e0: 1ffff100e5e41745 ffff88072c610000 ffff88072f20ba88 ffff880402cb9407
ffff88072f20ba00: ffffed00e5e427af ffff88072c610580 fffffbfff082adbc ffff88072f20bac8
ffff88072f20ba20: ffff88072f20ba88 0000000041b58ab3 ffffffff839f20d0 ffffffff81029260
ffff88072f20ba40: 0000004c8bbedff6 0000000000000001 ffff88072f20bb08 ffffffff832ad2e7
ffff88072f20ba60: 0000000000000000 ffff88072f20bb30 0000000000000000 0000000000000000
ffff88072f20ba80: ffff88072f20bdf8 0000000000000008 0000000000000000 0000000000000000
ffff88072f20baa0: dffffc0000000000 ffff88072f213440 1ffffffff082adb9 0000000000000390
ffff88072f20bac0: 0000000040000000 0000000000000000 0000000000000000 ffff88072f213d90
ffff88072f20bae0: ffffffff81029265 0000000000029fff ffff88072f20bb00 fffffffffffffff8
ffff88072f20bb00: 000000000000000f 0000000000000001 0000004c8bbee2c1 ffffffff83c16120
ffff88072f20bb20: 0000004c8bbedff6 ffff88072f213440 ffff88072f20bdf8 ffffffff8101e752
ffff88072f20bb40: ffff88072f213d78 1ffff100e5e41770 ffffed00e5e427af 0000006400000000
ffff88072f20bb60: 0000000000000000 ffff88072f20bef8 0000000000000000 0000000000000000
ffff88072f20bb80: 0000000041b58ab3 ffffffff839f1dd0 ffffffff8101e2a0 0000000000000000
ffff88072f20bba0: 0000000000000000 ...
ffff88072f20bcc0: ffff880402cb9498 ffff88072f20bd18 ffffffff81448fa2 0000000000000000
ffff88072f20bce0: ffff880402cb94a8 ffffffff8100a25d ffff88072f20bef8 0000004c8bbee2c1
ffff88072f20bd00: ffffffff83c16120 0000004c8bbedff6 ffffffff83c16130 ffff88072f20bdd0
ffff88072f20bd20: ffffffff832ad2e7 0000004c8bbedff6 ffffffff83c16130 ffff88072f20bdf0
ffff88072f20bd40: ffffffff832ad2e7 ffff88072f20be18 ffff88072f20bda0 ffffffff81448fa2
ffff88072f20bd60: 0000000000000000 ffffed00e5e426ca 00000000000004c1 0000004c8bbee2c1
ffff88072f20bd80: 000000000000001f 0000000000000000 000000002dd2f949 ffff88072f20bef8
ffff88072f20bda0: 000000000000001f ffffffff8101e2a5 ffff88072f20bef8 0000000000000000
ffff88072f20bdc0: ffff88072f20bf90 ffffffff8100a235 ffff88072f20bef8 0000004c8bbee2c1
ffff88072f20bde0: ffffffff83c16120 0000004c8bbedff6 ffffffff83c16130 ffff88072f20be18
ffff88072f20be00: ffffffff8100a25d dffffc0000000000 0000000000000000 ffff88072f20be80
ffff88072f20be20: ffffffff81150f5e 0000000000000000 ffffffff81150ec5 ffff88072f20bef8
ffff88072f20be40: 0000000000000000 ffffffff83c64ce8 0000000000000000 ffff88072f20bef8
ffff88072f20be60: 0000000000000000 0000000000000007 ffff88072c4c877c ffff88072f20bf90
ffff88072f20be80: ffff88072f20bea8 ffffffff81151a97 ffff88072f20bee8 ffff88072f20bef8
ffff88072f20bea0: ffff88072c4c8000 ffff88072f20bee8 ffffffff81151d52 0000000000000000
ffff88072f20bec0: 0000000000000001 ffffffff814f8270 ffff880421affd78 ffff880421affcc0
ffff88072f20bee0: ffffffff814e6a70 ffff88072f20bef9 ffffffff832ad08e ffffffff814e6a70
ffff88072f20bf00: ffff880421affcc0 ffff880421affd78 ffffffff814f8270 ffff88072f207f80
ffff88072f20bf20: ffff880421affd60 1ffff100e5e426ec 00000000ffffffff ffffed00e5e426ca
ffff88072f20bf40: 00000000000004c1 0000000000000000 1ffff1008435ffaf 0000000000000000
ffff88072f20bf60: 0000000000000000 ffff88072f224f10 ffffffffffffffff ffffffff814e6b83
ffff88072f20bf80: 0000000000000010 0000000000000806 ffff88072f207f60 0000000000000018
ffff88072f20bfa0: ffffffff814e6b83 0000000000000010 0000000000000806 ffff88072f207f60
ffff88072f20bfc0: 0000000000000018 0000000000000001 0000000000000000 ffffffff814e6b83
ffff88072f20bfe0: 0000000000000010 0000000000000806 ffff88072f207f60 0000000000000018
d <IRQ>
d flush_smp_call_function_queue+0x111/0x310
d generic_smp_call_function_single_interrupt+0x13/0x30
d smp_call_function_single_interrupt+0x64/0x90
d call_function_single_interrupt+0x90/0xa0
dRIP: 0010:cpuidle_enter_state+0x121/0x7a0
dRSP: 0018:ffff88072c4d7e28 EFLAGS: 00000246c ORIG_RAX: ffffffffffffff04
dRAX: 0000000000000000 RBX: ffff88072f22b720 RCX: 000000000000001f
dRDX: 1ffff100e5e442f9 RSI: 000000002dd2f949 RDI: ffff88072f2217c8
dRBP: ffff88072c4d7e88 R08: 0000000000000007 R09: ffffffff83f3f320
dR10: 071c71c71c71c71c R11: ffff88072f21dd04 R12: 0000004c8bbe92c9
dR13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000060
d <EOI>
ffff88072f200000: 0000000000000000 ...
ffff88072f207560: 0000000000000000 ffff880413a83468 ffff880413a83300 0000000000000000
ffff88072f207580: ffff88072f2075e0 ffffffff814ffc80 0000000000000000 ffff88072f2075e0
ffff88072f2075a0: ffffffff8101b2ee ffff88072f213440 0000000000000000 ffff88072f224ff4
ffff88072f2075c0: ffff88072f2075e0 ffffffff8101d12c ffffffff83c167c0 ffff88072f213440
ffff88072f2075e0: ffff88072f2075f0 ffffffff8101d2a0 ffff88072f207688 ffffffff8100de4e
ffff88072f207600: ffffed00e5e40eda ffff88072f213654 ffff880413a83300 ffff880413a83300
ffff88072f207620: ffffffff83c167c0 000000490bb75162 ffff88072f224f08 ffff88072f213658
ffff88072f207640: ffff88072f2076a0 ffff88072f213650 ffffffff00000000 ffff88072f213440
ffff88072f207660: ffffffff83c167c0 ffffffff83c167c0 ffff88072f224ff4 ffff880413a83310
ffff88072f207680: dffffc0000000000 ffff88072f2076a0 ffffffff814eaff9 ffff88072f224f00
ffff88072f2076a0: ffff88072f2076f8 ffffffff814f55aa 000000012f224f08 ffff88072f224f08
ffff88072f2076c0: ffff880413a83300 ffff88072f224ff0 ffff880413a83300 ffff88072f224f08
ffff88072f2076e0: 0000000000000000 ffff880413a83388 ffff88072f224f08 ffff88072f207740
ffff88072f207700: ffffffff814f5fc2 00000000e0a8761e 0000000000000000 ffff88072f224f08
ffff88072f207720: ffff88072f224f08 0000000000000000 ffff880413a83300 ffff880402cb9498
ffff88072f207740: ffff88072f207790 ffffffff81448fa2 ffff880421affd20 ffff880402cb94a8
ffff88072f207760: ffffffff812f57f4 ffff88072a547d10 000000000000000a ffff88072a547d00
ffff88072f207780: 0000000000000000 ffff88072f207d78 ffff88072f207848 ffff880402cb9498
ffff88072f2077a0: ffff88072f2077f0 ffffffff81448fa2 ffffffff814d88ec ffff880402cb94a8
ffff88072f2077c0: ffffffff812f6675 ffff88072f2079b8 ffff88072f207cd8 dffffc0000000000
ffff88072f2077e0: 0000000000000000 ffffed00e5e40f3c ffff88072f2078a8 ffffffff832ad2e7
ffff88072f207800: 1ffff100e54a8fac 000000000000000a 0000000000000004 0000000000000000
ffff88072f207820: ffff88072f207a60 ffff88072f207839 ffffffff832ac633 ffffed00e5e40f3c
ffff88072f207840: 0000000000000007 000000000000001e 0000000000000010 0000000000000007
ffff88072f207860: fffffbfff082b618 ffff880729fcbf98 ffff88072f215ca0 000000000000001e
ffff88072f207880: ffff88072f207890 ffffffff823d7aa5 ffff88072f2078d0 ffffffff8238dbde
ffff88072f2078a0: 0000000000000050 ffff88072f2079b8 ffff88072f207cd8 dffffc0000000000
ffff88072f2078c0: 0000000000000000 ffffed00e5e40f3c ffff88072f207a60 ffffffff812f6390
ffff88072f2078e0: ffffffff812da912 ffff88072c494200 ffff880402cb9498 ffff88072f207948
ffff88072f207900: ffffffff81448fa2 1ffff100e5e40f33 0000000002cb94a8 ffff88072f207d04
ffff88072f207920: ffff88072f2079e0 ffff88072f207ae8 ffff88072f207ae0 ffff88072f207cec
ffff88072f207940: ffffed00e5e40f9b ffff880729fcbf90 ffff880402cb9498 ffff88072f2079a8
ffff88072f207960: ffffffff81448fa2 ffff88072c471a00 ffff880402cb94a8 ffffffff8101a831
ffff88072f207980: 0000000000000000 ffff880405b11b10 0000000000000002 ffff88072c489f00
ffff88072f2079a0: dffffc0000000000 ffff88072f207a60 ffffffff832ad2e7 ffff88072f207a08
ffff88072f2079c0: ffffffff81448fa2 0000000000000040 ffff880402cb94a8 ffff880402cb9498
ffff88072f2079e0: ffff88072f207a30 ffffffff81448fa2 ffff88072f213968 ffff880402cb94a8
ffff88072f207a00: ffffffff8101c939 ffff880402cb9498 ffff88072f207a60 ffffffff81448fa2
ffff88072f207a20: ffff88072f207c80 ffff880402cb94a8 ffffffff810191fa 0000000000000000
ffff88072f207a40: ffff88072c49f980 ffff88072f213970 ffff88072f213440 0000000000000002
ffff88072f207a60: ffff88072f207b18 ffffffff832ad2e7 ffff88072f213440 0000000000000002
ffff88072f207a80: ffff88072f207b38 ffffffff832ad2e7 ffff88072f207b60 0000000200000005
ffff88072f207aa0: 0000000000000004 ffff88072f213b60 ffff88072f207a98 ffffffff81019250
ffff88072f207ac0: 0000000000000000 0000000000000000 0000000000000004 0000000000000001
ffff88072f207ae0: ffff88072c49f980 0000000000000000 ffffffff832ab465 0000000000000001
ffff88072f207b00: ffff88072f213440 0000000000000000 ffffffff810190d5 ffff88072f207b28
ffff88072f207b20: ffffffff832ab465 ffff88072f207b60 ffffffff810191fa 0000000000000000
ffff88072f207b40: ffff88072f207b60 ffffffff810191fa dffffc0000000000 ffff88072f207c88
ffff88072f207b60: ffff88072f207c48 ffffffff8100cac2 ffffffff832ad2e7 ffff88072f224f08
ffff88072f207b80: 1ffffffff082adc1 0000000000029fff 1ffff100e5e40f78 ffff88072f213b60
ffff88072f207ba0: ffff880400000002 ffff880700000000 ffff88072f207c80 ffff88072f213440
ffff88072f207bc0: 0000000041b58ab3 ffffffff839f6109 ffffffff8100c720 ffff88072f207c80
ffff88072f207be0: 0000000200000000 ffff88072f213440 0000000181448fa2 ffffffff8100c725
ffff88072f207c00: ffffffff839f60e8 ffffffff81009c50 0000000000000008 ffff880402cb9498
ffff88072f207c20: ffff88072f207c70 ffffffff81448fa2 ffff88072f213440 ffff880402cb94a8
ffff88072f207c40: ffffffff814ffc80 ffff880405b11980 0000000000000000 ffff880402cb9498
ffff88072f207c60: ffff88072f207cb0 ffffffff81448fa2 ffff88072f207d28 ffff880402cb94a8
ffff88072f207c80: ffffffff814ffc80 ffff880405b11980 0000000000000000 ffff880405b11b40
ffff88072f207ca0: ffff880405b11980 0000000000000001 ffff88072f207d68 ffffffff832ad2e7
ffff88072f207cc0: ffff880402cb9498 ffff88072f207d18 ffffffff81448fa2 ffffffff832ad2e7
ffff88072f207ce0: ffff880402cb94a8 ffffffff8101d12c ffff88072f213440 0000000000000000
ffff88072f207d00: dffffc0000000000 0000000000000000 0000000000000002 ffff88072f207dd0
ffff88072f207d20: ffffffff832ad2e7 0000000000000000 0000000000000002 ffff88072f207df0
ffff88072f207d40: ffffffff832ad2e7 ffff88072f207e18 ffffffff832ad2e7 ffffffff814ffc80
ffff88072f207d60: ffffffff8102bd25 ffffed00e5e426ca 00000000000004c1 0000000000000001
ffff88072f207d80: 00000000000003f1 0000000000000000 0000000000000000 0000000000000000
ffff88072f207da0: 0000000000000186 ffffffff8102e295 0000000000000000 0000000000000002
ffff88072f207dc0: 0000000000000400 ffffffff8101d105 ffff880402cb9498 ffff88072f207e28
ffff88072f207de0: ffffffff81448fa2 0000000000000000 ffff880402cb94a8 ffffffff814f840a
ffff88072f207e00: 0000000000000000 ffff880405b11980 ffff88072f224f10 0000000000000000
ffff88072f207e20: ffff88072f224f08 ffff88072f207ee0 ffffffff832ad2e7 0000000000000000
ffff88072f207e40: dffffc0000000000 0000000000000000 1ffffffff0782d03 ffff88072f207f50
ffff88072f207e60: ffffffff83c167c0 ffff88072f213654 ffff88072f213658 ffffed00e5e426ca
ffff88072f207e80: 00000000000004c1 0000000000000000 000000000000038f 1ffff100e5e426c8
ffff88072f207ea0: 0000000000000000 ffff88072f224f10 ffffffff83c167c0 ffffffff832ab465
ffff88072f207ec0: ffff88072f207ed8 ffffffff814eaff9 ffff88072f224f00 ffff88072f207f08
ffff88072f207ee0: ffff88072f207ef0 ffffffff832ab465 ffff88072f207f50 ffffffff814f840a
ffff88072f207f00: 0000000000000000 ffff88072f207f50 ffffffff814f840a ffff88072f224fe8
ffff88072f207f20: ffff88072f224f98 ffff880421affd60 ffffffff814f8270 ffff880421affd78
ffff88072f207f40: ffff880421affcc0 ffffffff814e6a70 ffff88072f207f80 ffffffff814e6b6e
ffff88072f207f60: 0000000000000001 ffff880421affca8 dffffc0000000000 0000000000000000
ffff88072f207f80: ffff88072f207fc0 ffffffff813b0dc1 ffff88072f207fa8 ffffffff83a77980
ffff88072f207fa0: 0000004c8bbe92c9 0000000000000001 0000000000000001 0000000000000060
ffff88072f207fc0: ffff88072f207fd0 ffffffff813b2693 ffff88072f207fe8 ffffffff8119e8f4
ffff88072f207fe0: ffff88072f22b720 ffff88072c4d7d79 ffffffff832adc10 ffff88072c4d7d78
d ? cpuidle_enter_state+0x11c/0x7a0
d ? cpuidle_enter_state+0x5/0x7a0
d ? cpuidle_enter+0x5/0x20
d cpuidle_enter+0x17/0x20
d call_cpuidle+0x47/0xc0
d ? call_cpuidle+0x5/0xc0
d cpu_startup_entry+0x1a6/0x2d0
d start_secondary+0x245/0x2d0
d start_cpu+0x5/0x14
ffff88072c4d0000: 0000000057ac6e9d 0000000000000000 0000000000000000 0000000000000000
ffff88072c4d0020: 0000000000000000 ...
ffff88072c4d7620: 0000000000000010 0000000000000000 ffff88072c4d7930 ffffffff81510c31
ffff88072c4d7640: 0000000000000000 ...
ffff88072c4d7660: 0000000000000010 0000000000000000 ffff88072c4d7970 ffffffff81510c31
ffff88072c4d7680: 0000000000000000 ffff88041fb1d8a8 0000000000000000 ffff88041fb1d800
ffff88072c4d76a0: 1ffff100e589aed8 ffffffe400000000 ffff88041e47cc80 ffff88072c4d79a0
ffff88072c4d76c0: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffff88041fb1d800
ffff88072c4d76e0: 1ffff100e589aee0 ffffffe400000000 ffff88041e47cc80 ffff88072c4d79e0
ffff88072c4d7700: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 0000000000000000
ffff88072c4d7720: 0000000000000000 ...
ffff88072c4d7800: 0000000041b58ab3 ffffffff83a0d6b0 ffffffff814e3a20 0000000000000000
ffff88072c4d7820: 0000000000000000 ...
ffff88072c4d7860: ffffffff8150cec6 0000000000000003 ffff88072f21bb20 1ffff100e589af38
ffff88072c4d7880: 0000000000000001 ffff880403093830 ffff88072c4d7948 ffffffff832ad2e7
ffff88072c4d78a0: ffff88072c4d78c8 ffffffff816213e5 ffff88072c4d79a0 ffff88072c4d7ce8
ffff88072c4d78c0: ffff88072c4d7bf8 ffff88072c4d7930 ffffffff8150f4b3 ffff880727eb3000
ffff88072c4d78e0: 0000000000000040 0000000000000000 dffffc0000000000 ffff880403093300
ffff88072c4d7900: 1ffff10080612706 0000000000000001 ffff88072c4c8380 ffff88072c4d79a0
ffff88072c4d7920: ffffffff81359c15 ffff88041e47ce88 ffff88072c4d7c08 ffffffff814e3bf8
ffff88072c4d7940: ffff88072c4d7970 ffff88072c4d7958 ffffffff81359c15 ffff88072c4d7bf8
ffff88072c4d7960: 0000000000000003 ffff88072f21bb20 ffffffff814f2df9 ffff88072c4c8000
ffff88072c4d7980: ffffffff814e3a20 ffff88072c4d7b40 dffffc0000000000 ffff880402cbcc90
ffff88072c4d79a0: ffff88072c4d7a58 ffffffff832ad2e7 ffffffff814f2df9 ffff88072c4c8000
ffff88072c4d79c0: ffffffff814e3a20 ffff88072c4d7b90 ffff88072c4d7ad0 ffff880402cbce88
ffff88072c4d79e0: ffff88072c4d7a98 ffffffff832ad2e7 ffff88072c4c8898 dffffc0000000000
ffff88072c4d7a00: dffffc0000000000 000000000000000a 1ffff100e5899112 ffff88072c4d7b40
ffff88072c4d7a20: 0000000000000000 0000000000000000 ffffffff81359c15 0000000000000000
ffff88072c4d7a40: dffffc0000000000 ffff880403093300 1ffff100e5899112 ffff88072c4d7a68
ffff88072c4d7a60: ffffffff81359c15 ffff88072c4d7ad0 ffff88072c4c8000 ffffffff814e3a20
ffff88072c4d7a80: ffff88072c4d7ad0 ffffffff814f2df9 ffff88072f220e5c ffffffff812a5c3b
ffff88072c4d7aa0: ffff88072c4c8000 ffff88072f220d80 ffff880421f61e80 0000000000000000
ffff88072c4d7ac0: ffff88072c4c847c ffff88072c4d7b80 ffffffff832ad2e7 ffffffff814fe10a
ffff88072c4d7ae0: ffff88072f21bb08 ffffffff8329e69c ffff88072f220d80 ffff88072c4d7bf8
ffff88072c4d7b00: ffff880421f61e80 ffff8804192d2400 ffff88072c4c8418 ffff88072c4c8898
ffff88072c4d7b20: dffffc0000000000 0000000000000003 ffff880402cb9498 ffff88072c4d7b88
ffff88072c4d7b40: ffffffff81448fa2 ffff88072f220d80 ffff880402cb94a8 ffffffff812faffe
ffff88072c4d7b60: 0000000100006f11 ffff88072a547d00 ffff88072f220d80 dffffc0000000000
ffff88072c4d7b80: 0000000000000004 ffff88072c4d7c40 ffffffff832ad2e7 ffffffff832ab608
ffff88072c4d7ba0: ffff88072c4c8000 ffff88072c4d7bf8 ffff880402cb9498 ffff88072c4d7c08
ffff88072c4d7bc0: ffffffff81448fa2 ffff880402cb9498 ffff88072c4d7c20 ffffffff81448fa2
ffff88072c4d7be0: ffff88083fffa140 ffff880402cb94a8 ffffffff815834a0 00000000000249af
ffff88072c4d7c00: ffff88072c4d7e50 ffff88072c4d7e50 0000000000000000 ffff88083fffc998
ffff88072c4d7c20: ffff88072c4d7cd8 ffffffff832ad2e7 ffff880402cb9498 ffff88072c4d7c88
ffff88072c4d7c40: ffffffff81448fa2 ffff880402cb9498 ffff880402cb94a8 ffffffff82d315e9
ffff88072c4d7c60: 0000000000188347 ffff88072f21dcc8 ffff88072f21dcc4 0000000000000000
ffff88072c4d7c80: 0000000000000001 ffff88072c4d7d40 ffffffff832ad2e7 ffff880402cb9498
ffff88072c4d7ca0: ffff88072c4d7cf0 ffff880402cb9498 ffff88072c4d7d00 ffff88072c4d7ea0
ffff88072c4d7cc0: ffffffff813a55db ffff880402cb94a8 ffffffff832ab076 ffffffff83f3f320
ffff88072c4d7ce0: ffff880402cb9498 ffff88072c4d7d38 ffffffff81448fa2 0000000000000000
ffff88072c4d7d00: ffff880402cb94a8 ffffffff82d2d9ae ffff88072f22b720 ffffffff83f3f320
ffff88072c4d7d20: 0000000000000001 0000000000000001 0000000000000060 ffff88072c4d7df0
ffff88072c4d7d40: ffffffff832ad2e7 ffffffff812b9035 ffff880402cb9498 ffff88072c4d7da8
ffff88072c4d7d60: ffffffff81448fa2 ffff88072c4d7e88 ffff880402cb94a8 0000000000000060
ffff88072c4d7d80: 0000000000000001 0000000000000001 0000004c8bbe92c9 ffff88072c4d7e88
ffff88072c4d7da0: ffff88072f22b720 ffff88072f21dd04 071c71c71c71c71c ffffffff83f3f320
ffff88072c4d7dc0: 0000000000000007 0000000000000000 000000000000001f 1ffff100e5e442f9
ffff88072c4d7de0: 000000002dd2f949 ffff88072f2217c8 ffffffffffffff04 ffffffff82d2d9d1
ffff88072c4d7e00: 0000000000000010 0000000000000246 ffff88072c4d7e28 0000000000000018
ffff88072c4d7e20: ffffffff82d2d9cc ffff88072f22b720 ffff88072f22b720 ffffffff82d2d8b5
ffff88072c4d7e40: 0000000000000007 ffffffff82d2e0d5 0000004c8ba3686a 0000004c8bbe92c9
ffff88072c4d7e60: ffff88072c4c8000 ffffffff83f3f320 000000000000000a ffff88072f22b720
ffff88072c4d7e80: ffffffff83f3f320 ffff88072c4d7e98 ffffffff82d2e0e7 ffff88072c4d7ec8
ffff88072c4d7ea0: ffffffff81312417 ffffffff813123d5 ffff88072c4d7f18 ffffed00e5899000
ffff88072c4d7ec0: ffff88072c4c8000 ffff88072c4d7f18 ffffffff81312756 ffffed00e5899000
ffff88072c4d7ee0: ffff88072c4c8000 ffff88072c4c8000 000000000000000a ffff88072f213100
ffff88072c4d7f00: 00000000002ab0ae 0000000000000000 0000000000000000 ffff88072c4d7f48
ffff88072c4d7f20: ffffffff811a0e95 0000000000000000 0000000000000000 0000000000000000
ffff88072c4d7f40: 0000000000000000 0000000000000000 ffffffff810001a5 0000000000000000
ffff88072c4d7f60: 0000000000000000 ...
3Memory state around the buggy address:
3 ffff88072f20bc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88072f20bd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3>ffff88072f20bd80: f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00
3 ^
3 ffff88072f20be00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88072f20be80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3==================================================================

2016-11-17 09:30:42

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 10:13:41AM +0100, Peter Zijlstra wrote:
> On Thu, Nov 17, 2016 at 10:04:46AM +0100, Peter Zijlstra wrote:
> > On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote:
> > > Peter or Vince, can you try to recreate with this patch? It dumps the
> > > raw stack contents during a stack dump. Hopefully that would give a
> > > clue about what's going wrong.
> >
> >
> > Here goes... I'll do another run and get you the results of that as
> > well.
>
> This one is funny, I've not seen that WARNING before. Let me do a third
> run.
>

3==================================================================
3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042f84baa0
3Read of size 8 by task perf_fuzzer/8638
0page:ffffea0010be12c0 count:1 mapcount:0 mapping: (null) index:0x0c
0flags: 0x2ffff8000000400(reserved)
1page dumped because: kasan: bad access detected
dCPU: 1 PID: 8638 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #4
dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
dCall Trace:
d <NMI>
d dump_stack+0x67/0x94
d kasan_report_error+0x4a1/0x4d0
d __asan_report_load8_noabort+0x43/0x50
d ? unwind_get_return_address+0x1fb/0x220
d unwind_get_return_address+0x1fb/0x220
d perf_callchain_kernel+0x356/0x550
d ? arch_perf_update_userpage+0x350/0x350
d ? insn_get_opcode.part.4+0x3ac/0x910
d get_perf_callchain+0x276/0x670
d ? early_serial_putc+0x24/0x70
d ? put_callchain_buffers+0x50/0x50
d ? early_serial_putc+0x1d/0x70
d ? insn_get_displacement.part.7+0x4a/0x500
d perf_callchain+0x128/0x1a0
d ? early_serial_putc+0x1d/0x70
d perf_prepare_sample+0x70e/0xfb0
d ? intel_pmu_pebs_fixup_ip+0x2c9/0x4c0
d perf_event_output_forward+0x93/0x110
d ? perf_prepare_sample+0xfb0/0xfb0
d ? early_serial_putc+0x1d/0x70
d ? early_serial_putc+0x24/0x70
d ? early_serial_putc+0x24/0x70
d ? sched_clock_cpu+0x11c/0x1a0
d __perf_event_overflow+0x1a3/0x570
d perf_event_overflow+0x14/0x20
d __intel_pmu_pebs_event+0x3ca/0x610
d ? pebs_update_state+0x310/0x310
d ? early_serial_putc+0x1d/0x70
d ? intel_pmu_disable_bts+0xc0/0xc0
d ? early_serial_putc+0x1d/0x70
d ? insn_get_prefixes.part.2+0x36d/0xc70
d ? insn_get_opcode.part.4+0x3ac/0x910
d ? number+0x71c/0xa70
d ? insn_get_opcode+0x42/0x50
d ? branch_type+0x122/0x3a0
d ? put_dec+0xb0/0xb0
d ? knc_pmu_handle_irq+0x3a0/0x3a0
d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
d ? __intel_pmu_pebs_event+0x610/0x610
d ? early_serial_write+0x7c/0xf0
d intel_pmu_handle_irq+0x4b2/0xa90
d ? intel_pmu_save_and_restart+0xe0/0xe0
d ? acpi_os_read_memory+0x228/0x262
d ? acpi_os_get_timer+0x1a/0x1a
d ? vunmap_page_range+0x269/0x400
d ? ghes_copy_tofrom_phys+0x149/0x270
d ? ghes_read_estatus+0x11e/0x6b0
d ? ghes_copy_tofrom_phys+0x270/0x270
d perf_event_nmi_handler+0x2d/0x50
d nmi_handle+0x9e/0x250
d default_do_nmi+0x111/0x180
d do_nmi+0x1a2/0x210
d end_repeat_nmi+0x1a/0x1e
dRIP: 0010:__intel_pmu_enable_all+0x11d/0x190
dRSP: 0018:ffff88042419f9b0 EFLAGS: 00000046c
dRAX: 0000000000000000 RBX: ffff88072594aa80 RCX: 000000000000038f
dRDX: 1ffff10085f0a6c8 RSI: 0000000000000000 RDI: ffff88042f853640
dRBP: ffff88042419f9b8 R08: 00000000000004c1 R09: ffffed0085f0a6ca
dR10: 00000000ffffffff R11: 1ffff10085f0a6ec R12: 000000070000000f
dR13: dffffc0000000000 R14: 0000000000000000 R15: 0000000000000002
d ? __intel_pmu_enable_all+0x11d/0x190
d ? __intel_pmu_enable_all+0x11d/0x190
d <EOE>
ffff88042f84a000: 0000000000000000 ...
ffff88042f84a560: ffff8804af84a955 ffff88042f84a95d 0000000000000060 1ffff10085f094b8
ffff88042f84a580: 00000000ffffffff 0020000000000000 0000000000000000 00000000fffffffd
ffff88042f84a5a0: 0000000000000001 0000000000000190 0000000000000010 ffff1060ffffff09
ffff88042f84a5c0: 0000000041b58ab3 ffff8804af84a9bd ffff88042f84a9c5 0000000000000060
ffff88042f84a5e0: 1ffff10085f094c5 00000000ffffffff 0020000000000000 0000000000000000
ffff88042f84a600: 00000000fffffffd 0000000000000001 0000000000000190 ffffffff8348be10
ffff88042f84a620: ffff1060ffffff09 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960
ffff88042f84a640: ffff88042f84a700 ffff88042f313930 ffffffff823b7c54 ffff88042f84a758
ffff88042f84a660: ffff88042f84a760 ffff88042f84a758 ffff88042f84a956 0000000000000001
ffff88042f84a680: ffffffff8348bec9 ffff88042f84a9c5 0000000000000018 dffffc0000000000
ffff88042f84a6a0: ffff88042f84a818 ffff88042f84a768 ffff88042f84a790 ffffffff823b7c54
ffff88042f84a6c0: ffff88042f84a7c0 ffff88042f84a7c8 ffff88042f84a7c0 ffff88042f84a9be
ffff88042f84a6e0: 1ffff10085f094e1 ffffed0085f094f9 000000007fffffff ffff8804af84a9bd
ffff88042f84a700: ffffffff8348beca 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0
ffff88042f84a720: 0000000000000000 ffff1060ffffff09 ffffffff823b8d95 0000000041b58ab3
ffff88042f84a740: ffffffff839ff69c ffffffff823b8d00 0000000000000000 0000000000000020
ffff88042f84a760: ffff88042f84a7e0 ffffed0085f094f4 1ffff10085f0950a ffff88042f84a930
ffff88042f84a780: ffffffff8101d21d ffffffffffffffff ffff88042f84a838 ffffffff823b8d95
ffff88042f84a7a0: 0000000041b58ab3 ffffffff839ff69c ffffffff823b8d00 ffffffff83892478
ffff88042f84a7c0: ffffffff00000020 ffff88042f84a848 ffff88042f84a800 ffff88042f84a870
ffff88042f84a7e0: ffff88042f84a9a8 dffffc0000000000 ffff88042f84a838 ffffffff813c861f
ffff88042f84a800: ffffffff8348beca 0000000000000000 000000000000011d 0000000000000190
ffff88042f84a820: ffffffff83892478 ffffffff83892478 ffff88042f84a9a8 ffff88042f84a958
ffff88042f84a840: ffffffff813c883f 0000000000000016 0000000041b58ab3 ffffffff83a0c758
ffff88042f84a860: ffffffff813c86d0 ffffed0085f09514 0000000000000000 ffff88042f84aa30
ffff88042f84a880: ffff0a00ffffff05 ffff88042f84a962 ffff88042f84ac9e ffff88042f84a963
ffff88042f84a8a0: 0000000000000000 ffff88042f84a900 000000000000011d 0000000000000000
ffff88042f84a8c0: 0000000000000020 1ffff10085f09524 ffff0a00ffffff05 1ffff10085f09524
ffff88042f84a8e0: ffff88042f84a940 ffff0a00ffffff05 ffff88042f84a9ca ffff88042f84acff
ffff88042f84a900: ffff88042f84a9cb 0000000000000000 ffff88042f84a968 ffffffff823adbcd
ffff88042f84a920: 0000000041b58ab3 ffffffff83a40559 1ffff10085f09531 ffff0a00ffffff05
ffff88042f84a940: 1ffff10085f09531 ffff88042f84a9a8 ffff88042f84acdd ffff88042f84aed8
ffff88042f84a960: ffff0a00ffffff05 ffff88042f84aaf0 ffffffff823af620 0000000000000000
ffff88042f84a980: 0000000000000000 0000000041b58ab3 ffffffff83a40559 ffffffff823af530
ffff88042f84a9a0: 0000000000000000 5f6c65746e695f5f 62616e655f756d70 302b6c6c615f656c
ffff88042f84a9c0: 3178302f64313178 0030300030003039 0000000000000000 0000000000000000
ffff88042f84a9e0: 0000000000000000 ffff0a00ffffff05 ffff88042f84aacd ffff88042f84ae22
ffff88042f84aa00: ffff88042f84aace ffff88042f84aed8 ffff88042f84acd8 ffffffff823adb30
ffff88042f84aa20: 1ffff10085f0954d 00000000ffffffff 0020f10085f09551 ffff0a00ffffff00
ffff88042f84aa40: 1ffff10000000010 ffff88042f84aa00 ffff88042f84aa20 ffff88042f84af10
ffff88042f84aa60: ffff103000001005 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960
ffff88042f84aa80: ffffffffffffffff 3266383461613630 6666666638383034 ffffffff00000010
ffff88042f84aaa0: 0000000000000000 0000000000000002 ffff88042f84af10 ffff103000001009
ffff88042f84aac0: 0000000041b58ab3 ffff88042f84aed8 ffff88042f84ad1d ffff88042f84ac30
ffff88042f84aae0: 1ffff10085f09565 ffffffffffffffff 002088042f84abb0 ffffffff823b6600
ffff88042f84ab00: ffffffff00000010 ffff88042f84ad00 ffffffff823b6600 ffff88042f84ac10
ffff88042f84ab20: ffff103000001009 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960
ffff88042f84ab40: ffffffff839f498b 3832336162393630 6666666666666666 ffff88042f84ac2f
ffff88042f84ab60: ffff88042f84abb0 ffffffff823ad785 ffff88042f84ac70 0000000000000001
ffff88042f84ab80: ffff88042f84ad2d ffff88042f84abb0 ffffffff816213e5 ffff88042f84ad2d
ffff88042f84aba0: ffffffff839f498f dffffc0000000000 ffff88042f84ac90 ffffffff823b72eb
ffff88042f84abc0: ffff88042f84afd0 ffff88042f84afd8 ffffffff839f4990 ffff88042f84acd8
ffff88042f84abe0: 1ffff10085f09581 ffffed0085f095fb 0000000000000200 ffff88042f84aed8
ffff88042f84ac00: 0000000000000001 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0
ffff88042f84ac20: ffff88042f84ac70 ffff103000001000 1ffff10085f0958a ffffffff83c88b40
ffff88042f84ac40: ffff88042f84aeb8 ffffffff81338af4 0000000041b58ab3 ffff88042f84ac78
ffff88042f84ac60: ffffffff811c0a51 ffff88042f84ad2e dffffc0000000000 ffff88042f84aca8
ffff88042f84ac80: ffffffff811c0afc ffff88042f84acd8 00000000ffffffff 1ffff10085f09597
ffff88042f84aca0: ffffffff83c88b40 ffff88042f84af20 ffffffff81338af4 0000000041b58ab3
ffff88042f84acc0: ffffffff83a03c41 ffffffff81338a00 ffff0a00ffffff00 3430383866666666
ffff88042f84ace0: 3063636134386632 666666666666203a 6333306133386666 6666666666203134
ffff88042f84ad00: 3236363636363636 3333333620613330 3833333331363033 3636362036363636
ffff88042f84ad20: 3336333032363336 1f000a3633363336 ffffffff83c88b40 ffff88042f84afb0
ffff88042f84ad40: ffffffff81338af4 0000000041b58ab3 ffffffff83a03c41 ffffffff81338a00
ffff88042f84ad60: 0000000041b58ab3 6177647261486401 3a656d616e206572 43206c65746e4920
ffff88042f84ad80: 697461726f70726f 3030363253206e6f 30303632532f5a47 534f4942202c5a47
ffff88042f84ada0: 3030364335455320 2e32302e4236382e 2e323030302e3230 3331303233323231
ffff88042f84adc0: 2f32312030313231 0a333130322f3332 3d3d3d3d3d3d3d00 6f20646165523301
ffff88042f84ade0: 3820657a69732066 6b73617420796220 75665f6672657020 3336382f72657a7a
ffff88042f84ae00: 5f7465675f000a38 615f6e7275746572 302b737365726464 3278302f62663178
ffff88042f84ae20: 6461207461203032 3866666666207264 6234386632343038 000000000a306161
ffff88042f84ae40: 0000000000000000 ...
ffff88042f84ae80: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
ffff88042f84aea0: ffff88042f84bef8 ffffffff8342d740 ffff88042f84af68 ffff88042f84af38
ffff88042f84aec0: ffffffff8133c807 0000000000000000 0000000000000000 0000000000000000
ffff88042f84aee0: 0000000000000000 0000000000000000 0000000000000000 00000000ffffffff
ffff88042f84af00: 0000000000000018 ffff88042f84bef8 ffffffff839f4970 ffff88042f84afd0
ffff88042f84af20: ffff88042f84afa0 ffffffff8133c807 0000000000000000 ffff88042f84afe8
ffff88042f84af40: ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c ffffffff8151fdf3
ffff88042f84af60: 0000000000000000 0000000000000000 000000002f84aff8 1ffff10085f095f6
ffff88042f84af80: 0000000000000018 ffff88042f84bef8 ffff88040672eac0 ffff88042f84c000
ffff88042f84afa0: ffff88042f84b050 ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c
ffff88042f84afc0: ffffffff8151fdf3 ffff88042f84b0b8 ffff880400000030 ffff88042f84b060
ffff88042f84afe0: ffff88042f84b010 ffff88042f84bef8 ffff88040672eac0 ffff88042f84bff8
ffff88042f84b000: ffff88042f84b028 ffffffff813b9d58 0000000000000000 ffff88042f84afe0
ffff88042f84b020: ffff88042f84b028 ffffffff813b9d58 0000000000000000 ffff88042f84afe0
ffff88042f84b040: ffff88042f84b020 0000000000000009 ffff88042f84b120 ffffffff811504e6
ffff88042f84b060: ffffffff8101d21d ffffffff839fbd23 000000002f84b0a0 0000000000000020
ffff88042f84b080: ffffffff839fbd5b ffffffff839fbd1f 0000000000000005 ffff88042f84a000
ffff88042f84b0a0: ffff88042f84c000 ffff88042419f9b0 0000000000000001 ffff880424198000
ffff88042f84b0c0: ffff8804241a0000 0000000000000000 0000000000000022 ffff88040672eac0
ffff88042f84b0e0: 0000000000000000 ffff88042419f9b8 0000000000000000 0000000000000086
ffff88042f84b100: 0000000000000000 ffff88042f84b3c8 ffffed0085f0967a dffffc0000000000
ffff88042f84b120: ffff88042f84b130 ffffffff811505c4 ffff88042f84b150 ffffffff8238dfb3
ffff88042f84b140: ffff88042f84b1e0 ffff88042f84baa0 ffff88042f84b1d0 ffffffff816221c1
ffff88042f84b160: 0000000000000000 0000000000000000 ffff88042f84b1a0 0000000000000097
ffff88042f84b180: ffff88042f84b2b8 ffff88042419f9b8 ffffed0085f09658 0000000000000010
ffff88042f84b1a0: ffff88042f84b1f0 ffff88042f84b2b8 ffff88042f84baa0 ffff88042f84b3c8
ffff88042f84b1c0: ffffed0085f0967a dffffc0000000000 ffff88042f84b208 ffffffff81622323
ffff88042f84b1e0: ffff88042f84baa0 ffff88042f84baa0 0000000000000008 ffff88042f84b200
ffff88042f84b200: ffffffff811c72cb ffff88042f84b248 ffffffff811c72cb ffff88042f84b2c8
ffff88042f84b220: ffff88042f84b2b8 ffff88042f84b2b8 ffff88042f84b338 ffff88042f84b3c8
ffff88042f84b240: ffffed0085f0967a ffff88042f84b360 ffffffff8100f356 0000000000000000
ffff88042f84b260: 0000000000000000 1ffff10085f09650 ffff88042f84b3d4 1ffff10085f09653
ffff88042f84b280: ffff88042f84b3da 0000000000000004 ffff88042f84b3d0 0000000041b58ab3
ffff88042f84b2a0: ffffffff839f613c ffffffff8100f000 ffffed0085f096a6 0000000000000005
ffff88042f84b2c0: ffff88042f84a000 ffff88042f84c000 ffff88042419f9b0 0000000000000020
ffff88042f84b2e0: ffff88040672eac0 0000000000000000 ffff88042f84ba98 0000000000000000
ffff88042f84b300: ffff88042f84b380 ffffffff823bbf3c 0000000000000000 0000000000000000
ffff88042f84b320: 0000000000000000 ffff88042f84b548 ffff88042f84b530 000000000001bdc0
ffff88042f84b340: ffff88042f84b408 ffff8804222eadc0 1ffff10085f09675 ffff8804222eadc0
ffff88042f84b360: ffff88042f84b430 ffffffff81513f16 0000000000000007 ffffffff811c0a34
ffff88042f84b380: ffff880400000000 ffff88042f84b6e0 0000000000000000 ffff880400000001
ffff88042f84b3a0: 000000000000000c 0000000041b58ab3 ffffffff83a0a369 ffffffff81513ca0
ffff88042f84b3c0: ffff88042f84b3e8 ffff8804222eadc0 000000010000007f ffff88042f000001
ffff88042f84b3e0: ffffffff811c0a2d ffff88042f84b410 ffffffff823bccda 00000000000021be
ffff88042f84b400: 0000000000000000 ffff88072594aa80 0000000000000040 ffff88042f84b6e0
ffff88042f84b420: ffff880419217c00 0000000000000000 ffff88042f84b478 ffffffff81514438
ffff88042f84b440: ffffffff00000001 ffffffff811c0a2d ffff88042f84b51c ffff88072594aa80
ffff88042f84b460: ffff88042f84b518 ffff88072594aa80 ffff88042f84b7c0 ffff88042f84b4e8
ffff88042f84b480: ffffffff8150ac7e ffff88042f84b5b0 ffffffff81026e69 ffff88042f84b6e0
ffff88042f84b4a0: 1ffff10085f09699 00000000000000a3 ffff88042f84b51e ffff88042f84b6e0
ffff88042f84b4c0: 1ffff10085f0969f ffff88072594aa80 ffff88042f84b7c0 ffff88042f84b6e0
ffff88042f84b4e0: ffff88042f84b6e0 ffff88042f84b5d8 ffffffff8150b5b3 0000000041b58ab3
ffff88042f84b500: ffffffff83a0d7b0 ffffffff8150b520 0000000100000000 0020400100000009
ffff88042f84b520: 0000000100000000 0000000000000000 0107080800004000 ffffffff811c0a2d
ffff88042f84b540: ffffffff811c0a34 ffffffff811c0a34 0000000000000000 0000000000000000
ffff88042f84b560: 0000000000000000 ffff88072594aa80 ffff88072594aa80 ffff88042f84b7c0
ffff88042f84b580: 000000000000464d ffff88042f84b5d8 ffffffff812c58ec ffff88042f84b7c0
ffff88042f84b5a0: ffff88072594aa80 00000000000000a3 ffff88072594aa80 ffff88072594aa80
ffff88042f84b5c0: ffff88042f84b7c0 ffff88072594ad74 0000000000000000 ffff88042f84b620
ffff88042f84b5e0: ffffffff814e7c13 ffff88042f84b7d0 ffff880400000000 ffff88072594aa80
ffff88042f84b600: dffffc0000000000 ffff88042c7e00b0 ffff88042f84b6e0 ffff88042f84b7c0
ffff88042f84b620: ffff88042f84b630 ffffffff8150cb54 ffff88042f84b9b0 ffffffff8102901a
ffff88042f84b640: 0000000000000000 ...
ffff88042f84b660: 0000000000000000 1ffff10085f096d8 0000000000000000 0000000000000000
ffff88042f84b680: 0000000000000000 0000000000000000 ffff88042f84b7c0 ffff88042f84bef8
ffff88042f84b6a0: ffff88072594aa80 0000000000000000 0000000000000000 ffff88042f84b6e0
ffff88042f84b6c0: 0000000041b58ab3 ffffffff839f20b0 ffffffff81028c50 0000000000000000
ffff88042f84b6e0: 0000000000000002 0000000000000000 dffffc0000000000 000000070000000f
ffff88042f84b700: ffff88042f84ba98 ffff88072594aa80 1ffff10085f0a6ec 00000000ffffffff
ffff88042f84b720: ffffed0085f0a6ca 00000000000004c1 0000000000000000 000000000000038f
ffff88042f84b740: 1ffff10085f0a6c8 0000000000000000 ffff88042f853640 ffffffffffffffff
ffff88042f84b760: ffffffff811c0a2d 0000000000000010 000000000000000e ffff88042f84ba88
ffff88042f84b780: 0000000000000018 ffffffff839f2108 ffffffff8102b200 0000000000000000
ffff88042f84b7a0: 0000000000000000 ...
ffff88042f84b7c0: 0000000000000000 0000000000000000 0000000000000000 000000000000090f
ffff88042f84b7e0: 0000000000000000 0000000000000000 0000000005080021 00000000000000a3
ffff88042f84b800: ffffffff811c0a2d 000021be000021be 0000000000000000 0000000000000000
ffff88042f84b820: 0000000000000000 0000000000000001 ffff88042f84b8a8 ffffffff823bb28d
ffff88042f84b840: 0000000000000000 ...
ffff88042f84b860: ffff88042f84b9c0 ffff88042f84ba18 ffff88042f84ba20 ffff88042f84b974
ffff88042f84b880: ffff88042f84b9c4 ffff88042f84b9c0 ffff88042f84ba60 0000000000000002
ffff88042f84b8a0: ffff88042f84b9c4 ffff88042f84b928 ffffffff823bbf3c ffffffff823ac07c
ffff88042f84b8c0: ffff88042f84b9c0 ffff88042f84b9c0 ffff88042f84ba20 ffff88042f84ba08
ffff88042f84b8e0: 0000000000000068 ffff88042f84b910 ffff88042f84b9d0 ffff88042f84ba18
ffff88042f84b900: ffff88042f84b9c0 ffff88042f84b9c0 ffff88042f84ba60 0000000000000002
ffff88042f84b920: 0000000000000002 ffff88042f84b940 ffffffff823beb42 1ffff10085f0972c
ffff88042f84b940: ffff88042f84ba88 ffffffff8102cf02 ffffffff83a40503 ffffffff823ab960
ffff88042f84b960: 0000000041b58ab3 ffffffff839f711e ffffffff8102cde0 0000000000000000
ffff88042f84b980: 0000000000000000 0000000000000000 dffffc0000000000 ffff88042f84bb08
ffff88042f84b9a0: ffff88072594aa80 ffff88042f853440 ffff88042f84bb30 ffffffff81029856
ffff88042f84b9c0: 0000000100000000 0000000100000000 ffff88042f853d78 ffff88042f84bef8
ffff88042f84b9e0: 1ffff10085f09745 ffff88042c7e0000 ffff88042f84ba88 0000000000000007
ffff88042f84ba00: ffffed0085f0a7af ffff88042c7e00b0 fffffbfff082adbc ffff88042f84bac8
ffff88042f84ba20: ffff88042f84ba88 0000000041b58ab3 ffffffff839f20d0 ffffffff81029260
ffff88042f84ba40: 0000000000000000 0000000000000001 0000000000000000 0000000000000000
ffff88042f84ba60: ffff88042f853f00 000000000000000f ffff88042f853440 dffffc0000000000
ffff88042f84ba80: 000000000000000f 0000000000000001 0000000000000000 ffff88042f84bac8
ffff88042f84baa0: ffffffff811c0afc fffffbfff082adc4 00000003ffffffff 1ffff10085f0975b
ffff88042f84bac0: ffffffff83c88b40 0000000000000000 0000000000000000 ffff88042f853d90
ffff88042f84bae0: ffffffff00000000 ffffffff00029ffe 0000000000000000 ffffed0085f0a7b2
ffff88042f84bb00: ffff88042f853f10 0000000000000001 000000c76b33c80d ffffffff83c16120
ffff88042f84bb20: 000000c76b33c7fa ffff88042f853440 ffff88042f84bdf8 ffffffff8101e752
ffff88042f84bb40: ffff88042f853d78 1ffff10085f09770 ffffed0085f0a7af 0000006400000000
ffff88042f84bb60: 0000000000000000 ffff88042f84bef8 0000000000000000 ffff88042f84bd60
ffff88042f84bb80: 0000000041b58ab3 ffffffff839f1dd0 ffffffff8101e2a0 ffff88042f84bc40
ffff88042f84bba0: ffffffff8256fd3e ffffc900000c5000 1ffff10085f09777 0000000041b58ab3
ffff88042f84bbc0: 0000000000000000 ffffffff8256fb16 0000000000000000 0000000000000000
ffff88042f84bbe0: 0000000000000000 ...
ffff88042f84bc00: 0000000000000000 0000000000000000 0000000000000000 1ffff10085f0978a
ffff88042f84bc20: ffff88042f84bcf0 ffffc900000cff94 ffff88042f84bd60 dffffc0000000000
ffff88042f84bc40: ffff88072ec02000 ffff88042f84bcb8 ffffffff815e0639 ffff88072ec02000
ffff88042f84bc60: ffff88042f62d000 ffffc900000e9fff ffffc900000ea000 ffffffff83c0bc98
ffff88042f84bc80: ffffc900000e9fff ffffc900000ea000 ffffc900000e9000 0000000000000000
ffff88042f84bca0: ffff88072951f4e0 00000000bdceb03c ffff880423ef2214 ffff88042f84bcc8
ffff88042f84bcc0: ffffc900000e9000 ffff88042f84bd18 ffffffff82609379 0000000000000014
ffff88042f84bce0: 0000000000000000 0010000000000001 ffff8807294f4300 1ffff10085f097a8
ffff88042f84bd00: ffffc900000cff80 ffff88042f84bda0 0000000000000001 ffff88042f84bdc8
ffff88042f84bd20: ffffffff826095be ffffffff83c16120 000000c76aa445a8 ffff8807294f4308
ffff88042f84bd40: 0000000041b58ab3 ffffffff83a49886 ffffffff826094a0 ffffffff83a03c64
ffff88042f84bd60: 00000000bdceb028 0000000000000000 0000000000000020 ffff88042f84bdf8
ffff88042f84bd80: ffff88042f84bdb0 ffff88042f84bdb0 ffffffff84c270c0 dffffc0000000000
ffff88042f84bda0: ffff8807294f4300 dffffc0000000000 ffffffff83f6e7c0 0000000000000000
ffff88042f84bdc0: ffffffff83f6ea20 ffff88042f84be18 ffff88042f84bef8 000000c76b33c80d
ffff88042f84bde0: ffffffff83c16120 000000c76b33c7fa ffffffff83c16130 ffff88042f84be18
ffff88042f84be00: ffffffff8100a25d dffffc0000000000 0000000000000001 ffff88042f84be80
ffff88042f84be20: ffffffff81150f5e 0000000000000000 0000000000000000 ffff88042f84bef8
ffff88042f84be40: 0000000000000000 ffffffff83c64ce8 00000000000004a1 ffff88042f84bef8
ffff88042f84be60: 0000000000000001 0000000000000007 ffff88040672f23c ffff88042f84bf90
ffff88042f84be80: ffff88042f84bea8 ffffffff81151b41 ffff88040672eac0 ffff88042f84bef8
ffff88042f84bea0: ffff88040672eac0 ffff88042f84bee8 ffffffff81151d52 0000000000000000
ffff88042f84bec0: 0000000000000001 000000070000000f dffffc0000000000 0000000000000000
ffff88042f84bee0: 0000000000000002 ffff88042f84bef9 ffffffff832ad08e 0000000000000002
ffff88042f84bf00: 0000000000000000 dffffc0000000000 000000070000000f ffff88042419f9b8
ffff88042f84bf20: ffff88072594aa80 1ffff10085f0a6ec 00000000ffffffff ffffed0085f0a6ca
ffff88042f84bf40: 00000000000004c1 0000000000000000 000000000000038f 1ffff10085f0a6c8
ffff88042f84bf60: 0000000000000000 ffff88042f853640 ffffffffffffffff ffffffff8101d21d
ffff88042f84bf80: 0000000000000010 0000000000000046 ffff88042419f9b0 0000000000000018
ffff88042f84bfa0: ffffffff8101d21d 0000000000000010 0000000000000046 ffff88042419f9b0
ffff88042f84bfc0: 0000000000000018 0000000000000001 1ffff10085f0a6c8 ffffffff8101d21d
ffff88042f84bfe0: 0000000000000010 0000000000000046 ffff88042419f9b0 0000000000000018
d intel_pmu_enable_all+0x10/0x20
d x86_pmu_enable+0x65e/0xb10
d ? ctx_sched_in+0xe8c/0x15a0
d perf_pmu_enable.part.97+0x29/0x40
d ctx_resched+0x147/0x1a0
d __perf_event_enable+0x714/0xb60
d ? wp_page_copy+0x857/0x13f0
d ? perf_mux_hrtimer_handler+0x670/0x670
d event_function+0x1de/0x380
d ? cpu_clock_event_read+0x30/0x30
d ? perf_cgroup_attach+0xb0/0xb0
d remote_function+0xfe/0x180
d generic_exec_single+0x1a3/0x300
d ? perf_cgroup_attach+0xb0/0xb0
d smp_call_function_single+0x1b2/0x310
d ? generic_exec_single+0x300/0x300
d ? generic_exec_single+0x113/0x300
d ? vm_normal_page+0x170/0x170
d task_function_call+0xb1/0x100
d ? perf_event_addr_filters_exec+0x360/0x360
d ? cpu_clock_event_read+0x30/0x30
d event_function_call+0x1ac/0x2e0
d ? perf_mux_hrtimer_handler+0x670/0x670
d ? task_function_call+0x100/0x100
d ? perf_mux_hrtimer_handler+0x670/0x670
d ? cpu_clock_event_read+0x30/0x30
d ? do_vfs_ioctl+0x192/0xf00
d ? _perf_event_disable+0xa0/0xa0
d _perf_event_enable+0x8f/0xc0
d perf_event_for_each_child+0x83/0x160
d perf_event_task_enable+0x8b/0x100
d SyS_prctl+0x761/0xb20
d ? SyS_umask+0x60/0x60
d ? trace_hardirqs_on_thunk+0x1a/0x1c
d entry_SYSCALL_64_fastpath+0x18/0xa8
dRIP: 0033:0x7f382ac68a4a
dRSP: 002b:00007ffc0d8588e8 EFLAGS: 00000202c ORIG_RAX: 000000000000009d
dRAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f382ac68a4a
dRDX: 0000000000000001 RSI: 00007ffc0d8588cc RDI: 0000000000000020
dRBP: 00007ffc0d858900 R08: 00007f382af1c0a4 R09: 00007f382af1c120
dR10: 00007f382af1c0a4 R11: 0000000000000202 R12: 0000000000401810
dR13: 00007ffc0d85ac80 R14: 0000000000000000 R15: 0000000000000000
ffff880424198000: 0000000057ac6e9d 0000000000000000 0000000000000000 0000000000000000
ffff880424198020: 0000000000000000 ...
ffff880424198820: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
ffff880424198840: 0000000000000000 ...
ffff8804241988a0: 0000000000000000 0000000000000000 ffff880424198000 0000000000000000
ffff8804241988c0: 0000000000000000 ...
ffff8804241990e0: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
ffff880424199100: 0000000000000000 ...
ffff880424199160: ffff8804241988b0 0000000000000000 0000000000000000 0000000000000000
ffff880424199180: 0000000000000000 ...
ffff880424199980: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
ffff8804241999a0: 0000000000000000 ...
ffff880424199a00: 0000000000000000 0000000000000000 ffff880424199160 0000000000000000
ffff880424199a20: 0000000000000000 ...
ffff88042419a240: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
ffff88042419a260: 0000000000000000 ...
ffff88042419a2c0: ffff880424199a10 0000000000000000 0000000000000000 0000000000000000
ffff88042419a2e0: 0000000000000000 ...
ffff88042419aae0: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
ffff88042419ab00: 0000000000000000 ...
ffff88042419ab60: 0000000000000000 0000000000000000 ffff88042419a2c0 0000000000000000
ffff88042419ab80: 0000000000000000 ...
ffff88042419b3a0: c1c000d000000001 c36000d000000001 0000000000000000 0000000000000000
ffff88042419b3c0: 0000000000000000 ...
ffff88042419b420: ffff88042419ab70 0000000000000000 0000000000000000 0000000000000000
ffff88042419b440: 0000000000000000 ...
ffff88042419bc40: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
ffff88042419bc60: 0000000000000000 ...
ffff88042419bcc0: 0000000000000000 0000000000000000 ffff88042419b420 0000000000000000
ffff88042419bce0: 0000000000000000 ...
ffff88042419c500: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
ffff88042419c520: 0000000000000000 ...
ffff88042419c580: ffff88042419bcd0 0000000000000000 0000000000000000 0000000000000000
ffff88042419c5a0: 0000000000000000 ...
ffff88042419cda0: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
ffff88042419cdc0: 0000000000000000 ...
ffff88042419ce20: 0000000000000000 0000000000000000 ffff88042419c580 0000000000000000
ffff88042419ce40: 0000000000000000 ...
ffff88042419d660: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
ffff88042419d680: 0000000000000000 ...
ffff88042419d6e0: ffff88042419ce30 0000000000000000 0000000000000000 0000000000000000
ffff88042419d700: 0000000000000000 ...
ffff88042419dec0: 0000000000000000 0000000000000000 0000000000000000 ffff880424190090
ffff88042419dee0: ffff880423db0098 dffffc0000000000 ffff880424190080 ffffed008483201a
ffff88042419df00: ffff88042419df60 ffffffff812c8399 c66000d000000001 c80000d000000001
ffff88042419df20: 0000000000000000 ffff880423db0080 ffff880423db0090 ffff88042fca0de8
ffff88042419df40: ffff88042fca0eb0 ffff88042fca0e18 ffff88042419dff8 ffffffff812e90f7
ffff88042419df60: ffff88042419dff8 ffffffff812e2eae 0000000000000000 0000000000000000
ffff88042419df80: 0000000000000000 0000000000000000 ffff88042fca0de8 ffff880423db0080
ffff88042419dfa0: ffff880423db0080 ffff880423db0080 ffff88042fca0e28 ffff880424190210
ffff88042419dfc0: ffff88042fca0eb0 0000000000000000 ffff880423db0000 ffff880423db0080
ffff88042419dfe0: dffffc0000000000 ffff880424190080 ffff88042fca0d80 ffff88042419e110
ffff88042419e000: ffffffff812fb6d1 ffff88042fca16b0 ffff88042fca0de8 1ffff10084833c11
ffff88042419e020: 0000000000000000 0000000000000000 0000000000000000 ffffed0085f942d6
ffff88042419e040: ffff88042fca0df8 0000000000000000 ffff880424190000 0000000000000000
ffff88042419e060: 0000000000000000 0000000000000000 0000000000000000 ffff880424190b40
ffff88042419e080: 1ffff10084833c11 0000000041b58ab3 ffffffff839fba92 ffffffff81144cf0
ffff88042419e0a0: 0000000000000000 ffff88042419e198 ffff88042419e190 ffff88042419e1f8
ffff88042419e0c0: 0000000000000001 ffff88042419e1f8 ffff88042419e1d0 ffff88042419e130
ffff88042419e0e0: ffffffff811c75da ffff88042fca0d80 0000000000000000 ffff88042419e198
ffff88042419e100: ffff88042419e190 ffff88042419ff48 0000000000000001 ffff88042419ff48
ffff88042419e120: ffff88042419e1d0 ffff88042419e180 ffffffff811c75da ffff88042419e190
ffff88042419e140: ffff88042419e180 ffffffff811c71d2 ffff88042419e1b8 ffff88042419e218
ffff88042419e160: ffff88042419e218 0000000000000000 0000000000000000 ffff880424190000
ffff88042419e180: ffff88042419e1f8 ffffffff8116dfd3 0000000000000001 ffffffff832ab679
ffff88042419e1a0: 00000000c019c80a 000000000000002e ffff88042419e208 ffffffff8242feaa
ffff88042419e1c0: 0000000000000190 ffff88042419e230 000000000009c80a 0000000000000000
ffff88042419e1e0: ffff88042419e2d0 ffff88042419e2c8 ffff88042419e330 0000000000000001
ffff88042419e200: ffff88042419e330 ffff88042419e308 ffff88042419e268 ffffffff811c75da
ffff88042419e220: ffffffff811c75da 0000000000000000 ffff88042419e2d0 ffff88042419e2c8
ffff88042419e240: ffff88042419ff48 0000000000000001 ffff88042419ff48 ffff88042419e308
ffff88042419e260: ffff88042419e2b8 ffffffff811c75da ffff88042419e2c8 ffff88042419e2b8
ffff88042419e280: ffffffff811c71d2 ffff88042419e2f0 ffff88042419e350 ffff88042419e350
ffff88042419e2a0: 0000000000000000 0000000000000000 ffff88040672eac0 ffff88042419e330
ffff88042419e2c0: ffffffff8116dfd3 00000000a897a839 0000000000000120 ffff88042419e368
ffff88042419e2e0: ffff88070a70d7e0 ffff88042419e340 ffffffff8242fc8f 0000000002408240
ffff88042419e300: ffff88042419e350 000000000007a839 0000000000000000 0000000002408240
ffff88042419e320: ffff88070c23b950 0000000002408240 ffff88070c23b950 ffff880727f22e80
ffff88042419e340: ffff88042419e570 ffffffff81621065 0000004000000024 ffff88042419e368
ffff88042419e360: 0000000000000000 ffffffff8116e03b ffffffff81620ff6 ffffffff8162126d
ffff88042419e380: ffffffff816217d2 ffffffff8161d1e0 ffffffff81ed2aa1 ffffffff81ed00cd
ffff88042419e3a0: ffffffff81f13e2b ffffffff81f14af4 ffffffff81da4df2 ffffffff81da51dc
ffff88042419e3c0: ffffffff81dbe1a2 ffffffff81d425f2 ffffffff81d43145 ffffffff81d4ce2d
ffff88042419e3e0: ffffffff81d4dfeb ffffffff81d50c7f ffffffff81d8f93d ffffffff81d91111
ffff88042419e400: ffffffff81d93171 ffffffff81e9fb75 ffffffff81e560ba ffffffff81e5a578
ffff88042419e420: ffffffff8154c563 ffffffff81e59732 ffffffff8154e6ea ffffffff81527f8a
ffff88042419e440: ffffffff8152814c ffffffff81eafe8d ffffffff81e7eee5 ffffffff8165fe8a
ffff88042419e460: ffffffff8166030e ffffffff8128a10a ffffffff81002a09 ffffffff81005265
ffff88042419e480: ffffffff832abcb8 ffff88041c85c8e0 ffffed008390b91c ffff880422c64d60
ffff88042419e4a0: ffff88070c23b930 1ffff10084833ca7 ffff88070ba04cb8 ffff88041c85c850
ffff88042419e4c0: ffff880422c64c40 ffff88042419e518 ffffffff81f15847 ffff88041c85c980
ffff88042419e4e0: 0000000600000007 ffff880422c64c6c ffff8806fbb5c818 1ffff10084833ca7
ffff88042419e500: ffff88042419e5d8 ffff880422c64c40 ffff88040b9780b0 ffff88042419e5f8
ffff88042419e520: ffffffff81dab421 0000000000000286 ffff88041c85c8e0 0000000041b58ab3
ffff88042419e540: ffffffff83a33fb7 ffffffff81dab2f0 00000000000009af 1ffff10084833caf
ffff88042419e560: ffff88042419e5f8 ffffffff81d9fb81 ffff88042419e5c0 0000000041b58ab3
ffff88042419e580: ffffffff83a34296 ffffffff81d9fa90 ffff880422c64c6c ffff8806875c0000
ffff88042419e5a0: 00000000000009af 0000000000000015 ffff880422c64c40 ffff88040b9780b0
ffff88042419e5c0: ffff88042419e5f8 ffffffff81dab2ca ffffffff81da10b9 ffff88042419e850
ffff88042419e5e0: ffff8806fbb5c818 ffff88040b978000 0000000000000015 ffff88042419e878
ffff88042419e600: ffffffff81dc3893 ffffffff81ed2aa1 ffff88040672ef80 ffff88040672eac0
ffff88042419e620: 1ffff10080ce5df0 ffff88040b978006 ffff8806fbb5c820 ffff88042419e770
ffff88042419e640: ffff88042419e9d0 ffff88042419e990 000000000359dc90 ffff88042419ea10
ffff88042419e660: ffff8806fbb5c828 1ffff10084833cd2 ffff880700000133 ffff88042419e950
ffff88042419e680: ffff8806fbb5c830 ffff88042419e778 ffff88042419e770 ffff88042419e788
ffff88042419e6a0: ffff88042419e780 ffff88042419e7e8 0000000000000001 ffff88042419e7e8
ffff88042419e6c0: ffff88042419e7c0 ffff88042419e720 ffffffff811c75da ffff88042419e778
ffff88042419e6e0: 0000000000000000 ffff88042419e788 ffff88042419e780 ffff88042419ff48
ffff88042419e700: 0000000000000001 ffff88042419ff48 ffff88042419e7c0 ffff88042419e770
ffff88042419e720: ffff88042419e810 ffff88042419e808 ffff88042419e870 0000000000000001
ffff88042419e740: ffff88042419e870 ffff88042419e848 ffff88042419e7a8 ffffffff811c75da
ffff88042419e760: 0000000000000000 0000000000000000 ffff88042419e810 ffff88042419e808
ffff88042419e780: ffff88042419ff48 0000000000000001 ffff88042419ff48 ffff88042419e848
ffff88042419e7a0: ffff88042419e7f8 ffffffff811c75da ffff88042419e808 ffff88042419e7f8
ffff88042419e7c0: ffffffff811c71d2 ffff88042419e830 ffff88042419e890 ffff88042419e890
ffff88042419e7e0: 0000000000000000 0000000000000000 ffff88040672eac0 ffff88042419e870
ffff88042419e800: ffffffff8116dfd3 00000000931ae748 00000000000000d8 ffff88042419e8a8
ffff88042419e820: ffff88070be15c80 ffff88042419e880 ffffffff8242fc8f 0000000002000000
ffff88042419e840: ffff88042419e890 00000000000ae748 0000000000000000 0000000002000000
ffff88042419e860: ffff8806fbb5c940 ffff8806fbb5ca28 ffffea001beed700 ffff8806fbb5c940
ffff88042419e880: ffff88042419eab0 ffffffff81621065 000000400000001b ffff88042419e8a8
ffff88042419e8a0: ffffffff00000000 ffffffff8116e03b ffffffff81620ff6 ffffffff81621851
ffff88042419e8c0: ffffffff8161d85e ffffffff81da8a5a ffffffff81d4ce60 ffffffff81d4dfeb
ffff88042419e8e0: ffffffff81d50c7f ffffffff81d8f93d ffffffff81d91111 ffffffff81d93171
ffff88042419e900: ffffffff81e9fb75 ffffffff81e560ba ffffffff81e5a578 ffffffff8154c563
ffff88042419e920: ffffffff81e59732 ffffffff8154e6ea ffffffff81527f8a ffffffff8152814c
ffff88042419e940: ffffffff81eafe8d ffffffff81e7eee5 ffffffff8165fe8a ffffffff8166030e
ffff88042419e960: ffffffff8128a10a ffffffff81002a09 ffffffff81005265 ffffffff832abcb8
ffff88042419e980: ffffffffffffffff 0000000000000000 ffff88042419ea50 0000000041b58ab3
ffff88042419e9a0: ffffffff83a33ff3 ffffffff81dbfe60 ffff880727f22880 ffff880422c648c0
ffff88042419e9c0: ffffffff8161d85e 0000000000000000 010000007e4d0700 ffff88042419ea78
ffff88042419e9e0: 0000000000000296 0000000000000000 0000000000000292 0000000000000000
ffff88042419ea00: 0000000000000000 0000000000000000 ffff880727f22e80 ffff88070c23b870
ffff88042419ea20: ffff88070c23b890 ffff88042419ea50 ffffffff81621867 ffff88070c23b870
ffff88042419ea40: ffffffff81ecdff4 ffff880727f22e80 ffff88042419ea80 0000000000000282
ffff88042419ea60: ffff8804135011b0 ffff8804135011b8 ffff88042419ea98 0000000000000282
ffff88042419ea80: 0000000000000000 0000000000000000 0000000000000000 ffff880727f22880
ffff88042419eaa0: ffff8806fbb5c940 ffff8806fbb5ca28 ffff88042419ead8 ffffffff81621867
ffff88042419eac0: ffff8806fbb5c940 ffffffff81da8a5a ffff880727f22880 ffff88042419eb08
ffff88042419eae0: ffffffff8161d85e 0000000000000002 ffff8806fbb5c9f5 1ffff100df76b93e
ffff88042419eb00: dffffc0000000000 ffff88042419eb58 ffffffff81da8a5a ffffffff81d53833
ffff88042419eb20: 000000000000001c ffffed00df76b928 ffff88042419f1f8 0000000000000001
ffff88042419eb40: 0000000000000000 ffff88042419efc8 ffff8806fbb5c818 ffff88042419eff0
ffff88042419eb60: ffffffff81d4ce60 ffff880400000001 ffff88042419ed08 ffffffff00000000
ffff88042419eb80: ffffffff81d8f93d ffffffff81d91111 ffffffff81d93171 ffffffff81e9fb75
ffff88042419eba0: ffffffff81e560ba ffffffff81e5a578 000000028154c563 ffff88042419f22c
ffff88042419ebc0: ffffed0084833e4d ffff88042419f269 1ffff10084833d85 ffff88042419f200
ffff88042419ebe0: ffff88042419f228 ffff88042419f208 ffff88042419f254 ffff88042419f234
ffff88042419ec00: ffff88042419f250 ffffffff81005265 ffff88042419ed88 ffff88040000000a
ffff88042419ec20: ffff880600000002 0000000041b58ab3 ffffffff83a1d770 ffffffff81d4b4f0
ffff88042419ec40: 1ffff10080ce5df0 ffff880700075765 ffff88070ba04cb8 0000000000000001
ffff88042419ec60: ffff88070a704980 ffff880413500fc0 ffff88042419ecb0 ffffffff81ed00cd
ffff88042419ec80: ffff8804135010b8 ffff880400075765 ffff88070ba04cb8 0000000000000001
ffff88042419eca0: 1ffff10084833dad ffff880413500fc0 ffff88042419ece0 ffffffff81f13e2b
ffff88042419ecc0: ffff88070ba04cb8 ffff880400000003 ffff88070ba04cb8 0000000000000000
ffff88042419ece0: ffff88042419ed28 ffffffff81f14af4 0000000100000000 ffff880400000000
ffff88042419ed00: ffff8804191f2200 0000000000000003 0000000000000000 0000000000000000
ffff88042419ed20: 0000000000000000 ffff880727f23000 ffff88070a704980 ffff88070a704a70
ffff88042419ed40: ffff88042419ed68 ffffffff81621867 ffff88070a704980 ffffffff81ee9d85
ffff88042419ed60: ffff880727f23000 ffff88042419ed98 ffffffff8161d85e ffff88070a704980
ffff88042419ed80: 0000000000000292 ffff880400000001 ffff880413500ff8 ffff88042419edc0
ffff88042419eda0: ffffffff8131f879 ffff880413501008 ffff88042419edf8 ffffffff81e6e5f1
ffff88042419edc0: ffff880400000000 ffff880400000000 ffff88070a704a00 ffff88070ba04cb8
ffff88042419ede0: ffff88070a704980 ffff88042419f030 ffff880413500fc0 ffff88042419ee30
ffff88042419ee00: ffffffff81f15244 ffff880400074d7d ffff88070ba04cb8 ffff88041c85c2f0
ffff88042419ee20: ffffed008390b85e ffff8804152619a0 ffff88070c23b7b0 ffff88042419ef88
ffff88042419ee40: ffff88070ba04cb8 ffff88041c85c260 ffff880415261880 ffff88042419eea8
ffff88042419ee60: ffffffff81f15847 ffff88042419ef70 0000003400000037 ffff8804152618ac
ffff88042419ee80: 0000000000000200 ffff88042419ef88 ffff88042419ef80 ffff88042419ef78
ffff88042419eea0: ffff88042419efe0 0000000000000001 ffff88042419efe0 ffff88042419efb8
ffff88042419eec0: ffff88042419ef18 ffffffff811c75da 0000000000000001 0000000000000000
ffff88042419eee0: ffff88042419ef80 ffff88042419ef78 ffff88042419ff48 0000000000000001
ffff88042419ef00: ffff88042419ff48 ffff88042419efb8 ffff88042419ef68 ffffffff811c75da
ffff88042419ef20: ffff88042419ef78 ffff88042419ef68 ffffffff811c71d2 ffff88042419efa0
ffff88042419ef40: ffff88042419f000 ffff88042419f000 0000000000000000 0000000000000000
ffff88042419ef60: ffff88040672eac0 ffff88042419efe0 ffffffff8116dfd3 00000000b1fe153b
ffff88042419ef80: 0000000000000120 ffff88042419f018 ffff88070ba2a6b0 ffff88042419eff0
ffff88042419efa0: ffffffff8242fc8f 0000000002091220 ffff88042419f000 ffff880709a1d018
ffff88042419efc0: ffff88042419f010 ffffffff81448fa2 ffff8807154d1640 ffff880709a1d028
ffff88042419efe0: ffffffff81510c31 0000000000000008 0000000000000000 ffff88042419f460
ffff88042419f000: ffff8804298a9100 ffff8804298a9308 ffff88042419f0c8 ffffffff832ad2e7
ffff88042419f020: ffffffff81510c31 0000000000000008 0000000000000000 ffff88042419f4a0
ffff88042419f040: ffff88042419f3f0 ffff8804298a9308 ffff88042419f108 ffffffff832ad2e7
ffff88042419f060: ffff88040672f358 ffff88042419f7d6 ffff8804298a9100 0000000000000000
ffff88042419f080: 1ffff10085315275 ffff8804298a9100 ffff8804298a93a8 fffffbfff082b618
ffff88042419f0a0: ffffffff81359c15 ffff88042419f826 ffff8804298a9100 0000000000000000
ffff88042419f0c0: 1ffff10085315275 ffff88042419f0d8 ffffffff81359c15 ffff88042419f3f0
ffff88042419f0e0: 0000000000000008 0000000000000000 ffff88042419f3f0 ffffffff81510c31
ffff88042419f100: ffffed0084833e43 ffff88042419f118 ffffffff81359c15 ffff88042419f430
ffff88042419f120: 0000000000000008 0000000000000000 ffff88042419f430 ffffffff81510c31
ffff88042419f140: 1ffff10084833e3a 000000012419f1a0 ffff88042419f53c ffff88042419f218
ffff88042419f160: 1ffff10084833e30 ffffffe42419f318 ffff8804298a9100 ffff88042419f460
ffff88042419f180: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 000000000716dfd3
ffff88042419f1a0: 1ffff10084833e38 ffffffe42419f548 ffff8804298a9100 ffff88042419f4a0
ffff88042419f1c0: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffffffff83a032f4
ffff88042419f1e0: ffffffff812f5fc0 0000000000000001 0000000000000001 000000000000000c
ffff88042419f200: 000000000000000c 0000000000000000 0000000000002e04 000000000000022c
ffff88042419f220: 0000001400000000 0000000000000014 0000000000000000 0000000000000000
ffff88042419f240: ffff88042419f250 ffff880709a1d018 ffff88042419f2a0 ffffffff81448fa2
ffff88042419f260: 0000000000000050 ffff880709a1d018 ffff88042419f2c0 ffffffff81448fa2
ffff88042419f280: ffff8800032b6000 ffff880709a1d028 ffffffff811d236d 80000000032b6161
ffff88042419f2a0: ffff880709a1d018 ffff88042419f2f8 ffffffff81448fa2 ffff880709a1d018
ffff88042419f2c0: 0000000041b58ab3 ffffffff83a0d680 ffffffff814e3680 dffffc0000000000
ffff88042419f2e0: 0000000000000012 ffff88042419f538 ffff880400000001 1ffff10084833e60
ffff88042419f300: 0000000041b58ab3 ffffffff83a0dbdc ffffffff815397d0 ffffffff832ad2e7
ffff88042419f320: ffffffff814e3bf8 ffff88042419f4c0 ffff8804298a91fa ffff88042419f4a0
ffff88042419f340: ffff88042419f6a8 0000000000314e86 0000000000000020 0000000000000028
ffff88042419f360: ffffffff81621106 0000000000000000 dffffc0000000000 ffff88042419f388
ffff88042419f380: ffffffff81621428 ffff88042419f470 ffffffff8153ccb5 ffff880709a1d028
ffff88042419f3a0: ffffffff81510c31 0000000000000008 ffffffff81621106 0000000000000001
ffff88042419f3c0: dffffc0000000000 ffff88042419f3d8 ffffffff81621428 ffff88042419f4c0
ffff88042419f3e0: ffffffff8153ccb5 ffff88043fffd010 0000000000000040 ffff88042f9a48e8
ffff88042419f400: ffff88043fffcb80 ffff88042419f4f8 ffffea00102b0e40 0000000000000246
ffff88042419f420: 024200ca00000001 ffff88042419f558 ffffed0084833ea1 0000000100000000
ffff88042419f440: 0000000000000040 0000000000000000 ffff88083fff9b80 ffff88042419f548
ffff88042419f460: ffffea001cbcdd40 0000000000000246 024200ca00000001 0000000000000000
ffff88042419f480: ffffed0084833eab 0000000100000000 ffff88083fffa700 0000000000000000
ffff88042419f4a0: 00000000024200ca ffff88042419f538 00000000024200ca ffff88042419f598
ffff88042419f4c0: ffff88042419f5c0 ffffffff815411d9 ffffffff83a0dc0c 1ffff10084833ea1
ffff88042419f4e0: 0000000000000000 ffff88040672f2f8 ffff88040672eac0 00000001024000c0
ffff88042419f500: 0000002000000000 0000000041b58ab3 ffffffff83a132e0 ffffffff00000000
ffff88042419f520: 1ffff10084833ea5 0000000041b58ab3 ffffffff83a0dbdc ffffffff815397d0
ffff88042419f540: 0000000000000000 ffff88083fffa700 ffff88042419f640 ffff88042419f638
ffff88042419f560: ffff88042419f650 ffff88042419f648 ffff88042419f6b0 0000000000000001
ffff88042419f580: ffff88042419f6b0 ffff88042419f688 ffff88042419f5e8 ffffffff811c75da
ffff88042419f5a0: ffff88042419f640 ffff88072c596600 dffffc0000000000 ffffffff00000000
ffff88042419f5c0: ffff88042419f820 ffff88042419f6d8 ffff88042419f5e0 ffffffff823d7aa5
ffff88042419f5e0: ffff88042419f700 ffffffff8100954c ffff88042419f6e0 ffff880400000005
ffff88042419f600: ffff880400000002 dffffc0000000000 1ffff10084833ec3 0000000041b58ab3
ffff88042419f620: ffffffff839f60d6 ffffffff81009250 ffffffff8101a831 0000000200000005
ffff88042419f640: 0000000000000002 ffff88042f853b60 0000000000000005 0000000000000021
ffff88042419f660: 0000000000000001 0000000200000001 0000000000000000 0000000000000000
ffff88042419f680: 0000000000000000 ...
ffff88042419f6a0: 0000000000000000 0000000000000000 ffff88042419f700 ffffffff8101c610
ffff88042419f6c0: ffff880400000001 ffff880700000001 ffff880700000040 0000000000000000
ffff88042419f6e0: ffff88042419f700 ffffffff810191fa dffffc0000000000 ffff88042419f828
ffff88042419f700: ffff88042419f7e8 ffffffff8100cac2 ffff88072594aa80 ffff880419217c00
ffff88042419f720: 1ffffffff082adc1 0000000000029ffe 1ffff10084833eec ffff88042f853b60
ffff88042419f740: ffff880700000002 ffff880400000000 ffff88042419f820 ffff88042f853440
ffff88042419f760: 0000000041b58ab3 ffffffff839f6109 ffffffff8100c720 ffff88042f853440
ffff88042419f780: 0000000200000000 ffffffff81009e18 0000000100000000 0000000041b58ab3
ffff88042419f7a0: ffffffff839f60e8 ffffffff81009c50 0000000000000008 ffff88042f853660
ffff88042419f7c0: ffff88042419f7e8 ffffffff816213e5 ffff88042f853440 ffff88042f853440
ffff88042419f7e0: 1ffff10084833f00 ffff88042419f968 ffffffff81009b68 ffffffff83c167c0
ffff88042419f800: 0000000041b58ab3 ffffffff839f60e8 ffffffff810099d0 ffff88042419f9b0
ffff88042419f820: 0000000000000021 ffffed0084833f21 000000007fffffff ffff8804a419f9af
ffff88042419f840: ffffffff81621106 0000000000000000 dffffc0000000000 ffff88042419f868
ffff88042419f860: ffffffff81621428 ffff88042419f950 ffffffff8153ccb5 1ffff10084833f1c
ffff88042419f880: 0000000000000001 0000000000000000 ffff88042419fd60 ffff88042419f9c0
ffff88042419f8a0: ffff88042c3f8168 ffff88042c3f8000 0000000000000000 ffff88042419f918
ffff88042419f8c0: ffffffff814ffc80 0000000000000021 ffff88042419f918 ffff88072594ab08
ffff88042419f8e0: ffff88072594aa80 0000000000000000 ffff88042419f950 ffffffff814ffc80
ffff88042419f900: ffff88042419f968 0000000000000082 0000000000000000 ffff88042419f978
ffff88042419f920: ffff88072594aa80 0000000000000000 ffff88042419f990 ffffffff814ffc80
ffff88042419f940: ffff88072594abd8 0000000000000001 ffff88042419f990 ffffffff8101e082
ffff88042419f960: dffffc0000000000 ffff88042f853648 0000000000000000 ffff88072594ac40
ffff88042419f980: ffff88042f853440 0000000000000000 dffffc0000000000 ffff88042419f9b8
ffff88042419f9a0: ffffffff8101d12c ffff88072594aa80 ffff88042f853440 ffff88042419f9c8
ffff88042419f9c0: ffffffff8101d2a0 ffff88042419fa60 ffffffff8100de4e ffff88042419fa50
ffff88042419f9e0: ffffffff814f793c 00000001812c58ec ffff88042f864f08 ffff880419217c00
ffff88042419fa00: ffffed0085f0c9fe ffff88042f853654 ffff88042f853658 ffffed0085f0a6ca
ffff88042419fa20: ffff88042f853650 0000000000000002 ffff88040672ea07 ffffffff83c167c0
ffff88042419fa40: ffffffff83c167c0 ffff880419217c00 ffffffff83c167c0 ffff88072594aa80
ffff88042419fa60: ffff88042419fa78 ffffffff814eaff9 ffff88042f864f00 ffff88042419faa8
ffff88042419fa80: ffffffff814f8217 ffff880419217c00 ffff880419217c00 ffff88042f864f08
ffff88042419faa0: dffffc0000000000 ffff88042419fb00 ffffffff814f92a4 ffff88070972d458
ffff88042419fac0: ffff88042419fe40 ffff88072594aaa0 ffffffff815af787 ffff88042f864f08
ffff88042419fae0: ffff880419217c00 ffff880419217c00 ffff88072594aa80 ffffffff814f8b90
ffff88042419fb00: ffff88042419fb48 ffffffff814e1b2e ffff88042419fd70 ffff88042f864f10
ffff88042419fb20: ffff88042419fca0 ffffffff814e1950 ffff88042419fcb8 0000000000000001
ffff88042419fb40: ffffffff814e6a70 ffff88042419fb78 ffffffff814e6b6e ffff88042419fb78
ffff88042419fb60: 0000000000000246 ffff88042419fc18 1ffff10084833f7c ffff88042419fbb8
ffff88042419fb80: ffffffff813b1163 ffff88042419fca0 ffffffff814e6a70 ffff88042419fc40
ffff88042419fba0: 0000000000000007 1ffff10084833f7c 0000000000000001 ffff88042419fc68
ffff88042419fbc0: ffffffff813b1472 ffff88042419fbf8 0000000000000246 ffffffff83a77980
ffff88042419fbe0: 0000000041b58ab3 ffffffff83a05fdf ffffffff813b12c0 ffff88042419fc18
ffff88042419fc00: 0000000000000000 ...
ffff88042419fc20: ffffffff813b10d3 ffffffff815b1920 1ffff10084833f88 ffff88042419fce0
ffff88042419fc40: ffffed0080ce5d5e 0000000000000007 ffff88042419fce0 1ffff10084833f90
ffff88042419fc60: ffff88040672eac0 ffff88042419fd08 ffffffff814e25e1 ffff88040672eaf4
ffff88042419fc80: 0000000041b58ab3 ffffffff839ff26b ffffffff814e2530 0000000000000000
ffff88042419fca0: ffff88040672eac0 ffffffff814e1950 ffff88042419fd70 00000000fffffffd
ffff88042419fcc0: ffff88042419fda0 00007ffc0d8588d0 0000000040082404 ffff88070972d458
ffff88042419fce0: ffff880419217c00 ffffed0083242f8f ffff88040672eac0 dffffc0000000000
ffff88042419fd00: ffff88072594aca8 ffff88042419fe18 ffffffff814e27dc ffff880419217c78
ffff88042419fd20: 1ffff10084833faa 0000000000000000 ffffffff814f8b90 ffff88072594aa80
ffff88042419fd40: ffff880419217c90 ffff88042419fd70 0000000041b58ab3 ffffffff83a0a00b
ffff88042419fd60: ffffffff814e2630 ffffffff83a12428 ffff88072594aa80 ffffffff814f8b90
ffff88042419fd80: 0000000000000000 0000000000000000 0000000041b58ab3 ffffffff83a142a0
ffff88042419fda0: ffff88070c3a8a00 1ffff10084833fc0 0000000000000000 ffffffff814e1950
ffff88042419fdc0: ffff88042419fd70 ffff880400000000 ffffffff81694cc2 ffff88042419fe10
ffff88042419fde0: ffff88041ed2b600 ffff8807282b0020 ffff88072594aa80 ffff880419217c08
ffff88042419fe00: ffffffff814e29b0 dffffc0000000000 ffff88072594aca8 ffff88042419fe38
ffff88042419fe20: ffffffff814e2a3f ffff880419217c00 ffff88072594aa80 ffff88042419fe70
ffff88042419fe40: ffffffff814e1d53 ffff88072594aa80 ffff880419217c00 ffff88040672f378
ffff88042419fe60: dffffc0000000000 00007f382af1c0a4 ffff88042419fea0 ffffffff814ff76b
ffff88042419fe80: 0000000000000001 1ffff10084833fd8 00007ffc0d8588cc 0000000000000020
ffff88042419fea0: ffff88042419ff48 ffffffff8126fb41 00007f382af1c0a4 ffff88040672eac0
ffff88042419fec0: 0000000041b58ab3 ffffffff83a00b7c ffffffff8126f3e0 ffff88040672eac0
ffff88042419fee0: 0000000000000000 0000000000000008 ffff88070972d458 ffffffff8100201a
ffff88042419ff00: ffff88040672eac0 00007f382af1c0a4 00007f382af1c120 00007f382af1c0a4
ffff88042419ff20: 0000000000000000 0000000000401810 00007ffc0d85ac80 0000000000000000
ffff88042419ff40: 0000000000000000 00007ffc0d858900 ffffffff832abc2a 0000000000000000
ffff88042419ff60: 0000000000000000 00007ffc0d85ac80 0000000000401810 00007ffc0d858900
ffff88042419ff80: 0000000000000000 0000000000000202 00007f382af1c0a4 00007f382af1c120
ffff88042419ffa0: 00007f382af1c0a4 ffffffffffffffda 00007f382ac68a4a 0000000000000001
ffff88042419ffc0: 00007ffc0d8588cc 0000000000000020 000000000000009d 00007f382ac68a4a
ffff88042419ffe0: 0000000000000033 0000000000000202 00007ffc0d8588e8 000000000000002b
3Memory state around the buggy address:
3 ffff88042f84b980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042f84ba00: 00 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2
3>ffff88042f84ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
3 ^
3 ffff88042f84bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3 ffff88042f84bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
3==================================================================

2016-11-17 09:48:58

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 10:30 AM, Peter Zijlstra <[email protected]> wrote:
>> > > Peter or Vince, can you try to recreate with this patch? It dumps the
>> > > raw stack contents during a stack dump. Hopefully that would give a
>> > > clue about what's going wrong.
>> >
>> >
>> > Here goes... I'll do another run and get you the results of that as
>> > well.
>>
>> This one is funny, I've not seen that WARNING before. Let me do a third
>> run.
>>
>
> 3==================================================================
> 3BUG: KASAN: stack-out-of-bounds in unwind_get_return_address+0x1fb/0x220 at addr ffff88042f84baa0
> 3Read of size 8 by task perf_fuzzer/8638
> 0page:ffffea0010be12c0 count:1 mapcount:0 mapping: (null) index:0x0c
> 0flags: 0x2ffff8000000400(reserved)
> 1page dumped because: kasan: bad access detected
> dCPU: 1 PID: 8638 Comm: perf_fuzzer Not tainted 4.9.0-rc5-00530-gd8866fc-dirty #4
> dHardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
> dCall Trace:
> d <NMI>
> d dump_stack+0x67/0x94
> d kasan_report_error+0x4a1/0x4d0
> d __asan_report_load8_noabort+0x43/0x50
> d ? unwind_get_return_address+0x1fb/0x220
> d unwind_get_return_address+0x1fb/0x220
> d perf_callchain_kernel+0x356/0x550
> d ? arch_perf_update_userpage+0x350/0x350
> d ? insn_get_opcode.part.4+0x3ac/0x910
> d get_perf_callchain+0x276/0x670
> d ? early_serial_putc+0x24/0x70
> d ? put_callchain_buffers+0x50/0x50
> d ? early_serial_putc+0x1d/0x70
> d ? insn_get_displacement.part.7+0x4a/0x500
> d perf_callchain+0x128/0x1a0
> d ? early_serial_putc+0x1d/0x70
> d perf_prepare_sample+0x70e/0xfb0
> d ? intel_pmu_pebs_fixup_ip+0x2c9/0x4c0
> d perf_event_output_forward+0x93/0x110
> d ? perf_prepare_sample+0xfb0/0xfb0
> d ? early_serial_putc+0x1d/0x70
> d ? early_serial_putc+0x24/0x70
> d ? early_serial_putc+0x24/0x70
> d ? sched_clock_cpu+0x11c/0x1a0
> d __perf_event_overflow+0x1a3/0x570
> d perf_event_overflow+0x14/0x20
> d __intel_pmu_pebs_event+0x3ca/0x610
> d ? pebs_update_state+0x310/0x310
> d ? early_serial_putc+0x1d/0x70
> d ? intel_pmu_disable_bts+0xc0/0xc0
> d ? early_serial_putc+0x1d/0x70
> d ? insn_get_prefixes.part.2+0x36d/0xc70
> d ? insn_get_opcode.part.4+0x3ac/0x910
> d ? number+0x71c/0xa70
> d ? insn_get_opcode+0x42/0x50
> d ? branch_type+0x122/0x3a0
> d ? put_dec+0xb0/0xb0
> d ? knc_pmu_handle_irq+0x3a0/0x3a0
> d intel_pmu_drain_pebs_nhm+0x5f6/0xbf0
> d ? __intel_pmu_pebs_event+0x610/0x610
> d ? early_serial_write+0x7c/0xf0
> d intel_pmu_handle_irq+0x4b2/0xa90
> d ? intel_pmu_save_and_restart+0xe0/0xe0
> d ? acpi_os_read_memory+0x228/0x262
> d ? acpi_os_get_timer+0x1a/0x1a
> d ? vunmap_page_range+0x269/0x400
> d ? ghes_copy_tofrom_phys+0x149/0x270
> d ? ghes_read_estatus+0x11e/0x6b0
> d ? ghes_copy_tofrom_phys+0x270/0x270
> d perf_event_nmi_handler+0x2d/0x50
> d nmi_handle+0x9e/0x250
> d default_do_nmi+0x111/0x180
> d do_nmi+0x1a2/0x210
> d end_repeat_nmi+0x1a/0x1e
> dRIP: 0010:__intel_pmu_enable_all+0x11d/0x190
> dRSP: 0018:ffff88042419f9b0 EFLAGS: 00000046c
> dRAX: 0000000000000000 RBX: ffff88072594aa80 RCX: 000000000000038f
> dRDX: 1ffff10085f0a6c8 RSI: 0000000000000000 RDI: ffff88042f853640
> dRBP: ffff88042419f9b8 R08: 00000000000004c1 R09: ffffed0085f0a6ca
> dR10: 00000000ffffffff R11: 1ffff10085f0a6ec R12: 000000070000000f
> dR13: dffffc0000000000 R14: 0000000000000000 R15: 0000000000000002
> d ? __intel_pmu_enable_all+0x11d/0x190
> d ? __intel_pmu_enable_all+0x11d/0x190
> d <EOE>
> ffff88042f84a000: 0000000000000000 ...
> ffff88042f84a560: ffff8804af84a955 ffff88042f84a95d 0000000000000060 1ffff10085f094b8
> ffff88042f84a580: 00000000ffffffff 0020000000000000 0000000000000000 00000000fffffffd
> ffff88042f84a5a0: 0000000000000001 0000000000000190 0000000000000010 ffff1060ffffff09
> ffff88042f84a5c0: 0000000041b58ab3 ffff8804af84a9bd ffff88042f84a9c5 0000000000000060
> ffff88042f84a5e0: 1ffff10085f094c5 00000000ffffffff 0020000000000000 0000000000000000
> ffff88042f84a600: 00000000fffffffd 0000000000000001 0000000000000190 ffffffff8348be10
> ffff88042f84a620: ffff1060ffffff09 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960
> ffff88042f84a640: ffff88042f84a700 ffff88042f313930 ffffffff823b7c54 ffff88042f84a758
> ffff88042f84a660: ffff88042f84a760 ffff88042f84a758 ffff88042f84a956 0000000000000001
> ffff88042f84a680: ffffffff8348bec9 ffff88042f84a9c5 0000000000000018 dffffc0000000000
> ffff88042f84a6a0: ffff88042f84a818 ffff88042f84a768 ffff88042f84a790 ffffffff823b7c54
> ffff88042f84a6c0: ffff88042f84a7c0 ffff88042f84a7c8 ffff88042f84a7c0 ffff88042f84a9be
> ffff88042f84a6e0: 1ffff10085f094e1 ffffed0085f094f9 000000007fffffff ffff8804af84a9bd
> ffff88042f84a700: ffffffff8348beca 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0
> ffff88042f84a720: 0000000000000000 ffff1060ffffff09 ffffffff823b8d95 0000000041b58ab3
> ffff88042f84a740: ffffffff839ff69c ffffffff823b8d00 0000000000000000 0000000000000020
> ffff88042f84a760: ffff88042f84a7e0 ffffed0085f094f4 1ffff10085f0950a ffff88042f84a930
> ffff88042f84a780: ffffffff8101d21d ffffffffffffffff ffff88042f84a838 ffffffff823b8d95
> ffff88042f84a7a0: 0000000041b58ab3 ffffffff839ff69c ffffffff823b8d00 ffffffff83892478
> ffff88042f84a7c0: ffffffff00000020 ffff88042f84a848 ffff88042f84a800 ffff88042f84a870
> ffff88042f84a7e0: ffff88042f84a9a8 dffffc0000000000 ffff88042f84a838 ffffffff813c861f
> ffff88042f84a800: ffffffff8348beca 0000000000000000 000000000000011d 0000000000000190
> ffff88042f84a820: ffffffff83892478 ffffffff83892478 ffff88042f84a9a8 ffff88042f84a958
> ffff88042f84a840: ffffffff813c883f 0000000000000016 0000000041b58ab3 ffffffff83a0c758
> ffff88042f84a860: ffffffff813c86d0 ffffed0085f09514 0000000000000000 ffff88042f84aa30
> ffff88042f84a880: ffff0a00ffffff05 ffff88042f84a962 ffff88042f84ac9e ffff88042f84a963
> ffff88042f84a8a0: 0000000000000000 ffff88042f84a900 000000000000011d 0000000000000000
> ffff88042f84a8c0: 0000000000000020 1ffff10085f09524 ffff0a00ffffff05 1ffff10085f09524
> ffff88042f84a8e0: ffff88042f84a940 ffff0a00ffffff05 ffff88042f84a9ca ffff88042f84acff
> ffff88042f84a900: ffff88042f84a9cb 0000000000000000 ffff88042f84a968 ffffffff823adbcd
> ffff88042f84a920: 0000000041b58ab3 ffffffff83a40559 1ffff10085f09531 ffff0a00ffffff05
> ffff88042f84a940: 1ffff10085f09531 ffff88042f84a9a8 ffff88042f84acdd ffff88042f84aed8
> ffff88042f84a960: ffff0a00ffffff05 ffff88042f84aaf0 ffffffff823af620 0000000000000000
> ffff88042f84a980: 0000000000000000 0000000041b58ab3 ffffffff83a40559 ffffffff823af530
> ffff88042f84a9a0: 0000000000000000 5f6c65746e695f5f 62616e655f756d70 302b6c6c615f656c
> ffff88042f84a9c0: 3178302f64313178 0030300030003039 0000000000000000 0000000000000000
> ffff88042f84a9e0: 0000000000000000 ffff0a00ffffff05 ffff88042f84aacd ffff88042f84ae22
> ffff88042f84aa00: ffff88042f84aace ffff88042f84aed8 ffff88042f84acd8 ffffffff823adb30
> ffff88042f84aa20: 1ffff10085f0954d 00000000ffffffff 0020f10085f09551 ffff0a00ffffff00
> ffff88042f84aa40: 1ffff10000000010 ffff88042f84aa00 ffff88042f84aa20 ffff88042f84af10
> ffff88042f84aa60: ffff103000001005 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960
> ffff88042f84aa80: ffffffffffffffff 3266383461613630 6666666638383034 ffffffff00000010
> ffff88042f84aaa0: 0000000000000000 0000000000000002 ffff88042f84af10 ffff103000001009
> ffff88042f84aac0: 0000000041b58ab3 ffff88042f84aed8 ffff88042f84ad1d ffff88042f84ac30
> ffff88042f84aae0: 1ffff10085f09565 ffffffffffffffff 002088042f84abb0 ffffffff823b6600
> ffff88042f84ab00: ffffffff00000010 ffff88042f84ad00 ffffffff823b6600 ffff88042f84ac10
> ffff88042f84ab20: ffff103000001009 0000000041b58ab3 ffffffff83a40503 ffffffff823ab960
> ffff88042f84ab40: ffffffff839f498b 3832336162393630 6666666666666666 ffff88042f84ac2f
> ffff88042f84ab60: ffff88042f84abb0 ffffffff823ad785 ffff88042f84ac70 0000000000000001
> ffff88042f84ab80: ffff88042f84ad2d ffff88042f84abb0 ffffffff816213e5 ffff88042f84ad2d
> ffff88042f84aba0: ffffffff839f498f dffffc0000000000 ffff88042f84ac90 ffffffff823b72eb
> ffff88042f84abc0: ffff88042f84afd0 ffff88042f84afd8 ffffffff839f4990 ffff88042f84acd8
> ffff88042f84abe0: 1ffff10085f09581 ffffed0085f095fb 0000000000000200 ffff88042f84aed8
> ffff88042f84ac00: 0000000000000001 0000000041b58ab3 ffffffff83a405aa ffffffff823b6ef0
> ffff88042f84ac20: ffff88042f84ac70 ffff103000001000 1ffff10085f0958a ffffffff83c88b40
> ffff88042f84ac40: ffff88042f84aeb8 ffffffff81338af4 0000000041b58ab3 ffff88042f84ac78
> ffff88042f84ac60: ffffffff811c0a51 ffff88042f84ad2e dffffc0000000000 ffff88042f84aca8
> ffff88042f84ac80: ffffffff811c0afc ffff88042f84acd8 00000000ffffffff 1ffff10085f09597
> ffff88042f84aca0: ffffffff83c88b40 ffff88042f84af20 ffffffff81338af4 0000000041b58ab3
> ffff88042f84acc0: ffffffff83a03c41 ffffffff81338a00 ffff0a00ffffff00 3430383866666666
> ffff88042f84ace0: 3063636134386632 666666666666203a 6333306133386666 6666666666203134
> ffff88042f84ad00: 3236363636363636 3333333620613330 3833333331363033 3636362036363636
> ffff88042f84ad20: 3336333032363336 1f000a3633363336 ffffffff83c88b40 ffff88042f84afb0
> ffff88042f84ad40: ffffffff81338af4 0000000041b58ab3 ffffffff83a03c41 ffffffff81338a00
> ffff88042f84ad60: 0000000041b58ab3 6177647261486401 3a656d616e206572 43206c65746e4920
> ffff88042f84ad80: 697461726f70726f 3030363253206e6f 30303632532f5a47 534f4942202c5a47
> ffff88042f84ada0: 3030364335455320 2e32302e4236382e 2e323030302e3230 3331303233323231
> ffff88042f84adc0: 2f32312030313231 0a333130322f3332 3d3d3d3d3d3d3d00 6f20646165523301
> ffff88042f84ade0: 3820657a69732066 6b73617420796220 75665f6672657020 3336382f72657a7a
> ffff88042f84ae00: 5f7465675f000a38 615f6e7275746572 302b737365726464 3278302f62663178
> ffff88042f84ae20: 6461207461203032 3866666666207264 6234386632343038 000000000a306161
> ffff88042f84ae40: 0000000000000000 ...
> ffff88042f84ae80: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
> ffff88042f84aea0: ffff88042f84bef8 ffffffff8342d740 ffff88042f84af68 ffff88042f84af38
> ffff88042f84aec0: ffffffff8133c807 0000000000000000 0000000000000000 0000000000000000
> ffff88042f84aee0: 0000000000000000 0000000000000000 0000000000000000 00000000ffffffff
> ffff88042f84af00: 0000000000000018 ffff88042f84bef8 ffffffff839f4970 ffff88042f84afd0
> ffff88042f84af20: ffff88042f84afa0 ffffffff8133c807 0000000000000000 ffff88042f84afe8
> ffff88042f84af40: ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c ffffffff8151fdf3
> ffff88042f84af60: 0000000000000000 0000000000000000 000000002f84aff8 1ffff10085f095f6
> ffff88042f84af80: 0000000000000018 ffff88042f84bef8 ffff88040672eac0 ffff88042f84c000
> ffff88042f84afa0: ffff88042f84b050 ffffffff8151fea9 0000000041b58ab3 ffffffff839ff69c
> ffff88042f84afc0: ffffffff8151fdf3 ffff88042f84b0b8 ffff880400000030 ffff88042f84b060
> ffff88042f84afe0: ffff88042f84b010 ffff88042f84bef8 ffff88040672eac0 ffff88042f84bff8
> ffff88042f84b000: ffff88042f84b028 ffffffff813b9d58 0000000000000000 ffff88042f84afe0
> ffff88042f84b020: ffff88042f84b028 ffffffff813b9d58 0000000000000000 ffff88042f84afe0
> ffff88042f84b040: ffff88042f84b020 0000000000000009 ffff88042f84b120 ffffffff811504e6
> ffff88042f84b060: ffffffff8101d21d ffffffff839fbd23 000000002f84b0a0 0000000000000020
> ffff88042f84b080: ffffffff839fbd5b ffffffff839fbd1f 0000000000000005 ffff88042f84a000
> ffff88042f84b0a0: ffff88042f84c000 ffff88042419f9b0 0000000000000001 ffff880424198000
> ffff88042f84b0c0: ffff8804241a0000 0000000000000000 0000000000000022 ffff88040672eac0
> ffff88042f84b0e0: 0000000000000000 ffff88042419f9b8 0000000000000000 0000000000000086
> ffff88042f84b100: 0000000000000000 ffff88042f84b3c8 ffffed0085f0967a dffffc0000000000
> ffff88042f84b120: ffff88042f84b130 ffffffff811505c4 ffff88042f84b150 ffffffff8238dfb3
> ffff88042f84b140: ffff88042f84b1e0 ffff88042f84baa0 ffff88042f84b1d0 ffffffff816221c1
> ffff88042f84b160: 0000000000000000 0000000000000000 ffff88042f84b1a0 0000000000000097
> ffff88042f84b180: ffff88042f84b2b8 ffff88042419f9b8 ffffed0085f09658 0000000000000010
> ffff88042f84b1a0: ffff88042f84b1f0 ffff88042f84b2b8 ffff88042f84baa0 ffff88042f84b3c8
> ffff88042f84b1c0: ffffed0085f0967a dffffc0000000000 ffff88042f84b208 ffffffff81622323
> ffff88042f84b1e0: ffff88042f84baa0 ffff88042f84baa0 0000000000000008 ffff88042f84b200
> ffff88042f84b200: ffffffff811c72cb ffff88042f84b248 ffffffff811c72cb ffff88042f84b2c8
> ffff88042f84b220: ffff88042f84b2b8 ffff88042f84b2b8 ffff88042f84b338 ffff88042f84b3c8
> ffff88042f84b240: ffffed0085f0967a ffff88042f84b360 ffffffff8100f356 0000000000000000
> ffff88042f84b260: 0000000000000000 1ffff10085f09650 ffff88042f84b3d4 1ffff10085f09653
> ffff88042f84b280: ffff88042f84b3da 0000000000000004 ffff88042f84b3d0 0000000041b58ab3
> ffff88042f84b2a0: ffffffff839f613c ffffffff8100f000 ffffed0085f096a6 0000000000000005
> ffff88042f84b2c0: ffff88042f84a000 ffff88042f84c000 ffff88042419f9b0 0000000000000020
> ffff88042f84b2e0: ffff88040672eac0 0000000000000000 ffff88042f84ba98 0000000000000000
> ffff88042f84b300: ffff88042f84b380 ffffffff823bbf3c 0000000000000000 0000000000000000
> ffff88042f84b320: 0000000000000000 ffff88042f84b548 ffff88042f84b530 000000000001bdc0
> ffff88042f84b340: ffff88042f84b408 ffff8804222eadc0 1ffff10085f09675 ffff8804222eadc0
> ffff88042f84b360: ffff88042f84b430 ffffffff81513f16 0000000000000007 ffffffff811c0a34
> ffff88042f84b380: ffff880400000000 ffff88042f84b6e0 0000000000000000 ffff880400000001
> ffff88042f84b3a0: 000000000000000c 0000000041b58ab3 ffffffff83a0a369 ffffffff81513ca0
> ffff88042f84b3c0: ffff88042f84b3e8 ffff8804222eadc0 000000010000007f ffff88042f000001
> ffff88042f84b3e0: ffffffff811c0a2d ffff88042f84b410 ffffffff823bccda 00000000000021be
> ffff88042f84b400: 0000000000000000 ffff88072594aa80 0000000000000040 ffff88042f84b6e0
> ffff88042f84b420: ffff880419217c00 0000000000000000 ffff88042f84b478 ffffffff81514438
> ffff88042f84b440: ffffffff00000001 ffffffff811c0a2d ffff88042f84b51c ffff88072594aa80
> ffff88042f84b460: ffff88042f84b518 ffff88072594aa80 ffff88042f84b7c0 ffff88042f84b4e8
> ffff88042f84b480: ffffffff8150ac7e ffff88042f84b5b0 ffffffff81026e69 ffff88042f84b6e0
> ffff88042f84b4a0: 1ffff10085f09699 00000000000000a3 ffff88042f84b51e ffff88042f84b6e0
> ffff88042f84b4c0: 1ffff10085f0969f ffff88072594aa80 ffff88042f84b7c0 ffff88042f84b6e0
> ffff88042f84b4e0: ffff88042f84b6e0 ffff88042f84b5d8 ffffffff8150b5b3 0000000041b58ab3
> ffff88042f84b500: ffffffff83a0d7b0 ffffffff8150b520 0000000100000000 0020400100000009
> ffff88042f84b520: 0000000100000000 0000000000000000 0107080800004000 ffffffff811c0a2d
> ffff88042f84b540: ffffffff811c0a34 ffffffff811c0a34 0000000000000000 0000000000000000
> ffff88042f84b560: 0000000000000000 ffff88072594aa80 ffff88072594aa80 ffff88042f84b7c0
> ffff88042f84b580: 000000000000464d ffff88042f84b5d8 ffffffff812c58ec ffff88042f84b7c0
> ffff88042f84b5a0: ffff88072594aa80 00000000000000a3 ffff88072594aa80 ffff88072594aa80
> ffff88042f84b5c0: ffff88042f84b7c0 ffff88072594ad74 0000000000000000 ffff88042f84b620
> ffff88042f84b5e0: ffffffff814e7c13 ffff88042f84b7d0 ffff880400000000 ffff88072594aa80
> ffff88042f84b600: dffffc0000000000 ffff88042c7e00b0 ffff88042f84b6e0 ffff88042f84b7c0
> ffff88042f84b620: ffff88042f84b630 ffffffff8150cb54 ffff88042f84b9b0 ffffffff8102901a
> ffff88042f84b640: 0000000000000000 ...
> ffff88042f84b660: 0000000000000000 1ffff10085f096d8 0000000000000000 0000000000000000
> ffff88042f84b680: 0000000000000000 0000000000000000 ffff88042f84b7c0 ffff88042f84bef8
> ffff88042f84b6a0: ffff88072594aa80 0000000000000000 0000000000000000 ffff88042f84b6e0
> ffff88042f84b6c0: 0000000041b58ab3 ffffffff839f20b0 ffffffff81028c50 0000000000000000
> ffff88042f84b6e0: 0000000000000002 0000000000000000 dffffc0000000000 000000070000000f
> ffff88042f84b700: ffff88042f84ba98 ffff88072594aa80 1ffff10085f0a6ec 00000000ffffffff
> ffff88042f84b720: ffffed0085f0a6ca 00000000000004c1 0000000000000000 000000000000038f
> ffff88042f84b740: 1ffff10085f0a6c8 0000000000000000 ffff88042f853640 ffffffffffffffff
> ffff88042f84b760: ffffffff811c0a2d 0000000000000010 000000000000000e ffff88042f84ba88
> ffff88042f84b780: 0000000000000018 ffffffff839f2108 ffffffff8102b200 0000000000000000
> ffff88042f84b7a0: 0000000000000000 ...
> ffff88042f84b7c0: 0000000000000000 0000000000000000 0000000000000000 000000000000090f
> ffff88042f84b7e0: 0000000000000000 0000000000000000 0000000005080021 00000000000000a3
> ffff88042f84b800: ffffffff811c0a2d 000021be000021be 0000000000000000 0000000000000000
> ffff88042f84b820: 0000000000000000 0000000000000001 ffff88042f84b8a8 ffffffff823bb28d
> ffff88042f84b840: 0000000000000000 ...
> ffff88042f84b860: ffff88042f84b9c0 ffff88042f84ba18 ffff88042f84ba20 ffff88042f84b974
> ffff88042f84b880: ffff88042f84b9c4 ffff88042f84b9c0 ffff88042f84ba60 0000000000000002
> ffff88042f84b8a0: ffff88042f84b9c4 ffff88042f84b928 ffffffff823bbf3c ffffffff823ac07c
> ffff88042f84b8c0: ffff88042f84b9c0 ffff88042f84b9c0 ffff88042f84ba20 ffff88042f84ba08
> ffff88042f84b8e0: 0000000000000068 ffff88042f84b910 ffff88042f84b9d0 ffff88042f84ba18
> ffff88042f84b900: ffff88042f84b9c0 ffff88042f84b9c0 ffff88042f84ba60 0000000000000002
> ffff88042f84b920: 0000000000000002 ffff88042f84b940 ffffffff823beb42 1ffff10085f0972c
> ffff88042f84b940: ffff88042f84ba88 ffffffff8102cf02 ffffffff83a40503 ffffffff823ab960
> ffff88042f84b960: 0000000041b58ab3 ffffffff839f711e ffffffff8102cde0 0000000000000000
> ffff88042f84b980: 0000000000000000 0000000000000000 dffffc0000000000 ffff88042f84bb08
> ffff88042f84b9a0: ffff88072594aa80 ffff88042f853440 ffff88042f84bb30 ffffffff81029856
> ffff88042f84b9c0: 0000000100000000 0000000100000000 ffff88042f853d78 ffff88042f84bef8
> ffff88042f84b9e0: 1ffff10085f09745 ffff88042c7e0000 ffff88042f84ba88 0000000000000007
> ffff88042f84ba00: ffffed0085f0a7af ffff88042c7e00b0 fffffbfff082adbc ffff88042f84bac8
> ffff88042f84ba20: ffff88042f84ba88 0000000041b58ab3 ffffffff839f20d0 ffffffff81029260
> ffff88042f84ba40: 0000000000000000 0000000000000001 0000000000000000 0000000000000000
> ffff88042f84ba60: ffff88042f853f00 000000000000000f ffff88042f853440 dffffc0000000000
> ffff88042f84ba80: 000000000000000f 0000000000000001 0000000000000000 ffff88042f84bac8
> ffff88042f84baa0: ffffffff811c0afc fffffbfff082adc4 00000003ffffffff 1ffff10085f0975b
> ffff88042f84bac0: ffffffff83c88b40 0000000000000000 0000000000000000 ffff88042f853d90
> ffff88042f84bae0: ffffffff00000000 ffffffff00029ffe 0000000000000000 ffffed0085f0a7b2
> ffff88042f84bb00: ffff88042f853f10 0000000000000001 000000c76b33c80d ffffffff83c16120
> ffff88042f84bb20: 000000c76b33c7fa ffff88042f853440 ffff88042f84bdf8 ffffffff8101e752
> ffff88042f84bb40: ffff88042f853d78 1ffff10085f09770 ffffed0085f0a7af 0000006400000000
> ffff88042f84bb60: 0000000000000000 ffff88042f84bef8 0000000000000000 ffff88042f84bd60
> ffff88042f84bb80: 0000000041b58ab3 ffffffff839f1dd0 ffffffff8101e2a0 ffff88042f84bc40
> ffff88042f84bba0: ffffffff8256fd3e ffffc900000c5000 1ffff10085f09777 0000000041b58ab3
> ffff88042f84bbc0: 0000000000000000 ffffffff8256fb16 0000000000000000 0000000000000000
> ffff88042f84bbe0: 0000000000000000 ...
> ffff88042f84bc00: 0000000000000000 0000000000000000 0000000000000000 1ffff10085f0978a
> ffff88042f84bc20: ffff88042f84bcf0 ffffc900000cff94 ffff88042f84bd60 dffffc0000000000
> ffff88042f84bc40: ffff88072ec02000 ffff88042f84bcb8 ffffffff815e0639 ffff88072ec02000
> ffff88042f84bc60: ffff88042f62d000 ffffc900000e9fff ffffc900000ea000 ffffffff83c0bc98
> ffff88042f84bc80: ffffc900000e9fff ffffc900000ea000 ffffc900000e9000 0000000000000000
> ffff88042f84bca0: ffff88072951f4e0 00000000bdceb03c ffff880423ef2214 ffff88042f84bcc8
> ffff88042f84bcc0: ffffc900000e9000 ffff88042f84bd18 ffffffff82609379 0000000000000014
> ffff88042f84bce0: 0000000000000000 0010000000000001 ffff8807294f4300 1ffff10085f097a8
> ffff88042f84bd00: ffffc900000cff80 ffff88042f84bda0 0000000000000001 ffff88042f84bdc8
> ffff88042f84bd20: ffffffff826095be ffffffff83c16120 000000c76aa445a8 ffff8807294f4308
> ffff88042f84bd40: 0000000041b58ab3 ffffffff83a49886 ffffffff826094a0 ffffffff83a03c64
> ffff88042f84bd60: 00000000bdceb028 0000000000000000 0000000000000020 ffff88042f84bdf8
> ffff88042f84bd80: ffff88042f84bdb0 ffff88042f84bdb0 ffffffff84c270c0 dffffc0000000000
> ffff88042f84bda0: ffff8807294f4300 dffffc0000000000 ffffffff83f6e7c0 0000000000000000
> ffff88042f84bdc0: ffffffff83f6ea20 ffff88042f84be18 ffff88042f84bef8 000000c76b33c80d
> ffff88042f84bde0: ffffffff83c16120 000000c76b33c7fa ffffffff83c16130 ffff88042f84be18
> ffff88042f84be00: ffffffff8100a25d dffffc0000000000 0000000000000001 ffff88042f84be80
> ffff88042f84be20: ffffffff81150f5e 0000000000000000 0000000000000000 ffff88042f84bef8
> ffff88042f84be40: 0000000000000000 ffffffff83c64ce8 00000000000004a1 ffff88042f84bef8
> ffff88042f84be60: 0000000000000001 0000000000000007 ffff88040672f23c ffff88042f84bf90
> ffff88042f84be80: ffff88042f84bea8 ffffffff81151b41 ffff88040672eac0 ffff88042f84bef8
> ffff88042f84bea0: ffff88040672eac0 ffff88042f84bee8 ffffffff81151d52 0000000000000000
> ffff88042f84bec0: 0000000000000001 000000070000000f dffffc0000000000 0000000000000000
> ffff88042f84bee0: 0000000000000002 ffff88042f84bef9 ffffffff832ad08e 0000000000000002
> ffff88042f84bf00: 0000000000000000 dffffc0000000000 000000070000000f ffff88042419f9b8
> ffff88042f84bf20: ffff88072594aa80 1ffff10085f0a6ec 00000000ffffffff ffffed0085f0a6ca
> ffff88042f84bf40: 00000000000004c1 0000000000000000 000000000000038f 1ffff10085f0a6c8
> ffff88042f84bf60: 0000000000000000 ffff88042f853640 ffffffffffffffff ffffffff8101d21d
> ffff88042f84bf80: 0000000000000010 0000000000000046 ffff88042419f9b0 0000000000000018
> ffff88042f84bfa0: ffffffff8101d21d 0000000000000010 0000000000000046 ffff88042419f9b0
> ffff88042f84bfc0: 0000000000000018 0000000000000001 1ffff10085f0a6c8 ffffffff8101d21d
> ffff88042f84bfe0: 0000000000000010 0000000000000046 ffff88042419f9b0 0000000000000018
> d intel_pmu_enable_all+0x10/0x20
> d x86_pmu_enable+0x65e/0xb10
> d ? ctx_sched_in+0xe8c/0x15a0
> d perf_pmu_enable.part.97+0x29/0x40
> d ctx_resched+0x147/0x1a0
> d __perf_event_enable+0x714/0xb60
> d ? wp_page_copy+0x857/0x13f0
> d ? perf_mux_hrtimer_handler+0x670/0x670
> d event_function+0x1de/0x380
> d ? cpu_clock_event_read+0x30/0x30
> d ? perf_cgroup_attach+0xb0/0xb0
> d remote_function+0xfe/0x180
> d generic_exec_single+0x1a3/0x300
> d ? perf_cgroup_attach+0xb0/0xb0
> d smp_call_function_single+0x1b2/0x310
> d ? generic_exec_single+0x300/0x300
> d ? generic_exec_single+0x113/0x300
> d ? vm_normal_page+0x170/0x170
> d task_function_call+0xb1/0x100
> d ? perf_event_addr_filters_exec+0x360/0x360
> d ? cpu_clock_event_read+0x30/0x30
> d event_function_call+0x1ac/0x2e0
> d ? perf_mux_hrtimer_handler+0x670/0x670
> d ? task_function_call+0x100/0x100
> d ? perf_mux_hrtimer_handler+0x670/0x670
> d ? cpu_clock_event_read+0x30/0x30
> d ? do_vfs_ioctl+0x192/0xf00
> d ? _perf_event_disable+0xa0/0xa0
> d _perf_event_enable+0x8f/0xc0
> d perf_event_for_each_child+0x83/0x160
> d perf_event_task_enable+0x8b/0x100
> d SyS_prctl+0x761/0xb20
> d ? SyS_umask+0x60/0x60
> d ? trace_hardirqs_on_thunk+0x1a/0x1c
> d entry_SYSCALL_64_fastpath+0x18/0xa8
> dRIP: 0033:0x7f382ac68a4a
> dRSP: 002b:00007ffc0d8588e8 EFLAGS: 00000202c ORIG_RAX: 000000000000009d
> dRAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f382ac68a4a
> dRDX: 0000000000000001 RSI: 00007ffc0d8588cc RDI: 0000000000000020
> dRBP: 00007ffc0d858900 R08: 00007f382af1c0a4 R09: 00007f382af1c120
> dR10: 00007f382af1c0a4 R11: 0000000000000202 R12: 0000000000401810
> dR13: 00007ffc0d85ac80 R14: 0000000000000000 R15: 0000000000000000
> ffff880424198000: 0000000057ac6e9d 0000000000000000 0000000000000000 0000000000000000
> ffff880424198020: 0000000000000000 ...
> ffff880424198820: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
> ffff880424198840: 0000000000000000 ...
> ffff8804241988a0: 0000000000000000 0000000000000000 ffff880424198000 0000000000000000
> ffff8804241988c0: 0000000000000000 ...
> ffff8804241990e0: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
> ffff880424199100: 0000000000000000 ...
> ffff880424199160: ffff8804241988b0 0000000000000000 0000000000000000 0000000000000000
> ffff880424199180: 0000000000000000 ...
> ffff880424199980: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
> ffff8804241999a0: 0000000000000000 ...
> ffff880424199a00: 0000000000000000 0000000000000000 ffff880424199160 0000000000000000
> ffff880424199a20: 0000000000000000 ...
> ffff88042419a240: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
> ffff88042419a260: 0000000000000000 ...
> ffff88042419a2c0: ffff880424199a10 0000000000000000 0000000000000000 0000000000000000
> ffff88042419a2e0: 0000000000000000 ...
> ffff88042419aae0: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
> ffff88042419ab00: 0000000000000000 ...
> ffff88042419ab60: 0000000000000000 0000000000000000 ffff88042419a2c0 0000000000000000
> ffff88042419ab80: 0000000000000000 ...
> ffff88042419b3a0: c1c000d000000001 c36000d000000001 0000000000000000 0000000000000000
> ffff88042419b3c0: 0000000000000000 ...
> ffff88042419b420: ffff88042419ab70 0000000000000000 0000000000000000 0000000000000000
> ffff88042419b440: 0000000000000000 ...
> ffff88042419bc40: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
> ffff88042419bc60: 0000000000000000 ...
> ffff88042419bcc0: 0000000000000000 0000000000000000 ffff88042419b420 0000000000000000
> ffff88042419bce0: 0000000000000000 ...
> ffff88042419c500: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
> ffff88042419c520: 0000000000000000 ...
> ffff88042419c580: ffff88042419bcd0 0000000000000000 0000000000000000 0000000000000000
> ffff88042419c5a0: 0000000000000000 ...
> ffff88042419cda0: 0000000000000000 0000000000000000 c66000d000000001 c80000d000000001
> ffff88042419cdc0: 0000000000000000 ...
> ffff88042419ce20: 0000000000000000 0000000000000000 ffff88042419c580 0000000000000000
> ffff88042419ce40: 0000000000000000 ...
> ffff88042419d660: c66000d000000001 c80000d000000001 0000000000000000 0000000000000000
> ffff88042419d680: 0000000000000000 ...
> ffff88042419d6e0: ffff88042419ce30 0000000000000000 0000000000000000 0000000000000000
> ffff88042419d700: 0000000000000000 ...
> ffff88042419dec0: 0000000000000000 0000000000000000 0000000000000000 ffff880424190090
> ffff88042419dee0: ffff880423db0098 dffffc0000000000 ffff880424190080 ffffed008483201a
> ffff88042419df00: ffff88042419df60 ffffffff812c8399 c66000d000000001 c80000d000000001
> ffff88042419df20: 0000000000000000 ffff880423db0080 ffff880423db0090 ffff88042fca0de8
> ffff88042419df40: ffff88042fca0eb0 ffff88042fca0e18 ffff88042419dff8 ffffffff812e90f7
> ffff88042419df60: ffff88042419dff8 ffffffff812e2eae 0000000000000000 0000000000000000
> ffff88042419df80: 0000000000000000 0000000000000000 ffff88042fca0de8 ffff880423db0080
> ffff88042419dfa0: ffff880423db0080 ffff880423db0080 ffff88042fca0e28 ffff880424190210
> ffff88042419dfc0: ffff88042fca0eb0 0000000000000000 ffff880423db0000 ffff880423db0080
> ffff88042419dfe0: dffffc0000000000 ffff880424190080 ffff88042fca0d80 ffff88042419e110
> ffff88042419e000: ffffffff812fb6d1 ffff88042fca16b0 ffff88042fca0de8 1ffff10084833c11
> ffff88042419e020: 0000000000000000 0000000000000000 0000000000000000 ffffed0085f942d6
> ffff88042419e040: ffff88042fca0df8 0000000000000000 ffff880424190000 0000000000000000
> ffff88042419e060: 0000000000000000 0000000000000000 0000000000000000 ffff880424190b40
> ffff88042419e080: 1ffff10084833c11 0000000041b58ab3 ffffffff839fba92 ffffffff81144cf0
> ffff88042419e0a0: 0000000000000000 ffff88042419e198 ffff88042419e190 ffff88042419e1f8
> ffff88042419e0c0: 0000000000000001 ffff88042419e1f8 ffff88042419e1d0 ffff88042419e130
> ffff88042419e0e0: ffffffff811c75da ffff88042fca0d80 0000000000000000 ffff88042419e198
> ffff88042419e100: ffff88042419e190 ffff88042419ff48 0000000000000001 ffff88042419ff48
> ffff88042419e120: ffff88042419e1d0 ffff88042419e180 ffffffff811c75da ffff88042419e190
> ffff88042419e140: ffff88042419e180 ffffffff811c71d2 ffff88042419e1b8 ffff88042419e218
> ffff88042419e160: ffff88042419e218 0000000000000000 0000000000000000 ffff880424190000
> ffff88042419e180: ffff88042419e1f8 ffffffff8116dfd3 0000000000000001 ffffffff832ab679
> ffff88042419e1a0: 00000000c019c80a 000000000000002e ffff88042419e208 ffffffff8242feaa
> ffff88042419e1c0: 0000000000000190 ffff88042419e230 000000000009c80a 0000000000000000
> ffff88042419e1e0: ffff88042419e2d0 ffff88042419e2c8 ffff88042419e330 0000000000000001
> ffff88042419e200: ffff88042419e330 ffff88042419e308 ffff88042419e268 ffffffff811c75da
> ffff88042419e220: ffffffff811c75da 0000000000000000 ffff88042419e2d0 ffff88042419e2c8
> ffff88042419e240: ffff88042419ff48 0000000000000001 ffff88042419ff48 ffff88042419e308
> ffff88042419e260: ffff88042419e2b8 ffffffff811c75da ffff88042419e2c8 ffff88042419e2b8
> ffff88042419e280: ffffffff811c71d2 ffff88042419e2f0 ffff88042419e350 ffff88042419e350
> ffff88042419e2a0: 0000000000000000 0000000000000000 ffff88040672eac0 ffff88042419e330
> ffff88042419e2c0: ffffffff8116dfd3 00000000a897a839 0000000000000120 ffff88042419e368
> ffff88042419e2e0: ffff88070a70d7e0 ffff88042419e340 ffffffff8242fc8f 0000000002408240
> ffff88042419e300: ffff88042419e350 000000000007a839 0000000000000000 0000000002408240
> ffff88042419e320: ffff88070c23b950 0000000002408240 ffff88070c23b950 ffff880727f22e80
> ffff88042419e340: ffff88042419e570 ffffffff81621065 0000004000000024 ffff88042419e368
> ffff88042419e360: 0000000000000000 ffffffff8116e03b ffffffff81620ff6 ffffffff8162126d
> ffff88042419e380: ffffffff816217d2 ffffffff8161d1e0 ffffffff81ed2aa1 ffffffff81ed00cd
> ffff88042419e3a0: ffffffff81f13e2b ffffffff81f14af4 ffffffff81da4df2 ffffffff81da51dc
> ffff88042419e3c0: ffffffff81dbe1a2 ffffffff81d425f2 ffffffff81d43145 ffffffff81d4ce2d
> ffff88042419e3e0: ffffffff81d4dfeb ffffffff81d50c7f ffffffff81d8f93d ffffffff81d91111
> ffff88042419e400: ffffffff81d93171 ffffffff81e9fb75 ffffffff81e560ba ffffffff81e5a578
> ffff88042419e420: ffffffff8154c563 ffffffff81e59732 ffffffff8154e6ea ffffffff81527f8a
> ffff88042419e440: ffffffff8152814c ffffffff81eafe8d ffffffff81e7eee5 ffffffff8165fe8a
> ffff88042419e460: ffffffff8166030e ffffffff8128a10a ffffffff81002a09 ffffffff81005265
> ffff88042419e480: ffffffff832abcb8 ffff88041c85c8e0 ffffed008390b91c ffff880422c64d60
> ffff88042419e4a0: ffff88070c23b930 1ffff10084833ca7 ffff88070ba04cb8 ffff88041c85c850
> ffff88042419e4c0: ffff880422c64c40 ffff88042419e518 ffffffff81f15847 ffff88041c85c980
> ffff88042419e4e0: 0000000600000007 ffff880422c64c6c ffff8806fbb5c818 1ffff10084833ca7
> ffff88042419e500: ffff88042419e5d8 ffff880422c64c40 ffff88040b9780b0 ffff88042419e5f8
> ffff88042419e520: ffffffff81dab421 0000000000000286 ffff88041c85c8e0 0000000041b58ab3
> ffff88042419e540: ffffffff83a33fb7 ffffffff81dab2f0 00000000000009af 1ffff10084833caf
> ffff88042419e560: ffff88042419e5f8 ffffffff81d9fb81 ffff88042419e5c0 0000000041b58ab3
> ffff88042419e580: ffffffff83a34296 ffffffff81d9fa90 ffff880422c64c6c ffff8806875c0000
> ffff88042419e5a0: 00000000000009af 0000000000000015 ffff880422c64c40 ffff88040b9780b0
> ffff88042419e5c0: ffff88042419e5f8 ffffffff81dab2ca ffffffff81da10b9 ffff88042419e850
> ffff88042419e5e0: ffff8806fbb5c818 ffff88040b978000 0000000000000015 ffff88042419e878
> ffff88042419e600: ffffffff81dc3893 ffffffff81ed2aa1 ffff88040672ef80 ffff88040672eac0
> ffff88042419e620: 1ffff10080ce5df0 ffff88040b978006 ffff8806fbb5c820 ffff88042419e770
> ffff88042419e640: ffff88042419e9d0 ffff88042419e990 000000000359dc90 ffff88042419ea10
> ffff88042419e660: ffff8806fbb5c828 1ffff10084833cd2 ffff880700000133 ffff88042419e950
> ffff88042419e680: ffff8806fbb5c830 ffff88042419e778 ffff88042419e770 ffff88042419e788
> ffff88042419e6a0: ffff88042419e780 ffff88042419e7e8 0000000000000001 ffff88042419e7e8
> ffff88042419e6c0: ffff88042419e7c0 ffff88042419e720 ffffffff811c75da ffff88042419e778
> ffff88042419e6e0: 0000000000000000 ffff88042419e788 ffff88042419e780 ffff88042419ff48
> ffff88042419e700: 0000000000000001 ffff88042419ff48 ffff88042419e7c0 ffff88042419e770
> ffff88042419e720: ffff88042419e810 ffff88042419e808 ffff88042419e870 0000000000000001
> ffff88042419e740: ffff88042419e870 ffff88042419e848 ffff88042419e7a8 ffffffff811c75da
> ffff88042419e760: 0000000000000000 0000000000000000 ffff88042419e810 ffff88042419e808
> ffff88042419e780: ffff88042419ff48 0000000000000001 ffff88042419ff48 ffff88042419e848
> ffff88042419e7a0: ffff88042419e7f8 ffffffff811c75da ffff88042419e808 ffff88042419e7f8
> ffff88042419e7c0: ffffffff811c71d2 ffff88042419e830 ffff88042419e890 ffff88042419e890
> ffff88042419e7e0: 0000000000000000 0000000000000000 ffff88040672eac0 ffff88042419e870
> ffff88042419e800: ffffffff8116dfd3 00000000931ae748 00000000000000d8 ffff88042419e8a8
> ffff88042419e820: ffff88070be15c80 ffff88042419e880 ffffffff8242fc8f 0000000002000000
> ffff88042419e840: ffff88042419e890 00000000000ae748 0000000000000000 0000000002000000
> ffff88042419e860: ffff8806fbb5c940 ffff8806fbb5ca28 ffffea001beed700 ffff8806fbb5c940
> ffff88042419e880: ffff88042419eab0 ffffffff81621065 000000400000001b ffff88042419e8a8
> ffff88042419e8a0: ffffffff00000000 ffffffff8116e03b ffffffff81620ff6 ffffffff81621851
> ffff88042419e8c0: ffffffff8161d85e ffffffff81da8a5a ffffffff81d4ce60 ffffffff81d4dfeb
> ffff88042419e8e0: ffffffff81d50c7f ffffffff81d8f93d ffffffff81d91111 ffffffff81d93171
> ffff88042419e900: ffffffff81e9fb75 ffffffff81e560ba ffffffff81e5a578 ffffffff8154c563
> ffff88042419e920: ffffffff81e59732 ffffffff8154e6ea ffffffff81527f8a ffffffff8152814c
> ffff88042419e940: ffffffff81eafe8d ffffffff81e7eee5 ffffffff8165fe8a ffffffff8166030e
> ffff88042419e960: ffffffff8128a10a ffffffff81002a09 ffffffff81005265 ffffffff832abcb8
> ffff88042419e980: ffffffffffffffff 0000000000000000 ffff88042419ea50 0000000041b58ab3
> ffff88042419e9a0: ffffffff83a33ff3 ffffffff81dbfe60 ffff880727f22880 ffff880422c648c0
> ffff88042419e9c0: ffffffff8161d85e 0000000000000000 010000007e4d0700 ffff88042419ea78
> ffff88042419e9e0: 0000000000000296 0000000000000000 0000000000000292 0000000000000000
> ffff88042419ea00: 0000000000000000 0000000000000000 ffff880727f22e80 ffff88070c23b870
> ffff88042419ea20: ffff88070c23b890 ffff88042419ea50 ffffffff81621867 ffff88070c23b870
> ffff88042419ea40: ffffffff81ecdff4 ffff880727f22e80 ffff88042419ea80 0000000000000282
> ffff88042419ea60: ffff8804135011b0 ffff8804135011b8 ffff88042419ea98 0000000000000282
> ffff88042419ea80: 0000000000000000 0000000000000000 0000000000000000 ffff880727f22880
> ffff88042419eaa0: ffff8806fbb5c940 ffff8806fbb5ca28 ffff88042419ead8 ffffffff81621867
> ffff88042419eac0: ffff8806fbb5c940 ffffffff81da8a5a ffff880727f22880 ffff88042419eb08
> ffff88042419eae0: ffffffff8161d85e 0000000000000002 ffff8806fbb5c9f5 1ffff100df76b93e
> ffff88042419eb00: dffffc0000000000 ffff88042419eb58 ffffffff81da8a5a ffffffff81d53833
> ffff88042419eb20: 000000000000001c ffffed00df76b928 ffff88042419f1f8 0000000000000001
> ffff88042419eb40: 0000000000000000 ffff88042419efc8 ffff8806fbb5c818 ffff88042419eff0
> ffff88042419eb60: ffffffff81d4ce60 ffff880400000001 ffff88042419ed08 ffffffff00000000
> ffff88042419eb80: ffffffff81d8f93d ffffffff81d91111 ffffffff81d93171 ffffffff81e9fb75
> ffff88042419eba0: ffffffff81e560ba ffffffff81e5a578 000000028154c563 ffff88042419f22c
> ffff88042419ebc0: ffffed0084833e4d ffff88042419f269 1ffff10084833d85 ffff88042419f200
> ffff88042419ebe0: ffff88042419f228 ffff88042419f208 ffff88042419f254 ffff88042419f234
> ffff88042419ec00: ffff88042419f250 ffffffff81005265 ffff88042419ed88 ffff88040000000a
> ffff88042419ec20: ffff880600000002 0000000041b58ab3 ffffffff83a1d770 ffffffff81d4b4f0
> ffff88042419ec40: 1ffff10080ce5df0 ffff880700075765 ffff88070ba04cb8 0000000000000001
> ffff88042419ec60: ffff88070a704980 ffff880413500fc0 ffff88042419ecb0 ffffffff81ed00cd
> ffff88042419ec80: ffff8804135010b8 ffff880400075765 ffff88070ba04cb8 0000000000000001
> ffff88042419eca0: 1ffff10084833dad ffff880413500fc0 ffff88042419ece0 ffffffff81f13e2b
> ffff88042419ecc0: ffff88070ba04cb8 ffff880400000003 ffff88070ba04cb8 0000000000000000
> ffff88042419ece0: ffff88042419ed28 ffffffff81f14af4 0000000100000000 ffff880400000000
> ffff88042419ed00: ffff8804191f2200 0000000000000003 0000000000000000 0000000000000000
> ffff88042419ed20: 0000000000000000 ffff880727f23000 ffff88070a704980 ffff88070a704a70
> ffff88042419ed40: ffff88042419ed68 ffffffff81621867 ffff88070a704980 ffffffff81ee9d85
> ffff88042419ed60: ffff880727f23000 ffff88042419ed98 ffffffff8161d85e ffff88070a704980
> ffff88042419ed80: 0000000000000292 ffff880400000001 ffff880413500ff8 ffff88042419edc0
> ffff88042419eda0: ffffffff8131f879 ffff880413501008 ffff88042419edf8 ffffffff81e6e5f1
> ffff88042419edc0: ffff880400000000 ffff880400000000 ffff88070a704a00 ffff88070ba04cb8
> ffff88042419ede0: ffff88070a704980 ffff88042419f030 ffff880413500fc0 ffff88042419ee30
> ffff88042419ee00: ffffffff81f15244 ffff880400074d7d ffff88070ba04cb8 ffff88041c85c2f0
> ffff88042419ee20: ffffed008390b85e ffff8804152619a0 ffff88070c23b7b0 ffff88042419ef88
> ffff88042419ee40: ffff88070ba04cb8 ffff88041c85c260 ffff880415261880 ffff88042419eea8
> ffff88042419ee60: ffffffff81f15847 ffff88042419ef70 0000003400000037 ffff8804152618ac
> ffff88042419ee80: 0000000000000200 ffff88042419ef88 ffff88042419ef80 ffff88042419ef78
> ffff88042419eea0: ffff88042419efe0 0000000000000001 ffff88042419efe0 ffff88042419efb8
> ffff88042419eec0: ffff88042419ef18 ffffffff811c75da 0000000000000001 0000000000000000
> ffff88042419eee0: ffff88042419ef80 ffff88042419ef78 ffff88042419ff48 0000000000000001
> ffff88042419ef00: ffff88042419ff48 ffff88042419efb8 ffff88042419ef68 ffffffff811c75da
> ffff88042419ef20: ffff88042419ef78 ffff88042419ef68 ffffffff811c71d2 ffff88042419efa0
> ffff88042419ef40: ffff88042419f000 ffff88042419f000 0000000000000000 0000000000000000
> ffff88042419ef60: ffff88040672eac0 ffff88042419efe0 ffffffff8116dfd3 00000000b1fe153b
> ffff88042419ef80: 0000000000000120 ffff88042419f018 ffff88070ba2a6b0 ffff88042419eff0
> ffff88042419efa0: ffffffff8242fc8f 0000000002091220 ffff88042419f000 ffff880709a1d018
> ffff88042419efc0: ffff88042419f010 ffffffff81448fa2 ffff8807154d1640 ffff880709a1d028
> ffff88042419efe0: ffffffff81510c31 0000000000000008 0000000000000000 ffff88042419f460
> ffff88042419f000: ffff8804298a9100 ffff8804298a9308 ffff88042419f0c8 ffffffff832ad2e7
> ffff88042419f020: ffffffff81510c31 0000000000000008 0000000000000000 ffff88042419f4a0
> ffff88042419f040: ffff88042419f3f0 ffff8804298a9308 ffff88042419f108 ffffffff832ad2e7
> ffff88042419f060: ffff88040672f358 ffff88042419f7d6 ffff8804298a9100 0000000000000000
> ffff88042419f080: 1ffff10085315275 ffff8804298a9100 ffff8804298a93a8 fffffbfff082b618
> ffff88042419f0a0: ffffffff81359c15 ffff88042419f826 ffff8804298a9100 0000000000000000
> ffff88042419f0c0: 1ffff10085315275 ffff88042419f0d8 ffffffff81359c15 ffff88042419f3f0
> ffff88042419f0e0: 0000000000000008 0000000000000000 ffff88042419f3f0 ffffffff81510c31
> ffff88042419f100: ffffed0084833e43 ffff88042419f118 ffffffff81359c15 ffff88042419f430
> ffff88042419f120: 0000000000000008 0000000000000000 ffff88042419f430 ffffffff81510c31
> ffff88042419f140: 1ffff10084833e3a 000000012419f1a0 ffff88042419f53c ffff88042419f218
> ffff88042419f160: 1ffff10084833e30 ffffffe42419f318 ffff8804298a9100 ffff88042419f460
> ffff88042419f180: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 000000000716dfd3
> ffff88042419f1a0: 1ffff10084833e38 ffffffe42419f548 ffff8804298a9100 ffff88042419f4a0
> ffff88042419f1c0: 0000000041b58ab3 ffffffff83a0d8d8 ffffffff81510890 ffffffff83a032f4
> ffff88042419f1e0: ffffffff812f5fc0 0000000000000001 0000000000000001 000000000000000c
> ffff88042419f200: 000000000000000c 0000000000000000 0000000000002e04 000000000000022c
> ffff88042419f220: 0000001400000000 0000000000000014 0000000000000000 0000000000000000
> ffff88042419f240: ffff88042419f250 ffff880709a1d018 ffff88042419f2a0 ffffffff81448fa2
> ffff88042419f260: 0000000000000050 ffff880709a1d018 ffff88042419f2c0 ffffffff81448fa2
> ffff88042419f280: ffff8800032b6000 ffff880709a1d028 ffffffff811d236d 80000000032b6161
> ffff88042419f2a0: ffff880709a1d018 ffff88042419f2f8 ffffffff81448fa2 ffff880709a1d018
> ffff88042419f2c0: 0000000041b58ab3 ffffffff83a0d680 ffffffff814e3680 dffffc0000000000
> ffff88042419f2e0: 0000000000000012 ffff88042419f538 ffff880400000001 1ffff10084833e60
> ffff88042419f300: 0000000041b58ab3 ffffffff83a0dbdc ffffffff815397d0 ffffffff832ad2e7
> ffff88042419f320: ffffffff814e3bf8 ffff88042419f4c0 ffff8804298a91fa ffff88042419f4a0
> ffff88042419f340: ffff88042419f6a8 0000000000314e86 0000000000000020 0000000000000028
> ffff88042419f360: ffffffff81621106 0000000000000000 dffffc0000000000 ffff88042419f388
> ffff88042419f380: ffffffff81621428 ffff88042419f470 ffffffff8153ccb5 ffff880709a1d028
> ffff88042419f3a0: ffffffff81510c31 0000000000000008 ffffffff81621106 0000000000000001
> ffff88042419f3c0: dffffc0000000000 ffff88042419f3d8 ffffffff81621428 ffff88042419f4c0
> ffff88042419f3e0: ffffffff8153ccb5 ffff88043fffd010 0000000000000040 ffff88042f9a48e8
> ffff88042419f400: ffff88043fffcb80 ffff88042419f4f8 ffffea00102b0e40 0000000000000246
> ffff88042419f420: 024200ca00000001 ffff88042419f558 ffffed0084833ea1 0000000100000000
> ffff88042419f440: 0000000000000040 0000000000000000 ffff88083fff9b80 ffff88042419f548
> ffff88042419f460: ffffea001cbcdd40 0000000000000246 024200ca00000001 0000000000000000
> ffff88042419f480: ffffed0084833eab 0000000100000000 ffff88083fffa700 0000000000000000
> ffff88042419f4a0: 00000000024200ca ffff88042419f538 00000000024200ca ffff88042419f598
> ffff88042419f4c0: ffff88042419f5c0 ffffffff815411d9 ffffffff83a0dc0c 1ffff10084833ea1
> ffff88042419f4e0: 0000000000000000 ffff88040672f2f8 ffff88040672eac0 00000001024000c0
> ffff88042419f500: 0000002000000000 0000000041b58ab3 ffffffff83a132e0 ffffffff00000000
> ffff88042419f520: 1ffff10084833ea5 0000000041b58ab3 ffffffff83a0dbdc ffffffff815397d0
> ffff88042419f540: 0000000000000000 ffff88083fffa700 ffff88042419f640 ffff88042419f638
> ffff88042419f560: ffff88042419f650 ffff88042419f648 ffff88042419f6b0 0000000000000001
> ffff88042419f580: ffff88042419f6b0 ffff88042419f688 ffff88042419f5e8 ffffffff811c75da
> ffff88042419f5a0: ffff88042419f640 ffff88072c596600 dffffc0000000000 ffffffff00000000
> ffff88042419f5c0: ffff88042419f820 ffff88042419f6d8 ffff88042419f5e0 ffffffff823d7aa5
> ffff88042419f5e0: ffff88042419f700 ffffffff8100954c ffff88042419f6e0 ffff880400000005
> ffff88042419f600: ffff880400000002 dffffc0000000000 1ffff10084833ec3 0000000041b58ab3
> ffff88042419f620: ffffffff839f60d6 ffffffff81009250 ffffffff8101a831 0000000200000005
> ffff88042419f640: 0000000000000002 ffff88042f853b60 0000000000000005 0000000000000021
> ffff88042419f660: 0000000000000001 0000000200000001 0000000000000000 0000000000000000
> ffff88042419f680: 0000000000000000 ...
> ffff88042419f6a0: 0000000000000000 0000000000000000 ffff88042419f700 ffffffff8101c610
> ffff88042419f6c0: ffff880400000001 ffff880700000001 ffff880700000040 0000000000000000
> ffff88042419f6e0: ffff88042419f700 ffffffff810191fa dffffc0000000000 ffff88042419f828
> ffff88042419f700: ffff88042419f7e8 ffffffff8100cac2 ffff88072594aa80 ffff880419217c00
> ffff88042419f720: 1ffffffff082adc1 0000000000029ffe 1ffff10084833eec ffff88042f853b60
> ffff88042419f740: ffff880700000002 ffff880400000000 ffff88042419f820 ffff88042f853440
> ffff88042419f760: 0000000041b58ab3 ffffffff839f6109 ffffffff8100c720 ffff88042f853440
> ffff88042419f780: 0000000200000000 ffffffff81009e18 0000000100000000 0000000041b58ab3
> ffff88042419f7a0: ffffffff839f60e8 ffffffff81009c50 0000000000000008 ffff88042f853660
> ffff88042419f7c0: ffff88042419f7e8 ffffffff816213e5 ffff88042f853440 ffff88042f853440
> ffff88042419f7e0: 1ffff10084833f00 ffff88042419f968 ffffffff81009b68 ffffffff83c167c0
> ffff88042419f800: 0000000041b58ab3 ffffffff839f60e8 ffffffff810099d0 ffff88042419f9b0
> ffff88042419f820: 0000000000000021 ffffed0084833f21 000000007fffffff ffff8804a419f9af
> ffff88042419f840: ffffffff81621106 0000000000000000 dffffc0000000000 ffff88042419f868
> ffff88042419f860: ffffffff81621428 ffff88042419f950 ffffffff8153ccb5 1ffff10084833f1c
> ffff88042419f880: 0000000000000001 0000000000000000 ffff88042419fd60 ffff88042419f9c0
> ffff88042419f8a0: ffff88042c3f8168 ffff88042c3f8000 0000000000000000 ffff88042419f918
> ffff88042419f8c0: ffffffff814ffc80 0000000000000021 ffff88042419f918 ffff88072594ab08
> ffff88042419f8e0: ffff88072594aa80 0000000000000000 ffff88042419f950 ffffffff814ffc80
> ffff88042419f900: ffff88042419f968 0000000000000082 0000000000000000 ffff88042419f978
> ffff88042419f920: ffff88072594aa80 0000000000000000 ffff88042419f990 ffffffff814ffc80
> ffff88042419f940: ffff88072594abd8 0000000000000001 ffff88042419f990 ffffffff8101e082
> ffff88042419f960: dffffc0000000000 ffff88042f853648 0000000000000000 ffff88072594ac40
> ffff88042419f980: ffff88042f853440 0000000000000000 dffffc0000000000 ffff88042419f9b8
> ffff88042419f9a0: ffffffff8101d12c ffff88072594aa80 ffff88042f853440 ffff88042419f9c8
> ffff88042419f9c0: ffffffff8101d2a0 ffff88042419fa60 ffffffff8100de4e ffff88042419fa50
> ffff88042419f9e0: ffffffff814f793c 00000001812c58ec ffff88042f864f08 ffff880419217c00
> ffff88042419fa00: ffffed0085f0c9fe ffff88042f853654 ffff88042f853658 ffffed0085f0a6ca
> ffff88042419fa20: ffff88042f853650 0000000000000002 ffff88040672ea07 ffffffff83c167c0
> ffff88042419fa40: ffffffff83c167c0 ffff880419217c00 ffffffff83c167c0 ffff88072594aa80
> ffff88042419fa60: ffff88042419fa78 ffffffff814eaff9 ffff88042f864f00 ffff88042419faa8
> ffff88042419fa80: ffffffff814f8217 ffff880419217c00 ffff880419217c00 ffff88042f864f08
> ffff88042419faa0: dffffc0000000000 ffff88042419fb00 ffffffff814f92a4 ffff88070972d458
> ffff88042419fac0: ffff88042419fe40 ffff88072594aaa0 ffffffff815af787 ffff88042f864f08
> ffff88042419fae0: ffff880419217c00 ffff880419217c00 ffff88072594aa80 ffffffff814f8b90
> ffff88042419fb00: ffff88042419fb48 ffffffff814e1b2e ffff88042419fd70 ffff88042f864f10
> ffff88042419fb20: ffff88042419fca0 ffffffff814e1950 ffff88042419fcb8 0000000000000001
> ffff88042419fb40: ffffffff814e6a70 ffff88042419fb78 ffffffff814e6b6e ffff88042419fb78
> ffff88042419fb60: 0000000000000246 ffff88042419fc18 1ffff10084833f7c ffff88042419fbb8
> ffff88042419fb80: ffffffff813b1163 ffff88042419fca0 ffffffff814e6a70 ffff88042419fc40
> ffff88042419fba0: 0000000000000007 1ffff10084833f7c 0000000000000001 ffff88042419fc68
> ffff88042419fbc0: ffffffff813b1472 ffff88042419fbf8 0000000000000246 ffffffff83a77980
> ffff88042419fbe0: 0000000041b58ab3 ffffffff83a05fdf ffffffff813b12c0 ffff88042419fc18
> ffff88042419fc00: 0000000000000000 ...
> ffff88042419fc20: ffffffff813b10d3 ffffffff815b1920 1ffff10084833f88 ffff88042419fce0
> ffff88042419fc40: ffffed0080ce5d5e 0000000000000007 ffff88042419fce0 1ffff10084833f90
> ffff88042419fc60: ffff88040672eac0 ffff88042419fd08 ffffffff814e25e1 ffff88040672eaf4
> ffff88042419fc80: 0000000041b58ab3 ffffffff839ff26b ffffffff814e2530 0000000000000000
> ffff88042419fca0: ffff88040672eac0 ffffffff814e1950 ffff88042419fd70 00000000fffffffd
> ffff88042419fcc0: ffff88042419fda0 00007ffc0d8588d0 0000000040082404 ffff88070972d458
> ffff88042419fce0: ffff880419217c00 ffffed0083242f8f ffff88040672eac0 dffffc0000000000
> ffff88042419fd00: ffff88072594aca8 ffff88042419fe18 ffffffff814e27dc ffff880419217c78
> ffff88042419fd20: 1ffff10084833faa 0000000000000000 ffffffff814f8b90 ffff88072594aa80
> ffff88042419fd40: ffff880419217c90 ffff88042419fd70 0000000041b58ab3 ffffffff83a0a00b
> ffff88042419fd60: ffffffff814e2630 ffffffff83a12428 ffff88072594aa80 ffffffff814f8b90
> ffff88042419fd80: 0000000000000000 0000000000000000 0000000041b58ab3 ffffffff83a142a0
> ffff88042419fda0: ffff88070c3a8a00 1ffff10084833fc0 0000000000000000 ffffffff814e1950
> ffff88042419fdc0: ffff88042419fd70 ffff880400000000 ffffffff81694cc2 ffff88042419fe10
> ffff88042419fde0: ffff88041ed2b600 ffff8807282b0020 ffff88072594aa80 ffff880419217c08
> ffff88042419fe00: ffffffff814e29b0 dffffc0000000000 ffff88072594aca8 ffff88042419fe38
> ffff88042419fe20: ffffffff814e2a3f ffff880419217c00 ffff88072594aa80 ffff88042419fe70
> ffff88042419fe40: ffffffff814e1d53 ffff88072594aa80 ffff880419217c00 ffff88040672f378
> ffff88042419fe60: dffffc0000000000 00007f382af1c0a4 ffff88042419fea0 ffffffff814ff76b
> ffff88042419fe80: 0000000000000001 1ffff10084833fd8 00007ffc0d8588cc 0000000000000020
> ffff88042419fea0: ffff88042419ff48 ffffffff8126fb41 00007f382af1c0a4 ffff88040672eac0
> ffff88042419fec0: 0000000041b58ab3 ffffffff83a00b7c ffffffff8126f3e0 ffff88040672eac0
> ffff88042419fee0: 0000000000000000 0000000000000008 ffff88070972d458 ffffffff8100201a
> ffff88042419ff00: ffff88040672eac0 00007f382af1c0a4 00007f382af1c120 00007f382af1c0a4
> ffff88042419ff20: 0000000000000000 0000000000401810 00007ffc0d85ac80 0000000000000000
> ffff88042419ff40: 0000000000000000 00007ffc0d858900 ffffffff832abc2a 0000000000000000
> ffff88042419ff60: 0000000000000000 00007ffc0d85ac80 0000000000401810 00007ffc0d858900
> ffff88042419ff80: 0000000000000000 0000000000000202 00007f382af1c0a4 00007f382af1c120
> ffff88042419ffa0: 00007f382af1c0a4 ffffffffffffffda 00007f382ac68a4a 0000000000000001
> ffff88042419ffc0: 00007ffc0d8588cc 0000000000000020 000000000000009d 00007f382ac68a4a
> ffff88042419ffe0: 0000000000000033 0000000000000202 00007ffc0d8588e8 000000000000002b
> 3Memory state around the buggy address:
> 3 ffff88042f84b980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff88042f84ba00: 00 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2
> 3>ffff88042f84ba80: f2 00 00 f4 f4 f2 f2 f2 f2 00 00 f4 f4 f3 f3 f3
> 3 ^
> 3 ffff88042f84bb00: f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 3 ffff88042f84bb80: f1 f1 f1 f1 f1 f1 f1 f1 00 f4 f4 f4 f2 f2 f2 f2
> 3==================================================================

Just in case, there is currently a known KASAN false positive related
to longjmp's on GPFs. When a syscall hits GPF stack is unwound to
kernel entry point, this leaves a bunch of stray poisoned redzones on
the thread stack. They later cause false stack-out-of-bounds reports.

But this does not seem to be the case here. Kernel is not tainted. And
shadow at the bottom of the reports looks sane.

But if that's the case somehow, we will need to add
kasan_unpoison_remaining_stack() call before a longjmp like we did for
jprobe_return():
https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ

2016-11-17 17:05:34

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 09:25:58AM -0500, Vince Weaver wrote:
> On Thu, 17 Nov 2016, Josh Poimboeuf wrote:
>
> > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote:
> > > Just in case, there is currently a known KASAN false positive related
> > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to
> > > kernel entry point, this leaves a bunch of stray poisoned redzones on
> > > the thread stack. They later cause false stack-out-of-bounds reports.
> > >
> > > But this does not seem to be the case here. Kernel is not tainted. And
> > > shadow at the bottom of the reports looks sane.
> > >
> > > But if that's the case somehow, we will need to add
> > > kasan_unpoison_remaining_stack() call before a longjmp like we did for
> > > jprobe_return():
> > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ
> >
> > I'm pretty sure this isn't a KASAN false positive. The unwinder does
> > actually seem to be accessing a bad area of the stack, in the middle of
> > a function's stack frame.
>
> I'm having trouble reproducing it on a few other machines I have fuzzing.
> So there might be some kernel option contributing, I need to compare
> .configs.
>
> Also the machine that easily triggers the problem I'm compiling with
> gcc-5.4 where the machines I can't are using gcc-4.9.

I believe KASAN only works with gcc 5 and later, so that would explain
why you aren't seeing it with gcc 4.9.

--
Josh

2016-11-17 17:13:43

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 09:18:48AM -0600, Josh Poimboeuf wrote:
> On Thu, Nov 17, 2016 at 10:04:46AM +0100, Peter Zijlstra wrote:
> > On Wed, Nov 16, 2016 at 10:48:28PM -0600, Josh Poimboeuf wrote:
> > > Peter or Vince, can you try to recreate with this patch? It dumps the
> > > raw stack contents during a stack dump. Hopefully that would give a
> > > clue about what's going wrong.
> >
> >
> > Here goes... I'll do another run and get you the results of that as
> > well.
>
> Thanks, I just waded through this and it turned up some good clues. And
> according to 'git blame', you might be able to help :-)
>
> It's not stack corruption. Instead it looks like
> __intel_pmu_pebs_event() is creating a bad or stale pt_regs which gets
> passed to the unwinder. Specifically, regs->bp points to a seemingly
> random address on the NMI stack. Which seems odd, considering the code
> itself is running on the same NMI stack.
>
> I don't know much about the PEBS code but it seems like it's passing
> some stale data. Either that or there's some NMI nesting going on.

Ooh, indeed. The PEBS record can be quite stale by the time we get to
the interrupt. Using those registers for an unwind is 'interesting' at
best.

Esp. with the multi-pebs stuff that's landed this can be very very
stale, but even single pebs can have a radically different stack at
interrupt time than we had at record time -- imagine a (i)ret happening
in between.

Let me consider that code, and what to do about this; its been a while
since I went over all that.

2016-11-17 17:14:04

[permalink] [raw]

Subject: [PATCH 2/2] dumpstack: prevent KASAN false positive warnings

The oops stack dump code scans the entire stack, which can cause KASAN
"stack-out-of-bounds" false positive warnings. Tell KASAN to ignore it.

Signed-off-by: Josh Poimboeuf <[email protected]>
---
arch/x86/kernel/dumpstack.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 9b7cf5c..85f854b 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -112,7 +112,7 @@ void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs,
for (; stack < stack_info.end; stack++) {
unsigned long real_addr;
int reliable = 0;
- unsigned long addr = *stack;
+ unsigned long addr = READ_ONCE_NOCHECK(*stack);
unsigned long *ret_addr_p =
unwind_get_return_address_ptr(&state);

--
2.7.4

2016-11-17 17:15:26

[permalink] [raw]

Subject: [PATCH 1/2] unwind: prevent KASAN false positive warnings in guess unwinder

The guess unwinder scans the entire stack, which can cause KASAN
"stack-out-of-bounds" false positive warnings. Tell KASAN to ignore it.

Reported-by: Peter Zijlstra <[email protected]>
Signed-off-by: Josh Poimboeuf <[email protected]>
---
arch/x86/kernel/unwind_guess.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/unwind_guess.c b/arch/x86/kernel/unwind_guess.c
index 2d721e5..b80e8bf 100644
--- a/arch/x86/kernel/unwind_guess.c
+++ b/arch/x86/kernel/unwind_guess.c
@@ -7,11 +7,13 @@

unsigned long unwind_get_return_address(struct unwind_state *state)
{
+ unsigned long addr = READ_ONCE_NOCHECK(*state->sp);
+
if (unwind_done(state))
return 0;

return ftrace_graph_ret_addr(state->task, &state->graph_idx,
- *state->sp, state->sp);
+ addr, state->sp);
}
EXPORT_SYMBOL_GPL(unwind_get_return_address);

@@ -23,8 +25,10 @@ bool unwind_next_frame(struct unwind_state *state)
return false;

do {
+ unsigned long addr = READ_ONCE_NOCHECK(*state->sp);
+
for (state->sp++; state->sp < info->end; state->sp++)
- if (__kernel_text_address(*state->sp))
+ if (__kernel_text_address(addr))
return true;

state->sp = info->next_sp;
--
2.7.4

2016-11-17 17:16:04

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, 17 Nov 2016, Dmitry Vyukov wrote:

> On Thu, Nov 17, 2016 at 3:36 PM, Josh Poimboeuf <[email protected]> wrote:
> > On Thu, Nov 17, 2016 at 09:25:58AM -0500, Vince Weaver wrote:
> >> On Thu, 17 Nov 2016, Josh Poimboeuf wrote:
> >>
> >> > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote:
> >> > > Just in case, there is currently a known KASAN false positive related
> >> > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to
> >> > > kernel entry point, this leaves a bunch of stray poisoned redzones on
> >> > > the thread stack. They later cause false stack-out-of-bounds reports.
> >> > >
> >> > > But this does not seem to be the case here. Kernel is not tainted. And
> >> > > shadow at the bottom of the reports looks sane.
> >> > >
> >> > > But if that's the case somehow, we will need to add
> >> > > kasan_unpoison_remaining_stack() call before a longjmp like we did for
> >> > > jprobe_return():
> >> > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ
> >> >
> >> > I'm pretty sure this isn't a KASAN false positive. The unwinder does
> >> > actually seem to be accessing a bad area of the stack, in the middle of
> >> > a function's stack frame.
> >>
> >> I'm having trouble reproducing it on a few other machines I have fuzzing.
> >> So there might be some kernel option contributing, I need to compare
> >> .configs.
> >>
> >> Also the machine that easily triggers the problem I'm compiling with
> >> gcc-5.4 where the machines I can't are using gcc-4.9.
> >
> > I believe KASAN only works with gcc 5 and later, so that would explain
> > why you aren't seeing it with gcc 4.9.
>
> Right. 4.9 has limited support for KASAN. It supports general
> instrumentation, but only with CONFIG_KASAN_OUTLINE, and it does not
> support stack poisoning. Which is required to detect stack OOBs.

I guess it's time to update the other machines to debian-unstable then. I
didn't really need to be able to run dmesg as non-root anyway.

I would actually be compiling the kernels with gcc-6.2 rather than gcc-5
but that seems to not work currently. Haven't had time to see if that's a
known issue or not.

Vince

2016-11-17 17:17:37

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 05:07:00PM +0100, Peter Zijlstra wrote:
> On Thu, Nov 17, 2016 at 09:18:48AM -0600, Josh Poimboeuf wrote:

> > Thanks, I just waded through this and it turned up some good clues. And
> > according to 'git blame', you might be able to help :-)
> >
> > It's not stack corruption. Instead it looks like
> > __intel_pmu_pebs_event() is creating a bad or stale pt_regs which gets
> > passed to the unwinder. Specifically, regs->bp points to a seemingly
> > random address on the NMI stack. Which seems odd, considering the code
> > itself is running on the same NMI stack.
> >
> > I don't know much about the PEBS code but it seems like it's passing
> > some stale data. Either that or there's some NMI nesting going on.

So the puzzle was BP,SP pointing into the NMI stack at random spots. But
I think I can explain this; if the event has a very _very_ short period,
then the tail __intel_pmu_enable_all() call from the PMI handler will
'insta' trigger a record and raise another PMI.

We then get back-to-back NMIs with a record pointing to a now
overwritten stack.

The other scenario, where there is an (i)ret between the record and the
interrupt would be less confusing but still wrong.

Solve this by always using iregs->{bp,sp} for callchains.

The below patch still copies the record BP,SP when !CALLCHAINS &&
SAMPLE_REGS; does this make sense?

The fuzzer is still running with this patch applied.. I'll let it run
for a while.

---
arch/x86/events/intel/ds.c | 35 +++++++++++++++++++++++------------
arch/x86/events/perf_event.h | 2 +-
2 files changed, 24 insertions(+), 13 deletions(-)

diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 0319311dbdbb..be202390bbd3 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1108,20 +1108,20 @@ static void setup_pebs_sample_data(struct perf_event *event,
}

/*
- * We use the interrupt regs as a base because the PEBS record
- * does not contain a full regs set, specifically it seems to
- * lack segment descriptors, which get used by things like
- * user_mode().
+ * We use the interrupt regs as a base because the PEBS record does not
+ * contain a full regs set, specifically it seems to lack segment
+ * descriptors, which get used by things like user_mode().
*
- * In the simple case fix up only the IP and BP,SP regs, for
- * PERF_SAMPLE_IP and PERF_SAMPLE_CALLCHAIN to function properly.
- * A possible PERF_SAMPLE_REGS will have to transfer all regs.
+ * In the simple case fix up only the IP for PERF_SAMPLE_IP.
+ *
+ * We must however always use BP,SP from iregs for the unwinder to stay
+ * sane; the record BP,SP can point into thin air when the record is
+ * from a previous PMI context or an (I)RET happend between the record
+ * and PMI.
*/
*regs = *iregs;
regs->flags = pebs->flags;
set_linear_ip(regs, pebs->ip);
- regs->bp = pebs->bp;
- regs->sp = pebs->sp;

if (sample_type & PERF_SAMPLE_REGS_INTR) {
regs->ax = pebs->ax;
@@ -1130,10 +1130,21 @@ static void setup_pebs_sample_data(struct perf_event *event,
regs->dx = pebs->dx;
regs->si = pebs->si;
regs->di = pebs->di;
- regs->bp = pebs->bp;
- regs->sp = pebs->sp;

- regs->flags = pebs->flags;
+ /*
+ * Per the above; only set BP,SP if we don't need callchains.
+ *
+ * XXX: does this make sense?
+ */
+ if (!(sample_type & PERF_SAMPLE_CALLCHAIN)) {
+ regs->bp = pebs->bp;
+ regs->sp = pebs->sp;
+ }
+
+ /*
+ * Preserve PERF_EFLAGS_VM from set_linear_ip().
+ */
+ regs->flags = pebs->flags | (regs->flags & PERF_EFLAGS_VM);
#ifndef CONFIG_X86_32
regs->r8 = pebs->r8;
regs->r9 = pebs->r9;
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index 5874d8de1f8d..a77ee026643d 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -113,7 +113,7 @@ struct debug_store {
* Per register state.
*/
struct er_account {
- raw_spinlock_t lock; /* per-core: protect structure */
+ raw_spinlock_t lock; /* per-core: protect structure */
u64 config; /* extra MSR config */
u64 reg; /* extra MSR number */
atomic_t ref; /* reference count */

2016-11-17 17:39:37

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote:
> Just in case, there is currently a known KASAN false positive related
> to longjmp's on GPFs. When a syscall hits GPF stack is unwound to
> kernel entry point, this leaves a bunch of stray poisoned redzones on
> the thread stack. They later cause false stack-out-of-bounds reports.
>
> But this does not seem to be the case here. Kernel is not tainted. And
> shadow at the bottom of the reports looks sane.
>
> But if that's the case somehow, we will need to add
> kasan_unpoison_remaining_stack() call before a longjmp like we did for
> jprobe_return():
> https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ

I'm pretty sure this isn't a KASAN false positive. The unwinder does
actually seem to be accessing a bad area of the stack, in the middle of
a function's stack frame.

--
Josh

2016-11-17 17:44:27

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

2016-11-17 17:45:19

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, Nov 17, 2016 at 3:36 PM, Josh Poimboeuf <[email protected]> wrote:
> On Thu, Nov 17, 2016 at 09:25:58AM -0500, Vince Weaver wrote:
>> On Thu, 17 Nov 2016, Josh Poimboeuf wrote:
>>
>> > On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote:
>> > > Just in case, there is currently a known KASAN false positive related
>> > > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to
>> > > kernel entry point, this leaves a bunch of stray poisoned redzones on
>> > > the thread stack. They later cause false stack-out-of-bounds reports.
>> > >
>> > > But this does not seem to be the case here. Kernel is not tainted. And
>> > > shadow at the bottom of the reports looks sane.
>> > >
>> > > But if that's the case somehow, we will need to add
>> > > kasan_unpoison_remaining_stack() call before a longjmp like we did for
>> > > jprobe_return():
>> > > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ
>> >
>> > I'm pretty sure this isn't a KASAN false positive. The unwinder does
>> > actually seem to be accessing a bad area of the stack, in the middle of
>> > a function's stack frame.
>>
>> I'm having trouble reproducing it on a few other machines I have fuzzing.
>> So there might be some kernel option contributing, I need to compare
>> .configs.
>>
>> Also the machine that easily triggers the problem I'm compiling with
>> gcc-5.4 where the machines I can't are using gcc-4.9.
>
> I believe KASAN only works with gcc 5 and later, so that would explain
> why you aren't seeing it with gcc 4.9.

Right. 4.9 has limited support for KASAN. It supports general
instrumentation, but only with CONFIG_KASAN_OUTLINE, and it does not
support stack poisoning. Which is required to detect stack OOBs.

2016-11-17 18:19:54

[permalink] [raw]

Subject: Re: perf: fuzzer KASAN unwind_get_return_address

On Thu, 17 Nov 2016, Josh Poimboeuf wrote:

> On Thu, Nov 17, 2016 at 10:48:27AM +0100, Dmitry Vyukov wrote:
> > Just in case, there is currently a known KASAN false positive related
> > to longjmp's on GPFs. When a syscall hits GPF stack is unwound to
> > kernel entry point, this leaves a bunch of stray poisoned redzones on
> > the thread stack. They later cause false stack-out-of-bounds reports.
> >
> > But this does not seem to be the case here. Kernel is not tainted. And
> > shadow at the bottom of the reports looks sane.
> >
> > But if that's the case somehow, we will need to add
> > kasan_unpoison_remaining_stack() call before a longjmp like we did for
> > jprobe_return():
> > https://groups.google.com/d/msg/kasan-dev/Hzox58yZ4MU/TOdFoWMuBQAJ
>
> I'm pretty sure this isn't a KASAN false positive. The unwinder does
> actually seem to be accessing a bad area of the stack, in the middle of
> a function's stack frame.

I'm having trouble reproducing it on a few other machines I have fuzzing.
So there might be some kernel option contributing, I need to compare
.configs.

Also the machine that easily triggers the problem I'm compiling with
gcc-5.4 where the machines I can't are using gcc-4.9.

Vince

2016-11-17 20:26:29