Greetings,
0day kernel testing robot got the below dmesg and the first bad commit is
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.x86/stackguards
commit 8b275b3754465d502d393f8ae8dd355b7067e73f
Author: Andy Lutomirski <[email protected]>
AuthorDate: Fri Jul 13 19:01:23 2018 -0700
Commit: Thomas Gleixner <[email protected]>
CommitDate: Fri Apr 5 17:04:10 2019 +0200
x86/irq/64: Remap the IRQ stack with guard pages
The IRQ stack lives in percpu space, so an IRQ handler that overflows it
will overwrite other data structures.
Use vmap() to remap the IRQ stack so that it will have the usual guard
pages that vmap/vmalloc allocations have. With this the kernel will panic
immediately on an IRQ stack overflow.
[ tglx: Move the map code to a proper place and invoke it only when a CPU
is about to be brought online. No point in installing the map at
early boot for all possible CPUs. Fail the CPU bringup if the vmap
fails as done for all other preparatory stages in cpu hotplug. ]
Signed-off-by: Andy Lutomirski <[email protected]>
Signed-off-by: Thomas Gleixner <[email protected]>
c8e0bbaa83 x86/irq/64: Split the IRQ stack into its own pages
8b275b3754 x86/irq/64: Remap the IRQ stack with guard pages
2bf08cce47 x86/irq/64: Remove stack overflow debug code
af9671e6ad Merge branch 'perf/urgent'
+-------------------------------------------------------+------------+------------+------------+------------+
| | c8e0bbaa83 | 8b275b3754 | 2bf08cce47 | af9671e6ad |
+-------------------------------------------------------+------------+------------+------------+------------+
| boot_successes | 76 | 0 | 0 | 32 |
| boot_failures | 44 | 37 | 35 | |
| BUG:kernel_in_stage | 41 | 1 | 1 | |
| BUG:kernel_reboot-without-warning_in_test_stage | 1 | | | |
| invoked_oom-killer:gfp_mask=0x | 1 | | | |
| Mem-Info | 1 | | | |
| BUG:kernel_timeout_in_boot_stage | 1 | | | |
| BUG:unable_to_handle_kernel | 0 | 36 | 34 | |
| Oops:#[##] | 0 | 36 | 34 | |
| RIP:slab_kernel_map | 0 | 36 | 34 | |
| RIP:default_idle | 0 | 27 | 33 | |
| Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 0 | 36 | 34 | |
| RIP:_raw_spin_unlock_irqrestore | 0 | 1 | | |
| RIP:lock_acquire | 0 | 1 | | |
| RIP:console_unlock | 0 | 1 | | |
| RIP:parameq | 0 | 2 | | |
| RIP:kfree | 0 | 2 | | |
| RIP:rcu_lockdep_current_cpu_online | 0 | 1 | | |
| RIP:_raw_spin_unlock_irq | 0 | 1 | | |
| RIP:queue_work_on | 0 | 0 | 1 | |
+-------------------------------------------------------+------------+------------+------------+------------+
[ 0.631053] x86/mm: Memory block size: 128MB
[ 0.635665] workqueue: round-robin CPU selection forced, expect performance impact
[ 0.639009] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.640882] futex hash table entries: 512 (order: 4, 65536 bytes)
[ 0.641951] xor: measuring software checksum speed
[ 0.661684] BUG: unable to handle kernel paging request at ffffb659000a1000
[ 0.663041] #PF error: [normal kernel read fault]
[ 0.663947] PGD 17f067 P4D 17f067 PUD 180067 PMD 182067 PTE 0
[ 0.665039] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC PTI
[ 0.665966] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.1.0-rc3-00028-g8b275b37 #1
[ 0.667385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
[ 0.668953] RIP: 0010:slab_kernel_map+0x8e/0x126
[ 0.669846] Code: 4c 8d 68 18 65 8b 0d 53 06 02 64 48 63 c9 4c 8d 74 24 08 41 83 ec 18 48 89 48 10 49 8d 46 07 a9 f8 3f 00 00 74 31 49 83 c6 08 <4d> 8b 7e f8 89 54 24 04 4c 89 ff e8 c0 89 eb ff 85 c0 8b 54 24 04
[ 0.671563] RSP: 0000:ffffb659000a0e18 EFLAGS: 00010082
[ 0.671563] RAX: ffffb659000a1007 RBX: ffff9691001a9d00 RCX: 00000000277047a4
[ 0.671563] RDX: 0000000000000000 RSI: 0000000000000093 RDI: ffff96911ee18600
[ 0.671563] RBP: ffff9691002d2000 R08: 0000000000000000 R09: 0000000000000001
[ 0.671563] R10: ffff96910027e000 R11: 0000000000000002 R12: 00000000000004f8
[ 0.671563] R13: ffff9691002d2af0 R14: ffffb659000a1008 R15: ffffb6590006be48
[ 0.671563] FS: 0000000000000000(0000) GS:ffff96911ee00000(0000) knlGS:0000000000000000
[ 0.671563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.671563] CR2: ffffb659000a1000 CR3: 0000000009016001 CR4: 00000000000206e0
[ 0.671563] Call Trace:
[ 0.671563] <IRQ>
[ 0.671563] ? __put_task_struct+0xf6/0xfd
[ 0.671563] ___cache_free+0x1da/0x399
[ 0.671563] ? cpumask_test_cpu+0x56/0x56
[ 0.671563] kmem_cache_free+0x53/0x14e
[ 0.671563] __put_task_struct+0xf6/0xfd
[ 0.671563] rcu_core+0x53f/0x7ac
[ 0.671563] __do_softirq+0x1b8/0x453
[ 0.671563] irq_exit+0x5c/0x78
[ 0.671563] smp_apic_timer_interrupt+0x1e3/0x226
[ 0.671563] apic_timer_interrupt+0xf/0x20
[ 0.671563] </IRQ>
[ 0.671563] RIP: 0010:default_idle+0x18/0x27
[ 0.671563] Code: c2 ff ff 48 89 ea 48 89 df 31 f6 5b 5d e9 ff 2f b9 ff 65 8b 35 21 ca 75 63 bf 01 00 00 00 e8 13 a3 56 ff e8 e3 49 69 ff fb f4 <65> 8b 35 09 ca 75 63 83 cf ff e9 fd a2 56 ff 55 53 be 15 00 00 00
[ 0.671563] RSP: 0000:ffffb6590006bef0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[ 0.671563] RAX: ffff96910027e000 RBX: ffff96910027e000 RCX: 0000000000000000
[ 0.671563] RDX: 0000000000000000 RSI: 0000000000000006 RDI: ffff96910027e000
[ 0.671563] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000001
[ 0.671563] R10: ffff96910027e000 R11: 0000000000000002 R12: 0000000000000001
[ 0.671563] R13: ffff96910027e000 R14: 0000000000000001 R15: 0000000000000000
[ 0.671563] do_idle+0xf1/0x18d
[ 0.671563] cpu_startup_entry+0x18/0x1a
[ 0.671563] start_secondary+0x183/0x192
[ 0.671563] secondary_startup_64+0xa4/0xb0
[ 0.671563] Modules linked in:
[ 0.671563] CR2: ffffb659000a1000
[ 0.671563] ---[ end trace 7729f5823914a43b ]---
[ 0.671563] RIP: 0010:slab_kernel_map+0x8e/0x126
# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start 78bdf542e65a70f559e19263057a3464711213f0 79a3aaa7b82e3106be97842dedfd8429248896e6 --
git bisect bad 5c2003406ed2f899e07cca0dc44a145af6700fff # 09:23 B 0 11 26 0 Merge 'rcu/rcu/next' into devel-hourly-2019040606
git bisect good a3e2049dd770c728114b7cc3592e2ed925112b0f # 09:39 G 11 0 3 3 Merge 'martineau/kbuild-mptcp-enabled' into devel-hourly-2019040606
git bisect bad bee60f35b21f91ef943b674f45972f955826f281 # 09:52 B 0 2 19 2 Merge 'usb-serial/usb-next' into devel-hourly-2019040606
git bisect bad 2fc6a889d225040f8a967fb6e492df1167d765ef # 10:08 B 0 26 41 0 Merge 'linux-review/Jonathan-Neusch-fer/docs-core-api-Drop-reference-to-flexible-arrays/20190401-131043' into devel-hourly-2019040606
git bisect good bdd89b465ef36a7cd5ef0606136334ba0ea48488 # 10:20 G 32 0 4 4 Merge 'regmap/for-linus' into devel-hourly-2019040606
git bisect good bfa8aee6d0f7a9c9766285d4bb6b14d77d1e37fb # 10:36 G 35 0 14 14 Merge 'linux-review/Cesar-Santos/staging-vt6655-upc-remove-double-blank-lines/20190405-124105' into devel-hourly-2019040606
git bisect good d48e6cc0f5fdfd0f723c5a47bb9e32edb040b72c # 10:46 G 33 0 5 5 Merge 'arm-tegra/for-5.2/arm64/dt' into devel-hourly-2019040606
git bisect bad 76206f239ffec92fcba409389ff2407af675b1d0 # 11:00 B 0 35 50 0 Merge 'linux-review/Axel-Lin/regulator-twl-Constify-regulator_ops/20190405-031410' into devel-hourly-2019040606
git bisect good 22064a8c3f5814c35ccee745707d811794175c69 # 11:13 G 33 0 6 6 Merge 'linux-review/Daniel-Lezcano/thermal-drivers-core-Remove-the-module-Kconfig-s-option/20190401-134608' into devel-hourly-2019040606
git bisect bad edbaf5663a97f2a6e06020e219d3dfaa786f1924 # 11:27 B 0 15 30 0 Merge 'tip/WIP.x86/stackguards' into devel-hourly-2019040606
git bisect good 8198896a7057e99fa63cb12566807ecdb2254291 # 11:42 G 35 0 4 4 x86/cpu: Prepare TSS.IST setup for guard pages
git bisect good c54dc58938f656f9c86df3b5cb002a4a9998b7c6 # 12:22 G 32 0 4 4 x86/irq/32: Make irq stack a character array
git bisect good a53328534aef5ad82df7deb25813b945879a4a75 # 12:42 G 32 0 7 7 x86/irq/32: Handle irq stack allocation failure proper
git bisect bad 8b275b3754465d502d393f8ae8dd355b7067e73f # 12:56 B 0 10 25 0 x86/irq/64: Remap the IRQ stack with guard pages
git bisect good c8e0bbaa8327848e273e86bb19703aaac5ef4a18 # 13:13 G 35 0 20 20 x86/irq/64: Split the IRQ stack into its own pages
# first bad commit: [8b275b3754465d502d393f8ae8dd355b7067e73f] x86/irq/64: Remap the IRQ stack with guard pages
git bisect good c8e0bbaa8327848e273e86bb19703aaac5ef4a18 # 13:19 G 97 0 21 41 x86/irq/64: Split the IRQ stack into its own pages
# extra tests with debug options
git bisect bad 8b275b3754465d502d393f8ae8dd355b7067e73f # 13:38 B 0 29 44 0 x86/irq/64: Remap the IRQ stack with guard pages
# extra tests on HEAD of linux-devel/devel-hourly-2019040606
git bisect bad 78bdf542e65a70f559e19263057a3464711213f0 # 13:38 B 0 10 31 3 0day head guard for 'devel-hourly-2019040606'
# extra tests on tree/branch tip/WIP.x86/stackguards
git bisect bad 2bf08cce47f7527518ca1d29a1fa3e1e4e9d4b39 # 13:51 B 0 34 49 0 x86/irq/64: Remove stack overflow debug code
# extra tests with first bad commit reverted
git bisect good 2cff997291fb30a10af1e6f059e67d6349a0ac36 # 14:13 G 33 0 5 5 Revert "x86/irq/64: Remap the IRQ stack with guard pages"
# extra tests on tree/branch tip/master
git bisect good af9671e6adba5250cf553a0024a9af773cd133ba # 14:36 G 35 0 3 3 Merge branch 'perf/urgent'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation
On Fri, Apr 5, 2019 at 11:38 PM kernel test robot <[email protected]> wrote:
>
> Greetings,
>
> 0day kernel testing robot got the below dmesg and the first bad commit is
>
> https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.x86/stackguards
>
> commit 8b275b3754465d502d393f8ae8dd355b7067e73f
> Author: Andy Lutomirski <[email protected]>
> AuthorDate: Fri Jul 13 19:01:23 2018 -0700
> Commit: Thomas Gleixner <[email protected]>
> CommitDate: Fri Apr 5 17:04:10 2019 +0200
>
> x86/irq/64: Remap the IRQ stack with guard pages
>
> The IRQ stack lives in percpu space, so an IRQ handler that overflows it
> will overwrite other data structures.
>
> Use vmap() to remap the IRQ stack so that it will have the usual guard
> pages that vmap/vmalloc allocations have. With this the kernel will panic
> immediately on an IRQ stack overflow.
>
> [ tglx: Move the map code to a proper place and invoke it only when a CPU
> is about to be brought online. No point in installing the map at
> early boot for all possible CPUs. Fail the CPU bringup if the vmap
> fails as done for all other preparatory stages in cpu hotplug. ]
>
> Signed-off-by: Andy Lutomirski <[email protected]>
> Signed-off-by: Thomas Gleixner <[email protected]>
I haven't spotted the actual bug yet, but the faulting instruction is:
2a: 65 8b 35 09 ca 75 63 mov %gs:*0x6375ca09(%rip),%esi
# 0x6375ca3a <-- trapping instruction
This seems to be faulting just above the top of the stack (the thing
in RSP), so I suspect that there is some path that is shoving the
remapped value into GSBASE, which is wrong.
Also, FWIW, there was some reason that I initialized all the virtual
mappings for all possible CPUs early. I don't remember what it was,
and it may not have been a good reason, but I put at least some
nonzero amount of thought into it :)
--Andy
On Sat, Apr 6, 2019 at 6:54 AM Andy Lutomirski <[email protected]> wrote:
>
> On Fri, Apr 5, 2019 at 11:38 PM kernel test robot <[email protected]> wrote:
> >
> > Greetings,
> >
> > 0day kernel testing robot got the below dmesg and the first bad commit is
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git WIP.x86/stackguards
> >
> > commit 8b275b3754465d502d393f8ae8dd355b7067e73f
> > Author: Andy Lutomirski <[email protected]>
> > AuthorDate: Fri Jul 13 19:01:23 2018 -0700
> > Commit: Thomas Gleixner <[email protected]>
> > CommitDate: Fri Apr 5 17:04:10 2019 +0200
> >
> > x86/irq/64: Remap the IRQ stack with guard pages
> >
> > The IRQ stack lives in percpu space, so an IRQ handler that overflows it
> > will overwrite other data structures.
> >
> > Use vmap() to remap the IRQ stack so that it will have the usual guard
> > pages that vmap/vmalloc allocations have. With this the kernel will panic
> > immediately on an IRQ stack overflow.
> >
> > [ tglx: Move the map code to a proper place and invoke it only when a CPU
> > is about to be brought online. No point in installing the map at
> > early boot for all possible CPUs. Fail the CPU bringup if the vmap
> > fails as done for all other preparatory stages in cpu hotplug. ]
> >
> > Signed-off-by: Andy Lutomirski <[email protected]>
> > Signed-off-by: Thomas Gleixner <[email protected]>
>
> I haven't spotted the actual bug yet, but the faulting instruction is:
>
> 2a: 65 8b 35 09 ca 75 63 mov %gs:*0x6375ca09(%rip),%esi
> # 0x6375ca3a <-- trapping instruction
>
Gah, -ETOOLITTLESLEEP. This is a bit strange:
e: 4c 8d 74 24 08 lea 0x8(%rsp),%r14
...
26: 49 83 c6 08 add $0x8,%r14
2a:* 4d 8b 7e f8 mov -0x8(%r14),%r15 <--
trapping instruction
Which is an access to the stack above RSP by a few bytes. But that
can't be an overflow, since it's *above* RSP. Is something possibly
screwy with the mapping?
I might have a chance to debug this for real this evening. Right now
I'm about to try to wrangle a sick kid through an airport.
On Sat, 6 Apr 2019, Andy Lutomirski wrote:
> I haven't spotted the actual bug yet, but the faulting instruction is:
>
> 2a: 65 8b 35 09 ca 75 63 mov %gs:*0x6375ca09(%rip),%esi
> # 0x6375ca3a <-- trapping instruction
>
> This seems to be faulting just above the top of the stack (the thing
> in RSP), so I suspect that there is some path that is shoving the
> remapped value into GSBASE, which is wrong.
>
> Also, FWIW, there was some reason that I initialized all the virtual
> mappings for all possible CPUs early. I don't remember what it was,
> and it may not have been a good reason, but I put at least some
> nonzero amount of thought into it :)
There is absolutely no reason to have irq stacks before init_IRQ(). 32bit
uses at runtime allocated irq stacks for years.
If the CPU takes a device interrupt before that, then there are way more
things which explode than just the irqstack pointer being NULL.
Thanks,
tglx
> On Apr 7, 2019, at 2:23 AM, Thomas Gleixner <[email protected]> wrote:
>
>> On Sat, 6 Apr 2019, Andy Lutomirski wrote:
>> I haven't spotted the actual bug yet, but the faulting instruction is:
>>
>> 2a: 65 8b 35 09 ca 75 63 mov %gs:*0x6375ca09(%rip),%esi
>> # 0x6375ca3a <-- trapping instruction
>>
>> This seems to be faulting just above the top of the stack (the thing
>> in RSP), so I suspect that there is some path that is shoving the
>> remapped value into GSBASE, which is wrong.
>>
>> Also, FWIW, there was some reason that I initialized all the virtual
>> mappings for all possible CPUs early. I don't remember what it was,
>> and it may not have been a good reason, but I put at least some
>> nonzero amount of thought into it :)
>
> There is absolutely no reason to have irq stacks before init_IRQ(). 32bit
> uses at runtime allocated irq stacks for years.
>
> If the CPU takes a device interrupt before that, then there are way more
> things which explode than just the irqstack pointer being NULL.
>
Fair enough. Although the patch I emailed in the other thread allows at least the entry code to survive on 64-bit at the cost of just a couple lines of code.
But the kernel does indeed seem to work fine without the change. Feel free to disregard that part :)