Hi all,
I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
These were triggered while injecting machine checks using einj.
1)
[321381.466885] ------------[ cut here ]------------
[321381.472144] WARNING: CPU: 136 PID: 0 at kernel/time/timer.c:1098 del_timer_sync+0x36/0x60()
[321381.481571] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) x86_pkg_temp_thermal(E) btrfs(E) intel_powerclamp(E) coretemp(E) kvm(E) xor(E) crct10dif_pclmul(E) raid6_pq(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) iTCO_wdt(E) iTCO_vendor_support(E) joydev(E) aesni_intel(E) lpc_ich(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) cryptd(E) pcspkr(E) mfd_core(E) i2c_i801(E) wmi(E) edac_core(E) shpchp(E) ipmi_si(E) ipmi_msghandler(E) processor(E) acpi_pad(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) ehci_pci(E) ehci_hcd(E) drm_kms_helper(E) ixgbe(E) ahci(E) igb(E) mdio(E) ttm(E) libahci(E) ptp(E) i2c_algo_bit(E) usbcore(E) pps_core(E) drm(E) libata(E) megaraid_sas(E) usb_common(E) dca(E) sg(E) scsi_mod(E) autofs4(E)
[321381.572300] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4
[321381.582117] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015
[321381.593777] ffffffff81818089 ffff88047fc88808 ffffffff8157d67e 0000000000000000
[321381.602184] 0000000000000000 ffff88047fc88848 ffffffff810637fa ffff88046e4bc740
[321381.610595] ffff88047fc888a8 ffff88047fc888a8 0000000104c6f0f8 ffff88047f5cdb00
[321381.619006] Call Trace:
[321381.621834] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65
[321381.628358] [<ffffffff810637fa>] warn_slowpath_common+0x8a/0xc0
[321381.635168] [<ffffffff810638ea>] warn_slowpath_null+0x1a/0x20
[321381.641775] [<ffffffff810cb316>] del_timer_sync+0x36/0x60
[321381.647995] [<ffffffff81582bf0>] schedule_timeout+0x150/0x280
[321381.654611] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0
[321381.660547] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80
[321381.667359] [<ffffffff810cb85c>] msleep+0x3c/0x50
[321381.672812] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200]
[321381.680404] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper]
[321381.689453] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper]
[321381.698706] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm]
[321381.706971] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper]
[321381.715244] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper]
[321381.724780] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper]
[321381.733144] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80
[321381.739859] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30
[321381.747252] [<ffffffff815792d4>] panic+0xee/0x1f5
[321381.752704] [<ffffffff8102d272>] mce_panic+0x1e2/0x200
[321381.758640] [<ffffffff8102d303>] mce_timed_out+0x73/0x80
[321381.764762] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0
[321381.771377] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
[321381.777499] [<ffffffff81585d49>] machine_check+0x29/0x50
[321381.783630] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
[321381.789760] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0
[321381.797457] [<ffffffff81450327>] cpuidle_enter+0x17/0x20
[321381.803586] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390
[321381.810297] [<ffffffff8103a203>] start_secondary+0x143/0x170
[321381.816814] ---[ end trace 9f2a977c4a9be24e ]---
[321381.822068] bad: scheduling from the idle thread!
[321381.827421] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4
[321381.837238] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015
[321381.848898] ffff88046e4bc740 ffff88047fc887a8 ffffffff8157d67e 0000000000000000
[321381.857305] ffff88047fc95300 ffff88047fc887c8 ffffffff81093675 ffff88047fc88808
[321381.865713] ffff88047fc95300 ffff88047fc887f8 ffffffff8108796c 0000000100000000
[321381.874124] Call Trace:
[321381.876951] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65
[321381.883483] [<ffffffff81093675>] dequeue_task_idle+0x35/0x50
[321381.890001] [<ffffffff8108796c>] dequeue_task+0x5c/0x80
[321381.896027] [<ffffffff8108c56b>] deactivate_task+0x2b/0x30
[321381.902352] [<ffffffff8157fcea>] __schedule+0x64a/0x910
[321381.908385] [<ffffffff8157ffee>] schedule+0x3e/0x90
[321381.914030] [<ffffffff81582be8>] schedule_timeout+0x148/0x280
[321381.920636] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0
[321381.926570] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80
[321381.933382] [<ffffffff810cb85c>] msleep+0x3c/0x50
[321381.938835] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200]
[321381.946428] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper]
[321381.955478] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper]
[321381.964731] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm]
[321381.973004] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper]
[321381.981277] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper]
[321381.990811] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper]
[321381.999174] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80
[321382.005887] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30
[321382.013280] [<ffffffff815792d4>] panic+0xee/0x1f5
[321382.018731] [<ffffffff8102d272>] mce_panic+0x1e2/0x200
[321382.024660] [<ffffffff8102d303>] mce_timed_out+0x73/0x80
[321382.030787] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0
[321382.037404] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
[321382.043533] [<ffffffff81585d49>] machine_check+0x29/0x50
[321382.049665] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
[321382.055794] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0
[321382.063491] [<ffffffff81450327>] cpuidle_enter+0x17/0x20
[321382.069623] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390
[321382.076335] [<ffffffff8103a203>] start_secondary+0x143/0x170
[321382.082877] ------------[ cut here ]------------
2)
bkd04sdp:~ # [58109.056018] Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
[58109.056058] mce: [Hardware Error]: Machine check events logged
[58110.109873] Shutting down cpus with NMI
[58110.176778] Kernel Offset: disabled
[58110.180667] drm_kms_helper: panic occurred, switching back to text console
[58110.188367] mga_delay choosing mdelay...
[58110.242399] mga_delay choosing mdelay...
[58110.266768] ------------[ cut here ]------------
[58110.271926] kernel BUG at mm/vmalloc.c:1335!
[58110.276695] invalid opcode: 0000 [#1] SMP
[58110.281289] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) btrfs(E) xor(E) raid6_pq(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm(E) joydev(E) iTCO_wdt(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) lpc_ich(E) cryptd(E) pcspkr(E) edac_core(E) mfd_core(E) i2c_i801(E) shpchp(E) wmi(E) ipmi_si(E) ipmi_msghandler(E) acpi_pad(E) processor(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) ahci(E) ehci_pci(E) sysimgblt(E) drm_kms_helper(E) ehci_hcd(E) ixgbe(E) igb(E) ttm(E) libahci(E) mdio(E) ptp(E) usbcore(E) pps_core(E) drm(E) libata(E) i2c_algo_bit(E) usb_common(E) dca(E) megaraid_sas(E) sg(E) scsi_mod(E) autofs4(E)
[58110.371884] CPU: 75 PID: 0 Comm: swapper/75 Tainted: G E 4.1.0-rc8-7-default+ #10
[58110.381506] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015
[58110.393063] task: ffff88046ea6d580 ti: ffff88046ea70000 task.ti: ffff88046ea70000
[58110.401422] RIP: 0010:[<ffffffff81189c65>] [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160
[58110.411156] RSP: 0018:ffff88047f7284b8 EFLAGS: 00010006
[58110.417091] RAX: 0000000080010003 RBX: 0000000091000000 RCX: ffffc90000000000
[58110.425065] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00000000000eb000
[58110.433038] RBP: ffff88047f7284f8 R08: ffffe8ffffffffff R09: 00000000ffffffff
[58110.441010] R10: ffff880036a6a700 R11: ffff880460ab69c0 R12: 00000000910eb000
[58110.448983] R13: 0000000000000001 R14: 0000000091000000 R15: 00000000000eb000
[58110.456955] FS: 0000000000000000(0000) GS:ffff88047f720000(0000) knlGS:0000000000000000
[58110.465994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[58110.472413] CR2: 00007f5bdb9ac095 CR3: 0000000001a0b000 CR4: 00000000001407e0
[58110.480386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[58110.488358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[58110.496331] Stack:
[58110.498577] ffff88047f728518 ffffc90000000000 00000000910eafff 0000000091000000
[58110.506885] 00000000910eb000 0000000000000001 0000000091000000 00000000000eb000
[58110.515192] ffff88047f728518 ffffffff8118ad20 00000000000000d0 ffffffffa0334f78
[58110.523499] Call Trace:
[58110.526229] <#MC>
[58110.528382] [<ffffffff8118ad20>] get_vm_area_caller+0x40/0x50
[58110.535111] [<ffffffffa0334f78>] ? ttm_mem_reg_ioremap+0xc8/0x110 [ttm]
[58110.542607] [<ffffffff81050f18>] __ioremap_caller+0x188/0x390
[58110.549127] [<ffffffff812d59b9>] ? find_next_bit+0x19/0x20
[58110.555353] [<ffffffff81051177>] ioremap_wc+0x17/0x20
[58110.561099] [<ffffffffa0334f78>] ttm_mem_reg_ioremap+0xc8/0x110 [ttm]
[58110.568398] [<ffffffffa0335351>] ttm_bo_move_memcpy+0xd1/0x700 [ttm]
[58110.575598] [<ffffffff811a6c55>] ? __kmalloc+0x4b5/0x4c0
[58110.581632] [<ffffffffa01d9b48>] mgag200_bo_move+0x18/0x20 [mgag200]
[58110.588830] [<ffffffffa0332ea0>] ttm_bo_handle_move_mem+0x260/0x590 [ttm]
[58110.596514] [<ffffffffa03337d2>] ? ttm_bo_mem_space+0xd2/0x320 [ttm]
[58110.603705] [<ffffffffa0333eb2>] ttm_bo_validate+0x1c2/0x1d0 [ttm]
[58110.610711] [<ffffffff8113f681>] ? irq_work_queue+0x11/0x90
[58110.617037] [<ffffffffa01da3d3>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200]
[58110.624915] [<ffffffffa01d5a26>] mga_crtc_do_set_base.isra.8.constprop.21+0x76/0x410 [mgag200]
[58110.634636] [<ffffffffa01d6e02>] mga_crtc_mode_set+0x1042/0x2140 [mgag200]
[58110.642416] [<ffffffffa01d5492>] ? mga_crtc_prepare+0x132/0x370 [mgag200]
[58110.650106] [<ffffffffa04ec8db>] drm_crtc_helper_set_mode+0x2fb/0x530 [drm_kms_helper]
[58110.659052] [<ffffffffa04ed896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper]
[58110.668217] [<ffffffffa00b5318>] drm_mode_set_config_internal+0x68/0x100 [drm]
[58110.676388] [<ffffffffa04f78b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper]
[58110.684558] [<ffffffffa04f7aa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper]
[58110.693989] [<ffffffffa04f86f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper]
[58110.702260] [<ffffffff81081bad>] notifier_call_chain+0x4d/0x80
[58110.708873] [<ffffffff81081c31>] atomic_notifier_call_chain+0x21/0x30
[58110.716169] [<ffffffff8156f954>] panic+0xee/0x1f5
[58110.721530] [<ffffffff8102d232>] mce_panic+0x1e2/0x200
[58110.727366] [<ffffffff8102d2c3>] mce_timed_out+0x73/0x80
[58110.733396] [<ffffffff8102e861>] do_machine_check+0x5f1/0xae0
[58110.739915] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130
[58110.745952] [<ffffffff8157c3c9>] machine_check+0x29/0x50
[58110.751984] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130
[58110.758017] <<EOE>>
[58110.760362] [<ffffffff814467f0>] cpuidle_enter_state+0x70/0x1f0
[58110.767275] [<ffffffff814469a7>] cpuidle_enter+0x17/0x20
[58110.773309] [<ffffffff810a4e18>] cpu_startup_entry+0x308/0x390
[58110.779916] [<ffffffff8103a163>] start_secondary+0x143/0x170
[58110.786325] Code: 00 00 48 0f bd cf 83 c1 01 83 f9 0c 0f 4c c8 b0 1e 83 f9 1e 0f 4f c8 49 d3 e6 e9 f8 fe ff ff 48 89 df e8 9f a8 01 00 31 c0 eb b8 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 49 89 c8 41
[58110.808146] RIP [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160
[58110.815257] RSP <ffff88047f7284b8>
[58110.820218] ---[ end trace ab0c230901a0ee95 ]---
Thanks
Rui
On Fri, Jun 26, 2015 at 9:55 AM, Rui Wang <[email protected]> wrote:
> Hi all,
>
> I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
This is the same for all drm drivers, the drm atomic handling with
fbcon/fbdev is totally broken. It would be serious work to fix this
properly.
-Daniel
>
> These were triggered while injecting machine checks using einj.
>
> 1)
>
> [321381.466885] ------------[ cut here ]------------
> [321381.472144] WARNING: CPU: 136 PID: 0 at kernel/time/timer.c:1098 del_timer_sync+0x36/0x60()
> [321381.481571] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) x86_pkg_temp_thermal(E) btrfs(E) intel_powerclamp(E) coretemp(E) kvm(E) xor(E) crct10dif_pclmul(E) raid6_pq(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) iTCO_wdt(E) iTCO_vendor_support(E) joydev(E) aesni_intel(E) lpc_ich(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) cryptd(E) pcspkr(E) mfd_core(E) i2c_i801(E) wmi(E) edac_core(E) shpchp(E) ipmi_si(E) ipmi_msghandler(E) processor(E) acpi_pad(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) sysimgblt(E) ehci_pci(E) ehci_hcd(E) drm_kms_helper(E) ixgbe(E) ahci(E) igb(E) mdio(E) ttm(E) libahci(E) ptp(E) i2c_algo_bit(E) usbcore(E) pps_core(E) drm(E) libata(E) megaraid_sas(E) usb_common(E) dca(E) sg(E) scsi_mod(E) autofs4(E)
> [321381.572300] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4
> [321381.582117] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015
> [321381.593777] ffffffff81818089 ffff88047fc88808 ffffffff8157d67e 0000000000000000
> [321381.602184] 0000000000000000 ffff88047fc88848 ffffffff810637fa ffff88046e4bc740
> [321381.610595] ffff88047fc888a8 ffff88047fc888a8 0000000104c6f0f8 ffff88047f5cdb00
> [321381.619006] Call Trace:
> [321381.621834] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65
> [321381.628358] [<ffffffff810637fa>] warn_slowpath_common+0x8a/0xc0
> [321381.635168] [<ffffffff810638ea>] warn_slowpath_null+0x1a/0x20
> [321381.641775] [<ffffffff810cb316>] del_timer_sync+0x36/0x60
> [321381.647995] [<ffffffff81582bf0>] schedule_timeout+0x150/0x280
> [321381.654611] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0
> [321381.660547] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80
> [321381.667359] [<ffffffff810cb85c>] msleep+0x3c/0x50
> [321381.672812] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200]
> [321381.680404] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper]
> [321381.689453] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper]
> [321381.698706] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm]
> [321381.706971] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper]
> [321381.715244] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper]
> [321381.724780] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper]
> [321381.733144] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80
> [321381.739859] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30
> [321381.747252] [<ffffffff815792d4>] panic+0xee/0x1f5
> [321381.752704] [<ffffffff8102d272>] mce_panic+0x1e2/0x200
> [321381.758640] [<ffffffff8102d303>] mce_timed_out+0x73/0x80
> [321381.764762] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0
> [321381.771377] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
> [321381.777499] [<ffffffff81585d49>] machine_check+0x29/0x50
> [321381.783630] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
> [321381.789760] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0
> [321381.797457] [<ffffffff81450327>] cpuidle_enter+0x17/0x20
> [321381.803586] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390
> [321381.810297] [<ffffffff8103a203>] start_secondary+0x143/0x170
> [321381.816814] ---[ end trace 9f2a977c4a9be24e ]---
> [321381.822068] bad: scheduling from the idle thread!
> [321381.827421] CPU: 136 PID: 0 Comm: swapper/136 Tainted: G W E 4.1.0-rc8-7-default+ #4
> [321381.837238] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015
> [321381.848898] ffff88046e4bc740 ffff88047fc887a8 ffffffff8157d67e 0000000000000000
> [321381.857305] ffff88047fc95300 ffff88047fc887c8 ffffffff81093675 ffff88047fc88808
> [321381.865713] ffff88047fc95300 ffff88047fc887f8 ffffffff8108796c 0000000100000000
> [321381.874124] Call Trace:
> [321381.876951] <#MC> [<ffffffff8157d67e>] dump_stack+0x4c/0x65
> [321381.883483] [<ffffffff81093675>] dequeue_task_idle+0x35/0x50
> [321381.890001] [<ffffffff8108796c>] dequeue_task+0x5c/0x80
> [321381.896027] [<ffffffff8108c56b>] deactivate_task+0x2b/0x30
> [321381.902352] [<ffffffff8157fcea>] __schedule+0x64a/0x910
> [321381.908385] [<ffffffff8157ffee>] schedule+0x3e/0x90
> [321381.914030] [<ffffffff81582be8>] schedule_timeout+0x148/0x280
> [321381.920636] [<ffffffff812cc9fb>] ? idr_alloc+0x7b/0xe0
> [321381.926570] [<ffffffff810c9c90>] ? internal_add_timer+0x80/0x80
> [321381.933382] [<ffffffff810cb85c>] msleep+0x3c/0x50
> [321381.938835] [<ffffffffa0145607>] mga_crtc_prepare+0x167/0x370 [mgag200]
> [321381.946428] [<ffffffffa04f38b6>] drm_crtc_helper_set_mode+0x2d6/0x530 [drm_kms_helper]
> [321381.955478] [<ffffffffa04f4896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper]
> [321381.964731] [<ffffffffa00a3318>] drm_mode_set_config_internal+0x68/0x100 [drm]
> [321381.973004] [<ffffffffa04fe8b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper]
> [321381.981277] [<ffffffffa04feaa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper]
> [321381.990811] [<ffffffffa04ff6f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper]
> [321381.999174] [<ffffffff8108270d>] notifier_call_chain+0x4d/0x80
> [321382.005887] [<ffffffff81082791>] atomic_notifier_call_chain+0x21/0x30
> [321382.013280] [<ffffffff815792d4>] panic+0xee/0x1f5
> [321382.018731] [<ffffffff8102d272>] mce_panic+0x1e2/0x200
> [321382.024660] [<ffffffff8102d303>] mce_timed_out+0x73/0x80
> [321382.030787] [<ffffffff8102e8a1>] do_machine_check+0x5f1/0xae0
> [321382.037404] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
> [321382.043533] [<ffffffff81585d49>] machine_check+0x29/0x50
> [321382.049665] [<ffffffff81348eaf>] ? intel_idle+0xbf/0x130
> [321382.055794] <<EOE>> [<ffffffff81450170>] cpuidle_enter_state+0x70/0x1f0
> [321382.063491] [<ffffffff81450327>] cpuidle_enter+0x17/0x20
> [321382.069623] [<ffffffff810a5968>] cpu_startup_entry+0x308/0x390
> [321382.076335] [<ffffffff8103a203>] start_secondary+0x143/0x170
> [321382.082877] ------------[ cut here ]------------
>
>
> 2)
>
> bkd04sdp:~ # [58109.056018] Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
> [58109.056058] mce: [Hardware Error]: Machine check events logged
> [58110.109873] Shutting down cpus with NMI
> [58110.176778] Kernel Offset: disabled
> [58110.180667] drm_kms_helper: panic occurred, switching back to text console
> [58110.188367] mga_delay choosing mdelay...
> [58110.242399] mga_delay choosing mdelay...
> [58110.266768] ------------[ cut here ]------------
> [58110.271926] kernel BUG at mm/vmalloc.c:1335!
> [58110.276695] invalid opcode: 0000 [#1] SMP
> [58110.281289] Modules linked in: einj(E) nmioe(E) iscsi_ibft(E) iscsi_boot_sysfs(E) af_packet(E) btrfs(E) xor(E) raid6_pq(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm(E) joydev(E) iTCO_wdt(E) iTCO_vendor_support(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) ghash_clmulni_intel(E) aesni_intel(E) aes_x86_64(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) lpc_ich(E) cryptd(E) pcspkr(E) edac_core(E) mfd_core(E) i2c_i801(E) shpchp(E) wmi(E) ipmi_si(E) ipmi_msghandler(E) acpi_pad(E) processor(E) button(E) dm_mod(E) ext4(E) crc16(E) mbcache(E) jbd2(E) hid_generic(E) usbhid(E) sr_mod(E) cdrom(E) sd_mod(E) mgag200(E) syscopyarea(E) sysfillrect(E) ahci(E) ehci_pci(E) sysimgblt(E) drm_kms_helper(E) ehci_hcd(E) ixgbe(E) igb(E) ttm(E) libahci(E) mdio(E) ptp(E) usbcore(E) pps_core(E) drm(E) libata(E) i2c_algo_bit(E) usb_common(E) dca(E) megaraid_sas(E) sg(E) scsi_mod(E) autofs4(E)
> [58110.371884] CPU: 75 PID: 0 Comm: swapper/75 Tainted: G E 4.1.0-rc8-7-default+ #10
> [58110.381506] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0059.R00.1501081238 01/08/2015
> [58110.393063] task: ffff88046ea6d580 ti: ffff88046ea70000 task.ti: ffff88046ea70000
> [58110.401422] RIP: 0010:[<ffffffff81189c65>] [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160
> [58110.411156] RSP: 0018:ffff88047f7284b8 EFLAGS: 00010006
> [58110.417091] RAX: 0000000080010003 RBX: 0000000091000000 RCX: ffffc90000000000
> [58110.425065] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00000000000eb000
> [58110.433038] RBP: ffff88047f7284f8 R08: ffffe8ffffffffff R09: 00000000ffffffff
> [58110.441010] R10: ffff880036a6a700 R11: ffff880460ab69c0 R12: 00000000910eb000
> [58110.448983] R13: 0000000000000001 R14: 0000000091000000 R15: 00000000000eb000
> [58110.456955] FS: 0000000000000000(0000) GS:ffff88047f720000(0000) knlGS:0000000000000000
> [58110.465994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [58110.472413] CR2: 00007f5bdb9ac095 CR3: 0000000001a0b000 CR4: 00000000001407e0
> [58110.480386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [58110.488358] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [58110.496331] Stack:
> [58110.498577] ffff88047f728518 ffffc90000000000 00000000910eafff 0000000091000000
> [58110.506885] 00000000910eb000 0000000000000001 0000000091000000 00000000000eb000
> [58110.515192] ffff88047f728518 ffffffff8118ad20 00000000000000d0 ffffffffa0334f78
> [58110.523499] Call Trace:
> [58110.526229] <#MC>
> [58110.528382] [<ffffffff8118ad20>] get_vm_area_caller+0x40/0x50
> [58110.535111] [<ffffffffa0334f78>] ? ttm_mem_reg_ioremap+0xc8/0x110 [ttm]
> [58110.542607] [<ffffffff81050f18>] __ioremap_caller+0x188/0x390
> [58110.549127] [<ffffffff812d59b9>] ? find_next_bit+0x19/0x20
> [58110.555353] [<ffffffff81051177>] ioremap_wc+0x17/0x20
> [58110.561099] [<ffffffffa0334f78>] ttm_mem_reg_ioremap+0xc8/0x110 [ttm]
> [58110.568398] [<ffffffffa0335351>] ttm_bo_move_memcpy+0xd1/0x700 [ttm]
> [58110.575598] [<ffffffff811a6c55>] ? __kmalloc+0x4b5/0x4c0
> [58110.581632] [<ffffffffa01d9b48>] mgag200_bo_move+0x18/0x20 [mgag200]
> [58110.588830] [<ffffffffa0332ea0>] ttm_bo_handle_move_mem+0x260/0x590 [ttm]
> [58110.596514] [<ffffffffa03337d2>] ? ttm_bo_mem_space+0xd2/0x320 [ttm]
> [58110.603705] [<ffffffffa0333eb2>] ttm_bo_validate+0x1c2/0x1d0 [ttm]
> [58110.610711] [<ffffffff8113f681>] ? irq_work_queue+0x11/0x90
> [58110.617037] [<ffffffffa01da3d3>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200]
> [58110.624915] [<ffffffffa01d5a26>] mga_crtc_do_set_base.isra.8.constprop.21+0x76/0x410 [mgag200]
> [58110.634636] [<ffffffffa01d6e02>] mga_crtc_mode_set+0x1042/0x2140 [mgag200]
> [58110.642416] [<ffffffffa01d5492>] ? mga_crtc_prepare+0x132/0x370 [mgag200]
> [58110.650106] [<ffffffffa04ec8db>] drm_crtc_helper_set_mode+0x2fb/0x530 [drm_kms_helper]
> [58110.659052] [<ffffffffa04ed896>] drm_crtc_helper_set_config+0x856/0xa70 [drm_kms_helper]
> [58110.668217] [<ffffffffa00b5318>] drm_mode_set_config_internal+0x68/0x100 [drm]
> [58110.676388] [<ffffffffa04f78b2>] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper]
> [58110.684558] [<ffffffffa04f7aa3>] drm_fb_helper_force_kernel_mode+0x73/0xb0 [drm_kms_helper]
> [58110.693989] [<ffffffffa04f86f9>] drm_fb_helper_panic+0x29/0x30 [drm_kms_helper]
> [58110.702260] [<ffffffff81081bad>] notifier_call_chain+0x4d/0x80
> [58110.708873] [<ffffffff81081c31>] atomic_notifier_call_chain+0x21/0x30
> [58110.716169] [<ffffffff8156f954>] panic+0xee/0x1f5
> [58110.721530] [<ffffffff8102d232>] mce_panic+0x1e2/0x200
> [58110.727366] [<ffffffff8102d2c3>] mce_timed_out+0x73/0x80
> [58110.733396] [<ffffffff8102e861>] do_machine_check+0x5f1/0xae0
> [58110.739915] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130
> [58110.745952] [<ffffffff8157c3c9>] machine_check+0x29/0x50
> [58110.751984] [<ffffffff8133f52f>] ? intel_idle+0xbf/0x130
> [58110.758017] <<EOE>>
> [58110.760362] [<ffffffff814467f0>] cpuidle_enter_state+0x70/0x1f0
> [58110.767275] [<ffffffff814469a7>] cpuidle_enter+0x17/0x20
> [58110.773309] [<ffffffff810a4e18>] cpu_startup_entry+0x308/0x390
> [58110.779916] [<ffffffff8103a163>] start_secondary+0x143/0x170
> [58110.786325] Code: 00 00 48 0f bd cf 83 c1 01 83 f9 0c 0f 4c c8 b0 1e 83 f9 1e 0f 4f c8 49 d3 e6 e9 f8 fe ff ff 48 89 df e8 9f a8 01 00 31 c0 eb b8 <0f> 0b 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 49 89 c8 41
> [58110.808146] RIP [<ffffffff81189c65>] __get_vm_area_node+0x155/0x160
> [58110.815257] RSP <ffff88047f7284b8>
> [58110.820218] ---[ end trace ab0c230901a0ee95 ]---
>
> Thanks
> Rui
>
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
>> I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
>
> This is the same for all drm drivers, the drm atomic handling with
> fbcon/fbdev is totally broken. It would be serious work to fix this
> properly.
It's a serious problem when a server crashes ... even worse when it hangs while doing so
because we have to rely on some other agent to notice the hung server and go poke it
with a stick.
If it is too hard to fix all of the drivers, is it possible to attack this in the allocator?
-Tony
????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay? ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m???? ????????I?
On Fri, Jun 26, 2015 at 8:30 PM, Luck, Tony <[email protected]> wrote:
>>> I'm here to report two panics which hang forever (the machine cannot reboot). It is because mgag200 doesn't work in panic context. It sleeps and allocates memory non-atomically.
>>
>> This is the same for all drm drivers, the drm atomic handling with
>> fbcon/fbdev is totally broken. It would be serious work to fix this
>> properly.
>
> It's a serious problem when a server crashes ... even worse when it hangs while doing so
> because we have to rely on some other agent to notice the hung server and go poke it
> with a stick.
>
> If it is too hard to fix all of the drivers, is it possible to attack this in the allocator?
Hm, what do you mean by fixing this in the allocator? I've made some
rough sketch of the problem space in
http://www.x.org/wiki/DRMJanitors/ under "Make panic handling work".
Problem is that the folks which know what to do (drm hackers) have
zero incentive to fix it (since if you blow up a drm driver any kind
of fbcon panic handling is hopeless anyway).
The other problem is is that this is a serious effort with tons of
little things all over to consider. My gut estimate is that probably
it'll take something of the order of a man year to fix this for real.
David Herrmann has supplied parts of the required puzzle to actually
be able to somewhat reliably show panics on drm modesetting drivers,
but that didn't contain any of the work to make fbdev not totally suck
at panic handling first. And I guess for general distros and servers
that's needed - developers simply disable all of fbdev to be able to
debug kms hangs.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Sat, Jun 27, 2015 at 03:52:56PM +0200, Daniel Vetter wrote:
> Hm, what do you mean by fixing this in the allocator? I've made some
> rough sketch of the problem space in
> http://www.x.org/wiki/DRMJanitors/ under "Make panic handling work".
> Problem is that the folks which know what to do (drm hackers) have
> zero incentive to fix it (since if you blow up a drm driver any kind
> of fbcon panic handling is hopeless anyway).
Ok, silly question: switching the GPU output to the simplest,
supported-by-all-GPUs mode before panicking is not that easy too, right?
Or does that mean, one needs to reinit GPU in order to even show
something...
Thanks.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
On Sat, Jun 27, 2015 at 4:12 PM, Borislav Petkov <[email protected]> wrote:
> On Sat, Jun 27, 2015 at 03:52:56PM +0200, Daniel Vetter wrote:
>> Hm, what do you mean by fixing this in the allocator? I've made some
>> rough sketch of the problem space in
>> http://www.x.org/wiki/DRMJanitors/ under "Make panic handling work".
>> Problem is that the folks which know what to do (drm hackers) have
>> zero incentive to fix it (since if you blow up a drm driver any kind
>> of fbcon panic handling is hopeless anyway).
>
> Ok, silly question: switching the GPU output to the simplest,
> supported-by-all-GPUs mode before panicking is not that easy too, right?
Anything that resembles doing a modeset (and the precise mode is
totally irrelevant) will be almost impossible: There's clocks to tune,
various links to train, a bunch of sidebind communication and all that
takes time, needs to happen in a very precise order and also needs
lots of mutexes (crossing subsystems sometimes even) to avoid
trampling on maybe the thread that just died and caused the panic.
In short you need specially-written codepaths for panic, which means
only the most minimal thing has a chance to even work. fbdev/fbcon and
the current drm panic handler are anything but that. Doing a full
modeset isn't it either. Heck even the just updating the displayed
buffer might need massive reconfiguration of the memory fetch
watermarks if e.g. the screen is display high bpc yuv and you want to
display the rgb fbcon buffer. And if you'd just try to take what's
currently being displayed there's the annoying problem that often
providing a cpu-contiguous view of that needs a vmalloc area, if not
first moving the buffer around a bit in vram with the copy engine.
That all involves lots of really complex code.
> Or does that mean, one needs to reinit GPU in order to even show
> something...
If the gpu is runtime suspended, then yes you'd get to even init the
entire thing first before you can display anything.
fbdev panic handling was designed for a world where you'd bash a few
values into a few registers and never wait for the CRTC to reach
stable timings and output a useful signal. Which could all happen very
much after the kernel made it's dying sigh. Display hw has long
stopped being this simple and display drivers also. The only vestige
is that we still call the display pipeline object in drm CRTC for
cathode ray tube controller ...
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Sat, Jun 27, 2015 at 07:56:19PM +0200, Daniel Vetter wrote:
<snip very useful explanation for !GPU people>
> Which could all happen very much after the kernel made it's dying
> sigh. Display hw has long stopped being this simple and display
> drivers also.
Thanks for the explanation. I was fearing that it would go in such
direction though. :\ Oh well.
So there are two observations to be ventured here:
* there's still no robust and simple way to save/show oops messages on
laptops. When a laptop crashes and the user has only this one machine,
catching the oops can be a serious PITA. Laptop most likely doesn't have
serial, modeset is hard to do, as you explained, and saving it into
persistent storage (UEFI) might brick it in some cases or whatever the
firmware will do to it. I guess we can keep wishing here for something
better.
* The server aspect might not really need GPU modesetting IMHO because
realistically, servers have serial so the oops can go out there. Which
begs the other question:
Shouldn't drm_fb_helper_panic() and that path below it simply be
removed?
I mean, if it needs to do all that hw-reconf on the panic path and if
that reconfiguration is not completely reliable, maybe we shouldn't do
it all?
This way, we will basically remain as quiet as possible in order not
to muddle the system unreasonably when we're about to die and the only
thing we want to catch is an oops or maybe kexec the crash kernel...
Oops will go out on serial anyway and server will kexec - GPU/fb stuff
doesn't do anything. Yes, no?
Hmmm.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
On Mon, Jun 29, 2015 at 10:09 AM, Borislav Petkov <[email protected]> wrote:
> On Sat, Jun 27, 2015 at 07:56:19PM +0200, Daniel Vetter wrote:
>
> <snip very useful explanation for !GPU people>
>
>> Which could all happen very much after the kernel made it's dying
>> sigh. Display hw has long stopped being this simple and display
>> drivers also.
>
> Thanks for the explanation. I was fearing that it would go in such
> direction though. :\ Oh well.
>
> So there are two observations to be ventured here:
>
> * there's still no robust and simple way to save/show oops messages on
> laptops. When a laptop crashes and the user has only this one machine,
> catching the oops can be a serious PITA. Laptop most likely doesn't have
> serial, modeset is hard to do, as you explained, and saving it into
> persistent storage (UEFI) might brick it in some cases or whatever the
> firmware will do to it. I guess we can keep wishing here for something
> better.
As long as the display is up and running we should have a fair stab at
showing the oops - it's just that no one has seriously bothered with
the necessary infastructure, automated testing (it won't work
otherwise) and driver work.
> * The server aspect might not really need GPU modesetting IMHO because
> realistically, servers have serial so the oops can go out there. Which
> begs the other question:
>
> Shouldn't drm_fb_helper_panic() and that path below it simply be
> removed?
>
> I mean, if it needs to do all that hw-reconf on the panic path and if
> that reconfiguration is not completely reliable, maybe we shouldn't do
> it all?
>
> This way, we will basically remain as quiet as possible in order not
> to muddle the system unreasonably when we're about to die and the only
> thing we want to catch is an oops or maybe kexec the crash kernel...
>
> Oops will go out on serial anyway and server will kexec - GPU/fb stuff
> doesn't do anything. Yes, no?
drm_fb_helper_panic isn't the only panic handler - fbdev/fbcon have
their own. They interfere, and fbdev blissfully assumes that it can
call almost any driver hook from hardirq context. Which means you'd
also need to consolidate the various hand-rolled (in drivers instead
of the fbdev helper) bits to offload fbdev callbacks to workers and
make sure we don't go boom in panics in there either.
It's really a big mess unfortunately :(
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Mon, Jun 29, 2015 at 11:25:17AM +0200, Daniel Vetter wrote:
> As long as the display is up and running we should have a fair stab at
> showing the oops
Yeah, that has the same problem as all the other methods for showing
oops - *if* it is still healthly. Like, for example, trying to catch an
oops in networking with netconsole :)
> - it's just that no one has seriously bothered with the necessary
> infastructure, automated testing (it won't work otherwise) and driver
> work.
Right.
> drm_fb_helper_panic isn't the only panic handler - fbdev/fbcon have
> their own. They interfere, and fbdev blissfully assumes that it can
> call almost any driver hook from hardirq context. Which means you'd
> also need to consolidate the various hand-rolled (in drivers instead
> of the fbdev helper) bits to offload fbdev callbacks to workers and
> make sure we don't go boom in panics in there either.
Hmmm, this would mean to go and tell them through the panic notifier
to not do anything "risky" anymore. Perhaps even stop them from doing
regular tasks which could become risky in the case of panic. I guess
this would mean, we should be doing the same in other subsystems too.
Hmm.
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--
On Mon, Jun 29, 2015 at 11:42 AM, Borislav Petkov <[email protected]> wrote:
>> drm_fb_helper_panic isn't the only panic handler - fbdev/fbcon have
>> their own. They interfere, and fbdev blissfully assumes that it can
>> call almost any driver hook from hardirq context. Which means you'd
>> also need to consolidate the various hand-rolled (in drivers instead
>> of the fbdev helper) bits to offload fbdev callbacks to workers and
>> make sure we don't go boom in panics in there either.
>
> Hmmm, this would mean to go and tell them through the panic notifier
> to not do anything "risky" anymore. Perhaps even stop them from doing
> regular tasks which could become risky in the case of panic. I guess
> this would mean, we should be doing the same in other subsystems too.
Not sure that applies to other subsystems, it's really just history.
fbdev was designed for a world where register banging and some writes
to a gart was all you had to do for graphics, and you can very well do
that from hardirq context. And fbdev will take care of some simple
locking itself if needed. The problem really is that drm drivers are
anything but simple, and there's a bit of room to share more code and
tricks (and fix up even more corner-cases) between drm drivers in the
shared fbdev emulation library.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Monday, June 29, 2015 5:25 PM, Daniel Vetter <[email protected]> wrote:
> As long as the display is up and running we should have a fair stab at
> showing the oops - it's just that no one has seriously bothered with
> the necessary infastructure, automated testing (it won't work
> otherwise) and driver work.
I think testing can be done by injecting a fatal machine check exception
via einj's debugfs interface. I can reproduce the hard hang every time.
I think It can be a simple script or C program do to the automated testing.
If anyone has any patch I'll be happy to help test it out.
Thanks
Rui
On Tue, Jun 30, 2015 at 4:53 AM, Rui Wang <[email protected]> wrote:
> On Monday, June 29, 2015 5:25 PM, Daniel Vetter <[email protected]> wrote:
>> As long as the display is up and running we should have a fair stab at
>> showing the oops - it's just that no one has seriously bothered with
>> the necessary infastructure, automated testing (it won't work
>> otherwise) and driver work.
>
> I think testing can be done by injecting a fatal machine check exception
> via einj's debugfs interface. I can reproduce the hard hang every time.
> I think It can be a simple script or C program do to the automated testing.
> If anyone has any patch I'll be happy to help test it out.
Testing shouldn't kill the machine ;-)
The idea I had is to just exercise the drm panic code (since we'd need
to shunt everything else), and that can be done my calling the
relevant functions from a hardirq context. And hardirq context is
simples to get with a IPI to the local cpu. This way we don't depend
upon the entire panic path to be recoverable, but only upon the drm
bits being sane.
The other thing that needs testing is pushing all the fbdev callbacks
into workers if run from non-process context. But that's something
fbdev likes doing anyway (it's just that most drivers don't need to
overwrite the hooks where this usually happens).
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Tuesday, June 30, 2015 2:37 PM, Daniel Vetter <[email protected]> wrote:
> On Tue, Jun 30, 2015 at 4:53 AM, Rui Wang <[email protected]> wrote:
> >
> > I think testing can be done by injecting a fatal machine check
> > exception via einj's debugfs interface. I can reproduce the hard hang every
> time.
> > I think It can be a simple script or C program do to the automated testing.
> > If anyone has any patch I'll be happy to help test it out.
>
> Testing shouldn't kill the machine ;-)
Yes :) What I assumed was that after applying a future patch the machine should
be able to reboot instead of hanging itself, so the testing can repeat.
>
> The idea I had is to just exercise the drm panic code (since we'd need to
> shunt everything else), and that can be done my calling the relevant
> functions from a hardirq context. And hardirq context is simples to get with a
> IPI to the local cpu. This way we don't depend upon the entire panic path to
> be recoverable, but only upon the drm bits being sane.
Yes If it can be tested without rebooting then it'll be more efficient.
But einj does something more than what an IPI can do, it injects hardware
errors which trigger exceptions in NMI context... and the exception handler
usually panics on fatal errors. And the display may be the only way to catch
what has happened. I'm just hoping that the future version may work in NMI
context.
Thanks
Rui
On Tue, Jun 30, 2015 at 9:23 AM, Rui Wang <[email protected]> wrote:
> On Tuesday, June 30, 2015 2:37 PM, Daniel Vetter <[email protected]> wrote:
>> On Tue, Jun 30, 2015 at 4:53 AM, Rui Wang <[email protected]> wrote:
>> >
>> > I think testing can be done by injecting a fatal machine check
>> > exception via einj's debugfs interface. I can reproduce the hard hang every
>> time.
>> > I think It can be a simple script or C program do to the automated testing.
>> > If anyone has any patch I'll be happy to help test it out.
>>
>> Testing shouldn't kill the machine ;-)
>
> Yes :) What I assumed was that after applying a future patch the machine should
> be able to reboot instead of hanging itself, so the testing can repeat.
>
>>
>> The idea I had is to just exercise the drm panic code (since we'd need to
>> shunt everything else), and that can be done my calling the relevant
>> functions from a hardirq context. And hardirq context is simples to get with a
>> IPI to the local cpu. This way we don't depend upon the entire panic path to
>> be recoverable, but only upon the drm bits being sane.
>
> Yes If it can be tested without rebooting then it'll be more efficient.
> But einj does something more than what an IPI can do, it injects hardware
> errors which trigger exceptions in NMI context... and the exception handler
> usually panics on fatal errors. And the display may be the only way to catch
> what has happened. I'm just hoping that the future version may work in NMI
> context.
NMI sounds ... ambigous ;-) But yeah if we can somehow inject
something as an NMI too then that would be even better. What I want to
avoid is forcing reboots, since that means you can't run a basic
modeset test afterwards to make sure nothing was trampled too badly.
Of course we'd have replace the screen contents, but the important
part is that the panic handler doen't touch anything if the driver is
in modeset code right now (because it'll massively increase the risk
of dying completely), and an easy way to check that it didn't step all
over modeset state unduly is to do a modeset afterwards. If that works
we'll be fine.
Also with that approach we can make sure that no real errors get into
dmesg (as opposed to a real panic), which means we can capture dmesg
afterwards and if there is a seroius log message (or even backtrace)
then drm panic handling has a bug.
All that isn't possible when we force a real panic to happen.
Actually thinking more about NMI that shouldn't be a problem. The
important thing with nmi vs. hardirq is that you can't even reliably
grab an irqsave spinlock, it's trylocks all the way down. But that
also holds for the panic handler, it's trylocks only. Could we somehow
just check that using lockdep - is there an NMI lockdep context
somewhere we could fake-grab? That's another upside of using an IPI
btw: Real panics kill lockdep ;-)
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Tuesday, June 30, 2015 11:24 PM, Daniel Vetter <[email protected]> wrote:
> On Tue, Jun 30, 2015 at 9:23 AM, Rui Wang <[email protected]> wrote:
> > But einj does something more than what an IPI can do, it injects hardware
> > errors which trigger exceptions in NMI context... and the exception handler
> > usually panics on fatal errors. And the display may be the only way to catch
> > what has happened. I'm just hoping that the future version may work in
> > NMI context.
>
> NMI sounds ... ambigous ;-) But yeah if we can somehow inject
> something as an NMI too then that would be even better. What I want to
> avoid is forcing reboots, since that means you can't run a basic
> modeset test afterwards to make sure nothing was trampled too badly.
> Of course we'd have replace the screen contents, but the important
> part is that the panic handler doen't touch anything if the driver is
> in modeset code right now (because it'll massively increase the risk
> of dying completely), and an easy way to check that it didn't step all
> over modeset state unduly is to do a modeset afterwards. If that works
> we'll be fine.
>
> Also with that approach we can make sure that no real errors get into
> dmesg (as opposed to a real panic), which means we can capture dmesg
> afterwards and if there is a seroius log message (or even backtrace)
> then drm panic handling has a bug.
>
> All that isn't possible when we force a real panic to happen.
>
> Actually thinking more about NMI that shouldn't be a problem. The
> important thing with nmi vs. hardirq is that you can't even reliably
> grab an irqsave spinlock, it's trylocks all the way down. But that
> also holds for the panic handler, it's trylocks only. Could we somehow
> just check that using lockdep - is there an NMI lockdep context
> somewhere we could fake-grab? That's another upside of using an IPI
> btw: Real panics kill lockdep ;-)
Einj is supported by ACPI in combination with the hardwre. The injected
errors result in true MCEs, truly non-maskable. Lockdep might not be useful
in this case. Corrected Errors (CEs) don't result in panic but I guess it
might be possible to let it invoke your future mode-setting code for testing
purpose, without rebooting. (Notice that MCEs can happen right from inside
your mode-setting code while accessing any memory address)
But anyway we're not looking for a 100% working solution so if it could only
work in normal irq or ipi context, it'd already be a big plus compared to
what we have now.
Thanks
Rui
On Wed, Jul 1, 2015 at 9:26 AM, Rui Wang <[email protected]> wrote:
> On Tuesday, June 30, 2015 11:24 PM, Daniel Vetter <[email protected]> wrote:
>> On Tue, Jun 30, 2015 at 9:23 AM, Rui Wang <[email protected]> wrote:
>> > But einj does something more than what an IPI can do, it injects hardware
>> > errors which trigger exceptions in NMI context... and the exception handler
>> > usually panics on fatal errors. And the display may be the only way to catch
>> > what has happened. I'm just hoping that the future version may work in
>> > NMI context.
>>
>> NMI sounds ... ambigous ;-) But yeah if we can somehow inject
>> something as an NMI too then that would be even better. What I want to
>> avoid is forcing reboots, since that means you can't run a basic
>> modeset test afterwards to make sure nothing was trampled too badly.
>> Of course we'd have replace the screen contents, but the important
>> part is that the panic handler doen't touch anything if the driver is
>> in modeset code right now (because it'll massively increase the risk
>> of dying completely), and an easy way to check that it didn't step all
>> over modeset state unduly is to do a modeset afterwards. If that works
>> we'll be fine.
>>
>> Also with that approach we can make sure that no real errors get into
>> dmesg (as opposed to a real panic), which means we can capture dmesg
>> afterwards and if there is a seroius log message (or even backtrace)
>> then drm panic handling has a bug.
>>
>> All that isn't possible when we force a real panic to happen.
>>
>> Actually thinking more about NMI that shouldn't be a problem. The
>> important thing with nmi vs. hardirq is that you can't even reliably
>> grab an irqsave spinlock, it's trylocks all the way down. But that
>> also holds for the panic handler, it's trylocks only. Could we somehow
>> just check that using lockdep - is there an NMI lockdep context
>> somewhere we could fake-grab? That's another upside of using an IPI
>> btw: Real panics kill lockdep ;-)
>
> Einj is supported by ACPI in combination with the hardwre. The injected
> errors result in true MCEs, truly non-maskable. Lockdep might not be useful
> in this case. Corrected Errors (CEs) don't result in panic but I guess it
> might be possible to let it invoke your future mode-setting code for testing
> purpose, without rebooting. (Notice that MCEs can happen right from inside
> your mode-setting code while accessing any memory address)
Yeah NMI can happen anywhere and that's about the worst-case panic
context we have. The problem is that NMI bugs are a giant pain to
debug, so for testing I think it'd be better to just have a hardirq
context + the help of lockdep (if possible) to make sure we only do
try_lock and lockless stuff.
> But anyway we're not looking for a 100% working solution so if it could only
> work in normal irq or ipi context, it'd already be a big plus compared to
> what we have now.
NMI vs ipi vs other stuff is just about what's the best debug/testing
strategy. Most of the work there will really be in writing tons of
testcases to race the drm panic handler against drm modeset ioctls.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch