My router/storage box suddenly stopped responding (originally noticed
because dnsmasq wasn't responding) and I had to reboot it. I checked
the systemd journal when it came back and these were the last thing in
there for the previous boot. Any ideas about pinning down the cause?
general protection fault: 0000 [#1] SMP
Modules linked in: zfs(PO) zunicode(PO) zavl(PO) zcommon(PO)
znvpair(PO) spl(O) xt_nat sit tunnel4 ip_tunnel sch_sfq 8021q
ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle
ipt_REJECT xt_tcpudp xt_LOG ip6t_rt xt_limit nf_conntrack_ipv6
nf_conntrack_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc
xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables
ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat
snd_hda_codec_hdmi snd_hda_codec_realtek iTCO_wdt
snd_hda_codec_generic iTCO_vendor_support raid0 raid1 md_mod
x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller
snd_hda_codec microcode r8169 i2c_i801 snd_hwdep acpi_cpufreq
snd_pcm mii snd_timer thermal snd fan soundcore lpc_ich battery
mfd_core evdev processor tun usbip_host(C) usbip_core(C) msr loop
kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif
crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel
libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915
video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp
pps_core ipmi_poweroff ipmi_msghandler button
CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>]
__kmalloc_track_caller+0x86/0x260
RSP: 0018:ffff8802f0abfc88 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff8802f0abfdae RCX: 00000000000016e8
RDX: 00000000000016e7 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff8802f0abfcc0 R08: 00000000000173c0 R09: ffffffff81a8e058
R10: ffff88040e803e00 R11: ffffea00101e4c00 R12: ffff00736b736174
R13: 00000000000000d0 R14: 0000000000000006 R15: ffff88040e803e00
FS: 00007f47b62ac780(0000) GS:ffff88041f2c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f47b6ac3568 CR3: 0000000359fb0000 CR4: 00000000001407e0
Stack:
ffff88040e803e00 ffffffff8123a4f4 ffff8802f0abfdae 0000000000000006
00000000000000d0 00000000000081a4 00000000000001a4 ffff8802f0abfce8
ffffffff8116fb11 ffff8802f0abfdae ffff880406631100 0000000000000002
Call Trace:
[<ffffffff8123a4f4>] ? __kernfs_new_node+0x34/0xf0
[<ffffffff8116fb11>] kstrdup+0x31/0x60
[<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
[<ffffffff8123b386>] kernfs_new_node+0x26/0x50
[<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
[<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
[<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
[<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
[<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
[<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
[<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
[<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
[<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00
49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
RSP <ffff8802f0abfc88>
general protection fault: 0000 [#2] SMP
Modules linked in: zfs(PO) zunicode(PO) zavl(PO) zcommon(PO)
znvpair(PO) spl(O) xt_nat sit tunnel4 ip_tunnel sch_sfq 8021q
ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle
ipt_REJECT xt_tcpudp xt_LOG ip6t_rt xt_limit nf_conntrack_ipv6
nf_conntrack_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc
xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables
ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat
snd_hda_codec_hdmi snd_hda_codec_realtek iTCO_wdt
snd_hda_codec_generic iTCO_vendor_support raid0 raid1 md_mod
x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller
snd_hda_codec microcode r8169 i2c_i801 snd_hwdep acpi_cpufreq
snd_pcm mii snd_timer thermal snd fan soundcore lpc_ich battery
mfd_core evdev processor tun usbip_host(C) usbip_core(C) msr loop
kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif
crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel
libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915
video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp
pps_core ipmi_poweroff ipmi_msghandler button
CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
RSP: 0018:ffff8802f0abf718 EFLAGS: 00010086
RAX: 0000000000000000 RBX: ffff8800373cb3c0 RCX: 00000000000016e8
RDX: 00000000000016e7 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff8802f0abf750 R08: 00000000000173c0 R09: ffff88040e803e00
R10: ffffffff8132d81f R11: 0000000000000000 R12: ffff00736b736174
sit: non-ECT from 95.208.229.33 with TOS=0x3
R13: 0000000000008020 R14: 0000000000000008 R15: ffff88040e803e00
FS: 00007f47b62ac780(0000) GS:ffff88041f2c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f47b6ac3568 CR3: 0000000359fb0000 CR4: 00000000001407e0
Stack:
0000000000000000 ffff8802f0abf828 ffff8800373cb3c0 ffff8802f0abf7b8
ffff88040e8634b0 0000000000000000 0000000000000001 ffff8802f0abf7a0
ffffffff8132d81f ffffffffa00ccc59 ffffffffa00ccc59 0000000000000021
Call Trace:
[<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
[<ffffffff8132db3a>] acpi_ns_get_node+0x79/0xe2
[<ffffffff81336827>] ? acpi_ut_allocate_object_desc_dbg+0x3e/0x6a
[<ffffffff813368c2>] ? acpi_ut_create_internal_object_dbg+0x23/0x87
[<ffffffff8132b531>] acpi_ns_evaluate+0x51/0x24d
[<ffffffff8132b531>] ? acpi_ns_evaluate+0x51/0x24d
[<ffffffff8132e319>] acpi_evaluate_object+0x189/0x285
[<ffffffff8130f0bc>] acpi_execute_simple_method+0x43/0x45
[<ffffffffa00cb63e>] acpi_video_register+0x3c1/0x593 [video]
[<ffffffffa00cb789>] acpi_video_register+0x50c/0x593 [video]
[<ffffffff81302599>] fb_notifier_callback+0x109/0x130
[<ffffffff8150fc7d>] notifier_call_chain+0x4d/0x70
[<ffffffff8108f137>] __blocking_notifier_call_chain+0x47/0x60
[<ffffffff8108f166>] blocking_notifier_call_chain+0x16/0x20
[<ffffffff81302f5b>] fb_notifier_call_chain+0x1b/0x20
[<ffffffff8130350e>] fb_blank+0x9e/0xc0
[<ffffffff812fa6e1>] fbcon_blank+0x1f1/0x300
[<ffffffff810c1044>] ? wake_up_klogd+0x34/0x50
[<ffffffff810c1259>] ? console_unlock+0x1f9/0x3d0
[<ffffffff81073c8b>] ? lock_timer_base.isra.26+0x2b/0x50
[<ffffffff8107219f>] ? internal_add_timer+0x2f/0x70
[<ffffffff81074415>] ? mod_timer+0x105/0x200
[<ffffffff8136d04a>] do_unblank_screen+0xba/0x1f0
[<ffffffff8136d190>] unblank_screen+0x10/0x20
[<ffffffff812ae8b9>] bust_spinlocks+0x19/0x40
[<ffffffff8150cb18>] oops_end+0x38/0x150
[<ffffffff8101639b>] die+0x4b/0x70
[<ffffffff8150c5fa>] do_general_protection+0xca/0x150
[<ffffffff8150bf68>] general_protection+0x28/0x30
[<ffffffff811ad226>] ? __kmalloc_track_caller+0x86/0x260
[<ffffffff811ad351>] ? __kmalloc_track_caller+0x1b1/0x260
[<ffffffff8123a4f4>] ? __kernfs_new_node+0x34/0xf0
[<ffffffff8116fb11>] kstrdup+0x31/0x60
[<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
[<ffffffff8123b386>] kernfs_new_node+0x26/0x50
[<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
[<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
[<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
[<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
[<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
[<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
[<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
[<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
[<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 64 01 00 00
49 83 78 10 00 0f 84 59 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
RIP [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
RSP <ffff8802f0abf718>
---[ end trace d11af9bda628c8d1 ]---
Looks like cgroup related. CCing cgrroup ML.
On Sun 20-07-14 21:27:42, Steven Noonan wrote:
> My router/storage box suddenly stopped responding (originally noticed
> because dnsmasq wasn't responding) and I had to reboot it. I checked
> the systemd journal when it came back and these were the last thing in
> there for the previous boot. Any ideas about pinning down the cause?
>
> general protection fault: 0000 [#1] SMP
> Modules linked in: zfs(PO) zunicode(PO) zavl(PO) zcommon(PO)
> znvpair(PO) spl(O) xt_nat sit tunnel4 ip_tunnel sch_sfq 8021q
> ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle
> ipt_REJECT xt_tcpudp xt_LOG ip6t_rt xt_limit nf_conntrack_ipv6
> nf_conntrack_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc
> xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables
> ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat
> snd_hda_codec_hdmi snd_hda_codec_realtek iTCO_wdt
> snd_hda_codec_generic iTCO_vendor_support raid0 raid1 md_mod
> x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
> glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller
> snd_hda_codec microcode r8169 i2c_i801 snd_hwdep acpi_cpufreq
> snd_pcm mii snd_timer thermal snd fan soundcore lpc_ich battery
> mfd_core evdev processor tun usbip_host(C) usbip_core(C) msr loop
> kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif
> crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel
> libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915
> video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp
> pps_core ipmi_poweroff ipmi_msghandler button
> CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
> Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
> task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
> RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>]
> __kmalloc_track_caller+0x86/0x260
> RSP: 0018:ffff8802f0abfc88 EFLAGS: 00010286
> RAX: 0000000000000000 RBX: ffff8802f0abfdae RCX: 00000000000016e8
> RDX: 00000000000016e7 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffff8802f0abfcc0 R08: 00000000000173c0 R09: ffffffff81a8e058
> R10: ffff88040e803e00 R11: ffffea00101e4c00 R12: ffff00736b736174
> R13: 00000000000000d0 R14: 0000000000000006 R15: ffff88040e803e00
> FS: 00007f47b62ac780(0000) GS:ffff88041f2c0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f47b6ac3568 CR3: 0000000359fb0000 CR4: 00000000001407e0
> Stack:
> ffff88040e803e00 ffffffff8123a4f4 ffff8802f0abfdae 0000000000000006
> 00000000000000d0 00000000000081a4 00000000000001a4 ffff8802f0abfce8
> ffffffff8116fb11 ffff8802f0abfdae ffff880406631100 0000000000000002
> Call Trace:
> [<ffffffff8123a4f4>] ? __kernfs_new_node+0x34/0xf0
> [<ffffffff8116fb11>] kstrdup+0x31/0x60
> [<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
> [<ffffffff8123b386>] kernfs_new_node+0x26/0x50
> [<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
> [<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
> [<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
> [<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
> [<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
> [<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
> [<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
> [<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
> [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
> Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00
> 49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
> RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
> RSP <ffff8802f0abfc88>
> general protection fault: 0000 [#2] SMP
> Modules linked in: zfs(PO) zunicode(PO) zavl(PO) zcommon(PO)
> znvpair(PO) spl(O) xt_nat sit tunnel4 ip_tunnel sch_sfq 8021q
> ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle
> ipt_REJECT xt_tcpudp xt_LOG ip6t_rt xt_limit nf_conntrack_ipv6
> nf_conntrack_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 bridge stp llc
> xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables
> ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat
> snd_hda_codec_hdmi snd_hda_codec_realtek iTCO_wdt
> snd_hda_codec_generic iTCO_vendor_support raid0 raid1 md_mod
> x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
> crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul
> glue_helper ablk_helper cryptd snd_hda_intel snd_hda_controller
> snd_hda_codec microcode r8169 i2c_i801 snd_hwdep acpi_cpufreq
> snd_pcm mii snd_timer thermal snd fan soundcore lpc_ich battery
> mfd_core evdev processor tun usbip_host(C) usbip_core(C) msr loop
> kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif
> crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel
> libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915
> video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp
> pps_core ipmi_poweroff ipmi_msghandler button
> CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
> Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
> task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
> RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
> RSP: 0018:ffff8802f0abf718 EFLAGS: 00010086
> RAX: 0000000000000000 RBX: ffff8800373cb3c0 RCX: 00000000000016e8
> RDX: 00000000000016e7 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffff8802f0abf750 R08: 00000000000173c0 R09: ffff88040e803e00
> R10: ffffffff8132d81f R11: 0000000000000000 R12: ffff00736b736174
> sit: non-ECT from 95.208.229.33 with TOS=0x3
> R13: 0000000000008020 R14: 0000000000000008 R15: ffff88040e803e00
> FS: 00007f47b62ac780(0000) GS:ffff88041f2c0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f47b6ac3568 CR3: 0000000359fb0000 CR4: 00000000001407e0
> Stack:
> 0000000000000000 ffff8802f0abf828 ffff8800373cb3c0 ffff8802f0abf7b8
> ffff88040e8634b0 0000000000000000 0000000000000001 ffff8802f0abf7a0
> ffffffff8132d81f ffffffffa00ccc59 ffffffffa00ccc59 0000000000000021
> Call Trace:
> [<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
> [<ffffffff8132db3a>] acpi_ns_get_node+0x79/0xe2
> [<ffffffff81336827>] ? acpi_ut_allocate_object_desc_dbg+0x3e/0x6a
> [<ffffffff813368c2>] ? acpi_ut_create_internal_object_dbg+0x23/0x87
> [<ffffffff8132b531>] acpi_ns_evaluate+0x51/0x24d
> [<ffffffff8132b531>] ? acpi_ns_evaluate+0x51/0x24d
> [<ffffffff8132e319>] acpi_evaluate_object+0x189/0x285
> [<ffffffff8130f0bc>] acpi_execute_simple_method+0x43/0x45
> [<ffffffffa00cb63e>] acpi_video_register+0x3c1/0x593 [video]
> [<ffffffffa00cb789>] acpi_video_register+0x50c/0x593 [video]
> [<ffffffff81302599>] fb_notifier_callback+0x109/0x130
> [<ffffffff8150fc7d>] notifier_call_chain+0x4d/0x70
> [<ffffffff8108f137>] __blocking_notifier_call_chain+0x47/0x60
> [<ffffffff8108f166>] blocking_notifier_call_chain+0x16/0x20
> [<ffffffff81302f5b>] fb_notifier_call_chain+0x1b/0x20
> [<ffffffff8130350e>] fb_blank+0x9e/0xc0
> [<ffffffff812fa6e1>] fbcon_blank+0x1f1/0x300
> [<ffffffff810c1044>] ? wake_up_klogd+0x34/0x50
> [<ffffffff810c1259>] ? console_unlock+0x1f9/0x3d0
> [<ffffffff81073c8b>] ? lock_timer_base.isra.26+0x2b/0x50
> [<ffffffff8107219f>] ? internal_add_timer+0x2f/0x70
> [<ffffffff81074415>] ? mod_timer+0x105/0x200
> [<ffffffff8136d04a>] do_unblank_screen+0xba/0x1f0
> [<ffffffff8136d190>] unblank_screen+0x10/0x20
> [<ffffffff812ae8b9>] bust_spinlocks+0x19/0x40
> [<ffffffff8150cb18>] oops_end+0x38/0x150
> [<ffffffff8101639b>] die+0x4b/0x70
> [<ffffffff8150c5fa>] do_general_protection+0xca/0x150
> [<ffffffff8150bf68>] general_protection+0x28/0x30
> [<ffffffff811ad226>] ? __kmalloc_track_caller+0x86/0x260
> [<ffffffff811ad351>] ? __kmalloc_track_caller+0x1b1/0x260
> [<ffffffff8123a4f4>] ? __kernfs_new_node+0x34/0xf0
> [<ffffffff8116fb11>] kstrdup+0x31/0x60
> [<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
> [<ffffffff8123b386>] kernfs_new_node+0x26/0x50
> [<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
> [<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
> [<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
> [<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
> [<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
> [<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
> [<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
> [<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
> [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
> Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 64 01 00 00
> 49 83 78 10 00 0f 84 59 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
> RIP [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
> RSP <ffff8802f0abf718>
> ---[ end trace d11af9bda628c8d1 ]---
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Michal Hocko
SUSE Labs
Hello, Steven.
On Sun, Jul 20, 2014 at 09:27:42PM -0700, Steven Noonan wrote:
> My router/storage box suddenly stopped responding (originally noticed
> because dnsmasq wasn't responding) and I had to reboot it. I checked
> the systemd journal when it came back and these were the last thing in
> there for the previous boot. Any ideas about pinning down the cause?
>
> general protection fault: 0000 [#1] SMP
...
> CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
> Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
> task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
> RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>]
> __kmalloc_track_caller+0x86/0x260
So, GFP in kmalloc,
> Call Trace:
> [<ffffffff8116fb11>] kstrdup+0x31/0x60
called from kstrdup()
> [<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
> [<ffffffff8123b386>] kernfs_new_node+0x26/0x50
which was invoked to copy the node name while creating a new kernfs
node.
> [<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
> [<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
> [<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
> [<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
> [<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
> [<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
> [<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
> [<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
> [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
> Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00
> 49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
> RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
> RSP <ffff8802f0abfc88>
followed by another GPF
> general protection fault: 0000 [#2] SMP
...
> RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
in __kmalloc()
> [<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
called from acpi to copy a different name.
I don't think the problem is anything cgroup / kernfs specific. The
allocator is GPFing inside it from multiple callers and it's not even
using a caller-provided cache. It looks like something screwed up the
memory allocator and it's now faulting on unrelated callers. Most
likely illegal free or use-after-free.
Steven, can you please post the full kernel log from boot till reboot?
It usually is a good idea to include full log when reporting bugs as
it's very easy to exclude the actually relevant part.
Thanks.
--
tejun
On Mon, Jul 21, 2014 at 6:29 AM, Tejun Heo <[email protected]> wrote:
> Hello, Steven.
>
> On Sun, Jul 20, 2014 at 09:27:42PM -0700, Steven Noonan wrote:
>> My router/storage box suddenly stopped responding (originally noticed
>> because dnsmasq wasn't responding) and I had to reboot it. I checked
>> the systemd journal when it came back and these were the last thing in
>> there for the previous boot. Any ideas about pinning down the cause?
>>
>> general protection fault: 0000 [#1] SMP
> ...
>> CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
>> Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
>> task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
>> RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>]
>> __kmalloc_track_caller+0x86/0x260
>
> So, GFP in kmalloc,
>
>> Call Trace:
>> [<ffffffff8116fb11>] kstrdup+0x31/0x60
>
> called from kstrdup()
>
>> [<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
>> [<ffffffff8123b386>] kernfs_new_node+0x26/0x50
>
> which was invoked to copy the node name while creating a new kernfs
> node.
>
>> [<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
>> [<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
>> [<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
>> [<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
>> [<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
>> [<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
>> [<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
>> [<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
>> [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
>> Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00
>> 49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
>> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
>> RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
>> RSP <ffff8802f0abfc88>
>
> followed by another GPF
>
>> general protection fault: 0000 [#2] SMP
> ...
>> RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
>
> in __kmalloc()
>
>> [<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
>
> called from acpi to copy a different name.
>
> I don't think the problem is anything cgroup / kernfs specific. The
> allocator is GPFing inside it from multiple callers and it's not even
> using a caller-provided cache. It looks like something screwed up the
> memory allocator and it's now faulting on unrelated callers. Most
> likely illegal free or use-after-free.
>
> Steven, can you please post the full kernel log from boot till reboot?
> It usually is a good idea to include full log when reporting bugs as
> it's very easy to exclude the actually relevant part.
>
I would if I could, but I've had to set up some rather draconian
limits on my systemd journal sizes because of some incessant kernel
messages filling up the logs (related to 6to4 SIT tunnels) -- this has
unfortunately truncated most of the log. Are there any particular
kernel config options I should enable to make tracking this down
easier if it comes up again?
- Steven
(Oops, LKML doesn't like rich text, resending. Was trying to avoid
GMail's bad line wrapping. Going to use Mutt instead.)
I'm starting to wonder if it's bad RAM or something. Just got a couple of
worrying warnings on boot from the same system (after it spontaneously
rebooted, with nothing revealing in the previous boot's logs).
Full kernel message log since boot available here: http://pastebin.com/ZzJjK862
Jul 23 09:43:20 orcus kernel: ------------[ cut here ]------------
Jul 23 09:43:20 orcus kernel: WARNING: CPU: 4 PID: 270 at include/linux/kref.h:47 kobject_get+0x46/0x60()
Jul 23 09:43:20 orcus kernel: Modules linked in: snd_hda_intel snd_hda_controller microcode(+) i2c_i801 r8169(+) snd_hda_codec snd_hwdep mii snd_pcm snd_timer thermal(+) fan snd acpi_cpufreq(+) battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
Jul 23 09:43:20 orcus kernel: CPU: 4 PID: 270 Comm: systemd-udevd Tainted: P C O 3.15.6-1-ec2 #1
Jul 23 09:43:20 orcus kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
Jul 23 09:43:20 orcus kernel: 0000000000000009 ffff8804067b7858 ffffffff81505dd6 0000000000000000
Jul 23 09:43:20 orcus kernel: ffff8804067b7890 ffffffff81066a3d ffff880405d80c00 0000000000000202
Jul 23 09:43:20 orcus kernel: ffffffffffffffff ffff880406076c00 ffff880402b15000 ffff8804067b78a0
Jul 23 09:43:20 orcus kernel: Call Trace:
Jul 23 09:43:20 orcus kernel: [<ffffffff81505dd6>] dump_stack+0x45/0x56
Jul 23 09:43:20 orcus kernel: [<ffffffff81066a3d>] warn_slowpath_common+0x7d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff81066b1a>] warn_slowpath_null+0x1a/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3d26>] kobject_get+0x46/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff813d0717>] cpufreq_cpu_get+0x77/0xc0
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1909>] cpufreq_get_policy+0x19/0x70
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c059b>] acpi_processor_power_exit+0x164/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3116>] ? idr_get_empty_slot+0x1a6/0x3d0
Jul 23 09:43:20 orcus kernel: [<ffffffff812a2602>] ? get_from_free_list+0x42/0x50
Jul 23 09:43:20 orcus kernel: [<ffffffff812a369d>] ? ida_get_new_above+0x1ed/0x220
Jul 23 09:43:20 orcus kernel: [<ffffffff812a2d74>] ? ida_pre_get+0x54/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3755>] ? ida_simple_get+0x85/0x100
Jul 23 09:43:20 orcus kernel: [<ffffffff8123a818>] ? kernfs_next_descendant_post+0x48/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810b4637>] ? down_timeout+0x37/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c06ed>] acpi_processor_power_exit+0x2b6/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c0739>] acpi_processor_power_exit+0x302/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813cc95e>] thermal_zone_bind_cooling_device+0xce/0x370
Jul 23 09:43:20 orcus kernel: [<ffffffff8130eb63>] ? acpi_os_signal_semaphore+0x21/0x2d
Jul 23 09:43:20 orcus kernel: [<ffffffff8133665b>] ? acpi_ut_release_mutex+0x5f/0x63
Jul 23 09:43:20 orcus kernel: [<ffffffff8132e16b>] ? acpi_get_data_full+0x7d/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffffa085addb>] 0xffffffffa085adda
Jul 23 09:43:20 orcus kernel: [<ffffffff8138d1c6>] ? device_create_file+0x46/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085af22>] 0xffffffffa085af21
Jul 23 09:43:20 orcus kernel: [<ffffffff813ce949>] thermal_zone_device_register+0x669/0x8b0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085b164>] 0xffffffffa085b163
Jul 23 09:43:20 orcus kernel: [<ffffffff8123db21>] ? sysfs_do_create_link_sd.isra.2+0x71/0xe0
Jul 23 09:43:20 orcus kernel: [<ffffffff8131217d>] acpi_device_probe+0x43/0xe9
Jul 23 09:43:20 orcus kernel: [<ffffffff81391bee>] driver_probe_device+0x8e/0x270
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e9b>] __driver_attach+0x8b/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e10>] ? __device_attach+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff8138fd2b>] bus_for_each_dev+0x6b/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffff813917fe>] driver_attach+0x1e/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff81391328>] bus_add_driver+0x178/0x230
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81392574>] driver_register+0x64/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81312a28>] acpi_bus_register_driver+0x40/0x42
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d067>] init_module+0x67/0x81 [thermal]
Jul 23 09:43:20 orcus kernel: [<ffffffff8100216a>] do_one_initcall+0xfa/0x160
Jul 23 09:43:20 orcus kernel: [<ffffffff8108f142>] ? __blocking_notifier_call_chain+0x52/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810e3f51>] load_module+0x1a11/0x2300
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1260>] ? store_uevent+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1eb1>] ? copy_module_from_fd.isra.39+0x111/0x170
Jul 23 09:43:20 orcus kernel: [<ffffffff810e497e>] SyS_finit_module+0x7e/0x80
Jul 23 09:43:20 orcus kernel: [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Jul 23 09:43:20 orcus kernel: ---[ end trace 71a1e508f45dbd18 ]---
Jul 23 09:43:20 orcus kernel: ------------[ cut here ]------------
Jul 23 09:43:20 orcus kernel: WARNING: CPU: 4 PID: 270 at lib/kobject.c:670 kobject_put+0x58/0x60()
Jul 23 09:43:20 orcus kernel: kobject: '(null)' (ffff880405d80cb0): is not initialized, yet kobject_put() is being called.
Jul 23 09:43:20 orcus kernel: Modules linked in: snd_hda_intel snd_hda_controller microcode(+) i2c_i801 r8169(+) snd_hda_codec snd_hwdep mii snd_pcm snd_timer thermal(+) fan snd acpi_cpufreq(+) battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
Jul 23 09:43:20 orcus kernel: CPU: 4 PID: 270 Comm: systemd-udevd Tainted: P WC O 3.15.6-1-ec2 #1
Jul 23 09:43:20 orcus kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
Jul 23 09:43:20 orcus kernel: 0000000000000009 ffff8804067b7818 ffffffff81505dd6 ffff8804067b7860
Jul 23 09:43:20 orcus kernel: ffff8804067b7850 ffffffff81066a3d ffff880405d80cb0 ffffffffffffffff
Jul 23 09:43:20 orcus kernel: ffffffffffffffff ffff880406076c00 ffff880402b15000 ffff8804067b78b0
Jul 23 09:43:20 orcus kernel: Call Trace:
Jul 23 09:43:20 orcus kernel: [<ffffffff81505dd6>] dump_stack+0x45/0x56
Jul 23 09:43:20 orcus kernel: [<ffffffff81066a3d>] warn_slowpath_common+0x7d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff81066aac>] warn_slowpath_fmt+0x4c/0x50
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3e18>] kobject_put+0x58/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff813d0780>] cpufreq_cpu_put+0x20/0x30
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1949>] cpufreq_get_policy+0x59/0x70
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c059b>] acpi_processor_power_exit+0x164/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1cd0>] ? cpufreq_update_policy+0x140/0x140
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c06ed>] acpi_processor_power_exit+0x2b6/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c0739>] acpi_processor_power_exit+0x302/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813cc95e>] thermal_zone_bind_cooling_device+0xce/0x370
Jul 23 09:43:20 orcus kernel: [<ffffffff8130eb63>] ? acpi_os_signal_semaphore+0x21/0x2d
Jul 23 09:43:20 orcus kernel: [<ffffffff8133665b>] ? acpi_ut_release_mutex+0x5f/0x63
Jul 23 09:43:20 orcus kernel: [<ffffffff8132e16b>] ? acpi_get_data_full+0x7d/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffffa085addb>] 0xffffffffa085adda
Jul 23 09:43:20 orcus kernel: [<ffffffff8138d1c6>] ? device_create_file+0x46/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085af22>] 0xffffffffa085af21
Jul 23 09:43:20 orcus kernel: [<ffffffff813ce949>] thermal_zone_device_register+0x669/0x8b0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085b164>] 0xffffffffa085b163
Jul 23 09:43:20 orcus kernel: [<ffffffff8123db21>] ? sysfs_do_create_link_sd.isra.2+0x71/0xe0
Jul 23 09:43:20 orcus kernel: [<ffffffff8131217d>] acpi_device_probe+0x43/0xe9
Jul 23 09:43:20 orcus kernel: [<ffffffff81391bee>] driver_probe_device+0x8e/0x270
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e9b>] __driver_attach+0x8b/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e10>] ? __device_attach+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff8138fd2b>] bus_for_each_dev+0x6b/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffff813917fe>] driver_attach+0x1e/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff81391328>] bus_add_driver+0x178/0x230
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81392574>] driver_register+0x64/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81312a28>] acpi_bus_register_driver+0x40/0x42
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d067>] init_module+0x67/0x81 [thermal]
Jul 23 09:43:20 orcus kernel: [<ffffffff8100216a>] do_one_initcall+0xfa/0x160
Jul 23 09:43:20 orcus kernel: [<ffffffff8108f142>] ? __blocking_notifier_call_chain+0x52/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810e3f51>] load_module+0x1a11/0x2300
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1260>] ? store_uevent+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1eb1>] ? copy_module_from_fd.isra.39+0x111/0x170
Jul 23 09:43:20 orcus kernel: [<ffffffff810e497e>] SyS_finit_module+0x7e/0x80
Jul 23 09:43:20 orcus kernel: [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Jul 23 09:43:20 orcus kernel: ---[ end trace 71a1e508f45dbd19 ]---
Jul 23 09:43:20 orcus kernel: ------------[ cut here ]------------
Jul 23 09:43:20 orcus kernel: WARNING: CPU: 4 PID: 270 at lib/kobject.c:670 kobject_put+0x58/0x60()
Jul 23 09:43:20 orcus kernel: kobject: '(null)' (ffff880405d80cb0): is not initialized, yet kobject_put() is being called.
Jul 23 09:43:20 orcus kernel: Modules linked in: snd_hda_intel snd_hda_controller microcode(+) i2c_i801 r8169(+) snd_hda_codec snd_hwdep mii snd_pcm snd_timer thermal(+) fan snd acpi_cpufreq(+) battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
Jul 23 09:43:20 orcus kernel: CPU: 4 PID: 270 Comm: systemd-udevd Tainted: P WC O 3.15.6-1-ec2 #1
Jul 23 09:43:20 orcus kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
Jul 23 09:43:20 orcus kernel: 0000000000000009 ffff8804067b7808 ffffffff81505dd6 ffff8804067b7850
Jul 23 09:43:20 orcus kernel: ffff8804067b7840 ffffffff81066a3d ffff880405d80cb0 ffff880408619400
Jul 23 09:43:20 orcus kernel: 0000000000000001 0000000000000007 ffff880402b15318 ffff8804067b78a0
Jul 23 09:43:20 orcus kernel: Call Trace:
Jul 23 09:43:20 orcus kernel: [<ffffffff81505dd6>] dump_stack+0x45/0x56
Jul 23 09:43:20 orcus kernel: [<ffffffff81066a3d>] warn_slowpath_common+0x7d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff81066aac>] warn_slowpath_fmt+0x4c/0x50
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3e18>] kobject_put+0x58/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff813d0780>] cpufreq_cpu_put+0x20/0x30
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1949>] cpufreq_get_policy+0x59/0x70
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c059b>] acpi_processor_power_exit+0x164/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1cd0>] ? cpufreq_update_policy+0x140/0x140
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c05e2>] acpi_processor_power_exit+0x1ab/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813cf3fc>] thermal_zone_trip_update+0xcc/0x260
Jul 23 09:43:20 orcus kernel: [<ffffffff813cf5a6>] step_wise_throttle+0x16/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff813cdeec>] handle_thermal_trip+0x4c/0x150
Jul 23 09:43:20 orcus kernel: [<ffffffff813ce06d>] thermal_zone_device_update+0x7d/0xc0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cea8d>] thermal_zone_device_register+0x7ad/0x8b0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085b164>] 0xffffffffa085b163
Jul 23 09:43:20 orcus kernel: [<ffffffff8123db21>] ? sysfs_do_create_link_sd.isra.2+0x71/0xe0
Jul 23 09:43:20 orcus kernel: [<ffffffff8131217d>] acpi_device_probe+0x43/0xe9
Jul 23 09:43:20 orcus kernel: [<ffffffff81391bee>] driver_probe_device+0x8e/0x270
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e9b>] __driver_attach+0x8b/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e10>] ? __device_attach+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff8138fd2b>] bus_for_each_dev+0x6b/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffff813917fe>] driver_attach+0x1e/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff81391328>] bus_add_driver+0x178/0x230
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81392574>] driver_register+0x64/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81312a28>] acpi_bus_register_driver+0x40/0x42
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d067>] init_module+0x67/0x81 [thermal]
Jul 23 09:43:20 orcus kernel: [<ffffffff8100216a>] do_one_initcall+0xfa/0x160
Jul 23 09:43:20 orcus kernel: [<ffffffff8108f142>] ? __blocking_notifier_call_chain+0x52/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810e3f51>] load_module+0x1a11/0x2300
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1260>] ? store_uevent+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1eb1>] ? copy_module_from_fd.isra.39+0x111/0x170
Jul 23 09:43:20 orcus kernel: [<ffffffff810e497e>] SyS_finit_module+0x7e/0x80
Jul 23 09:43:20 orcus kernel: [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Jul 23 09:43:20 orcus kernel: ---[ end trace 71a1e508f45dbd1a ]---
Jul 23 09:43:20 orcus kernel: ------------[ cut here ]------------
Jul 23 09:43:20 orcus kernel: WARNING: CPU: 4 PID: 270 at lib/kobject.c:670 kobject_put+0x58/0x60()
Jul 23 09:43:20 orcus kernel: kobject: '(null)' (ffff880405d80cb0): is not initialized, yet kobject_put() is being called.
Jul 23 09:43:20 orcus kernel: Modules linked in: snd_hda_intel snd_hda_controller microcode(+) i2c_i801 r8169(+) snd_hda_codec snd_hwdep mii snd_pcm snd_timer thermal(+) fan snd acpi_cpufreq(+) battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
Jul 23 09:43:20 orcus kernel: CPU: 4 PID: 270 Comm: systemd-udevd Tainted: P WC O 3.15.6-1-ec2 #1
Jul 23 09:43:20 orcus kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
Jul 23 09:43:20 orcus kernel: 0000000000000009 ffff8804067b7850 ffffffff81505dd6 ffff8804067b7898
Jul 23 09:43:20 orcus kernel: ffff8804067b7888 ffffffff81066a3d ffff880405d80cb0 ffff880406076c00
Jul 23 09:43:20 orcus kernel: 0000000000000000 ffff880402b15018 0000000000000000 ffff8804067b78e8
Jul 23 09:43:20 orcus kernel: Call Trace:
Jul 23 09:43:20 orcus kernel: [<ffffffff81505dd6>] dump_stack+0x45/0x56
Jul 23 09:43:20 orcus kernel: [<ffffffff81066a3d>] warn_slowpath_common+0x7d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff81066aac>] warn_slowpath_fmt+0x4c/0x50
Jul 23 09:43:20 orcus kernel: [<ffffffff813d078c>] ? cpufreq_cpu_put+0x2c/0x30
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3e18>] kobject_put+0x58/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff813d0780>] cpufreq_cpu_put+0x20/0x30
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1949>] cpufreq_get_policy+0x59/0x70
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c059b>] acpi_processor_power_exit+0x164/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1cd0>] ? cpufreq_update_policy+0x140/0x140
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c077f>] acpi_processor_power_exit+0x348/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813cc34d>] thermal_cdev_update+0x8d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cf5e9>] step_wise_throttle+0x59/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff813cdeec>] handle_thermal_trip+0x4c/0x150
Jul 23 09:43:20 orcus kernel: [<ffffffff813ce06d>] thermal_zone_device_update+0x7d/0xc0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cea8d>] thermal_zone_device_register+0x7ad/0x8b0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085b164>] 0xffffffffa085b163
Jul 23 09:43:20 orcus kernel: [<ffffffff8123db21>] ? sysfs_do_create_link_sd.isra.2+0x71/0xe0
Jul 23 09:43:20 orcus kernel: [<ffffffff8131217d>] acpi_device_probe+0x43/0xe9
Jul 23 09:43:20 orcus kernel: [<ffffffff81391bee>] driver_probe_device+0x8e/0x270
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e9b>] __driver_attach+0x8b/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e10>] ? __device_attach+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff8138fd2b>] bus_for_each_dev+0x6b/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffff813917fe>] driver_attach+0x1e/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff81391328>] bus_add_driver+0x178/0x230
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81392574>] driver_register+0x64/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81312a28>] acpi_bus_register_driver+0x40/0x42
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d067>] init_module+0x67/0x81 [thermal]
Jul 23 09:43:20 orcus kernel: [<ffffffff8100216a>] do_one_initcall+0xfa/0x160
Jul 23 09:43:20 orcus kernel: [<ffffffff8108f142>] ? __blocking_notifier_call_chain+0x52/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810e3f51>] load_module+0x1a11/0x2300
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1260>] ? store_uevent+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1eb1>] ? copy_module_from_fd.isra.39+0x111/0x170
Jul 23 09:43:20 orcus kernel: [<ffffffff810e497e>] SyS_finit_module+0x7e/0x80
Jul 23 09:43:20 orcus kernel: [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Jul 23 09:43:20 orcus kernel: ---[ end trace 71a1e508f45dbd1b ]---
Jul 23 09:43:20 orcus kernel: ------------[ cut here ]------------
Jul 23 09:43:20 orcus kernel: WARNING: CPU: 4 PID: 270 at lib/kobject.c:670 kobject_put+0x58/0x60()
Jul 23 09:43:20 orcus kernel: kobject: '(null)' (ffff880405d80cb0): is not initialized, yet kobject_put() is being called.
Jul 23 09:43:20 orcus kernel: Modules linked in: snd_hda_intel snd_hda_controller microcode(+) i2c_i801 r8169(+) snd_hda_codec snd_hwdep mii snd_pcm snd_timer thermal(+) fan snd acpi_cpufreq(+) battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
Jul 23 09:43:20 orcus kernel: CPU: 4 PID: 270 Comm: systemd-udevd Tainted: P WC O 3.15.6-1-ec2 #1
Jul 23 09:43:20 orcus kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
Jul 23 09:43:20 orcus kernel: 0000000000000009 ffff8804067b7838 ffffffff81505dd6 ffff8804067b7880
Jul 23 09:43:20 orcus kernel: ffff8804067b7870 ffffffff81066a3d ffff880405d80cb0 0000000000000003
Jul 23 09:43:20 orcus kernel: 0000000000000000 ffff880402b15018 0000000000000000 ffff8804067b78d0
Jul 23 09:43:20 orcus kernel: Call Trace:
Jul 23 09:43:20 orcus kernel: [<ffffffff81505dd6>] dump_stack+0x45/0x56
Jul 23 09:43:20 orcus kernel: [<ffffffff81066a3d>] warn_slowpath_common+0x7d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff81066aac>] warn_slowpath_fmt+0x4c/0x50
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3e18>] kobject_put+0x58/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff813d0780>] cpufreq_cpu_put+0x20/0x30
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1949>] cpufreq_get_policy+0x59/0x70
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c059b>] acpi_processor_power_exit+0x164/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1cd0>] ? cpufreq_update_policy+0x140/0x140
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c06ed>] acpi_processor_power_exit+0x2b6/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c0795>] acpi_processor_power_exit+0x35e/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813cc34d>] thermal_cdev_update+0x8d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cf5e9>] step_wise_throttle+0x59/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff813cdeec>] handle_thermal_trip+0x4c/0x150
Jul 23 09:43:20 orcus kernel: [<ffffffff813ce06d>] thermal_zone_device_update+0x7d/0xc0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cea8d>] thermal_zone_device_register+0x7ad/0x8b0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085b164>] 0xffffffffa085b163
Jul 23 09:43:20 orcus kernel: [<ffffffff8123db21>] ? sysfs_do_create_link_sd.isra.2+0x71/0xe0
Jul 23 09:43:20 orcus kernel: [<ffffffff8131217d>] acpi_device_probe+0x43/0xe9
Jul 23 09:43:20 orcus kernel: [<ffffffff81391bee>] driver_probe_device+0x8e/0x270
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e9b>] __driver_attach+0x8b/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e10>] ? __device_attach+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff8138fd2b>] bus_for_each_dev+0x6b/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffff813917fe>] driver_attach+0x1e/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff81391328>] bus_add_driver+0x178/0x230
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81392574>] driver_register+0x64/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81312a28>] acpi_bus_register_driver+0x40/0x42
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d067>] init_module+0x67/0x81 [thermal]
Jul 23 09:43:20 orcus kernel: [<ffffffff8100216a>] do_one_initcall+0xfa/0x160
Jul 23 09:43:20 orcus kernel: [<ffffffff8108f142>] ? __blocking_notifier_call_chain+0x52/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810e3f51>] load_module+0x1a11/0x2300
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1260>] ? store_uevent+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1eb1>] ? copy_module_from_fd.isra.39+0x111/0x170
Jul 23 09:43:20 orcus kernel: [<ffffffff810e497e>] SyS_finit_module+0x7e/0x80
Jul 23 09:43:20 orcus kernel: [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Jul 23 09:43:20 orcus kernel: ---[ end trace 71a1e508f45dbd1c ]---
Jul 23 09:43:20 orcus kernel: ------------[ cut here ]------------
Jul 23 09:43:20 orcus kernel: WARNING: CPU: 4 PID: 270 at lib/kobject.c:670 kobject_put+0x58/0x60()
Jul 23 09:43:20 orcus kernel: kobject: '(null)' (ffff880405d80cb0): is not initialized, yet kobject_put() is being called.
Jul 23 09:43:20 orcus kernel: Modules linked in: snd_hda_intel snd_hda_controller microcode(+) i2c_i801 r8169(+) snd_hda_codec snd_hwdep mii snd_pcm snd_timer thermal(+) fan snd acpi_cpufreq(+) battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
Jul 23 09:43:20 orcus kernel: CPU: 4 PID: 270 Comm: systemd-udevd Tainted: P WC O 3.15.6-1-ec2 #1
Jul 23 09:43:20 orcus kernel: Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
Jul 23 09:43:20 orcus kernel: 0000000000000009 ffff8804067b7820 ffffffff81505dd6 ffff8804067b7868
Jul 23 09:43:20 orcus kernel: ffff8804067b7858 ffffffff81066a3d ffff880405d80cb0 0000000000000000
Jul 23 09:43:20 orcus kernel: 0000000000000007 ffff880402b15018 0000000000000000 ffff8804067b78b8
Jul 23 09:43:20 orcus kernel: Call Trace:
Jul 23 09:43:20 orcus kernel: [<ffffffff81505dd6>] dump_stack+0x45/0x56
Jul 23 09:43:20 orcus kernel: [<ffffffff81066a3d>] warn_slowpath_common+0x7d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff81066aac>] warn_slowpath_fmt+0x4c/0x50
Jul 23 09:43:20 orcus kernel: [<ffffffff812a3e18>] kobject_put+0x58/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff813d0780>] cpufreq_cpu_put+0x20/0x30
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1949>] cpufreq_get_policy+0x59/0x70
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c059b>] acpi_processor_power_exit+0x164/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813d1cd0>] ? cpufreq_update_policy+0x140/0x140
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c0645>] acpi_processor_power_exit+0x20e/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffffa07c07d0>] acpi_processor_power_exit+0x399/0x3c6 [processor]
Jul 23 09:43:20 orcus kernel: [<ffffffff813cc34d>] thermal_cdev_update+0x8d/0xa0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cf5e9>] step_wise_throttle+0x59/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff813cdeec>] handle_thermal_trip+0x4c/0x150
Jul 23 09:43:20 orcus kernel: [<ffffffff813ce06d>] thermal_zone_device_update+0x7d/0xc0
Jul 23 09:43:20 orcus kernel: [<ffffffff813cea8d>] thermal_zone_device_register+0x7ad/0x8b0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085b164>] 0xffffffffa085b163
Jul 23 09:43:20 orcus kernel: [<ffffffff8123db21>] ? sysfs_do_create_link_sd.isra.2+0x71/0xe0
Jul 23 09:43:20 orcus kernel: [<ffffffff8131217d>] acpi_device_probe+0x43/0xe9
Jul 23 09:43:20 orcus kernel: [<ffffffff81391bee>] driver_probe_device+0x8e/0x270
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e9b>] __driver_attach+0x8b/0x90
Jul 23 09:43:20 orcus kernel: [<ffffffff81391e10>] ? __device_attach+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff8138fd2b>] bus_for_each_dev+0x6b/0xb0
Jul 23 09:43:20 orcus kernel: [<ffffffff813917fe>] driver_attach+0x1e/0x20
Jul 23 09:43:20 orcus kernel: [<ffffffff81391328>] bus_add_driver+0x178/0x230
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81392574>] driver_register+0x64/0xf0
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d000>] ? 0xffffffffa085cfff
Jul 23 09:43:20 orcus kernel: [<ffffffff81312a28>] acpi_bus_register_driver+0x40/0x42
Jul 23 09:43:20 orcus kernel: [<ffffffffa085d067>] init_module+0x67/0x81 [thermal]
Jul 23 09:43:20 orcus kernel: [<ffffffff8100216a>] do_one_initcall+0xfa/0x160
Jul 23 09:43:20 orcus kernel: [<ffffffff8108f142>] ? __blocking_notifier_call_chain+0x52/0x60
Jul 23 09:43:20 orcus kernel: [<ffffffff810e3f51>] load_module+0x1a11/0x2300
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1260>] ? store_uevent+0x40/0x40
Jul 23 09:43:20 orcus kernel: [<ffffffff810e1eb1>] ? copy_module_from_fd.isra.39+0x111/0x170
Jul 23 09:43:20 orcus kernel: [<ffffffff810e497e>] SyS_finit_module+0x7e/0x80
Jul 23 09:43:20 orcus kernel: [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
Jul 23 09:43:20 orcus kernel: ---[ end trace 71a1e508f45dbd1d ]---
On Mon, Jul 21, 2014 at 10:41:45AM -0700, Steven Noonan wrote:
> On Mon, Jul 21, 2014 at 6:29 AM, Tejun Heo <[email protected]> wrote:
> > Hello, Steven.
> >
> > On Sun, Jul 20, 2014 at 09:27:42PM -0700, Steven Noonan wrote:
> >> My router/storage box suddenly stopped responding (originally noticed
> >> because dnsmasq wasn't responding) and I had to reboot it. I checked
> >> the systemd journal when it came back and these were the last thing in
> >> there for the previous boot. Any ideas about pinning down the cause?
> >>
> >> general protection fault: 0000 [#1] SMP
> > ...
> >> CPU: 3 PID: 8881 Comm: systemd Tainted: P WC O 3.15.6 #1
> >> Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
> >> task: ffff8802f473d880 ti: ffff8802f0abc000 task.ti: ffff8802f0abc000
> >> RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>]
> >> __kmalloc_track_caller+0x86/0x260
> >
> > So, GFP in kmalloc,
> >
> >> Call Trace:
> >> [<ffffffff8116fb11>] kstrdup+0x31/0x60
> >
> > called from kstrdup()
> >
> >> [<ffffffff8123a4f4>] __kernfs_new_node+0x34/0xf0
> >> [<ffffffff8123b386>] kernfs_new_node+0x26/0x50
> >
> > which was invoked to copy the node name while creating a new kernfs
> > node.
> >
> >> [<ffffffff8123cc59>] __kernfs_create_file+0x39/0xa0
> >> [<ffffffff810edb60>] cgroup_addrm_files+0x110/0x250
> >> [<ffffffff810ee9ab>] cgroup_mkdir+0x21b/0x540
> >> [<ffffffff8125ca36>] ? security_inode_notifysecctx+0x16/0x20
> >> [<ffffffff8123b30a>] kernfs_iop_mkdir+0x5a/0x90
> >> [<ffffffff811d3120>] vfs_mkdir+0xe0/0x180
> >> [<ffffffff811d7bea>] SyS_mkdirat+0xaa/0xe0
> >> [<ffffffff811d7c39>] SyS_mkdir+0x19/0x20
> >> [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
> >> Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00
> >> 49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49>
> >> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
> >> RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
> >> RSP <ffff8802f0abfc88>
> >
> > followed by another GPF
> >
> >> general protection fault: 0000 [#2] SMP
> > ...
> >> RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
> >
> > in __kmalloc()
> >
> >> [<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
> >
> > called from acpi to copy a different name.
> >
> > I don't think the problem is anything cgroup / kernfs specific. The
> > allocator is GPFing inside it from multiple callers and it's not even
> > using a caller-provided cache. It looks like something screwed up the
> > memory allocator and it's now faulting on unrelated callers. Most
> > likely illegal free or use-after-free.
> >
> > Steven, can you please post the full kernel log from boot till reboot?
> > It usually is a good idea to include full log when reporting bugs as
> > it's very easy to exclude the actually relevant part.
> >
>
> I would if I could, but I've had to set up some rather draconian
> limits on my systemd journal sizes because of some incessant kernel
> messages filling up the logs (related to 6to4 SIT tunnels) -- this has
> unfortunately truncated most of the log. Are there any particular
> kernel config options I should enable to make tracking this down
> easier if it comes up again?
>
> - Steven
Am 23.07.2014 19:50, schrieb Steven Noonan:
> (Oops, LKML doesn't like rich text, resending. Was trying to avoid
> GMail's bad line wrapping. Going to use Mutt instead.)
>
> I'm starting to wonder if it's bad RAM or something. Just got a couple of
> worrying warnings on boot from the same system (after it spontaneously
> rebooted, with nothing revealing in the previous boot's logs).
I once had such too and since then I'm using memtest=3 in my kernel
command line on x86* machines. Depending on the amount of RAM it will
slow down boot by a few seconds, but if you don't care if your machine
comes up in 5 or 10 seconds, it is a no-brainer.
Regards,
Alexander Holler
On Thu, Jul 24, 2014 at 12:06 AM, Alexander Holler <[email protected]> wrote:
> Am 23.07.2014 19:50, schrieb Steven Noonan:
>
>> (Oops, LKML doesn't like rich text, resending. Was trying to avoid
>> GMail's bad line wrapping. Going to use Mutt instead.)
>>
>> I'm starting to wonder if it's bad RAM or something. Just got a couple of
>> worrying warnings on boot from the same system (after it spontaneously
>> rebooted, with nothing revealing in the previous boot's logs).
So the spontaneous reboot was apparently caused by a power outage. All
my boxes had identical uptimes of less than a couple days when I checked
them.
>
>
> I once had such too and since then I'm using memtest=3 in my kernel command
> line on x86* machines. Depending on the amount of RAM it will slow down boot
> by a few seconds, but if you don't care if your machine comes up in 5 or 10
> seconds, it is a no-brainer.
>
However, I got another general protection fault. This time it happened
when doing 'find' on an NFS mount point. Tried booting with 'memtest=16'
to see if that would catch anything, but it passed without finding any
bad regions. I'm running memtest86 right now to be a bit more thorough
and ensure it's not just bad hardware, but so far it's not found
anything (1 full pass done so far).
Here's the latest backtraces. I only managed to copy/paste this before
the system hung and I had to reboot it, but there should be a more
complete kernel log in the systemd journal that I can grab once it's
done with memtest86.
[212326.408380] general protection fault: 0000 [#1] SMP
[212326.409183] Modules linked in: rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd fscache sunrpc macvlan xt_nat sit tunnel4 ip_tunnel sch_sfq ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT xt_limit 8021q nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_tcpudp bridge ip6t_rt nf_conntrack_ipv6 stp llc nf_defrag_ipv6 xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat x86_pkg_temp_thermal iTCO_wdt intel_powerclamp raid1 iTCO_vendor_support raid0 coretemp crct10dif_pclmul md_mod snd_hda_codec_hdmi crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul snd_hda_codec_realtek glue_helper ablk_helper cryptd snd_hda_codec_generic snd_hda_intel snd_hda_controller microcode i2c_i801 r8169 snd_hda_codec
[212326.411879] snd_hwdep mii snd_pcm snd_timer thermal fan snd acpi_cpufreq battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
[212326.414577] CPU: 5 PID: 30360 Comm: find Tainted: P WC O 3.15.6-1-ec2 #1
[212326.415457] Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
[212326.416352] task: ffff8801275bbb00 ti: ffff88030f80c000 task.ti: ffff88030f80c000
[212326.417261] RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
[212326.418194] RSP: 0018:ffff88030f80fb78 EFLAGS: 00010282
[212326.419130] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 00000000000035ee
[212326.420081] RDX: 00000000000035ed RSI: 0000000000000000 RDI: 0000000000000000
[212326.421021] RBP: ffff88030f80fbb0 R08: 00000000000173c0 R09: ffff8801eb6ae160
[212326.421958] R10: ffff88040e803e00 R11: 0000000000000004 R12: ff0074726f707262
[212326.422887] R13: 00000000000000d0 R14: 0000000000000004 R15: ffff88040e803e00
[212326.423808] FS: 00007f3b98919700(0000) GS:ffff88041f340000(0000) knlGS:0000000000000000
[212326.424752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[212326.425698] CR2: 0000000000ef0010 CR3: 00000003ffd3c000 CR4: 00000000001407e0
[212326.426659] Stack:
[212326.427620] ffff88040e803e00 ffffffffa0211d75 0000000000000004 ffff8803607f0558
[212326.428609] 0000000000000009 ffff8801eb6ae000 ffff8801eb6ae140 ffff88030f80fbd0
[212326.429630] ffffffff8116fb60 ffff88030f80fd40 ffff88030f80fe58 ffff88030f80fcc8
[212326.430640] Call Trace:
[212326.431651] [<ffffffffa0211d75>] ? nfs_permission+0x405/0xfb0 [nfs]
[212326.432681] [<ffffffff8116fb60>] kmemdup+0x20/0x50
[212326.433717] [<ffffffffa0211d75>] nfs_permission+0x405/0xfb0 [nfs]
[212326.434760] [<ffffffffa0212277>] nfs_permission+0x907/0xfb0 [nfs]
[212326.435810] [<ffffffffa0212350>] ? nfs_permission+0x9e0/0xfb0 [nfs]
[212326.436863] [<ffffffffa0212372>] nfs_permission+0xa02/0xfb0 [nfs]
[212326.437924] [<ffffffff8115300e>] do_read_cache_page+0x7e/0x1a0
[212326.438990] [<ffffffff8115314c>] read_cache_page+0x1c/0x20
[212326.440078] [<ffffffffa021252b>] nfs_permission+0xbbb/0xfb0 [nfs]
[212326.441159] [<ffffffffa0787690>] ? nfs4_proc_secinfo+0x63a0/0x63a0 [nfsv4]
[212326.442251] [<ffffffff811d9f16>] iterate_dir+0xa6/0xe0
[212326.443347] [<ffffffff811da359>] SyS_getdents+0x89/0x100
[212326.444448] [<ffffffff811da020>] ? fillonedir+0xd0/0xd0
[212326.445552] [<ffffffff810ff216>] ? __audit_syscall_exit+0x236/0x2e0
[212326.446666] [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
[212326.447783] Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00 49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
[212326.449050] RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
[212326.450277] RSP <ffff88030f80fb78>
[212326.451513] general protection fault: 0000 [#2] SMP
[212326.452755] Modules linked in: rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd fscache sunrpc macvlan xt_nat sit tunnel4 ip_tunnel sch_sfq ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT xt_limit 8021q nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_tcpudp bridge ip6t_rt nf_conntrack_ipv6 stp llc nf_defrag_ipv6 xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat x86_pkg_temp_thermal iTCO_wdt intel_powerclamp raid1 iTCO_vendor_support raid0 coretemp crct10dif_pclmul md_mod snd_hda_codec_hdmi crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul snd_hda_codec_realtek glue_helper ablk_helper cryptd snd_hda_codec_generic snd_hda_intel snd_hda_controller microcode i2c_i801 r8169 snd_hda_codec
[212326.457001] snd_hwdep mii snd_pcm snd_timer thermal fan snd acpi_cpufreq battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
[212326.461578] CPU: 5 PID: 30360 Comm: find Tainted: P WC O 3.15.6-1-ec2 #1
[212326.463122] Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
[212326.464678] task: ffff8801275bbb00 ti: ffff88030f80c000 task.ti: ffff88030f80c000
[212326.466248] RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
[212326.467835] RSP: 0018:ffff88030f80f608 EFLAGS: 00010082
[212326.469445] RAX: 0000000000000000 RBX: ffff88030faa9000 RCX: 00000000000035ee
[212326.471051] RDX: 00000000000035ed RSI: 0000000000000000 RDI: 0000000000000000
[212326.472666] RBP: ffff88030f80f640 R08: 00000000000173c0 R09: ffff88040e803e00
[212326.474272] R10: ffffffff8132d81f R11: 0000000000000000 R12: ff0074726f707262
[212326.475873] R13: 0000000000008020 R14: 0000000000000008 R15: ffff88040e803e00
[212326.477454] FS: 00007f3b98919700(0000) GS:ffff88041f340000(0000) knlGS:0000000000000000
[212326.479005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[212326.480565] CR2: 0000000000ef0010 CR3: 00000003ffd3c000 CR4: 00000000001407e0
[212326.482143] Stack:
[212326.483720] 0000000000000000 ffff88030f80f718 ffff88030faa9000 ffff88030f80f6a8
[212326.485307] ffff88040e8634b0 0000000000000000 0000000000000001 ffff88030f80f690
[212326.486899] ffffffff8132d81f ffffffffa00ccc59 ffffffffa00ccc59 0000000000000021
[212326.488505] Call Trace:
[212326.490099] [<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
[212326.491703] [<ffffffff8132db3a>] acpi_ns_get_node+0x79/0xe2
[212326.493299] [<ffffffff81336827>] ? acpi_ut_allocate_object_desc_dbg+0x3e/0x6a
[212326.494937] [<ffffffff813368c2>] ? acpi_ut_create_internal_object_dbg+0x23/0x87
[212326.496542] [<ffffffff8132b531>] acpi_ns_evaluate+0x51/0x24d
[212326.498143] [<ffffffff8132b531>] ? acpi_ns_evaluate+0x51/0x24d
[212326.499733] [<ffffffff8132e319>] acpi_evaluate_object+0x189/0x285
[212326.501312] [<ffffffff8130f0bc>] acpi_execute_simple_method+0x43/0x45
[212326.502856] [<ffffffffa00cb63e>] acpi_video_register+0x3c1/0x593 [video]
[212326.504361] [<ffffffffa00cb789>] acpi_video_register+0x50c/0x593 [video]
[212326.505815] [<ffffffff81302599>] fb_notifier_callback+0x109/0x130
[212326.507231] [<ffffffff8150fc7d>] notifier_call_chain+0x4d/0x70
[212326.508607] [<ffffffff8108f137>] __blocking_notifier_call_chain+0x47/0x60
[212326.509965] [<ffffffff8108f166>] blocking_notifier_call_chain+0x16/0x20
[212326.511285] [<ffffffff81302f5b>] fb_notifier_call_chain+0x1b/0x20
[212326.512602] [<ffffffff8130350e>] fb_blank+0x9e/0xc0
[212326.513908] [<ffffffff812fa6e1>] fbcon_blank+0x1f1/0x300
[212326.515203] [<ffffffff810c1044>] ? wake_up_klogd+0x34/0x50
[212326.516490] [<ffffffff810c1259>] ? console_unlock+0x1f9/0x3d0
[212326.517770] [<ffffffff81073c8b>] ? lock_timer_base.isra.26+0x2b/0x50
[212326.519050] [<ffffffff8107219f>] ? internal_add_timer+0x2f/0x70
[212326.520324] [<ffffffff81074415>] ? mod_timer+0x105/0x200
[212326.521593] [<ffffffff8136d04a>] do_unblank_screen+0xba/0x1f0
[212326.522860] [<ffffffff8136d190>] unblank_screen+0x10/0x20
[212326.524118] [<ffffffff812ae8b9>] bust_spinlocks+0x19/0x40
[212326.525366] [<ffffffff8150cb18>] oops_end+0x38/0x150
[212326.526605] [<ffffffff8101639b>] die+0x4b/0x70
[212326.527834] [<ffffffff8150c5fa>] do_general_protection+0xca/0x150
[212326.529061] [<ffffffff8150bf68>] general_protection+0x28/0x30
[212326.530282] [<ffffffff811ad226>] ? __kmalloc_track_caller+0x86/0x260
[212326.531504] [<ffffffff811ad351>] ? __kmalloc_track_caller+0x1b1/0x260
[212326.532713] [<ffffffffa0211d75>] ? nfs_permission+0x405/0xfb0 [nfs]
[212326.533917] [<ffffffff8116fb60>] kmemdup+0x20/0x50
[212326.535117] [<ffffffffa0211d75>] nfs_permission+0x405/0xfb0 [nfs]
[212326.536320] [<ffffffffa0212277>] nfs_permission+0x907/0xfb0 [nfs]
[212326.537522] [<ffffffffa0212350>] ? nfs_permission+0x9e0/0xfb0 [nfs]
[212326.538726] [<ffffffffa0212372>] nfs_permission+0xa02/0xfb0 [nfs]
[212326.539928] [<ffffffff8115300e>] do_read_cache_page+0x7e/0x1a0
[212326.541128] [<ffffffff8115314c>] read_cache_page+0x1c/0x20
[212326.542329] [<ffffffffa021252b>] nfs_permission+0xbbb/0xfb0 [nfs]
[212326.543531] [<ffffffffa0787690>] ? nfs4_proc_secinfo+0x63a0/0x63a0 [nfsv4]
[212326.544735] [<ffffffff811d9f16>] iterate_dir+0xa6/0xe0
[212326.545935] [<ffffffff811da359>] SyS_getdents+0x89/0x100
[212326.547137] [<ffffffff811da020>] ? fillonedir+0xd0/0xd0
[212326.548336] [<ffffffff810ff216>] ? __audit_syscall_exit+0x236/0x2e0
[212326.549557] [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
[212326.550758] Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 64 01 00 00 49 83 78 10 00 0f 84 59 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
[212326.552107] RIP [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
[212326.553311] RSP <ffff88030f80f608>
[212326.554506] ---[ end trace 71a1e508f45dbd1e ]---
I'm thinking I should start turning on some of the more invasive debug
kernel configs to get to the bottom of this...
On Fri, Jul 25, 2014 at 9:42 PM, Steven Noonan <[email protected]> wrote:
> On Thu, Jul 24, 2014 at 12:06 AM, Alexander Holler <[email protected]> wrote:
>> Am 23.07.2014 19:50, schrieb Steven Noonan:
>>
>>> (Oops, LKML doesn't like rich text, resending. Was trying to avoid
>>> GMail's bad line wrapping. Going to use Mutt instead.)
>>>
>>> I'm starting to wonder if it's bad RAM or something. Just got a couple of
>>> worrying warnings on boot from the same system (after it spontaneously
>>> rebooted, with nothing revealing in the previous boot's logs).
>
> So the spontaneous reboot was apparently caused by a power outage. All
> my boxes had identical uptimes of less than a couple days when I checked
> them.
>
>>
>>
>> I once had such too and since then I'm using memtest=3 in my kernel command
>> line on x86* machines. Depending on the amount of RAM it will slow down boot
>> by a few seconds, but if you don't care if your machine comes up in 5 or 10
>> seconds, it is a no-brainer.
>>
>
> However, I got another general protection fault. This time it happened
> when doing 'find' on an NFS mount point. Tried booting with 'memtest=16'
> to see if that would catch anything, but it passed without finding any
> bad regions. I'm running memtest86 right now to be a bit more thorough
> and ensure it's not just bad hardware, but so far it's not found
> anything (1 full pass done so far).
>
> Here's the latest backtraces. I only managed to copy/paste this before
> the system hung and I had to reboot it, but there should be a more
> complete kernel log in the systemd journal that I can grab once it's
> done with memtest86.
>
> [212326.408380] general protection fault: 0000 [#1] SMP
> [212326.409183] Modules linked in: rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd fscache sunrpc macvlan xt_nat sit tunnel4 ip_tunnel sch_sfq ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT xt_limit 8021q nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_tcpudp bridge ip6t_rt nf_conntrack_ipv6 stp llc nf_defrag_ipv6 xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat x86_pkg_temp_thermal iTCO_wdt intel_powerclamp raid1 iTCO_vendor_support raid0 coretemp crct10dif_pclmul md_mod snd_hda_codec_hdmi crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul snd_hda_codec_realtek glue_helper ablk_helper cryptd snd_hda_codec_generic snd_hda_intel snd_hda_controller microcode i2c_i801 r8169 snd_hda_codec
> [212326.411879] snd_hwdep mii snd_pcm snd_timer thermal fan snd acpi_cpufreq battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
> [212326.414577] CPU: 5 PID: 30360 Comm: find Tainted: P WC O 3.15.6-1-ec2 #1
> [212326.415457] Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
> [212326.416352] task: ffff8801275bbb00 ti: ffff88030f80c000 task.ti: ffff88030f80c000
> [212326.417261] RIP: 0010:[<ffffffff811ad226>] [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
> [212326.418194] RSP: 0018:ffff88030f80fb78 EFLAGS: 00010282
> [212326.419130] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 00000000000035ee
> [212326.420081] RDX: 00000000000035ed RSI: 0000000000000000 RDI: 0000000000000000
> [212326.421021] RBP: ffff88030f80fbb0 R08: 00000000000173c0 R09: ffff8801eb6ae160
> [212326.421958] R10: ffff88040e803e00 R11: 0000000000000004 R12: ff0074726f707262
> [212326.422887] R13: 00000000000000d0 R14: 0000000000000004 R15: ffff88040e803e00
> [212326.423808] FS: 00007f3b98919700(0000) GS:ffff88041f340000(0000) knlGS:0000000000000000
> [212326.424752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [212326.425698] CR2: 0000000000ef0010 CR3: 00000003ffd3c000 CR4: 00000000001407e0
> [212326.426659] Stack:
> [212326.427620] ffff88040e803e00 ffffffffa0211d75 0000000000000004 ffff8803607f0558
> [212326.428609] 0000000000000009 ffff8801eb6ae000 ffff8801eb6ae140 ffff88030f80fbd0
> [212326.429630] ffffffff8116fb60 ffff88030f80fd40 ffff88030f80fe58 ffff88030f80fcc8
> [212326.430640] Call Trace:
> [212326.431651] [<ffffffffa0211d75>] ? nfs_permission+0x405/0xfb0 [nfs]
> [212326.432681] [<ffffffff8116fb60>] kmemdup+0x20/0x50
> [212326.433717] [<ffffffffa0211d75>] nfs_permission+0x405/0xfb0 [nfs]
> [212326.434760] [<ffffffffa0212277>] nfs_permission+0x907/0xfb0 [nfs]
> [212326.435810] [<ffffffffa0212350>] ? nfs_permission+0x9e0/0xfb0 [nfs]
> [212326.436863] [<ffffffffa0212372>] nfs_permission+0xa02/0xfb0 [nfs]
> [212326.437924] [<ffffffff8115300e>] do_read_cache_page+0x7e/0x1a0
> [212326.438990] [<ffffffff8115314c>] read_cache_page+0x1c/0x20
> [212326.440078] [<ffffffffa021252b>] nfs_permission+0xbbb/0xfb0 [nfs]
> [212326.441159] [<ffffffffa0787690>] ? nfs4_proc_secinfo+0x63a0/0x63a0 [nfsv4]
> [212326.442251] [<ffffffff811d9f16>] iterate_dir+0xa6/0xe0
> [212326.443347] [<ffffffff811da359>] SyS_getdents+0x89/0x100
> [212326.444448] [<ffffffff811da020>] ? fillonedir+0xd0/0xd0
> [212326.445552] [<ffffffff810ff216>] ? __audit_syscall_exit+0x236/0x2e0
> [212326.446666] [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
> [212326.447783] Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 50 01 00 00 49 83 78 10 00 0f 84 45 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
> [212326.449050] RIP [<ffffffff811ad226>] __kmalloc_track_caller+0x86/0x260
> [212326.450277] RSP <ffff88030f80fb78>
> [212326.451513] general protection fault: 0000 [#2] SMP
> [212326.452755] Modules linked in: rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd fscache sunrpc macvlan xt_nat sit tunnel4 ip_tunnel sch_sfq ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT xt_limit 8021q nf_conntrack_ipv4 nf_defrag_ipv4 xt_LOG xt_tcpudp bridge ip6t_rt nf_conntrack_ipv6 stp llc nf_defrag_ipv6 xt_conntrack nf_conntrack iptable_filter ip6table_filter ip6_tables ip_tables x_tables it87 hwmon_vid nls_cp437 vfat fat x86_pkg_temp_thermal iTCO_wdt intel_powerclamp raid1 iTCO_vendor_support raid0 coretemp crct10dif_pclmul md_mod snd_hda_codec_hdmi crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul snd_hda_codec_realtek glue_helper ablk_helper cryptd snd_hda_codec_generic snd_hda_intel snd_hda_controller microcode i2c_i801 r8169 snd_hda_codec
> [212326.457001] snd_hwdep mii snd_pcm snd_timer thermal fan snd acpi_cpufreq battery soundcore lpc_ich mfd_core evdev processor zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) tun usbip_host(C) usbip_core(C) msr loop kvm_intel kvm efivarfs ext4 crc16 jbd2 mbcache sd_mod crc_t10dif crct10dif_common hid_generic usbhid hid ahci libahci crc32c_intel ehci_pci libata xhci_hcd ehci_hcd scsi_mod usbcore usb_common i915 video intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core e1000e ptp pps_core ipmi_poweroff ipmi_msghandler button
> [212326.461578] CPU: 5 PID: 30360 Comm: find Tainted: P WC O 3.15.6-1-ec2 #1
> [212326.463122] Hardware name: Shuttle Inc. SH67H/FH67H, BIOS 2.04 04/10/2013
> [212326.464678] task: ffff8801275bbb00 ti: ffff88030f80c000 task.ti: ffff88030f80c000
> [212326.466248] RIP: 0010:[<ffffffff811aa26a>] [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
> [212326.467835] RSP: 0018:ffff88030f80f608 EFLAGS: 00010082
> [212326.469445] RAX: 0000000000000000 RBX: ffff88030faa9000 RCX: 00000000000035ee
> [212326.471051] RDX: 00000000000035ed RSI: 0000000000000000 RDI: 0000000000000000
> [212326.472666] RBP: ffff88030f80f640 R08: 00000000000173c0 R09: ffff88040e803e00
> [212326.474272] R10: ffffffff8132d81f R11: 0000000000000000 R12: ff0074726f707262
> [212326.475873] R13: 0000000000008020 R14: 0000000000000008 R15: ffff88040e803e00
> [212326.477454] FS: 00007f3b98919700(0000) GS:ffff88041f340000(0000) knlGS:0000000000000000
> [212326.479005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [212326.480565] CR2: 0000000000ef0010 CR3: 00000003ffd3c000 CR4: 00000000001407e0
> [212326.482143] Stack:
> [212326.483720] 0000000000000000 ffff88030f80f718 ffff88030faa9000 ffff88030f80f6a8
> [212326.485307] ffff88040e8634b0 0000000000000000 0000000000000001 ffff88030f80f690
> [212326.486899] ffffffff8132d81f ffffffffa00ccc59 ffffffffa00ccc59 0000000000000021
> [212326.488505] Call Trace:
> [212326.490099] [<ffffffff8132d81f>] acpi_ns_internalize_name+0x68/0xad
> [212326.491703] [<ffffffff8132db3a>] acpi_ns_get_node+0x79/0xe2
> [212326.493299] [<ffffffff81336827>] ? acpi_ut_allocate_object_desc_dbg+0x3e/0x6a
> [212326.494937] [<ffffffff813368c2>] ? acpi_ut_create_internal_object_dbg+0x23/0x87
> [212326.496542] [<ffffffff8132b531>] acpi_ns_evaluate+0x51/0x24d
> [212326.498143] [<ffffffff8132b531>] ? acpi_ns_evaluate+0x51/0x24d
> [212326.499733] [<ffffffff8132e319>] acpi_evaluate_object+0x189/0x285
> [212326.501312] [<ffffffff8130f0bc>] acpi_execute_simple_method+0x43/0x45
> [212326.502856] [<ffffffffa00cb63e>] acpi_video_register+0x3c1/0x593 [video]
> [212326.504361] [<ffffffffa00cb789>] acpi_video_register+0x50c/0x593 [video]
> [212326.505815] [<ffffffff81302599>] fb_notifier_callback+0x109/0x130
> [212326.507231] [<ffffffff8150fc7d>] notifier_call_chain+0x4d/0x70
> [212326.508607] [<ffffffff8108f137>] __blocking_notifier_call_chain+0x47/0x60
> [212326.509965] [<ffffffff8108f166>] blocking_notifier_call_chain+0x16/0x20
> [212326.511285] [<ffffffff81302f5b>] fb_notifier_call_chain+0x1b/0x20
> [212326.512602] [<ffffffff8130350e>] fb_blank+0x9e/0xc0
> [212326.513908] [<ffffffff812fa6e1>] fbcon_blank+0x1f1/0x300
> [212326.515203] [<ffffffff810c1044>] ? wake_up_klogd+0x34/0x50
> [212326.516490] [<ffffffff810c1259>] ? console_unlock+0x1f9/0x3d0
> [212326.517770] [<ffffffff81073c8b>] ? lock_timer_base.isra.26+0x2b/0x50
> [212326.519050] [<ffffffff8107219f>] ? internal_add_timer+0x2f/0x70
> [212326.520324] [<ffffffff81074415>] ? mod_timer+0x105/0x200
> [212326.521593] [<ffffffff8136d04a>] do_unblank_screen+0xba/0x1f0
> [212326.522860] [<ffffffff8136d190>] unblank_screen+0x10/0x20
> [212326.524118] [<ffffffff812ae8b9>] bust_spinlocks+0x19/0x40
> [212326.525366] [<ffffffff8150cb18>] oops_end+0x38/0x150
> [212326.526605] [<ffffffff8101639b>] die+0x4b/0x70
> [212326.527834] [<ffffffff8150c5fa>] do_general_protection+0xca/0x150
> [212326.529061] [<ffffffff8150bf68>] general_protection+0x28/0x30
> [212326.530282] [<ffffffff811ad226>] ? __kmalloc_track_caller+0x86/0x260
> [212326.531504] [<ffffffff811ad351>] ? __kmalloc_track_caller+0x1b1/0x260
> [212326.532713] [<ffffffffa0211d75>] ? nfs_permission+0x405/0xfb0 [nfs]
> [212326.533917] [<ffffffff8116fb60>] kmemdup+0x20/0x50
> [212326.535117] [<ffffffffa0211d75>] nfs_permission+0x405/0xfb0 [nfs]
> [212326.536320] [<ffffffffa0212277>] nfs_permission+0x907/0xfb0 [nfs]
> [212326.537522] [<ffffffffa0212350>] ? nfs_permission+0x9e0/0xfb0 [nfs]
> [212326.538726] [<ffffffffa0212372>] nfs_permission+0xa02/0xfb0 [nfs]
> [212326.539928] [<ffffffff8115300e>] do_read_cache_page+0x7e/0x1a0
> [212326.541128] [<ffffffff8115314c>] read_cache_page+0x1c/0x20
> [212326.542329] [<ffffffffa021252b>] nfs_permission+0xbbb/0xfb0 [nfs]
> [212326.543531] [<ffffffffa0787690>] ? nfs4_proc_secinfo+0x63a0/0x63a0 [nfsv4]
> [212326.544735] [<ffffffff811d9f16>] iterate_dir+0xa6/0xe0
> [212326.545935] [<ffffffff811da359>] SyS_getdents+0x89/0x100
> [212326.547137] [<ffffffff811da020>] ? fillonedir+0xd0/0xd0
> [212326.548336] [<ffffffff810ff216>] ? __audit_syscall_exit+0x236/0x2e0
> [212326.549557] [<ffffffff8151496d>] system_call_fastpath+0x1a/0x1f
> [212326.550758] Code: 25 88 dd 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 64 01 00 00 49 83 78 10 00 0f 84 59 01 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <49> 8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 bb 49 63
> [212326.552107] RIP [<ffffffff811aa26a>] __kmalloc+0x8a/0x280
> [212326.553311] RSP <ffff88030f80f608>
> [212326.554506] ---[ end trace 71a1e508f45dbd1e ]---
>
> I'm thinking I should start turning on some of the more invasive debug
> kernel configs to get to the bottom of this...
Stopped memtest86 mid-way through the 2nd pass so I could get the full
kernel log:
http://pastebin.com/raw.php?i=qkZ0LNCr
NMI watchdog kicked in while it was hung.