Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752758Ab1FORye (ORCPT ); Wed, 15 Jun 2011 13:54:34 -0400 Received: from smtp5-g21.free.fr ([212.27.42.5]:43858 "EHLO smtp5-g21.free.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752088Ab1FORyc (ORCPT ); Wed, 15 Jun 2011 13:54:32 -0400 Message-ID: <4DF8F1D0.7000907@corp.free.fr> Date: Wed, 15 Jun 2011 19:54:24 +0200 From: Yohan User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10 MIME-Version: 1.0 To: linux-kernel@vger.kernel.org Subject: Kernel panic on java mail server Content-Type: multipart/mixed; boundary="------------080303030300050101050100" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 32828 Lines: 514 This is a multi-part message in MIME format. --------------080303030300050101050100 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hello, I have a stange kernel panic on a zimbra mail server (mysqld + server in java)... when the java process starts the system freeze... This is the first time that i have that kind of problem over 70 same kind servers. The hardware has been changed 3 times (2 times on AMD server, 1 time on Intel server). I join the traces on the intel platform, Tyan S7012 MB, 1 CPU Intel E5645, 48GB RAM, kernels 2.6.32.41 and 2.6.39.1 without debugging enabled and the output with the 2.6.32.41 with debugging enabled. It was tested on 2.6.32.41 / 2.6.35.13 / 2.6.38.4 / 2.6.38.8 / 2.6.39.1.... same thing. Thanks for helping. Yohan --------------080303030300050101050100 Content-Type: text/plain; name="2.6.39.1.txt" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="2.6.39.1.txt" [ 180.154239] general protection fault: 0000 [#1] SMP [ 180.247372] last sysfs file: /sys/devices/system/machinecheck/machinecheck0/trigger [ 180.373040] CPU 6 [ 180.430767] Pid: 4636, comm: mysqld Not tainted 2.6.39.1-zimbraI #1 empty empty/S7012 [ 180.559863] RIP: 0010:[] [] free_pcppages_bulk+0x2c3/0x380 [ 180.699707] RSP: 0000:ffff880c0a0d5958 EFLAGS: 00010002 [ 180.798726] RAX: 2e74736f682e7367 RBX: 0000000000000002 RCX: ffffea00297498a0 [ 180.919653] RDX: ffffea00297c49d0 RSI: ffff880c3fffbe88 RDI: ffffea00297c49a8 [ 181.040598] RBP: ffff880c0a0d59a8 R08: 0000000000000000 R09: 00000000000005e2 [ 181.161760] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880c3fffbe00 [ 181.283022] R13: ffff880c3f2d21d8 R14: 0000000000000019 R15: 0000000000000002 [ 181.404549] FS: 0000000000000000(0000) GS:ffff880c3f2c0000(0000) knlGS:0000000000000000 [ 181.538463] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 181.644584] CR2: 00000000005297e0 CR3: 0000000001475000 CR4: 00000000000006e0 [ 181.767912] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 181.891518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 182.015193] Process mysqld (pid: 4636, threadinfo ffff880c0a0d4000, task ffff880c0e5cef40) [ 182.153504] Stack: [ 182.216759] 2e74736f682e7367 7473675f3d656d67 e0726f20656d616e 0000000000000002 [ 182.346623] 000000196f207373 657672657320736f 30757465730a2e72 657384d0736f6834 [ 182.477249] 302e73676e697474 696d64413d747275 2c6f158d6f43206e 0a3a74726f50206b [ 182.608362] Call Trace: [ 182.679209] Code: 50 68 4b 8d 04 80 49 8d 04 40 49 83 84 c4 b8 00 00 00 01 83 6d d4 01 0f 84 88 00 00 00 41 83 ee 01 0f 84 a6 fd ff ff 48 8b 45 b0 [ 182.844257] 39 00 0f 85 c4 fd ff ff 0f 1f 40 00 e9 90 fd ff ff 4d 63 da [ 183.017325] RIP [] free_pcppages_bulk+0x2c3/0x380 [ 183.137398] RSP [ 183.222984] ---[ end trace 786e33d9c56c26b4 ]--- [ 183.222986] java: Corrupted page table at address 32c671428 [ 183.222988] PGD c0a13e067 PUD 2065970b65636557 BAD [ 183.222994] Bad pagetable: 0009 [#2] SMP [ 183.222995] last sysfs file: /sys/devices/system/machinecheck/machinecheck0/trigger [ 183.222997] CPU 0 [ 183.222999] Pid: 4629, comm: java Tainted: G D 2.6.39.1-zimbraI #1 empty empty/S7012 [ 183.223002] RIP: 0010:[] [] task_tick_fair+0x36/0xf0 [ 183.223006] RSP: 0018:ffff880c3f203ea8 EFLAGS: 00010086 [ 183.223008] RAX: 0000000075636f69 RBX: ffff880c0baf37a0 RCX: 00000000fffdc9b7 [ 183.223010] RDX: 0000000000000000 RSI: ffff880c0baf37a0 RDI: ffff880c3f2110c0 [ 183.223012] RBP: ffff880c3f203ed8 R08: 0000000000000800 R09: 0000000000000005 [ 183.223013] R10: 0000000000000000 R11: 0000000000000004 R12: 00000000000110c0 [ 183.223015] R13: 00000000000110c0 R14: ffff880c0baf37d8 R15: 0000000000000000 [ 183.223018] FS: 000000004034d950(0063) GS:ffff880c3f200000(0000) knlGS:0000000000000000 [ 183.223020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 183.223021] CR2: ffff9f0b65636b18 CR3: 0000000c0bb4b000 CR4: 00000000000006f0 [ 183.223023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 183.223025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 183.223027] Process java (pid: 4629, threadinfo ffff880c0a20e000, task ffff880c0baf37a0) [ 183.223028] Stack: [ 183.223029] ffffffff815bc1a0 ffff880c3f2110c0 0000000000000010 00000000000110c0 [ 183.223032] ffff880c0baf37a0 0000000000000000 ffff880c3f203f18 ffffffff81031252 [ 183.223034] 0000000000000000 0000000000000000 0000000000000000 ffff880c0baf37a0 [ 183.223037] Call Trace: [ 183.223038] [ 183.223042] [] scheduler_tick+0x162/0x230 [ 183.223047] [] update_process_times+0x67/0x80 [ 183.223050] [] tick_periodic+0x27/0x70 [ 183.223053] [] tick_handle_periodic+0x21/0x80 [ 183.223057] [] smp_apic_timer_interrupt+0x66/0xa0 [ 183.223060] [] apic_timer_interrupt+0x13/0x20 [ 183.223062] [ 183.223066] [] ? memcpy+0xb/0x120 [ 183.223070] Code: f0 49 89 f6 48 89 5d d8 49 83 c6 38 4c 89 65 e0 4c 89 6d e8 4c 89 7d f8 48 89 f3 74 79 48 8b 46 08 49 c7 c4 c0 10 01 00 8b 40 18 <4c> 03 24 c5 e0 98 4b 81 4d 8d 6c 24 68 4c 89 ef e8 d5 ed ff ff [ 183.223086] RIP [] task_tick_fair+0x36/0xf0 [ 183.223089] RSP [ 183.223090] ---[ end trace 786e33d9c56c26b5 ]--- [ 183.223092] Kernel panic - not syncing: Fatal exception in interrupt [ 183.223094] Pid: 4629, comm: java Tainted: G D 2.6.39.1-zimbraI #1 [ 183.223095] Call Trace: [ 183.223096] [] panic+0xba/0x1e0 [ 183.223102] [] ? show_registers+0x86/0x280 [ 183.223105] [] ? kmsg_dump+0x4b/0x100 [ 183.223109] [] ? down_trylock+0x36/0x50 [ 183.223112] [] oops_end+0x9f/0xb0 [ 183.223116] [] pgtable_bad+0x94/0xb0 [ 183.223118] [] do_page_fault+0x2f4/0x450 [ 183.223121] [] ? __kfree_skb+0x3a/0xa0 [ 183.223123] [] ? consume_skb+0x19/0x40 [ 183.223128] [] ? dev_kfree_skb_any+0x35/0x40 [ 183.223132] [] ? igb_poll+0x8d2/0xd30 [ 183.223135] [] ? kobject_put+0x27/0x60 [ 183.223140] [] ? put_device+0x12/0x20 [ 183.223144] [] page_fault+0x1f/0x30 [ 183.223147] [] ? task_tick_fair+0x36/0xf0 [ 183.223149] [] scheduler_tick+0x162/0x230 [ 183.223152] [] update_process_times+0x67/0x80 [ 183.223155] [] tick_periodic+0x27/0x70 [ 183.223157] [] tick_handle_periodic+0x21/0x80 [ 183.223159] [] smp_apic_timer_interrupt+0x66/0xa0 [ 183.223162] [] apic_timer_interrupt+0x13/0x20 [ 183.223163] [] ? memcpy+0xb/0x120 [ 190.682307] Kernel panic - not syncing: Fatal exception in interrupt [ 190.798751] Pid: 4636, comm: mysqld Tainted: G D 2.6.39.1-zimbraI #1 [ 190.924945] Call Trace: [ 190.994578] [] panic+0xba/0x1e0 [ 191.092143] [] ? kmsg_dump+0x4b/0x100 [ 191.195554] [] oops_end+0x9f/0xb0 [ 191.294317] [] die+0x56/0x90 [ 191.387021] [] do_general_protection+0x152/0x160 [ 191.500374] [] general_protection+0x1f/0x30 [ 191.608294] [] ? free_pcppages_bulk+0x2c3/0x380 --------------080303030300050101050100 Content-Type: text/plain; name="2.6.32.41.txt" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="2.6.32.41.txt" [ 261.993172] BUG: soft lockup - CPU#3 stuck for 61s! [syslogd:2418] [ 262.109164] BUG: soft lockup - CPU#11 stuck for 61s! [cron:4555] [ 262.221917] BUG: soft lockup - CPU#8 stuck for 61s! [java:4458] [ 262.332433] BUG: soft lockup - CPU#2 stuck for 61s! [flush-9:1:2356] [ 262.446477] BUG: soft lockup - CPU#10 stuck for 61s! [munin-node:4554] [ 264.140716] BUG: soft lockup - CPU#0 stuck for 61s! [java:4486] [ 264.246744] BUG: soft lockup - CPU#6 stuck for 61s! [sync_supers:148] [ 264.358389] BUG: soft lockup - CPU#9 stuck for 61s! [mysqld:3000] [ 281.127736] BUG: spinlock lockup on CPU#10, munin-node/4554, ffffffff814ae0c0 [ 287.057652] BUG: spinlock lockup on CPU#8, java/4458, ffffffff814ae0c0 [ 290.221802] BUG: spinlock lockup on CPU#6, sync_supers/148, ffffffff814ae0c0 [ 291.878297] BUG: spinlock lockup on CPU#3, syslogd/2418, ffffffff814ae0c0 [ 293.198109] BUG: spinlock lockup on CPU#2, flush-9:1/2356, ffffffff814ae0c0 [ 294.429820] BUG: spinlock lockup on CPU#9, mysqld/3000, ffffffff814ae0c0 [ 302.842471] BUG: spinlock lockup on CPU#11, cron/4555, ffffffff814ae0c0 [ 334.077742] BUG: spinlock lockup on CPU#7, sshd/4556, ffffffff814ae0c0 [ 345.346860] BUG: spinlock lockup on CPU#4, mdadm/2560, ffffffff814ae0c0 [ 375.919077] BUG: spinlock lockup on CPU#5, cron/4557, ffffffff814ae0c0 [ 379.232633] INFO: task java:4439 blocked for more than 120 seconds. [ 379.344135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 379.475150] INFO: task java:4457 blocked for more than 120 seconds. [ 379.588143] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 379.720604] INFO: task java:4461 blocked for more than 120 seconds. [ 379.834390] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 379.967781] INFO: task java:4485 blocked for more than 120 seconds. [ 380.082147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 380.216339] INFO: task java:4508 blocked for more than 120 seconds. [ 380.332092] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 380.467907] INFO: task java:4513 blocked for more than 120 seconds. [ 380.585320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 380.722442] INFO: task java:4514 blocked for more than 120 seconds. [ 380.841406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 380.980238] INFO: task java:4515 blocked for more than 120 seconds. [ 381.100689] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 381.240764] INFO: task java:4516 blocked for more than 120 seconds. [ 381.361822] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 381.502291] INFO: task java:4517 blocked for more than 120 seconds. [ 381.624433] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. --------------080303030300050101050100 Content-Type: text/plain; name="2.6.32.41d.txt" Content-Transfer-Encoding: 8bit Content-Disposition: attachment; filename="2.6.32.41d.txt" [ 1528.346126] BUG: unable to handle kernel NULL pointer dereference at 0000000000000026 [ 1528.477707] IP: [] hrtimer_run_queues+0x128/0x210 [ 1528.590822] PGD c38b78067 PUD 0 [ 1528.667288] Thread overran stack, or stack corrupted [ 1528.764395] Oops: 0000 [#1] SMP [ 1528.840863] last sysfs file: /sys/devices/virtual/block/md1/md/mismatch_cnt [ 1528.962815] CPU 2 [ 1529.025583] Pid: 1514, comm: Not tainted 2.6.32.41-zimbraI #1 empty [ 1529.141162] RIP: 0010:[] [] hrtimer_run_queues+0x128/0x210 [ 1529.286085] RSP: 0018:ffff880052643ea8 EFLAGS: 00010002 [ 1529.390732] RAX: ffff880c34831728 RBX: 0000000000000000 RCX: ffffffffb2072bdd [ 1529.517860] RDX: 0000000000000038 RSI: 0000000022795c7a RDI: ffff88005264ce00 [ 1529.645131] RBP: ffff880052643f08 R08: 0000000000000001 R09: 0000000000000002 [ 1529.773096] R10: ffff880c38725fd8 R11: 0000000000000001 R12: ffff880c34831728 [ 1529.900970] R13: ffff880c3630c920 R14: 0000000000000026 R15: ffff88005264ce00 [ 1530.029136] FS: 00007fb9e34ea6e0(0000) GS:ffff880052640000(0000) knlGS:0000000000000000 [ 1530.169664] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1530.282344] CR2: 0000000000000026 CR3: 0000000c3677b000 CR4: 00000000000006e0 [ 1530.412026] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1530.541058] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1530.669764] Process (pid: 1514, threadinfo ffff880c360f7b80, task ffff880c3630c920) [ 1530.806400] Stack: [ 1530.874307] ffffffff8106a060 0000000000000038 000000004df8da13 0000000022795c7a [ 1530.962336] <0> 000000004df8da13 0000000022795c7a 0000000000000046 0000000000000000 [ 1531.099922] <0> 0000000000000002 ffff880c3630c920 0000000000000000 ffff880c36966180 [ 1531.284610] Call Trace: [ 1531.359372] [ 1531.430254] [] ? __rcu_process_callbacks+0x80/0x320 [ 1531.555186] [] run_local_timers+0x9/0x20 [ 1531.668331] [] update_process_times+0x37/0x70 [ 1531.787615] [] tick_periodic+0x27/0x70 [ 1531.898111] [] tick_handle_periodic+0x21/0x80 [ 1532.015833] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1532.137969] [] apic_timer_interrupt+0x13/0x20 [ 1532.255831] [ 1532.279106] Code: 00 48 8b 58 18 4a 39 5c 3a 38 49 89 c4 7f 17 e9 8a 00 00 00 48 8b 50 18 49 89 c4 48 8b 45 a8 4a 39 54 38 38 7e 78 4d 8b 74 24 30 <4d> 8b 2e 9c 58 f6 c4 02 0f 85 89 00 00 00 31 c9 ba 02 00 00 00 [ 1532.657167] RIP [] hrtimer_run_queues+0x128/0x210 [ 1532.780880] RSP [ 1532.870474] CR2: 0000000000000026 [ 1533.096616] ---[ end trace e16f43a7fa6f9a78 ]--- [ 1533.096619] general protection fault: 0000 [#2] SMP [ 1533.096623] last sysfs file: /sys/devices/virtual/block/md1/md/mismatch_cnt [ 1533.096625] CPU 0 [ 1533.096628] Pid: 9093, comm: java Tainted: G D 2.6.32.41-zimbraI #1 empty [ 1533.096630] RIP: 0010:[] [] hrtimer_run_queues+0x14d/0x210 [ 1533.096636] RSP: 0018:ffff880052603ea8 EFLAGS: 00010097 [ 1533.096638] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 1533.096640] RDX: 0000000000000000 RSI: ffffffff810b4f00 RDI: ffff880c3637d968 [ 1533.096642] RBP: ffff880052603f08 R08: 00000000ffffffff R09: ffffffffffdb6030 [ 1533.096645] R10: 00000000164799e8 R11: fffffffffff7d54a R12: ffff880c3637d968 [ 1533.096647] R13: 20ec8348e5894855 R14: ffffffff810b4ef0 R15: ffff88005260ce00 [ 1533.096650] FS: 0000000041ef2950(0063) GS:ffff880052600000(0000) knlGS:0000000000000000 [ 1533.096652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1533.096654] CR2: 00007fb8e53f7000 CR3: 0000000c3677b000 CR4: 00000000000006f0 [ 1533.096656] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1533.096659] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1533.096661] Process java (pid: 9093, threadinfo ffff880c38640000, task ffff880c385b5a00) [ 1533.096663] Stack: [ 1533.096664] ffffffff8106a060 0000000000000038 000000004df8da13 0000000022d4e764 [ 1533.096667] <0> 000000004df8da13 0000000022d4e764 ffff880052603f08 0000000000000000 [ 1533.096670] <0> 0000000000000000 ffff880c385b5a00 0000000000001000 ffff880bd9cd4000 [ 1533.096673] Call Trace: [ 1533.096674] [ 1533.096678] [] ? __rcu_process_callbacks+0x80/0x320 [ 1533.096682] [] run_local_timers+0x9/0x20 [ 1533.096684] [] update_process_times+0x37/0x70 [ 1533.096688] [] tick_periodic+0x27/0x70 [ 1533.096691] [] tick_handle_periodic+0x21/0x80 [ 1533.096694] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1533.096697] [] apic_timer_interrupt+0x13/0x20 [ 1533.096699] [ 1533.096703] [] ? memcpy_c+0xb/0x20 [ 1533.096706] [] ? memmove+0x41/0x50 [ 1533.096710] [] ? leaf_copy_items_entirely+0x147/0x230 [ 1533.096714] [] ? leaf_move_items+0x366/0x370 [ 1533.096717] [] ? leaf_shift_right+0x2d/0x70 [ 1533.096720] [] ? do_balance+0x15d4/0x2580 [ 1533.096723] [] ? bit_waitqueue+0x12/0xd0 [ 1533.096726] [] ? wake_up_bit+0x28/0x40 [ 1533.096731] [] ? unlock_buffer+0x12/0x20 [ 1533.096735] [] ? clear_all_dirty_bits+0xb/0x10 [ 1533.096738] [] ? fix_nodes+0x434/0x8d0 [ 1533.096741] [] ? reiserfs_delete_solid_item+0x1ab/0x300 [ 1533.096745] [] ? reiserfs_delete_object+0x58/0x80 [ 1533.096748] [] ? reiserfs_delete_inode+0xa2/0xe0 [ 1533.096752] [] ? iput+0x2b/0x70 [ 1533.096755] [] ? reiserfs_delete_inode+0x0/0xe0 [ 1533.096758] [] ? generic_delete_inode+0x85/0x120 [ 1533.096761] [] ? generic_drop_inode+0x18/0x70 [ 1533.096763] [] ? iput+0x5d/0x70 [ 1533.096767] [] ? do_unlinkat+0x113/0x1b0 [ 1533.096771] [] ? do_page_fault+0x144/0x260 [ 1533.096775] [] ? sys_unlink+0x11/0x20 [ 1533.096777] [] ? system_call_fastpath+0x16/0x1b [ 1533.096779] Code: 78 4d 8b 74 24 30 4d 8b 2e 9c 58 f6 c4 02 0f 85 89 00 00 00 31 c9 ba 02 00 00 00 4c 89 f6 4c 89 e7 e8 28 fc ff ff 49 8b 44 24 28 <41> fe 45 00 4c 89 e7 ff d0 4c 89 ef 89 c3 e8 b0 e1 2e 00 85 db [ 1533.096798] RIP [] hrtimer_run_queues+0x14d/0x210 [ 1533.096801] RSP [ 1533.096803] ---[ end trace e16f43a7fa6f9a79 ]--- [ 1533.096805] ------------[ cut here ]------------ [ 1533.096808] Kernel panic - not syncing: Fatal exception in interrupt [ 1533.096811] kernel BUG at kernel/sched.c:1165! [ 1533.096814] Pid: 9093, comm: java Tainted: G D 2.6.32.41-zimbraI #1 [ 1533.096817] invalid opcode: 0000 [#3] Call Trace: [ 1533.096820] SMP [ 1533.096822] last sysfs file: /sys/devices/virtual/block/md1/md/mismatch_cnt [ 1533.096826] [] panic+0xa0/0x170 [ 1533.096828] CPU 3 [ 1533.096832] [] ? show_registers+0x86/0x260 [ 1533.096835] Pid: 2538, comm: munin-node Tainted: G D 2.6.32.41-zimbraI #1 empty [ 1533.096838] RIP: 0010:[] [] ? down_trylock+0x36/0x50 [ 1533.096846] [] resched_task+0x61/0x70 [ 1533.096848] RSP: 0000:ffff880052663e88 EFLAGS: 00010046 [ 1533.096851] [] ? console_unblank+0x1f/0x80 [ 1533.096854] RAX: 000000000000006c RBX: ffff880c38782760 RCX: ffff880c38640000 [ 1533.096857] RDX: 0000000000006c6c RSI: 0000000000000400 RDI: ffff880c385b5a00 [ 1533.096861] [] ? print_oops_end_marker+0x1e/0x20 [ 1533.096864] RBP: ffff880052663e88 R08: 0000000000000001 R09: 0000000000000010 [ 1533.096867] R10: 0000000000000000 R11: 0000000000030001 R12: ffff880052610dc0 [ 1533.096870] [] oops_end+0x9f/0xb0 [ 1533.096873] R13: ffff880052610e18 R14: ffff880c38782798 R15: 0000000000988372 [ 1533.096876] FS: 00007f668eaf76e0(0000) GS:ffff880052660000(0000) knlGS:0000000000000000 [ 1533.096879] [] die+0x56/0x90 [ 1533.096882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1533.096884] CR2: 0000000000000000 CR3: 0000000c37011000 CR4: 00000000000006e0 [ 1533.096888] [] ? __pollwait+0x0/0x100 [ 1533.096890] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1533.096893] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1533.096896] [] do_general_protection+0x148/0x150 [ 1533.096900] Process munin-node (pid: 2538, threadinfo ffff880c36382000, task ffff880c38782760) [ 1533.096902] Stack: [ 1533.096906] [] general_protection+0x1f/0x30 [ 1533.096908] ffff880052663ec8 ffffffff81035c0d [] ? __pollwait+0x0/0x100 [ 1533.096913] ffff880c36383fd8 0000000000000003 [] ? __pollwait+0x10/0x100 [ 1533.096918] [ 1533.096918] <0> 00000163d8e3d4a3 [] ? hrtimer_run_queues+0x14d/0x210 [ 1533.096924] ffff880052670dc0 0000000000010dc0 [] ? hrtimer_run_queues+0x148/0x210 [ 1533.096929] ffff880c38782760 [ 1533.096931] <0> [] ? __rcu_process_callbacks+0x80/0x320 [ 1533.096935] ffff880052663f18 ffffffff8103b013 [] run_local_timers+0x9/0x20 [ 1533.096940] 0000000000000003 0000000000000003 [] update_process_times+0x37/0x70 [ 1533.096945] [ 1533.096947] Call Trace: [ 1533.096949] [] tick_periodic+0x27/0x70 [ 1533.096951] [ 1533.096954] [] tick_handle_periodic+0x21/0x80 [ 1533.096958] [] task_tick_fair+0xdd/0xf0 [ 1533.096962] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1533.096966] [] scheduler_tick+0x123/0x180 [ 1533.096969] [] apic_timer_interrupt+0x13/0x20 [ 1533.096973] [] update_process_times+0x4b/0x70 [ 1533.096975] [ 1533.096978] [] tick_periodic+0x27/0x70 [ 1533.096981] [] ? memcpy_c+0xb/0x20 [ 1533.096985] [] tick_handle_periodic+0x21/0x80 [ 1533.096988] [] ? memmove+0x41/0x50 [ 1533.096992] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1533.096996] [] ? leaf_copy_items_entirely+0x147/0x230 [ 1533.097000] [] apic_timer_interrupt+0x13/0x20 [ 1533.097002] [] ? leaf_move_items+0x366/0x370 [ 1533.097008] [] ? lock_kernel+0x2f/0x40 [ 1533.097012] [] ? leaf_shift_right+0x2d/0x70 [ 1533.097016] [] ? __dequeue_signal+0x110/0x160 [ 1533.097020] [] ? do_balance+0x15d4/0x2580 [ 1533.097023] [] ? recalc_sigpending+0x12/0x30 [ 1533.097027] [] ? bit_waitqueue+0x12/0xd0 [ 1533.097030] [] ? get_signal_to_deliver+0x185/0x340 [ 1533.097034] [] ? wake_up_bit+0x28/0x40 [ 1533.097038] [] ? do_notify_resume+0xb4/0x7d0 [ 1533.097041] [] ? unlock_buffer+0x12/0x20 [ 1533.097045] [] ? __bad_area_nosemaphore+0xbf/0x1c0 [ 1533.097049] [] ? clear_all_dirty_bits+0xb/0x10 [ 1533.097054] [] ? destroy_compound_page+0x70/0xb0 [ 1533.097058] [] ? fix_nodes+0x434/0x8d0 [ 1533.097061] [] ? __free_pages+0x18/0x30 [ 1533.097065] [] ? reiserfs_delete_solid_item+0x1ab/0x300 [ 1533.097068] [] ? __bad_area+0x4f/0x70 [ 1533.097071] [] ? reiserfs_delete_object+0x58/0x80 [ 1533.097075] [] ? bad_area+0xe/0x10 [ 1533.097078] [] ? reiserfs_delete_inode+0xa2/0xe0 [ 1533.097082] [] ? do_page_fault+0x1b4/0x260 [ 1533.097085] [] ? iput+0x2b/0x70 [ 1533.097088] [] ? retint_signal+0x3d/0x83 [ 1533.097092] [] ? reiserfs_delete_inode+0x0/0xe0 [ 1533.097094] Code: 18 [] ? generic_delete_inode+0x85/0x120 [ 1533.097099] 65 8b [] ? generic_drop_inode+0x18/0x70 [ 1533.097104] 04 25 [] ? iput+0x5d/0x70 [ 1533.097108] 50 b8 [] ? do_unlinkat+0x113/0x1b0 [ 1533.097113] 00 00 [] ? do_page_fault+0x144/0x260 [ 1533.097117] 39 c2 [] ? sys_unlink+0x11/0x20 [ 1533.097122] 74 0d [] ? system_call_fastpath+0x16/0x1b [ 1533.097127] 0f ae f0 48 8b 47 08 f6 40 14 04 74 02 c9 c3 89 d7 ff 15 26 7b 40 00 c9 0f 1f 44 00 00 c3 <0f> 0b eb fe 66 66 2e 0f 1f 84 00 00 00 00 00 55 4c 8b 8f 88 07 [ 1533.097137] RIP [] resched_task+0x61/0x70 [ 1533.097140] RSP [ 1533.097141] ---[ end trace e16f43a7fa6f9a7a ]--- [ 1533.097143] general protection fault: 0000 [#4] [ 1533.097146] Kernel panic - not syncing: Fatal exception in interrupt [ 1533.097148] SMP Pid: 2538, comm: munin-node Tainted: G D 2.6.32.41-zimbraI #1 [ 1533.097152] [ 1533.097153] Call Trace: [ 1533.097154] last sysfs file: /sys/devices/virtual/block/md1/md/mismatch_cnt [ 1533.097157] CPU 8 [ 1533.097161] [] panic+0xa0/0x170 [ 1533.097163] Pid: 0, comm: swapper Tainted: G D 2.6.32.41-zimbraI #1 empty [ 1533.097167] RIP: 0010:[] [] ? show_registers+0x86/0x260 [ 1533.097174] [] hrtimer_run_queues+0x14d/0x210 [ 1533.097177] RSP: 0018:ffff880052703ea8 EFLAGS: 00010097 [ 1533.097181] [] ? down_trylock+0x36/0x50 [ 1533.097183] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 1533.097187] RDX: 0000000000000000 RSI: ffffffff810b4f00 RDI: ffff880c36375968 [ 1533.097191] [] ? console_unblank+0x1f/0x80 [ 1533.097193] RBP: ffff880052703f08 R08: 00000000ffffffff R09: 0000000000000000 [ 1533.097196] R10: 0000000000000000 R11: 00000000ffffffff R12: ffff880c36375968 [ 1533.097200] [] ? print_oops_end_marker+0x1e/0x20 [ 1533.097203] R13: 20ec8348e5894855 R14: ffffffff810b4ef0 R15: ffff88005270ce00 [ 1533.097206] FS: 0000000000000000(0000) GS:ffff880052700000(0000) knlGS:0000000000000000 [ 1533.097211] [] oops_end+0x9f/0xb0 [ 1533.097213] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 1533.097216] CR2: 0000000000000000 CR3: 0000000c3677b000 CR4: 00000000000006e0 [ 1533.097220] [] die+0x56/0x90 [ 1533.097222] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1533.097226] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1533.097229] [] do_trap+0x130/0x150 [ 1533.097232] Process swapper (pid: 0, threadinfo ffff880c39d5a000, task ffff880c39d41c20) [ 1533.097235] Stack: [ 1533.097237] [] do_invalid_op+0x90/0xb0 [ 1533.097239] ffffffff8106a060 0000000000000038 [] ? resched_task+0x61/0x70 [ 1533.097244] 000000004df8da13 0000000022d4e764 [] ? __enqueue_entity+0x80/0x90 [ 1533.097250] [ 1533.097251] <0> 000000004df8da13 [] ? enqueue_task_fair+0x103/0x140 [ 1533.097256] 0000000022d4e764 0000000000000046 [] invalid_op+0x15/0x20 [ 1533.097260] 0000000000000000 [ 1533.097262] <0> [] ? resched_task+0x61/0x70 [ 1533.097266] 0000000000000008 ffff880c39d41c20 [] task_tick_fair+0xdd/0xf0 [ 1533.097271] 0000000000000000 0000000000000000 [] scheduler_tick+0x123/0x180 [ 1533.097276] [ 1533.097277] Call Trace: [ 1533.097280] [] update_process_times+0x4b/0x70 [ 1533.097282] [ 1533.097285] [] tick_periodic+0x27/0x70 [ 1533.097289] [] ? __rcu_process_callbacks+0x80/0x320 [ 1533.097293] [] tick_handle_periodic+0x21/0x80 [ 1533.097296] [] run_local_timers+0x9/0x20 [ 1533.097300] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1533.097303] [] update_process_times+0x37/0x70 [ 1533.097307] [] apic_timer_interrupt+0x13/0x20 [ 1533.097310] [] tick_periodic+0x27/0x70 [ 1533.097312] [ 1533.097315] [] tick_handle_periodic+0x21/0x80 [ 1533.097319] [] ? lock_kernel+0x2f/0x40 [ 1533.097322] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1533.097326] [] ? __dequeue_signal+0x110/0x160 [ 1533.097330] [] apic_timer_interrupt+0x13/0x20 [ 1533.097333] [] ? recalc_sigpending+0x12/0x30 [ 1533.097335] [] ? get_signal_to_deliver+0x185/0x340 [ 1533.097341] [] ? mwait_idle+0x52/0x70 [ 1533.097344] [] ? do_notify_resume+0xb4/0x7d0 [ 1533.097347] [] ? enter_idle+0x20/0x30 [ 1533.097351] [] ? __bad_area_nosemaphore+0xbf/0x1c0 [ 1533.097354] [] ? destroy_compound_page+0x70/0xb0 [ 1533.097357] [] ? cpu_idle+0x45/0x70 [ 1533.097361] [] ? __free_pages+0x18/0x30 [ 1533.097366] [] ? start_secondary+0x168/0x1c0 [ 1533.097370] [] ? __bad_area+0x4f/0x70 [ 1533.097372] Code: 78 [] ? bad_area+0xe/0x10 [ 1533.097376] 4d 8b [] ? do_page_fault+0x1b4/0x260 [ 1533.097380] 74 24 [] ? retint_signal+0x3d/0x83 [ 1533.097384] 30 4d 8b 2e 9c 58 f6 c4 02 0f 85 89 00 00 00 31 c9 ba 02 00 00 00 4c 89 f6 4c 89 e7 e8 28 fc ff ff 49 8b 44 24 28 <41> fe 45 00 4c 89 e7 ff d0 4c 89 ef 89 c3 e8 b0 e1 2e 00 85 db [ 1533.097398] RIP [] hrtimer_run_queues+0x14d/0x210 [ 1533.097401] RSP [ 1533.097402] ---[ end trace e16f43a7fa6f9a7b ]--- [ 1533.097403] Kernel panic - not syncing: Fatal exception in interrupt [ 1533.097405] Pid: 0, comm: swapper Tainted: G D 2.6.32.41-zimbraI #1 [ 1533.097406] Call Trace: [ 1533.097407] [] panic+0xa0/0x170 [ 1533.097412] [] ? show_registers+0x86/0x260 [ 1533.097415] [] ? down_trylock+0x36/0x50 [ 1533.097417] [] ? console_unblank+0x1f/0x80 [ 1533.097419] [] ? print_oops_end_marker+0x1e/0x20 [ 1533.097422] [] oops_end+0x9f/0xb0 [ 1533.097424] [] die+0x56/0x90 [ 1533.097426] [] ? __pollwait+0x0/0x100 [ 1533.097429] [] do_general_protection+0x148/0x150 [ 1533.097432] [] general_protection+0x1f/0x30 [ 1533.097434] [] ? __pollwait+0x0/0x100 [ 1533.097436] [] ? __pollwait+0x10/0x100 [ 1533.097438] [] ? hrtimer_run_queues+0x14d/0x210 [ 1533.097441] [] ? hrtimer_run_queues+0x148/0x210 [ 1533.097444] [] ? __rcu_process_callbacks+0x80/0x320 [ 1533.097446] [] run_local_timers+0x9/0x20 [ 1533.097449] [] update_process_times+0x37/0x70 [ 1533.097451] [] tick_periodic+0x27/0x70 [ 1533.097453] [] tick_handle_periodic+0x21/0x80 [ 1533.097456] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1533.097458] [] apic_timer_interrupt+0x13/0x20 [ 1533.097459] [] ? mwait_idle+0x52/0x70 [ 1533.097463] [] ? enter_idle+0x20/0x30 [ 1533.097465] [] ? cpu_idle+0x45/0x70 [ 1533.097467] [] ? start_secondary+0x168/0x1c0 [ 1562.890899] Kernel panic - not syncing: Fatal exception in interrupt [ 1563.004269] Pid: 1514, comm: Tainted: G D 2.6.32.41-zimbraI #1 [ 1563.120857] Call Trace: [ 1563.187061] [] panic+0xa0/0x170 [ 1563.289293] [] ? up+0x35/0x50 [ 1563.381992] [] ? release_console_sem+0x1c0/0x1e0 [ 1563.494825] [] ? console_unblank+0x65/0x80 [ 1563.601234] [] ? print_oops_end_marker+0x1e/0x20 [ 1563.714056] [] oops_end+0x9f/0xb0 [ 1563.811354] [] no_context+0x15a/0x250 [ 1563.912768] [] __bad_area_nosemaphore+0xdb/0x1c0 [ 1564.025633] [] bad_area_nosemaphore+0xe/0x10 [ 1564.134185] [] do_page_fault+0x9f/0x260 [ 1564.237438] [] page_fault+0x1f/0x30 [ 1564.336434] [] ? hrtimer_run_queues+0x128/0x210 [ 1564.448054] [] ? hrtimer_run_queues+0xec/0x210 [ 1564.558378] [] ? __rcu_process_callbacks+0x80/0x320 [ 1564.674098] [] run_local_timers+0x9/0x20 [ 1564.777966] [] update_process_times+0x37/0x70 [ 1564.886688] [] tick_periodic+0x27/0x70 [ 1564.988144] [] tick_handle_periodic+0x21/0x80 [ 1565.096979] [] smp_apic_timer_interrupt+0x67/0xa0 [ 1565.210093] [] apic_timer_interrupt+0x13/0x20 --------------080303030300050101050100-- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/