Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934759Ab3E1QMh (ORCPT ); Tue, 28 May 2013 12:12:37 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43977 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934619Ab3E1QMf (ORCPT ); Tue, 28 May 2013 12:12:35 -0400 Date: Tue, 28 May 2013 12:12:30 -0400 From: Dave Jones To: xfs@oss.sgi.com Cc: Linux Kernel Subject: 3.10-rc3 xfs mount/recovery failure & ext fsck hang. Message-ID: <20130528161230.GA7577@redhat.com> Mail-Followup-To: Dave Jones , xfs@oss.sgi.com, Linux Kernel MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 12464 Lines: 174 box crashed, and needed rebooting. On next bootup, when it found the dirty partition, xfs chose to spew and then hang instead of replaying the journal and mounting :( Dave [ 14.694731] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, debug enabled [ 14.722328] XFS (sda2): Mounting Filesystem [ 14.757801] XFS (sda2): Starting recovery (logdev: internal) [ 14.782049] XFS: Assertion failed: fs_is_ok, file: fs/xfs/xfs_dir2_data.c, line: 169 [ 14.783989] ------------[ cut here ]------------ [ 14.785547] kernel BUG at fs/xfs/xfs_message.c:108! [ 14.787169] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 14.789038] Modules linked in: xfs libcrc32c coretemp hwmon kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr snd_hda_codec_realtek snd_hda_codec_hdmi e1000e usb_debug snd_hda_intel snd_hda_codec snd_seq ptp snd_seq_device snd_hwdep pps_core snd_pcm snd_page_alloc snd_timer snd soundcore binfmt_misc [ 14.795850] CPU: 1 PID: 300 Comm: mount Not tainted 3.10.0-rc3+ #39 [ 14.800660] task: ffff88022b9e4a40 ti: ffff880240e1a000 task.ti: ffff880240e1a000 [ 14.802737] RIP: 0010:[] [] assfail+0x22/0x30 [xfs] [ 14.802738] RSP: 0018:ffff880240e1b868 EFLAGS: 00010296 [ 14.802739] RAX: 0000000000000048 RBX: ffff88022b653020 RCX: 0000000000000006 [ 14.802739] RDX: 0000000000003900 RSI: ffff88022b9e51c0 RDI: ffff88022b9e4a40 [ 14.802740] RBP: ffff880240e1b868 R08: 0000000000000001 R09: 0000000000000000 [ 14.802740] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880241301bd8 [ 14.802741] R13: 0000000000000000 R14: ffff88022b653ed8 R15: ffff88022b653ed8 [ 14.802741] FS: 00007fecff34a880(0000) GS:ffff880244a00000(0000) knlGS:0000000000000000 [ 14.802742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 14.802743] CR2: 00007f5bc7bfca60 CR3: 000000022b9f4000 CR4: 00000000001407e0 [ 14.802743] Stack: [ 14.802745] ffff880240e1b8d8 ffffffffa029a1a0 000000022b5dcb40 ffff88022b653000 [ 14.802746] ffff88022b653ff8 ffff88022b653004 ffff880200000004 ffff88022b653029 [ 14.802747] ffff880200000002 ffff88022a86de00 ffff88022b653000 ffff88022b653000 [ 14.802748] Call Trace: [ 14.802761] [] __xfs_dir3_data_check+0x5a0/0x6c0 [xfs] [ 14.802773] [] xfs_dir3_block_verify+0x7a/0x90 [xfs] [ 14.802783] [] xfs_dir3_block_write_verify+0x35/0x120 [xfs] [ 14.802791] [] ? xfs_bdstrat_cb+0x55/0x1a0 [xfs] [ 14.802797] [] _xfs_buf_ioapply+0x6b/0x370 [xfs] [ 14.802801] [] ? wake_up_state+0x20/0x20 [ 14.802807] [] ? xfs_bdstrat_cb+0x55/0x1a0 [xfs] [ 14.802814] [] xfs_buf_iorequest+0x6b/0x1b0 [xfs] [ 14.802820] [] xfs_bdstrat_cb+0x55/0x1a0 [xfs] [ 14.802826] [] __xfs_buf_delwri_submit+0x1ed/0x2f0 [xfs] [ 14.802833] [] xfs_buf_delwri_submit+0x30/0x90 [xfs] [ 14.802847] [] xlog_recover_commit_trans+0xda/0x130 [xfs] [ 14.802858] [] xlog_recover_process_data+0x179/0x2a0 [xfs] [ 14.802870] [] xlog_do_recovery_pass+0x4dd/0x760 [xfs] [ 14.802880] [] xlog_do_log_recovery+0x9b/0x130 [xfs] [ 14.802890] [] xlog_do_recover+0x16/0x1c0 [xfs] [ 14.802900] [] xlog_recover+0x82/0xc0 [xfs] [ 14.802913] [] xfs_log_mount+0xa0/0x1a0 [xfs] [ 14.802924] [] xfs_mountfs+0x3b9/0x800 [xfs] [ 14.802933] [] xfs_fs_fill_super+0x308/0x390 [xfs] [ 14.802936] [] mount_bdev+0x1c0/0x200 [ 14.802944] [] ? xfs_finish_flags+0x130/0x130 [xfs] [ 14.802946] [] ? __kmalloc_track_caller+0x218/0x360 [ 14.802955] [] xfs_fs_mount+0x15/0x20 [xfs] [ 14.802957] [] mount_fs+0x39/0x1b0 [ 14.802959] [] ? __alloc_percpu+0x10/0x20 [ 14.802961] [] vfs_kern_mount+0x63/0xf0 [ 14.802963] [] do_mount+0x23e/0xa20 [ 14.802965] [] ? strndup_user+0x4b/0xf0 [ 14.802967] [] SyS_mount+0x83/0xc0 [ 14.802969] [] tracesys+0xdd/0xe2 [ 14.802984] Code: 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 f1 41 89 d0 48 89 e5 48 89 fa 48 c7 c6 20 67 2e a0 31 ff 31 c0 e8 ce fb ff ff <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 [ 14.802992] RIP [] assfail+0x22/0x30 [xfs] [ 14.802992] RSP [ 40.642521] BUG: soft lockup - CPU#0 stuck for 22s! [fsck.ext2:294] [ 40.644139] Modules linked in: xfs libcrc32c coretemp hwmon kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr snd_hda_codec_realtek snd_hda_codec_hdmi e1000e usb_debug snd_hda_intel snd_hda_codec snd_seq ptp snd_seq_device snd_hwdep pps_core snd_pcm snd_page_alloc snd_timer snd soundcore binfmt_misc [ 40.649456] irq event stamp: 141855 [ 40.650742] hardirqs last enabled at (141855): [] retint_swapgs+0xe/0x13 [ 40.652618] hardirqs last disabled at (141854): [] int_with_check+0x53/0x73 [ 40.654515] softirqs last enabled at (137936): [] __do_softirq+0x185/0x430 [ 40.656410] softirqs last disabled at (137913): [] irq_exit+0xcd/0xe0 [ 40.658222] CPU: 0 PID: 294 Comm: fsck.ext2 Tainted: G D 3.10.0-rc3+ #39 [ 40.662692] task: ffff88023d5fca40 ti: ffff8802413e2000 task.ti: ffff8802413e2000 [ 40.664421] RIP: 0010:[] [] smp_call_function_many+0x2be/0x360 [ 40.666343] RSP: 0018:ffff8802413e3d10 EFLAGS: 00000202 [ 40.667808] RAX: 0000000000000001 RBX: ffffffff816ecd20 RCX: ffff880244bd8008 [ 40.669484] RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000 [ 40.671159] RBP: ffff8802413e3d68 R08: ffff8802444287e0 R09: 0000000000000000 [ 40.672860] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8802413e3c88 [ 40.674536] R13: ffff8802449d3f00 R14: ffff8802413e2000 R15: ffff88023d5fca40 [ 40.676215] FS: 00007f784e690780(0000) GS:ffff880244800000(0000) knlGS:0000000000000000 [ 40.678014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 40.679539] CR2: 0000003159c4b102 CR3: 00000002413b7000 CR4: 00000000001407f0 [ 40.681225] Stack: [ 40.682307] 0000000144404240 00000000001d4a80 0000000000000000 ffffffff811e2590 [ 40.684074] ffff880244fd4a80 0000000000000282 ffff88022a760fc0 0000000000000000 [ 40.685838] ffffffff811e2590 0000000000000000 0000000000000001 ffff8802413e3da0 [ 40.687607] Call Trace: [ 40.688754] [] ? __bforget+0x70/0x70 [ 40.690222] [] ? __bforget+0x70/0x70 [ 40.691684] [] on_each_cpu_mask+0x42/0xb0 [ 40.693241] [] ? mark_buffer_async_write+0x20/0x20 [ 40.694847] [] on_each_cpu_cond+0xe4/0x250 [ 40.696366] [] ? __bforget+0x70/0x70 [ 40.697828] [] invalidate_bh_lrus+0x29/0x30 [ 40.699358] [] kill_bdev+0x21/0x30 [ 40.700800] [] __blkdev_put+0x69/0x1b0 [ 40.702279] [] blkdev_put+0x50/0x160 [ 40.703745] [] blkdev_close+0x28/0x30 [ 40.705209] [] __fput+0xf5/0x2d0 [ 40.706619] [] ____fput+0xe/0x10 [ 40.708022] [] task_work_run+0xac/0xe0 [ 40.709487] [] do_notify_resume+0x69/0x90 [ 40.710983] [] int_signal+0x12/0x17 [ 40.712417] Code: 00 41 89 c4 39 f0 0f 8d d5 fd ff ff 48 63 d0 48 8b 0b 48 03 0c d5 40 50 cf 81 f6 41 20 01 74 14 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 e1 6a c3 00 83 f8 ff 48 8b 7b 08 74 b0 39 c6 77 [ 44.790350] ---[ end trace b71cb3732bfc4a50 ]--- [ 44.791921] ------------[ cut here ]------------ [ 44.793459] WARNING: at kernel/exit.c:715 do_exit+0x55/0xcd0() [ 44.795138] Modules linked in: xfs libcrc32c coretemp hwmon kvm_intel kvm crc32c_intel ghash_clmulni_intel microcode pcspkr snd_hda_codec_realtek snd_hda_codec_hdmi e1000e usb_debug snd_hda_intel snd_hda_codec snd_seq ptp snd_seq_device snd_hwdep pps_core snd_pcm snd_page_alloc snd_timer [ 44.800315] [sched_delayed] sched: RT throttling activated [ 44.804099] snd soundcore binfmt_misc [ 44.806032] CPU: 1 PID: 300 Comm: mount Tainted: G D 3.10.0-rc3+ #39 [ 44.811587] ffffffff81a0e953 ffff880240e1b588 ffffffff816e375b ffff880240e1b5c0 [ 44.813612] ffffffff8104a1b1 000000000000000b ffff880240e1b7b8 ffff88022b9e4a40 [ 44.816016] 0000000000000000 0000000000000000 ffff880240e1b5d0 ffffffff8104a28a [ 44.818374] Call Trace: [ 44.819778] [] dump_stack+0x19/0x1b [ 44.821662] [] warn_slowpath_common+0x61/0x80 [ 44.823554] [] warn_slowpath_null+0x1a/0x20 [ 44.825390] [] do_exit+0x55/0xcd0 [ 44.827120] [] ? __const_udelay+0x29/0x30 [ 44.828875] [] ? __rcu_read_unlock+0x54/0xa0 [ 44.831417] [] ? kmsg_dump+0x1bd/0x230 [ 44.833132] [] ? kmsg_dump+0x25/0x230 [ 44.834803] [] oops_end+0x90/0xd0 [ 44.836404] [] die+0x4b/0x70 [ 44.838067] [] do_trap+0x60/0x170 [ 44.839693] [] do_invalid_op+0xa8/0xe0 [ 44.841508] [] ? assfail+0x22/0x30 [xfs] [ 44.841510] [] ? trace_hardirqs_off_thunk+0x3a/0x3c [ 44.841514] [] ? restore_args+0x30/0x30 [ 44.841515] [] invalid_op+0x18/0x20 [ 44.841524] [] ? assfail+0x22/0x30 [xfs] [ 44.841532] [] ? assfail+0x22/0x30 [xfs] [ 44.841545] [] __xfs_dir3_data_check+0x5a0/0x6c0 [xfs] [ 44.841557] [] xfs_dir3_block_verify+0x7a/0x90 [xfs] [ 44.841568] [] xfs_dir3_block_write_verify+0x35/0x120 [xfs] [ 44.841575] [] ? xfs_bdstrat_cb+0x55/0x1a0 [xfs] [ 44.841582] [] _xfs_buf_ioapply+0x6b/0x370 [xfs] [ 44.841585] [] ? wake_up_state+0x20/0x20 [ 44.841591] [] ? xfs_bdstrat_cb+0x55/0x1a0 [xfs] [ 44.841598] [] xfs_buf_iorequest+0x6b/0x1b0 [xfs] [ 44.841604] [] xfs_bdstrat_cb+0x55/0x1a0 [xfs] [ 44.841611] [] __xfs_buf_delwri_submit+0x1ed/0x2f0 [xfs] [ 44.841618] [] xfs_buf_delwri_submit+0x30/0x90 [xfs] [ 44.841632] [] xlog_recover_commit_trans+0xda/0x130 [xfs] [ 44.841644] [] xlog_recover_process_data+0x179/0x2a0 [xfs] [ 44.841656] [] xlog_do_recovery_pass+0x4dd/0x760 [xfs] [ 44.841667] [] xlog_do_log_recovery+0x9b/0x130 [xfs] [ 44.841678] [] xlog_do_recover+0x16/0x1c0 [xfs] [ 44.841688] [] xlog_recover+0x82/0xc0 [xfs] [ 44.841700] [] xfs_log_mount+0xa0/0x1a0 [xfs] [ 44.841712] [] xfs_mountfs+0x3b9/0x800 [xfs] [ 44.841721] [] xfs_fs_fill_super+0x308/0x390 [xfs] [ 44.841723] [] mount_bdev+0x1c0/0x200 [ 44.841732] [] ? xfs_finish_flags+0x130/0x130 [xfs] [ 44.841734] [] ? __kmalloc_track_caller+0x218/0x360 [ 44.841743] [] xfs_fs_mount+0x15/0x20 [xfs] [ 44.841745] [] mount_fs+0x39/0x1b0 [ 44.841747] [] ? __alloc_percpu+0x10/0x20 [ 44.841749] [] vfs_kern_mount+0x63/0xf0 [ 44.841751] [] do_mount+0x23e/0xa20 [ 44.841753] [] ? strndup_user+0x4b/0xf0 [ 44.841755] [] SyS_mount+0x83/0xc0 [ 44.841757] [] tracesys+0xdd/0xe2 [ 44.841795] ---[ end trace b71cb3732bfc4a51 ]--- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/