Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1422981AbXBAUSq (ORCPT ); Thu, 1 Feb 2007 15:18:46 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1422982AbXBAUSq (ORCPT ); Thu, 1 Feb 2007 15:18:46 -0500 Received: from omx2-ext.sgi.com ([192.48.171.19]:59344 "EHLO omx2.sgi.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1422984AbXBAUSp (ORCPT ); Thu, 1 Feb 2007 15:18:45 -0500 Date: Thu, 1 Feb 2007 12:18:01 -0800 (PST) From: Christoph Lameter To: Jens Axboe cc: Andrew Morton , David Chinner , linux-kernel@vger.kernel.org Subject: Re: 2.6.20-rc6-mm3 In-Reply-To: <20070201191857.GQ10305@kernel.dk> Message-ID: References: <20070129204528.eb8d695e.akpm@osdl.org> <20070131162422.6bccc52c.akpm@osdl.org> <20070131163638.290f40c1.akpm@osdl.org> <20070201062018.GC33919298@melbourne.sgi.com> <20070131231253.fdebc9f5.akpm@osdl.org> <20070201191857.GQ10305@kernel.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 18664 Lines: 360 On Thu, 1 Feb 2007, Jens Axboe wrote: > for xfs_buf_wait_unpin() and xfs_buf_lock(). Does this fix it? No it still hangs consistently. This time at an earlier spot. All OS INIT slaves have reached rendezvous Processes interrupted by INIT - 0 (cpu 0 task 0xa000000100b24000) 0 (cpu 1 task 0xe00000b003bd8000) 0 (cpu 2 task 0xe000023c38248000) 0 (cpu 3 task 0xe00000b003d00000) 0 (cpu 4 task 0xe000023c38258000) 0 (cpu 5 task 0xe00000b003d10000) 0 (cpu 6 task 0xe000023c38268000) 0 (cpu 7 task 0xe00000b003d20000) 0 (cpu 8 task 0xe000023c382e8000) 0 (cpu 9 task 0xe00000b003d30000) 0 (cpu 10 task 0xe000023c38380000) 0 (cpu 11 task 0xe00000b003d40000) Backtrace of pid 223 (pdflush) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e0000030156879d0 bsp=e0000030156811a8 [] synchronize_qrcu+0x170/0x1e0 sp=e000003015687ae0 bsp=e000003015681170 [] __make_request+0x160/0x880 sp=e000003015687b10 bsp=e000003015681130 [] generic_make_request+0x4a0/0x520 sp=e000003015687b30 bsp=e0000030156810f8 [] submit_bio+0x2f0/0x320 sp=e000003015687b50 bsp=e0000030156810b0 [] xfs_buf_iorequest+0x740/0x820 sp=e000003015687b70 bsp=e000003015681040 [] xlog_bdstrat_cb+0x50/0xe0 sp=e000003015687ba0 bsp=e000003015681020 [] xlog_state_release_iclog+0x770/0xcc0 sp=e000003015687ba0 bsp=e000003015680fc0 [] xlog_state_sync_all+0x1c0/0x460 sp=e000003015687ba0 bsp=e000003015680f60 [] _xfs_log_force+0xd0/0x5c0 sp=e000003015687bd0 bsp=e000003015680f00 [] xfs_syncsub+0x40/0x520 sp=e000003015687c00 bsp=e000003015680eb0 [] xfs_sync+0x70/0xa0 sp=e000003015687c00 bsp=e000003015680e88 [] vfs_sync+0xa0/0xc0 sp=e000003015687c00 bsp=e000003015680e58 [] xfs_fs_write_super+0x70/0xa0 sp=e000003015687c00 bsp=e000003015680e38 [] sync_supers+0x150/0x260 sp=e000003015687c00 bsp=e000003015680e08 [] wb_kupdate+0x60/0x280 sp=e000003015687c00 bsp=e000003015680dc8 [] pdflush+0x330/0x4e0 sp=e000003015687c50 bsp=e000003015680d90 [] kthread+0x220/0x2a0 sp=e000003015687d50 bsp=e000003015680d48 [] kernel_thread_helper+0xd0/0x100 sp=e000003015687e30 bsp=e000003015680d20 [] start_kernel_thread+0x20/0x40 sp=e000003015687e30 bsp=e000003015680d20 Backtrace of pid 1006 (xfsbufd) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00002bc3a23fc00 bsp=e00002bc3a238e40 [] schedule_timeout+0x110/0x180 sp=e00002bc3a23fd10 bsp=e00002bc3a238e10 [] schedule_timeout_interruptible+0x30/0x60 sp=e00002bc3a23fd40 bsp=e00002bc3a238de8 [] xfsbufd+0x1b0/0x5e0 sp=e00002bc3a23fd40 bsp=e00002bc3a238d90 [] kthread+0x220/0x2a0 sp=e00002bc3a23fd50 bsp=e00002bc3a238d48 [] kernel_thread_helper+0xd0/0x100 sp=e00002bc3a23fe30 bsp=e00002bc3a238d20 [] start_kernel_thread+0x20/0x40 sp=e00002bc3a23fe30 bsp=e00002bc3a238d20 Backtrace of pid 1007 (xfssyncd) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00002bc3a29fc00 bsp=e00002bc3a298e20 [] schedule_timeout+0x110/0x180 sp=e00002bc3a29fd10 bsp=e00002bc3a298de8 [] schedule_timeout_interruptible+0x30/0x60 sp=e00002bc3a29fd40 bsp=e00002bc3a298dc8 [] xfssyncd+0xb0/0x400 sp=e00002bc3a29fd40 bsp=e00002bc3a298d90 [] kthread+0x220/0x2a0 sp=e00002bc3a29fd50 bsp=e00002bc3a298d48 [] kernel_thread_helper+0xd0/0x100 sp=e00002bc3a29fe30 bsp=e00002bc3a298d20 [] start_kernel_thread+0x20/0x40 sp=e00002bc3a29fe30 bsp=e00002bc3a298d20 Backtrace of pid 1010 (boot) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00000b078797bd0 bsp=e00000b078791090 [] pipe_wait+0xc0/0x120 sp=e00000b078797ce0 bsp=e00000b078791068 [] pipe_read+0x730/0x820 sp=e00000b078797d10 bsp=e00000b078790fb0 [] do_sync_read+0x180/0x200 sp=e00000b078797d10 bsp=e00000b078790f78 [] vfs_read+0x1b0/0x340 sp=e00000b078797e20 bsp=e00000b078790f28 [] sys_read+0x70/0xe0 sp=e00000b078797e20 bsp=e00000b078790eb0 [] ia64_ret_from_syscall+0x0/0x20 sp=e00000b078797e30 bsp=e00000b078790eb0 [] __kernel_syscall_via_break+0x0/0x20 sp=e00000b078798000 bsp=e00000b078790eb0 Backtrace of pid 1031 (startpar) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e000003015867950 bsp=e000003015861000 [] schedule_timeout+0x40/0x180 sp=e000003015867a60 bsp=e000003015860fc8 [] do_select+0x360/0x840 sp=e000003015867a90 bsp=e000003015860ee8 [] sys_select+0x610/0x9e0 sp=e000003015867ce0 bsp=e000003015860e50 [] ia64_ret_from_syscall+0x0/0x20 sp=e000003015867e30 bsp=e000003015860e50 [] __kernel_syscall_via_break+0x0/0x20 sp=e000003015868000 bsp=e000003015860e50 Backtrace of pid 1063 (udevd) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e000003015cbf950 bsp=e000003015cb8f78 [] schedule_timeout+0x40/0x180 sp=e000003015cbfa60 bsp=e000003015cb8f48 [] do_select+0x360/0x840 sp=e000003015cbfa90 bsp=e000003015cb8e68 [] sys_select+0x610/0x9e0 sp=e000003015cbfce0 bsp=e000003015cb8dc8 [] ia64_ret_from_syscall+0x0/0x20 sp=e000003015cbfe30 bsp=e000003015cb8dc8 [] __kernel_syscall_via_break+0x0/0x20 sp=e000003015cc0000 bsp=e000003015cb8dc8 Backtrace of pid 2061 (boot.md) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00000300b97fcf0 bsp=e00000300b979028 [] do_wait+0x1c50/0x20e0 sp=e00000300b97fe00 bsp=e00000300b978fa8 [] sys_wait4+0x60/0x80 sp=e00000300b97fe30 bsp=e00000300b978f50 [] ia64_ret_from_syscall+0x0/0x20 sp=e00000300b97fe30 bsp=e00000300b978f50 [] __kernel_syscall_via_break+0x0/0x20 sp=e00000300b980000 bsp=e00000300b978f50 Backtrace of pid 2063 (lk) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00000300bc8fcf0 bsp=e00000300bc89028 [] do_wait+0x1c50/0x20e0 sp=e00000300bc8fe00 bsp=e00000300bc88fa8 [] sys_wait4+0x60/0x80 sp=e00000300bc8fe30 bsp=e00000300bc88f50 [] ia64_ret_from_syscall+0x0/0x20 sp=e00000300bc8fe30 bsp=e00000300bc88f50 [] __kernel_syscall_via_break+0x0/0x20 sp=e00000300bc90000 bsp=e00000300bc88f50 Backtrace of pid 2089 (lk_bios) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00001b03852f8b0 bsp=e00001b038529618 [] schedule_timeout+0x40/0x180 sp=e00001b03852f9c0 bsp=e00001b0385295e0 [] _xfs_log_force+0x500/0x5c0 sp=e00001b03852f9f0 bsp=e00001b038529580 [] xfs_alloc_search_busy+0x190/0x1e0 sp=e00001b03852fa20 bsp=e00001b038529538 [] xfs_alloc_ag_vextent+0x2250/0x2420 sp=e00001b03852fa20 bsp=e00001b0385294a8 [] xfs_alloc_vextent+0x690/0x9c0 sp=e00001b03852fa60 bsp=e00001b038529428 [] xfs_bmapi+0x1e00/0x33a0 sp=e00001b03852fa60 bsp=e00001b0385292d8 [] xfs_iomap_write_allocate+0x3f0/0x760 sp=e00001b03852fba0 bsp=e00001b038529248 [] xfs_iomap+0x670/0xb00 sp=e00001b03852fc30 bsp=e00001b0385291d0 [] xfs_bmap+0x40/0x60 sp=e00001b03852fc80 bsp=e00001b038529188 [] xfs_map_blocks+0xa0/0x120 sp=e00001b03852fc80 bsp=e00001b038529148 [] xfs_page_state_convert+0x540/0x1a40 sp=e00001b03852fc90 bsp=e00001b038529080 [] xfs_vm_writepage+0x180/0x220 sp=e00001b03852fd50 bsp=e00001b038529040 [] generic_writepages+0x420/0x800 sp=e00001b03852fd60 bsp=e00001b038528fc0 [] xfs_vm_writepages+0x90/0xc0 sp=e00001b03852fdf0 bsp=e00001b038528f88 [] do_writepages+0xb0/0x120 sp=e00001b03852fdf0 bsp=e00001b038528f58 [] __filemap_fdatawrite_range+0xb0/0xe0 sp=e00001b03852fdf0 bsp=e00001b038528f20 [] filemap_fdatawrite+0x40/0x60 sp=e00001b03852fe30 bsp=e00001b038528f00 [] fs_flush_pages+0xc0/0x100 sp=e00001b03852fe30 bsp=e00001b038528eb8 [] xfs_close+0x190/0x1e0 sp=e00001b03852fe30 bsp=e00001b038528e80 [] xfs_file_close+0xa0/0xc0 sp=e00001b03852fe30 bsp=e00001b038528e60 [] filp_close+0xd0/0x140 sp=e00001b03852fe30 bsp=e00001b038528e30 [] sys_close+0x140/0x1e0 sp=e00001b03852fe30 bsp=e00001b038528db0 [] ia64_ret_from_syscall+0x0/0x20 sp=e00001b03852fe30 bsp=e00001b038528db0 [] __kernel_syscall_via_break+0x0/0x20 sp=e00001b038530000 bsp=e00001b038528db0 Backtrace of pid 2091 (mdrun) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e000013038907bd0 bsp=e000013038901090 [] pipe_wait+0xc0/0x120 sp=e000013038907ce0 bsp=e000013038901068 [] pipe_read+0x730/0x820 sp=e000013038907d10 bsp=e000013038900fb0 [] do_sync_read+0x180/0x200 sp=e000013038907d10 bsp=e000013038900f78 [] vfs_read+0x1b0/0x340 sp=e000013038907e20 bsp=e000013038900f28 [] sys_read+0x70/0xe0 sp=e000013038907e20 bsp=e000013038900eb0 [] ia64_ret_from_syscall+0x0/0x20 sp=e000013038907e30 bsp=e000013038900eb0 [] __kernel_syscall_via_break+0x0/0x20 sp=e000013038908000 bsp=e000013038900eb0 Backtrace of pid 2134 (mdrun) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00002b83b9d7cf0 bsp=e00002b83b9d1028 [] do_wait+0x1c50/0x20e0 sp=e00002b83b9d7e00 bsp=e00002b83b9d0fa8 [] sys_wait4+0x60/0x80 sp=e00002b83b9d7e30 bsp=e00002b83b9d0f50 [] ia64_ret_from_syscall+0x0/0x20 sp=e00002b83b9d7e30 bsp=e00002b83b9d0f50 [] __kernel_syscall_via_break+0x0/0x20 sp=e00002b83b9d8000 bsp=e00002b83b9d0f50 Backtrace of pid 2135 (mdadm) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e000003015c07b40 bsp=e000003015c01250 [] io_schedule+0xf0/0x120 sp=e000003015c07c50 bsp=e000003015c01228 [] io_wait_schedule+0x50/0x80 sp=e000003015c07c50 bsp=e000003015c01208 [] sleep_on_page+0x20/0x40 sp=e000003015c07c50 bsp=e000003015c011d8 [] __wait_on_bit_lock+0xb0/0x1c0 sp=e000003015c07c50 bsp=e000003015c01190 [] lock_page_blocking+0x70/0xa0 sp=e000003015c07c50 bsp=e000003015c01168 [] generic_writepages+0x2d0/0x800 sp=e000003015c07c50 bsp=e000003015c010f0 [] xfs_vm_writepages+0x90/0xc0 sp=e000003015c07ce0 bsp=e000003015c010b8 [] do_writepages+0xb0/0x120 sp=e000003015c07ce0 bsp=e000003015c01088 [] __writeback_single_inode+0x450/0x8a0 sp=e000003015c07ce0 bsp=e000003015c01028 [] generic_sync_sb_inodes+0x4d0/0x740 sp=e000003015c07d20 bsp=e000003015c00fc0 [] sync_sb_inodes+0x90/0xc0 sp=e000003015c07d20 bsp=e000003015c00f98 [] sync_inodes_sb+0x120/0x140 sp=e000003015c07d20 bsp=e000003015c00f70 [] __fsync_super+0x30/0x1c0 sp=e000003015c07d60 bsp=e000003015c00f50 [] fsync_super+0x30/0x60 sp=e000003015c07d60 bsp=e000003015c00f30 [] fsync_bdev+0x50/0xc0 sp=e000003015c07d60 bsp=e000003015c00f08 [] blkdev_ioctl+0x100/0x11e0 sp=e000003015c07d60 bsp=e000003015c00e98 [] block_ioctl+0x40/0x60 sp=e000003015c07e10 bsp=e000003015c00e68 [] do_ioctl+0x90/0x180 sp=e000003015c07e10 bsp=e000003015c00e28 [] vfs_ioctl+0x880/0x8e0 sp=e000003015c07e10 bsp=e000003015c00dd8 [] sys_ioctl+0x60/0xc0 sp=e000003015c07e20 bsp=e000003015c00d60 [] ia64_ret_from_syscall+0x0/0x20 sp=e000003015c07e30 bsp=e000003015c00d60 [] __kernel_syscall_via_break+0x0/0x20 sp=e000003015c08000 bsp=e000003015c00d60 Backtrace of pid 2136 (grep) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e00000300bd67bd0 bsp=e00000300bd60f08 [] pipe_wait+0xc0/0x120 sp=e00000300bd67ce0 bsp=e00000300bd60ee0 [] pipe_read+0x730/0x820 sp=e00000300bd67d10 bsp=e00000300bd60e30 [] do_sync_read+0x180/0x200 sp=e00000300bd67d10 bsp=e00000300bd60df0 [] vfs_read+0x1b0/0x340 sp=e00000300bd67e20 bsp=e00000300bd60da0 [] sys_read+0x70/0xe0 sp=e00000300bd67e20 bsp=e00000300bd60d28 [] ia64_ret_from_syscall+0x0/0x20 sp=e00000300bd67e30 bsp=e00000300bd60d28 [] __kernel_syscall_via_break+0x0/0x20 sp=e00000300bd68000 bsp=e00000300bd60d28 Backtrace of pid 2137 (sed) Call Trace: [] schedule+0x1bf0/0x1ec0 sp=e000003015acfbd0 bsp=e000003015ac9008 [] pipe_wait+0xc0/0x120 sp=e000003015acfce0 bsp=e000003015ac8fd8 [] pipe_read+0x730/0x820 sp=e000003015acfd10 bsp=e000003015ac8f28 [] do_sync_read+0x180/0x200 sp=e000003015acfd10 bsp=e000003015ac8ef0 [] vfs_read+0x1b0/0x340 sp=e000003015acfe20 bsp=e000003015ac8ea0 [] sys_read+0x70/0xe0 sp=e000003015acfe20 bsp=e000003015ac8e28 [] ia64_ret_from_syscall+0x0/0x20 sp=e000003015acfe30 bsp=e000003015ac8e28 [] __kernel_syscall_via_break+0x0/0x20 sp=e000003015ad0000 bsp=e000003015ac8e28 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/