Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754697Ab1CQPvf (ORCPT ); Thu, 17 Mar 2011 11:51:35 -0400 Received: from mx1.redhat.com ([209.132.183.28]:24628 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754226Ab1CQPvc (ORCPT ); Thu, 17 Mar 2011 11:51:32 -0400 Date: Thu, 17 Mar 2011 11:51:21 -0400 From: Mike Snitzer To: Jens Axboe Cc: "linux-kernel@vger.kernel.org" , "hch@infradead.org" Subject: Re: [PATCH 05/10] block: remove per-queue plugging Message-ID: <20110317155120.GA9220@redhat.com> References: <1295659049-2688-1-git-send-email-jaxboe@fusionio.com> <1295659049-2688-6-git-send-email-jaxboe@fusionio.com> <20110303221353.GA10366@redhat.com> <4D761E0D.8050200@fusionio.com> <20110308202100.GA31744@redhat.com> <4D76912C.9040705@fusionio.com> <20110308220526.GA393@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110308220526.GA393@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 29337 Lines: 496 On Tue, Mar 08 2011 at 5:05pm -0500, Mike Snitzer wrote: > On Tue, Mar 08 2011 at 3:27pm -0500, > Jens Axboe wrote: > > > On 2011-03-08 21:21, Mike Snitzer wrote: > > > On Tue, Mar 08 2011 at 7:16am -0500, > > > Jens Axboe wrote: > > > > > >> On 2011-03-03 23:13, Mike Snitzer wrote: > > >>> I'm now hitting a lockdep issue, while running a 'for-2.6.39/stack-plug' > > >>> kernel, when I try an fsync heavy workload to a request-based mpath > > >>> device (the kernel ultimately goes down in flames, I've yet to look at > > >>> the crashdump I took) > > >> > > >> Mike, can you re-run with the current stack-plug branch? I've fixed the > > >> !CONFIG_BLOCK and rebase issues, and also added a change for this flush > > >> on schedule event. It's run outside of the runqueue lock now, so > > >> hopefully that should solve this one. > > > > > > Works for me, thanks. > > > > Super, thanks! Out of curiousity, did you use dm/md? > > Yes, I've been using a request-based DM multipath device. Against latest 'for-2.6.39/core', I just ran that same fsync heavy workload against XFS (ontop of a DM multipath volume). ffsb induced the following hangs (ripple effect causing NetworkManager to get hung up on this data-only XFS volume, etc): XFS mounting filesystem dm-0 Ending clean XFS mount for filesystem: dm-0 mount used greatest stack depth: 3296 bytes left ffsb used greatest stack depth: 2592 bytes left INFO: task kswapd0:23 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D ffff880037b8f6e0 3656 23 2 0x00000000 ffff880037b8f6d0 0000000000000046 ffff880037b8f630 ffffffff8107012f ffff880037b8e010 ffff880037b8ffd8 00000000001d21c0 ffff880037b90600 ffff880037b90998 ffff880037b90990 00000000001d21c0 00000000001d21c0 Call Trace: [] ? trace_hardirqs_off+0xd/0xf [] xlog_wait+0x60/0x78 [xfs] [] ? default_wake_function+0x0/0x14 [] ? _raw_spin_lock+0x62/0x69 [] xlog_state_get_iclog_space+0x9e/0x22c [xfs] [] xlog_write+0x171/0x4ae [xfs] [] ? kmem_alloc+0x69/0xb1 [xfs] [] ? __kmalloc+0x14e/0x160 [] xfs_log_write+0x54/0x7e [xfs] [] xfs_trans_commit_iclog+0x195/0x2d8 [xfs] [] ? trace_hardirqs_on+0xd/0xf [] _xfs_trans_commit+0xc9/0x206 [xfs] [] xfs_itruncate_finish+0x1fd/0x2bd [xfs] [] xfs_free_eofblocks+0x1ac/0x1f1 [xfs] [] xfs_inactive+0x108/0x3a6 [xfs] [] ? lockdep_init_map+0xa6/0x11b [] xfs_fs_evict_inode+0xf6/0xfe [xfs] [] evict+0x24/0x8c [] dispose_list+0x31/0xaf [] shrink_icache_memory+0x1e5/0x215 [] shrink_slab+0xe0/0x164 [] kswapd+0x5e7/0x9dc [] ? kswapd+0x0/0x9dc [] kthread+0xa0/0xa8 [] ? trace_hardirqs_on_caller+0x11d/0x141 [] kernel_thread_helper+0x4/0x10 [] ? restore_args+0x0/0x30 [] ? kthread+0x0/0xa8 [] ? kernel_thread_helper+0x0/0x10 4 locks held by kswapd0/23: #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x3d/0x164 #1: (iprune_sem){++++.-}, at: [] shrink_icache_memory+0x4a/0x215 #2: (xfs_iolock_reclaimable){+.+.-.}, at: [] xfs_ilock+0x30/0xb9 [xfs] #3: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] INFO: task NetworkManager:958 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. NetworkManager D ffff88007a481288 3312 958 1 0x00000000 ffff88007a481278 0000000000000046 ffff88007a4811d8 ffffffff8107012f ffff88007a480010 ffff88007a481fd8 00000000001d21c0 ffff88007b4f0f80 ffff88007b4f1318 ffff88007b4f1310 00000000001d21c0 00000000001d21c0 Call Trace: [] ? trace_hardirqs_off+0xd/0xf [] xlog_wait+0x60/0x78 [xfs] [] ? default_wake_function+0x0/0x14 [] ? _raw_spin_lock+0x62/0x69 [] xlog_state_get_iclog_space+0x9e/0x22c [xfs] [] xlog_write+0x171/0x4ae [xfs] [] ? kmem_alloc+0x69/0xb1 [xfs] [] ? __kmalloc+0x14e/0x160 [] xfs_log_write+0x54/0x7e [xfs] [] xfs_trans_commit_iclog+0x195/0x2d8 [xfs] [] ? mark_held_locks+0x52/0x70 [] ? kmem_cache_alloc+0xd1/0x145 [] ? trace_hardirqs_on_caller+0x11d/0x141 [] ? trace_hardirqs_on+0xd/0xf [] _xfs_trans_commit+0xc9/0x206 [xfs] [] xfs_bmap_finish+0x87/0x16a [xfs] [] xfs_itruncate_finish+0x19e/0x2bd [xfs] [] xfs_free_eofblocks+0x1ac/0x1f1 [xfs] [] xfs_inactive+0x108/0x3a6 [xfs] [] ? lockdep_init_map+0xa6/0x11b [] xfs_fs_evict_inode+0xf6/0xfe [xfs] [] evict+0x24/0x8c [] dispose_list+0x31/0xaf [] shrink_icache_memory+0x1e5/0x215 [] shrink_slab+0xe0/0x164 [] try_to_free_pages+0x27f/0x495 [] __alloc_pages_nodemask+0x4e3/0x767 [] ? trace_hardirqs_off_caller+0x1f/0x9e [] alloc_pages_current+0xa7/0xca [] __page_cache_alloc+0x85/0x8c [] __do_page_cache_readahead+0xdb/0x1df [] ra_submit+0x21/0x25 [] filemap_fault+0x176/0x396 [] ? trace_hardirqs_on+0xd/0xf [] __do_fault+0x54/0x354 [] ? mark_lock+0x2d/0x22d [] handle_pte_fault+0x2cf/0x6e8 [] ? __pte_alloc+0xc3/0xd0 [] handle_mm_fault+0xda/0xed [] do_page_fault+0x3b4/0x3d6 [] ? fsnotify_perm+0x69/0x75 [] ? security_file_permission+0x2e/0x33 [] ? trace_hardirqs_off_thunk+0x3a/0x3c [] page_fault+0x25/0x30 5 locks held by NetworkManager/958: #0: (&mm->mmap_sem){++++++}, at: [] do_page_fault+0x1c2/0x3d6 #1: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x3d/0x164 #2: (iprune_sem){++++.-}, at: [] shrink_icache_memory+0x4a/0x215 #3: (xfs_iolock_reclaimable){+.+.-.}, at: [] xfs_ilock+0x30/0xb9 [xfs] #4: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] INFO: task xfssyncd/dm-0:1346 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. xfssyncd/dm-0 D ffff880072cb1a20 4824 1346 2 0x00000000 ffff880072cb1a10 0000000000000046 ffff880072cb1970 ffffffff8107012f ffff880072cb0010 ffff880072cb1fd8 00000000001d21c0 ffff88007b22ca00 ffff88007b22cd98 ffff88007b22cd90 00000000001d21c0 00000000001d21c0 Call Trace: [] ? trace_hardirqs_off+0xd/0xf [] xlog_wait+0x60/0x78 [xfs] [] ? default_wake_function+0x0/0x14 [] ? _raw_spin_lock+0x62/0x69 [] xlog_state_get_iclog_space+0x9e/0x22c [xfs] [] xlog_write+0x171/0x4ae [xfs] [] ? sched_clock_local+0x1c/0x82 [] ? trace_hardirqs_off_caller+0x1f/0x9e [] ? mark_lock+0x2d/0x22d [] xfs_log_write+0x54/0x7e [xfs] [] xfs_trans_commit_iclog+0x195/0x2d8 [xfs] [] ? trace_hardirqs_on+0xd/0xf [] ? kmem_zone_alloc+0x69/0xb1 [xfs] [] ? xfs_trans_add_item+0x50/0x5c [xfs] [] _xfs_trans_commit+0xc9/0x206 [xfs] [] xfs_fs_log_dummy+0x76/0x7d [xfs] [] xfs_sync_worker+0x37/0x6f [xfs] [] xfssyncd+0x15b/0x1b1 [xfs] [] ? xfssyncd+0x0/0x1b1 [xfs] [] kthread+0xa0/0xa8 [] ? trace_hardirqs_on_caller+0x11d/0x141 [] kernel_thread_helper+0x4/0x10 [] ? restore_args+0x0/0x30 [] ? kthread+0x0/0xa8 [] ? kernel_thread_helper+0x0/0x10 no locks held by xfssyncd/dm-0/1346. INFO: task ffsb:1355 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ffsb D 000000010002503a 3648 1355 1322 0x00000000 ffff88007baffae8 0000000000000046 ffff88007baffa48 ffffffff00000000 ffff88007bafe010 ffff88007bafffd8 00000000001d21c0 ffff880071df4680 ffff880071df4a18 ffff880071df4a10 00000000001d21c0 00000000001d21c0 Call Trace: [] xlog_wait+0x60/0x78 [xfs] [] ? default_wake_function+0x0/0x14 [] ? _raw_spin_lock+0x62/0x69 [] xlog_state_get_iclog_space+0x9e/0x22c [xfs] [] xlog_write+0x171/0x4ae [xfs] [] ? __lock_acquire+0x3bc/0xd26 [] ? local_clock+0x41/0x5a [] ? pvclock_clocksource_read+0x4b/0xb4 [] xfs_log_write+0x54/0x7e [xfs] [] ? sched_clock_local+0x1c/0x82 [] xfs_trans_commit_iclog+0x195/0x2d8 [xfs] [] ? mark_held_locks+0x52/0x70 [] ? kmem_cache_alloc+0xd1/0x145 [] ? trace_hardirqs_on_caller+0x11d/0x141 [] ? trace_hardirqs_on+0xd/0xf [] ? kmem_zone_alloc+0x69/0xb1 [xfs] [] _xfs_trans_commit+0xc9/0x206 [xfs] [] xfs_file_fsync+0x166/0x1e6 [xfs] [] vfs_fsync_range+0x54/0x7c [] vfs_fsync+0x1c/0x1e [] do_fsync+0x2e/0x43 [] sys_fsync+0x10/0x14 [] system_call_fastpath+0x16/0x1b 2 locks held by ffsb/1355: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] INFO: task ffsb:1364 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ffsb D 0000000100024ff6 3776 1364 1322 0x00000000 ffff880027d25ae8 0000000000000046 ffff880027d25a48 ffffffff00000000 ffff880027d24010 ffff880027d25fd8 00000000001d21c0 ffff88002839c8c0 ffff88002839cc58 ffff88002839cc50 00000000001d21c0 00000000001d21c0 Call Trace: [] xlog_wait+0x60/0x78 [xfs] [] ? default_wake_function+0x0/0x14 [] ? _raw_spin_lock+0x62/0x69 [] xlog_state_get_iclog_space+0x9e/0x22c [xfs] [] xlog_write+0x171/0x4ae [xfs] [] ? kmem_alloc+0x69/0xb1 [xfs] [] ? __kmalloc+0x14e/0x160 [] xfs_log_write+0x54/0x7e [xfs] [] xfs_trans_commit_iclog+0x195/0x2d8 [xfs] [] ? mark_held_locks+0x52/0x70 [] ? kmem_cache_alloc+0xd1/0x145 [] ? trace_hardirqs_on_caller+0x11d/0x141 [] ? trace_hardirqs_on+0xd/0xf [] ? kmem_zone_alloc+0x69/0xb1 [xfs] [] _xfs_trans_commit+0xc9/0x206 [xfs] [] xfs_file_fsync+0x166/0x1e6 [xfs] [] vfs_fsync_range+0x54/0x7c [] vfs_fsync+0x1c/0x1e [] do_fsync+0x2e/0x43 [] sys_fsync+0x10/0x14 [] system_call_fastpath+0x16/0x1b 2 locks held by ffsb/1364: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] (and many more ffsb processes hung similar to the 2 above) I just attempted a git command against the root volume, it hung: git D 0000000100252022 3440 1471 1461 0x00000004 ffff88003ad611d8 0000000000000046 ffff88003ad61138 ffffffff00000000 ffff88003ad60010 ffff88003ad61fd8 00000000001d21c0 ffff88003b498d40 ffff88003b4990d8 ffff88003b4990d0 00000000001d21c0 00000000001d21c0 Call Trace: [] xlog_wait+0x60/0x78 [xfs] [] ? default_wake_function+0x0/0x14 [] ? _raw_spin_lock+0x62/0x69 [] xlog_state_get_iclog_space+0x9e/0x22c [xfs] [] xlog_write+0x171/0x4ae [xfs] [] ? kmem_alloc+0x69/0xb1 [xfs] [] ? __kmalloc+0x14e/0x160 [] xfs_log_write+0x54/0x7e [xfs] [] xfs_trans_commit_iclog+0x195/0x2d8 [xfs] [] ? mark_held_locks+0x52/0x70 [] ? kmem_cache_alloc+0xd1/0x145 [] ? trace_hardirqs_on_caller+0x11d/0x141 [] ? trace_hardirqs_on+0xd/0xf [] _xfs_trans_commit+0xc9/0x206 [xfs] [] xfs_bmap_finish+0x87/0x16a [xfs] [] xfs_itruncate_finish+0x19e/0x2bd [xfs] [] xfs_free_eofblocks+0x1ac/0x1f1 [xfs] [] xfs_inactive+0x108/0x3a6 [xfs] [] ? lockdep_init_map+0xa6/0x11b [] xfs_fs_evict_inode+0xf6/0xfe [xfs] [] evict+0x24/0x8c [] dispose_list+0x31/0xaf [] shrink_icache_memory+0x1e5/0x215 [] shrink_slab+0xe0/0x164 [] try_to_free_pages+0x27f/0x495 [] __alloc_pages_nodemask+0x4e3/0x767 [] ? trace_hardirqs_off_caller+0x1f/0x9e [] alloc_pages_current+0xa7/0xca [] __page_cache_alloc+0x85/0x8c [] __do_page_cache_readahead+0xdb/0x1df [] ? lock_release_holdtime+0x2c/0xd7 [] ra_submit+0x21/0x25 [] ondemand_readahead+0x1e3/0x1f6 [] page_cache_async_readahead+0x79/0x82 [] filemap_fault+0xc6/0x396 [] ? trace_hardirqs_on+0xd/0xf [] __do_fault+0x54/0x354 [] ? mark_lock+0x2d/0x22d [] handle_pte_fault+0x2cf/0x6e8 [] ? __pte_alloc+0xc3/0xd0 [] handle_mm_fault+0xda/0xed [] do_page_fault+0x3b4/0x3d6 [] ? trace_hardirqs_off_caller+0x1f/0x9e [] ? trace_hardirqs_off+0xd/0xf [] ? local_clock+0x41/0x5a [] ? trace_hardirqs_off_thunk+0x3a/0x3c [] page_fault+0x25/0x30 And here is the summary of all the locks (via sysrq-t): Showing all locks held in the system: 2 locks held by kworker/0:1/10: #0: (xfsdatad){++++..}, at: [] process_one_work+0x18a/0x37f #1: ((&ioend->io_work)){+.+...}, at: [] process_one_work+0x18a/0x37f 4 locks held by kswapd0/23: #0: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x3d/0x164 #1: (iprune_sem){++++.-}, at: [] shrink_icache_memory+0x4a/0x215 #2: (xfs_iolock_reclaimable){+.+.-.}, at: [] xfs_ilock+0x30/0xb9 [xfs] #3: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by multipathd/659: #0: (&u->readlock){+.+.+.}, at: [] unix_dgram_recvmsg+0x5a/0x27f 5 locks held by NetworkManager/958: #0: (&mm->mmap_sem){++++++}, at: [] do_page_fault+0x1c2/0x3d6 #1: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x3d/0x164 #2: (iprune_sem){++++.-}, at: [] shrink_icache_memory+0x4a/0x215 #3: (xfs_iolock_reclaimable){+.+.-.}, at: [] xfs_ilock+0x30/0xb9 [xfs] #4: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by agetty/1099: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by mingetty/1101: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by mingetty/1103: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by mingetty/1105: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by mingetty/1107: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by mingetty/1109: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by mingetty/1111: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 1 lock held by bash/1313: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x284/0x7ba 2 locks held by ffsb/1355: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1358: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by ffsb/1359: #0: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1362: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1364: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1365: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1366: #0: (xfs_iolock_active){++++.+}, at: [] xfs_ilock_nowait+0x2b/0xdf [xfs] #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1367: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by ffsb/1368: #0: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1371: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1372: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1373: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1374: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1375: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1376: #0: (xfs_iolock_active){++++.+}, at: [] xfs_ilock_nowait+0x2b/0xdf [xfs] #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1377: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1378: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1380: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1381: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1383: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 4 locks held by ffsb/1384: #0: (&sb->s_type->i_mutex_key#13/1){+.+.+.}, at: [] do_unlinkat+0x67/0x165 #1: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_unlink+0x4f/0xcc #2: (&(&ip->i_lock)->mr_lock/2){+.+...}, at: [] xfs_ilock+0x63/0xb9 [xfs] #3: (&(&ip->i_lock)->mr_lock/3){+.+...}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1385: #0: (xfs_iolock_active){++++.+}, at: [] xfs_ilock_nowait+0x2b/0xdf [xfs] #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1386: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1387: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1388: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: 2 locks held by ffsb/1389: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by ffsb/1390: #0: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1391: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1392: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1393: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by ffsb/1394: #0: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1395: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1396: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1397: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1398: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1399: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1400: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1402: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1403: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 1 lock held by ffsb/1404: #0: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1405: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1406: #0: (xfs_iolock_active){++++.+}, at: [] xfs_ilock_nowait+0x2b/0xdf [xfs] #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1407: #0: (xfs_iolock_active){++++.+}, at: [] xfs_ilock_nowait+0x2b/0xdf [xfs] #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1409: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1410: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1411: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1412: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1413: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1414: #0: (xfs_iolock_active){++++.+}, at: [] xfs_ilock_nowait+0x2b/0xdf [xfs] #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1416: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by ffsb/1417: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] vfs_fsync_range+0x47/0x7c #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 3 locks held by ffsb/1418: #0: (&sb->s_type->i_mutex_key#13){+.+.+.}, at: [] do_last+0xb8/0x2f9 #1: (&(&ip->i_lock)->mr_lock/1){+.+.+.}, at: [] xfs_ilock+0x63/0xb9 [xfs] #2: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock_nowait+0x59/0xdf [xfs] 2 locks held by flush-253:0/1350: #0: (&type->s_umount_key#24){.+.+.+}, at: [] writeback_inodes_wb+0xce/0x13d #1: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 5 locks held by git/1471: #0: (&mm->mmap_sem){++++++}, at: [] do_page_fault+0x1c2/0x3d6 #1: (shrinker_rwsem){++++..}, at: [] shrink_slab+0x3d/0x164 #2: (iprune_sem){++++.-}, at: [] shrink_icache_memory+0x4a/0x215 #3: (xfs_iolock_reclaimable){+.+.-.}, at: [] xfs_ilock+0x30/0xb9 [xfs] #4: (&(&ip->i_lock)->mr_lock){++++--}, at: [] xfs_ilock+0x63/0xb9 [xfs] 2 locks held by bash/1472: #0: (sysrq_key_table_lock){......}, at: [] __handle_sysrq+0x28/0x15c #1: (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x52/0x19b -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/