2007-08-24 21:00:48

by folkert

[permalink] [raw]
Subject: [2.6.22] circular lock detected

Hi,

2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).

[ 346.314640] =======================================================
[ 346.314758] [ INFO: possible circular locking dependency detected ]
[ 346.314815] 2.6.22 #5
[ 346.314862] -------------------------------------------------------
[ 346.314920] tor/2421 is trying to acquire lock:
[ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[ 346.315193]
[ 346.315195] but task is already holding lock:
[ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
[ 346.316457]
[ 346.316458] which lock already depends on the new lock.
[ 346.316461]
[ 346.316585]
[ 346.316586] the existing dependency chain (in reverse order) is:
[ 346.316675]
[ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
[ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.317873] [<c103d786>] lock_acquire+0x62/0x81
[ 346.318195] [<c10341af>] down_read+0x2b/0x3d
[ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189
[ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228
[ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78
[ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270
[ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161
[ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4
[ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22
[ 346.322131] [<c1003eea>] syscall_call+0x7/0xb
[ 346.322446] [<ffffffff>] 0xffffffff
[ 346.322779]
[ 346.322780] -> #3 (&ei->truncate_mutex){--..}:
[ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.323919] [<c103d786>] lock_acquire+0x62/0x81
[ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468
[ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116
[ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c
[ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f
[ 346.326479] [<c1087cb0>] notify_change+0x352/0x386
[ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f
[ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231
[ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2
[ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53
[ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8
[ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e
[ 346.328727] [<c1003eea>] syscall_call+0x7/0xb
[ 346.329042] [<ffffffff>] 0xffffffff
[ 346.329368]
[ 346.329370] -> #2 (&inode->i_alloc_sem){--..}:
[ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.330508] [<c103d786>] lock_acquire+0x62/0x81
[ 346.330826] [<c1034245>] down_write+0x2b/0x45
[ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386
[ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f
[ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231
[ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2
[ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53
[ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8
[ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e
[ 346.333600] [<c1003eea>] syscall_call+0x7/0xb
[ 346.333912] [<ffffffff>] 0xffffffff
[ 346.334232]
[ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
[ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281
[ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.335361] [<c103d786>] lock_acquire+0x62/0x81
[ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1
[ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
[ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78
[ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16
[ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62
[ 346.337888] [<c1156dc5>] device_del+0x82/0x291
[ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15
[ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a
[ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
[ 346.339142] [<c1149075>] con_close+0x5e/0x6b
[ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce
[ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c
[ 346.340078] [<c1074479>] __fput+0x145/0x16e
[ 346.340392] [<c1074332>] fput+0x19/0x1b
[ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75
[ 346.341012] [<c1072c55>] sys_close+0x69/0xb4
[ 346.341325] [<c1003eea>] syscall_call+0x7/0xb
[ 346.341892] [<ffffffff>] 0xffffffff
[ 346.342213]
[ 346.342214] -> #0 (tty_mutex){--..}:
[ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281
[ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.343333] [<c103d786>] lock_acquire+0x62/0x81
[ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d
[ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189
[ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3
[ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415
[ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b
[ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
[ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
[ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
[ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
[ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba
[ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116
[ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d
[ 346.349246] [<c1073788>] sys_write+0x3d/0x64
[ 346.349560] [<c1003eea>] syscall_call+0x7/0xb
[ 346.349869] [<ffffffff>] 0xffffffff
[ 346.350438]
[ 346.350440] other info that might help us debug this:
[ 346.350442]
[ 346.350562] 3 locks held by tor/2421:
[ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
[ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
[ 346.351274]
[ 346.351276] stack backtrace:
[ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30
[ 346.351446] [<c1004cd4>] show_trace+0x12/0x14
[ 346.351526] [<c1004dce>] dump_stack+0x16/0x18
[ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71
[ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281
[ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8
[ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51
[ 346.351921] [<c103d786>] lock_acquire+0x62/0x81
[ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
[ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa
[ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d
[ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189
[ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
[ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
[ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
[ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
[ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3
[ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415
[ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b
[ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
[ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
[ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
[ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
[ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba
[ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116
[ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d
[ 346.353444] [<c1073788>] sys_write+0x3d/0x64
[ 346.353524] [<c1003eea>] syscall_call+0x7/0xb
[ 346.353603] =======================


Folkert van Heusden

--
http://www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg
ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of
HP/UX en win een vlaai naar keuze
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com


2007-08-26 23:03:33

by Michal Piotrowski

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

Hi,

[Adding fsdevel and linux-ext4 to CC]

On 24/08/07, Folkert van Heusden <[email protected]> wrote:
> Hi,
>
> 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
>
> [ 346.314640] =======================================================
> [ 346.314758] [ INFO: possible circular locking dependency detected ]
> [ 346.314815] 2.6.22 #5
> [ 346.314862] -------------------------------------------------------
> [ 346.314920] tor/2421 is trying to acquire lock:
> [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.315193]
> [ 346.315195] but task is already holding lock:
> [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [ 346.316457]
> [ 346.316458] which lock already depends on the new lock.
> [ 346.316461]
> [ 346.316585]
> [ 346.316586] the existing dependency chain (in reverse order) is:
> [ 346.316675]
> [ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
> [ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.317873] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.318195] [<c10341af>] down_read+0x2b/0x3d
> [ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189
> [ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228
> [ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78
> [ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270
> [ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161
> [ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4
> [ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22
> [ 346.322131] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.322446] [<ffffffff>] 0xffffffff
> [ 346.322779]
> [ 346.322780] -> #3 (&ei->truncate_mutex){--..}:
> [ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.323919] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468
> [ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116
> [ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c
> [ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f
> [ 346.326479] [<c1087cb0>] notify_change+0x352/0x386
> [ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f
> [ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231
> [ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2
> [ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53
> [ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8
> [ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e
> [ 346.328727] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.329042] [<ffffffff>] 0xffffffff
> [ 346.329368]
> [ 346.329370] -> #2 (&inode->i_alloc_sem){--..}:
> [ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.330508] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.330826] [<c1034245>] down_write+0x2b/0x45
> [ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386
> [ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f
> [ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231
> [ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2
> [ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53
> [ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8
> [ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e
> [ 346.333600] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.333912] [<ffffffff>] 0xffffffff
> [ 346.334232]
> [ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
> [ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.335361] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1
> [ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
> [ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78
> [ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16
> [ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62
> [ 346.337888] [<c1156dc5>] device_del+0x82/0x291
> [ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15
> [ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a
> [ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
> [ 346.339142] [<c1149075>] con_close+0x5e/0x6b
> [ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce
> [ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c
> [ 346.340078] [<c1074479>] __fput+0x145/0x16e
> [ 346.340392] [<c1074332>] fput+0x19/0x1b
> [ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75
> [ 346.341012] [<c1072c55>] sys_close+0x69/0xb4
> [ 346.341325] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.341892] [<ffffffff>] 0xffffffff
> [ 346.342213]
> [ 346.342214] -> #0 (tty_mutex){--..}:
> [ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281
> [ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.343333] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d
> [ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189
> [ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3
> [ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415
> [ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b
> [ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba
> [ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116
> [ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d
> [ 346.349246] [<c1073788>] sys_write+0x3d/0x64
> [ 346.349560] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.349869] [<ffffffff>] 0xffffffff
> [ 346.350438]
> [ 346.350440] other info that might help us debug this:
> [ 346.350442]
> [ 346.350562] 3 locks held by tor/2421:
> [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [ 346.351274]
> [ 346.351276] stack backtrace:
> [ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30
> [ 346.351446] [<c1004cd4>] show_trace+0x12/0x14
> [ 346.351526] [<c1004dce>] dump_stack+0x16/0x18
> [ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71
> [ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281
> [ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.351921] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d
> [ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189
> [ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3
> [ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415
> [ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b
> [ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba
> [ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116
> [ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d
> [ 346.353444] [<c1073788>] sys_write+0x3d/0x64
> [ 346.353524] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.353603] =======================
>
>
> Folkert van Heusden
>
> --
> http://www.vanheusden.com/multitail - win een vlaai van multivlaai! zorg
> ervoor dat multitail opgenomen wordt in Fedora Core, AIX, Solaris of
> HP/UX en win een vlaai naar keuze
> ----------------------------------------------------------------------
> Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

Regards,
Michal

--
LOG
http://www.stardust.webpages.pl/log/

2007-09-02 11:55:58

by Andrew Morton

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

> On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
> Hi,
>
> 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
>
> [ 346.314640] =======================================================
> [ 346.314758] [ INFO: possible circular locking dependency detected ]
> [ 346.314815] 2.6.22 #5
> [ 346.314862] -------------------------------------------------------
> [ 346.314920] tor/2421 is trying to acquire lock:
> [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.315193]
> [ 346.315195] but task is already holding lock:
> [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [ 346.316457]
> [ 346.316458] which lock already depends on the new lock.
> [ 346.316461]
> [ 346.316585]
> [ 346.316586] the existing dependency chain (in reverse order) is:
> [ 346.316675]
> [ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
> [ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.317873] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.318195] [<c10341af>] down_read+0x2b/0x3d
> [ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189
> [ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228
> [ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78
> [ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270
> [ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161
> [ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4
> [ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22
> [ 346.322131] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.322446] [<ffffffff>] 0xffffffff
> [ 346.322779]
> [ 346.322780] -> #3 (&ei->truncate_mutex){--..}:
> [ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.323919] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468
> [ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116
> [ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c
> [ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f
> [ 346.326479] [<c1087cb0>] notify_change+0x352/0x386
> [ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f
> [ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231
> [ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2
> [ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53
> [ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8
> [ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e
> [ 346.328727] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.329042] [<ffffffff>] 0xffffffff
> [ 346.329368]
> [ 346.329370] -> #2 (&inode->i_alloc_sem){--..}:
> [ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.330508] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.330826] [<c1034245>] down_write+0x2b/0x45
> [ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386
> [ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f
> [ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231
> [ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2
> [ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53
> [ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8
> [ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e
> [ 346.333600] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.333912] [<ffffffff>] 0xffffffff
> [ 346.334232]
> [ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
> [ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281
> [ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.335361] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1
> [ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
> [ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78
> [ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16
> [ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62
> [ 346.337888] [<c1156dc5>] device_del+0x82/0x291
> [ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15
> [ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a
> [ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
> [ 346.339142] [<c1149075>] con_close+0x5e/0x6b
> [ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce
> [ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c
> [ 346.340078] [<c1074479>] __fput+0x145/0x16e
> [ 346.340392] [<c1074332>] fput+0x19/0x1b
> [ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75
> [ 346.341012] [<c1072c55>] sys_close+0x69/0xb4
> [ 346.341325] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.341892] [<ffffffff>] 0xffffffff
> [ 346.342213]
> [ 346.342214] -> #0 (tty_mutex){--..}:
> [ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281
> [ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.343333] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d
> [ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189
> [ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3
> [ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415
> [ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b
> [ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba
> [ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116
> [ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d
> [ 346.349246] [<c1073788>] sys_write+0x3d/0x64
> [ 346.349560] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.349869] [<ffffffff>] 0xffffffff
> [ 346.350438]
> [ 346.350440] other info that might help us debug this:
> [ 346.350442]
> [ 346.350562] 3 locks held by tor/2421:
> [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> [ 346.351274]
> [ 346.351276] stack backtrace:
> [ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30
> [ 346.351446] [<c1004cd4>] show_trace+0x12/0x14
> [ 346.351526] [<c1004dce>] dump_stack+0x16/0x18
> [ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71
> [ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281
> [ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8
> [ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> [ 346.351921] [<c103d786>] lock_acquire+0x62/0x81
> [ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> [ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa
> [ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d
> [ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189
> [ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> [ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> [ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> [ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> [ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3
> [ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415
> [ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b
> [ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> [ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> [ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> [ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> [ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba
> [ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116
> [ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d
> [ 346.353444] [<c1073788>] sys_write+0x3d/0x64
> [ 346.353524] [<c1003eea>] syscall_call+0x7/0xb
> [ 346.353603] =======================
>

Has been reported before, but I don't recall whether we fixed it. Jan,
do you know>?

2007-09-02 17:11:48

by folkert

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

> > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> >
> > [ 346.314640] =======================================================
...
> > [ 346.349869] [<ffffffff>] 0xffffffff
> > [ 346.350438]
> > [ 346.350440] other info that might help us debug this:
> > [ 346.350442]
> > [ 346.350562] 3 locks held by tor/2421:
> > [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [ 346.351274]
> > [ 346.351276] stack backtrace:
...
> > [ 346.353603] =======================
>
> Has been reported before, but I don't recall whether we fixed it. Jan,
> do you know>?

It is repeatable. Yesterday I changed from ata to sata and it still
occurs altough in a different proces:

[ 7382.917927]

Oh and it also seems it takes longer.

[ 7382.917929] =======================================================
[ 7382.918024] [ INFO: possible circular locking dependency detected ]
[ 7382.918072] 2.6.22 #2
[ 7382.918114] -------------------------------------------------------
[ 7382.918159] moo/17202 is trying to acquire lock:
[ 7382.918204] (tty_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa
[ 7382.918392]
[ 7382.918393] but task is already holding lock:
[ 7382.918474] (&s->s_dquot.dqptr_sem){----}, at: [<c10a47a3>] dquot_alloc_space+0x50/0x189
[ 7382.918665]
[ 7382.918666] which lock already depends on the new lock.
[ 7382.918668]
[ 7382.918788]
[ 7382.918789] the existing dependency chain (in reverse order) is:
[ 7382.918873]
[ 7382.918874] -> #4 (&s->s_dquot.dqptr_sem){----}:
[ 7382.919061] [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.919378] [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.919690] [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.920004] [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.920322] [<c10340f3>] down_read+0x2b/0x3d
[ 7382.920635] [<c10a47a3>] dquot_alloc_space+0x50/0x189
[ 7382.920957] [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2
[ 7382.921280] [<c10b8128>] ext3_alloc_blocks+0x40/0xdf
[ 7382.921592] [<c10b8217>] ext3_alloc_branch+0x50/0x21b
[ 7382.921906] [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7382.922222] [<c10b8a1e>] ext3_getblk+0x97/0x228
[ 7382.922534] [<c10b8bc9>] ext3_bread+0x1a/0x78
[ 7382.922844] [<c10bdda8>] ext3_mkdir+0xf4/0x270
[ 7382.923160] [<c107cd3a>] vfs_mkdir+0xb3/0x161
[ 7382.923474] [<c107ce74>] sys_mkdirat+0x8c/0xc4
[ 7382.923789] [<c107cecc>] sys_mkdir+0x20/0x22
[ 7382.924106] [<c1003eea>] syscall_call+0x7/0xb
[ 7382.924425] [<ffffffff>] 0xffffffff
[ 7382.924752]
[ 7382.924753] -> #3 (&ei->truncate_mutex){--..}:
[ 7382.924939] [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.925257] [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.925571] [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.925886] [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.926202] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.926782] [<c1228353>] mutex_lock+0x8/0xa
[ 7382.927101] [<c10ba2c4>] ext3_truncate+0x170/0x468
[ 7382.927414] [<c1060dde>] vmtruncate+0xa6/0x116
[ 7382.927726] [<c108787b>] inode_setattr+0x145/0x16c
[ 7382.928041] [<c10bb245>] ext3_setattr+0x150/0x22f
[ 7382.928358] [<c1087bf4>] notify_change+0x352/0x386
[ 7382.928672] [<c1071a38>] do_truncate+0x5c/0x7f
[ 7382.928988] [<c107c227>] may_open+0x1ec/0x231
[ 7382.929289] [<c107c3e8>] open_namei+0xdd/0x5a2
[ 7382.929298] [<c107269f>] do_filp_open+0x2c/0x53
[ 7382.929308] [<c10729d4>] do_sys_open+0x52/0xd8
[ 7382.929316] [<c1072a76>] sys_open+0x1c/0x1e
[ 7382.929323] [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929332] [<ffffffff>] 0xffffffff
[ 7382.929351]
[ 7382.929352] -> #2 (&inode->i_alloc_sem){--..}:
[ 7382.929358] [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.929367] [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929374] [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929382] [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929390] [<c1034189>] down_write+0x2b/0x45
[ 7382.929401] [<c1087b7b>] notify_change+0x2d9/0x386
[ 7382.929409] [<c1071a38>] do_truncate+0x5c/0x7f
[ 7382.929416] [<c107c227>] may_open+0x1ec/0x231
[ 7382.929423] [<c107c3e8>] open_namei+0xdd/0x5a2
[ 7382.929430] [<c107269f>] do_filp_open+0x2c/0x53
[ 7382.929441] [<c10729d4>] do_sys_open+0x52/0xd8
[ 7382.929449] [<c1072a76>] sys_open+0x1c/0x1e
[ 7382.929455] [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929463] [<ffffffff>] 0xffffffff
[ 7382.929471]
[ 7382.929472] -> #1 (&sysfs_inode_imutex_key){--..}:
[ 7382.929481] [<c103b232>] check_prev_add+0x15b/0x281
[ 7382.929489] [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929496] [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929505] [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929512] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.929523] [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929531] [<c10b276c>] remove_dir+0x2c/0xf1
[ 7382.929538] [<c10b2839>] sysfs_remove_subdir+0x8/0xa
[ 7382.929547] [<c10b3c13>] sysfs_remove_group+0x67/0x78
[ 7382.929554] [<c115cd9c>] dpm_sysfs_remove+0x12/0x16
[ 7382.929563] [<c115c4b3>] device_pm_remove+0x21/0x62
[ 7382.929570] [<c1156f09>] device_del+0x82/0x291
[ 7382.929578] [<c1157123>] device_unregister+0xb/0x15
[ 7382.929585] [<c115731e>] device_destroy+0x8d/0x9a
[ 7382.929592] [<c1142856>] vcs_remove_sysfs+0x1c/0x3a
[ 7382.929600] [<c11491b9>] con_close+0x5e/0x6b
[ 7382.929607] [<c113aba2>] release_dev+0x4c4/0x6ce
[ 7382.929614] [<c113b21a>] tty_release+0x12/0x1c
[ 7382.929620] [<c10743bd>] __fput+0x145/0x16e
[ 7382.929628] [<c1074276>] fput+0x19/0x1b
[ 7382.929634] [<c1072af7>] filp_close+0x3c/0x75
[ 7382.929641] [<c1072b99>] sys_close+0x69/0xb4
[ 7382.929648] [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929654] [<ffffffff>] 0xffffffff
[ 7382.929665]
[ 7382.929666] -> #0 (tty_mutex){--..}:
[ 7382.929670] [<c103b10b>] check_prev_add+0x34/0x281
[ 7382.929677] [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929684] [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929691] [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929698] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.929706] [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929712] [<c10a4119>] print_warning+0x8c/0x15d
[ 7382.929720] [<c10a48d7>] dquot_alloc_space+0x184/0x189
[ 7382.929727] [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2
[ 7382.929735] [<c10b8128>] ext3_alloc_blocks+0x40/0xdf
[ 7382.929741] [<c10b8217>] ext3_alloc_branch+0x50/0x21b
[ 7382.929748] [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7382.929755] [<c10b891c>] ext3_get_block+0x78/0xe3
[ 7382.929761] [<c1094746>] __block_prepare_write+0x168/0x415
[ 7382.929770] [<c1095252>] block_prepare_write+0x28/0x3b
[ 7382.929777] [<c10b8dc8>] ext3_prepare_write+0xe3/0x17e
[ 7382.929784] [<c1055235>] generic_file_buffered_write+0x1b4/0x614
[ 7382.929792] [<c10558cf>] __generic_file_aio_write_nolock+0x23a/0x533
[ 7382.929800] [<c1055cd1>] generic_file_aio_write+0x58/0xc4
[ 7382.929807] [<c10b69d7>] ext3_file_write+0x2d/0xba
[ 7382.929815] [<c107347f>] do_sync_write+0xc7/0x116
[ 7382.929822] [<c1073626>] vfs_write+0x158/0x15d
[ 7382.929829] [<c10736cc>] sys_write+0x3d/0x64
[ 7382.929835] [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929842] [<ffffffff>] 0xffffffff
[ 7382.929851]
[ 7382.929852] other info that might help us debug this:
[ 7382.929853]
[ 7382.929855] 3 locks held by moo/17202:
[ 7382.929857] #0: (&inode->i_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929864] #1: (&ei->truncate_mutex){--..}, at: [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929871] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a47a3>] dquot_alloc_space+0x50/0x189
[ 7382.929877]
[ 7382.929878] stack backtrace:
[ 7382.929880] [<c1004cac>] show_trace_log_lvl+0x1a/0x30
[ 7382.929885] [<c1004cd4>] show_trace+0x12/0x14
[ 7382.929888] [<c1004dce>] dump_stack+0x16/0x18
[ 7382.929892] [<c103aa6d>] print_circular_bug_tail+0x6f/0x71
[ 7382.929896] [<c103b10b>] check_prev_add+0x34/0x281
[ 7382.929900] [<c103b3e3>] check_prevs_add+0x8b/0xe8
[ 7382.929903] [<c103cd6e>] __lock_acquire+0x65d/0xb51
[ 7382.929907] [<c103d6ca>] lock_acquire+0x62/0x81
[ 7382.929911] [<c12283d4>] __mutex_lock_slowpath+0x75/0x28c
[ 7382.929915] [<c1228353>] mutex_lock+0x8/0xa
[ 7382.929919] [<c10a4119>] print_warning+0x8c/0x15d
[ 7382.929923] [<c10a48d7>] dquot_alloc_space+0x184/0x189
[ 7382.929926] [<c10b5b99>] ext3_new_blocks+0x44b/0x5a2
[ 7382.929931] [<c10b8128>] ext3_alloc_blocks+0x40/0xdf
[ 7382.929934] [<c10b8217>] ext3_alloc_branch+0x50/0x21b
[ 7382.929937] [<c10b86f5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7382.929941] [<c10b891c>] ext3_get_block+0x78/0xe3
[ 7382.929944] [<c1094746>] __block_prepare_write+0x168/0x415
[ 7382.929949] [<c1095252>] block_prepare_write+0x28/0x3b
[ 7382.929953] [<c10b8dc8>] ext3_prepare_write+0xe3/0x17e
[ 7382.929957] [<c1055235>] generic_file_buffered_write+0x1b4/0x614
[ 7382.929961] [<c10558cf>] __generic_file_aio_write_nolock+0x23a/0x533
[ 7382.929966] [<c1055cd1>] generic_file_aio_write+0x58/0xc4
[ 7382.929970] [<c10b69d7>] ext3_file_write+0x2d/0xba
[ 7382.929974] [<c107347f>] do_sync_write+0xc7/0x116
[ 7382.929978] [<c1073626>] vfs_write+0x158/0x15d
[ 7382.929981] [<c10736cc>] sys_write+0x3d/0x64
[ 7382.929985] [<c1003eea>] syscall_call+0x7/0xb
[ 7382.929988] =======================


Folkert van Heusden

--

Multitail - gibkaja utilita po sledovaniju log-fajlov i vyvoda
kommand. Fil'trovanie, raskra?ivanie, slijanie, vizual'noe sravnenie,
i t.d. http://www.vanheusden.com/multitail/
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, http://www.vanheusden.com

2007-09-03 12:27:19

by Jan Kara

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

> > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
> > Hi,
> >
> > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> >
> > [ 346.314640] =======================================================
> > [ 346.314758] [ INFO: possible circular locking dependency detected ]
> > [ 346.314815] 2.6.22 #5
> > [ 346.314862] -------------------------------------------------------
> > [ 346.314920] tor/2421 is trying to acquire lock:
> > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.315193]
> > [ 346.315195] but task is already holding lock:
> > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [ 346.316457]
> > [ 346.316458] which lock already depends on the new lock.
> > [ 346.316461]
> > [ 346.316585]
> > [ 346.316586] the existing dependency chain (in reverse order) is:
> > [ 346.316675]
> > [ 346.316676] -> #4 (&s->s_dquot.dqptr_sem){----}:
> > [ 346.316871] [<c103b2ee>] check_prev_add+0x15b/0x281
> > [ 346.317204] [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [ 346.317535] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [ 346.317873] [<c103d786>] lock_acquire+0x62/0x81
> > [ 346.318195] [<c10341af>] down_read+0x2b/0x3d
> > [ 346.318527] [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [ 346.318854] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> > [ 346.319187] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> > [ 346.319513] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> > [ 346.319850] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> > [ 346.320184] [<c10b8ada>] ext3_getblk+0x97/0x228
> > [ 346.320530] [<c10b8c85>] ext3_bread+0x1a/0x78
> > [ 346.320870] [<c10bde64>] ext3_mkdir+0xf4/0x270
> > [ 346.321188] [<c107cdf6>] vfs_mkdir+0xb3/0x161
> > [ 346.321506] [<c107cf30>] sys_mkdirat+0x8c/0xc4
> > [ 346.321820] [<c107cf88>] sys_mkdir+0x20/0x22
> > [ 346.322131] [<c1003eea>] syscall_call+0x7/0xb
> > [ 346.322446] [<ffffffff>] 0xffffffff
> > [ 346.322779]
> > [ 346.322780] -> #3 (&ei->truncate_mutex){--..}:
> > [ 346.322965] [<c103b2ee>] check_prev_add+0x15b/0x281
> > [ 346.323278] [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [ 346.323602] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [ 346.323919] [<c103d786>] lock_acquire+0x62/0x81
> > [ 346.324234] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [ 346.324857] [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.325177] [<c10ba380>] ext3_truncate+0x170/0x468
> > [ 346.325501] [<c1060e9a>] vmtruncate+0xa6/0x116
> > [ 346.325828] [<c1087937>] inode_setattr+0x145/0x16c
> > [ 346.326150] [<c10bb301>] ext3_setattr+0x150/0x22f
> > [ 346.326479] [<c1087cb0>] notify_change+0x352/0x386
> > [ 346.326802] [<c1071af4>] do_truncate+0x5c/0x7f
> > [ 346.327116] [<c107c2e3>] may_open+0x1ec/0x231
> > [ 346.327434] [<c107c4a4>] open_namei+0xdd/0x5a2
> > [ 346.327770] [<c107275b>] do_filp_open+0x2c/0x53
> > [ 346.328090] [<c1072a90>] do_sys_open+0x52/0xd8
> > [ 346.328408] [<c1072b32>] sys_open+0x1c/0x1e
> > [ 346.328727] [<c1003eea>] syscall_call+0x7/0xb
> > [ 346.329042] [<ffffffff>] 0xffffffff
> > [ 346.329368]
> > [ 346.329370] -> #2 (&inode->i_alloc_sem){--..}:
> > [ 346.329556] [<c103b2ee>] check_prev_add+0x15b/0x281
> > [ 346.329879] [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [ 346.330192] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [ 346.330508] [<c103d786>] lock_acquire+0x62/0x81
> > [ 346.330826] [<c1034245>] down_write+0x2b/0x45
> > [ 346.331137] [<c1087c37>] notify_change+0x2d9/0x386
> > [ 346.331453] [<c1071af4>] do_truncate+0x5c/0x7f
> > [ 346.331769] [<c107c2e3>] may_open+0x1ec/0x231
> > [ 346.332087] [<c107c4a4>] open_namei+0xdd/0x5a2
> > [ 346.332401] [<c107275b>] do_filp_open+0x2c/0x53
> > [ 346.332720] [<c1072a90>] do_sys_open+0x52/0xd8
> > [ 346.333037] [<c1072b32>] sys_open+0x1c/0x1e
> > [ 346.333600] [<c1003eea>] syscall_call+0x7/0xb
> > [ 346.333912] [<ffffffff>] 0xffffffff
> > [ 346.334232]
> > [ 346.334233] -> #1 (&sysfs_inode_imutex_key){--..}:
> > [ 346.334422] [<c103b2ee>] check_prev_add+0x15b/0x281
> > [ 346.334734] [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [ 346.335046] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [ 346.335361] [<c103d786>] lock_acquire+0x62/0x81
> > [ 346.335685] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [ 346.336001] [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.336313] [<c10b2828>] remove_dir+0x2c/0xf1
> > [ 346.336629] [<c10b28f5>] sysfs_remove_subdir+0x8/0xa
> > [ 346.336941] [<c10b3ccf>] sysfs_remove_group+0x67/0x78
> > [ 346.337256] [<c115cc58>] dpm_sysfs_remove+0x12/0x16
> > [ 346.337574] [<c115c36f>] device_pm_remove+0x21/0x62
> > [ 346.337888] [<c1156dc5>] device_del+0x82/0x291
> > [ 346.338199] [<c1156fdf>] device_unregister+0xb/0x15
> > [ 346.338516] [<c11571da>] device_destroy+0x8d/0x9a
> > [ 346.338827] [<c1142712>] vcs_remove_sysfs+0x1c/0x3a
> > [ 346.339142] [<c1149075>] con_close+0x5e/0x6b
> > [ 346.339458] [<c113aa5e>] release_dev+0x4c4/0x6ce
> > [ 346.339767] [<c113b0d6>] tty_release+0x12/0x1c
> > [ 346.340078] [<c1074479>] __fput+0x145/0x16e
> > [ 346.340392] [<c1074332>] fput+0x19/0x1b
> > [ 346.340701] [<c1072bb3>] filp_close+0x3c/0x75
> > [ 346.341012] [<c1072c55>] sys_close+0x69/0xb4
> > [ 346.341325] [<c1003eea>] syscall_call+0x7/0xb
> > [ 346.341892] [<ffffffff>] 0xffffffff
> > [ 346.342213]
> > [ 346.342214] -> #0 (tty_mutex){--..}:
> > [ 346.342397] [<c103b1c7>] check_prev_add+0x34/0x281
> > [ 346.342708] [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [ 346.343017] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [ 346.343333] [<c103d786>] lock_acquire+0x62/0x81
> > [ 346.343646] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [ 346.343956] [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.344263] [<c10a41d5>] print_warning+0x8c/0x15d
> > [ 346.344575] [<c10a4993>] dquot_alloc_space+0x184/0x189
> > [ 346.344883] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> > [ 346.345193] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> > [ 346.345507] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> > [ 346.345816] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> > [ 346.346127] [<c10b89d8>] ext3_get_block+0x78/0xe3
> > [ 346.346441] [<c1094802>] __block_prepare_write+0x168/0x415
> > [ 346.346750] [<c109530e>] block_prepare_write+0x28/0x3b
> > [ 346.347057] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> > [ 346.347369] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> > [ 346.347689] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> > [ 346.348002] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> > [ 346.348314] [<c10b6a93>] ext3_file_write+0x2d/0xba
> > [ 346.348627] [<c107353b>] do_sync_write+0xc7/0x116
> > [ 346.348935] [<c10736e2>] vfs_write+0x158/0x15d
> > [ 346.349246] [<c1073788>] sys_write+0x3d/0x64
> > [ 346.349560] [<c1003eea>] syscall_call+0x7/0xb
> > [ 346.349869] [<ffffffff>] 0xffffffff
> > [ 346.350438]
> > [ 346.350440] other info that might help us debug this:
> > [ 346.350442]
> > [ 346.350562] 3 locks held by tor/2421:
> > [ 346.350604] #0: (&inode->i_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.350827] #1: (&ei->truncate_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.351049] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > [ 346.351274]
> > [ 346.351276] stack backtrace:
> > [ 346.351363] [<c1004cac>] show_trace_log_lvl+0x1a/0x30
> > [ 346.351446] [<c1004cd4>] show_trace+0x12/0x14
> > [ 346.351526] [<c1004dce>] dump_stack+0x16/0x18
> > [ 346.351605] [<c103ab29>] print_circular_bug_tail+0x6f/0x71
> > [ 346.351686] [<c103b1c7>] check_prev_add+0x34/0x281
> > [ 346.351764] [<c103b49f>] check_prevs_add+0x8b/0xe8
> > [ 346.351843] [<c103ce2a>] __lock_acquire+0x65d/0xb51
> > [ 346.351921] [<c103d786>] lock_acquire+0x62/0x81
> > [ 346.352002] [<c121ab04>] __mutex_lock_slowpath+0x75/0x28c
> > [ 346.352083] [<c121aa83>] mutex_lock+0x8/0xa
> > [ 346.352160] [<c10a41d5>] print_warning+0x8c/0x15d
> > [ 346.352240] [<c10a4993>] dquot_alloc_space+0x184/0x189
> > [ 346.352323] [<c10b5c55>] ext3_new_blocks+0x44b/0x5a2
> > [ 346.352402] [<c10b81e4>] ext3_alloc_blocks+0x40/0xdf
> > [ 346.352479] [<c10b82d3>] ext3_alloc_branch+0x50/0x21b
> > [ 346.352558] [<c10b87b1>] ext3_get_blocks_handle+0x1b8/0x367
> > [ 346.352638] [<c10b89d8>] ext3_get_block+0x78/0xe3
> > [ 346.352720] [<c1094802>] __block_prepare_write+0x168/0x415
> > [ 346.352801] [<c109530e>] block_prepare_write+0x28/0x3b
> > [ 346.352881] [<c10b8e84>] ext3_prepare_write+0xe3/0x17e
> > [ 346.352961] [<c10552f1>] generic_file_buffered_write+0x1b4/0x614
> > [ 346.353041] [<c105598b>] __generic_file_aio_write_nolock+0x23a/0x533
> > [ 346.353121] [<c1055d8d>] generic_file_aio_write+0x58/0xc4
> > [ 346.353200] [<c10b6a93>] ext3_file_write+0x2d/0xba
> > [ 346.353281] [<c107353b>] do_sync_write+0xc7/0x116
> > [ 346.353364] [<c10736e2>] vfs_write+0x158/0x15d
> > [ 346.353444] [<c1073788>] sys_write+0x3d/0x64
> > [ 346.353524] [<c1003eea>] syscall_call+0x7/0xb
> > [ 346.353603] =======================
> >
> Has been reported before, but I don't recall whether we fixed it. Jan,
> do you know>?
I think we at least found a solution: Teach lockdep that I_MUTEX for
different filesystems is different. Peter Zilstra wrote a patch for that
and Folkert even confirmed that it fixes the problem for him. I'm not
sure what happened with the patch afterwards though. Adding Peter to CC
:).

Honza
--
Jan Kara <[email protected]>
SuSE CR Labs

2007-09-03 12:50:26

by Andrew Morton

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

> On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <[email protected]> wrote:
> > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
> > > Hi,
> > >
> > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> > >
> > > [ 346.314640] =======================================================
> > > [ 346.314758] [ INFO: possible circular locking dependency detected ]
> > > [ 346.314815] 2.6.22 #5
> > > [ 346.314862] -------------------------------------------------------
> > > [ 346.314920] tor/2421 is trying to acquire lock:
> > > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > > [ 346.315193]
> > > [ 346.315195] but task is already holding lock:
> > > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > > [ 346.316457]
> > > [ 346.316458] which lock already depends on the new lock.
>
> ...
>
> > > [ 346.353603] =======================
> > >
> > Has been reported before, but I don't recall whether we fixed it. Jan,
> > do you know>?
> I think we at least found a solution: Teach lockdep that I_MUTEX for
> different filesystems is different. Peter Zilstra wrote a patch for that
> and Folkert even confirmed that it fixes the problem for him. I'm not
> sure what happened with the patch afterwards though. Adding Peter to CC
> :).

But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?

2007-09-03 13:42:19

by Jan Kara

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <[email protected]> wrote:
> > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
> > > > Hi,
> > > >
> > > > 2.6.22 kernel with hyperthreading enabled only ext3 filesystems (2).
> > > >
> > > > [ 346.314640] =======================================================
> > > > [ 346.314758] [ INFO: possible circular locking dependency detected ]
> > > > [ 346.314815] 2.6.22 #5
> > > > [ 346.314862] -------------------------------------------------------
> > > > [ 346.314920] tor/2421 is trying to acquire lock:
> > > > [ 346.314973] (tty_mutex){--..}, at: [<c121aa83>] mutex_lock+0x8/0xa
> > > > [ 346.315193]
> > > > [ 346.315195] but task is already holding lock:
> > > > [ 346.316203] (&s->s_dquot.dqptr_sem){----}, at: [<c10a485f>] dquot_alloc_space+0x50/0x189
> > > > [ 346.316457]
> > > > [ 346.316458] which lock already depends on the new lock.
> >
> > ...
> >
> > > > [ 346.353603] =======================
> > > >
> > > Has been reported before, but I don't recall whether we fixed it. Jan,
> > > do you know>?
> > I think we at least found a solution: Teach lockdep that I_MUTEX for
> > different filesystems is different. Peter Zilstra wrote a patch for that
> > and Folkert even confirmed that it fixes the problem for him. I'm not
> > sure what happened with the patch afterwards though. Adding Peter to CC
> > :).
>
> But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?
The final report is for this ranking but the locking chain (if I understand it
right) is:
tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)

So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
complain about tty_mutex vs i_mutex) but the wrong link in the chain is
that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
[ext3] are different and should never depend on each other...

Honza

--
Jan Kara <[email protected]>
SuSE CR Labs

2007-09-25 08:03:05

by Peter Zijlstra

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <[email protected]> wrote:

> On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <[email protected]> wrote:
> > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:

> > > > Has been reported before, but I don't recall whether we fixed it. Jan,
> > > > do you know>?
> > > I think we at least found a solution: Teach lockdep that I_MUTEX for
> > > different filesystems is different. Peter Zilstra wrote a patch for that
> > > and Folkert even confirmed that it fixes the problem for him. I'm not
> > > sure what happened with the patch afterwards though. Adding Peter to CC
> > > :).
> >
> > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?
> The final report is for this ranking but the locking chain (if I understand it
> right) is:
> tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
> i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
> truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
>
> So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
> complain about tty_mutex vs i_mutex) but the wrong link in the chain is
> that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
> [ext3] are different and should never depend on each other...
>

Found it again.

---

Give each filesystem its own inode lock class. The various filesystems have
different locking order wrt the inode locks; esp. the pseudo filesystems
differ from the rest.

Signed-off-by: Peter Zijlstra <[email protected]>
---
fs/inode.c | 12 +++++++++---
include/linux/fs.h | 5 +++++
2 files changed, 14 insertions(+), 3 deletions(-)

Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct
return NULL;
}

+ spin_lock_init(&inode->i_lock);
+ lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
+
+ mutex_init(&inode->i_mutex);
+ lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
+
+ init_rwsem(&inode->i_alloc_sem);
+ lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
+
mapping->a_ops = &empty_aops;
mapping->host = inode;
mapping->flags = 0;
@@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
INIT_HLIST_NODE(&inode->i_hash);
INIT_LIST_HEAD(&inode->i_dentry);
INIT_LIST_HEAD(&inode->i_devices);
- mutex_init(&inode->i_mutex);
- init_rwsem(&inode->i_alloc_sem);
INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
rwlock_init(&inode->i_data.tree_lock);
spin_lock_init(&inode->i_data.i_mmap_lock);
@@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
spin_lock_init(&inode->i_data.private_lock);
INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
- spin_lock_init(&inode->i_lock);
i_size_ordered_init(inode);
#ifdef CONFIG_INOTIFY
INIT_LIST_HEAD(&inode->inotify_watches);
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -1302,8 +1302,13 @@ struct file_system_type {
struct module *owner;
struct file_system_type * next;
struct list_head fs_supers;
+
struct lock_class_key s_lock_key;
struct lock_class_key s_umount_key;
+
+ struct lock_class_key i_lock_key;
+ struct lock_class_key i_mutex_key;
+ struct lock_class_key i_alloc_sem_key;
};

extern int get_sb_bdev(struct file_system_type *fs_type,

2007-09-25 08:25:25

by Jan Kara

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <[email protected]> wrote:
>
> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <[email protected]> wrote:
> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
>
> > > > > Has been reported before, but I don't recall whether we fixed it. Jan,
> > > > > do you know>?
> > > > I think we at least found a solution: Teach lockdep that I_MUTEX for
> > > > different filesystems is different. Peter Zilstra wrote a patch for that
> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
> > > > sure what happened with the patch afterwards though. Adding Peter to CC
> > > > :).
> > >
> > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?
> > The final report is for this ranking but the locking chain (if I understand it
> > right) is:
> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
> >
> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
> > [ext3] are different and should never depend on each other...
> >
>
> Found it again.
Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
the false lockdep warnings from the quota code. If I recall correctly, one
of the reporters even confirmed it fixes the problem for him.
The patch looks fine. You can add:
Signed-off-by: Jan Kara <[email protected]>

Honza

> Give each filesystem its own inode lock class. The various filesystems have
> different locking order wrt the inode locks; esp. the pseudo filesystems
> differ from the rest.
>
> Signed-off-by: Peter Zijlstra <[email protected]>
> ---
> fs/inode.c | 12 +++++++++---
> include/linux/fs.h | 5 +++++
> 2 files changed, 14 insertions(+), 3 deletions(-)
>
> Index: linux-2.6/fs/inode.c
> ===================================================================
> --- linux-2.6.orig/fs/inode.c
> +++ linux-2.6/fs/inode.c
> @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct
> return NULL;
> }
>
> + spin_lock_init(&inode->i_lock);
> + lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
> +
> + mutex_init(&inode->i_mutex);
> + lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
> +
> + init_rwsem(&inode->i_alloc_sem);
> + lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
> +
> mapping->a_ops = &empty_aops;
> mapping->host = inode;
> mapping->flags = 0;
> @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
> INIT_HLIST_NODE(&inode->i_hash);
> INIT_LIST_HEAD(&inode->i_dentry);
> INIT_LIST_HEAD(&inode->i_devices);
> - mutex_init(&inode->i_mutex);
> - init_rwsem(&inode->i_alloc_sem);
> INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
> rwlock_init(&inode->i_data.tree_lock);
> spin_lock_init(&inode->i_data.i_mmap_lock);
> @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
> spin_lock_init(&inode->i_data.private_lock);
> INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
> INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
> - spin_lock_init(&inode->i_lock);
> i_size_ordered_init(inode);
> #ifdef CONFIG_INOTIFY
> INIT_LIST_HEAD(&inode->inotify_watches);
> Index: linux-2.6/include/linux/fs.h
> ===================================================================
> --- linux-2.6.orig/include/linux/fs.h
> +++ linux-2.6/include/linux/fs.h
> @@ -1302,8 +1302,13 @@ struct file_system_type {
> struct module *owner;
> struct file_system_type * next;
> struct list_head fs_supers;
> +
> struct lock_class_key s_lock_key;
> struct lock_class_key s_umount_key;
> +
> + struct lock_class_key i_lock_key;
> + struct lock_class_key i_mutex_key;
> + struct lock_class_key i_alloc_sem_key;
> };
>
> extern int get_sb_bdev(struct file_system_type *fs_type,
>
--
Jan Kara <[email protected]>
SUSE Labs, CR

2008-01-04 07:41:11

by Simon Arlott

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected

On 25/09/07 09:46, Jan Kara wrote:
> On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
>> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <[email protected]> wrote:
>>
>> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
>> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <[email protected]> wrote:
>> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
>>
>> > > > > Has been reported before, but I don't recall whether we fixed it. Jan,
>> > > > > do you know>?
>> > > > I think we at least found a solution: Teach lockdep that I_MUTEX for
>> > > > different filesystems is different. Peter Zilstra wrote a patch for that
>> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
>> > > > sure what happened with the patch afterwards though. Adding Peter to CC
>> > > > :).
>> > >
>> > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?
>> > The final report is for this ranking but the locking chain (if I understand it
>> > right) is:
>> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
>> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
>> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
>> >
>> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
>> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
>> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
>> > [ext3] are different and should never depend on each other...
>> >
>>
>> Found it again.
> Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
> the false lockdep warnings from the quota code. If I recall correctly, one
> of the reporters even confirmed it fixes the problem for him.
> The patch looks fine. You can add:
> Signed-off-by: Jan Kara <[email protected]>
>
> Honza
>
>> Give each filesystem its own inode lock class. The various filesystems have
>> different locking order wrt the inode locks; esp. the pseudo filesystems
>> differ from the rest.
>>
>> Signed-off-by: Peter Zijlstra <[email protected]>

This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed...

03 00:31:34 [1905219.899008] =======================================================
03 00:31:34 [1905219.907136] [ INFO: possible circular locking dependency detected ]
03 00:31:34 [1905219.913569] 2.6.23.9-git #43
03 00:31:34 [1905219.916624] -------------------------------------------------------
03 00:31:34 [1905219.923059] bacula-sd/32639 is trying to acquire lock:
03 00:31:34 [1905219.928364] (tty_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905219.935047]
03 00:31:34 [1905219.935049] but task is already holding lock:
03 00:31:34 [1905219.941238] (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905219.949646]
03 00:31:34 [1905219.949649] which lock already depends on the new lock.
03 00:31:34 [1905219.949652]
03 00:31:34 [1905219.958367]
03 00:31:34 [1905219.958369] the existing dependency chain (in reverse order) is:
03 00:31:34 [1905219.966204]
03 00:31:34 [1905219.966206] -> #4 (&s->s_dquot.dqptr_sem){----}:
03 00:31:34 [1905219.972714] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905219.978511] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905219.983957] [<8012ac3c>] down_read+0x3a/0x4b
03 00:31:34 [1905219.989252] [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905219.995217] [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.001009] [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.007495] [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.013113] [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.019603] [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.025654] [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.031705] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.038630] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.045894] [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.052292] [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.057996] [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.063615] [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.068885] [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.074071] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.080046] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.084456]
03 00:31:34 [1905220.084458] -> #3 (&ei->truncate_mutex){--..}:
03 00:31:34 [1905220.090781] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.096573] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.102020] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.108418] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.113690] [<80196110>] ext3_truncate+0x49e/0x7ec
03 00:31:34 [1905220.119397] [<80148496>] vmtruncate+0x161/0x187
03 00:31:34 [1905220.124847] [<80169794>] inode_setattr+0x73/0x149
03 00:31:34 [1905220.130470] [<8019720c>] ext3_setattr+0x185/0x1e1
03 00:31:34 [1905220.136088] [<8016999a>] notify_change+0x130/0x2d2
03 00:31:34 [1905220.141883] [<80156c7b>] do_truncate+0x60/0x79
03 00:31:34 [1905220.147245] [<8015f377>] may_open+0x1b9/0x201
03 00:31:34 [1905220.152522] [<801612e3>] open_namei+0x240/0x539
03 00:31:34 [1905220.157965] [<8015651b>] do_filp_open+0x26/0x3b
03 00:31:34 [1905220.163411] [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.168857] [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.173955] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.179830] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.184233]
03 00:31:34 [1905220.184235] -> #2 (&inode->i_alloc_sem){--..}:
03 00:31:34 [1905220.190559] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.196436] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.201884] [<8012acd6>] down_write+0x3a/0x53
03 00:31:34 [1905220.207152] [<80169952>] notify_change+0xe8/0x2d2
03 00:31:34 [1905220.212772] [<80156c7b>] do_truncate+0x60/0x79
03 00:31:34 [1905220.218130] [<8015f377>] may_open+0x1b9/0x201
03 00:31:34 [1905220.223485] [<801612e3>] open_namei+0x240/0x539
03 00:31:34 [1905220.228926] [<8015651b>] do_filp_open+0x26/0x3b
03 00:31:34 [1905220.234372] [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.239727] [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.244819] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.250760] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.255156]
03 00:31:34 [1905220.255158] -> #1 (&inode->i_mutex){--..}:
03 00:31:34 [1905220.261135] [<80131c5b>] __lock_acquire+0x9ba/0xb96
03 00:31:34 [1905220.268005] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.273448] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.279752] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.285024] [<80190cf6>] get_node+0x1d/0x46
03 00:31:34 [1905220.290119] [<80190e84>] devpts_get_tty+0xb/0x3b
03 00:31:34 [1905220.295646] [<80214fb5>] init_dev+0x21/0x4a2
03 00:31:34 [1905220.300837] [<80217c95>] ptmx_open+0xfd/0x1bc
03 00:31:34 [1905220.306102] [<80159f91>] chrdev_open+0x14a/0x180
03 00:31:34 [1905220.311618] [<801563b2>] __dentry_open+0xfd/0x1af
03 00:31:34 [1905220.317232] [<801564e5>] nameidata_to_filp+0x27/0x37
03 00:31:34 [1905220.323097] [<80156528>] do_filp_open+0x33/0x3b
03 00:31:34 [1905220.328535] [<80156575>] do_sys_open+0x45/0xc9
03 00:31:34 [1905220.333874] [<80156631>] sys_open+0x1c/0x1e
03 00:31:34 [1905220.338981] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.344855] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.349256]
03 00:31:34 [1905220.349258] -> #0 (tty_mutex){--..}:
03 00:31:34 [1905220.354717] [<80131b4b>] __lock_acquire+0x8aa/0xb96
03 00:31:34 [1905220.360505] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.365956] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.372258] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.377517] [<80182ea9>] print_warning+0x92/0x137
03 00:31:34 [1905220.383139] [<80183ab8>] dquot_alloc_space+0x13b/0x15b
03 00:31:34 [1905220.389176] [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.394961] [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.401429] [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.407047] [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.413455] [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.419506] [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.425553] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.432465] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.439724] [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.446027] [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.451728] [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.457338] [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.462600] [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.467771] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.473639] [<ffffffff>] 0xffffffff
03 00:31:34 [1905220.478040]
03 00:31:34 [1905220.478043] other info that might help us debug this:
03 00:31:34 [1905220.478046]
03 00:31:34 [1905220.486588] 3 locks held by bacula-sd/32639:
03 00:31:34 [1905220.491025] #0: (&inode->i_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.498668] #1: (&ei->truncate_mutex){--..}, at: [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.506659] #2: (&s->s_dquot.dqptr_sem){----}, at: [<801839bf>] dquot_alloc_space+0x42/0x15b
03 00:31:34 [1905220.515511]
03 00:31:34 [1905220.515513] stack backtrace:
03 00:31:34 [1905220.520230] [<80103694>] show_trace_log_lvl+0x1a/0x2f
03 00:31:34 [1905220.525555] [<8010421b>] show_trace+0x12/0x14
03 00:31:34 [1905220.530185] [<80104232>] dump_stack+0x15/0x17
03 00:31:34 [1905220.534817] [<801303c4>] print_circular_bug_tail+0x5e/0x67
03 00:31:34 [1905220.540580] [<80131b4b>] __lock_acquire+0x8aa/0xb96
03 00:31:34 [1905220.545731] [<80132200>] lock_acquire+0x5d/0x75
03 00:31:34 [1905220.550536] [<80389e25>] __mutex_lock_slowpath+0xdb/0x271
03 00:31:34 [1905220.556203] [<80389fd7>] mutex_lock+0x1c/0x1f
03 00:31:34 [1905220.560838] [<80182ea9>] print_warning+0x92/0x137
03 00:31:34 [1905220.565819] [<80183ab8>] dquot_alloc_space+0x13b/0x15b
03 00:31:34 [1905220.571230] [<801920e1>] ext3_new_blocks+0x83/0x5ba
03 00:31:34 [1905220.576381] [<80195111>] ext3_get_blocks_handle+0x386/0x822
03 00:31:34 [1905220.582226] [<80195889>] ext3_get_block+0xac/0xc5
03 00:31:34 [1905220.587205] [<80174e54>] __block_prepare_write+0x151/0x3c6
03 00:31:34 [1905220.592963] [<801750ed>] block_prepare_write+0x24/0x32
03 00:31:34 [1905220.598372] [<80196a43>] ext3_prepare_write+0x98/0x153
03 00:31:34 [1905220.603785] [<8013cbbe>] generic_file_buffered_write+0x219/0x57c
03 00:31:34 [1905220.610063] [<8013d384>] __generic_file_aio_write_nolock+0x463/0x4b3
03 00:31:34 [1905220.616687] [<8013d42a>] generic_file_aio_write+0x56/0xb4
03 00:31:34 [1905220.622362] [<8019311b>] ext3_file_write+0x27/0x99
03 00:31:34 [1905220.627426] [<80157742>] do_sync_write+0xc4/0x101
03 00:31:34 [1905220.632401] [<80157f03>] vfs_write+0xaf/0x138
03 00:31:34 [1905220.637033] [<8015840f>] sys_write+0x3d/0x61
03 00:31:34 [1905220.641574] [<80102702>] sysenter_past_esp+0x5f/0x99
03 00:31:34 [1905220.646811] =======================

>> ---
>> fs/inode.c | 12 +++++++++---
>> include/linux/fs.h | 5 +++++
>> 2 files changed, 14 insertions(+), 3 deletions(-)
>>
>> Index: linux-2.6/fs/inode.c
>> ===================================================================
>> --- linux-2.6.orig/fs/inode.c
>> +++ linux-2.6/fs/inode.c
>> @@ -142,6 +142,15 @@ static struct inode *alloc_inode(struct
>> return NULL;
>> }
>>
>> + spin_lock_init(&inode->i_lock);
>> + lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);
>> +
>> + mutex_init(&inode->i_mutex);
>> + lockdep_set_class(&inode->i_mutex, &sb->s_type->i_mutex_key);
>> +
>> + init_rwsem(&inode->i_alloc_sem);
>> + lockdep_set_class(&inode->i_alloc_sem, &sb->s_type->i_alloc_sem_key);
>> +
>> mapping->a_ops = &empty_aops;
>> mapping->host = inode;
>> mapping->flags = 0;
>> @@ -190,8 +199,6 @@ void inode_init_once(struct inode *inode
>> INIT_HLIST_NODE(&inode->i_hash);
>> INIT_LIST_HEAD(&inode->i_dentry);
>> INIT_LIST_HEAD(&inode->i_devices);
>> - mutex_init(&inode->i_mutex);
>> - init_rwsem(&inode->i_alloc_sem);
>> INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
>> rwlock_init(&inode->i_data.tree_lock);
>> spin_lock_init(&inode->i_data.i_mmap_lock);
>> @@ -199,7 +206,6 @@ void inode_init_once(struct inode *inode
>> spin_lock_init(&inode->i_data.private_lock);
>> INIT_RAW_PRIO_TREE_ROOT(&inode->i_data.i_mmap);
>> INIT_LIST_HEAD(&inode->i_data.i_mmap_nonlinear);
>> - spin_lock_init(&inode->i_lock);
>> i_size_ordered_init(inode);
>> #ifdef CONFIG_INOTIFY
>> INIT_LIST_HEAD(&inode->inotify_watches);
>> Index: linux-2.6/include/linux/fs.h
>> ===================================================================
>> --- linux-2.6.orig/include/linux/fs.h
>> +++ linux-2.6/include/linux/fs.h
>> @@ -1302,8 +1302,13 @@ struct file_system_type {
>> struct module *owner;
>> struct file_system_type * next;
>> struct list_head fs_supers;
>> +
>> struct lock_class_key s_lock_key;
>> struct lock_class_key s_umount_key;
>> +
>> + struct lock_class_key i_lock_key;
>> + struct lock_class_key i_mutex_key;
>> + struct lock_class_key i_alloc_sem_key;
>> };
>>
>> extern int get_sb_bdev(struct file_system_type *fs_type,
>>


--
Simon Arlott

2008-01-04 09:22:13

by Peter Zijlstra

[permalink] [raw]
Subject: Re: [2.6.22] circular lock detected


On Fri, 2008-01-04 at 07:40 +0000, Simon Arlott wrote:
> On 25/09/07 09:46, Jan Kara wrote:
> > On Tue 25-09-07 10:02:43, Peter Zijlstra wrote:
> >> On Mon, 3 Sep 2007 16:01:35 +0200 Jan Kara <[email protected]> wrote:
> >>
> >> > On Mon 03-09-07 05:49:59, Andrew Morton wrote:
> >> > > > On Mon, 3 Sep 2007 14:27:02 +0200 Jan Kara <[email protected]> wrote:
> >> > > > > > On Fri, 24 Aug 2007 23:00:33 +0200 Folkert van Heusden <[email protected]> wrote:
> >>
> >> > > > > Has been reported before, but I don't recall whether we fixed it. Jan,
> >> > > > > do you know>?
> >> > > > I think we at least found a solution: Teach lockdep that I_MUTEX for
> >> > > > different filesystems is different. Peter Zilstra wrote a patch for that
> >> > > > and Folkert even confirmed that it fixes the problem for him. I'm not
> >> > > > sure what happened with the patch afterwards though. Adding Peter to CC
> >> > > > :).
> >> > >
> >> > > But this is a tty_lock-versus-dqptr_sem ranking error. Unrelated to i_mutex?
> >> > The final report is for this ranking but the locking chain (if I understand it
> >> > right) is:
> >> > tty_mutex (con_close) -> i_mutex (sysfs: remove_subdir)
> >> > i_mutex (do_truncate) -> i_alloc_sem (notify_change) -> truncate_mutex (ext3_truncate)
> >> > truncate_mutex (ext3_get_blocks_handle) -> dqptr_sem (dquot_alloc_space)
> >> >
> >> > So it complains about tty_mutex vs dqptr_sem (I don't know why it does not
> >> > complain about tty_mutex vs i_mutex) but the wrong link in the chain is
> >> > that i_mutex from remove_subdir() [sysfs] and i_mutex from do_truncate()
> >> > [ext3] are different and should never depend on each other...
> >> >
> >>
> >> Found it again.
> > Cool, thanks Peter. Andrew, would you put it into -mm? This should take care of
> > the false lockdep warnings from the quota code. If I recall correctly, one
> > of the reporters even confirmed it fixes the problem for him.
> > The patch looks fine. You can add:
> > Signed-off-by: Jan Kara <[email protected]>
> >
> > Honza
> >
> >> Give each filesystem its own inode lock class. The various filesystems have
> >> different locking order wrt the inode locks; esp. the pseudo filesystems
> >> differ from the rest.
> >>
> >> Signed-off-by: Peter Zijlstra <[email protected]>
>
> This patch still doesn't exist in 2.6.23.9 and the warning isn't fixed...

It would have been if you'd used .24-rc :-)


Attachments:
signature.asc (189.00 B)
This is a digitally signed message part