2008-11-03 13:33:26

by Alexander Beregalov

[permalink] [raw]
Subject: Deadlock at io_schedule? (Re: linux-next: Tree for November 3)

Hi

2.6.28-rc3-next-20081103 on sparc64

SysRq : Emergency Sync
SysRq : Emergency Sync
SysRq : Emergency Sync
SysRq : Show Locks Held

Showing all locks held in the system:
1 lock held by pdflush/163:
#0: (&type->s_umount_key#13){----}, at: [<00000000004cf148>]
__sync_inodes+0x5c/0xd4
1 lock held by pdflush/164:
#0: (&type->s_umount_key#13){----}, at: [<00000000004cf148>]
__sync_inodes+0x5c/0xd4
1 lock held by metalog/1413:
#0: (&sb->s_type->i_mutex_key#3){--..}, at: [<0000000000488c88>]
generic_file_aio_write+0x44/0xc8
1 lock held by agetty/1630:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
1 lock held by agetty/1632:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
1 lock held by agetty/1634:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
1 lock held by agetty/1636:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
1 lock held by agetty/1638:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
1 lock held by agetty/1640:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
2 locks held by bash/1671:
#0: (sysrq_key_table_lock){....}, at: [<00000000005fc700>]
__handle_sysrq+0x14/0x164
#1: (tasklist_lock){..--}, at: [<0000000000471510>]
debug_show_all_locks+0x5c/0x1a8
1 lock held by bash/1676:
#0: (&tty->atomic_read_lock){--..}, at: [<00000000005ead9c>]
n_tty_read+0x20c/0x648
2 locks held by touch/1317:
#0: (&sb->s_type->i_mutex_key#3){--..}, at: [<00000000004d2170>]
utimes_common+0x164/0x194
#1: (jbd_handle){--..}, at: [<000000000053d788>] journal_start+0xe8/0x118
1 lock held by bash/1333:
#0: (&p->cred_exec_mutex){--..}, at: [<00000000004e61c0>]
compat_do_execve+0x38/0x194
1 lock held by init/1336:
#0: (&sb->s_type->i_mutex_key#3){--..}, at: [<0000000000488c88>]
generic_file_aio_write+0x44/0xc8

=============================================

SysRq : Show Blocked State
task PC stack pid father
pdflush D 00000000004878c8 0 163 2
Call Trace:
[00000000006d9f74] io_schedule+0x20/0x3c
[00000000004878c8] sync_page+0x84/0x94
[00000000006da2f8] __wait_on_bit+0x64/0xc0
[0000000000487b20] wait_on_page_bit+0x8c/0x9c
[00000000004880bc] wait_on_page_writeback_range+0x68/0x124
[00000000004881a4] filemap_fdatawait+0x2c/0x40
[000000000048844c] filemap_write_and_wait+0x38/0x50
[00000000004d41fc] sync_blockdev+0x1c/0x30
[00000000004cf168] __sync_inodes+0x7c/0xd4
[00000000004cf1d8] sync_inodes+0x18/0x40
[00000000004d1f88] do_sync+0x1c/0x7c
[000000000048ff64] pdflush+0x110/0x1c0
[00000000004655e4] kthread+0x48/0x7c
[00000000004271a0] kernel_thread+0x3c/0x54
[0000000000465450] kthreadd+0xe0/0x184
pdflush D 00000000004878c8 0 164 2
Call Trace:
[00000000006d9f74] io_schedule+0x20/0x3c
[00000000004878c8] sync_page+0x84/0x94
[00000000006da2f8] __wait_on_bit+0x64/0xc0
[0000000000487b20] wait_on_page_bit+0x8c/0x9c
[00000000004880bc] wait_on_page_writeback_range+0x68/0x124
[00000000004881a4] filemap_fdatawait+0x2c/0x40
[000000000048844c] filemap_write_and_wait+0x38/0x50
[00000000004d41fc] sync_blockdev+0x1c/0x30
[00000000004cf168] __sync_inodes+0x7c/0xd4
[00000000004cf1d8] sync_inodes+0x18/0x40
[00000000004d1f88] do_sync+0x1c/0x7c
[000000000048ff64] pdflush+0x110/0x1c0
[00000000004655e4] kthread+0x48/0x7c
[00000000004271a0] kernel_thread+0x3c/0x54
[0000000000465450] kthreadd+0xe0/0x184
kjournald D 00000000005443d4 0 405 2
Call Trace:
[000000000053f394] journal_commit_transaction+0x1c4/0x1648
[00000000005443d4] kjournald+0x134/0x2fc
[00000000004655e4] kthread+0x48/0x7c
[00000000004271a0] kernel_thread+0x3c/0x54
[0000000000465450] kthreadd+0xe0/0x184
metalog D 000000000053d758 0 1413 1
Call Trace:
[000000000053d4b0] start_this_handle+0x2b0/0x4a0
[000000000053d758] journal_start+0xb8/0x118
[00000000005117c0] ext3_journal_start_sb+0x54/0x64
[000000000050b09c] ext3_dirty_inode+0x18/0xc8
[00000000004cf318] __mark_inode_dirty+0x2c/0x180
[00000000004c6674] file_update_time+0xe0/0x118
[0000000000488b44] __generic_file_aio_write_nolock+0x26c/0x36c
[0000000000488c9c] generic_file_aio_write+0x58/0xc8
[0000000000505a30] ext3_file_write+0x28/0xd8
[00000000004b2d88] do_sync_write+0x90/0xe0
[00000000004b35e8] vfs_write+0x7c/0x118
[00000000004b3a18] sys_write+0x38/0x68
[0000000000406154] linux_sparc_syscall32+0x34/0x40
touch D 00000000004d4400 0 1317 1
Call Trace:
[00000000006d9f74] io_schedule+0x20/0x3c
[00000000004d4400] sync_buffer+0x4c/0x5c
[00000000006da1d8] __wait_on_bit_lock+0x64/0xa4
[00000000006da280] out_of_line_wait_on_bit_lock+0x68/0x7c
[00000000004d474c] __lock_buffer+0x34/0x44
[000000000053cacc] do_get_write_access+0xd4/0x65c
[000000000053d078] journal_get_write_access+0x24/0x40
[0000000000516a08] __ext3_journal_get_write_access+0x14/0x50
[0000000000507db4] ext3_reserve_inode_write+0x40/0x88
[0000000000507e28] ext3_mark_inode_dirty+0x2c/0x5c
[000000000050b12c] ext3_dirty_inode+0xa8/0xc8
[00000000004cf318] __mark_inode_dirty+0x2c/0x180
[00000000004c6b6c] inode_setattr+0x120/0x138
[000000000050b034] ext3_setattr+0x174/0x1c4
[00000000004c6d4c] notify_change+0x1c8/0x328
[00000000004d217c] utimes_common+0x170/0x194
init D 000000000053d758 0 1336 1
Call Trace:
[000000000053d4b0] start_this_handle+0x2b0/0x4a0
[000000000053d758] journal_start+0xb8/0x118
[00000000005117c0] ext3_journal_start_sb+0x54/0x64
[000000000050b09c] ext3_dirty_inode+0x18/0xc8
[00000000004cf318] __mark_inode_dirty+0x2c/0x180
[00000000004c6674] file_update_time+0xe0/0x118
[0000000000488b44] __generic_file_aio_write_nolock+0x26c/0x36c
[0000000000488c9c] generic_file_aio_write+0x58/0xc8
[0000000000505a30] ext3_file_write+0x28/0xd8
[00000000004b2d88] do_sync_write+0x90/0xe0
[00000000004b35e8] vfs_write+0x7c/0x118
[00000000004b3a18] sys_write+0x38/0x68
[0000000000406154] linux_sparc_syscall32+0x34/0x40


2008-11-13 12:56:51

by Alexander Beregalov

[permalink] [raw]
Subject: Re: Deadlock at io_schedule? (Re: linux-next: Tree for November 3)

2008/11/3 Alexander Beregalov <[email protected]>:
> Hi
>
> 2.6.28-rc3-next-20081103 on sparc64

The same on SMP x86_64, 2.6.28-rc4-next-20081113
Please have a look anybody


[ 5383.663175] Showing all locks held in the system:
[ 5383.663187] 1 lock held by agetty/1958:
[ 5383.663190] #0: (&tty->atomic_read_lock){--..}, at:
[<ffffffff80400ded>] n_tty_read+0x29b/0x7bd
[ 5383.663203] 1 lock held by agetty/1959:
[ 5383.663205] #0: (&tty->atomic_read_lock){--..}, at:
[<ffffffff80400ded>] n_tty_read+0x29b/0x7bd
[ 5383.663214] 1 lock held by agetty/1960:
[ 5383.663216] #0: (&tty->atomic_read_lock){--..}, at:
[<ffffffff80400ded>] n_tty_read+0x29b/0x7bd
[ 5383.663224] 1 lock held by agetty/1963:
[ 5383.663226] #0: (&tty->atomic_read_lock){--..}, at:
[<ffffffff80400ded>] n_tty_read+0x29b/0x7bd
[ 5383.663234] 1 lock held by agetty/1964:
[ 5383.663236] #0: (&tty->atomic_read_lock){--..}, at:
[<ffffffff80400ded>] n_tty_read+0x29b/0x7bd
[ 5383.663244] 1 lock held by agetty/1966:
[ 5383.663246] #0: (&tty->atomic_read_lock){--..}, at:
[<ffffffff80400ded>] n_tty_read+0x29b/0x7bd
[ 5383.663255] 2 locks held by bash/2011:
[ 5383.663257] #0: (sysrq_key_table_lock){....}, at:
[<ffffffff804102fb>] __handle_sysrq+0x26/0x158
[ 5383.663264] #1: (tasklist_lock){..--}, at: [<ffffffff802569d8>]
debug_show_all_locks+0x4d/0x17f
[ 5383.663274] 1 lock held by bash/2028:
[ 5383.663276] #0: (&(&ip->i_iolock)->mr_lock){----}, at:
[<ffffffff8037646b>] xfs_ilock+0x44/0x79
[ 5383.663285]
[ 5383.663287] =============================================
[ 5383.663288]
[ 5394.063573] SysRq : Show Blocked State
[ 5394.063608] task PC stack pid father
[ 5394.063624] pdflush D ffff880004d9a740 3256 313 2
[ 5394.063631] ffff88007d265d50 0000000000000046 ffff88007dac4a80
ffff88007dac4a80
[ 5394.063638] ffffffff80930740 ffffffff80930740 ffff88007dac4a80
ffff88007d8092a0
[ 5394.063645] ffff88007dac4dc8 0000000180231acf ffff88007d265ce0
ffff88007dac4dc8
[ 5394.063652] Call Trace:
[ 5394.063661] [<ffffffff8052792b>] ? _spin_unlock_irqrestore+0x66/0x74
[ 5394.063667] [<ffffffff80240834>] ? __mod_timer+0xd3/0xe5
[ 5394.063671] [<ffffffff80525c34>] schedule_timeout+0x8d/0xb4
[ 5394.063675] [<ffffffff802401cb>] ? process_timeout+0x0/0xb
[ 5394.063679] [<ffffffff8024077e>] ? __mod_timer+0x1d/0xe5
[ 5394.063683] [<ffffffff80524f75>] __sched_text_start+0x2d/0x39
[ 5394.063688] [<ffffffff80282c15>] congestion_wait+0x6b/0x87
[ 5394.063692] [<ffffffff8024a5db>] ? autoremove_wake_function+0x0/0x38
[ 5394.063697] [<ffffffff802b80f8>] ? writeback_inodes+0xec/0xf4
[ 5394.063702] [<ffffffff8027bc65>] wb_kupdate+0xc4/0x11e
[ 5394.063706] [<ffffffff8027c6bf>] pdflush+0x11a/0x1cd
[ 5394.063710] [<ffffffff8027bba1>] ? wb_kupdate+0x0/0x11e
[ 5394.063713] [<ffffffff8027c5a5>] ? pdflush+0x0/0x1cd
[ 5394.063717] [<ffffffff8024a1e9>] kthread+0x49/0x76
[ 5394.063722] [<ffffffff8020c8a9>] child_rip+0xa/0x11
[ 5394.063727] [<ffffffff80231966>] ? finish_task_switch+0x0/0xb9
[ 5394.063730] [<ffffffff8020bd98>] ? restore_args+0x0/0x30
[ 5394.063734] [<ffffffff8024a1a0>] ? kthread+0x0/0x76
[ 5394.063737] [<ffffffff8020c89f>] ? child_rip+0x0/0x11
[ 5394.063741] xfsbufd D ffff880005138740 6032 695 2
[ 5394.063748] ffff88007cf35c80 0000000000000046 ffff88007cf35bf0
ffffffff80255c61
[ 5394.063754] ffffffff80930740 ffffffff80930740 ffff88007cf70000
ffff88007dfa4a80
[ 5394.063760] ffff88007cf70348 0000000380255c61 ffff88007cf70000
ffff88007cf70348
[ 5394.063765] Call Trace:
[ 5394.063770] [<ffffffff80255c61>] ? get_lock_stats+0x2d/0x5c
[ 5394.063773] [<ffffffff80255c9e>] ? put_lock_stats+0xe/0x27
[ 5394.063777] [<ffffffff802300af>] ? get_parent_ip+0x11/0x41
[ 5394.063781] [<ffffffff80525a88>] io_schedule+0x2d/0x39
[ 5394.063786] [<ffffffff803a8cec>] get_request_wait+0x10a/0x1a6
[ 5394.063790] [<ffffffff8024a5db>] ? autoremove_wake_function+0x0/0x38
[ 5394.063794] [<ffffffff803a51c5>] ? elv_merge+0x16d/0x19e
[ 5394.063798] [<ffffffff803a913b>] __make_request+0x3b3/0x4b1
[ 5394.063802] [<ffffffff80277502>] ? mempool_alloc+0x56/0x10e
[ 5394.063806] [<ffffffff803a792c>] generic_make_request+0x2f4/0x337
[ 5394.063810] [<ffffffff80231acf>] ? sub_preempt_count+0xb0/0xc4
[ 5394.063813] [<ffffffff803a7a08>] submit_bio+0x99/0xa2
[ 5394.063818] [<ffffffff803939fa>] _xfs_buf_ioapply+0x215/0x243
[ 5394.063822] [<ffffffff8039477d>] xfs_buf_iorequest+0x44/0x75
[ 5394.063826] [<ffffffff803983c7>] xfs_bdstrat_cb+0x19/0x3e
[ 5394.063830] [<ffffffff803947c0>] xfs_buf_iostrategy+0x12/0x1b
[ 5394.063833] [<ffffffff803949e6>] xfsbufd+0xa8/0xe9
[ 5394.063837] [<ffffffff8039493e>] ? xfsbufd+0x0/0xe9
[ 5394.063840] [<ffffffff8024a1e9>] kthread+0x49/0x76
[ 5394.063844] [<ffffffff8020c8a9>] child_rip+0xa/0x11
[ 5394.063848] [<ffffffff80231966>] ? finish_task_switch+0x0/0xb9
[ 5394.063856] [<ffffffff8020bd98>] ? restore_args+0x0/0x30
[ 5394.063858] [<ffffffff8024a1a0>] ? kthread+0x0/0x76
[ 5394.063861] [<ffffffff8020c89f>] ? child_rip+0x0/0x11
[ 5394.063863] bash D ffff880004d9a740 5240 2028 2015
[ 5394.063868] ffff8800730f1a88 0000000000000046 0000000000000001
ffff88007ca26430
[ 5394.063872] ffffffff80930740 ffffffff80930740 ffff88007ca25d20
ffff88007df62540
[ 5394.063876] ffff88007ca26068 0000000180aacde0 ffff8800730f1a38
ffff88007ca26068
[ 5394.063880] Call Trace:
[ 5394.063883] [<ffffffff803a8721>] ? generic_unplug_device+0x1d/0x46
[ 5394.063885] [<ffffffff80525a88>] io_schedule+0x2d/0x39
[ 5394.063888] [<ffffffff80275742>] sync_page+0x69/0x70
[ 5394.063890] [<ffffffff80275757>] sync_page_killable+0xe/0x46
[ 5394.063893] [<ffffffff80525cf1>] __wait_on_bit_lock+0x45/0x79
[ 5394.063895] [<ffffffff80275749>] ? sync_page_killable+0x0/0x46
[ 5394.063897] [<ffffffff802755ed>] __lock_page_killable+0x63/0x6a
[ 5394.063900] [<ffffffff8024a613>] ? wake_bit_function+0x0/0x2a
[ 5394.063902] [<ffffffff80275510>] ? find_get_page+0x75/0x85
[ 5394.063905] [<ffffffff80275626>] lock_page_killable+0x32/0x3a
[ 5394.063907] [<ffffffff80276de8>] generic_file_aio_read+0x39d/0x581
[ 5394.063910] [<ffffffff8037646b>] ? xfs_ilock+0x44/0x79
[ 5394.063913] [<ffffffff80399285>] xfs_read+0x17b/0x1f2
[ 5394.063915] [<ffffffff8039576a>] xfs_file_aio_read+0x51/0x53
[ 5394.063919] [<ffffffff8029e678>] do_sync_read+0xe7/0x12d
[ 5394.063921] [<ffffffff80231acf>] ? sub_preempt_count+0xb0/0xc4
[ 5394.063924] [<ffffffff8024a5db>] ? autoremove_wake_function+0x0/0x38
[ 5394.063926] [<ffffffff80255d69>] ? lock_release_holdtime+0xb2/0xb7
[ 5394.063929] [<ffffffff8029ef37>] vfs_read+0xa4/0xde
[ 5394.063932] [<ffffffff802a3692>] kernel_read+0x43/0x5b
[ 5394.063935] [<ffffffff80283f31>] ? might_fault+0x4d/0xa1
[ 5394.063937] [<ffffffff802a3787>] prepare_binprm+0xdd/0xe1
[ 5394.063939] [<ffffffff802a3ddc>] do_execve+0xde/0x1da
[ 5394.063942] [<ffffffff80209ce6>] sys_execve+0x3e/0x60
[ 5394.063944] [<ffffffff8020bb8a>] stub_execve+0x6a/0xc0