From: Chris Mason Subject: Re: [BUG] fatal hang untarring 90GB file, possibly writeback related. Date: Wed, 27 Apr 2011 13:34:03 -0400 Message-ID: <1303925374-sup-7968@think> References: <1303920553.2583.7.camel@mulgrave.site> <1303921583-sup-4021@think> <1303923000.2583.8.camel@mulgrave.site> <1303923177-sup-2603@think> <1303924902.2583.13.camel@mulgrave.site> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: linux-fsdevel , linux-mm , linux-kernel , "linux-ext4" To: James Bottomley Return-path: In-reply-to: <1303924902.2583.13.camel@mulgrave.site> Sender: owner-linux-mm@kvack.org List-Id: linux-ext4.vger.kernel.org [ cc'd linux-ext4 ] James is hitting softlockups in kswapd while doing writes to a large ext4 file. Excerpts from James Bottomley's message of 2011-04-27 13:21:42 -0400: > On Wed, 2011-04-27 at 12:54 -0400, Chris Mason wrote: > > Ok, I'd try turning it on so we catch the sleeping with a spinlock he= ld > > case better. >=20 > Will do, that's CONFIG_PREEMPT (rather than CONFIG_PREEMPT_VOLUNTARY)? >=20 > This is the trace with sysrq-l and sysrq-w >=20 > The repro this time doesn't have a soft lockup, just the tar is hung an= d > one of my CPUs is in 99% system. > [ 454.742935] flush-253:2 D 0000000000000000 0 793 2 0x= 00000000 > [ 454.745425] ffff88006355b710 0000000000000046 ffff88006355b6b0 ffff= ffff00000000 > [ 454.747955] ffff880037ee9700 ffff88006355bfd8 ffff88006355bfd8 0000= 000000013b40 > [ 454.750506] ffffffff81a0b020 ffff880037ee9700 ffff88006355b710 0000= 00018106e7c3 > [ 454.753048] Call Trace: > [ 454.755537] [] do_get_write_access+0x1c6/0x38d > [ 454.758071] [] ? autoremove_wake_function+0x3d/0x= 3d > [ 454.760644] [] jbd2_journal_get_write_access+0x2b= /0x42 > [ 454.763206] [] ? ext4_read_block_bitmap+0x54/0x2d= 0 > [ 454.765770] [] __ext4_journal_get_write_access+0x= 58/0x66 > [ 454.768353] [] ext4_mb_mark_diskspace_used+0x70/0= x2ae > [ 454.770942] [] ext4_mb_new_blocks+0x1c8/0x3c2 > [ 454.773501] [] ext4_ext_map_blocks+0x1961/0x1c04 > [ 454.776082] [] ? radix_tree_gang_lookup_tag_slot+= 0x81/0xa2 > [ 454.778711] [] ? find_get_pages_tag+0x3b/0xd6 > [ 454.781323] [] ext4_map_blocks+0x112/0x1e7 > [ 454.783894] [] mpage_da_map_and_submit+0x93/0x2cd > [ 454.786491] [] ext4_da_writepages+0x2c1/0x44d > [ 454.789090] [] do_writepages+0x21/0x2a So our flusher threads are stuck waiting in jbd2_journal_get_write_access, which means they aren't cleaning dirty pages. In order to get write access, they probably need the transaction to commit: > [ 454.828711] jbd2/dm-2-8 D 0000000000000000 0 799 2 0x= 00000000 > [ 454.831390] ffff88006d59db10 0000000000000046 ffff88006d59daa0 ffff= ffff00000000 > [ 454.834094] ffff88006deb4500 ffff88006d59dfd8 ffff88006d59dfd8 0000= 000000013b40 > [ 454.836788] ffffffff81a0b020 ffff88006deb4500 ffff88006d59dad0 0000= 00016d59dad0 > [ 454.839453] Call Trace: > [ 454.842098] [] ? lock_page+0x3e/0x3e > [ 454.844738] [] ? lock_page+0x3e/0x3e > [ 454.847303] [] io_schedule+0x63/0x7e > [ 454.849877] [] sleep_on_page+0xe/0x12 > [ 454.852469] [] __wait_on_bit+0x48/0x7b > [ 454.855021] [] wait_on_page_bit+0x72/0x74 > [ 454.857583] [] ? autoremove_wake_function+0x3d/0x= 3d > [ 454.860171] [] filemap_fdatawait_range+0x84/0x163 > [ 454.862744] [] filemap_fdatawait+0x24/0x26 > [ 454.865299] [] jbd2_journal_commit_transaction+0x= 922/0x1194 But that seems to be waiting for a page lock. Probably the same page lock held by the flusher thread? Looks like tar is stuck in the same boat down below. > [ 454.867892] [] ? __switch_to+0xc6/0x220 > [ 454.870496] [] kjournald2+0xc9/0x20a > [ 454.873103] [] ? remove_wait_queue+0x3a/0x3a > [ 454.875690] [] ? commit_timeout+0x10/0x10 > [ 454.878327] [] kthread+0x84/0x8c > [ 454.880961] [] kernel_thread_helper+0x4/0x10 > [ 454.883604] [] ? kthread_worker_fn+0x148/0x148 > [ 454.886262] [] ? gs_change+0x13/0x13 > [ 454.888875] tar D ffff88006e573af8 0 991 838 0x= 00000000 > [ 454.891546] ffff880037f5b8a8 0000000000000086 ffff8801002a1d40 0000= 000000000282 > [ 454.894213] ffff88006d644500 ffff880037f5bfd8 ffff880037f5bfd8 0000= 000000013b40 > [ 454.896889] ffff8801002b4500 ffff88006d644500 ffff880037f5b8a8 ffff= ffff8106e7c3 > [ 454.899530] Call Trace: > [ 454.902118] [] ? prepare_to_wait+0x6c/0x78 > [ 454.904724] [] do_get_write_access+0x1c6/0x38d > [ 454.907344] [] ? autoremove_wake_function+0x3d/0x= 3d > [ 454.909967] [] ? ext4_dirty_inode+0x33/0x4c > [ 454.912574] [] jbd2_journal_get_write_access+0x2b= /0x42 > [ 454.915192] [] __ext4_journal_get_write_access+0x= 58/0x66 > [ 454.917819] [] ext4_reserve_inode_write+0x41/0x83 > [ 454.920459] [] ext4_mark_inode_dirty+0x7c/0x1f0 > [ 454.923070] [] ext4_dirty_inode+0x33/0x4c > [ 454.925660] [] __mark_inode_dirty+0x2f/0x175 > [ 454.928247] [] generic_write_end+0x6c/0x7e > [ 454.930865] [] ext4_da_write_end+0x1a5/0x204 > [ 454.933454] [] generic_file_buffered_write+0x17e/= 0x23a > [ 454.936062] [] __generic_file_aio_write+0x242/0x2= 72 > [ 454.938648] [] generic_file_aio_write+0x61/0xba > [ 454.941288] [] ext4_file_write+0x1dc/0x234 > [ 454.943909] [] do_sync_write+0xbf/0xff > [ 454.946501] [] ? fsnotify+0x1eb/0x217 > [ 454.949114] [] ? selinux_file_permission+0x58/0xb= 4 > [ 454.951736] [] ? security_file_permission+0x2e/0x= 33 > [ 454.954349] [] ? rw_verify_area+0xb0/0xcd > [ 454.956943] [] vfs_write+0xac/0xf3 > [ 454.959530] [] sys_write+0x4a/0x6e > [ 454.962129] [] system_call_fastpath+0x16/0x1b -chris -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org