Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760712AbYFMIGA (ORCPT ); Fri, 13 Jun 2008 04:06:00 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1753420AbYFMIFl (ORCPT ); Fri, 13 Jun 2008 04:05:41 -0400 Received: from dwdmx4.dwd.de ([141.38.3.230]:49251 "EHLO dwdmx4.dwd.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754186AbYFMIFW (ORCPT ); Fri, 13 Jun 2008 04:05:22 -0400 Date: Fri, 13 Jun 2008 08:05:12 +0000 (GMT) From: Holger Kiehl X-X-Sender: kiehl@diagnostix.dwd.de To: "Aneesh Kumar K.V" Cc: Theodore Tso , Solofo.Ramangalahy@bull.net, Nick Dokos , linux-ext4@vger.kernel.org, linux-kernel Subject: Re: Performance of ext4 In-Reply-To: Message-ID: References: <20080611105945.GB9008@skywalker> <18563.1213215457@alphaville.zko.hp.com> <18513.345.553912.449710@frecb006361.adech.frec.bull.fr> <20080612131928.GB18229@mit.edu> <20080612180605.GD22481@skywalker> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 64471 Lines: 1345 On Thu, 12 Jun 2008, Holger Kiehl wrote: > On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote: > >> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote: >>> This time there is no OOPS and system is still up running without any >>> problem (except any process wanting to write something to this filesystem >>> gets stuck forever). >>> >>> What can I do to help find the problem? The system is still up with all >>> those >>> process hanging in D-state. >>> >> >> if you can login to the system get the dmesg output after >> >> echo t > /proc/sysrq-trigger >> > Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to > reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set. > After recompiling, rebooting and run afdbench first I got an OOPS and the system hanged up solid. The only thing I was able to catch is this: RIP [] jbd2_journal_release_jbd_inode+0xcb/0x100 RSP This was copied by hand. And this I cut 'cut and past' from my terminal: kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89 df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01 So I rebooted run bonnie and then afdbench and I get the same problem as yesterday. All process trying to write something to this filesystem hang in D-state. But now I was able to do the echo t > /proc/sysrq-trigger. Here the dmesg output: node_dirty+0x134/0x147 [] jbd2_journal_start+0x9a/0xce [] ext4_dirty_inode+0x28/0x7b [] __mark_inode_dirty+0x29/0x187 [] file_update_time+0xaf/0xf4 [] __generic_file_aio_write_nolock+0x265/0x37e [] touch_atime+0x12/0xfa [] __wake_up_sync+0x3a/0x56 [] generic_file_aio_write+0x64/0xc4 [] ext4_file_write+0x96/0x11a [] do_sync_write+0xce/0x113 [] autoremove_wake_function+0x0/0x2e [] hrtick_set+0xa1/0x10a [] vfs_write+0xad/0x136 [] sys_write+0x45/0x6e [] tracesys+0xd5/0xda trans_db_log S 00000000ffffffff 0 8010 8005 ffff81016f58d9e8 0000000000000082 0000000000000002 0000000000000000 ffffffff80600cc0 ffff810180001600 0000000000000000 ffff81017f6c1640 ffff81007fb43210 ffff81017f6c1878 0000000280266f65 ffff81017f6c1878 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] ext4_mark_inode_dirty+0x134/0x147 [] __ext4_journal_dirty_metadata+0x1e/0x46 [] bit_waitqueue+0x10/0xa0 [] do_get_write_access+0x3ab/0x3ed [] __getblk+0x1e/0x28d [] find_lock_page+0x29/0x87 [] filemap_fault+0x1cb/0x329 [] __do_fault+0x321/0x369 [] core_sys_select+0x1ab/0x253 [] handle_mm_fault+0x348/0x6a3 [] do_page_fault+0x471/0x825 [] do_filp_open+0x3f2/0x832 [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda archive_watch S 00000000ffffffff 0 8011 8005 ffff81017f79b9e8 0000000000000086 ffff810000002618 0000000100000000 ffffffff80600cc0 0000000000000000 ffff810000002608 ffff81017f6c10b0 ffffffff8054d350 ffff81017f6c12e8 0000000000001580 ffff81017f6c12e8 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] __find_get_block+0x14c/0x15c [] __getblk+0x1e/0x28d [] ext4_getblk+0xb4/0x170 [] dput+0x21/0x10a [] __d_lookup+0xbd/0x107 [] ext4fs_dirhash+0x10d/0x1d2 [] ext4_htree_store_dirent+0xe6/0xf1 [] htree_dirblock_to_tree+0x12c/0x13e [] core_sys_select+0x1ab/0x253 [] filldir+0x68/0xb7 [] filldir+0x0/0xb7 [] ext4_readdir+0x1c1/0x524 [] filldir+0x0/0xb7 [] do_filp_open+0x3da/0x832 [] cp_new_stat+0xe9/0xfc [] mnt_want_write+0x25/0x77 [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda input_log D 00000000ffffffff 0 8012 8005 ffff81017f79db58 0000000000000086 0000000000000008 0000000000000000 ffffffff80600cc0 0000000000000008 0000000000000000 ffff81017f6c5370 ffff81007fb442c0 ffff81017f6c55a8 0000000380290dd5 ffff81017f6c55a8 Call Trace: [] default_wake_function+0x0/0xe [] start_this_handle+0x2ad/0x370 [] autoremove_wake_function+0x0/0x2e [] ext4_mark_inode_dirty+0x134/0x147 [] jbd2_journal_start+0x9a/0xce [] ext4_dirty_inode+0x28/0x7b [] __mark_inode_dirty+0x29/0x187 [] file_update_time+0xaf/0xf4 [] __generic_file_aio_write_nolock+0x265/0x37e [] touch_atime+0x12/0xfa [] __wake_up_sync+0x3a/0x56 [] generic_file_aio_write+0x64/0xc4 [] ext4_file_write+0x96/0x11a [] do_sync_write+0xce/0x113 [] autoremove_wake_function+0x0/0x2e [] hrtick_set+0xa1/0x10a [] vfs_write+0xad/0x136 [] sys_write+0x45/0x6e [] tracesys+0xd5/0xda output_log D 00000000ffffffff 0 8013 8005 ffff81016f7b9b58 0000000000000086 0000000000000008 0000000000000000 ffffffff80600cc0 0000000000000008 0000000000000000 ffff81017f6c4de0 ffff81007fb42160 ffff81017f6c5018 0000000180290dd5 ffff81017f6c5018 Call Trace: [] default_wake_function+0x0/0xe [] start_this_handle+0x2ad/0x370 [] autoremove_wake_function+0x0/0x2e [] ext4_mark_inode_dirty+0x134/0x147 [] jbd2_journal_start+0x9a/0xce [] ext4_dirty_inode+0x28/0x7b [] __mark_inode_dirty+0x29/0x187 [] file_update_time+0xaf/0xf4 [] __generic_file_aio_write_nolock+0x265/0x37e [] touch_atime+0x12/0xfa [] __wake_up_sync+0x3a/0x56 [] generic_file_aio_write+0x64/0xc4 [] ext4_file_write+0x96/0x11a [] do_sync_write+0xce/0x113 [] autoremove_wake_function+0x0/0x2e [] hrtick_set+0xa1/0x10a [] vfs_write+0xad/0x136 [] sys_write+0x45/0x6e [] tracesys+0xd5/0xda delete_log S 0000000000000000 0 8014 8005 ffff81017f7099e8 0000000000000086 0000000000000002 0000000000000000 ffffffff80600cc0 ffff810000002600 0000000000000000 ffff81017f6c26f0 ffff81017f784850 ffff81017f6c2928 0000000080266f65 ffff81017f6c2928 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] ext4_mark_inode_dirty+0x134/0x147 [] __ext4_journal_dirty_metadata+0x1e/0x46 [] zone_statistics+0x3c/0x90 [] get_page_from_freelist+0x461/0x5d5 [] zone_statistics+0x3c/0x90 [] get_page_from_freelist+0x461/0x5d5 [] __alloc_pages_internal+0xd2/0x40c [] core_sys_select+0x1ab/0x253 [] handle_mm_fault+0x2c9/0x6a3 [] do_page_fault+0x471/0x825 [] mmap_region+0x41a/0x505 [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda production_lo S 00000000ffffffff 0 8015 8005 ffff81017f70b9e8 0000000000000082 0000000000000002 0000000000000000 ffffffff80600cc0 ffff810200001600 0000000000000000 ffff81017f6c42c0 ffff81007fb43210 ffff81017f6c44f8 00000002794b5d20 ffff81017f6c44f8 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] __ext4_journal_stop+0x1f/0x3d [] __mark_inode_dirty+0xed/0x187 [] generic_file_buffered_write+0x22c/0x658 [] __mark_inode_dirty+0xed/0x187 [] __generic_file_aio_write_nolock+0x34a/0x37e [] __do_fault+0x321/0x369 [] core_sys_select+0x1ab/0x253 [] do_sync_write+0xce/0x113 [] do_page_fault+0x471/0x825 [] mmap_region+0x41a/0x505 [] autoremove_wake_function+0x0/0x2e [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda distribution_ S 00000000ffffffff 0 8016 8005 ffff81016f5959e8 0000000000000086 0000000000080780 ffffe20008ba3f98 ffffffff80600cc0 ffffffff802a3869 0000fee20000011d ffff81017f6c3d30 ffff81007fb442c0 ffff81017f6c3f68 00000003794b5d20 ffff81017f6c3f68 Call Trace: [] __find_get_block_slow+0xcf/0xdb [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] __ext4_journal_stop+0x1f/0x3d [] __mark_inode_dirty+0x29/0x187 [] generic_file_buffered_write+0x22c/0x658 [] ext4_mark_inode_dirty+0x134/0x147 [] current_fs_time+0x1e/0x24 [] remove_suid+0x18/0x4b [] __generic_file_aio_write_nolock+0x34a/0x37e [] touch_atime+0x12/0xfa [] __wake_up_sync+0x3a/0x56 [] core_sys_select+0x1ab/0x253 [] do_sync_write+0xce/0x113 [] autoremove_wake_function+0x0/0x2e [] hrtick_set+0xa1/0x10a [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda amg S 00000000ffffffff 0 8017 8005 ffff81016f5e59e8 0000000000000082 ffff81023ec9a070 0000000000000282 ffffffff80600cc0 ffff81007f2e4ac0 0000000000011210 ffff81017f6c37a0 ffffffff8054d350 ffff81017f6c39d8 0000000000000040 ffff81017f6c39d8 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] default_wake_function+0x0/0xe [] __getblk+0x1e/0x28d [] zone_statistics+0x3c/0x90 [] __d_lookup+0xbd/0x107 [] do_lookup+0x63/0x1a1 [] dput+0x21/0x10a [] __d_lookup+0xbd/0x107 [] do_lookup+0x63/0x1a1 [] dput+0x21/0x10a [] __link_path_walk+0xcb1/0xdd9 [] core_sys_select+0x1ab/0x253 [] getname+0x13e/0x1a0 [] mntput_no_expire+0x27/0x11e [] remove_wait_queue+0x12/0x41 [] do_wait+0xae4/0xb88 [] cp_new_stat+0xe9/0xfc [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda afdd S 00000000ffffffff 0 8018 8005 ffff81016f5e79e8 0000000000000086 ffff810200001618 0000000200000000 ffffffff80600cc0 0000000000000000 ffff810200001608 ffff81017f6c0000 ffff81007fb442c0 ffff81017f6c0238 0000000300000b00 ffff81017f6c0238 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] extract_buf+0x84/0xfc [] mix_pool_bytes_extract+0x58/0x148 [] extract_buf+0x84/0xfc [] core_sys_select+0x1ab/0x253 [] account+0xd6/0xe6 [] __wake_up+0x38/0x4f [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda dir_check D ffff810027160690 0 8019 8017 ffff81007e8b1d70 0000000000000086 ffff81007e8b1d38 ffff81007e8a0e00 ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8100707910b0 ffff8100469a2160 ffff8100707912e8 000000028028c8f9 ffff8100707912e8 Call Trace: [] dput+0x21/0x10a [] __link_path_walk+0xcb1/0xdd9 [] mntput_no_expire+0x27/0x11e [] __mutex_lock_slowpath+0x6a/0xa0 [] mutex_lock+0xa/0xb [] lock_rename+0x35/0xc5 [] do_rename+0x98/0x1b2 [] cp_new_stat+0xe9/0xfc [] audit_syscall_exit+0x2ee/0x30c [] sys_renameat+0x4c/0x6b [] tracesys+0xd5/0xda afd_stat S 00000000ffffffff 0 8020 8005 ffff81016f7fd9e8 0000000000000086 0000000000000000 000000000009829c ffffffff80600cc0 ffffffff80323d50 ffff810201e124a0 ffff81017f6c0590 ffff81007fb442c0 ffff81017f6c07c8 0000000300000000 ffff81017f6c07c8 Call Trace: [] rb_insert_color+0x66/0xe2 [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] bit_waitqueue+0x10/0xa0 [] __pollwait+0x0/0xe3 [] do_get_write_access+0x3ab/0x3ed [] __getblk+0x1e/0x28d [] wake_bit_function+0x0/0x23 [] __ext4_journal_dirty_metadata+0x1e/0x46 [] ext4_mark_iloc_dirty+0x415/0x47e [] ext4_mark_inode_dirty+0x134/0x147 [] walk_page_buffers+0x65/0x8b [] ext4_bh_unmapped+0x0/0xe [] __up_read+0x13/0x8a [] __prop_inc_single+0x1d/0x36 [] do_wp_page+0x47d/0x4ba [] core_sys_select+0x1ab/0x253 [] handle_mm_fault+0x62d/0x6a3 [] hrtick_set+0xa1/0x10a [] __up_read+0x13/0x8a [] do_page_fault+0x471/0x825 [] do_notify_resume+0x81f/0x840 [] cp_new_stat+0xe9/0xfc [] autoremove_wake_function+0x0/0x2e [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda fd S 00000000ffffffff 0 8021 8005 ffff81016f5139e8 0000000000000082 0000000000000000 0000000000000000 ffffffff80600cc0 0000000000000001 0000000180001608 ffff81017f6c3210 ffff81007fb442c0 ffff81017f6c3448 0000000300000b00 ffff81017f6c3448 Call Trace: [] lock_timer_base+0x26/0x4b [] __mod_timer+0xaa/0xb8 [] schedule_timeout+0xa6/0xc9 [] process_timeout+0x0/0x5 [] do_select+0x414/0x482 [] __pollwait+0x0/0xe3 [] default_wake_function+0x0/0xe [] default_wake_function+0x0/0xe [] default_wake_function+0x0/0xe [] default_wake_function+0x0/0xe [] default_wake_function+0x0/0xe [] default_wake_function+0x0/0xe [] dput+0x21/0x10a [] __d_lookup+0xbd/0x107 [] mntput_no_expire+0x27/0x11e [] __link_path_walk+0xdbb/0xdd9 [] core_sys_select+0x1ab/0x253 [] getname+0x13e/0x1a0 [] __user_walk_fd+0x41/0x4c [] vfs_stat_fd+0x1b/0x4a [] sys_select+0xb7/0x167 [] tracesys+0xd5/0xda show_bench_st S 00000000ffffffff 0 8043 2879 ffff810023a77ea8 0000000000000082 00007fff24bbfe80 00007f631cabd000 ffffffff80600cc0 ffffffff80242f85 ffff810023a77ed8 ffff810070793d30 ffff81007fb43210 ffff810070793f68 0000000200000000 ffff810070793f68 Call Trace: [] getnstimeofday+0x38/0x84 [] hrtimer_start+0x100/0x122 [] hrtimer_try_to_cancel+0x61/0x6a [] do_nanosleep+0x64/0xa6 [] hrtimer_nanosleep+0x54/0xbf [] hrtimer_wakeup+0x0/0x22 [] sys_nanosleep+0x4c/0x62 [] tracesys+0xd5/0xda tee D 0000000000000000 0 8044 2879 ffff810046939b58 0000000000000086 ffff81001af65240 ffff810049075630 ffffffff80600cc0 ffffffff80324411 ffff810049075630 ffff8100707942c0 ffff81007e84de90 ffff8100707944f8 00000000ff8254a0 ffff8100707944f8 Call Trace: [] __up_read+0x13/0x8a [] __do_fault+0x321/0x369 [] start_this_handle+0x2ad/0x370 [] autoremove_wake_function+0x0/0x2e [] jbd2_journal_start+0x9a/0xce [] ext4_dirty_inode+0x28/0x7b [] __mark_inode_dirty+0x29/0x187 [] file_update_time+0xaf/0xf4 [] __generic_file_aio_write_nolock+0x265/0x37e [] generic_file_aio_write+0x64/0xc4 [] ext4_file_write+0x96/0x11a [] do_sync_write+0xce/0x113 [] autoremove_wake_function+0x0/0x2e [] tty_write+0x200/0x21b [] vfs_write+0xad/0x136 [] sys_write+0x45/0x6e [] tracesys+0xd5/0xda dir_check ? ffff810070796518 0 8413 8019 ffff81003453bef8 0000000000000046 ffff810070462908 0000000000000056 ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070796420 ffff81017f6c42c0 ffff810070796658 0000000200000000 ffff810070796658 Call Trace: [] do_notify_parent+0x18c/0x199 [] do_exit+0x661/0x665 [] do_group_exit+0x66/0x93 [] tracesys+0xd5/0xda dir_check ? ffff810070796aa8 0 8415 8019 ffff8100492c9ef8 0000000000000046 ffff810070462908 0000000000000056 ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff8100707969b0 ffff8101ff6fbd30 ffff810070796be8 0000000000000000 ffff810070796be8 Call Trace: [] do_notify_parent+0x18c/0x199 [] do_exit+0x661/0x665 [] do_group_exit+0x66/0x93 [] tracesys+0xd5/0xda dir_check ? ffff81016f416aa8 0 8460 7996 ffff81016f6c1ef8 0000000000000046 ffff810070461048 0000000000000056 ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff81016f4169b0 ffff81017f7826f0 ffff81016f416be8 0000000100000000 ffff81016f416be8 Call Trace: [] do_notify_parent+0x18c/0x199 [] do_exit+0x661/0x665 [] do_group_exit+0x66/0x93 [] tracesys+0xd5/0xda dir_check ? ffff81016f417038 0 8463 7996 ffff81016f429ef8 0000000000000046 ffff810070461048 0000000000000056 ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff81016f416f40 ffff81027759d370 ffff81016f417178 0000000100000003 ffff81016f417178 Call Trace: [] do_notify_parent+0x18c/0x199 [] do_exit+0x661/0x665 [] do_group_exit+0x66/0x93 [] tracesys+0xd5/0xda dir_check ? ffff810070790c18 0 8492 7973 ffff81002382def8 0000000000000046 ffff8101ff70fb88 0000000000000056 ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070790b20 ffff81027f148000 ffff810070790d58 0000000100000000 ffff810070790d58 Call Trace: [] do_notify_parent+0x18c/0x199 [] do_exit+0x661/0x665 [] do_group_exit+0x66/0x93 [] tracesys+0xd5/0xda dir_check ? 00000000ffffffff 0 8515 7973 ffff81006e4a7ef8 0000000000000046 ffff8101ff70fb88 0000000000000056 ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070791640 ffff81007fb43210 ffff810070791878 0000000200000000 ffff810070791878 Call Trace: [] do_notify_parent+0x18c/0x199 [] do_exit+0x661/0x665 [] do_group_exit+0x66/0x93 [] tracesys+0xd5/0xda sf_ftp D 0000000000000000 0 9928 8021 ffff81027851fbe8 0000000000000086 ffff81017f72de90 0000000300000000 ffffffff80600cc0 0000000000000000 0000000000000000 ffff81027f2bcde0 ffff81017f6c1bd0 ffff81027f2bd018 0000000100000000 ffff81027f2bd018 Call Trace: [] :nf_conntrack:nf_conntrack_in+0x455/0x508 [] _read_lock_bh+0x9/0x19 [] start_this_handle+0x2ad/0x370 [] autoremove_wake_function+0x0/0x2e [] jbd2_journal_start+0x9a/0xce [] ext4_dirty_inode+0x28/0x7b [] __mark_inode_dirty+0x29/0x187 [] file_update_time+0xaf/0xf4 [] pipe_write+0x4bc/0x4ce [] path_walk+0xaf/0xb9 [] do_sync_write+0xce/0x113 [] __dentry_open+0x141/0x226 [] autoremove_wake_function+0x0/0x2e [] vfs_write+0xad/0x136 [] sys_write+0x45/0x6e [] tracesys+0xd5/0xda vsftpd S ffff810052a38000 0 9929 2487 ffff810046a57bc8 0000000000000086 0000000000000096 ffffffff80224edc ffffffff80600cc0 0000000000000001 0000000000000096 ffff8100469a37a0 ffff8101b97f8590 ffff8100469a39d8 0000000100000001 ffff8100469a39d8 Call Trace: [] try_to_wake_up+0x155/0x166 [] memcpy_fromiovec+0x34/0x63 [] skb_queue_tail+0x17/0x3e [] sock_def_readable+0x10/0x5d [] unix_stream_sendmsg+0x293/0x353 [] schedule_timeout+0x1e/0xc9 [] prepare_to_wait+0x15/0x5f [] unix_stream_recvmsg+0x282/0x582 [] __wake_up_sync+0x3a/0x56 [] autoremove_wake_function+0x0/0x2e [] sock_aio_read+0x107/0x112 [] do_sync_read+0xce/0x113 [] autoremove_wake_function+0x0/0x2e [] vfs_read+0xbd/0x132 [] sys_read+0x45/0x6e [] tracesys+0xd5/0xda vsftpd D 00000000ffffffff 0 9937 9929 ffff8101f8e99dd0 0000000000000086 ffffffff8040820c ffff8101f8e99d34 ffffffff80600cc0 0000000ed348224f ffff81007e8a0ec8 ffff8101b97f8590 ffff81007fb42160 ffff8101b97f87c8 0000000100000000 ffff8101b97f87c8 Call Trace: [] sock_common_recvmsg+0x30/0x45 [] path_walk+0xaf/0xb9 [] __mutex_lock_slowpath+0x6a/0xa0 [] mutex_lock+0xa/0xb [] do_filp_open+0x10c/0x832 [] get_unused_fd_flags+0x7f/0x10e [] do_sys_open+0x46/0xc3 [] tracesys+0xd5/0xda sf_ftp D 0000000000000000 0 9945 8021 ffff81027f295be8 0000000000000082 ffff81007e856290 ffffffff8042e6b5 ffffffff80600cc0 ffff81027f295b90 ffff8100370343f0 ffff81027f2bd900 ffff81027f2ba160 ffff81027f2bdb38 0000000037008438 ffff81027f2bdb38 Call Trace: [] ip_queue_xmit+0x2a2/0x2f8 [] __getblk+0x1e/0x28d [] lock_timer_base+0x26/0x4b [] start_this_handle+0x2ad/0x370 [] autoremove_wake_function+0x0/0x2e [] check_preempt_wakeup+0x7f/0xbc [] jbd2_journal_start+0x9a/0xce [] ext4_dirty_inode+0x28/0x7b [] __mark_inode_dirty+0x29/0x187 [] file_update_time+0xaf/0xf4 [] pipe_write+0x4bc/0x4ce [] do_sync_write+0xce/0x113 [] do_page_fault+0x471/0x825 [] autoremove_wake_function+0x0/0x2e [] free_pages_and_swap_cache+0x70/0x8b [] unmap_region+0x10f/0x125 [] vfs_write+0xad/0x136 [] sys_write+0x45/0x6e [] tracesys+0xd5/0xda vsftpd S ffff81016f680000 0 9946 2487 ffff8100706ddbc8 0000000000000082 0000000000000096 ffffffff80224edc ffffffff80600cc0 0000000000000001 0000000000000096 ffff8100469a5900 ffff81016f4137a0 ffff8100469a5b38 0000000000000001 ffff8100469a5b38 Call Trace: [] try_to_wake_up+0x155/0x166 [] memcpy_fromiovec+0x34/0x63 [] skb_queue_tail+0x17/0x3e [] sock_def_readable+0x10/0x5d [] unix_stream_sendmsg+0x293/0x353 [] schedule_timeout+0x1e/0xc9 [] prepare_to_wait+0x15/0x5f [] unix_stream_recvmsg+0x282/0x582 [