From: "Nanakos V. Chrysostomos" Subject: Re: Ext4 Kernel Crash Date: Tue, 31 Jan 2012 17:04:10 +0200 (EET) Message-ID: <14806.194.177.215.120.1328022250.squirrel@www.wired-net.gr> References: <20120129175057.GA4699@dinofilaria.home> Reply-To: nanakos@wired-net.gr Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7BIT Cc: "Lukas Czerner" , "Jesper Juhl" , linux-ext4@vger.kernel.org To: "Lukas Czerner" Return-path: Received: from mail.eleftheriou.eu ([86.35.2.10]:58802 "EHLO enigma.wired-net.gr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753526Ab2AaPEV (ORCPT ); Tue, 31 Jan 2012 10:04:21 -0500 In-Reply-To: Sender: linux-ext4-owner@vger.kernel.org List-ID: Hi again, for those two machines it is a hang, for some other machines as I wrote in my first email is a crash. Maybe the email subject should have been Crash/Hang.. Cheers, Chris. > Just a follow-up, if you are a Red Hat customer, please, refer this > issue trough the appropriate channels so we can get this issue solved > for you. > > Thanks! > -Lukas > > On Tue, 31 Jan 2012, Lukas Czerner wrote: > >> Hi, >> >> this problem is likely to be already resolved in the recent distribution >> kernel and it is unlikely to be an upstream issue. >> >> Also there is no crash but just a hang. >> >> Thanks! >> -Lukas >> >> On Sun, 29 Jan 2012, Jesper Juhl wrote: >> >> > On Sun, 29 Jan 2012, Nanakos Chrysostomos wrote: >> > >> > > Dear Linux Kernel Developers, >> > > I am writing to report a kernel or ext4 bug. >> > > I have two servers populated with 2 LUNs from the main storage >> facility, >> > > each server has one LUN which is about 1.2TB. On top of that I am >> > > utilizing glusterfs and samba to mount the 2.4TB created space to a >> > > Windows server and other Linux machines (via glusterfs native >> client). >> > > Yesterday I tried to copy to the new storage glusterfs space about >> 788GB >> > > and in the morning I got from both machines the attached kernel >> messages. >> > > I have to mention that the first machine runs samba and mounts the >> > > allocated glusterfs space with the help of FUSE glusterfs driver. >> The >> > > second machine doesn't mount/export anything and simply runs the >> glusterfs >> > > service. >> > > I have to also add that similar problems have happened to another >> system >> > > which is used as a TIF to JP2 converter. The conversion path starts >> from >> > > the conversion of a pdf file to png and then to jp2. As you can >> imagine >> > > this process produces hundreds of thousands of files. Similarly the >> > > machine crashes and I get similar kernel messages concerning ext4. I >> can't >> > > give you more information on that because the machine crashes, I >> have to >> > > take a photo and send it back to you next time. >> > > What do you believe about the kernel messages? Is this a ext4 bug >> when >> > > thousands of files are created? How can I help resolve that? >> > > >> > > Please Cc me because I am not in the list. >> > > >> > > With kind regards, >> > > Chris. >> > > >> > >> > 1. Adding ext4 people to CC. >> > >> > 2. Took a quick look (without knowing much about ext4, mind you), and >> I >> > noticed something that is, probably unrelated, but just in case it is >> not: >> > >> > In ext4_new_inode(), at the 'got:' label we have this code: >> > >> > [...] >> > got: >> > /* We may have to initialize the block bitmap if it isn't >> already */ >> > if (EXT4_HAS_RO_COMPAT_FEATURE(sb, >> EXT4_FEATURE_RO_COMPAT_GDT_CSUM) && >> > gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) { >> > struct buffer_head *block_bitmap_bh; >> > >> > block_bitmap_bh = ext4_read_block_bitmap(sb, group); >> > BUFFER_TRACE(block_bitmap_bh, "get block bitmap >> access"); >> > err = ext4_journal_get_write_access(handle, >> block_bitmap_bh); >> > if (err) { >> > brelse(block_bitmap_bh); >> > goto fail; >> > } >> > >> > BUFFER_TRACE(block_bitmap_bh, "dirty block bitmap"); >> > err = ext4_handle_dirty_metadata(handle, NULL, >> block_bitmap_bh); >> > [...] >> > >> > In the above, ext4_read_block_bitmap() may return NULL. If it does, >> then >> > the calls to ext4_journal_get_write_access() and >> > ext4_handle_dirty_metadata() are not safe, since both those functions >> may >> > call other functions that may end up dereferencing their argument (the >> > ext4_read_block_bitmap() return value). >> > >> > Whether or not that will ever actually happen and whether or not it's >> > related to your problem, I don't know. >> > But I thought I'd share the info. >> > >> > > >> > > PS: Are there any ext4 kernel optimization's for avoiding that or >> > > narrowing down the problem if this is not so obvious? >> > > >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 2070 1 0x00000000 >> > > ffff88003daa5c68 0000000000000082 0000000000000000 000000010000000b >> > > ffff88003daa5d18 ffff88003daa5c98 ffff88003daa5bf8 ffffffff81090957 >> > > ffff88003a912638 ffff88003daa5fd8 000000000000f4e8 ffff88003a912638 >> > > Call Trace: >> > > [] ? bit_waitqueue+0x17/0xd0 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] ? __ext4_handle_dirty_metadata+0x7b/0x100 >> [ext4] >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_orphan_add+0xb9/0x1f0 [ext4] >> > > [] ? ext4_delete_entry+0xf9/0x110 [ext4] >> > > [] ext4_unlink+0x29b/0x2b0 [ext4] >> > > [] vfs_unlink+0x9f/0xe0 >> > > [] ? lookup_hash+0x3a/0x50 >> > > [] do_unlinkat+0x183/0x1c0 >> > > [] ? path_put+0x31/0x40 >> > > [] ? sys_lgetxattr+0x61/0x80 >> > > [] sys_unlink+0x16/0x20 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:2322 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 2322 1 0x00000000 >> > > ffff88003c70dc48 0000000000000082 0000000000000000 00000001810949db >> > > 0000000000000000 0000000000000000 ffff88003c70dcb0 0000000000000000 >> > > ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8 >> > > Call Trace: >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2585 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000000 0 2585 1998 0x00000000 >> > > ffff88003b299bb8 0000000000000082 000000000000007a 0000000000000286 >> > > ffff88003b23b8c0 000000000000007e ffff88003b299b98 ffff88003b23b8c0 >> > > ffff88003a913ab8 ffff88003b299fd8 000000000000f4e8 ffff88003a913ab8 >> > > Call Trace: >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] ? d_free+0x3f/0x60 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2586 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000000 0 2586 1998 0x00000000 >> > > ffff88003b1e7bb8 0000000000000082 0000000000000088 0000000000000286 >> > > ffff88003b619380 000000000000008c ffff88003b1e7b98 ffff88003b619380 >> > > ffff880037a59af8 ffff88003b1e7fd8 000000000000f4e8 ffff880037a59af8 >> > > Call Trace: >> > > [] ? release_sock+0xce/0xe0 >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2587 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000001 0 2587 1998 0x00000000 >> > > ffff88003d5efbb8 0000000000000086 0000000000000000 0000000000000286 >> > > ffff880037047cc0 000000000000007e ffff88003d5efb98 ffff880037047cc0 >> > > ffff88003b04f038 ffff88003d5effd8 000000000000f4e8 ffff88003b04f038 >> > > Call Trace: >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2588 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000001 0 2588 1998 0x00000000 >> > > ffff88003b093bb8 0000000000000082 0000000000000000 0000000000000286 >> > > ffff88003a9e69c0 000000000000007e ffff88003b093b98 ffff88003a9e69c0 >> > > ffff8800376f9b38 ffff88003b093fd8 000000000000f4e8 ffff8800376f9b38 >> > > Call Trace: >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2589 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000001 0 2589 1998 0x00000000 >> > > ffff8800374c1bb8 0000000000000086 0000000000000088 0000000000000286 >> > > ffff880037bf50c0 000000000000008c ffff8800374c1b98 ffff880037bf50c0 >> > > ffff88003b18a678 ffff8800374c1fd8 000000000000f4e8 ffff88003b18a678 >> > > Call Trace: >> > > [] ? release_sock+0xce/0xe0 >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:2322 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000001 0 2322 1 0x00000000 >> > > ffff88003c70da28 0000000000000082 ffff880000020f58 ffff88003b8615a8 >> > > ffff88003c70d9a8 ffffffff81467515 ffff88003b8615a8 ffff88003b8615a8 >> > > ffff88003b18baf8 ffff88003c70dfd8 000000000000f4e8 ffff88003b18baf8 >> > > Call Trace: >> > > [] ? ip_local_out+0x25/0x30 >> > > [] ? prepare_to_wait+0x4e/0x80 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ? ext4_init_security+0x2d/0x296 [ext4] >> > > [] ext4_ext_tree_init+0x3e/0x60 [ext4] >> > > [] ext4_new_inode+0x1150/0x1190 [ext4] >> > > [] ext4_create+0xc0/0x150 [ext4] >> > > [] ? generic_permission+0x23/0xb0 >> > > [] vfs_create+0xb4/0xe0 >> > > [] do_filp_open+0xb2f/0xd60 >> > > [] ? alloc_fd+0x92/0x160 >> > > [] do_sys_open+0x69/0x140 >> > > [] ? sys_lgetxattr+0x61/0x80 >> > > [] sys_open+0x20/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2614 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000001 0 2614 1998 0x00000000 >> > > ffff88003c63dbb8 0000000000000086 0000000000000000 0000000000000286 >> > > ffff88003b6014c0 000000000000007c ffff88003c63db98 ffff88003b6014c0 >> > > ffff88003a913ab8 ffff88003c63dfd8 000000000000f4e8 ffff88003a913ab8 >> > > Call Trace: >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] ? __d_lookup+0xa7/0x150 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task smbd:2615 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > smbd D 0000000000000001 0 2615 1998 0x00000000 >> > > ffff88003d9bbbb8 0000000000000086 0000000000000000 0000000000000286 >> > > ffff88003720f580 000000000000008a ffff88003d9bbb98 ffff88003720f580 >> > > ffff880037a59af8 ffff88003d9bbfd8 000000000000f4e8 ffff880037a59af8 >> > > Call Trace: >> > > [] __mutex_lock_slowpath+0x13e/0x180 >> > > [] mutex_lock+0x2b/0x50 >> > > [] do_lookup+0xef/0x1e0 >> > > [] __link_path_walk+0x734/0x1030 >> > > [] ? path_to_nameidata+0x45/0x60 >> > > [] path_walk+0x6a/0xe0 >> > > [] do_path_lookup+0x5b/0xa0 >> > > [] user_path_at+0x57/0xa0 >> > > [] ? __do_page_fault+0x1e4/0x480 >> > > [] ? putname+0x35/0x50 >> > > [] ? user_path_at+0x62/0xa0 >> > > [] vfs_fstatat+0x46/0x80 >> > > [] ? mntput_no_expire+0x30/0x110 >> > > [] vfs_stat+0x1b/0x20 >> > > [] sys_newstat+0x24/0x50 >> > > [] ? do_page_fault+0x3e/0xa0 >> > > [] ? page_fault+0x25/0x30 >> > > [] system_call_fastpath+0x16/0x1b >> > > [root@gluster00 ~]# >> > > >> > > >> > > [root@gluster00 ~]# mount >> > > /dev/sda3 on / type ext4 (rw) >> > > proc on /proc type proc (rw) >> > > sysfs on /sys type sysfs (rw) >> > > devpts on /dev/pts type devpts (rw,gid=5,mode=620) >> > > tmpfs on /dev/shm type tmpfs (rw) >> > > /dev/sda1 on /boot type ext4 (rw) >> > > /dev/sdb1 on /GLUSTER00 type ext4 (rw) >> > > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) >> > > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) >> > > gluster00:/mm-volume on /mmvolume type fuse.glusterfs >> (rw,allow_other,default_permissions,max_read=131072) >> > > >> > > [root@gluster00 ~]# df -h >> > > Filesystem Size Used Avail Use% Mounted on >> > > /dev/sda3 18G 2.0G 15G 13% / >> > > tmpfs 499M 0 499M 0% /dev/shm >> > > /dev/sda1 248M 103M 133M 44% /boot >> > > /dev/sdb1 960G 200M 911G 1% /GLUSTER00 >> > > gluster00:/mm-volume 2.4T 397M 2.2T 1% /mmvolume >> > > >> > > >> > > [root@gluster00 ~]# gluster -V >> > > glusterfs 3.2.5 built on Dec 7 2011 18:41:39 >> > > Repository revision: git://git.gluster.com/glusterfs.git >> > > Copyright (c) 2006-2011 Gluster Inc. >> > > GlusterFS comes with ABSOLUTELY NO WARRANTY. >> > > You may redistribute copies of GlusterFS under the terms of the GNU >> General Public License. >> > > >> > > [root@gluster00 ~]# uname -a >> > > Linux gluster00.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 >> 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux >> > > >> > > [root@gluster00 ~]# smbd -V >> > > Version 3.5.10-114.el6 >> > > >> > > >> > > THE MACHINE RUNS CENTOS 6.2 >> > > >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b >> > > ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] ? bit_waitqueue+0x17/0xd0 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] ? __ext4_handle_dirty_metadata+0x7b/0x100 >> [ext4] >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_orphan_add+0xb9/0x1f0 [ext4] >> > > [] ? ext4_delete_entry+0xf9/0x110 [ext4] >> > > [] ext4_unlink+0x29b/0x2b0 [ext4] >> > > [] vfs_unlink+0x9f/0xe0 >> > > [] ? lookup_hash+0x3a/0x50 >> > > [] do_unlinkat+0x183/0x1c0 >> > > [] ? path_put+0x31/0x40 >> > > [] ? sys_lgetxattr+0x61/0x80 >> > > [] sys_unlink+0x16/0x20 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b >> > > ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] ? bit_waitqueue+0x17/0xd0 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] ? __ext4_handle_dirty_metadata+0x7b/0x100 >> [ext4] >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_orphan_add+0xb9/0x1f0 [ext4] >> > > [] ? ext4_delete_entry+0xf9/0x110 [ext4] >> > > [] ext4_unlink+0x29b/0x2b0 [ext4] >> > > [] vfs_unlink+0x9f/0xe0 >> > > [] ? lookup_hash+0x3a/0x50 >> > > [] do_unlinkat+0x183/0x1c0 >> > > [] ? path_put+0x31/0x40 >> > > [] ? sys_lgetxattr+0x61/0x80 >> > > [] sys_unlink+0x16/0x20 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b >> > > ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] ? bit_waitqueue+0x17/0xd0 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] ? __ext4_handle_dirty_metadata+0x7b/0x100 >> [ext4] >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_orphan_add+0xb9/0x1f0 [ext4] >> > > [] ? ext4_delete_entry+0xf9/0x110 [ext4] >> > > [] ext4_unlink+0x29b/0x2b0 [ext4] >> > > [] vfs_unlink+0x9f/0xe0 >> > > [] ? lookup_hash+0x3a/0x50 >> > > [] do_unlinkat+0x183/0x1c0 >> > > [] ? path_put+0x31/0x40 >> > > [] ? sys_lgetxattr+0x61/0x80 >> > > [] sys_unlink+0x16/0x20 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c68 0000000000000086 0000000000000000 000000010000000b >> > > ffff88003b683d18 ffff88003b683c98 ffff88003b683bf8 ffffffff81090957 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] ? bit_waitqueue+0x17/0xd0 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] ? __ext4_handle_dirty_metadata+0x7b/0x100 >> [ext4] >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_orphan_add+0xb9/0x1f0 [ext4] >> > > [] ? ext4_delete_entry+0xf9/0x110 [ext4] >> > > [] ext4_unlink+0x29b/0x2b0 [ext4] >> > > [] vfs_unlink+0x9f/0xe0 >> > > [] ? lookup_hash+0x3a/0x50 >> > > [] do_unlinkat+0x183/0x1c0 >> > > [] ? path_put+0x31/0x40 >> > > [] ? sys_lgetxattr+0x61/0x80 >> > > [] sys_unlink+0x16/0x20 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1754 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1754 1 0x00000000 >> > > ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580 >> > > 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0 >> > > ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38 >> > > Call Trace: >> > > [] ? prepare_to_wait+0x4e/0x80 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1754 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1754 1 0x00000000 >> > > ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580 >> > > 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0 >> > > ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38 >> > > Call Trace: >> > > [] ? prepare_to_wait+0x4e/0x80 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1754 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1754 1 0x00000000 >> > > ffff88003b479c48 0000000000000086 ffff880000000001 000000013712d580 >> > > 0000000000000000 ffff88003f810080 ffff88003faacc00 ffff88003f8224c0 >> > > ffff88003712db38 ffff88003b479fd8 000000000000f4e8 ffff88003712db38 >> > > Call Trace: >> > > [] ? prepare_to_wait+0x4e/0x80 >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db >> > > 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db >> > > 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > INFO: task glusterfsd:1878 blocked for more than 120 seconds. >> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >> message. >> > > glusterfsd D 0000000000000000 0 1878 1 0x00000000 >> > > ffff88003b683c48 0000000000000086 0000000000000000 00000001810949db >> > > 0000000000000000 0000000000000000 ffff88003b683cb0 0000000000000000 >> > > ffff880037a9e6b8 ffff88003b683fd8 000000000000f4e8 ffff880037a9e6b8 >> > > Call Trace: >> > > [] do_get_write_access+0x29d/0x520 [jbd2] >> > > [] ? wake_bit_function+0x0/0x50 >> > > [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] >> > > [] __ext4_journal_get_write_access+0x38/0x80 >> [ext4] >> > > [] ext4_reserve_inode_write+0x73/0xa0 [ext4] >> > > [] ? call_filldir+0x78/0xe0 [ext4] >> > > [] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4] >> > > [] ext4_dirty_inode+0x40/0x60 [ext4] >> > > [] __mark_inode_dirty+0x3b/0x160 >> > > [] touch_atime+0x12d/0x170 >> > > [] ? filldir+0x0/0xe0 >> > > [] vfs_readdir+0xd6/0xe0 >> > > [] sys_getdents+0x89/0xf0 >> > > [] system_call_fastpath+0x16/0x1b >> > > [root@gluster01 ~]# >> > > >> > > >> > > [root@gluster01 ~]# mount >> > > /dev/sda3 on / type ext4 (rw) >> > > proc on /proc type proc (rw) >> > > sysfs on /sys type sysfs (rw) >> > > devpts on /dev/pts type devpts (rw,gid=5,mode=620) >> > > tmpfs on /dev/shm type tmpfs (rw) >> > > /dev/sda1 on /boot type ext4 (rw) >> > > /dev/sdb1 on /GLUSTER01 type ext4 (rw) >> > > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) >> > > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) >> > > >> > > [root@gluster01 ~]# df -h >> > > Filesystem Size Used Avail Use% Mounted on >> > > /dev/sda3 18G 1.8G 15G 11% / >> > > tmpfs 499M 0 499M 0% /dev/shm >> > > /dev/sda1 248M 103M 133M 44% /boot >> > > /dev/sdb1 1.4T 198M 1.4T 1% /GLUSTER01 >> > > >> > > [root@gluster01 ~]# gluster -V >> > > glusterfs 3.2.5 built on Dec 7 2011 18:41:39 >> > > Repository revision: git://git.gluster.com/glusterfs.git >> > > Copyright (c) 2006-2011 Gluster Inc. >> > > GlusterFS comes with ABSOLUTELY NO WARRANTY. >> > > You may redistribute copies of GlusterFS under the terms of the GNU >> General Public License. >> > > >> > > [root@gluster01 ~]# uname -a >> > > Linux gluster01.prod 2.6.32-220.4.1.el6.x86_64 #1 SMP Tue Jan 24 >> 02:13:44 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux >> > > >> > > >> > > THE MACHINE RUNS CENTOS 6.2 >> > > -- >> > > To unsubscribe from this list: send the line "unsubscribe >> linux-kernel" in >> > > the body of a message to majordomo@vger.kernel.org >> > > More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > Please read the FAQ at http://www.tux.org/lkml/ >> > > >> > >> > >> >> > > -- >