Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754881AbXKNM71 (ORCPT ); Wed, 14 Nov 2007 07:59:27 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751556AbXKNM7T (ORCPT ); Wed, 14 Nov 2007 07:59:19 -0500 Received: from mail.fieldses.org ([66.93.2.214]:60111 "EHLO fieldses.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751476AbXKNM7S (ORCPT ); Wed, 14 Nov 2007 07:59:18 -0500 Date: Wed, 14 Nov 2007 07:59:07 -0500 To: Benny Halevy Cc: Chris Wedgwood , linux-xfs@oss.sgi.com, LKML , Christian Kujau Subject: Re: 2.6.24-rc2 XFS nfsd hang Message-ID: <20071114125907.GB4010@fieldses.org> References: <20071114070400.GA25708@puku.stupidest.org> <473AA72C.6020308@panasas.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <473AA72C.6020308@panasas.com> User-Agent: Mutt/1.5.17 (2007-11-01) From: "J. Bruce Fields" Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 7236 Lines: 128 On Wed, Nov 14, 2007 at 09:43:40AM +0200, Benny Halevy wrote: > I wonder if this is a similar hang to what Christian was seeing here: > http://lkml.org/lkml/2007/11/13/319 Ah, thanks for noticing that. Christian Kujau, is /data an xfs partition? There are a bunch of xfs commits in ^92d15c2ccbb3e31a3fc71ad28fdb55e1319383c0 ^291702f017efdfe556cb87b8530eb7d1ff08cbae ^1d677a6dfaac1d1cf51a7f58847077240985faf2 ^fba956c46a72f9e7503fd464ffee43c632307e31 ^bbf25010f1a6b761914430f5fca081ec8c7accd1 6e800af233e0bdf108efb7bd23c11ea6fa34cdeb 7b1915a989ea4d426d0fd98974ab80f30ef1d779 c223701cf6c706f42840631c1ca919a18e6e2800 f77bf01425b11947eeb3b5b54685212c302741b8 which was the range remaining for him to bisect. --b. > > Benny > > On Nov. 14, 2007, 9:04 +0200, Chris Wedgwood wrote: > > With 2.6.24-rc2 (amd64) I sometimes (usually but perhaps not always) > > see a hang when accessing some NFS exported XFS filesystems. Local > > access to these filesystems ahead of time works without problems. > > > > This does not occur with 2.6.23.1. The filesystem does not appear to > > be corrupt. > > > > > > The call chain for the wedged process is: > > > > [ 1462.911256] nfsd D ffffffff80547840 4760 2966 2 > > [ 1462.911283] ffff81010414d4d0 0000000000000046 0000000000000000 ffff81010414d610 > > [ 1462.911322] ffff810104cbc6e0 ffff81010414d480 ffffffff80746dc0 ffffffff80746dc0 > > [ 1462.911360] ffffffff80744020 ffffffff80746dc0 ffff81010129c140 ffff8101000ad100 > > [ 1462.911391] Call Trace: > > [ 1462.911417] [] __down+0xe9/0x101 > > [ 1462.911437] [] default_wake_function+0x0/0xe > > [ 1462.911458] [] __down_failed+0x35/0x3a > > [ 1462.911480] [] _xfs_buf_find+0x84/0x24d > > [ 1462.911501] [] _xfs_buf_find+0x193/0x24d > > [ 1462.911522] [] xfs_buf_lock+0x43/0x45 > > [ 1462.911543] [] _xfs_buf_find+0x1ba/0x24d > > [ 1462.911564] [] xfs_buf_get_flags+0x5a/0x14b > > [ 1462.911586] [] xfs_buf_read_flags+0x12/0x86 > > [ 1462.911607] [] xfs_trans_read_buf+0x4c/0x2cf > > [ 1462.911629] [] xfs_da_do_buf+0x41b/0x65b > > [ 1462.911652] [] xfs_da_read_buf+0x24/0x29 > > [ 1462.911673] [] xfs_dir2_block_lookup_int+0x4d/0x1ab > > [ 1462.911694] [] xfs_dir2_block_lookup_int+0x4d/0x1ab > > [ 1462.911717] [] xfs_dir2_block_lookup+0x15/0x8e > > [ 1462.911738] [] xfs_dir_lookup+0xd2/0x12c > > [ 1462.911761] [] submit_bio+0x10d/0x114 > > [ 1462.911781] [] xfs_dir_lookup_int+0x2c/0xc5 > > [ 1462.911802] [] lockdep_init_map+0x90/0x495 > > [ 1462.911823] [] xfs_lookup+0x44/0x6f > > [ 1462.911843] [] xfs_vn_lookup+0x29/0x60 > > [ 1462.915246] [] __lookup_hash+0xe5/0x109 > > [ 1462.915267] [] lookup_one_len+0x41/0x4e > > [ 1462.915289] [] compose_entry_fh+0xc1/0x117 > > [ 1462.915311] [] encode_entry+0x17c/0x38b > > [ 1462.915333] [] find_or_create_page+0x3f/0xc9 > > [ 1462.915355] [] _xfs_buf_lookup_pages+0x2c1/0x2f6 > > [ 1462.915377] [] _spin_unlock+0x1f/0x49 > > [ 1462.915399] [] cache_alloc_refill+0x1ba/0x4b9 > > [ 1462.915424] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.915448] [] nfs3svc_encode_entry_plus+0x10/0x13 > > [ 1462.915469] [] xfs_dir2_block_getdents+0x15b/0x1e2 > > [ 1462.915491] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.915514] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.915534] [] xfs_readdir+0x91/0xb6 > > [ 1462.915557] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.915579] [] xfs_file_readdir+0x31/0x40 > > [ 1462.915599] [] vfs_readdir+0x61/0x93 > > [ 1462.915619] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.915642] [] nfsd_readdir+0x6d/0xc5 > > [ 1462.915663] [] nfsd3_proc_readdirplus+0x114/0x204 > > [ 1462.915686] [] nfsd_dispatch+0xde/0x1b6 > > [ 1462.915706] [] svc_process+0x3f8/0x717 > > [ 1462.915729] [] nfsd+0x1a9/0x2c1 > > [ 1462.915749] [] child_rip+0xa/0x12 > > [ 1462.915769] [] __svc_create_thread+0xea/0x1eb > > [ 1462.915792] [] nfsd+0x0/0x2c1 > > [ 1462.915812] [] child_rip+0x0/0x12 > > > > Over time other processes pile up beind this. > > > > [ 1462.910728] nfsd D ffffffffffffffff 5440 2965 2 > > [ 1462.910769] ffff8101040cdd40 0000000000000046 0000000000000001 ffff810103471900 > > [ 1462.910812] ffff8101029a72c0 ffff8101040cdcf0 ffffffff80746dc0 ffffffff80746dc0 > > [ 1462.910852] ffffffff80744020 ffffffff80746dc0 ffff81010008e0c0 ffff8101012a1040 > > [ 1462.910882] Call Trace: > > [ 1462.910909] [] nfsd_permission+0x95/0xeb > > [ 1462.910931] [] vfs_readdir+0x46/0x93 > > [ 1462.910950] [] mutex_lock_nested+0x165/0x27c > > [ 1462.910971] [] _spin_unlock+0x1f/0x49 > > [ 1462.910994] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.911015] [] vfs_readdir+0x46/0x93 > > [ 1462.911037] [] nfs3svc_encode_entry_plus+0x0/0x13 > > [ 1462.911057] [] nfsd_readdir+0x6d/0xc5 > > [ 1462.911079] [] nfsd3_proc_readdirplus+0x114/0x204 > > [ 1462.911102] [] nfsd_dispatch+0xde/0x1b6 > > [ 1462.911122] [] svc_process+0x3f8/0x717 > > [ 1462.911143] [] nfsd+0x1a9/0x2c1 > > [ 1462.911165] [] child_rip+0xa/0x12 > > [ 1462.911184] [] __svc_create_thread+0xea/0x1eb > > [ 1462.911206] [] nfsd+0x0/0x2c1 > > [ 1462.911225] [] child_rip+0x0/0x12 > > > > > > Any suggestions other than to bisect this? (Bisection might be > > painful as it crosses the x86-merge.) > > - > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > > - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/