Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756980AbYFVWTq (ORCPT ); Sun, 22 Jun 2008 18:19:46 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754187AbYFVWTf (ORCPT ); Sun, 22 Jun 2008 18:19:35 -0400 Received: from ipmail01.adl6.internode.on.net ([203.16.214.146]:7547 "EHLO ipmail01.adl6.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753801AbYFVWTf (ORCPT ); Sun, 22 Jun 2008 18:19:35 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApoEANNqXkh5LG+u/2dsb2JhbACtQg X-IronPort-AV: E=Sophos;i="4.27,686,1204464600"; d="scan'208";a="132492337" Date: Mon, 23 Jun 2008 08:19:30 +1000 From: Dave Chinner To: Daniel J Blueman Cc: Christoph Lameter , Mel Gorman , Linus Torvalds , Alexander Beregalov , Linux Kernel , xfs@oss.sgi.com Subject: Re: [2.6.26-rc7] shrink_icache from pagefault locking (nee: nfsd hangs for a few sec)... Message-ID: <20080622221930.GA11558@disturbed> Mail-Followup-To: Daniel J Blueman , Christoph Lameter , Mel Gorman , Linus Torvalds , Alexander Beregalov , Linux Kernel , xfs@oss.sgi.com References: <6278d2220806220256g674304ectb945c14e7e09fede@mail.gmail.com> <6278d2220806220258p28de00c1x615ad7b2f708e3f8@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <6278d2220806220258p28de00c1x615ad7b2f708e3f8@mail.gmail.com> User-Agent: Mutt/1.5.17+20080114 (2008-01-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 5072 Lines: 123 [added xfs@oss.sgi.com to cc] On Sun, Jun 22, 2008 at 10:58:56AM +0100, Daniel J Blueman wrote: > I'm seeing a similar issue [2] to what was recently reported [1] by > Alexander, but with another workload involving XFS and memory > pressure. > > SLUB allocator is in use and config is at http://quora.org/config-client-debug . > > Let me know if you'd like more details/vmlinux objdump etc. > > Thanks, > Daniel > > --- [1] > > http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/e673c9173d45a735/db9213ef39e4e11c > > --- [2] > > ======================================================= > [ INFO: possible circular locking dependency detected ] > 2.6.26-rc7-210c #2 > ------------------------------------------------------- > AutopanoPro/4470 is trying to acquire lock: > (iprune_mutex){--..}, at: [] shrink_icache_memory+0x7d/0x290 > > but task is already holding lock: > (&mm->mmap_sem){----}, at: [] do_page_fault+0x255/0x890 > > which lock already depends on the new lock. > > > the existing dependency chain (in reverse order) is: > > -> #2 (&mm->mmap_sem){----}: > [] __lock_acquire+0xbdd/0x1020 > [] lock_acquire+0x65/0x90 > [] down_read+0x3b/0x70 > [] do_page_fault+0x27c/0x890 > [] error_exit+0x0/0xa9 > [] 0xffffffffffffffff > > -> #1 (&(&ip->i_iolock)->mr_lock){----}: > [] __lock_acquire+0xbdd/0x1020 > [] lock_acquire+0x65/0x90 > [] down_write_nested+0x46/0x80 > [] xfs_ilock+0x99/0xa0 > [] xfs_ireclaim+0x3f/0x90 > [] xfs_finish_reclaim+0x59/0x1a0 > [] xfs_reclaim+0x109/0x110 > [] xfs_fs_clear_inode+0xe1/0x110 > [] clear_inode+0x7d/0x110 > [] dispose_list+0x2a/0x100 > [] shrink_icache_memory+0x22f/0x290 > [] shrink_slab+0x168/0x1d0 > [] kswapd+0x3b6/0x560 > [] kthread+0x4d/0x80 > [] child_rip+0xa/0x12 > [] 0xffffffffffffffff You may as well ignore anything invlving this path in XFS until lockdep gets fixed. The kswapd reclaim path is inverted over the synchronous reclaim path that is xfs_ilock -> run out of memory -> prune_icache and then potentially another -> xfs_ilock. In this case, XFS can *never* deadlock because the second xfs_ilock is on a different, unreferenced, unlocked inode, but without turning off lockdep there is nothing in XFS that can be done to prevent this warning. Therxp eis a similar bug in the VM w.r.t the mmap_sem in that the mmap_sem is held across a call to put_filp() which can result in inversions between the xfs_ilock and mmap_sem. Both of these cases cannot be solved by changing XFS - lockdep needs to be made aware of paths that can invert normal locking order (like prune_icache) so it doesn't give false positives like this. > -> #0 (iprune_mutex){--..}: > [] __lock_acquire+0xa47/0x1020 > [] lock_acquire+0x65/0x90 > [] mutex_lock_nested+0xb5/0x300 > [] shrink_icache_memory+0x7d/0x290 > [] shrink_slab+0x168/0x1d0 > [] try_to_free_pages+0x268/0x3a0 > [] __alloc_pages_internal+0x206/0x4b0 > [] __alloc_pages_nodemask+0x9/0x10 > [] alloc_page_vma+0x72/0x1b0 > [] handle_mm_fault+0x462/0x7b0 > [] do_page_fault+0x30c/0x890 > [] error_exit+0x0/0xa9 > [] 0xffffffffffffffff This case is different in that it įs complaining about mmap_sem vs iprune_mutex, so I think that we can pretty much ignore the XFS side of things here - the problem is higher level code.... > [] try_to_free_pages+0x268/0x3a0 > [] ? isolate_pages_global+0x0/0x40 > [] __alloc_pages_internal+0x206/0x4b0 > [] __alloc_pages_nodemask+0x9/0x10 > [] alloc_page_vma+0x72/0x1b0 > [] handle_mm_fault+0x462/0x7b0 FWIW, should page allocation in a page fault be allowed to recurse into the filesystem? If I follow the spaghetti of inline and compiler inlined functions correctly, this is a GFP_HIGHUSER_MOVABLE allocation, right? Should we be allowing shrink_icache_memory() to be called at all in the page fault path? Cheers, Dave. -- Dave Chinner david@fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/