=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (iprune_mutex){--..}:
[<c012fe17>] lock_acquire+0x4b/0x6c
[<c0326cc4>] __mutex_lock_slowpath+0xb3/0x200
[<c0326e32>] mutex_lock+0x21/0x24
[<c017638a>] shrink_icache_memory+0x36/0x1b2
[<c014cf50>] shrink_slab+0xd0/0x123
[<c014d70e>] try_to_free_pages+0x114/0x1ba
[<c0149b78>] __alloc_pages+0x18b/0x279
[<c0146b10>] generic_file_buffered_write+0x167/0x546
[<c014722f>] __generic_file_aio_write_nolock+0x340/0x38a
[<c01472d5>] generic_file_aio_write+0x5c/0xaf
[<f9c9e1cf>] nfs_file_write+0x7b/0x97 [nfs]
[<c015f976>] do_sync_write+0xaf/0xe4
[<c0160229>] vfs_write+0xab/0x157
[<c0160760>] sys_write+0x3b/0x60
[<c0102df3>] syscall_call+0x7/0xb
-> #0 (&inode->i_mutex){--..}:
[<c012fe17>] lock_acquire+0x4b/0x6c
[<c0326cc4>] __mutex_lock_slowpath+0xb3/0x200
[<c0326e32>] mutex_lock+0x21/0x24
[<f921706d>] ntfs_put_inode+0x3d/0x75 [ntfs]
[<c0175acd>] iput+0x33/0x6a
[<f9216eaa>] ntfs_clear_big_inode+0x99/0xb2 [ntfs]
[<c0175fdc>] clear_inode+0xd8/0x129
[<c01762cf>] dispose_list+0x4c/0xd1
[<c01764de>] shrink_icache_memory+0x18a/0x1b2
[<c014cf50>] shrink_slab+0xd0/0x123
[<c014d2a3>] kswapd+0x260/0x336
[<c012a4b7>] kthread+0xb0/0xdd
[<c0101005>] kernel_thread_helper+0x5/0xb
other info that might help us debug this:
2 locks held by kswapd0/186:
#0: (shrinker_rwsem){----}, at: [<c014cea5>] shrink_slab+0x25/0x123
#1: (iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
stack backtrace:
[<c0103fe1>] show_trace+0xd/0x10
[<c010447d>] dump_stack+0x19/0x1b
[<c012ef1e>] print_circular_bug_tail+0x59/0x64
[<c012f70b>] __lock_acquire+0x7e2/0x986
[<c012fe17>] lock_acquire+0x4b/0x6c
[<c0326cc4>] __mutex_lock_slowpath+0xb3/0x200
[<c0326e32>] mutex_lock+0x21/0x24
[<f921706d>] ntfs_put_inode+0x3d/0x75 [ntfs]
[<c0175acd>] iput+0x33/0x6a
[<f9216eaa>] ntfs_clear_big_inode+0x99/0xb2 [ntfs]
[<c0175fdc>] clear_inode+0xd8/0x129
[<c01762cf>] dispose_list+0x4c/0xd1
[<c01764de>] shrink_icache_memory+0x18a/0x1b2
[<c014cf50>] shrink_slab+0xd0/0x123
[<c014d2a3>] kswapd+0x260/0x336
[<c012a4b7>] kthread+0xb0/0xdd
[<c0101005>] kernel_thread_helper+0x5/0xb
Index: linux-2.6/include/linux/pagemap.h
===================================================================
--- linux-2.6.orig/include/linux/pagemap.h 2006-10-17 00:29:40.000000000 +1000
+++ linux-2.6/include/linux/pagemap.h 2006-10-17 00:29:50.000000000 +1000
@@ -57,7 +57,7 @@ extern struct page *page_cache_alloc_col
#else
static inline struct page *page_cache_alloc(struct address_space *x)
{
- return alloc_pages(mapping_gfp_mask(x), 0);
+ return alloc_pages(mapping_gfp_mask(x)&~__GFP_FS, 0);
}
static inline struct page *page_cache_alloc_cold(struct address_space *x)
Index: linux-2.6/mm/filemap.c
===================================================================
--- linux-2.6.orig/mm/filemap.c 2006-10-17 00:29:49.000000000 +1000
+++ linux-2.6/mm/filemap.c 2006-10-17 00:29:50.000000000 +1000
@@ -471,9 +471,9 @@ struct page *page_cache_alloc(struct add
{
if (cpuset_do_page_mem_spread()) {
int n = cpuset_mem_spread_node();
- return alloc_pages_node(n, mapping_gfp_mask(x), 0);
+ return alloc_pages_node(n, mapping_gfp_mask(x)&~__GFP_FS, 0);
}
- return alloc_pages(mapping_gfp_mask(x), 0);
+ return alloc_pages(mapping_gfp_mask(x)&~__GFP_FS, 0);
}
EXPORT_SYMBOL(page_cache_alloc);
@@ -1864,7 +1864,7 @@ repeat:
return NULL;
}
err = add_to_page_cache(*cached_page, mapping,
- index, GFP_KERNEL);
+ index, GFP_KERNEL&~__GFP_FS);
if (err == -EEXIST)
goto repeat;
if (err == 0) {
On Tue, 17 Oct 2006 00:32:44 +1000 Nick Piggin wrote:
> alpha @ steudten Engineering wrote:
> > =======================================================
> > [ INFO: possible circular locking dependency detected ]
> > 2.6.18-1.2189self #1
> > -------------------------------------------------------
> > kswapd0/186 is trying to acquire lock:
> > (&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
> >
> > but task is already holding lock:
> > (iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
> >
> > which lock already depends on the new lock.
>
> Thanks. __grab_cache_page wants to clear __GFP_FS, because it is
> holding the i_mutex so we don't want to reenter the filesystem in
> page reclaim.
>
> This would be an easy two liner, except those funny page_cache_alloc
> routines which take a mapping rather than a gfp_t argument :P
and it would be only one email, but you forgot spaces there,
so it's too ugly to use. ;) i.e., please add spaces around
the '&'.
and it's an attachment :(
> Anyway, I'll get around to writing the real patch and queue it up
> with my other buffered write deadlock fixes. It should be fairly
> unlikely to cause you a deadlock. You could give this quick patch a
> try, though. Does it fix your problem?
---
~Randy
Randy Dunlap wrote:
> On Tue, 17 Oct 2006 00:32:44 +1000 Nick Piggin wrote:
>
>
>>alpha @ steudten Engineering wrote:
>>
>>>=======================================================
>>>[ INFO: possible circular locking dependency detected ]
>>>2.6.18-1.2189self #1
>>>-------------------------------------------------------
>>>kswapd0/186 is trying to acquire lock:
>>> (&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
>>>
>>>but task is already holding lock:
>>> (iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
>>>
>>>which lock already depends on the new lock.
>>
>>Thanks. __grab_cache_page wants to clear __GFP_FS, because it is
>>holding the i_mutex so we don't want to reenter the filesystem in
>>page reclaim.
>>
>>This would be an easy two liner, except those funny page_cache_alloc
>>routines which take a mapping rather than a gfp_t argument :P
>
>
> and it would be only one email, but you forgot spaces there,
> so it's too ugly to use. ;) i.e., please add spaces around
> the '&'.
>
> and it's an attachment :(
Oh yeah, it was an attachment because I didn't want anyone to see how
ugly it is ;)
It's just a quick hack to see if it works. I'll send out a real patch
when I get time to code and test it properly.
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com
On Tue, 17 Oct 2006 00:32:44 +1000
Nick Piggin <[email protected]> wrote:
> alpha @ steudten Engineering wrote:
> > =======================================================
> > [ INFO: possible circular locking dependency detected ]
> > 2.6.18-1.2189self #1
> > -------------------------------------------------------
> > kswapd0/186 is trying to acquire lock:
> > (&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
> >
> > but task is already holding lock:
> > (iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
> >
> > which lock already depends on the new lock.
>
> Thanks. __grab_cache_page wants to clear __GFP_FS, because it is
> holding the i_mutex so we don't want to reenter the filesystem in
> page reclaim.
We want to be able to enter page reclaim while holding i_mutex. Think what
the effect of not doing this would be upon write() (!)
This warning is more fallout from ntfs's insistence on taking i_mutex in
its clear_inode(). See lengthy and unproductive discussion at
http://lkml.org/lkml/2006/7/26/185 .
Andrew Morton wrote:
> On Tue, 17 Oct 2006 00:32:44 +1000
> Nick Piggin <[email protected]> wrote:
>
>
>>alpha @ steudten Engineering wrote:
>>
>>>=======================================================
>>>[ INFO: possible circular locking dependency detected ]
>>>2.6.18-1.2189self #1
>>>-------------------------------------------------------
>>>kswapd0/186 is trying to acquire lock:
>>> (&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
>>>
>>>but task is already holding lock:
>>> (iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24
>>>
>>>which lock already depends on the new lock.
>>
>>Thanks. __grab_cache_page wants to clear __GFP_FS, because it is
>>holding the i_mutex so we don't want to reenter the filesystem in
>>page reclaim.
>
>
> We want to be able to enter page reclaim while holding i_mutex. Think what
> the effect of not doing this would be upon write() (!)
>
> This warning is more fallout from ntfs's insistence on taking i_mutex in
> its clear_inode(). See lengthy and unproductive discussion at
> http://lkml.org/lkml/2006/7/26/185 .
Yeah you're right. It will be a hot allocation + reclaim path for high
bandwidth writes.
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com