2007-02-23 06:40:36

by Ingo Molnar

[permalink] [raw]
Subject: circular locking: /proc/sys/vm/drop_caches


Andrew,

i just got the lockdep warning below when doing:

echo 3 > /proc/sys/vm/drop_caches

is this a known quirk in the drop_caches code?

Ingo

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.20-syslet #434
-------------------------------------------------------
bash/12166 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<c01c82f5>] journal_try_to_free_buffers+0xd5/0x18f

but task is already holding lock:
(inode_lock){--..}, at: [<c01947a3>] drop_pagecache+0x45/0xc8

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (inode_lock){--..}:
[<c0144940>] __lock_acquire+0x984/0xa94
[<c0144d6f>] lock_acquire+0x67/0x8c
[<c0424a9f>] _spin_lock+0x35/0x42
[<c0193ccb>] __mark_inode_dirty+0xe3/0x15e
[<c016253b>] __set_page_dirty_nobuffers+0xb4/0xbf
[<c0197361>] mark_buffer_dirty+0x1e/0x20
[<c01c7af2>] __journal_temp_unlink_buffer+0x176/0x17d
[<c01c7d23>] __journal_unfile_buffer+0xb/0x15
[<c01c7d97>] __journal_refile_buffer+0x6a/0xe3
[<c01cac34>] journal_commit_transaction+0xf46/0x11da
[<c01cd32c>] kjournald+0xab/0x1e8
[<c013b935>] kthread+0xb2/0xd7
[<c0104e5f>] kernel_thread_helper+0x7/0x10
[<ffffffff>] 0xffffffff

-> #0 (&journal->j_list_lock){--..}:
[<c014483b>] __lock_acquire+0x87f/0xa94
[<c0144d6f>] lock_acquire+0x67/0x8c
[<c0424a9f>] _spin_lock+0x35/0x42
[<c01c82f5>] journal_try_to_free_buffers+0xd5/0x18f
[<c01b944e>] ext3_releasepage+0x68/0x74
[<c015d027>] try_to_release_page+0x33/0x44
[<c016406e>] invalidate_mapping_pages+0x69/0xca
[<c01640dc>] invalidate_inode_pages+0xd/0x11
[<c01947bf>] drop_pagecache+0x61/0xc8
[<c019485c>] drop_caches_sysctl_handler+0x36/0x4e
[<c012efdc>] do_rw_proc+0xaf/0xf1
[<c012f078>] proc_writesys+0x20/0x25
[<c017b3de>] vfs_write+0xb1/0x165
[<c017bbc6>] sys_write+0x3d/0x61
[<c01041a8>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff

other info that might help us debug this:

2 locks held by bash/12166:
#0: (&type->s_umount_key#17){----}, at: [<c0194793>] drop_pagecache+0x35/0xc8
#1: (inode_lock){--..}, at: [<c01947a3>] drop_pagecache+0x45/0xc8

stack backtrace:
[<c010510d>] show_trace_log_lvl+0x19/0x2e
[<c0105206>] show_trace+0x12/0x14
[<c010521c>] dump_stack+0x14/0x16
[<c01433ff>] print_circular_bug_tail+0x5f/0x68
[<c014483b>] __lock_acquire+0x87f/0xa94
[<c0144d6f>] lock_acquire+0x67/0x8c
[<c0424a9f>] _spin_lock+0x35/0x42
[<c01c82f5>] journal_try_to_free_buffers+0xd5/0x18f
[<c01b944e>] ext3_releasepage+0x68/0x74
[<c015d027>] try_to_release_page+0x33/0x44
[<c016406e>] invalidate_mapping_pages+0x69/0xca
[<c01640dc>] invalidate_inode_pages+0xd/0x11
[<c01947bf>] drop_pagecache+0x61/0xc8
[<c019485c>] drop_caches_sysctl_handler+0x36/0x4e
[<c012efdc>] do_rw_proc+0xaf/0xf1
[<c012f078>] proc_writesys+0x20/0x25
[<c017b3de>] vfs_write+0xb1/0x165
[<c017bbc6>] sys_write+0x3d/0x61
[<c01041a8>] syscall_call+0x7/0xb
=======================


2007-02-23 09:41:18

by Frederik Deweerdt

[permalink] [raw]
Subject: Re: circular locking: /proc/sys/vm/drop_caches

On Fri, Feb 23, 2007 at 07:35:38AM +0100, Ingo Molnar wrote:
>
> Andrew,
>
> i just got the lockdep warning below when doing:
>
> echo 3 > /proc/sys/vm/drop_caches
>
> is this a known quirk in the drop_caches code?
>
This is known, yes: http://lkml.org/lkml/2006/12/4/19

Regards,
Frederik

2007-02-23 10:08:03

by Andrew Morton

[permalink] [raw]
Subject: Re: circular locking: /proc/sys/vm/drop_caches

> On Fri, 23 Feb 2007 07:35:38 +0100 Ingo Molnar <[email protected]> wrote:
> i just got the lockdep warning below when doing:
>
> echo 3 > /proc/sys/vm/drop_caches
>
> is this a known quirk in the drop_caches code?

It is - we're taking inode_lock at just the wrong level there and I don't
know how to fix it. The basic problem is that we're doing a list_head
walk: if we drop the lock we have to start again.

Well, I do know how to fix it: rip the various inode lists and use a tagged
radix tree instead - make writeback's per-superblock inode walk work the
same as writeback's per-inode page walk. That will also alow us to drop
the lock and then resume the drop_caches thing at the point where we left
off.

But it's a large change and has some quite tricky issues which need to be
carried over from the existing code and I don't have time to do it.

The only (lame) thing I can say for now is that drop_caches is a
debug-only, root-only thing :(

2007-02-23 11:41:14

by Ingo Molnar

[permalink] [raw]
Subject: Re: circular locking: /proc/sys/vm/drop_caches


* Andrew Morton <[email protected]> wrote:

> The only (lame) thing I can say for now is that drop_caches is a
> debug-only, root-only thing :(

ok, that's fair enough.

Ingo