Hi Al,
In the following configuration, I met a deadlock condition like below.
Kernel: 3.14-rc3
Workload: fsstress with 10 threads
Reproducible scenario: N/A
Is it related to this patch?
commit 1370e97bb2eb1ef2df7355204e5a4ba13e12b861
Author: Waiman Long <[email protected]>
Date: Thu Sep 12 10:55:34 2013 -0400
seqlock: Add a new locking reader type
In d_walk(),
/*
* might go back up the wrong parent if we have had a
rename
* or deletion
*/
if (this_parent != child->d_parent ||
(child->d_flags & DCACHE_DENTRY_KILLED) ||
--> I suspect that the upper conditions can trigger rename_retry even
though rename_retry was done once before.
need_seqretry(&rename_lock, seq)) {
spin_unlock(&this_parent->d_lock);
rcu_read_unlock();
goto rename_retry;
}
Thanks,
=============================================
[ INFO: possible recursive locking detected ]
3.14.0-rc3.f2fs+ #10 Tainted: G O
---------------------------------------------
fsstress/10709 is trying to acquire lock:
(rename_lock){+.+...}, at: [<ffffffff811cdfbd>] d_walk+0x4d/0x3c0
but task is already holding lock:
(rename_lock){+.+...}, at: [<ffffffff811cdfbd>] d_walk+0x4d/0x3c0
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(rename_lock);
lock(rename_lock);
*** DEADLOCK ***
May be due to missing lock nesting notation
4 locks held by fsstress/10709:
#0: (sb_writers#11){.+.+.+}, at: [<ffffffff811d84d4>] mnt_want_write
+0x24/0x50
#1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff811c4c44>]
do_rmdir+0x134/0x1f0
#2: (&type->i_mutex_dir_key#3){+.+.+.}, at: [<ffffffff811c4a47>]
vfs_rmdir+0x57/0x120
#3: (rename_lock){+.+...}, at: [<ffffffff811cdfbd>] d_walk+0x4d/0x3c0
stack backtrace:
CPU: 3 PID: 10709 Comm: fsstress Tainted: G O 3.14.0-rc3.f2fs+
#10
Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77
Extreme4, BIOS P2.30 09/21/2012
ffffffff82173620 ffff88002b1f1bc8 ffffffff81719b27 0000000000000007
ffffffff82173620 ffff88002b1f1cc8 ffffffff810a72f5 ffff88002b1f1bf8
0000000000021810 ffff880036cb8000 0000000000000000 ffff88002b1f1d48
Call Trace:
[<ffffffff81719b27>] dump_stack+0x4e/0x68
[<ffffffff810a72f5>] __lock_acquire+0x715/0x1e20
[<ffffffff810c59d8>] ? rcu_irq_exit+0x68/0xb0
[<ffffffff81726ddc>] ? retint_restore_args+0xe/0xe
[<ffffffff810a905e>] lock_acquire+0x8e/0x110
[<ffffffff811cdfbd>] ? d_walk+0x4d/0x3c0
[<ffffffff811ce230>] ? d_walk+0x2c0/0x3c0
[<ffffffff811cd190>] ? umount_collect+0x130/0x130
[<ffffffff81725d7e>] _raw_spin_lock+0x3e/0x80
[<ffffffff811cdfbd>] ? d_walk+0x4d/0x3c0
[<ffffffff811cdfbd>] d_walk+0x4d/0x3c0
[<ffffffff811ce081>] ? d_walk+0x111/0x3c0
[<ffffffff811cd190>] ? umount_collect+0x130/0x130
[<ffffffff811ce8a8>] shrink_dcache_parent+0x68/0x80
[<ffffffff811c4a97>] vfs_rmdir+0xa7/0x120
[<ffffffff813365c9>] ? apparmor_path_rmdir+0x19/0x20
[<ffffffff811c4cd3>] do_rmdir+0x1c3/0x1f0
[<ffffffff81388bbe>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff811c80e6>] SyS_rmdir+0x16/0x20
[<ffffffff8172fb12>] system_call_fastpath+0x16/0x1b
BUG: soft lockup - CPU#0 stuck for 22s! [fsstress:10715]
Modules linked in: f2fs(O) snd_hda_codec_hdmi snd_hda_codec_realtek
snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep i915 snd_pcm
snd_timer drm_kms_helper psmouse drm snd serio_raw soundcore
i2c_algo_bit lpc_ich mac_hid mxm_wmi tpm_tis video wmi lp parport tg3
ptp pps_core [last unloaded: f2fs]
irq event stamp: 11620621
hardirqs last enabled at (11620621): [<ffffffff810a557d>]
debug_check_no_locks_freed+0x9d/0x170
hardirqs last disabled at (11620620): [<ffffffff810a5520>]
debug_check_no_locks_freed+0x40/0x170
softirqs last enabled at (11617658): [<ffffffff810575e3>] __do_softirq
+0x1d3/0x2f0
softirqs last disabled at (11617625): [<ffffffff810579d5>] irq_exit
+0xb5/0xc0
CPU: 0 PID: 10715 Comm: fsstress Tainted: G O 3.14.0-rc3.f2fs+
#10
Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77
Extreme4, BIOS P2.30 09/21/2012
task: ffff880036fcc840 ti: ffff88000ecbe000 task.ti: ffff88000ecbe000
RIP: 0010:[<ffffffff8138771f>] [<ffffffff8138771f>] delay_tsc+0x4f/0x80
RSP: 0018:ffff88000ecbfc88 EFLAGS: 00000202
RAX: 00000000be2c718a RBX: ffffffff81726ddc RCX: 000000000000001f
RDX: 0000000000035c36 RSI: 00000000be2c716b RDI: 0000000000000001
RBP: ffff88000ecbfc88 R08: 0000000000000000 R09: 0000000000000000
R10: 57ffe8b10853be00 R11: 0000000000000000 R12: ffff88000ecbfbf8
R13: ffff88007cfd4040 R14: ffff88000ecbe000 R15: ffff880036fcc840
FS: 00007f0bdf19d700(0000) GS:ffff88007ce00000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5f111f09f1 CR3: 00000000188f0000 CR4: 00000000001407f0
Stack:
ffff88000ecbfc98 ffffffff8138760f ffff88000ecbfcc8 ffffffff810acd64
ffffffff81c045d0 ffffffff81c045b8 ffffffff81c045d0 ffff88003ee0dfd0
ffff88000ecbfcf8 ffffffff81725d9e ffffffff811d0112 0000000000000000
Call Trace:
[<ffffffff8138760f>] __delay+0xf/0x20
[<ffffffff810acd64>] do_raw_spin_lock+0xf4/0x160
[<ffffffff81725d9e>] _raw_spin_lock+0x5e/0x80
[<ffffffff811d0112>] ? d_move+0x22/0x90
[<ffffffff811d0112>] d_move+0x22/0x90
[<ffffffff811c52ef>] vfs_rename+0x5ef/0x600
[<ffffffff811c05bd>] ? lookup_real+0x1d/0x60
[<ffffffff811c566d>] SYSC_renameat+0x36d/0x3c0
[<ffffffff811d7f76>] ? mntput_no_expire+0x56/0x190
[<ffffffff811d7f8e>] ? mntput_no_expire+0x6e/0x190
[<ffffffff811d7f37>] ? mntput_no_expire+0x17/0x190
[<ffffffff811d80d4>] ? mntput+0x24/0x40
[<ffffffff81388bbe>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff811c889b>] SyS_rename+0x1b/0x20
[<ffffffff8172fb12>] system_call_fastpath+0x16/0x1b
--
Jaegeuk Kim
Samsung
On 02/27/2014 07:45 PM, Jaegeuk Kim wrote:
> Hi Al,
>
> In the following configuration, I met a deadlock condition like below.
>
> Kernel: 3.14-rc3
> Workload: fsstress with 10 threads
> Reproducible scenario: N/A
>
> Is it related to this patch?
> commit 1370e97bb2eb1ef2df7355204e5a4ba13e12b861
> Author: Waiman Long<[email protected]>
> Date: Thu Sep 12 10:55:34 2013 -0400
>
> seqlock: Add a new locking reader type
>
> In d_walk(),
> /*
> * might go back up the wrong parent if we have had a
> rename
> * or deletion
> */
> if (this_parent != child->d_parent ||
> (child->d_flags& DCACHE_DENTRY_KILLED) ||
>
> --> I suspect that the upper conditions can trigger rename_retry even
> though rename_retry was done once before.
>
> need_seqretry(&rename_lock, seq)) {
> spin_unlock(&this_parent->d_lock);
> rcu_read_unlock();
> goto rename_retry;
> }
>
> Thanks,
>
>
It seems like the rename_lock may not be able to fully protect against
the setting of the DCACHE_DENTRY_KILLED flag. Al, should this case be
handled separately? I am 100% sure if we could just release the lock and
let it try again without causing infinite loop.
-Longman