The driver may sleep under a read rcu lock, and function call path is:
xfs_iget (acquire the lock by rcu_read_lock)
"goto out_error_or_again" after xfs_iget_cache_hit
delay
schedule_timeout_uninterruptible --> may sleep
Meanwhile, the rcu_read_lock will be called twice in this situation.
To fix it, the lock is released before "goto".
Signed-off-by: Jia-Ju Bai <[email protected]>
---
fs/xfs/xfs_icache.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
index f61c84f8..c2a4722 100644
--- a/fs/xfs/xfs_icache.c
+++ b/fs/xfs/xfs_icache.c
@@ -600,8 +600,10 @@ struct xfs_inode *
if (ip) {
error = xfs_iget_cache_hit(pag, ip, ino, flags, lock_flags);
- if (error)
+ if (error) {
+ rcu_read_unlock();
goto out_error_or_again;
+ }
} else {
rcu_read_unlock();
XFS_STATS_INC(mp, xs_ig_missed);
--
1.7.9.5
On 2017年06月05日 16:17, Jia-Ju Bai wrote:
> The driver may sleep under a read rcu lock, and function call path is:
> xfs_iget (acquire the lock by rcu_read_lock)
> "goto out_error_or_again" after xfs_iget_cache_hit
> delay
> schedule_timeout_uninterruptible --> may sleep
> Meanwhile, the rcu_read_lock will be called twice in this situation.
>
> To fix it, the lock is released before "goto".
>
> Signed-off-by: Jia-Ju Bai <[email protected]>
> ---
> fs/xfs/xfs_icache.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
> index f61c84f8..c2a4722 100644
> --- a/fs/xfs/xfs_icache.c
> +++ b/fs/xfs/xfs_icache.c
> @@ -600,8 +600,10 @@ struct xfs_inode *
>
> if (ip) {
> error = xfs_iget_cache_hit(pag, ip, ino, flags, lock_flags);
> - if (error)
> + if (error) {
> + rcu_read_unlock();
Seems you are going to double unlock by doing this, since it is unlocked
in the xfs_iget_cache_hit.
Thanks
Shan Hai
> goto out_error_or_again;
> + }
> } else {
> rcu_read_unlock();
> XFS_STATS_INC(mp, xs_ig_missed);
On 06/05/2017 04:32 PM, Shan Hai wrote:
>
>
> On 2017年06月05日 16:17, Jia-Ju Bai wrote:
>> The driver may sleep under a read rcu lock, and function call path is:
>> xfs_iget (acquire the lock by rcu_read_lock)
>> "goto out_error_or_again" after xfs_iget_cache_hit
>> delay
>> schedule_timeout_uninterruptible --> may sleep
>> Meanwhile, the rcu_read_lock will be called twice in this situation.
>>
>> To fix it, the lock is released before "goto".
>>
>> Signed-off-by: Jia-Ju Bai <[email protected]>
>> ---
>> fs/xfs/xfs_icache.c | 4 +++-
>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
>> index f61c84f8..c2a4722 100644
>> --- a/fs/xfs/xfs_icache.c
>> +++ b/fs/xfs/xfs_icache.c
>> @@ -600,8 +600,10 @@ struct xfs_inode *
>> if (ip) {
>> error = xfs_iget_cache_hit(pag, ip, ino, flags, lock_flags);
>> - if (error)
>> + if (error) {
>> + rcu_read_unlock();
>
> Seems you are going to double unlock by doing this, since it is
> unlocked in the xfs_iget_cache_hit.
>
> Thanks
> Shan Hai
>
>> goto out_error_or_again;
>> + }
>> } else {
>> rcu_read_unlock();
>> XFS_STATS_INC(mp, xs_ig_missed);
>
I think you are right. Please ignore my patch.
Thanks,
Jia-Ju Bai
FYI, we noticed the following commit:
commit: 8ba626f5b236103c890c10d2a920f67f232bd122 ("fs: xfs: Fix a lock-twice and sleep-in-atomic bug in xfs_iget")
url: https://github.com/0day-ci/linux/commits/Jia-Ju-Bai/fs-xfs-Fix-a-lock-twice-and-sleep-in-atomic-bug-in-xfs_iget/20170605-180225
in testcase: xfstests
with following parameters:
disk: 4HDD
fs: xfs
test: generic-mid2
test-description: xfstests is a regression test suite for xfs and other files ystems.
test-url: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
on test machine: 4 threads Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz with 6G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+---------------------------------------------------------------------------+----------+------------+
| | v4.9-rc8 | 8ba626f5b2 |
+---------------------------------------------------------------------------+----------+------------+
| boot_successes | 1221 | 13 |
| boot_failures | 66 | 4 |
| BUG:sleeping_function_called_from_invalid_context_at_kernel/irq/manage.c | 17 | |
| calltrace:SyS_write | 8 | |
| calltrace:init_netconsole | 9 | |
| calltrace:SyS_finit_module | 14 | 1 |
| WARNING:at_fs/sysfs/dir.c:#sysfs_warn_dup | 5 | 1 |
| calltrace:parport_pc_init | 5 | 1 |
| WARNING:at_lib/kobject.c:#kobject_add_internal | 5 | 1 |
| invoked_oom-killer:gfp_mask=0x | 22 | |
| Mem-Info | 22 | |
| Kernel_panic-not_syncing:Out_of_memory_and_no_killable_processes | 22 | |
| BUG:kernel_hang_in_test_stage | 10 | |
| WARNING:at_net/mac80211/driver-ops.h:#ieee80211_set_default_key[mac80211] | 1 | |
| calltrace:SyS_sendmsg | 1 | |
| WARNING:at_net/mac80211/driver-ops.h:#ieee80211_key_replace[mac80211] | 1 | |
| BUG:kernel_hang_in_boot_stage | 2 | |
| BUG:kernel_in_stage | 11 | |
| BUG:scheduling_while_atomic | 0 | 3 |
| calltrace:SyS_ioctl | 0 | 4 |
| BUG:sleeping_function_called_from_invalid_context_at_mm/slab.h | 0 | 4 |
| kernel_BUG_at_mm/vmalloc.c | 0 | 4 |
| invalid_opcode:#[##] | 0 | 4 |
| RIP:vfree | 0 | 4 |
| Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 0 | 4 |
+---------------------------------------------------------------------------+----------+------------+
[ 212.428936] BUG: scheduling while atomic: fsstress/25384/0x00000000
[ 212.436169] Modules linked in: dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver netconsole btrfs xor raid6_pq sd_mod sg ata_generic pata_acpi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel eeepc_wmi ghash_clmulni_intel asus_wmi snd_hda_codec_hdmi sparse_keymap aesni_intel lrw gf128mul glue_helper ppdev snd_hda_codec_realtek rfkill ablk_helper snd_hda_codec_generic cryptd parport_pc serio_raw pcspkr ata_piix parport tpm_infineon pata_via libata snd_hda_intel i915 video snd_hda_codec drm_kms_helper snd_hda_core syscopyarea sysfillrect snd_hwdep sysimgblt fb_sys_fops shpchp snd_pcm snd_timer snd drm soundcore wmi ip_tables
[ 212.505041] CPU: 1 PID: 25384 Comm: fsstress Not tainted 4.9.0-rc8-00001-g8ba626f #1
[ 212.513623] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1002 04/01/2011
[ 212.523610] ffffc900022cf9d0 ffffffff81477159 0000000000000000 ffff8801b32993c0
[ 212.531964] ffffc900022cf9e0 ffffffff810a7d04 ffffc900022cfa38 ffffffff81962ed4
[ 212.540310] ffff88017ccf9d98 0000000000000000 ffff8801b32993c0 ffffc900022cfa80
[ 212.548667] Call Trace:
[ 212.551996] [<ffffffff81477159>] dump_stack+0x63/0x8a
[ 212.557995] [<ffffffff810a7d04>] __schedule_bug+0x54/0x70
[ 212.564322] [<ffffffff81962ed4>] __schedule+0x554/0x6f0
[ 212.570469] [<ffffffff819630ad>] schedule+0x3d/0x90
[ 212.576275] [<ffffffff819666f6>] schedule_timeout+0x1d6/0x3f0
[ 212.582950] [<ffffffff810f0340>] ? del_timer_sync+0x50/0x50
[ 212.589446] [<ffffffff8147d11d>] ? radix_tree_lookup+0xd/0x10
[ 212.596102] [<ffffffff8196699a>] schedule_timeout_uninterruptible+0x2a/0x30
[ 212.603983] [<ffffffff813a8058>] xfs_iget+0x1b8/0x960
[ 212.609966] [<ffffffff813b4040>] ? xfs_internal_inum+0x50/0x50
[ 212.616733] [<ffffffff813b43a3>] xfs_bulkstat_one_int+0x83/0x320
[ 212.623675] [<ffffffff813b4660>] xfs_bulkstat_one+0x20/0x30
[ 212.630181] [<ffffffff813b4b01>] xfs_bulkstat+0x491/0x640
[ 212.636513] [<ffffffff813b4640>] ? xfs_bulkstat_one_int+0x320/0x320
[ 212.643709] [<ffffffff813a90ba>] xfs_ioc_bulkstat+0xfa/0x1d0
[ 212.650297] [<ffffffff813ab98c>] xfs_file_ioctl+0x89c/0xc10
[ 212.656788] [<ffffffff81232082>] ? do_filp_open+0x92/0xe0
[ 212.663109] [<ffffffff810a81da>] ? __might_sleep+0x4a/0x80
[ 212.669511] [<ffffffff81235135>] do_vfs_ioctl+0x95/0x6c0
[ 212.675743] [<ffffffff812357d9>] SyS_ioctl+0x79/0x90
[ 212.681641] [<ffffffff819681b7>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 219.828342] BUG: scheduling while atomic: fsstress/25382/0x00000000
[ 219.835812] Modules linked in: dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver netconsole btrfs xor raid6_pq sd_mod sg ata_generic pata_acpi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel eeepc_wmi ghash_clmulni_intel asus_wmi snd_hda_codec_hdmi sparse_keymap aesni_intel lrw gf128mul glue_helper ppdev snd_hda_codec_realtek rfkill ablk_helper snd_hda_codec_generic cryptd parport_pc serio_raw pcspkr ata_piix parport tpm_infineon pata_via libata snd_hda_intel i915 video snd_hda_codec drm_kms_helper snd_hda_core syscopyarea sysfillrect snd_hwdep sysimgblt fb_sys_fops shpchp snd_pcm snd_timer snd drm soundcore wmi ip_tables
[ 219.905898] CPU: 1 PID: 25382 Comm: fsstress Tainted: G W 4.9.0-rc8-00001-g8ba626f #1
[ 219.915881] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1002 04/01/2011
[ 219.926050] ffffc900022bf9d0 ffffffff81477159 0000000000000000 ffff8801b32993c0
[ 219.934573] ffffc900022bf9e0 ffffffff810a7d04 ffffc900022bfa38 ffffffff81962ed4
[ 219.943107] ffff88017cf39c98 0000000000000000 ffff8801b32993c0 ffffc900022bfa80
[ 219.951642] Call Trace:
[ 219.955149] [<ffffffff81477159>] dump_stack+0x63/0x8a
[ 219.961345] [<ffffffff810a7d04>] __schedule_bug+0x54/0x70
[ 219.967866] [<ffffffff81962ed4>] __schedule+0x554/0x6f0
[ 219.974184] [<ffffffff819630ad>] schedule+0x3d/0x90
[ 219.980130] [<ffffffff819666f6>] schedule_timeout+0x1d6/0x3f0
[ 219.986927] [<ffffffff810f0340>] ? del_timer_sync+0x50/0x50
[ 219.993525] [<ffffffff8147d11d>] ? radix_tree_lookup+0xd/0x10
[ 220.000271] [<ffffffff8196699a>] schedule_timeout_uninterruptible+0x2a/0x30
[ 220.008229] [<ffffffff813a8058>] xfs_iget+0x1b8/0x960
[ 220.014265] [<ffffffff813b4040>] ? xfs_internal_inum+0x50/0x50
[ 220.021065] [<ffffffff813b43a3>] xfs_bulkstat_one_int+0x83/0x320
[ 220.028038] [<ffffffff813b4660>] xfs_bulkstat_one+0x20/0x30
[ 220.034572] [<ffffffff813b4b01>] xfs_bulkstat+0x491/0x640
[ 220.040919] [<ffffffff813b4640>] ? xfs_bulkstat_one_int+0x320/0x320
[ 220.048130] [<ffffffff813a90ba>] xfs_ioc_bulkstat+0xfa/0x1d0
[ 220.054736] [<ffffffff813ab98c>] xfs_file_ioctl+0x89c/0xc10
[ 220.061251] [<ffffffff813b099f>] ? xfs_iunlock+0xbf/0x120
[ 220.067598] [<ffffffff81232082>] ? do_filp_open+0x92/0xe0
[ 220.073944] [<ffffffff810a81da>] ? __might_sleep+0x4a/0x80
[ 220.080382] [<ffffffff81235135>] do_vfs_ioctl+0x95/0x6c0
[ 220.086648] [<ffffffff812357d9>] SyS_ioctl+0x79/0x90
[ 220.092561] [<ffffffff819681b7>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 220.099876] BUG: scheduling while atomic: fsstress/25382/0x00000000
[ 220.107445] Modules linked in: dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver netconsole btrfs xor raid6_pq sd_mod sg ata_generic pata_acpi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel eeepc_wmi ghash_clmulni_intel asus_wmi snd_hda_codec_hdmi sparse_keymap aesni_intel lrw gf128mul glue_helper ppdev snd_hda_codec_realtek rfkill ablk_helper snd_hda_codec_generic cryptd parport_pc serio_raw pcspkr ata_piix parport tpm_infineon pata_via libata snd_hda_intel i915 video snd_hda_codec drm_kms_helper snd_hda_core syscopyarea sysfillrect snd_hwdep sysimgblt fb_sys_fops shpchp snd_pcm snd_timer snd drm soundcore wmi ip_tables
[ 220.178175] CPU: 1 PID: 25382 Comm: fsstress Tainted: G W 4.9.0-rc8-00001-g8ba626f #1
[ 220.188211] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1002 04/01/2011
[ 220.198437] ffffc900022bf9d0 ffffffff81477159 0000000000000000 ffff8801b32993c0
[ 220.207026] ffffc900022bf9e0 ffffffff810a7d04 ffffc900022bfa38 ffffffff81962ed4
[ 220.215618] ffffc900022bfa80 00ff8801b3290040 ffff8801b32993c0 ffffc900022bfa80
[ 220.224211] Call Trace:
[ 220.227767] [<ffffffff81477159>] dump_stack+0x63/0x8a
[ 220.233993] [<ffffffff810a7d04>] __schedule_bug+0x54/0x70
[ 220.240548] [<ffffffff81962ed4>] __schedule+0x554/0x6f0
[ 220.246894] [<ffffffff819630ad>] schedule+0x3d/0x90
[ 220.252867] [<ffffffff819666f6>] schedule_timeout+0x1d6/0x3f0
[ 220.259691] [<ffffffff810f0340>] ? del_timer_sync+0x50/0x50
[ 220.266311] [<ffffffff8147d11d>] ? radix_tree_lookup+0xd/0x10
[ 220.273086] [<ffffffff8196699a>] schedule_timeout_uninterruptible+0x2a/0x30
[ 220.281070] [<ffffffff813a8058>] xfs_iget+0x1b8/0x960
[ 220.287127] [<ffffffff813b4040>] ? xfs_internal_inum+0x50/0x50
[ 220.293966] [<ffffffff813b43a3>] xfs_bulkstat_one_int+0x83/0x320
[ 220.300971] [<ffffffff813b4660>] xfs_bulkstat_one+0x20/0x30
[ 220.307563] [<ffffffff813b4b01>] xfs_bulkstat+0x491/0x640
[ 220.313949] [<ffffffff813b4640>] ? xfs_bulkstat_one_int+0x320/0x320
[ 220.321198] [<ffffffff813a90ba>] xfs_ioc_bulkstat+0xfa/0x1d0
[ 220.327841] [<ffffffff813ab98c>] xfs_file_ioctl+0x89c/0xc10
[ 220.334399] [<ffffffff813b099f>] ? xfs_iunlock+0xbf/0x120
[ 220.340810] [<ffffffff81232082>] ? do_filp_open+0x92/0xe0
[ 220.347195] [<ffffffff810a81da>] ? __might_sleep+0x4a/0x80
[ 220.353664] [<ffffffff81235135>] do_vfs_ioctl+0x95/0x6c0
[ 220.359960] [<ffffffff812357d9>] SyS_ioctl+0x79/0x90
[ 220.365980] [<ffffffff819681b7>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 255.148165] BUG: sleeping function called from invalid context at mm/slab.h:393
[ 255.155477] ------------[ cut here ]------------
[ 255.155479] kernel BUG at mm/vmalloc.c:1530!
[ 255.155481] invalid opcode: 0000 [#1] SMP
[ 255.155509] Modules linked in: dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver netconsole btrfs xor raid6_pq sd_mod sg ata_generic pata_acpi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel eeepc_wmi ghash_clmulni_intel asus_wmi snd_hda_codec_hdmi sparse_keymap aesni_intel lrw gf128mul glue_helper ppdev snd_hda_codec_realtek rfkill ablk_helper snd_hda_codec_generic cryptd parport_pc serio_raw pcspkr ata_piix parport tpm_infineon pata_via libata snd_hda_intel i915 video snd_hda_codec drm_kms_helper snd_hda_core syscopyarea sysfillrect snd_hwdep sysimgblt fb_sys_fops shpchp snd_pcm snd_timer snd drm soundcore wmi ip_tables
[ 255.155512] CPU: 3 PID: 25380 Comm: fsstress Tainted: G W 4.9.0-rc8-00001-g8ba626f #1
[ 255.155513] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1002 04/01/2011
[ 255.155515] task: ffff880171fc4b80 task.stack: ffffc900022ac000
[ 255.155521] RIP: 0010:[<ffffffff811d9a52>] [<ffffffff811d9a52>] vfree+0x32/0x70
[ 255.155522] RSP: 0018:ffffc900022afc20 EFLAGS: 00010206
[ 255.155523] RAX: 000000007fffffff RBX: 0000000000000040 RCX: 0000000000000004
[ 255.155524] RDX: 00001ffffffffffe RSI: 00000000000001fa RDI: ffffc90009005000
[ 255.155525] RBP: ffffc900022afc28 R08: 000000000001c8a0 R09: ffffffff811b2eba
[ 255.155526] R10: 0000000000000088 R11: ffff880150b6edc8 R12: 0000000000000088
[ 255.155527] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000
[ 255.155529] FS: 00007ff073697700(0000) GS:ffff8801b3380000(0000) knlGS:0000000000000000
[ 255.155530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 255.155531] CR2: 00007ff073694000 CR3: 000000017491a000 CR4: 00000000000406e0
[ 255.155532] Stack:
[ 255.155535] ffffffff811b2ec1 ffffc900022afd00 ffffffff813b49b3 ffffc900022afcd0
[ 255.155537] ffffc90009009000 0000013500000000 ffffc900022afd20 ffffc900022afd18
[ 255.155539] ffffc90009005000 00000004012afc90 ffffc90009005050 ffffffff813b4640
[ 255.155539] Call Trace:
[ 255.155543] [<ffffffff811b2ec1>] ? kvfree+0x31/0x40
[ 255.155547] [<ffffffff813b49b3>] xfs_bulkstat+0x343/0x640
[ 255.155550] [<ffffffff813b4640>] ? xfs_bulkstat_one_int+0x320/0x320
[ 255.155552] [<ffffffff813a90ba>] xfs_ioc_bulkstat+0xfa/0x1d0
[ 255.155554] [<ffffffff813ab98c>] xfs_file_ioctl+0x89c/0xc10
[ 255.155557] [<ffffffff810a81da>] ? __might_sleep+0x4a/0x80
[ 255.155559] [<ffffffff81232082>] ? do_filp_open+0x92/0xe0
[ 255.155561] [<ffffffff810a81da>] ? __might_sleep+0x4a/0x80
[ 255.155563] [<ffffffff81235135>] do_vfs_ioctl+0x95/0x6c0
[ 255.155565] [<ffffffff812357d9>] SyS_ioctl+0x79/0x90
[ 255.155569] [<ffffffff819681b7>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 255.155592] Code: 05 24 38 e3 7e a9 00 00 10 00 75 1f 48 85 ff 74 18 55 a9 00 ff 1f 00 48 89 e5 53 75 10 be 01 00 00 00 e8 c2 fe ff ff 5b 5d f3 c3 <0f> 0b 48 c7 c3 a0 47 01 00 65 48 03 1d cd 06 e3 7e 48 89 da 48
[ 255.155594] RIP [<ffffffff811d9a52>] vfree+0x32/0x70
[ 255.155595] RSP <ffffc900022afc20>
[ 255.155599] ---[ end trace 8866e5ac89d8da8e ]---
To reproduce:
git clone https://github.com/01org/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
Thanks,
Xiaolong