We found the following deadlock situations in low memory scenarios:
Thread A Thread B
- __writeback_single_inode
- fuse_write_inode
- fuse_simple_request
- __fuse_request_send
- request_wait_answer
- fuse_dev_splice_read
- fuse_copy_fill
- __alloc_pages_direct_reclaim
- do_shrink_slab
- super_cache_scan
- shrink_dentry_list
- dentry_unlink_inode
- iput_final
- inode_wait_for_writeback
The request and inode processed by Thread A and B are the same, which
causes a deadlock. To avoid this, we remove the __GFP_FS flag when
allocating memory in fuse_copy_fill, so there will be no memory
reclaimation in super_cache_scan.
Signed-off-by: Huang Jianan <[email protected]>
Signed-off-by: Guo Weichao <[email protected]>
---
fs/fuse/dev.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 588f8d1240aa..e580b9d04c25 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -721,7 +721,7 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
if (cs->nr_segs >= cs->pipe->max_usage)
return -EIO;
- page = alloc_page(GFP_HIGHUSER);
+ page = alloc_page(GFP_HIGHUSER & ~__GFP_FS);
if (!page)
return -ENOMEM;
--
2.25.1
Hi all,
This patch works well in our product, but I am not sure this is the correct
way to solve this problem. I think that the inode->i_count shouldn't be
zero after iput is executed in dentry_unlink_inode, then the inode won't
be writeback. But i haven't found where iget is missing.
Thanks,
Jianan
On 2021/2/2 12:08, Huang Jianan wrote:
> We found the following deadlock situations in low memory scenarios:
> Thread A Thread B
> - __writeback_single_inode
> - fuse_write_inode
> - fuse_simple_request
> - __fuse_request_send
> - request_wait_answer
> - fuse_dev_splice_read
> - fuse_copy_fill
> - __alloc_pages_direct_reclaim
> - do_shrink_slab
> - super_cache_scan
> - shrink_dentry_list
> - dentry_unlink_inode
> - iput_final
> - inode_wait_for_writeback
>
> The request and inode processed by Thread A and B are the same, which
> causes a deadlock. To avoid this, we remove the __GFP_FS flag when
> allocating memory in fuse_copy_fill, so there will be no memory
> reclaimation in super_cache_scan.
>
> Signed-off-by: Huang Jianan <[email protected]>
> Signed-off-by: Guo Weichao <[email protected]>
> ---
> fs/fuse/dev.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
> index 588f8d1240aa..e580b9d04c25 100644
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -721,7 +721,7 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)
> if (cs->nr_segs >= cs->pipe->max_usage)
> return -EIO;
>
> - page = alloc_page(GFP_HIGHUSER);
> + page = alloc_page(GFP_HIGHUSER & ~__GFP_FS);
> if (!page)
> return -ENOMEM;
>
friendly ping ... ????
On 2021/2/2 12:11, Huang Jianan via fuse-devel wrote:
> Hi all,
>
>
> This patch works well in our product, but I am not sure this is the
> correct
>
> way to solve this problem. I think that the inode->i_count shouldn't be
>
> zero after iput is executed in dentry_unlink_inode, then the inode won't
>
> be writeback. But i haven't found where iget is missing.
>
>
> Thanks,
>
> Jianan
>
> On 2021/2/2 12:08, Huang Jianan wrote:
>> We found the following deadlock situations in low memory scenarios:
>> Thread A Thread B
>> - __writeback_single_inode
>> - fuse_write_inode
>> - fuse_simple_request
>> - __fuse_request_send
>> - request_wait_answer
>> - fuse_dev_splice_read
>> - fuse_copy_fill
>> - __alloc_pages_direct_reclaim
>> - do_shrink_slab
>> - super_cache_scan
>> - shrink_dentry_list
>> - dentry_unlink_inode
>> - iput_final
>> - inode_wait_for_writeback
>>
>> The request and inode processed by Thread A and B are the same, which
>> causes a deadlock. To avoid this, we remove the __GFP_FS flag when
>> allocating memory in fuse_copy_fill, so there will be no memory
>> reclaimation in super_cache_scan.
>>
>> Signed-off-by: Huang Jianan <[email protected]>
>> Signed-off-by: Guo Weichao <[email protected]>
>> ---
>> fs/fuse/dev.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
>> index 588f8d1240aa..e580b9d04c25 100644
>> --- a/fs/fuse/dev.c
>> +++ b/fs/fuse/dev.c
>> @@ -721,7 +721,7 @@ static int fuse_copy_fill(struct fuse_copy_state
>> *cs)
>> if (cs->nr_segs >= cs->pipe->max_usage)
>> return -EIO;
>> - page = alloc_page(GFP_HIGHUSER);
>> + page = alloc_page(GFP_HIGHUSER & ~__GFP_FS);
>> if (!page)
>> return -ENOMEM;
>
>
On Tue, Feb 2, 2021 at 5:41 AM Huang Jianan via fuse-devel
<[email protected]> wrote:
>
> We found the following deadlock situations in low memory scenarios:
> Thread A Thread B
> - __writeback_single_inode
> - fuse_write_inode
> - fuse_simple_request
> - __fuse_request_send
> - request_wait_answer
> - fuse_dev_splice_read
> - fuse_copy_fill
> - __alloc_pages_direct_reclaim
> - do_shrink_slab
> - super_cache_scan
> - shrink_dentry_list
> - dentry_unlink_inode
> - iput_final
> - inode_wait_for_writeback
On what kernel are you seeing this?
I don't see how it can happen on upstream kernels, since there's a
"write_inode_now(inode, 1)" call in fuse_release() and nothing can
dirty the inode after the file has been released.
Thanks,
Miklos
Hi, Miklos
The similar issue occurs in our Android device(4G RAM + 3G zram + 8 arm cores + kernel-4.14) too.
Under the monkey test, kswapd and fuse daemon thread deadlocked when free pages is extreme low
(less than 1/2 of the min watermark), the backtrace of the 2 threads is as follows. kswapd
try to evict inode to free some memory(blocked at inode_wait_for_writeback), and fuse daemon thread
handle the fuse inode write request, which is throttled when do direct reclaim in page allocation
slow path(blocked at throttle_direct_reclaim). As the __GFP_FS is set, the thread is throttled until
kswapd free enough pages until watermark ok(check allow_direct_reclaim), which cause the deadlock.
Although the kernel version is 4.14, the same issue exists in the upstream kernel too.
kswapd0 D 26485194.538158 157 1287917 23577482 0x1a20840 0x0 157 438599862461462
<ffffff8beec866b4> __switch_to+0x134/0x150
<ffffff8befb838cc> __schedule+0xd5c/0x1100
<ffffff8befb83ce0> schedule+0x70/0x90
<ffffff8befb849b4> bit_wait+0x14/0x54
<ffffff8befb84350> __wait_on_bit+0x74/0xe0
<ffffff8beeeae0b4> inode_wait_for_writeback+0xa0/0xe4
<ffffff8beee9b95c> evict+0xa4/0x284
<ffffff8beee99b58> iput+0x25c/0x2ac
<ffffff8beee9602c> dentry_unlink_inode+0xd8/0xe4
<ffffff8beee93274> __dentry_kill+0xe8/0x22c
<ffffff8beee9374c> shrink_dentry_list+0x19c/0x3b0
<ffffff8beee9340c> prune_dcache_sb+0x54/0x80
<ffffff8beee79c50> super_cache_scan+0x114/0x164
<ffffff8beee16504> shrink_slab+0x454/0x528
<ffffff8beee1b81c> shrink_node+0x144/0x318
<ffffff8beee1a100> kswapd+0x830/0x9e0
<ffffff8beecde9f0> kthread+0x17c/0x18c
<ffffff8beec856a4> ret_from_fork+0x10/0x18
<ffffffffffffffff> 0xffffffffffffffff
Thread-19 D 7542.719029 2888 24823 5064 0x1404840 0x1000008 24235 438599754021693
<ffffff8beec866b4> __switch_to+0x134/0x150
<ffffff8befb838cc> __schedule+0xd5c/0x1100
<ffffff8befb83ce0> schedule+0x70/0x90
<ffffff8beee18258> try_to_free_pages+0x264/0x4b0
<ffffff8beee06978> __alloc_pages_nodemask+0x7a4/0x10d0
<ffffff8beefac784> fuse_copy_fill+0x15c/0x210
<ffffff8beefabbcc> fuse_dev_do_read+0x434/0xc24
<ffffff8beefab56c> fuse_dev_splice_read+0x84/0x1d8
<ffffff8beeeb5788> SyS_splice+0x67c/0x8bc
<ffffff8beec83fc0> el0_svc_naked+0x34/0x38
<ffffffffffffffff> 0xffffffffffffffff
code snippet:
static bool throttle_direct_reclaim(...)
{
...
/*
* If the caller cannot enter the filesystem, it's possible that it
* is due to the caller holding an FS lock or performing a journal
* transaction in the case of a filesystem like ext[3|4]. In this case,
* it is not safe to block on pfmemalloc_wait as kswapd could be
* blocked waiting on the same lock. Instead, throttle for up to a
* second before continuing.
*/
if (!(gfp_mask & __GFP_FS)) {
wait_event_interruptible_timeout(pgdat->pfmemalloc_wait,
allow_direct_reclaim(pgdat), HZ);
goto check_pending;
}
/* Throttle until kswapd wakes the process */
wait_event_killable(zone->zone_pgdat->pfmemalloc_wait,
allow_direct_reclaim(pgdat));
...
}
Thanks,
yanwu
On Wed, 24 Mar 2021 16:28:35 +0100 Miklos Szeredi via <[email protected]> wrote:
> On what kernel are you seeing this?
> I don't see how it can happen on upstream kernels, since there's a
>"write_inode_now(inode, 1)" call in fuse_release() and nothing can
> dirty the inode after the file has been released.
> Thanks,
> Miklos
>On Tue, Feb 2, 2021 at 5:41 AM Huang Jianan via fuse-devel
><[email protected]> wrote:
>>
>> We found the following deadlock situations in low memory scenarios:
>> Thread A Thread B
>> - __writeback_single_inode
>> - fuse_write_inode
>> - fuse_simple_request
>> - __fuse_request_send
>> - request_wait_answer
>> - fuse_dev_splice_read
>> - fuse_copy_fill
>> - __alloc_pages_direct_reclaim
>> - do_shrink_slab
>> - super_cache_scan
>> - shrink_dentry_list
>> - dentry_unlink_inode
>> - iput_final
>> - inode_wait_for_writeback
On Thu, 10 Mar 2022 at 12:11, Rokudo Yan <[email protected]> wrote:
>
> Hi, Miklos
>
> The similar issue occurs in our Android device(4G RAM + 3G zram + 8 arm cores + kernel-4.14) too.
> Under the monkey test, kswapd and fuse daemon thread deadlocked when free pages is extreme low
> (less than 1/2 of the min watermark), the backtrace of the 2 threads is as follows. kswapd
> try to evict inode to free some memory(blocked at inode_wait_for_writeback), and fuse daemon thread
> handle the fuse inode write request, which is throttled when do direct reclaim in page allocation
> slow path(blocked at throttle_direct_reclaim). As the __GFP_FS is set, the thread is throttled until
> kswapd free enough pages until watermark ok(check allow_direct_reclaim), which cause the deadlock.
> Although the kernel version is 4.14, the same issue exists in the upstream kernel too.
>
> kswapd0 D 26485194.538158 157 1287917 23577482 0x1a20840 0x0 157 438599862461462
> <ffffff8beec866b4> __switch_to+0x134/0x150
> <ffffff8befb838cc> __schedule+0xd5c/0x1100
> <ffffff8befb83ce0> schedule+0x70/0x90
> <ffffff8befb849b4> bit_wait+0x14/0x54
> <ffffff8befb84350> __wait_on_bit+0x74/0xe0
> <ffffff8beeeae0b4> inode_wait_for_writeback+0xa0/0xe4
This is the one I don't understand. Fuse inodes must never be dirty
on eviction for the reason stated in my previous reply:
> > I don't see how it can happen on upstream kernels, since there's a
> >"write_inode_now(inode, 1)" call in fuse_release() and nothing can
> > dirty the inode after the file has been released.
If you could trace the source of this dirtyness I think that would
explain this deadlock.
Thanks,
Miklos