I tried testing to see if this fixed [1], and it appears to be
triggering a lockdep warning[2] at this line in the patch:
[1] https://syzkaller.appspot.com/bug?extid=f4582777a19ec422b517
[2] https://syzkaller.appspot.com/x/report.txt?x=17260843280000
> diff --git a/fs/ext4/file.c b/fs/ext4/file.c
> index d101b3b0c7da..9df82d72eb90 100644
> --- a/fs/ext4/file.c
> +++ b/fs/ext4/file.c
> @@ -808,6 +809,27 @@ static int ext4_file_mmap(struct file *file, struct vm_area_struct *vma)
> if (!daxdev_mapping_supported(vma, dax_dev))
> return -EOPNOTSUPP;
>
> + /*
> + * Writing via mmap has no logic to handle inline data, so we
> + * need to call ext4_convert_inline_data() to convert the inode
> + * to normal format before doing so, otherwise a BUG_ON will be
> + * triggered in ext4_writepages() due to the
> + * EXT4_STATE_MAY_INLINE_DATA flag. Moreover, we need to grab
> + * i_rwsem during conversion, since clearing and setting the
> + * inline data flag may race with ext4_buffered_write_iter()
> + * to trigger a BUG_ON.
> + */
> + if (ext4_has_feature_inline_data(sb) &&
> + vma->vm_flags & VM_SHARED && vma->vm_flags & VM_MAYWRITE) {
> + int err;
> +
> + inode_lock(inode); <=================== LOCKDEP warning
> + err = ext4_convert_inline_data(inode);
> + inode_unlock(inode);
> + if (err)
> + return err;
> + }
The details of the lockdep warning from [2], which appears to be a
mmap(2) racing with a buffered write(2) are below. Could you take a
look?
Thanks!
- Ted
======================================================
WARNING: possible circular locking dependency detected
6.4.0-rc4-syzkaller-geb1f822c76be-dirty #0 Not tainted
------------------------------------------------------
syz-executor.4/5589 is trying to acquire lock:
ffff888024228168 (&mm->mmap_lock){++++}-{3:3}, at: mmap_read_lock include/linux/mmap_lock.h:142 [inline]
ffff888024228168 (&mm->mmap_lock){++++}-{3:3}, at: do_user_addr_fault+0xb3d/0x1210 arch/x86/mm/fault.c:1391
but task is already holding lock:
ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: inode_lock include/linux/fs.h:775 [inline]
ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xb0/0x460 fs/ext4/file.c:283
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&sb->s_type->i_mutex_key#8){++++}-{3:3}:
down_write+0x92/0x200 kernel/locking/rwsem.c:1573
inode_lock include/linux/fs.h:775 [inline]
ext4_file_mmap+0x62e/0x800 fs/ext4/file.c:826
call_mmap include/linux/fs.h:1873 [inline]
mmap_region+0x694/0x28d0 mm/mmap.c:2652
do_mmap+0x831/0xf60 mm/mmap.c:1394
vm_mmap_pgoff+0x1a2/0x3b0 mm/util.c:543
ksys_mmap_pgoff+0x41f/0x5a0 mm/mmap.c:1440
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
-> #0 (&mm->mmap_lock){++++}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3113 [inline]
check_prevs_add kernel/locking/lockdep.c:3232 [inline]
validate_chain kernel/locking/lockdep.c:3847 [inline]
__lock_acquire+0x2fcd/0x5f30 kernel/locking/lockdep.c:5088
lock_acquire kernel/locking/lockdep.c:5705 [inline]
lock_acquire+0x1b1/0x520 kernel/locking/lockdep.c:5670
down_read+0x9c/0x480 kernel/locking/rwsem.c:1520
mmap_read_lock include/linux/mmap_lock.h:142 [inline]
do_user_addr_fault+0xb3d/0x1210 arch/x86/mm/fault.c:1391
handle_page_fault arch/x86/mm/fault.c:1534 [inline]
exc_page_fault+0x98/0x170 arch/x86/mm/fault.c:1590
asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:570
fault_in_readable+0x1a5/0x210 mm/gup.c:1856
fault_in_iov_iter_readable+0x252/0x2c0 lib/iov_iter.c:362
generic_perform_write+0x1ae/0x570 mm/filemap.c:3913
ext4_buffered_write_iter+0x15b/0x460 fs/ext4/file.c:289
ext4_file_write_iter+0xbe0/0x1740 fs/ext4/file.c:710
call_write_iter include/linux/fs.h:1868 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x945/0xd50 fs/read_write.c:584
ksys_write+0x12b/0x250 fs/read_write.c:637
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&sb->s_type->i_mutex_key#8);
lock(&mm->mmap_lock);
lock(&sb->s_type->i_mutex_key#8);
rlock(&mm->mmap_lock);
*** DEADLOCK ***
3 locks held by syz-executor.4/5589:
#0: ffff88802a7fe0e8 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe7/0x100 fs/file.c:1047
#1: ffff888021fe0460 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x12b/0x250 fs/read_write.c:637
#2: ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: inode_lock include/linux/fs.h:775 [inline]
#2: ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xb0/0x460 fs/ext4/file.c:283
stack backtrace:
CPU: 0 PID: 5589 Comm: syz-executor.4 Not tainted 6.4.0-rc4-syzkaller-geb1f822c76be-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xd9/0x150 lib/dump_stack.c:106
check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2188
check_prev_add kernel/locking/lockdep.c:3113 [inline]
check_prevs_add kernel/locking/lockdep.c:3232 [inline]
validate_chain kernel/locking/lockdep.c:3847 [inline]
__lock_acquire+0x2fcd/0x5f30 kernel/locking/lockdep.c:5088
lock_acquire kernel/locking/lockdep.c:5705 [inline]
lock_acquire+0x1b1/0x520 kernel/locking/lockdep.c:5670
down_read+0x9c/0x480 kernel/locking/rwsem.c:1520
mmap_read_lock include/linux/mmap_lock.h:142 [inline]
do_user_addr_fault+0xb3d/0x1210 arch/x86/mm/fault.c:1391
handle_page_fault arch/x86/mm/fault.c:1534 [inline]
exc_page_fault+0x98/0x170 arch/x86/mm/fault.c:1590
asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:570
RIP: 0010:fault_in_readable+0x1a5/0x210 mm/gup.c:1856
Code: fc ff df 48 c7 04 02 00 00 00 00 48 83 c4 48 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 45 31 e4 eb ce e8 ae 51 c4 ff 45 31 f6 <41> 8a 45 00 31 ff 44 89 f6 88 44 24 28 e8 b9 4d c4 ff 45 85 f6 75
RSP: 0018:ffffc90006187a38 EFLAGS: 00050246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff888026905940 RSI: ffffffff81bff672 RDI: 0000000000000007
RBP: 00000000200002cc R08: 0000000000000007 R09: 0000000000000000
R10: 00000000000002c0 R11: 1ffffffff219cbe3 R12: 000000000000000c
R13: 00000000200002c0 R14: 0000000000000000 R15: 1ffff92000c30f48
fault_in_iov_iter_readable+0x252/0x2c0 lib/iov_iter.c:362
generic_perform_write+0x1ae/0x570 mm/filemap.c:3913
ext4_buffered_write_iter+0x15b/0x460 fs/ext4/file.c:289
ext4_file_write_iter+0xbe0/0x1740 fs/ext4/file.c:710
call_write_iter include/linux/fs.h:1868 [inline]
new_sync_write fs/read_write.c:491 [inline]
vfs_write+0x945/0xd50 fs/read_write.c:584
ksys_write+0x12b/0x250 fs/read_write.c:637
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f2359a8c169
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f235a721168 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 00007f2359bac050 RCX: 00007f2359a8c169
RDX: 000000000000000c RSI: 00000000200002c0 RDI: 0000000000000004
RBP: 00007f2359ae7ca1 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffea89659df R14: 00007f235a721300 R15: 0000000000022000
</TASK>
----------------
Code disassembly (best guess), 2 bytes skipped:
0: df 48 c7 fisttps -0x39(%rax)
3: 04 02 add $0x2,%al
5: 00 00 add %al,(%rax)
7: 00 00 add %al,(%rax)
9: 48 83 c4 48 add $0x48,%rsp
d: 4c 89 e0 mov %r12,%rax
10: 5b pop %rbx
11: 5d pop %rbp
12: 41 5c pop %r12
14: 41 5d pop %r13
16: 41 5e pop %r14
18: 41 5f pop %r15
1a: c3 retq
1b: 45 31 e4 xor %r12d,%r12d
1e: eb ce jmp 0xffffffee
20: e8 ae 51 c4 ff callq 0xffc451d3
25: 45 31 f6 xor %r14d,%r14d
* 28: 41 8a 45 00 mov 0x0(%r13),%al <-- trapping instruction
2c: 31 ff xor %edi,%edi
2e: 44 89 f6 mov %r14d,%esi
31: 88 44 24 28 mov %al,0x28(%rsp)
35: e8 b9 4d c4 ff callq 0xffc44df3
3a: 45 85 f6 test %r14d,%r14d
3d: 75 .byte 0x75
On Sat, Jun 03, 2023 at 11:04:45PM -0400, Theodore Ts'o wrote:
> I tried testing to see if this fixed [1], and it appears to be
> triggering a lockdep warning[2] at this line in the patch:
>
> [1] https://syzkaller.appspot.com/bug?extid=f4582777a19ec422b517
> [2] https://syzkaller.appspot.com/x/report.txt?x=17260843280000
Looking at this more closely, the fundamental problem is by the time
ext4_file_mmap() is called, the mm layer has already taken
current->mm->mmap_lock, and when we try to take the inode_lock, this
causes locking ordering problems with how buffered write path works,
which take the inode_lock first, and then in some cases, may end up
taking the mmap_lock if there is a page fault for the buffer used for
the buffered write.
If we're going to stick with the approach in this patch, I think what
we would need is to add a pre_mmap() function to file_operations
struct, which would get called by the mmap path *before* taking
current->mm->mmap_lock, so we can do the inline conversion before we
take the mmap_lock.
I'm not sure how the mm folks would react to such a proposal, though.
I could be seen as a bit hacky, and it's not clear that any file
system other than ext4 would need something like this. Willy, as
someone who does a lot of work in both mm and fs worlds --- I'm
curious what you think about this idea?
Thanks,
- Ted
> > diff --git a/fs/ext4/file.c b/fs/ext4/file.c
> > index d101b3b0c7da..9df82d72eb90 100644
> > --- a/fs/ext4/file.c
> > +++ b/fs/ext4/file.c
> > @@ -808,6 +809,27 @@ static int ext4_file_mmap(struct file *file, struct vm_area_struct *vma)
> > if (!daxdev_mapping_supported(vma, dax_dev))
> > return -EOPNOTSUPP;
> >
> > + /*
> > + * Writing via mmap has no logic to handle inline data, so we
> > + * need to call ext4_convert_inline_data() to convert the inode
> > + * to normal format before doing so, otherwise a BUG_ON will be
> > + * triggered in ext4_writepages() due to the
> > + * EXT4_STATE_MAY_INLINE_DATA flag. Moreover, we need to grab
> > + * i_rwsem during conversion, since clearing and setting the
> > + * inline data flag may race with ext4_buffered_write_iter()
> > + * to trigger a BUG_ON.
> > + */
> > + if (ext4_has_feature_inline_data(sb) &&
> > + vma->vm_flags & VM_SHARED && vma->vm_flags & VM_MAYWRITE) {
> > + int err;
> > +
> > + inode_lock(inode); <=================== LOCKDEP warning
> > + err = ext4_convert_inline_data(inode);
> > + inode_unlock(inode);
> > + if (err)
> > + return err;
> > + }
>
>
> The details of the lockdep warning from [2], which appears to be a
> mmap(2) racing with a buffered write(2) are below. Could you take a
> look?
>
> Thanks!
>
> - Ted
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 6.4.0-rc4-syzkaller-geb1f822c76be-dirty #0 Not tainted
> ------------------------------------------------------
> syz-executor.4/5589 is trying to acquire lock:
> ffff888024228168 (&mm->mmap_lock){++++}-{3:3}, at: mmap_read_lock include/linux/mmap_lock.h:142 [inline]
> ffff888024228168 (&mm->mmap_lock){++++}-{3:3}, at: do_user_addr_fault+0xb3d/0x1210 arch/x86/mm/fault.c:1391
>
> but task is already holding lock:
> ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: inode_lock include/linux/fs.h:775 [inline]
> ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xb0/0x460 fs/ext4/file.c:283
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #1 (&sb->s_type->i_mutex_key#8){++++}-{3:3}:
> down_write+0x92/0x200 kernel/locking/rwsem.c:1573
> inode_lock include/linux/fs.h:775 [inline]
> ext4_file_mmap+0x62e/0x800 fs/ext4/file.c:826
> call_mmap include/linux/fs.h:1873 [inline]
> mmap_region+0x694/0x28d0 mm/mmap.c:2652
> do_mmap+0x831/0xf60 mm/mmap.c:1394
> vm_mmap_pgoff+0x1a2/0x3b0 mm/util.c:543
> ksys_mmap_pgoff+0x41f/0x5a0 mm/mmap.c:1440
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> -> #0 (&mm->mmap_lock){++++}-{3:3}:
> check_prev_add kernel/locking/lockdep.c:3113 [inline]
> check_prevs_add kernel/locking/lockdep.c:3232 [inline]
> validate_chain kernel/locking/lockdep.c:3847 [inline]
> __lock_acquire+0x2fcd/0x5f30 kernel/locking/lockdep.c:5088
> lock_acquire kernel/locking/lockdep.c:5705 [inline]
> lock_acquire+0x1b1/0x520 kernel/locking/lockdep.c:5670
> down_read+0x9c/0x480 kernel/locking/rwsem.c:1520
> mmap_read_lock include/linux/mmap_lock.h:142 [inline]
> do_user_addr_fault+0xb3d/0x1210 arch/x86/mm/fault.c:1391
> handle_page_fault arch/x86/mm/fault.c:1534 [inline]
> exc_page_fault+0x98/0x170 arch/x86/mm/fault.c:1590
> asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:570
> fault_in_readable+0x1a5/0x210 mm/gup.c:1856
> fault_in_iov_iter_readable+0x252/0x2c0 lib/iov_iter.c:362
> generic_perform_write+0x1ae/0x570 mm/filemap.c:3913
> ext4_buffered_write_iter+0x15b/0x460 fs/ext4/file.c:289
> ext4_file_write_iter+0xbe0/0x1740 fs/ext4/file.c:710
> call_write_iter include/linux/fs.h:1868 [inline]
> new_sync_write fs/read_write.c:491 [inline]
> vfs_write+0x945/0xd50 fs/read_write.c:584
> ksys_write+0x12b/0x250 fs/read_write.c:637
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
>
> other info that might help us debug this:
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&sb->s_type->i_mutex_key#8);
> lock(&mm->mmap_lock);
> lock(&sb->s_type->i_mutex_key#8);
> rlock(&mm->mmap_lock);
>
> *** DEADLOCK ***
>
> 3 locks held by syz-executor.4/5589:
> #0: ffff88802a7fe0e8 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe7/0x100 fs/file.c:1047
> #1: ffff888021fe0460 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x12b/0x250 fs/read_write.c:637
> #2: ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: inode_lock include/linux/fs.h:775 [inline]
> #2: ffff88806a066800 (&sb->s_type->i_mutex_key#8){++++}-{3:3}, at: ext4_buffered_write_iter+0xb0/0x460 fs/ext4/file.c:283
>
> stack backtrace:
> CPU: 0 PID: 5589 Comm: syz-executor.4 Not tainted 6.4.0-rc4-syzkaller-geb1f822c76be-dirty #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/25/2023
> Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0xd9/0x150 lib/dump_stack.c:106
> check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2188
> check_prev_add kernel/locking/lockdep.c:3113 [inline]
> check_prevs_add kernel/locking/lockdep.c:3232 [inline]
> validate_chain kernel/locking/lockdep.c:3847 [inline]
> __lock_acquire+0x2fcd/0x5f30 kernel/locking/lockdep.c:5088
> lock_acquire kernel/locking/lockdep.c:5705 [inline]
> lock_acquire+0x1b1/0x520 kernel/locking/lockdep.c:5670
> down_read+0x9c/0x480 kernel/locking/rwsem.c:1520
> mmap_read_lock include/linux/mmap_lock.h:142 [inline]
> do_user_addr_fault+0xb3d/0x1210 arch/x86/mm/fault.c:1391
> handle_page_fault arch/x86/mm/fault.c:1534 [inline]
> exc_page_fault+0x98/0x170 arch/x86/mm/fault.c:1590
> asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:570
> RIP: 0010:fault_in_readable+0x1a5/0x210 mm/gup.c:1856
> Code: fc ff df 48 c7 04 02 00 00 00 00 48 83 c4 48 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 45 31 e4 eb ce e8 ae 51 c4 ff 45 31 f6 <41> 8a 45 00 31 ff 44 89 f6 88 44 24 28 e8 b9 4d c4 ff 45 85 f6 75
> RSP: 0018:ffffc90006187a38 EFLAGS: 00050246
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: ffff888026905940 RSI: ffffffff81bff672 RDI: 0000000000000007
> RBP: 00000000200002cc R08: 0000000000000007 R09: 0000000000000000
> R10: 00000000000002c0 R11: 1ffffffff219cbe3 R12: 000000000000000c
> R13: 00000000200002c0 R14: 0000000000000000 R15: 1ffff92000c30f48
> fault_in_iov_iter_readable+0x252/0x2c0 lib/iov_iter.c:362
> generic_perform_write+0x1ae/0x570 mm/filemap.c:3913
> ext4_buffered_write_iter+0x15b/0x460 fs/ext4/file.c:289
> ext4_file_write_iter+0xbe0/0x1740 fs/ext4/file.c:710
> call_write_iter include/linux/fs.h:1868 [inline]
> new_sync_write fs/read_write.c:491 [inline]
> vfs_write+0x945/0xd50 fs/read_write.c:584
> ksys_write+0x12b/0x250 fs/read_write.c:637
> do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
> entry_SYSCALL_64_after_hwframe+0x63/0xcd
> RIP: 0033:0x7f2359a8c169
> Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007f235a721168 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> RAX: ffffffffffffffda RBX: 00007f2359bac050 RCX: 00007f2359a8c169
> RDX: 000000000000000c RSI: 00000000200002c0 RDI: 0000000000000004
> RBP: 00007f2359ae7ca1 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> R13: 00007ffea89659df R14: 00007f235a721300 R15: 0000000000022000
> </TASK>
> ----------------
> Code disassembly (best guess), 2 bytes skipped:
> 0: df 48 c7 fisttps -0x39(%rax)
> 3: 04 02 add $0x2,%al
> 5: 00 00 add %al,(%rax)
> 7: 00 00 add %al,(%rax)
> 9: 48 83 c4 48 add $0x48,%rsp
> d: 4c 89 e0 mov %r12,%rax
> 10: 5b pop %rbx
> 11: 5d pop %rbp
> 12: 41 5c pop %r12
> 14: 41 5d pop %r13
> 16: 41 5e pop %r14
> 18: 41 5f pop %r15
> 1a: c3 retq
> 1b: 45 31 e4 xor %r12d,%r12d
> 1e: eb ce jmp 0xffffffee
> 20: e8 ae 51 c4 ff callq 0xffc451d3
> 25: 45 31 f6 xor %r14d,%r14d
> * 28: 41 8a 45 00 mov 0x0(%r13),%al <-- trapping instruction
> 2c: 31 ff xor %edi,%edi
> 2e: 44 89 f6 mov %r14d,%esi
> 31: 88 44 24 28 mov %al,0x28(%rsp)
> 35: e8 b9 4d c4 ff callq 0xffc44df3
> 3a: 45 85 f6 test %r14d,%r14d
> 3d: 75 .byte 0x75
>
On Sun, Jun 04, 2023 at 05:08:21PM -0400, Theodore Ts'o wrote:
> On Sat, Jun 03, 2023 at 11:04:45PM -0400, Theodore Ts'o wrote:
> > I tried testing to see if this fixed [1], and it appears to be
> > triggering a lockdep warning[2] at this line in the patch:
> >
> > [1] https://syzkaller.appspot.com/bug?extid=f4582777a19ec422b517
> > [2] https://syzkaller.appspot.com/x/report.txt?x=17260843280000
>
> Looking at this more closely, the fundamental problem is by the time
> ext4_file_mmap() is called, the mm layer has already taken
> current->mm->mmap_lock, and when we try to take the inode_lock, this
> causes locking ordering problems with how buffered write path works,
> which take the inode_lock first, and then in some cases, may end up
> taking the mmap_lock if there is a page fault for the buffer used for
> the buffered write.
>
> If we're going to stick with the approach in this patch, I think what
> we would need is to add a pre_mmap() function to file_operations
> struct, which would get called by the mmap path *before* taking
> current->mm->mmap_lock, so we can do the inline conversion before we
> take the mmap_lock.
>
> I'm not sure how the mm folks would react to such a proposal, though.
> I could be seen as a bit hacky, and it's not clear that any file
> system other than ext4 would need something like this. Willy, as
> someone who does a lot of work in both mm and fs worlds --- I'm
> curious what you think about this idea?
I'm probably missing something here, but why do we need to convert inline
data in page_mkwrite? mmap() can't change i_size (stores past i_size are
discarded), so we should be able to simply copy the data from the page
cache into the inode and write the inode when it comes to writepages()
time.
Unless somebody does a truncate() or write() that expands i_size, but we
should be able to do the conversion then without the mmap_lock held. No?
I'm not too familiar with inline data.
On Mon 05-06-23 02:58:15, Matthew Wilcox wrote:
> On Sun, Jun 04, 2023 at 05:08:21PM -0400, Theodore Ts'o wrote:
> > On Sat, Jun 03, 2023 at 11:04:45PM -0400, Theodore Ts'o wrote:
> > > I tried testing to see if this fixed [1], and it appears to be
> > > triggering a lockdep warning[2] at this line in the patch:
> > >
> > > [1] https://syzkaller.appspot.com/bug?extid=f4582777a19ec422b517
> > > [2] https://syzkaller.appspot.com/x/report.txt?x=17260843280000
> >
> > Looking at this more closely, the fundamental problem is by the time
> > ext4_file_mmap() is called, the mm layer has already taken
> > current->mm->mmap_lock, and when we try to take the inode_lock, this
> > causes locking ordering problems with how buffered write path works,
> > which take the inode_lock first, and then in some cases, may end up
> > taking the mmap_lock if there is a page fault for the buffer used for
> > the buffered write.
> >
> > If we're going to stick with the approach in this patch, I think what
> > we would need is to add a pre_mmap() function to file_operations
> > struct, which would get called by the mmap path *before* taking
> > current->mm->mmap_lock, so we can do the inline conversion before we
> > take the mmap_lock.
> >
> > I'm not sure how the mm folks would react to such a proposal, though.
> > I could be seen as a bit hacky, and it's not clear that any file
> > system other than ext4 would need something like this. Willy, as
> > someone who does a lot of work in both mm and fs worlds --- I'm
> > curious what you think about this idea?
>
> I'm probably missing something here, but why do we need to convert inline
> data in page_mkwrite? mmap() can't change i_size (stores past i_size are
> discarded), so we should be able to simply copy the data from the page
> cache into the inode and write the inode when it comes to writepages()
> time.
>
> Unless somebody does a truncate() or write() that expands i_size, but we
> should be able to do the conversion then without the mmap_lock held. No?
> I'm not too familiar with inline data.
Yeah, I agree, that is also the conclusion I have arrived at when thinking
about this problem now. We should be able to just remove the conversion
from ext4_page_mkwrite() and rely on write(2) or truncate(2) doing it when
growing i_size.
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Mon 05-06-23 11:16:55, Jan Kara wrote:
> On Mon 05-06-23 02:58:15, Matthew Wilcox wrote:
> > On Sun, Jun 04, 2023 at 05:08:21PM -0400, Theodore Ts'o wrote:
> > > On Sat, Jun 03, 2023 at 11:04:45PM -0400, Theodore Ts'o wrote:
> > > > I tried testing to see if this fixed [1], and it appears to be
> > > > triggering a lockdep warning[2] at this line in the patch:
> > > >
> > > > [1] https://syzkaller.appspot.com/bug?extid=f4582777a19ec422b517
> > > > [2] https://syzkaller.appspot.com/x/report.txt?x=17260843280000
> > >
> > > Looking at this more closely, the fundamental problem is by the time
> > > ext4_file_mmap() is called, the mm layer has already taken
> > > current->mm->mmap_lock, and when we try to take the inode_lock, this
> > > causes locking ordering problems with how buffered write path works,
> > > which take the inode_lock first, and then in some cases, may end up
> > > taking the mmap_lock if there is a page fault for the buffer used for
> > > the buffered write.
> > >
> > > If we're going to stick with the approach in this patch, I think what
> > > we would need is to add a pre_mmap() function to file_operations
> > > struct, which would get called by the mmap path *before* taking
> > > current->mm->mmap_lock, so we can do the inline conversion before we
> > > take the mmap_lock.
> > >
> > > I'm not sure how the mm folks would react to such a proposal, though.
> > > I could be seen as a bit hacky, and it's not clear that any file
> > > system other than ext4 would need something like this. Willy, as
> > > someone who does a lot of work in both mm and fs worlds --- I'm
> > > curious what you think about this idea?
> >
> > I'm probably missing something here, but why do we need to convert inline
> > data in page_mkwrite? mmap() can't change i_size (stores past i_size are
> > discarded), so we should be able to simply copy the data from the page
> > cache into the inode and write the inode when it comes to writepages()
> > time.
> >
> > Unless somebody does a truncate() or write() that expands i_size, but we
> > should be able to do the conversion then without the mmap_lock held. No?
> > I'm not too familiar with inline data.
>
> Yeah, I agree, that is also the conclusion I have arrived at when thinking
> about this problem now. We should be able to just remove the conversion
> from ext4_page_mkwrite() and rely on write(2) or truncate(2) doing it when
> growing i_size.
OK, thinking more about this and searching through the history, I've
realized why the conversion is originally in ext4_page_mkwrite(). The
problem is described in commit 7b4cc9787fe35b ("ext4: evict inline data
when writing to memory map") but essentially it boils down to the fact that
ext4 writeback code does not expect dirty page for a file with inline data
because ext4_write_inline_data_end() should have copied the data into the
inode and cleared the folio's dirty flag.
Indeed messing with xattrs from the writeback path to copy page contents
into inline data xattr would be ... interesting. Hum, out of good ideas for
now :-|.
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Mon, Jun 05, 2023 at 02:21:41PM +0200, Jan Kara wrote:
> On Mon 05-06-23 11:16:55, Jan Kara wrote:
> > Yeah, I agree, that is also the conclusion I have arrived at when thinking
> > about this problem now. We should be able to just remove the conversion
> > from ext4_page_mkwrite() and rely on write(2) or truncate(2) doing it when
> > growing i_size.
>
> OK, thinking more about this and searching through the history, I've
> realized why the conversion is originally in ext4_page_mkwrite(). The
> problem is described in commit 7b4cc9787fe35b ("ext4: evict inline data
> when writing to memory map") but essentially it boils down to the fact that
> ext4 writeback code does not expect dirty page for a file with inline data
> because ext4_write_inline_data_end() should have copied the data into the
> inode and cleared the folio's dirty flag.
>
> Indeed messing with xattrs from the writeback path to copy page contents
> into inline data xattr would be ... interesting. Hum, out of good ideas for
> now :-|.
Is it so bad? Now that we don't have writepage in ext4, only
writepages, it seems like we have a considerably more benign locking
environment to work in.
On Mon 05-06-23 15:55:35, Matthew Wilcox wrote:
> On Mon, Jun 05, 2023 at 02:21:41PM +0200, Jan Kara wrote:
> > On Mon 05-06-23 11:16:55, Jan Kara wrote:
> > > Yeah, I agree, that is also the conclusion I have arrived at when thinking
> > > about this problem now. We should be able to just remove the conversion
> > > from ext4_page_mkwrite() and rely on write(2) or truncate(2) doing it when
> > > growing i_size.
> >
> > OK, thinking more about this and searching through the history, I've
> > realized why the conversion is originally in ext4_page_mkwrite(). The
> > problem is described in commit 7b4cc9787fe35b ("ext4: evict inline data
> > when writing to memory map") but essentially it boils down to the fact that
> > ext4 writeback code does not expect dirty page for a file with inline data
> > because ext4_write_inline_data_end() should have copied the data into the
> > inode and cleared the folio's dirty flag.
> >
> > Indeed messing with xattrs from the writeback path to copy page contents
> > into inline data xattr would be ... interesting. Hum, out of good ideas for
> > now :-|.
>
> Is it so bad? Now that we don't have writepage in ext4, only
> writepages, it seems like we have a considerably more benign locking
> environment to work in.
Well, yes, without ->writepage() it might be *possible*. But still rather
ugly. The problem is that in ->writepages() i_size is not stable. Thus also
whether the inode data is inline or not is not stable. We'd need inode_lock
for that but that is not easily doable in the writeback path - inode lock
would then become fs_reclaim unsafe...
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR