2018-01-10 06:58:08

by syzbot

[permalink] [raw]
Subject: possible deadlock in shmem_file_llseek

Hello,

syzkaller hit the following crash on
d476c5334f1dee122534b29639f8d46a85ecbb9d
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.
C reproducer is attached
syzkaller reproducer is attached. See https://goo.gl/kgGztJ
for information about syzkaller reproducers


IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: [email protected]
It will help syzbot understand when the bug is fixed. See footer for
details.
If you forward the report, please keep this part and the footer.


======================================================
audit: type=1400 audit(1515535006.781:8): avc: denied { map } for
pid=3497 comm="syzkaller303685" path="/dev/ashmem" dev="devtmpfs" ino=178
scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1
WARNING: possible circular locking dependency detected
4.15.0-rc7+ #255 Not tainted
------------------------------------------------------
syzkaller303685/3497 is trying to acquire lock:
(&sb->s_type->i_mutex_key#11){++++}, at: [<000000006886b3fb>] inode_lock
include/linux/fs.h:713 [inline]
(&sb->s_type->i_mutex_key#11){++++}, at: [<000000006886b3fb>]
shmem_file_llseek+0xef/0x240 mm/shmem.c:2579

but task is already holding lock:
(ashmem_mutex){+.+.}, at: [<00000000a44304a8>] ashmem_llseek+0x56/0x1f0
drivers/staging/android/ashmem.c:334

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (ashmem_mutex){+.+.}:
__mutex_lock_common kernel/locking/mutex.c:756 [inline]
__mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
ashmem_mmap+0x53/0x410 drivers/staging/android/ashmem.c:370
call_mmap include/linux/fs.h:1777 [inline]
mmap_region+0xa99/0x15a0 mm/mmap.c:1705
do_mmap+0x6c0/0xe00 mm/mmap.c:1483
do_mmap_pgoff include/linux/mm.h:2217 [inline]
vm_mmap_pgoff+0x1de/0x280 mm/util.c:333
SYSC_mmap_pgoff mm/mmap.c:1533 [inline]
SyS_mmap_pgoff+0x462/0x5f0 mm/mmap.c:1491
SYSC_mmap arch/x86/kernel/sys_x86_64.c:100 [inline]
SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:91
entry_SYSCALL_64_fastpath+0x23/0x9a

-> #1 (&mm->mmap_sem){++++}:
__might_fault+0x13a/0x1d0 mm/memory.c:4529
_copy_to_user+0x2c/0xc0 lib/usercopy.c:25
copy_to_user include/linux/uaccess.h:155 [inline]
filldir+0x1a7/0x320 fs/readdir.c:196
dir_emit_dot include/linux/fs.h:3374 [inline]
dir_emit_dots include/linux/fs.h:3385 [inline]
dcache_readdir+0x12d/0x5e0 fs/libfs.c:192
iterate_dir+0x1ca/0x530 fs/readdir.c:51
SYSC_getdents fs/readdir.c:231 [inline]
SyS_getdents+0x225/0x450 fs/readdir.c:212
entry_SYSCALL_64_fastpath+0x23/0x9a

-> #0 (&sb->s_type->i_mutex_key#11){++++}:
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3914
down_write+0x87/0x120 kernel/locking/rwsem.c:70
inode_lock include/linux/fs.h:713 [inline]
shmem_file_llseek+0xef/0x240 mm/shmem.c:2579
vfs_llseek+0xa2/0xd0 fs/read_write.c:300
ashmem_llseek+0xe7/0x1f0 drivers/staging/android/ashmem.c:346
vfs_llseek fs/read_write.c:300 [inline]
SYSC_lseek fs/read_write.c:313 [inline]
SyS_lseek+0xeb/0x170 fs/read_write.c:304
entry_SYSCALL_64_fastpath+0x23/0x9a

other info that might help us debug this:

Chain exists of:
&sb->s_type->i_mutex_key#11 --> &mm->mmap_sem --> ashmem_mutex

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(ashmem_mutex);
lock(&mm->mmap_sem);
lock(ashmem_mutex);
lock(&sb->s_type->i_mutex_key#11);

*** DEADLOCK ***

1 lock held by syzkaller303685/3497:
#0: (ashmem_mutex){+.+.}, at: [<00000000a44304a8>]
ashmem_llseek+0x56/0x1f0 drivers/staging/android/ashmem.c:334

stack backtrace:
CPU: 0 PID: 3497 Comm: syzkaller303685 Not tainted 4.15.0-rc7+ #255
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x194/0x257 lib/dump_stack.c:53
print_circular_bug.isra.37+0x2cd/0x2dc kernel/locking/lockdep.c:1218
check_prev_add kernel/locking/lockdep.c:1858 [inline]
check_prevs_add kernel/locking/lockdep.c:1971 [inline]
validate_chain kernel/locking/lockdep.c:2412 [inline]
__lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3426
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3914
down_write+0x87/0x120 kernel/locking/rwsem.c:70
inode_lock include/linux/fs.h:713 [inline]
shmem_file_llseek+0xef/0x240 mm/shmem.c:2579
vfs_llseek+0xa2/0xd0 fs/read_write.c:300
ashmem_llseek+0xe7/0x1f0 drivers/staging/android/ashmem.c:346
vfs_llseek fs/read_write.c:300 [inline]
SYSC_lseek fs/read_write.c:313 [inline]
SyS_lseek+0xeb/0x170 fs/read_write.c:304
entry_SYSCALL_64_fastpath+0x23/0x9a
RIP: 0033:0x444aa9
RSP: 002b:00007ffd2bec96f8 EFLAGS: 00000217 ORIG_RAX: 0000000000000008
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000444aa9
RDX: 0000000000000003 RSI: fffffffffffffffc RDI: 0000000000000004
RBP: 00000000


---
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to [email protected].

syzbot will keep track of this bug report.
If you forgot to add the Reported-by tag, once the fix for this bug is
merged
into any tree, please reply to this email with:
#syz fix: exact-commit-title
If you want to test a patch for this bug, please reply with:
#syz test: git://repo/address.git branch
and provide the patch inline or as an attachment.
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.
Note: all commands must start from beginning of the line in the email body.


Attachments:
config.txt (131.01 kB)
raw.log (6.64 kB)
repro.txt (503.00 B)
repro.c (5.20 kB)
Download all attachments

2018-01-10 08:02:25

by Dmitry Vyukov

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek

On Wed, Jan 10, 2018 at 7:58 AM, syzbot
<[email protected]> wrote:
> Hello,
>
> syzkaller hit the following crash on
> d476c5334f1dee122534b29639f8d46a85ecbb9d
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/master
> compiler: gcc (GCC) 7.1.1 20170620
> .config is attached
> Raw console output is attached.
> C reproducer is attached
> syzkaller reproducer is attached. See https://goo.gl/kgGztJ
> for information about syzkaller reproducers

I think this is drivers/staging/android/ashmem.c, +android maintainers.


> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: [email protected]
> It will help syzbot understand when the bug is fixed. See footer for
> details.
> If you forward the report, please keep this part and the footer.
>
>
> ======================================================
> audit: type=1400 audit(1515535006.781:8): avc: denied { map } for
> pid=3497 comm="syzkaller303685" path="/dev/ashmem" dev="devtmpfs" ino=178
> scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1
> WARNING: possible circular locking dependency detected
> 4.15.0-rc7+ #255 Not tainted
> ------------------------------------------------------
> syzkaller303685/3497 is trying to acquire lock:
> (&sb->s_type->i_mutex_key#11){++++}, at: [<000000006886b3fb>] inode_lock
> include/linux/fs.h:713 [inline]
> (&sb->s_type->i_mutex_key#11){++++}, at: [<000000006886b3fb>]
> shmem_file_llseek+0xef/0x240 mm/shmem.c:2579
>
> but task is already holding lock:
> (ashmem_mutex){+.+.}, at: [<00000000a44304a8>] ashmem_llseek+0x56/0x1f0
> drivers/staging/android/ashmem.c:334
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #2 (ashmem_mutex){+.+.}:
> __mutex_lock_common kernel/locking/mutex.c:756 [inline]
> __mutex_lock+0x16f/0x1a80 kernel/locking/mutex.c:893
> mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
> ashmem_mmap+0x53/0x410 drivers/staging/android/ashmem.c:370
> call_mmap include/linux/fs.h:1777 [inline]
> mmap_region+0xa99/0x15a0 mm/mmap.c:1705
> do_mmap+0x6c0/0xe00 mm/mmap.c:1483
> do_mmap_pgoff include/linux/mm.h:2217 [inline]
> vm_mmap_pgoff+0x1de/0x280 mm/util.c:333
> SYSC_mmap_pgoff mm/mmap.c:1533 [inline]
> SyS_mmap_pgoff+0x462/0x5f0 mm/mmap.c:1491
> SYSC_mmap arch/x86/kernel/sys_x86_64.c:100 [inline]
> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:91
> entry_SYSCALL_64_fastpath+0x23/0x9a
>
> -> #1 (&mm->mmap_sem){++++}:
> __might_fault+0x13a/0x1d0 mm/memory.c:4529
> _copy_to_user+0x2c/0xc0 lib/usercopy.c:25
> copy_to_user include/linux/uaccess.h:155 [inline]
> filldir+0x1a7/0x320 fs/readdir.c:196
> dir_emit_dot include/linux/fs.h:3374 [inline]
> dir_emit_dots include/linux/fs.h:3385 [inline]
> dcache_readdir+0x12d/0x5e0 fs/libfs.c:192
> iterate_dir+0x1ca/0x530 fs/readdir.c:51
> SYSC_getdents fs/readdir.c:231 [inline]
> SyS_getdents+0x225/0x450 fs/readdir.c:212
> entry_SYSCALL_64_fastpath+0x23/0x9a
>
> -> #0 (&sb->s_type->i_mutex_key#11){++++}:
> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3914
> down_write+0x87/0x120 kernel/locking/rwsem.c:70
> inode_lock include/linux/fs.h:713 [inline]
> shmem_file_llseek+0xef/0x240 mm/shmem.c:2579
> vfs_llseek+0xa2/0xd0 fs/read_write.c:300
> ashmem_llseek+0xe7/0x1f0 drivers/staging/android/ashmem.c:346
> vfs_llseek fs/read_write.c:300 [inline]
> SYSC_lseek fs/read_write.c:313 [inline]
> SyS_lseek+0xeb/0x170 fs/read_write.c:304
> entry_SYSCALL_64_fastpath+0x23/0x9a
>
> other info that might help us debug this:
>
> Chain exists of:
> &sb->s_type->i_mutex_key#11 --> &mm->mmap_sem --> ashmem_mutex
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(ashmem_mutex);
> lock(&mm->mmap_sem);
> lock(ashmem_mutex);
> lock(&sb->s_type->i_mutex_key#11);
>
> *** DEADLOCK ***
>
> 1 lock held by syzkaller303685/3497:
> #0: (ashmem_mutex){+.+.}, at: [<00000000a44304a8>]
> ashmem_llseek+0x56/0x1f0 drivers/staging/android/ashmem.c:334
>
> stack backtrace:
> CPU: 0 PID: 3497 Comm: syzkaller303685 Not tainted 4.15.0-rc7+ #255
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:17 [inline]
> dump_stack+0x194/0x257 lib/dump_stack.c:53
> print_circular_bug.isra.37+0x2cd/0x2dc kernel/locking/lockdep.c:1218
> check_prev_add kernel/locking/lockdep.c:1858 [inline]
> check_prevs_add kernel/locking/lockdep.c:1971 [inline]
> validate_chain kernel/locking/lockdep.c:2412 [inline]
> __lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3426
> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3914
> down_write+0x87/0x120 kernel/locking/rwsem.c:70
> inode_lock include/linux/fs.h:713 [inline]
> shmem_file_llseek+0xef/0x240 mm/shmem.c:2579
> vfs_llseek+0xa2/0xd0 fs/read_write.c:300
> ashmem_llseek+0xe7/0x1f0 drivers/staging/android/ashmem.c:346
> vfs_llseek fs/read_write.c:300 [inline]
> SYSC_lseek fs/read_write.c:313 [inline]
> SyS_lseek+0xeb/0x170 fs/read_write.c:304
> entry_SYSCALL_64_fastpath+0x23/0x9a
> RIP: 0033:0x444aa9
> RSP: 002b:00007ffd2bec96f8 EFLAGS: 00000217 ORIG_RAX: 0000000000000008
> RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000444aa9
> RDX: 0000000000000003 RSI: fffffffffffffffc RDI: 0000000000000004
> RBP: 00000000
>
>
> ---
> This bug is generated by a dumb bot. It may contain errors.
> See https://goo.gl/tpsmEJ for details.
> Direct all questions to [email protected].
>
> syzbot will keep track of this bug report.
> If you forgot to add the Reported-by tag, once the fix for this bug is
> merged
> into any tree, please reply to this email with:
> #syz fix: exact-commit-title
> If you want to test a patch for this bug, please reply with:
> #syz test: git://repo/address.git branch
> and provide the patch inline or as an attachment.
> To mark this as a duplicate of another syzbot report, please reply with:
> #syz dup: exact-subject-of-another-report
> If it's a one-off invalid bug report, please reply with:
> #syz invalid
> Note: if the crash happens again, it will cause creation of a new bug
> report.
> Note: all commands must start from beginning of the line in the email body.
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/001a1144d6e854b3c90562668d74%40google.com.
> For more options, visit https://groups.google.com/d/optout.

2018-01-24 17:49:10

by Joel Fernandes

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek


#syz test: https://github.com/joelagnel/linux.git test-ashmem

--->8----
From 55dd7e77e0e46af101ef2195d618d5f56af1023d Mon Sep 17 00:00:00 2001
From: Joel Fernandes <[email protected]>
Date: Tue, 23 Jan 2018 17:45:04 -0800
Subject: [PATCH] ashmem: Fix lockdep issue during llseek

ashmem_mutex create a chain of dependencies like so:

(1)
mmap syscall ->
mmap_sem -> (acquired)
ashmem_mmap
ashmem_mutex (try to acquire)
(block)

(2)
llseek syscall ->
ashmem_llseek ->
ashmem_mutex -> (acquired)
inode_lock ->
inode->i_rwsem (try to acquire)
(block)

(3)
getdents ->
iterate_dir ->
inode_lock ->
inode->i_rwsem (acquired)
copy_to_user ->
mmap_sem (try to acquire)

There is a lock ordering created between mmap_sem and inode->i_rwsem
during a syzcaller test, this patch fixes the issue by releasing the
ashmem_mutex before the call the vfs_llseek, and reacquiring it after.

Reported-by: [email protected]
Signed-off-by: Joel Fernandes <[email protected]>
---
drivers/staging/android/ashmem.c | 2 ++
1 file changed, 2 insertions(+)

diff --git a/drivers/staging/android/ashmem.c b/drivers/staging/android/ashmem.c
index 0f695df14c9d..248983cf2db1 100644
--- a/drivers/staging/android/ashmem.c
+++ b/drivers/staging/android/ashmem.c
@@ -343,7 +343,9 @@ static loff_t ashmem_llseek(struct file *file, loff_t offset, int origin)
goto out;
}

+ mutex_unlock(&ashmem_mutex);
ret = vfs_llseek(asma->file, offset, origin);
+ mutex_lock(&ashmem_mutex);
if (ret < 0)
goto out;

--
2.16.0.rc1.238.g530d649a79-goog



2018-01-24 18:41:41

by Dmitry Vyukov

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek

On Wed, Jan 24, 2018 at 6:47 PM, Joel Fernandes <[email protected]> wrote:
>
> #syz test: https://github.com/joelagnel/linux.git test-ashmem


Oops, this email somehow ended up without Content-Type header, which
was unexpected on syzbot side. Now should be fixed with:
https://github.com/google/syzkaller/commit/866f1102f786c19a67e3857f891eaf5107550663

Let's try again:

#syz test: https://github.com/joelagnel/linux.git test-ashmem


Attachments:
patch (450.00 B)

2018-01-24 18:43:27

by syzbot

[permalink] [raw]
Subject: possible deadlock in shmem_file_llseek

Hello,

syzbot tried to test the proposed patch but build/boot failed:

patch is already applied


Tested on https://github.com/joelagnel/linux.git/test-ashmem commit
32f813bb0d06c1e189ac336f8c3c7377f85c71f0 (Wed Jan 24 01:45:04 2018 +0000)
ashmem: Fix lockdep issue during llseek

compiler: gcc (GCC) 7.1.1 20170620
Patch is attached.




Attachments:
patch.diff (329.00 B)

2018-01-24 18:45:01

by Dmitry Vyukov

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek

On Wed, Jan 24, 2018 at 7:42 PM, syzbot
<[email protected]> wrote:
> Hello,
>
> syzbot tried to test the proposed patch but build/boot failed:
>
> patch is already applied
>
>
> Tested on https://github.com/joelagnel/linux.git/test-ashmem commit
> 32f813bb0d06c1e189ac336f8c3c7377f85c71f0 (Wed Jan 24 01:45:04 2018 +0000)
> ashmem: Fix lockdep issue during llseek
>
> compiler: gcc (GCC) 7.1.1 20170620
> Patch is attached.

Right. We probably want:

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
master


Attachments:
patch (450.00 B)

2018-01-24 19:03:01

by syzbot

[permalink] [raw]
Subject: possible deadlock in shmem_file_llseek

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger
crash:

Reported-and-tested-by:
[email protected]

Note: the tag will also help syzbot to understand when the bug is fixed.

Tested on upstream commit
03fae44b41062419aedcf2ae4ad68034f440f862 (Wed Jan 24 18:08:16 2018 +0000)
Merge tag 'trace-v4.15-rc9' of
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace

compiler: gcc (GCC) 7.1.1 20170620
Patch is attached.
Kernel config is attached.


---
There is no WARRANTY for the result, to the extent permitted by applicable
law.
Except when otherwise stated in writing syzbot provides the result "AS IS"
without warranty of any kind, either expressed or implied, but not limited
to,
the implied warranties of merchantability and fittness for a particular
purpose.
The entire risk as to the quality of the result is with you. Should the
result
prove defective, you assume the cost of all necessary servicing, repair or
correction.


Attachments:
patch.diff (329.00 B)
config.txt (131.28 kB)
Download all attachments

2018-01-24 21:12:29

by Joel Fernandes

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek

On Wed, Jan 24, 2018 at 10:40 AM, Dmitry Vyukov <[email protected]> wrote:
> On Wed, Jan 24, 2018 at 6:47 PM, Joel Fernandes <[email protected]> wrote:
>>
>> #syz test: https://github.com/joelagnel/linux.git test-ashmem
>
>
> Oops, this email somehow ended up without Content-Type header, which
> was unexpected on syzbot side. Now should be fixed with:
> https://github.com/google/syzkaller/commit/866f1102f786c19a67e3857f891eaf5107550663
>
> Let's try again:
>
> #syz test: https://github.com/joelagnel/linux.git test-ashmem

Hehe glad I could trigger the syzcaller side bug ;D I actually edited
the email in plain text to be more git send-email friendly... sorry
I'm old fashioned but in this case it worked out for the better ;)

- Joel

2018-01-30 03:05:41

by Joel Fernandes

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek

On Wed, Jan 24, 2018 at 10:40 AM, Dmitry Vyukov <[email protected]> wrote:
> On Wed, Jan 24, 2018 at 6:47 PM, Joel Fernandes <[email protected]> wrote:
>>
>> #syz test: https://github.com/joelagnel/linux.git test-ashmem

Here's an updated patch. Just wanted to test it once more.

thanks,

- Joel


Attachments:
0001-ashmem-Fix-lockdep-issue-during-llseek.patch (2.10 kB)

2018-01-31 17:25:11

by Joel Fernandes

[permalink] [raw]
Subject: Re: possible deadlock in shmem_file_llseek

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
master


Attachments:
0001-ashmem-Fix-lockdep-issue-during-llseek.patch (2.10 kB)

2018-01-31 17:55:58

by syzbot

[permalink] [raw]
Subject: possible deadlock in shmem_file_llseek

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger
crash:

Reported-and-tested-by:
[email protected]

Note: the tag will also help syzbot to understand when the bug is fixed.

Tested on upstream commit
3da90b159b146672f830bcd2489dd3a1f4e9e089 (Wed Jan 31 03:07:32 2018 +0000)
Merge tag 'f2fs-for-4.16-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs

compiler: gcc (GCC) 7.1.1 20170620
Patch is attached.
Kernel config is attached.


---
There is no WARRANTY for the result, to the extent permitted by applicable
law.
Except when otherwise stated in writing syzbot provides the result "AS IS"
without warranty of any kind, either expressed or implied, but not limited
to,
the implied warranties of merchantability and fittness for a particular
purpose.
The entire risk as to the quality of the result is with you. Should the
result
prove defective, you assume the cost of all necessary servicing, repair or
correction.


Attachments:
patch.diff (508.00 B)
config.txt (131.37 kB)
Download all attachments