2018-01-19 19:30:56

by Shankara Pailoor

Subject: Possible Memory Leak in KCOV Linux 4.15-rc1

Hi Dmitry,

I added support for kcov in strace and I have been tracing a fairly
large program but after a little while, I notice that when I mmap a
new cover buffer, the call fails with ENOMEM. After killing the
program, I try and rerun and I notice that there is nearly no memory
on the system. When I do a kmemleak scan I get the following reports:

I believe the problem occurs when I try and setup the kcov buffer
again after an exec. Instead of reusing the old file descriptor I open
kcov again within that process. In that case, I don't know what
happens to the old kcov struct.

I don't see a maintainers list for kcov so I decided to email you
directly. Let me know what more information I can provide.

unreferenced object 0xffff8800633f6378 (size 96):
comm "runltp", pid 1847, jiffies 4294957922 (age 190.320s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc90008679000 (size 134217728):
comm "runltp", pid 1847, jiffies 4294957981 (age 190.261s)
hex dump (first 32 bytes):
e0 d2 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ..F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff8800633f6a58 (size 96):
comm "runltp", pid 1848, jiffies 4294958935 (age 189.307s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc9001067a000 (size 134217728):
comm "runltp", pid 1848, jiffies 4294958994 (age 189.248s)
hex dump (first 32 bytes):
76 db 46 00 00 00 00 00 22 05 73 81 ff ff ff ff v.F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88006516c6e8 (size 96):
comm "runltp", pid 1849, jiffies 4294960010 (age 188.242s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc9001867b000 (size 134217728):
comm "runltp", pid 1849, jiffies 4294960071 (age 188.181s)
hex dump (first 32 bytes):
5d ee 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ].F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88003ed519d0 (size 96):
comm "runltp", pid 1850, jiffies 4294961128 (age 187.124s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc900206bd000 (size 134217728):
comm "runltp", pid 1850, jiffies 4294961186 (age 187.066s)
hex dump (first 32 bytes):
b8 e3 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ..F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88003ed50c10 (size 96):
comm "runltp", pid 1851, jiffies 4294962256 (age 186.004s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc900286ff000 (size 134217728):
comm "runltp", pid 1851, jiffies 4294962315 (age 185.946s)
hex dump (first 32 bytes):
15 af 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ..F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
root@syzkaller:~# cat /sys/kernel/debug/kmemleak
unreferenced object 0xffff8800633f6378 (size 96):
comm "runltp", pid 1847, jiffies 4294957922 (age 362.452s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc90008679000 (size 134217728):
comm "runltp", pid 1847, jiffies 4294957981 (age 362.393s)
hex dump (first 32 bytes):
e0 d2 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ..F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff8800633f6a58 (size 96):
comm "runltp", pid 1848, jiffies 4294958935 (age 361.439s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc9001067a000 (size 134217728):
comm "runltp", pid 1848, jiffies 4294958994 (age 361.380s)
hex dump (first 32 bytes):
76 db 46 00 00 00 00 00 22 05 73 81 ff ff ff ff v.F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88006516c6e8 (size 96):
comm "runltp", pid 1849, jiffies 4294960010 (age 360.369s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc9001867b000 (size 134217728):
comm "runltp", pid 1849, jiffies 4294960071 (age 360.308s)
hex dump (first 32 bytes):
5d ee 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ].F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88003ed519d0 (size 96):
comm "runltp", pid 1850, jiffies 4294961128 (age 359.251s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc900206bd000 (size 134217728):
comm "runltp", pid 1850, jiffies 4294961186 (age 359.193s)
hex dump (first 32 bytes):
b8 e3 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ..F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff88003ed50c10 (size 96):
comm "runltp", pid 1851, jiffies 4294962256 (age 358.132s)
hex dump (first 32 bytes):
01 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
ff ff ff ff 00 00 00 00 ff ff ff ff ff ff ff ff ................
backtrace:
[<ffffffff9acb9ea6>] kmem_cache_alloc_trace+0x146/0x2e0
[<ffffffff9aa13575>] kcov_open+0x25/0x80
[<ffffffff9b28d8f4>] open_proxy_open+0x1e4/0x2b0
[<ffffffff9acd7f52>] do_dentry_open+0x682/0xd70
[<ffffffff9acdbc17>] vfs_open+0x107/0x230
[<ffffffff9ad1edd7>] path_openat+0x1157/0x3520
[<ffffffff9ad2424b>] do_filp_open+0x25b/0x3b0
[<ffffffff9acdcaac>] do_sys_open+0x4fc/0x6c0
[<ffffffff9acdcc9d>] SyS_open+0x2d/0x40
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffffc900286ff000 (size 134217728):
comm "runltp", pid 1851, jiffies 4294962315 (age 358.073s)
hex dump (first 32 bytes):
15 af 46 00 00 00 00 00 22 05 73 81 ff ff ff ff ..F.....".s.....
47 16 73 81 ff ff ff ff 11 16 73 81 ff ff ff ff G.s.......s.....
backtrace:
[<ffffffff9ac67fc7>] __vmalloc_node_range+0x387/0x6a0
[<ffffffff9ac6844c>] vmalloc_user+0x6c/0x140
[<ffffffff9aa1320e>] kcov_mmap+0x2e/0x170
[<ffffffff9ac48ecc>] mmap_region+0xa9c/0x15a0
[<ffffffff9ac4a096>] do_mmap+0x6c6/0xe10
[<ffffffff9abd773e>] vm_mmap_pgoff+0x1de/0x270
[<ffffffff9ac40889>] SyS_mmap_pgoff+0x469/0x610
[<ffffffff9a66d066>] SyS_mmap+0x16/0x20
[<ffffffff9a60733b>] do_syscall_64+0x23b/0x820
[<ffffffff9cf8e01d>] return_from_SYSCALL_64+0x0/0x75
[<ffffffffffffffff>] 0xffffffffffffffff

2018-01-20 12:40:58

by Dmitry Vyukov

Subject: Re: Possible Memory Leak in KCOV Linux 4.15-rc1

On Fri, Jan 19, 2018 at 8:29 PM, Shankara Pailoor <[email protected]> wrote:
> Hi Dmitry,
>
> I added support for kcov in strace and I have been tracing a fairly
> large program but after a little while, I notice that when I mmap a
> new cover buffer, the call fails with ENOMEM. After killing the
> program, I try and rerun and I notice that there is nearly no memory
> on the system. When I do a kmemleak scan I get the following reports:
>
> I believe the problem occurs when I try and setup the kcov buffer
> again after an exec. Instead of reusing the old file descriptor I open
> kcov again within that process. In that case, I don't know what
> happens to the old kcov struct.
>
> I don't see a maintainers list for kcov so I decided to email you
> directly. Let me know what more information I can provide.


Hi Shankara,

Looks bad. Can you provide a reproducer?
We extensively use kcov with syzkaller, but have not observed such
leaks. Also I don't see anything obvious in the code.

Thanks

2018-01-20 15:03:23

by Shankara Pailoor

Subject: Re: Possible Memory Leak in KCOV Linux 4.15-rc1

Hi Dmitry,

I will try and get something to you tomorrow. Just wondering, but what
happens to the old struct kcov if a task opens /sys/kernel/debug/kcov
twice? I am looking here
https://elixir.free-electrons.com/linux/v4.15-rc8/source/kernel/kcov.c#L381
and I don't see where the previous struct would get freed.

Regards,
Shankara

On Sat, Jan 20, 2018 at 4:38 AM, Dmitry Vyukov <[email protected]> wrote:
> On Fri, Jan 19, 2018 at 8:29 PM, Shankara Pailoor <[email protected]> wrote:
>> Hi Dmitry,
>>
>> I added support for kcov in strace and I have been tracing a fairly
>> large program but after a little while, I notice that when I mmap a
>> new cover buffer, the call fails with ENOMEM. After killing the
>> program, I try and rerun and I notice that there is nearly no memory
>> on the system. When I do a kmemleak scan I get the following reports:
>>
>> I believe the problem occurs when I try and setup the kcov buffer
>> again after an exec. Instead of reusing the old file descriptor I open
>> kcov again within that process. In that case, I don't know what
>> happens to the old kcov struct.
>>
>> I don't see a maintainers list for kcov so I decided to email you
>> directly. Let me know what more information I can provide.
>
>
> Hi Shankara,
>
> Looks bad. Can you provide a reproducer?
> We extensively use kcov with syzkaller, but have not observed such
> leaks. Also I don't see anything obvious in the code.
>
> Thanks

2018-01-20 15:08:55

by Dmitry Vyukov

Subject: Re: Possible Memory Leak in KCOV Linux 4.15-rc1

On Sat, Jan 20, 2018 at 4:01 PM, Shankara Pailoor <[email protected]> wrote:
> Hi Dmitry,
>
> I will try and get something to you tomorrow. Just wondering, but what
> happens to the old struct kcov if a task opens /sys/kernel/debug/kcov
> twice? I am looking here
> https://elixir.free-electrons.com/linux/v4.15-rc8/source/kernel/kcov.c#L381
> and I don't see where the previous struct would get freed.

Good question. Perhaps we need something like:

diff --git a/kernel/kcov.c b/kernel/kcov.c
index 7594c033d98a..c76498018500 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -358,7 +358,7 @@ static int kcov_ioctl_locked(struct kcov *kcov,
unsigned int cmd,
*/
if (kcov->mode != KCOV_MODE_INIT || !kcov->area)
return -EINVAL;
- if (kcov->t != NULL)
+ if (kcov->t != NULL || t->kcov != NULL)
return -EBUSY;
if (arg == KCOV_TRACE_PC)
kcov->mode = KCOV_MODE_TRACE_PC;



> On Sat, Jan 20, 2018 at 4:38 AM, Dmitry Vyukov <[email protected]> wrote:
>> On Fri, Jan 19, 2018 at 8:29 PM, Shankara Pailoor <[email protected]> wrote:
>>> Hi Dmitry,
>>>
>>> I added support for kcov in strace and I have been tracing a fairly
>>> large program but after a little while, I notice that when I mmap a
>>> new cover buffer, the call fails with ENOMEM. After killing the
>>> program, I try and rerun and I notice that there is nearly no memory
>>> on the system. When I do a kmemleak scan I get the following reports:
>>>
>>> I believe the problem occurs when I try and setup the kcov buffer
>>> again after an exec. Instead of reusing the old file descriptor I open
>>> kcov again within that process. In that case, I don't know what
>>> happens to the old kcov struct.
>>>
>>> I don't see a maintainers list for kcov so I decided to email you
>>> directly. Let me know what more information I can provide.
>>
>>
>> Hi Shankara,
>>
>> Looks bad. Can you provide a reproducer?
>> We extensively use kcov with syzkaller, but have not observed such
>> leaks. Also I don't see anything obvious in the code.
>>
>> Thanks
>

2018-01-21 09:12:59

by Shankara Pailoor

Subject: Re: Possible Memory Leak in KCOV Linux 4.15-rc1

Hi Dmitry,

The leaks went away when I disabled and closed the old file
descriptors before opening new ones.

The patch you sent wouldn't work because t is not initialized at the
line. This seems to work for me

diff --git a/kernel/kcov.c b/kernel/kcov.c
index 7594c03..1397006 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -371,6 +371,8 @@ static int kcov_ioctl_locked(struct kcov *kcov,
unsigned int cmd,
else
return -EINVAL;
t = current;
+ if (!t->kcov)
+ return -EBUSY;
/* Cache in task struct for performance. */
t->kcov_size = kcov->size;
t->kcov_area = kcov->area;

On Sat, Jan 20, 2018 at 7:06 AM, Dmitry Vyukov <[email protected]> wrote:
> On Sat, Jan 20, 2018 at 4:01 PM, Shankara Pailoor <[email protected]> wrote:
>> Hi Dmitry,
>>
>> I will try and get something to you tomorrow. Just wondering, but what
>> happens to the old struct kcov if a task opens /sys/kernel/debug/kcov
>> twice? I am looking here
>> https://elixir.free-electrons.com/linux/v4.15-rc8/source/kernel/kcov.c#L381
>> and I don't see where the previous struct would get freed.
>
> Good question. Perhaps we need something like:
>
> diff --git a/kernel/kcov.c b/kernel/kcov.c
> index 7594c033d98a..c76498018500 100644
> --- a/kernel/kcov.c
> +++ b/kernel/kcov.c
> @@ -358,7 +358,7 @@ static int kcov_ioctl_locked(struct kcov *kcov,
> unsigned int cmd,
> */
> if (kcov->mode != KCOV_MODE_INIT || !kcov->area)
> return -EINVAL;
> - if (kcov->t != NULL)
> + if (kcov->t != NULL || t->kcov != NULL)
> return -EBUSY;
> if (arg == KCOV_TRACE_PC)
> kcov->mode = KCOV_MODE_TRACE_PC;
>
>
>
>> On Sat, Jan 20, 2018 at 4:38 AM, Dmitry Vyukov <[email protected]> wrote:
>>> On Fri, Jan 19, 2018 at 8:29 PM, Shankara Pailoor <[email protected]> wrote:
>>>> Hi Dmitry,
>>>>
>>>> I added support for kcov in strace and I have been tracing a fairly
>>>> large program but after a little while, I notice that when I mmap a
>>>> new cover buffer, the call fails with ENOMEM. After killing the
>>>> program, I try and rerun and I notice that there is nearly no memory
>>>> on the system. When I do a kmemleak scan I get the following reports:
>>>>
>>>> I believe the problem occurs when I try and setup the kcov buffer
>>>> again after an exec. Instead of reusing the old file descriptor I open
>>>> kcov again within that process. In that case, I don't know what
>>>> happens to the old kcov struct.
>>>>
>>>> I don't see a maintainers list for kcov so I decided to email you
>>>> directly. Let me know what more information I can provide.
>>>
>>>
>>> Hi Shankara,
>>>
>>> Looks bad. Can you provide a reproducer?
>>> We extensively use kcov with syzkaller, but have not observed such
>>> leaks. Also I don't see anything obvious in the code.
>>>
>>> Thanks
>>

2018-01-21 17:54:48

by Shankara Pailoor

Subject: Re: Possible Memory Leak in KCOV Linux 4.15-rc1

Below is a reproducer.

#define _GNU_SOURCE
#include <fcntl.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <unistd.h>

#define KCOV_INIT_TRACE _IOR('c', 1, unsigned long)
#define KCOV_ENABLE _IO('c', 100)
#define KCOV_DISABLE _IO('c', 101)
#define COVER_SIZE (16 << 20)


void kcov_setup() {
unsigned long *cover;
int fd;

fd = open("/sys/kernel/debug/kcov", O_RDWR);
if (fd == -1)
perror("open"), exit(1);
if (ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE))
perror("ioctl"), exit(1);
cover = (unsigned long*)mmap(NULL,
COVER_SIZE * sizeof(unsigned long),
PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if ((void*)cover == MAP_FAILED)
perror("mmap"), exit(1);
if (ioctl(fd, KCOV_ENABLE, 0))
perror("ioctl"), exit(1);

}

void main() {
int i;
for (i = 0; i < 4; i++)
kcov_setup();
sleep(10);
}

On Sun, Jan 21, 2018 at 1:11 AM, Shankara Pailoor <[email protected]> wrote:
> Hi Dmitry,
>
> The leaks went away when I disabled and closed the old file
> descriptors before opening new ones.
>
> The patch you sent wouldn't work because t is not initialized at the
> line. This seems to work for me
>
> diff --git a/kernel/kcov.c b/kernel/kcov.c
> index 7594c03..1397006 100644
> --- a/kernel/kcov.c
> +++ b/kernel/kcov.c
> @@ -371,6 +371,8 @@ static int kcov_ioctl_locked(struct kcov *kcov,
> unsigned int cmd,
> else
> return -EINVAL;
> t = current;
> + if (!t->kcov)
> + return -EBUSY;
> /* Cache in task struct for performance. */
> t->kcov_size = kcov->size;
> t->kcov_area = kcov->area;
>
> On Sat, Jan 20, 2018 at 7:06 AM, Dmitry Vyukov <[email protected]> wrote:
>> On Sat, Jan 20, 2018 at 4:01 PM, Shankara Pailoor <[email protected]> wrote:
>>> Hi Dmitry,
>>>
>>> I will try and get something to you tomorrow. Just wondering, but what
>>> happens to the old struct kcov if a task opens /sys/kernel/debug/kcov
>>> twice? I am looking here
>>> https://elixir.free-electrons.com/linux/v4.15-rc8/source/kernel/kcov.c#L381
>>> and I don't see where the previous struct would get freed.
>>
>> Good question. Perhaps we need something like:
>>
>> diff --git a/kernel/kcov.c b/kernel/kcov.c
>> index 7594c033d98a..c76498018500 100644
>> --- a/kernel/kcov.c
>> +++ b/kernel/kcov.c
>> @@ -358,7 +358,7 @@ static int kcov_ioctl_locked(struct kcov *kcov,
>> unsigned int cmd,
>> */
>> if (kcov->mode != KCOV_MODE_INIT || !kcov->area)
>> return -EINVAL;
>> - if (kcov->t != NULL)
>> + if (kcov->t != NULL || t->kcov != NULL)
>> return -EBUSY;
>> if (arg == KCOV_TRACE_PC)
>> kcov->mode = KCOV_MODE_TRACE_PC;
>>
>>
>>
>>> On Sat, Jan 20, 2018 at 4:38 AM, Dmitry Vyukov <[email protected]> wrote:
>>>> On Fri, Jan 19, 2018 at 8:29 PM, Shankara Pailoor <[email protected]> wrote:
>>>>> Hi Dmitry,
>>>>>
>>>>> I added support for kcov in strace and I have been tracing a fairly
>>>>> large program but after a little while, I notice that when I mmap a
>>>>> new cover buffer, the call fails with ENOMEM. After killing the
>>>>> program, I try and rerun and I notice that there is nearly no memory
>>>>> on the system. When I do a kmemleak scan I get the following reports:
>>>>>
>>>>> I believe the problem occurs when I try and setup the kcov buffer
>>>>> again after an exec. Instead of reusing the old file descriptor I open
>>>>> kcov again within that process. In that case, I don't know what
>>>>> happens to the old kcov struct.
>>>>>
>>>>> I don't see a maintainers list for kcov so I decided to email you
>>>>> directly. Let me know what more information I can provide.
>>>>
>>>>
>>>> Hi Shankara,
>>>>
>>>> Looks bad. Can you provide a reproducer?
>>>> We extensively use kcov with syzkaller, but have not observed such
>>>> leaks. Also I don't see anything obvious in the code.
>>>>
>>>> Thanks
>>>

2018-01-22 08:30:59

by Dmitry Vyukov

Subject: Re: Possible Memory Leak in KCOV Linux 4.15-rc1

On Sun, Jan 21, 2018 at 6:54 PM, Shankara Pailoor <[email protected]> wrote:
> Below is a reproducer.
>
> #define _GNU_SOURCE
> #include <fcntl.h>
> #include <sys/ioctl.h>
> #include <sys/mman.h>
> #include <stdlib.h>
> #include <unistd.h>
>
> #define KCOV_INIT_TRACE _IOR('c', 1, unsigned long)
> #define KCOV_ENABLE _IO('c', 100)
> #define KCOV_DISABLE _IO('c', 101)
> #define COVER_SIZE (16 << 20)
>
>
> void kcov_setup() {
> unsigned long *cover;
> int fd;
>
> fd = open("/sys/kernel/debug/kcov", O_RDWR);
> if (fd == -1)
> perror("open"), exit(1);
> if (ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE))
> perror("ioctl"), exit(1);
> cover = (unsigned long*)mmap(NULL,
> COVER_SIZE * sizeof(unsigned long),
> PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
> if ((void*)cover == MAP_FAILED)
> perror("mmap"), exit(1);
> if (ioctl(fd, KCOV_ENABLE, 0))
> perror("ioctl"), exit(1);
>
> }
>
> void main() {
> int i;
> for (i = 0; i < 4; i++)
> kcov_setup();
> sleep(10);
> }


Thanks!

I've just mailed a fix for this:
https://groups.google.com/d/msg/syzkaller/Hclbq5Elfs8/J5V_FnC4DgAJ

With that patch the second KCOV_ENABLE return EBUSY.
To make it clear, it won't fix your code per se, because it's a misuse
of the API. You need to either call KCOV_DISABLE first, or reuse the
first kcov descriptor and mapping, you can just write 0 to the first
work of the mapping to reset current coverage (which should be much
faster).