High order memory stuff within trace could introduce OOM, use kvmalloc instead.
Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low.
traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=2, oom_score_adj=-1
traced_probes cpuset=system-background mems_allowed=0
CPU: 3 PID: 588 Comm: traced_probes Tainted: G W O 4.14.181 #1
Hardware name: Generic DT based system
(unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
(show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
(dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
(dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
(oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
(out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
(__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
(kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
(kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
(tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
(do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
(vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
(path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
(do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
(do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
(SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)
Signed-off-by: Zhaoyang Huang <[email protected]>
---
changes since v1: change kfree to kvfree
---
kernel/trace/trace.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index ca1ee65..8d70c79 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
if (trace_array_get(tr) < 0)
return -ENODEV;
- info = kzalloc(sizeof(*info), GFP_KERNEL);
+ info = kvmalloc(sizeof(*info), GFP_KERNEL);
if (!info) {
trace_array_put(tr);
return -ENOMEM;
@@ -7017,7 +7017,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
if (info->spare)
ring_buffer_free_read_page(iter->trace_buffer->buffer,
info->spare_cpu, info->spare);
- kfree(info);
+ kvfree(info);
mutex_unlock(&trace_types_lock);
--
1.9.1
On Thu, 30 Jul 2020 19:04:12 +0800
Zhaoyang Huang <[email protected]> wrote:
> High order memory stuff within trace could introduce OOM, use kvmalloc instead.
>
> Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low.
Please limit your column width in the description of patches to 76
characters.
>
> traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=2, oom_score_adj=-1
>
What does this traced_probes thing do?
> traced_probes cpuset=system-background mems_allowed=0
> CPU: 3 PID: 588 Comm: traced_probes Tainted: G W O 4.14.181 #1
> Hardware name: Generic DT based system
> (unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
> (show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
> (dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
> (dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
> (oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
> (out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
> (__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
> (kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
> (kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
> (tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
> (do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
> (vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
> (path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
> (do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
> (do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
> (SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)
>
> Signed-off-by: Zhaoyang Huang <[email protected]>
> ---
> changes since v1: change kfree to kvfree
> ---
> kernel/trace/trace.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index ca1ee65..8d70c79 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
> if (trace_array_get(tr) < 0)
> return -ENODEV;
>
> - info = kzalloc(sizeof(*info), GFP_KERNEL);
> + info = kvmalloc(sizeof(*info), GFP_KERNEL);
The above is a bug. It converts kzalloc() to kvmalloc() instead of
kvzalloc().
-- Steve
> if (!info) {
> trace_array_put(tr);
> return -ENOMEM;
> @@ -7017,7 +7017,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
> if (info->spare)
> ring_buffer_free_read_page(iter->trace_buffer->buffer,
> info->spare_cpu, info->spare);
> - kfree(info);
> + kvfree(info);
>
> mutex_unlock(&trace_types_lock);
>
This is also the second version of the patch. Please add a v2 to the
subject, otherwise it gets confusing to know which one to pull in.
"[PATCH v2] ..."
-- Steve
On Thu, 30 Jul 2020 19:04:12 +0800
Zhaoyang Huang <[email protected]> wrote:
> High order memory stuff within trace could introduce OOM, use kvmalloc instead.
>
> Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low.
>
On Thu, Jul 30, 2020 at 9:58 PM Steven Rostedt <[email protected]> wrote:
>
> On Thu, 30 Jul 2020 19:04:12 +0800
> Zhaoyang Huang <[email protected]> wrote:
>
> > High order memory stuff within trace could introduce OOM, use kvmalloc instead.
> >
> > Please find the bellowing for the call stack we run across in an android system. The scenario happens when traced_probes is woken up to get a large quantity of trace even if free memory is even higher than watermark_low.
>
> Please limit your column width in the description of patches to 76
> characters.
>
> >
> > traced_probes invoked oom-killer: gfp_mask=0x140c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=2, oom_score_adj=-1
> >
>
> What does this traced_probes thing do?
traced_probes is an android exe which reads ftrace and writes them to
files. I think kzalloc works fine for most of context but will cause
OOM in high ratio ftrace access in memory constraint system, like
mobilephone etc.
>
> > traced_probes cpuset=system-background mems_allowed=0
> > CPU: 3 PID: 588 Comm: traced_probes Tainted: G W O 4.14.181 #1
> > Hardware name: Generic DT based system
> > (unwind_backtrace) from [<c010d824>] (show_stack+0x20/0x24)
> > (show_stack) from [<c0b2e174>] (dump_stack+0xa8/0xec)
> > (dump_stack) from [<c027d584>] (dump_header+0x9c/0x220)
> > (dump_header) from [<c027cfe4>] (oom_kill_process+0xc0/0x5c4)
> > (oom_kill_process) from [<c027cb94>] (out_of_memory+0x220/0x310)
> > (out_of_memory) from [<c02816bc>] (__alloc_pages_nodemask+0xff8/0x13a4)
> > (__alloc_pages_nodemask) from [<c02a6a1c>] (kmalloc_order+0x30/0x48)
> > (kmalloc_order) from [<c02a6a64>] (kmalloc_order_trace+0x30/0x118)
> > (kmalloc_order_trace) from [<c0223d7c>] (tracing_buffers_open+0x50/0xfc)
> > (tracing_buffers_open) from [<c02e6f58>] (do_dentry_open+0x278/0x34c)
> > (do_dentry_open) from [<c02e70d0>] (vfs_open+0x50/0x70)
> > (vfs_open) from [<c02f7c24>] (path_openat+0x5fc/0x169c)
> > (path_openat) from [<c02f75c4>] (do_filp_open+0x94/0xf8)
> > (do_filp_open) from [<c02e7650>] (do_sys_open+0x168/0x26c)
> > (do_sys_open) from [<c02e77bc>] (SyS_openat+0x34/0x38)
> > (SyS_openat) from [<c0108bc0>] (ret_fast_syscall+0x0/0x28)
> >
> > Signed-off-by: Zhaoyang Huang <[email protected]>
> > ---
> > changes since v1: change kfree to kvfree
> > ---
> > kernel/trace/trace.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> > index ca1ee65..8d70c79 100644
> > --- a/kernel/trace/trace.c
> > +++ b/kernel/trace/trace.c
> > @@ -6891,7 +6891,7 @@ static int tracing_buffers_open(struct inode *inode, struct file *filp)
> > if (trace_array_get(tr) < 0)
> > return -ENODEV;
> >
> > - info = kzalloc(sizeof(*info), GFP_KERNEL);
> > + info = kvmalloc(sizeof(*info), GFP_KERNEL);
>
> The above is a bug. It converts kzalloc() to kvmalloc() instead of
> kvzalloc().
fixed and resend with patch v2
>
> -- Steve
>
>
>
> > if (!info) {
> > trace_array_put(tr);
> > return -ENOMEM;
> > @@ -7017,7 +7017,7 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)
> > if (info->spare)
> > ring_buffer_free_read_page(iter->trace_buffer->buffer,
> > info->spare_cpu, info->spare);
> > - kfree(info);
> > + kvfree(info);
> >
> > mutex_unlock(&trace_types_lock);
> >
>