2020-09-24 13:58:59

by Gaurav Kohli

[permalink] [raw]
Subject: [PATCH v1] trace: Fix race in trace_open and buffer resize call

Below race can come, if trace_open and resize of
cpu buffer is running parallely on different cpus
CPUX CPUY
ring_buffer_resize
atomic_read(&buffer->resize_disabled)
tracing_open
tracing_reset_online_cpus
ring_buffer_reset_cpu
rb_reset_cpu
rb_update_pages
remove/insert pages
resetting pointer
This race can cause data abort or some times infinte loop in
rb_remove_pages and rb_insert_pages while checking pages
for sanity.

Signed-off-by: Gaurav Kohli <[email protected]>
Cc: [email protected]
---
Changes since v0:
-Addressed Steven's review comments.

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 93ef0ab..15bf28b 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -4866,6 +4866,9 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
if (!cpumask_test_cpu(cpu, buffer->cpumask))
return;

+ /* prevent another thread from changing buffer sizes */
+ mutex_lock(&buffer->mutex);
+
atomic_inc(&cpu_buffer->resize_disabled);
atomic_inc(&cpu_buffer->record_disabled);

@@ -4876,6 +4879,8 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)

atomic_dec(&cpu_buffer->record_disabled);
atomic_dec(&cpu_buffer->resize_disabled);
+
+ mutex_unlock(&buffer->mutex);
}
EXPORT_SYMBOL_GPL(ring_buffer_reset_cpu);

@@ -4889,6 +4894,9 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
struct ring_buffer_per_cpu *cpu_buffer;
int cpu;

+ /* prevent another thread from changing buffer sizes */
+ mutex_lock(&buffer->mutex);
+
for_each_online_buffer_cpu(buffer, cpu) {
cpu_buffer = buffer->buffers[cpu];

@@ -4907,6 +4915,8 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
atomic_dec(&cpu_buffer->record_disabled);
atomic_dec(&cpu_buffer->resize_disabled);
}
+
+ mutex_unlock(&buffer->mutex);
}

/**
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project


2020-10-05 04:41:20

by Gaurav Kohli

[permalink] [raw]
Subject: Re: [PATCH v1] trace: Fix race in trace_open and buffer resize call

Hi Steven,

please let us know, if below looks good to you or need modifications.

Thanks
Gaurav

On 9/24/2020 7:25 PM, Gaurav Kohli wrote:
> Below race can come, if trace_open and resize of
> cpu buffer is running parallely on different cpus
> CPUX CPUY
> ring_buffer_resize
> atomic_read(&buffer->resize_disabled)
> tracing_open
> tracing_reset_online_cpus
> ring_buffer_reset_cpu
> rb_reset_cpu
> rb_update_pages
> remove/insert pages
> resetting pointer
> This race can cause data abort or some times infinte loop in
> rb_remove_pages and rb_insert_pages while checking pages
> for sanity.
>
> Signed-off-by: Gaurav Kohli <[email protected]>
> Cc: [email protected]
> ---
> Changes since v0:
> -Addressed Steven's review comments.
>
> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 93ef0ab..15bf28b 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c
> @@ -4866,6 +4866,9 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
> if (!cpumask_test_cpu(cpu, buffer->cpumask))
> return;
>
> + /* prevent another thread from changing buffer sizes */
> + mutex_lock(&buffer->mutex);
> +
> atomic_inc(&cpu_buffer->resize_disabled);
> atomic_inc(&cpu_buffer->record_disabled);
>
> @@ -4876,6 +4879,8 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
>
> atomic_dec(&cpu_buffer->record_disabled);
> atomic_dec(&cpu_buffer->resize_disabled);
> +
> + mutex_unlock(&buffer->mutex);
> }
> EXPORT_SYMBOL_GPL(ring_buffer_reset_cpu);
>
> @@ -4889,6 +4894,9 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
> struct ring_buffer_per_cpu *cpu_buffer;
> int cpu;
>
> + /* prevent another thread from changing buffer sizes */
> + mutex_lock(&buffer->mutex);
> +
> for_each_online_buffer_cpu(buffer, cpu) {
> cpu_buffer = buffer->buffers[cpu];
>
> @@ -4907,6 +4915,8 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
> atomic_dec(&cpu_buffer->record_disabled);
> atomic_dec(&cpu_buffer->resize_disabled);
> }
> +
> + mutex_unlock(&buffer->mutex);
> }
>
> /**
>

--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.

2020-10-05 14:29:25

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH v1] trace: Fix race in trace_open and buffer resize call

On Mon, 5 Oct 2020 10:09:34 +0530
Gaurav Kohli <[email protected]> wrote:

> Hi Steven,
>
> please let us know, if below looks good to you or need modifications.

Strange, I don't have your original email in my inbox. I do have it in my
LKML folder, but that's way too big for me to read. I checked my server
logs. I found where I received this from LKML, but there's no log that I
received this directly.

How are you sending out your patches? If it doesn't land in my inbox, I'll
never see it.

-- Steve


2020-10-05 14:31:25

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH v1] trace: Fix race in trace_open and buffer resize call

On Mon, 5 Oct 2020 10:25:15 -0400
Steven Rostedt <[email protected]> wrote:

> On Mon, 5 Oct 2020 10:09:34 +0530
> Gaurav Kohli <[email protected]> wrote:
>
> > Hi Steven,
> >
> > please let us know, if below looks good to you or need modifications.
>
> Strange, I don't have your original email in my inbox. I do have it in my
> LKML folder, but that's way too big for me to read. I checked my server
> logs. I found where I received this from LKML, but there's no log that I
> received this directly.
>
> How are you sending out your patches? If it doesn't land in my inbox, I'll
> never see it.
>

BTW, this is the second time this has happened to me. The first time I
assumed it may have been me accidentally deleting it. Now I'm thinking it's
on your end.

I have yet to received a patch from you directly. Last time I had to pull
it from my LKML folder after you replied to me (letting me know you sent
it).

-- Steve

2020-10-05 17:06:55

by Gaurav Kohli

[permalink] [raw]
Subject: Re: [PATCH v1] trace: Fix race in trace_open and buffer resize call



On 10/5/2020 7:57 PM, Steven Rostedt wrote:
> On Mon, 5 Oct 2020 10:25:15 -0400
> Steven Rostedt <[email protected]> wrote:
>
>> On Mon, 5 Oct 2020 10:09:34 +0530
>> Gaurav Kohli <[email protected]> wrote:
>>
>>> Hi Steven,
>>>
>>> please let us know, if below looks good to you or need modifications.
>>
>> Strange, I don't have your original email in my inbox. I do have it in my
>> LKML folder, but that's way too big for me to read. I checked my server
>> logs. I found where I received this from LKML, but there's no log that I
>> received this directly.
>>
>> How are you sending out your patches? If it doesn't land in my inbox, I'll
>> never see it.
>>
>
> BTW, this is the second time this has happened to me. The first time I
> assumed it may have been me accidentally deleting it. Now I'm thinking it's
> on your end.
>
> I have yet to received a patch from you directly. Last time I had to pull
> it from my LKML folder after you replied to me (letting me know you sent
> it).
>

> -- Steve
>

Hi Steven,

I am using normal git send-email(never saw problem with this), Not sure
what is wrong. In my older mail i have kept you in to and rest in cc.

Let me try to resent it.

--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.

2020-10-05 17:41:06

by Gaurav Kohli

[permalink] [raw]
Subject: Re: [PATCH v1] trace: Fix race in trace_open and buffer resize call



On 10/5/2020 10:02 PM, Steven Rostedt wrote:
> On Mon, 5 Oct 2020 21:59:02 +0530
> Gaurav Kohli <[email protected]> wrote:
>
>> Hi Steven,
>>
>> I am using normal git send-email(never saw problem with this), Not sure
>> what is wrong. In my older mail i have kept you in to and rest in cc.
>>
>> Let me try to resent it.
>
> The Cc is working (I got it in my LKML box), but I don't see any connection
> in my server. Note, the [email protected] is a server at my home.
>
> -- Steve
>

Thanks for update, i will verify my account settings and will send my
patch tomorrow.
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.

2020-10-05 17:55:59

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH v1] trace: Fix race in trace_open and buffer resize call

On Mon, 5 Oct 2020 21:59:02 +0530
Gaurav Kohli <[email protected]> wrote:

> Hi Steven,
>
> I am using normal git send-email(never saw problem with this), Not sure
> what is wrong. In my older mail i have kept you in to and rest in cc.
>
> Let me try to resent it.

The Cc is working (I got it in my LKML box), but I don't see any connection
in my server. Note, the [email protected] is a server at my home.

-- Steve