2020-04-14 17:54:13

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH] tracing: Fix the race between registering 'snapshot' event trigger and triggering 'snapshot' operation

On Mon, 13 Apr 2020 15:12:52 +0800
Xiao Yang <[email protected]> wrote:

> Traced event can trigger 'snapshot' operation(i.e. calls snapshot_trigger()
> or snapshot_count_trigger()) when register_snapshot_trigger() has completed
> registration but doesn't allocate spare buffer for 'snapshot' event trigger.
> 'snapshot' operation always detects the lack of allocated buffer in the rare
> case so make register_snapshot_trigger() allocate spare buffer first.
>
> trigger-snapshot.tc in kselftest reproduces the issue on slow vm:
> -----------------------------------------------------------
> cat trace
> ...
> ftracetest-3028 [002] .... 236.784290: sched_process_fork: comm=ftracetest pid=3028 child_comm=ftracetest child_pid=3036
> <...>-2875 [003] .... 240.460335: tracing_snapshot_instance_cond: *** SNAPSHOT NOT ALLOCATED ***
> <...>-2875 [003] .... 240.460338: tracing_snapshot_instance_cond: *** stopping trace here! ***
> -----------------------------------------------------------
>
> Signed-off-by: Xiao Yang <[email protected]>
> ---
> kernel/trace/trace_events_trigger.c | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
> index dd34a1b46a86..00e54cdcef3e 100644
> --- a/kernel/trace/trace_events_trigger.c
> +++ b/kernel/trace/trace_events_trigger.c
> @@ -1088,9 +1088,13 @@ register_snapshot_trigger(char *glob, struct event_trigger_ops *ops,
> struct event_trigger_data *data,
> struct trace_event_file *file)
> {
> - int ret = register_trigger(glob, ops, data, file);
> + int alloc_ret, ret;
>
> - if (ret > 0 && tracing_alloc_snapshot_instance(file->tr) != 0) {
> + alloc_ret = tracing_alloc_snapshot_instance(file->tr);
> +
> + ret = register_trigger(glob, ops, data, file);
> +
> + if (ret > 0 && alloc_ret != 0) {
> unregister_trigger(glob, ops, data, file);
> ret = 0;
> }


Why register if the allocation failed? Just switch the logic:

int ret = tracing_alloc_snapshot_instance(file->tr);

if (ret != 0)
return 0;

return register_trigger(glob, ops, data, file);


-- Steve


2020-04-15 06:49:43

by Xiao Yang

[permalink] [raw]
Subject: Re: [PATCH] tracing: Fix the race between registering 'snapshot' event trigger and triggering 'snapshot' operation

On 2020/4/13 23:02, Steven Rostedt wrote:
> On Mon, 13 Apr 2020 15:12:52 +0800
> Xiao Yang<[email protected]> wrote:
>
>> Traced event can trigger 'snapshot' operation(i.e. calls snapshot_trigger()
>> or snapshot_count_trigger()) when register_snapshot_trigger() has completed
>> registration but doesn't allocate spare buffer for 'snapshot' event trigger.
>> 'snapshot' operation always detects the lack of allocated buffer in the rare
>> case so make register_snapshot_trigger() allocate spare buffer first.
>>
>> trigger-snapshot.tc in kselftest reproduces the issue on slow vm:
>> -----------------------------------------------------------
>> cat trace
>> ...
>> ftracetest-3028 [002] .... 236.784290: sched_process_fork: comm=ftracetest pid=3028 child_comm=ftracetest child_pid=3036
>> <...>-2875 [003] .... 240.460335: tracing_snapshot_instance_cond: *** SNAPSHOT NOT ALLOCATED ***
>> <...>-2875 [003] .... 240.460338: tracing_snapshot_instance_cond: *** stopping trace here! ***
>> -----------------------------------------------------------
>>
>> Signed-off-by: Xiao Yang<[email protected]>
>> ---
>> kernel/trace/trace_events_trigger.c | 8 ++++++--
>> 1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
>> index dd34a1b46a86..00e54cdcef3e 100644
>> --- a/kernel/trace/trace_events_trigger.c
>> +++ b/kernel/trace/trace_events_trigger.c
>> @@ -1088,9 +1088,13 @@ register_snapshot_trigger(char *glob, struct event_trigger_ops *ops,
>> struct event_trigger_data *data,
>> struct trace_event_file *file)
>> {
>> - int ret = register_trigger(glob, ops, data, file);
>> + int alloc_ret, ret;
>>
>> - if (ret> 0&& tracing_alloc_snapshot_instance(file->tr) != 0) {
>> + alloc_ret = tracing_alloc_snapshot_instance(file->tr);
>> +
>> + ret = register_trigger(glob, ops, data, file);
>> +
>> + if (ret> 0&& alloc_ret != 0) {
>> unregister_trigger(glob, ops, data, file);
>> ret = 0;
>> }
>
>
> Why register if the allocation failed? Just switch the logic:
>
> int ret = tracing_alloc_snapshot_instance(file->tr);
>
> if (ret != 0)
> return 0;
>
> return register_trigger(glob, ops, data, file);
Hi Steve,

It looks simpler and better.
I think we can drop the unnecessary ret variable as well.

Thanks,
Xiao Yang
>
>
> -- Steve
>
>
> .
>