2022-11-24 14:17:25

by Yang Jihong

[permalink] [raw]
Subject: [PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

print_trace_line may overflow seq_file buffer. If the event is not
consumed, the while loop keeps peeking this event, causing a infinite loop.

Signed-off-by: Yang Jihong <[email protected]>
---
kernel/trace/trace.c | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index a7fe0e115272..55733224fa88 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -6787,7 +6787,27 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,

ret = print_trace_line(iter);
if (ret == TRACE_TYPE_PARTIAL_LINE) {
- /* don't print partial lines */
+ /*
+ * If one trace_line of the tracer overflows seq_file
+ * buffer, trace_seq_to_user returns -EBUSY.
+ * In this case, we need to consume it, otherwise,
+ * while loop will peek this event next time,
+ * resulting in an infinite loop.
+ */
+ if (trace_seq_has_overflowed(&iter->seq)) {
+ /*
+ * Here we only consider the case that one
+ * print_trace_line() fills the entire trace_seq
+ * in one shot, in that case, iter->seq.seq.len is zero,
+ * we simply output a log of too long line to inform the user.
+ */
+ iter->seq.full = 0;
+ trace_seq_puts(&iter->seq, "[LINE TOO BIG]\n");
+ trace_consume(iter);
+ break;
+ }
+
+ /* In other cases, don't print partial lines */
iter->seq.seq.len = save_len;
break;
}
--
2.30.GIT


2022-11-28 17:06:39

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line

On Thu, 24 Nov 2022 20:58:50 +0800
Yang Jihong <[email protected]> wrote:

> print_trace_line may overflow seq_file buffer. If the event is not
> consumed, the while loop keeps peeking this event, causing a infinite loop.
>
> Signed-off-by: Yang Jihong <[email protected]>
> ---
> kernel/trace/trace.c | 22 +++++++++++++++++++++-
> 1 file changed, 21 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index a7fe0e115272..55733224fa88 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -6787,7 +6787,27 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
>
> ret = print_trace_line(iter);
> if (ret == TRACE_TYPE_PARTIAL_LINE) {
> - /* don't print partial lines */
> + /*
> + * If one trace_line of the tracer overflows seq_file
> + * buffer, trace_seq_to_user returns -EBUSY.
> + * In this case, we need to consume it, otherwise,
> + * while loop will peek this event next time,
> + * resulting in an infinite loop.
> + */
> + if (trace_seq_has_overflowed(&iter->seq)) {

The only way to get here is if the above is true, and that is not going to
cause the infinite loop. What does is if save_len == 0. In fact, that's
all you need to check for:

if (save_len == 0) {

Should do the trick.

-- Steve



> + /*
> + * Here we only consider the case that one
> + * print_trace_line() fills the entire trace_seq
> + * in one shot, in that case, iter->seq.seq.len is zero,
> + * we simply output a log of too long line to inform the user.
> + */
> + iter->seq.full = 0;
> + trace_seq_puts(&iter->seq, "[LINE TOO BIG]\n");
> + trace_consume(iter);
> + break;
> + }
> +
> + /* In other cases, don't print partial lines */
> iter->seq.seq.len = save_len;
> break;
> }

2022-11-29 11:55:15

by Yang Jihong

[permalink] [raw]
Subject: Re: [PATCH v3] tracing: Fix infinite loop in tracing_read_pipe on overflowed print_trace_line



On 2022/11/29 0:46, Steven Rostedt wrote:
> On Thu, 24 Nov 2022 20:58:50 +0800
> Yang Jihong <[email protected]> wrote:
>
>> print_trace_line may overflow seq_file buffer. If the event is not
>> consumed, the while loop keeps peeking this event, causing a infinite loop.
>>
>> Signed-off-by: Yang Jihong <[email protected]>
>> ---
>> kernel/trace/trace.c | 22 +++++++++++++++++++++-
>> 1 file changed, 21 insertions(+), 1 deletion(-)
>>
>> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
>> index a7fe0e115272..55733224fa88 100644
>> --- a/kernel/trace/trace.c
>> +++ b/kernel/trace/trace.c
>> @@ -6787,7 +6787,27 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
>>
>> ret = print_trace_line(iter);
>> if (ret == TRACE_TYPE_PARTIAL_LINE) {
>> - /* don't print partial lines */
>> + /*
>> + * If one trace_line of the tracer overflows seq_file
>> + * buffer, trace_seq_to_user returns -EBUSY.
>> + * In this case, we need to consume it, otherwise,
>> + * while loop will peek this event next time,
>> + * resulting in an infinite loop.
>> + */
>> + if (trace_seq_has_overflowed(&iter->seq)) {
>
> The only way to get here is if the above is true, and that is not going to
> cause the infinite loop. What does is if save_len == 0. In fact, that's
> all you need to check for:
>
> if (save_len == 0) {
>
> Should do the trick.
Yes, Has been tested to be feasible, will change in next version.

By the way, the problem mentioned in the v1 patch:
"Anyway, what is triggering this?"

The problem is caused by the print_line callback function
(blk_tracer_print_line) of blktrace.
The blktrace does not filter out the events whose type is !=TRACK_BLK.
It parses and prints all trace events in the ring buffer as blktrace
events. As a result, the seq_file buffer may overflow in the
blk_log_dump_pdu function:

static void blk_log_dump_pdu()
{
...
for (i = 0; i < pdu_len; i++) { // pdu_len may exceed the
seq_file buffer size.

trace_seq_printf(s, "%s%02x",
i == 0 ? "" : " ", pdu_buf[i]);

/*
* stop when the rest is just zeros and indicate so
* with a ".." appended
*/
if (i == end && end != pdu_len - 1) {
trace_seq_puts(s, " ..) ");
return;
}
}
...
}

The fixed patch has been sent:
https://lore.kernel.org/lkml/[email protected]/T/


Thanks,
Yang