2023-03-10 10:05:12

by Tero Kristo

[permalink] [raw]
Subject: [PATCHv2 0/2] trace/hwlat: Prevent startup of multiple per-cpu threads

Hi,

v2 of this series just splits the patch into two, and adds the
reproducer script into the commit of patch #2. Separately, neither of
the patches fix the issue completely, both are needed.

-Tero




2023-03-10 10:05:15

by Tero Kristo

[permalink] [raw]
Subject: [PATCHv2 1/2] trace/hwlat: Do not wipe the contents of per-cpu thread data

Do not wipe the contents of the per-cpu kthread data when starting the
tracer, as this will completely forget about already running instances
and can later start new additional per-cpu threads.

Signed-off-by: Tero Kristo <[email protected]>
---
v2:
* split into separate patch

kernel/trace/trace_hwlat.c | 3 ---
1 file changed, 3 deletions(-)

diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
index d440ddd5fd8b..edc26dc22c3f 100644
--- a/kernel/trace/trace_hwlat.c
+++ b/kernel/trace/trace_hwlat.c
@@ -584,9 +584,6 @@ static int start_per_cpu_kthreads(struct trace_array *tr)
*/
cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);

- for_each_online_cpu(cpu)
- per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
-
for_each_cpu(cpu, current_mask) {
retval = start_cpu_kthread(cpu);
if (retval)
--
2.25.1


2023-03-10 10:05:19

by Tero Kristo

[permalink] [raw]
Subject: [PATCHv2 2/2] trace/hwlat: Do not start per-cpu thread if it is already running

The hwlatd tracer will end up starting multiple per-cpu threads with
the following script:

#!/bin/sh
cd /sys/kernel/debug/tracing
echo 0 > tracing_on
echo hwlat > current_tracer
echo per-cpu > hwlat_detector/mode
echo 100000 > hwlat_detector/width
echo 200000 > hwlat_detector/window
echo 1 > tracing_on

To fix the issue, check if the hwlatd thread for the cpu is already
running, before starting a new one. Along with the previous patch, this
avoids running multiple instances of the same CPU thread on the system.

Signed-off-by: Tero Kristo <[email protected]>
---
v2:
* split into separate patch
* added reproducer script into commit log

kernel/trace/trace_hwlat.c | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/kernel/trace/trace_hwlat.c b/kernel/trace/trace_hwlat.c
index edc26dc22c3f..c4945f8adc11 100644
--- a/kernel/trace/trace_hwlat.c
+++ b/kernel/trace/trace_hwlat.c
@@ -492,6 +492,10 @@ static int start_cpu_kthread(unsigned int cpu)
{
struct task_struct *kthread;

+ /* Do not start a new hwlatd thread if it is already running */
+ if (per_cpu(hwlat_per_cpu_data, cpu).kthread)
+ return 0;
+
kthread = kthread_run_on_cpu(kthread_fn, NULL, cpu, "hwlatd/%u");
if (IS_ERR(kthread)) {
pr_err(BANNER "could not start sampling thread\n");
--
2.25.1


Subject: Re: [PATCHv2 1/2] trace/hwlat: Do not wipe the contents of per-cpu thread data

On 3/10/23 11:04, Tero Kristo wrote:
> Do not wipe the contents of the per-cpu kthread data when starting the
> tracer, as this will completely forget about already running instances
> and can later start new additional per-cpu threads.
>
> Signed-off-by: Tero Kristo <[email protected]>

Acked-by: Daniel Bristot de Oliveira <[email protected]>
-- Daniel

Subject: Re: [PATCHv2 2/2] trace/hwlat: Do not start per-cpu thread if it is already running

On 3/10/23 11:04, Tero Kristo wrote:
> The hwlatd tracer will end up starting multiple per-cpu threads with
> the following script:
>
> #!/bin/sh
> cd /sys/kernel/debug/tracing
> echo 0 > tracing_on
> echo hwlat > current_tracer
> echo per-cpu > hwlat_detector/mode
> echo 100000 > hwlat_detector/width
> echo 200000 > hwlat_detector/window
> echo 1 > tracing_on
>
> To fix the issue, check if the hwlatd thread for the cpu is already
> running, before starting a new one. Along with the previous patch, this
> avoids running multiple instances of the same CPU thread on the system.
>
> Signed-off-by: Tero Kristo <[email protected]>

Acked-by: Daniel Bristot de Oliveira <[email protected]>

-- Daniel