This is a series of improvements for rtla, mainly as a result of our
daily usage of the tool debugging problems at red hat.
The cgroup support and house keeping options are from our usage
of the tool debugging containers.
The auto-analysis overhead reduction is needed when we go to
large boxes - but it is really hand in practice, as it gives an idea
of the problem without having to look at the trace.
Running hwnoise 100 % of CPU time might cause some systems
to slow down too much. Reduce its utilization to 75% by default to
avoid problems for people using it for the first time.
Finally, it adds support for running timerlat user-space threads,
and to collect the additional field via rtla timerlat top/hist.
Changes from V2:
- Add timerlat hist -u option
- Link: https://lore.kernel.org/lkml/[email protected]/
Changes from V1:
- Add the user-space thread support to rtla timerlat top
- Link: https://lore.kernel.org/lkml/[email protected]/
Daniel Bristot de Oliveira (11):
rtla: Add -C cgroup support
rtla: Add --house-keeping option
rtla: Change monitored_cpus from char * to cpu_set_t
rtla: Automatically move rtla to a house-keeping cpu
rtla/timerlat: Give timerlat auto analysis its own instance
rtla/timerlat_hist: Add auto-analysis support
rtla: Start the tracers after creating all instances
rtla/hwnoise: Reduce runtime to 75%
rtla: Add timerlat user-space support for timerlat top
rtla: Add timerlat user-space support for
Documentation: Add tools/rtla timerlat -u option documentation
Documentation/tools/rtla/common_options.rst | 8 +
.../tools/rtla/common_timerlat_aa.rst | 7 -
.../tools/rtla/common_timerlat_options.rst | 7 +
.../tools/rtla/rtla-timerlat-hist.rst | 7 +-
.../tools/rtla/rtla-timerlat-top.rst | 7 +
tools/tracing/rtla/src/osnoise.c | 65 ++++
tools/tracing/rtla/src/osnoise.h | 5 +
tools/tracing/rtla/src/osnoise_hist.c | 90 ++++-
tools/tracing/rtla/src/osnoise_top.c | 83 ++++-
tools/tracing/rtla/src/timerlat_aa.c | 35 +-
tools/tracing/rtla/src/timerlat_aa.h | 5 +-
tools/tracing/rtla/src/timerlat_hist.c | 262 ++++++++++++--
tools/tracing/rtla/src/timerlat_top.c | 229 +++++++++++--
tools/tracing/rtla/src/timerlat_u.c | 224 ++++++++++++
tools/tracing/rtla/src/timerlat_u.h | 18 +
tools/tracing/rtla/src/utils.c | 324 +++++++++++++++++-
tools/tracing/rtla/src/utils.h | 7 +
17 files changed, 1277 insertions(+), 106 deletions(-)
create mode 100644 tools/tracing/rtla/src/timerlat_u.c
create mode 100644 tools/tracing/rtla/src/timerlat_u.h
--
2.38.1
Add auto-analysis to timerlat hist, including the --no-aa option to
reduce overhead and --dump-task. --aa-only was not added as it is
already on timerlat top.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
.../tools/rtla/common_timerlat_aa.rst | 7 ---
.../tools/rtla/rtla-timerlat-hist.rst | 7 ++-
.../tools/rtla/rtla-timerlat-top.rst | 7 +++
tools/tracing/rtla/src/timerlat_hist.c | 51 ++++++++++++++++++-
4 files changed, 61 insertions(+), 11 deletions(-)
diff --git a/Documentation/tools/rtla/common_timerlat_aa.rst b/Documentation/tools/rtla/common_timerlat_aa.rst
index 795b9fbcbc6d..077029e6b289 100644
--- a/Documentation/tools/rtla/common_timerlat_aa.rst
+++ b/Documentation/tools/rtla/common_timerlat_aa.rst
@@ -5,10 +5,3 @@
**--no-aa**
disable auto-analysis, reducing rtla timerlat cpu usage
-
-**--aa-only** *us*
-
- Set stop tracing conditions and run without collecting and displaying statistics.
- Print the auto-analysis if the system hits the stop tracing condition. This option
- is useful to reduce rtla timerlat CPU, enabling the debug without the overhead of
- collecting the statistics.
diff --git a/Documentation/tools/rtla/rtla-timerlat-hist.rst b/Documentation/tools/rtla/rtla-timerlat-hist.rst
index 6bf7f0ca4556..057db78d4095 100644
--- a/Documentation/tools/rtla/rtla-timerlat-hist.rst
+++ b/Documentation/tools/rtla/rtla-timerlat-hist.rst
@@ -29,15 +29,18 @@ OPTIONS
.. include:: common_options.rst
+.. include:: common_timerlat_aa.rst
+
EXAMPLE
=======
In the example below, **rtla timerlat hist** is set to run for *10* minutes,
in the cpus *0-4*, *skipping zero* only lines. Moreover, **rtla timerlat
hist** will change the priority of the *timerlat* threads to run under
*SCHED_DEADLINE* priority, with a *10us* runtime every *1ms* period. The
-*1ms* period is also passed to the *timerlat* tracer::
+*1ms* period is also passed to the *timerlat* tracer. Auto-analysis is disabled
+to reduce overhead ::
- [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms
+ [root@alien ~]# timerlat hist -d 10m -c 0-4 -P d:100us:1ms -p 1ms --no-aa
# RTLA timerlat histogram
# Time unit is microseconds (us)
# Duration: 0 00:10:00
diff --git a/Documentation/tools/rtla/rtla-timerlat-top.rst b/Documentation/tools/rtla/rtla-timerlat-top.rst
index 73799c1150ad..1b7cf4e3eafe 100644
--- a/Documentation/tools/rtla/rtla-timerlat-top.rst
+++ b/Documentation/tools/rtla/rtla-timerlat-top.rst
@@ -32,6 +32,13 @@ OPTIONS
.. include:: common_timerlat_aa.rst
+**--aa-only** *us*
+
+ Set stop tracing conditions and run without collecting and displaying statistics.
+ Print the auto-analysis if the system hits the stop tracing condition. This option
+ is useful to reduce rtla timerlat CPU, enabling the debug without the overhead of
+ collecting the statistics.
+
EXAMPLE
=======
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 1675d54cae81..e720c8908906 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -16,6 +16,7 @@
#include "utils.h"
#include "osnoise.h"
#include "timerlat.h"
+#include "timerlat_aa.h"
struct timerlat_hist_params {
char *cpus;
@@ -34,6 +35,8 @@ struct timerlat_hist_params {
int dma_latency;
int cgroup;
int hk_cpus;
+ int no_aa;
+ int dump_tasks;
cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
@@ -438,7 +441,7 @@ static void timerlat_hist_usage(char *usage)
" usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\",
" [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
" [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\",
- " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]]",
+ " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]] [--no-aa] [--dump-task]",
"",
" -h/--help: print this menu",
" -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit",
@@ -450,12 +453,14 @@ static void timerlat_hist_usage(char *usage)
" -H/--house-keeping cpus: run rtla control threads only on the given cpus",
" -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
" -d/--duration time[m|h|d]: duration of the session in seconds",
+ " --dump-tasks: prints the task running on all CPUs if stop conditions are met (depends on !--no-aa)",
" -D/--debug: print debug info",
" -t/--trace[=file]: save the stopped trace to [file|timerlat_trace.txt]",
" -e/--event <sys:event>: enable the <sys:event> in the trace instance, multiple -e are allowed",
" --filter <filter>: enable a trace event filter to the previous -e event",
" --trigger <trigger>: enable a trace event trigger to the previous -e event",
" -n/--nano: display data in nanoseconds",
+ " --no-aa: disable auto-analysis, reducing rtla timerlat cpu usage",
" -b/--bucket-size N: set the histogram bucket size (default 1)",
" -E/--entries N: set the number of entries of the histogram (default 256)",
" --no-irq: ignore IRQ latencies",
@@ -537,13 +542,15 @@ static struct timerlat_hist_params
{"trigger", required_argument, 0, '6'},
{"filter", required_argument, 0, '7'},
{"dma-latency", required_argument, 0, '8'},
+ {"no-aa", no_argument, 0, '9'},
+ {"dump-task", no_argument, 0, '\1'},
{0, 0, 0, 0}
};
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:",
+ c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:9\1",
long_options, &option_index);
/* detect the end of the options. */
@@ -556,6 +563,7 @@ static struct timerlat_hist_params
/* set thread stop to auto_thresh */
params->stop_total_us = auto_thresh;
+ params->stop_us = auto_thresh;
/* get stack trace */
params->print_stack = auto_thresh;
@@ -699,6 +707,12 @@ static struct timerlat_hist_params
exit(EXIT_FAILURE);
}
break;
+ case '9':
+ params->no_aa = 1;
+ break;
+ case '\1':
+ params->dump_tasks = 1;
+ break;
default:
timerlat_hist_usage("Invalid option");
}
@@ -715,6 +729,12 @@ static struct timerlat_hist_params
if (params->no_index && !params->with_zeros)
timerlat_hist_usage("no-index set with with-zeros is not set - it does not make sense");
+ /*
+ * Auto analysis only happens if stop tracing, thus:
+ */
+ if (!params->stop_us && !params->stop_total_us)
+ params->no_aa = 1;
+
return params;
}
@@ -848,6 +868,7 @@ int timerlat_hist_main(int argc, char *argv[])
struct timerlat_hist_params *params;
struct osnoise_tool *record = NULL;
struct osnoise_tool *tool = NULL;
+ struct osnoise_tool *aa = NULL;
struct trace_instance *trace;
int dma_latency_fd = -1;
int return_value = 1;
@@ -919,6 +940,26 @@ int timerlat_hist_main(int argc, char *argv[])
trace_instance_start(&record->trace);
}
+ if (!params->no_aa) {
+ aa = osnoise_init_tool("timerlat_aa");
+ if (!aa)
+ goto out_hist;
+
+ retval = timerlat_aa_init(aa, params->dump_tasks);
+ if (retval) {
+ err_msg("Failed to enable the auto analysis instance\n");
+ goto out_hist;
+ }
+
+ retval = enable_timerlat(&aa->trace);
+ if (retval) {
+ err_msg("Failed to enable timerlat tracer\n");
+ goto out_hist;
+ }
+
+ trace_instance_start(&aa->trace);
+ }
+
tool->start_time = time(NULL);
timerlat_hist_set_signals(params);
@@ -946,6 +987,10 @@ int timerlat_hist_main(int argc, char *argv[])
if (trace_is_off(&tool->trace, &record->trace)) {
printf("rtla timerlat hit stop tracing\n");
+
+ if (!params->no_aa)
+ timerlat_auto_analysis(params->stop_us, params->stop_total_us);
+
if (params->trace_output) {
printf(" Saving trace to %s\n", params->trace_output);
save_trace_to_file(record->trace.inst, params->trace_output);
@@ -953,12 +998,14 @@ int timerlat_hist_main(int argc, char *argv[])
}
out_hist:
+ timerlat_aa_destroy();
if (dma_latency_fd >= 0)
close(dma_latency_fd);
trace_events_destroy(&record->trace, params->events);
params->events = NULL;
out_free:
timerlat_free_histogram(tool->data);
+ osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(tool);
free(params);
--
2.38.1
Use a cpumask instead of a char *, reducing memory footprint and code.
No functional change, and in preparation for auto house-keeping.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/osnoise_hist.c | 18 ++++----
tools/tracing/rtla/src/osnoise_top.c | 6 +--
tools/tracing/rtla/src/timerlat_hist.c | 18 ++++----
tools/tracing/rtla/src/timerlat_top.c | 6 +--
tools/tracing/rtla/src/utils.c | 63 --------------------------
5 files changed, 24 insertions(+), 87 deletions(-)
diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c
index d2b68177ffac..b616a72d5c0a 100644
--- a/tools/tracing/rtla/src/osnoise_hist.c
+++ b/tools/tracing/rtla/src/osnoise_hist.c
@@ -19,7 +19,7 @@
struct osnoise_hist_params {
char *cpus;
- char *monitored_cpus;
+ cpu_set_t monitored_cpus;
char *trace_output;
char *cgroup_name;
unsigned long long runtime;
@@ -274,7 +274,7 @@ static void osnoise_hist_header(struct osnoise_tool *tool)
trace_seq_printf(s, "Index");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -305,7 +305,7 @@ osnoise_print_summary(struct osnoise_hist_params *params,
trace_seq_printf(trace->seq, "count:");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -319,7 +319,7 @@ osnoise_print_summary(struct osnoise_hist_params *params,
trace_seq_printf(trace->seq, "min: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -334,7 +334,7 @@ osnoise_print_summary(struct osnoise_hist_params *params,
trace_seq_printf(trace->seq, "avg: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -352,7 +352,7 @@ osnoise_print_summary(struct osnoise_hist_params *params,
trace_seq_printf(trace->seq, "max: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -387,7 +387,7 @@ osnoise_print_stats(struct osnoise_hist_params *params, struct osnoise_tool *too
bucket * data->bucket_size);
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -411,7 +411,7 @@ osnoise_print_stats(struct osnoise_hist_params *params, struct osnoise_tool *too
trace_seq_printf(trace->seq, "over: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].count)
@@ -559,7 +559,7 @@ static struct osnoise_hist_params
osnoise_hist_usage("Bucket size needs to be > 0 and <= 1000000\n");
break;
case 'c':
- retval = parse_cpu_list(optarg, ¶ms->monitored_cpus);
+ retval = parse_cpu_set(optarg, ¶ms->monitored_cpus);
if (retval)
osnoise_hist_usage("\nInvalid -c cpu list\n");
params->cpus = optarg;
diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c
index fcf6c14ce1bc..0e2f3b216d34 100644
--- a/tools/tracing/rtla/src/osnoise_top.c
+++ b/tools/tracing/rtla/src/osnoise_top.c
@@ -26,7 +26,7 @@ enum osnoise_mode {
*/
struct osnoise_top_params {
char *cpus;
- char *monitored_cpus;
+ cpu_set_t monitored_cpus;
char *trace_output;
char *cgroup_name;
unsigned long long runtime;
@@ -263,7 +263,7 @@ osnoise_print_stats(struct osnoise_top_params *params, struct osnoise_tool *top)
osnoise_top_header(top);
for (i = 0; i < nr_cpus; i++) {
- if (params->cpus && !params->monitored_cpus[i])
+ if (params->cpus && !CPU_ISSET(i, ¶ms->monitored_cpus))
continue;
osnoise_top_print(top, i);
}
@@ -397,7 +397,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
break;
case 'c':
- retval = parse_cpu_list(optarg, ¶ms->monitored_cpus);
+ retval = parse_cpu_set(optarg, ¶ms->monitored_cpus);
if (retval)
osnoise_top_usage(params, "\nInvalid -c cpu list\n");
params->cpus = optarg;
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index d48c05d238f9..00287e96f22e 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -19,7 +19,7 @@
struct timerlat_hist_params {
char *cpus;
- char *monitored_cpus;
+ cpu_set_t monitored_cpus;
char *trace_output;
char *cgroup_name;
unsigned long long runtime;
@@ -227,7 +227,7 @@ static void timerlat_hist_header(struct osnoise_tool *tool)
trace_seq_printf(s, "Index");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -263,7 +263,7 @@ timerlat_print_summary(struct timerlat_hist_params *params,
trace_seq_printf(trace->seq, "count:");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -283,7 +283,7 @@ timerlat_print_summary(struct timerlat_hist_params *params,
trace_seq_printf(trace->seq, "min: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -303,7 +303,7 @@ timerlat_print_summary(struct timerlat_hist_params *params,
trace_seq_printf(trace->seq, "avg: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -331,7 +331,7 @@ timerlat_print_summary(struct timerlat_hist_params *params,
trace_seq_printf(trace->seq, "max: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -371,7 +371,7 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t
bucket * data->bucket_size);
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -405,7 +405,7 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t
trace_seq_printf(trace->seq, "over: ");
for (cpu = 0; cpu < data->nr_cpus; cpu++) {
- if (params->cpus && !params->monitored_cpus[cpu])
+ if (params->cpus && !CPU_ISSET(cpu, ¶ms->monitored_cpus))
continue;
if (!data->hist[cpu].irq_count && !data->hist[cpu].thread_count)
@@ -565,7 +565,7 @@ static struct timerlat_hist_params
break;
case 'c':
- retval = parse_cpu_list(optarg, ¶ms->monitored_cpus);
+ retval = parse_cpu_set(optarg, ¶ms->monitored_cpus);
if (retval)
timerlat_hist_usage("\nInvalid -c cpu list\n");
params->cpus = optarg;
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 5395d1c5921e..920f2f6ef842 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -21,7 +21,7 @@
struct timerlat_top_params {
char *cpus;
- char *monitored_cpus;
+ cpu_set_t monitored_cpus;
char *trace_output;
char *cgroup_name;
unsigned long long runtime;
@@ -271,7 +271,7 @@ timerlat_print_stats(struct timerlat_top_params *params, struct osnoise_tool *to
timerlat_top_header(top);
for (i = 0; i < nr_cpus; i++) {
- if (params->cpus && !params->monitored_cpus[i])
+ if (params->cpus && !CPU_ISSET(i, ¶ms->monitored_cpus))
continue;
timerlat_top_print(top, i);
}
@@ -422,7 +422,7 @@ static struct timerlat_top_params
params->aa_only = 1;
break;
case 'c':
- retval = parse_cpu_list(optarg, ¶ms->monitored_cpus);
+ retval = parse_cpu_set(optarg, ¶ms->monitored_cpus);
if (retval)
timerlat_top_usage("\nInvalid -c cpu list\n");
params->cpus = optarg;
diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c
index ee6fab09acae..8f9ad8f01e0f 100644
--- a/tools/tracing/rtla/src/utils.c
+++ b/tools/tracing/rtla/src/utils.c
@@ -88,69 +88,6 @@ void get_duration(time_t start_time, char *output, int output_size)
tm_info->tm_sec);
}
-/*
- * parse_cpu_list - parse a cpu_list filling a char vector with cpus set
- *
- * Receives a cpu list, like 1-3,5 (cpus 1, 2, 3, 5), and then set the char
- * in the monitored_cpus.
- *
- * XXX: convert to a bitmask.
- */
-int parse_cpu_list(char *cpu_list, char **monitored_cpus)
-{
- char *mon_cpus;
- const char *p;
- int end_cpu;
- int nr_cpus;
- int cpu;
- int i;
-
- nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
-
- mon_cpus = calloc(nr_cpus, sizeof(char));
- if (!mon_cpus)
- goto err;
-
- for (p = cpu_list; *p; ) {
- cpu = atoi(p);
- if (cpu < 0 || (!cpu && *p != '0') || cpu >= nr_cpus)
- goto err;
-
- while (isdigit(*p))
- p++;
- if (*p == '-') {
- p++;
- end_cpu = atoi(p);
- if (end_cpu < cpu || (!end_cpu && *p != '0') || end_cpu >= nr_cpus)
- goto err;
- while (isdigit(*p))
- p++;
- } else
- end_cpu = cpu;
-
- if (cpu == end_cpu) {
- debug_msg("cpu_list: adding cpu %d\n", cpu);
- mon_cpus[cpu] = 1;
- } else {
- for (i = cpu; i <= end_cpu; i++) {
- debug_msg("cpu_list: adding cpu %d\n", i);
- mon_cpus[i] = 1;
- }
- }
-
- if (*p == ',')
- p++;
- }
-
- *monitored_cpus = mon_cpus;
-
- return 0;
-
-err:
- debug_msg("Error parsing the cpu list %s", cpu_list);
- return 1;
-}
-
/*
* parse_cpu_set - parse a cpu_list filling cpu_set_t argument
*
--
2.38.1
osnoise runs 100% of time by default. It makes sense because osnoise
is preemptive. hwnoise checks preemption once a second, so it
reduces system progress.
Reduce runtime to 75% to avoid problems by default. I added a Fixes
as it might avoid problems for first time users as it lands on distros.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Fixes: 1f428356c38d ("rtla: Add hwnoise tool")
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/osnoise_top.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c
index 85abba568faf..f7c959be8677 100644
--- a/tools/tracing/rtla/src/osnoise_top.c
+++ b/tools/tracing/rtla/src/osnoise_top.c
@@ -348,8 +348,14 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
if (!params)
exit(1);
- if (strcmp(argv[0], "hwnoise") == 0)
+ if (strcmp(argv[0], "hwnoise") == 0) {
params->mode = MODE_HWNOISE;
+ /*
+ * Reduce CPU usage for 75% to avoid killing the system.
+ */
+ params->runtime = 750000;
+ params->period = 1000000;
+ }
while (1) {
static struct option long_options[] = {
--
2.38.1
Add the support for running timerlat threads in user-space. In this
mode, enabled with -u/--user-threads, timerlat dispatches user-space
processes that will loop in the timerlat_fd, measuring the overhead
for going to user-space and then returning to the kernel - in addition
to the existing measurements.
Here is one example of the tool's output with -u enabled:
$ sudo timerlat hist -u -c 1-3 -d 600
# RTLA timerlat histogram
# Time unit is microseconds (us)
# Duration: 0 00:10:01
Index IRQ-001 Thr-001 Usr-001 IRQ-002 Thr-002 Usr-002 IRQ-003 Thr-003 Usr-003
0 477555 0 0 425287 0 0 474357 0 0
1 122385 7998 0 174616 1921 0 125412 3138 0
2 47 587376 492150 89 594717 447830 147 593463 454872
3 11 2549 101930 7 2682 145580 64 2530 138680
4 3 1954 2833 1 463 4917 11 548 4656
5 0 60 1037 0 138 1117 6 179 1130
6 0 26 1837 0 38 277 1 76 339
7 0 15 143 0 28 147 2 37 156
8 0 10 23 0 11 75 0 12 80
9 0 7 17 0 0 26 0 11 42
10 0 2 11 0 0 18 0 2 20
11 0 0 7 0 1 8 0 2 12
12 0 0 6 0 1 4 0 2 8
13 0 1 3 0 0 0 0 0 1
14 0 1 0 0 0 1 0 0 2
15 0 1 0 0 0 0 0 0 2
16 0 1 2 0 0 0 0 0 0
17 0 0 1 0 0 0 0 0 0
19 0 0 1 0 0 0 0 0 0
over: 0 0 0 0 0 0 0 0 0
count: 600001 600001 600001 600000 600000 600000 600000 600000 600000
min: 0 1 2 0 1 2 0 1 2
avg: 0 1 2 0 2 2 0 2 2
max: 4 16 19 4 12 14 7 12 15
The tuning setup like -p or -C work for the user-space threads as well.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/timerlat_hist.c | 129 ++++++++++++++++++++++---
1 file changed, 118 insertions(+), 11 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index f431cf43246c..47d3d8b53cb2 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -12,11 +12,13 @@
#include <stdio.h>
#include <time.h>
#include <sched.h>
+#include <pthread.h>
#include "utils.h"
#include "osnoise.h"
#include "timerlat.h"
#include "timerlat_aa.h"
+#include "timerlat_u.h"
struct timerlat_hist_params {
char *cpus;
@@ -37,6 +39,7 @@ struct timerlat_hist_params {
int hk_cpus;
int no_aa;
int dump_tasks;
+ int user_hist;
cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
@@ -53,9 +56,11 @@ struct timerlat_hist_params {
struct timerlat_hist_cpu {
int *irq;
int *thread;
+ int *user;
int irq_count;
int thread_count;
+ int user_count;
unsigned long long min_irq;
unsigned long long sum_irq;
@@ -64,6 +69,10 @@ struct timerlat_hist_cpu {
unsigned long long min_thread;
unsigned long long sum_thread;
unsigned long long max_thread;
+
+ unsigned long long min_user;
+ unsigned long long sum_user;
+ unsigned long long max_user;
};
struct timerlat_hist_data {
@@ -88,6 +97,10 @@ timerlat_free_histogram(struct timerlat_hist_data *data)
if (data->hist[cpu].thread)
free(data->hist[cpu].thread);
+
+ if (data->hist[cpu].user)
+ free(data->hist[cpu].user);
+
}
/* one set of histograms per CPU */
@@ -124,15 +137,21 @@ static struct timerlat_hist_data
data->hist[cpu].irq = calloc(1, sizeof(*data->hist->irq) * (entries + 1));
if (!data->hist[cpu].irq)
goto cleanup;
+
data->hist[cpu].thread = calloc(1, sizeof(*data->hist->thread) * (entries + 1));
if (!data->hist[cpu].thread)
goto cleanup;
+
+ data->hist[cpu].user = calloc(1, sizeof(*data->hist->user) * (entries + 1));
+ if (!data->hist[cpu].user)
+ goto cleanup;
}
/* set the min to max */
for (cpu = 0; cpu < nr_cpus; cpu++) {
data->hist[cpu].min_irq = ~0;
data->hist[cpu].min_thread = ~0;
+ data->hist[cpu].min_user = ~0;
}
return data;
@@ -147,7 +166,7 @@ static struct timerlat_hist_data
*/
static void
timerlat_hist_update(struct osnoise_tool *tool, int cpu,
- unsigned long long thread,
+ unsigned long long context,
unsigned long long latency)
{
struct timerlat_hist_params *params = tool->params;
@@ -162,18 +181,24 @@ timerlat_hist_update(struct osnoise_tool *tool, int cpu,
if (data->bucket_size)
bucket = latency / data->bucket_size;
- if (!thread) {
+ if (!context) {
hist = data->hist[cpu].irq;
data->hist[cpu].irq_count++;
update_min(&data->hist[cpu].min_irq, &latency);
update_sum(&data->hist[cpu].sum_irq, &latency);
update_max(&data->hist[cpu].max_irq, &latency);
- } else {
+ } else if (context == 1) {
hist = data->hist[cpu].thread;
data->hist[cpu].thread_count++;
update_min(&data->hist[cpu].min_thread, &latency);
update_sum(&data->hist[cpu].sum_thread, &latency);
update_max(&data->hist[cpu].max_thread, &latency);
+ } else { /* user */
+ hist = data->hist[cpu].user;
+ data->hist[cpu].user_count++;
+ update_min(&data->hist[cpu].min_user, &latency);
+ update_sum(&data->hist[cpu].sum_user, &latency);
+ update_max(&data->hist[cpu].max_user, &latency);
}
if (bucket < entries)
@@ -190,16 +215,16 @@ timerlat_hist_handler(struct trace_seq *s, struct tep_record *record,
struct tep_event *event, void *data)
{
struct trace_instance *trace = data;
- unsigned long long thread, latency;
+ unsigned long long context, latency;
struct osnoise_tool *tool;
int cpu = record->cpu;
tool = container_of(trace, struct osnoise_tool, trace);
- tep_get_field_val(s, event, "context", record, &thread, 1);
+ tep_get_field_val(s, event, "context", record, &context, 1);
tep_get_field_val(s, event, "timer_latency", record, &latency, 1);
- timerlat_hist_update(tool, cpu, thread, latency);
+ timerlat_hist_update(tool, cpu, context, latency);
return 0;
}
@@ -241,6 +266,9 @@ static void timerlat_hist_header(struct osnoise_tool *tool)
if (!params->no_thread)
trace_seq_printf(s, " Thr-%03d", cpu);
+
+ if (params->user_hist)
+ trace_seq_printf(s, " Usr-%03d", cpu);
}
trace_seq_printf(s, "\n");
@@ -279,6 +307,10 @@ timerlat_print_summary(struct timerlat_hist_params *params,
if (!params->no_thread)
trace_seq_printf(trace->seq, "%9d ",
data->hist[cpu].thread_count);
+
+ if (params->user_hist)
+ trace_seq_printf(trace->seq, "%9d ",
+ data->hist[cpu].user_count);
}
trace_seq_printf(trace->seq, "\n");
@@ -299,6 +331,10 @@ timerlat_print_summary(struct timerlat_hist_params *params,
if (!params->no_thread)
trace_seq_printf(trace->seq, "%9llu ",
data->hist[cpu].min_thread);
+
+ if (params->user_hist)
+ trace_seq_printf(trace->seq, "%9llu ",
+ data->hist[cpu].min_user);
}
trace_seq_printf(trace->seq, "\n");
@@ -323,7 +359,15 @@ timerlat_print_summary(struct timerlat_hist_params *params,
if (!params->no_thread) {
if (data->hist[cpu].thread_count)
trace_seq_printf(trace->seq, "%9llu ",
- data->hist[cpu].sum_thread / data->hist[cpu].thread_count);
+ data->hist[cpu].sum_thread / data->hist[cpu].thread_count);
+ else
+ trace_seq_printf(trace->seq, " - ");
+ }
+
+ if (params->user_hist) {
+ if (data->hist[cpu].user_count)
+ trace_seq_printf(trace->seq, "%9llu ",
+ data->hist[cpu].sum_user / data->hist[cpu].user_count);
else
trace_seq_printf(trace->seq, " - ");
}
@@ -347,6 +391,10 @@ timerlat_print_summary(struct timerlat_hist_params *params,
if (!params->no_thread)
trace_seq_printf(trace->seq, "%9llu ",
data->hist[cpu].max_thread);
+
+ if (params->user_hist)
+ trace_seq_printf(trace->seq, "%9llu ",
+ data->hist[cpu].max_user);
}
trace_seq_printf(trace->seq, "\n");
trace_seq_do_printf(trace->seq);
@@ -392,6 +440,12 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t
data->hist[cpu].thread[bucket]);
}
+ if (params->user_hist) {
+ total += data->hist[cpu].user[bucket];
+ trace_seq_printf(trace->seq, "%9d ",
+ data->hist[cpu].user[bucket]);
+ }
+
}
if (total == 0 && !params->with_zeros) {
@@ -421,6 +475,10 @@ timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *t
if (!params->no_thread)
trace_seq_printf(trace->seq, "%9d ",
data->hist[cpu].thread[data->entries]);
+
+ if (params->user_hist)
+ trace_seq_printf(trace->seq, "%9d ",
+ data->hist[cpu].user[data->entries]);
}
trace_seq_printf(trace->seq, "\n");
trace_seq_do_printf(trace->seq);
@@ -441,7 +499,7 @@ static void timerlat_hist_usage(char *usage)
" usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\",
" [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
" [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\",
- " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]] [--no-aa] [--dump-task]",
+ " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]] [--no-aa] [--dump-task] [-u]",
"",
" -h/--help: print this menu",
" -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit",
@@ -476,6 +534,7 @@ static void timerlat_hist_usage(char *usage)
" f:prio - use SCHED_FIFO with prio",
" d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
" in nanoseconds",
+ " -u/--user-threads: use rtla user-space threads instead of in-kernel timerlat threads",
NULL,
};
@@ -532,6 +591,7 @@ static struct timerlat_hist_params
{"stack", required_argument, 0, 's'},
{"thread", required_argument, 0, 'T'},
{"trace", optional_argument, 0, 't'},
+ {"user-threads", no_argument, 0, 'u'},
{"event", required_argument, 0, 'e'},
{"no-irq", no_argument, 0, '0'},
{"no-thread", no_argument, 0, '1'},
@@ -550,7 +610,7 @@ static struct timerlat_hist_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:9\1",
+ c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:u0123456:7:8:9\1",
long_options, &option_index);
/* detect the end of the options. */
@@ -660,6 +720,9 @@ static struct timerlat_hist_params
else
params->trace_output = "timerlat_trace.txt";
break;
+ case 'u':
+ params->user_hist = 1;
+ break;
case '0': /* no irq */
params->no_irq = 1;
break;
@@ -744,7 +807,7 @@ static struct timerlat_hist_params
static int
timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_params *params)
{
- int retval;
+ int retval, i;
if (!params->sleep_time)
params->sleep_time = 1;
@@ -755,6 +818,9 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param
err_msg("Failed to apply CPUs config\n");
goto out_err;
}
+ } else {
+ for (i = 0; i < sysconf(_SC_NPROCESSORS_CONF); i++)
+ CPU_SET(i, ¶ms->monitored_cpus);
}
if (params->stop_us) {
@@ -807,6 +873,14 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param
auto_house_keeping(¶ms->monitored_cpus);
}
+ if (params->user_hist) {
+ retval = osnoise_set_workload(tool->context, 0);
+ if (retval) {
+ err_msg("Failed to set OSNOISE_WORKLOAD option\n");
+ goto out_err;
+ }
+ }
+
return 0;
out_err:
@@ -867,11 +941,13 @@ int timerlat_hist_main(int argc, char *argv[])
{
struct timerlat_hist_params *params;
struct osnoise_tool *record = NULL;
+ struct timerlat_u_params params_u;
struct osnoise_tool *tool = NULL;
struct osnoise_tool *aa = NULL;
struct trace_instance *trace;
int dma_latency_fd = -1;
int return_value = 1;
+ pthread_t timerlat_u;
int retval;
params = timerlat_hist_parse_args(argc, argv);
@@ -906,7 +982,7 @@ int timerlat_hist_main(int argc, char *argv[])
}
}
- if (params->cgroup) {
+ if (params->cgroup && !params->user_hist) {
retval = set_comm_cgroup("timerlat/", params->cgroup_name);
if (!retval) {
err_msg("Failed to move threads to cgroup\n");
@@ -970,6 +1046,25 @@ int timerlat_hist_main(int argc, char *argv[])
tool->start_time = time(NULL);
timerlat_hist_set_signals(params);
+ if (params->user_hist) {
+ /* rtla asked to stop */
+ params_u.should_run = 1;
+ /* all threads left */
+ params_u.stopped_running = 0;
+
+ params_u.set = ¶ms->monitored_cpus;
+ if (params->set_sched)
+ params_u.sched_param = ¶ms->sched_param;
+ else
+ params_u.sched_param = NULL;
+
+ params_u.cgroup_name = params->cgroup_name;
+
+ retval = pthread_create(&timerlat_u, NULL, timerlat_u_dispatcher, ¶ms_u);
+ if (retval)
+ err_msg("Error creating timerlat user-space threads\n");
+ }
+
while (!stop_tracing) {
sleep(params->sleep_time);
@@ -986,6 +1081,18 @@ int timerlat_hist_main(int argc, char *argv[])
if (trace_is_off(&tool->trace, &record->trace))
break;
+
+ /* is there still any user-threads ? */
+ if (params->user_hist) {
+ if (params_u.stopped_running) {
+ debug_msg("timerlat user-space threads stopped!\n");
+ break;
+ }
+ }
+ }
+ if (params->user_hist && !params_u.stopped_running) {
+ params_u.should_run = 0;
+ sleep(1);
}
timerlat_print_stats(params, tool);
--
2.38.1
Currently, the auto-analysis is attached to the timerlat top instance.
The idea was to avoid creating another instance just for that, so one
instance could be reused.
The drawback is that, by doing so, the auto-analysis run for the entire
session, consuming CPU time. On my 24 box CPUs for timerlat with a 100
us period consumed 50 % with auto analysis, but only 16 % without.
By creating an instance for auto-analysis, we can keep the processing
stopped until a stop tracing condition is hit. Once it happens,
timerlat auto-analysis can use its own trace instance to parse only
the end of the trace.
By doing so, auto-analysis stop consuming cpu time when it is not
needed.
If the --aa-only is passed, the timerlat top instance is reused for
auto analysis.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/timerlat_aa.c | 35 +++++++++++++++++--
tools/tracing/rtla/src/timerlat_aa.h | 5 +--
tools/tracing/rtla/src/timerlat_top.c | 48 ++++++++++++++++++---------
3 files changed, 67 insertions(+), 21 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat_aa.c b/tools/tracing/rtla/src/timerlat_aa.c
index 1843fff66da5..e0ffe69c271c 100644
--- a/tools/tracing/rtla/src/timerlat_aa.c
+++ b/tools/tracing/rtla/src/timerlat_aa.c
@@ -8,6 +8,7 @@
#include "utils.h"
#include "osnoise.h"
#include "timerlat.h"
+#include <unistd.h>
enum timelat_state {
TIMERLAT_INIT = 0,
@@ -233,7 +234,7 @@ static int timerlat_aa_thread_latency(struct timerlat_aa_data *taa_data,
*
* Returns 0 on success, -1 otherwise.
*/
-int timerlat_aa_handler(struct trace_seq *s, struct tep_record *record,
+static int timerlat_aa_handler(struct trace_seq *s, struct tep_record *record,
struct tep_event *event, void *context)
{
struct timerlat_aa_context *taa_ctx = timerlat_aa_get_ctx();
@@ -665,6 +666,25 @@ static void timerlat_thread_analysis(struct timerlat_aa_data *taa_data, int cpu,
ns_to_usf(total));
}
+static int timerlat_auto_analysis_collect_trace(struct timerlat_aa_context *taa_ctx)
+{
+ struct trace_instance *trace = &taa_ctx->tool->trace;
+ int retval;
+
+ retval = tracefs_iterate_raw_events(trace->tep,
+ trace->inst,
+ NULL,
+ 0,
+ collect_registered_events,
+ trace);
+ if (retval < 0) {
+ err_msg("Error iterating on events\n");
+ return 0;
+ }
+
+ return 1;
+}
+
/**
* timerlat_auto_analysis - Analyze the collected data
*/
@@ -677,6 +697,8 @@ void timerlat_auto_analysis(int irq_thresh, int thread_thresh)
struct tep_handle *tep;
int cpu;
+ timerlat_auto_analysis_collect_trace(taa_ctx);
+
/* bring stop tracing to the ns scale */
irq_thresh = irq_thresh * 1000;
thread_thresh = thread_thresh * 1000;
@@ -838,6 +860,10 @@ static int timerlat_aa_init_seqs(struct timerlat_aa_context *taa_ctx)
*/
static void timerlat_aa_unregister_events(struct osnoise_tool *tool, int dump_tasks)
{
+
+ tep_unregister_event_handler(tool->trace.tep, -1, "ftrace", "timerlat",
+ timerlat_aa_handler, tool);
+
tracefs_event_disable(tool->trace.inst, "osnoise", NULL);
tep_unregister_event_handler(tool->trace.tep, -1, "osnoise", "nmi_noise",
@@ -875,6 +901,10 @@ static int timerlat_aa_register_events(struct osnoise_tool *tool, int dump_tasks
{
int retval;
+ tep_register_event_handler(tool->trace.tep, -1, "ftrace", "timerlat",
+ timerlat_aa_handler, tool);
+
+
/*
* register auto-analysis handlers.
*/
@@ -955,8 +985,9 @@ void timerlat_aa_destroy(void)
*
* Returns 0 on success, -1 otherwise.
*/
-int timerlat_aa_init(struct osnoise_tool *tool, int nr_cpus, int dump_tasks)
+int timerlat_aa_init(struct osnoise_tool *tool, int dump_tasks)
{
+ int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
struct timerlat_aa_context *taa_ctx;
int retval;
diff --git a/tools/tracing/rtla/src/timerlat_aa.h b/tools/tracing/rtla/src/timerlat_aa.h
index d4f6ca7e342a..cea4bb1531a8 100644
--- a/tools/tracing/rtla/src/timerlat_aa.h
+++ b/tools/tracing/rtla/src/timerlat_aa.h
@@ -3,10 +3,7 @@
* Copyright (C) 2023 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
*/
-int timerlat_aa_init(struct osnoise_tool *tool, int nr_cpus, int dump_task);
+int timerlat_aa_init(struct osnoise_tool *tool, int dump_task);
void timerlat_aa_destroy(void);
-int timerlat_aa_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context);
-
void timerlat_auto_analysis(int irq_thresh, int thread_thresh);
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index f0c6d9735e2a..d6b5a382569e 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -156,9 +156,6 @@ timerlat_top_handler(struct trace_seq *s, struct tep_record *record,
timerlat_top_update(top, cpu, thread, latency);
}
- if (!params->no_aa)
- timerlat_aa_handler(s, record, event, context);
-
return 0;
}
@@ -644,7 +641,6 @@ static struct osnoise_tool
{
struct osnoise_tool *top;
int nr_cpus;
- int retval;
nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
@@ -661,16 +657,6 @@ static struct osnoise_tool
tep_register_event_handler(top->trace.tep, -1, "ftrace", "timerlat",
timerlat_top_handler, top);
- /*
- * If no auto analysis, we are ready.
- */
- if (params->no_aa)
- return top;
-
- retval = timerlat_aa_init(top, nr_cpus, params->dump_tasks);
- if (retval)
- goto out_err;
-
return top;
out_err:
@@ -702,6 +688,7 @@ int timerlat_top_main(int argc, char *argv[])
struct timerlat_top_params *params;
struct osnoise_tool *record = NULL;
struct osnoise_tool *top = NULL;
+ struct osnoise_tool *aa = NULL;
struct trace_instance *trace;
int dma_latency_fd = -1;
int return_value = 1;
@@ -774,6 +761,35 @@ int timerlat_top_main(int argc, char *argv[])
trace_instance_start(&record->trace);
}
+ if (!params->no_aa) {
+ if (params->aa_only) {
+ /* as top is not used for display, use it for aa */
+ aa = top;
+ } else {
+ /* otherwise, a new instance is needed */
+ aa = osnoise_init_tool("timerlat_aa");
+ if (!aa)
+ goto out_top;
+ }
+
+ retval = timerlat_aa_init(aa, params->dump_tasks);
+ if (retval) {
+ err_msg("Failed to enable the auto analysis instance\n");
+ goto out_top;
+ }
+
+ /* if it is re-using the main instance, there is no need to start it */
+ if (aa != top) {
+ retval = enable_timerlat(&aa->trace);
+ if (retval) {
+ err_msg("Failed to enable timerlat tracer\n");
+ goto out_top;
+ }
+
+ trace_instance_start(&aa->trace);
+ }
+ }
+
top->start_time = time(NULL);
timerlat_top_set_signals(params);
@@ -829,13 +845,15 @@ int timerlat_top_main(int argc, char *argv[])
}
out_top:
+ timerlat_aa_destroy();
if (dma_latency_fd >= 0)
close(dma_latency_fd);
trace_events_destroy(&record->trace, params->events);
params->events = NULL;
out_free:
timerlat_free_top(top->data);
- timerlat_aa_destroy();
+ if (aa && aa != top)
+ osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(top);
free(params);
--
2.38.1
When the user sets -c <cpu-list> try to move rtla out of the <cpu-list>,
even without an -H option. This is useful to avoid having rtla
interfering with the workload.
This works by removing <cpu-list> from rtla's current affinity.
If rtla fails to move itself away it is not that of a problem as this
is an automatic measure.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/osnoise_hist.c | 9 +++++
tools/tracing/rtla/src/osnoise_top.c | 9 +++++
tools/tracing/rtla/src/timerlat_hist.c | 9 +++++
tools/tracing/rtla/src/timerlat_top.c | 9 +++++
tools/tracing/rtla/src/utils.c | 50 ++++++++++++++++++++++++++
tools/tracing/rtla/src/utils.h | 1 +
6 files changed, 87 insertions(+)
diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c
index b616a72d5c0a..dfbcb5ca7ecb 100644
--- a/tools/tracing/rtla/src/osnoise_hist.c
+++ b/tools/tracing/rtla/src/osnoise_hist.c
@@ -753,6 +753,15 @@ osnoise_hist_apply_config(struct osnoise_tool *tool, struct osnoise_hist_params
err_msg("Failed to set rtla to the house keeping CPUs\n");
goto out_err;
}
+ } else if (params->cpus) {
+ /*
+ * Even if the user do not set a house-keeping CPU, try to
+ * move rtla to a CPU set different to the one where the user
+ * set the workload to run.
+ *
+ * No need to check results as this is an automatic attempt.
+ */
+ auto_house_keeping(¶ms->monitored_cpus);
}
return 0;
diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c
index 0e2f3b216d34..0833537bb2eb 100644
--- a/tools/tracing/rtla/src/osnoise_top.c
+++ b/tools/tracing/rtla/src/osnoise_top.c
@@ -582,6 +582,15 @@ osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *p
err_msg("Failed to set rtla to the house keeping CPUs\n");
goto out_err;
}
+ } else if (params->cpus) {
+ /*
+ * Even if the user do not set a house-keeping CPU, try to
+ * move rtla to a CPU set different to the one where the user
+ * set the workload to run.
+ *
+ * No need to check results as this is an automatic attempt.
+ */
+ auto_house_keeping(¶ms->monitored_cpus);
}
return 0;
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 00287e96f22e..1675d54cae81 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -776,6 +776,15 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param
err_msg("Failed to set rtla to the house keeping CPUs\n");
goto out_err;
}
+ } else if (params->cpus) {
+ /*
+ * Even if the user do not set a house-keeping CPU, try to
+ * move rtla to a CPU set different to the one where the user
+ * set the workload to run.
+ *
+ * No need to check results as this is an automatic attempt.
+ */
+ auto_house_keeping(¶ms->monitored_cpus);
}
return 0;
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 920f2f6ef842..f0c6d9735e2a 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -619,6 +619,15 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *
err_msg("Failed to set rtla to the house keeping CPUs\n");
goto out_err;
}
+ } else if (params->cpus) {
+ /*
+ * Even if the user do not set a house-keeping CPU, try to
+ * move rtla to a CPU set different to the one where the user
+ * set the workload to run.
+ *
+ * No need to check results as this is an automatic attempt.
+ */
+ auto_house_keeping(¶ms->monitored_cpus);
}
return 0;
diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c
index 8f9ad8f01e0f..3e25f0277fb9 100644
--- a/tools/tracing/rtla/src/utils.c
+++ b/tools/tracing/rtla/src/utils.c
@@ -709,3 +709,53 @@ int set_comm_cgroup(const char *comm_prefix, const char *cgroup)
close(cg_fd);
return 0;
}
+
+/**
+ * auto_house_keeping - Automatically move rtla out of measurement threads
+ *
+ * Try to move rtla away from the tracer, if possible.
+ *
+ * Returns 1 on success, 0 otherwise.
+ */
+int auto_house_keeping(cpu_set_t *monitored_cpus)
+{
+ cpu_set_t rtla_cpus, house_keeping_cpus;
+ int retval;
+
+ /* first get the CPUs in which rtla can actually run. */
+ retval = sched_getaffinity(getpid(), sizeof(rtla_cpus), &rtla_cpus);
+ if (retval == -1) {
+ debug_msg("Could not get rtla affinity, rtla might run with the threads!\n");
+ return 0;
+ }
+
+ /* then check if the existing setup is already good. */
+ CPU_AND(&house_keeping_cpus, &rtla_cpus, monitored_cpus);
+ if (!CPU_COUNT(&house_keeping_cpus)) {
+ debug_msg("rtla and the monitored CPUs do not share CPUs.");
+ debug_msg("Skipping auto house-keeping\n");
+ return 1;
+ }
+
+ /* remove the intersection */
+ CPU_XOR(&house_keeping_cpus, &rtla_cpus, monitored_cpus);
+
+ /* get only those that rtla can run */
+ CPU_AND(&house_keeping_cpus, &house_keeping_cpus, &rtla_cpus);
+
+ /* is there any cpu left? */
+ if (!CPU_COUNT(&house_keeping_cpus)) {
+ debug_msg("Could not find any CPU for auto house-keeping\n");
+ return 0;
+ }
+
+ retval = sched_setaffinity(getpid(), sizeof(house_keeping_cpus), &house_keeping_cpus);
+ if (retval == -1) {
+ debug_msg("Could not set affinity for auto house-keeping\n");
+ return 0;
+ }
+
+ debug_msg("rtla automatically moved to an auto house-keeping cpu set\n");
+
+ return 1;
+}
diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h
index 9ab2f0d7bc1c..dec59163cfbc 100644
--- a/tools/tracing/rtla/src/utils.h
+++ b/tools/tracing/rtla/src/utils.h
@@ -60,6 +60,7 @@ int parse_cpu_set(char *cpu_list, cpu_set_t *set);
int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr);
int set_comm_cgroup(const char *comm_prefix, const char *cgroup);
int set_cpu_dma_latency(int32_t latency);
+int auto_house_keeping(cpu_set_t *monitored_cpus);
#define ns_to_usf(x) (((double)x/1000))
#define ns_to_per(total, part) ((part * 100) / (double)total)
--
2.38.1
To avoid having rtla interfering with the measurement threads, add an
option for the user to set the CPUs in which rtla should run. For
instance:
# rtla timerlat top -H 0 -C 1-7
Will place rtla in the CPU 0, while running the measurement threads in
the CPU 1-7.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Suggested-by: Juri Lelli <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
Documentation/tools/rtla/common_options.rst | 4 ++
tools/tracing/rtla/src/osnoise_hist.c | 29 +++++++++--
tools/tracing/rtla/src/osnoise_top.c | 27 +++++++++-
tools/tracing/rtla/src/timerlat_hist.c | 27 +++++++++-
tools/tracing/rtla/src/timerlat_top.c | 27 +++++++++-
tools/tracing/rtla/src/utils.c | 58 +++++++++++++++++++++
tools/tracing/rtla/src/utils.h | 3 ++
7 files changed, 166 insertions(+), 9 deletions(-)
diff --git a/Documentation/tools/rtla/common_options.rst b/Documentation/tools/rtla/common_options.rst
index ede07359d93c..aeb91ff3bd68 100644
--- a/Documentation/tools/rtla/common_options.rst
+++ b/Documentation/tools/rtla/common_options.rst
@@ -2,6 +2,10 @@
Set the osnoise tracer to run the sample threads in the cpu-list.
+**-H**, **--house-keeping** *cpu-list*
+
+ Run rtla control threads only on the given cpu-list.
+
**-d**, **--duration** *time[s|m|h|d]*
Set the duration of the session.
diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c
index 076f4c6af3dd..d2b68177ffac 100644
--- a/tools/tracing/rtla/src/osnoise_hist.c
+++ b/tools/tracing/rtla/src/osnoise_hist.c
@@ -3,6 +3,7 @@
* Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
*/
+#define _GNU_SOURCE
#include <getopt.h>
#include <stdlib.h>
#include <string.h>
@@ -11,6 +12,7 @@
#include <errno.h>
#include <stdio.h>
#include <time.h>
+#include <sched.h>
#include "utils.h"
#include "osnoise.h"
@@ -30,6 +32,8 @@ struct osnoise_hist_params {
int set_sched;
int output_divisor;
int cgroup;
+ int hk_cpus;
+ cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
@@ -434,8 +438,8 @@ static void osnoise_hist_usage(char *usage)
"",
" usage: rtla osnoise hist [-h] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\",
" [-T us] [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] \\",
- " [-c cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] [--no-index] \\",
- " [--with-zeros] [-C[=cgroup_name]]",
+ " [-c cpu-list] [-H cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] \\",
+ " [--no-index] [--with-zeros] [-C[=cgroup_name]]",
"",
" -h/--help: print this menu",
" -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit",
@@ -445,6 +449,7 @@ static void osnoise_hist_usage(char *usage)
" -S/--stop-total us: stop trace if the total sample is higher than the argument in us",
" -T/--threshold us: the minimum delta to be considered a noise",
" -c/--cpus cpu-list: list of cpus to run osnoise threads",
+ " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
" -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
" -d/--duration time[s|m|h|d]: duration of the session",
" -D/--debug: print debug info",
@@ -507,6 +512,7 @@ static struct osnoise_hist_params
{"cgroup", optional_argument, 0, 'C'},
{"debug", no_argument, 0, 'D'},
{"duration", required_argument, 0, 'd'},
+ {"house-keeping", required_argument, 0, 'H'},
{"help", no_argument, 0, 'h'},
{"period", required_argument, 0, 'p'},
{"priority", required_argument, 0, 'P'},
@@ -528,7 +534,7 @@ static struct osnoise_hist_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhp:P:r:s:S:t::T:01234:5:",
+ c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:p:P:r:s:S:t::T:01234:5:",
long_options, &option_index);
/* detect the end of the options. */
@@ -597,6 +603,14 @@ static struct osnoise_hist_params
case '?':
osnoise_hist_usage(NULL);
break;
+ case 'H':
+ params->hk_cpus = 1;
+ retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
+ if (retval) {
+ err_msg("Error parsing house keeping CPUs\n");
+ exit(EXIT_FAILURE);
+ }
+ break;
case 'p':
params->period = get_llong_from_str(optarg);
if (params->period > 10000000)
@@ -732,6 +746,15 @@ osnoise_hist_apply_config(struct osnoise_tool *tool, struct osnoise_hist_params
}
}
+ if (params->hk_cpus) {
+ retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
+ ¶ms->hk_cpu_set);
+ if (retval == -1) {
+ err_msg("Failed to set rtla to the house keeping CPUs\n");
+ goto out_err;
+ }
+ }
+
return 0;
out_err:
diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c
index 139d8d392540..fcf6c14ce1bc 100644
--- a/tools/tracing/rtla/src/osnoise_top.c
+++ b/tools/tracing/rtla/src/osnoise_top.c
@@ -3,6 +3,7 @@
* Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
*/
+#define _GNU_SOURCE
#include <getopt.h>
#include <stdlib.h>
#include <string.h>
@@ -10,6 +11,7 @@
#include <unistd.h>
#include <stdio.h>
#include <time.h>
+#include <sched.h>
#include "osnoise.h"
#include "utils.h"
@@ -37,6 +39,8 @@ struct osnoise_top_params {
int quiet;
int set_sched;
int cgroup;
+ int hk_cpus;
+ cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
enum osnoise_mode mode;
@@ -278,7 +282,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage)
static const char * const msg[] = {
" [-h] [-q] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\",
" [-T us] [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] \\",
- " [-c cpu-list] [-P priority] [-C[=cgroup_name]]",
+ " [-c cpu-list] [-H cpu-list] [-P priority] [-C[=cgroup_name]]",
"",
" -h/--help: print this menu",
" -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit",
@@ -288,6 +292,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage)
" -S/--stop-total us: stop trace if the total sample is higher than the argument in us",
" -T/--threshold us: the minimum delta to be considered a noise",
" -c/--cpus cpu-list: list of cpus to run osnoise threads",
+ " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
" -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
" -d/--duration time[s|m|h|d]: duration of the session",
" -D/--debug: print debug info",
@@ -354,6 +359,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
{"debug", no_argument, 0, 'D'},
{"duration", required_argument, 0, 'd'},
{"event", required_argument, 0, 'e'},
+ {"house-keeping", required_argument, 0, 'H'},
{"help", no_argument, 0, 'h'},
{"period", required_argument, 0, 'p'},
{"priority", required_argument, 0, 'P'},
@@ -371,7 +377,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::d:De:hp:P:qr:s:S:t::T:0:1:",
+ c = getopt_long(argc, argv, "a:c:C::d:De:hH:p:P:qr:s:S:t::T:0:1:",
long_options, &option_index);
/* Detect the end of the options. */
@@ -430,6 +436,14 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
case '?':
osnoise_top_usage(params, NULL);
break;
+ case 'H':
+ params->hk_cpus = 1;
+ retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
+ if (retval) {
+ err_msg("Error parsing house keeping CPUs\n");
+ exit(EXIT_FAILURE);
+ }
+ break;
case 'p':
params->period = get_llong_from_str(optarg);
if (params->period > 10000000)
@@ -561,6 +575,15 @@ osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *p
}
}
+ if (params->hk_cpus) {
+ retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
+ ¶ms->hk_cpu_set);
+ if (retval == -1) {
+ err_msg("Failed to set rtla to the house keeping CPUs\n");
+ goto out_err;
+ }
+ }
+
return 0;
out_err:
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 459c159923e8..d48c05d238f9 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -3,6 +3,7 @@
* Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
*/
+#define _GNU_SOURCE
#include <getopt.h>
#include <stdlib.h>
#include <string.h>
@@ -10,6 +11,7 @@
#include <unistd.h>
#include <stdio.h>
#include <time.h>
+#include <sched.h>
#include "utils.h"
#include "osnoise.h"
@@ -31,6 +33,8 @@ struct timerlat_hist_params {
int set_sched;
int dma_latency;
int cgroup;
+ int hk_cpus;
+ cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
char no_irq;
@@ -432,7 +436,7 @@ static void timerlat_hist_usage(char *usage)
char *msg[] = {
"",
" usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\",
- " [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] \\",
+ " [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
" [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\",
" [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]]",
"",
@@ -443,6 +447,7 @@ static void timerlat_hist_usage(char *usage)
" -T/--thread us: stop trace if the thread latency is higher than the argument in us",
" -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
" -c/--cpus cpus: run the tracer only on the given cpus",
+ " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
" -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
" -d/--duration time[m|h|d]: duration of the session in seconds",
" -D/--debug: print debug info",
@@ -513,6 +518,7 @@ static struct timerlat_hist_params
{"debug", no_argument, 0, 'D'},
{"entries", required_argument, 0, 'E'},
{"duration", required_argument, 0, 'd'},
+ {"house-keeping", required_argument, 0, 'H'},
{"help", no_argument, 0, 'h'},
{"irq", required_argument, 0, 'i'},
{"nano", no_argument, 0, 'n'},
@@ -537,7 +543,7 @@ static struct timerlat_hist_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhi:np:P:s:t::T:0123456:7:8:",
+ c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:",
long_options, &option_index);
/* detect the end of the options. */
@@ -608,6 +614,14 @@ static struct timerlat_hist_params
case '?':
timerlat_hist_usage(NULL);
break;
+ case 'H':
+ params->hk_cpus = 1;
+ retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
+ if (retval) {
+ err_msg("Error parsing house keeping CPUs\n");
+ exit(EXIT_FAILURE);
+ }
+ break;
case 'i':
params->stop_us = get_llong_from_str(optarg);
break;
@@ -755,6 +769,15 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param
}
}
+ if (params->hk_cpus) {
+ retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
+ ¶ms->hk_cpu_set);
+ if (retval == -1) {
+ err_msg("Failed to set rtla to the house keeping CPUs\n");
+ goto out_err;
+ }
+ }
+
return 0;
out_err:
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index a19cbc2aa1f4..5395d1c5921e 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -3,6 +3,7 @@
* Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
*/
+#define _GNU_SOURCE
#include <getopt.h>
#include <stdlib.h>
#include <string.h>
@@ -11,6 +12,7 @@
#include <stdio.h>
#include <time.h>
#include <errno.h>
+#include <sched.h>
#include "utils.h"
#include "osnoise.h"
@@ -37,6 +39,8 @@ struct timerlat_top_params {
int aa_only;
int dump_tasks;
int cgroup;
+ int hk_cpus;
+ cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
};
@@ -286,7 +290,7 @@ static void timerlat_top_usage(char *usage)
static const char *const msg[] = {
"",
" usage: rtla timerlat [top] [-h] [-q] [-a us] [-d s] [-D] [-n] [-p us] [-i us] [-T us] [-s us] \\",
- " [[-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] \\",
+ " [[-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
" [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]]",
"",
" -h/--help: print this menu",
@@ -297,6 +301,7 @@ static void timerlat_top_usage(char *usage)
" -T/--thread us: stop trace if the thread latency is higher than the argument in us",
" -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
" -c/--cpus cpus: run the tracer only on the given cpus",
+ " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
" -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
" -d/--duration time[m|h|d]: duration of the session in seconds",
" -D/--debug: print debug info",
@@ -360,6 +365,7 @@ static struct timerlat_top_params
{"duration", required_argument, 0, 'd'},
{"event", required_argument, 0, 'e'},
{"help", no_argument, 0, 'h'},
+ {"house-keeping", required_argument, 0, 'H'},
{"irq", required_argument, 0, 'i'},
{"nano", no_argument, 0, 'n'},
{"period", required_argument, 0, 'p'},
@@ -380,7 +386,7 @@ static struct timerlat_top_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::d:De:hi:np:P:qs:t::T:0:1:2:345:",
+ c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:0:1:2:345:",
long_options, &option_index);
/* detect the end of the options. */
@@ -454,6 +460,14 @@ static struct timerlat_top_params
case '?':
timerlat_top_usage(NULL);
break;
+ case 'H':
+ params->hk_cpus = 1;
+ retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
+ if (retval) {
+ err_msg("Error parsing house keeping CPUs\n");
+ exit(EXIT_FAILURE);
+ }
+ break;
case 'i':
params->stop_us = get_llong_from_str(optarg);
break;
@@ -598,6 +612,15 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *
}
}
+ if (params->hk_cpus) {
+ retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
+ ¶ms->hk_cpu_set);
+ if (retval == -1) {
+ err_msg("Failed to set rtla to the house keeping CPUs\n");
+ goto out_err;
+ }
+ }
+
return 0;
out_err:
diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c
index bcc0a9f39cfe..ee6fab09acae 100644
--- a/tools/tracing/rtla/src/utils.c
+++ b/tools/tracing/rtla/src/utils.c
@@ -3,6 +3,7 @@
* Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
*/
+#define _GNU_SOURCE
#include <dirent.h>
#include <stdarg.h>
#include <stdlib.h>
@@ -150,6 +151,63 @@ int parse_cpu_list(char *cpu_list, char **monitored_cpus)
return 1;
}
+/*
+ * parse_cpu_set - parse a cpu_list filling cpu_set_t argument
+ *
+ * Receives a cpu list, like 1-3,5 (cpus 1, 2, 3, 5), and then set
+ * filling cpu_set_t argument.
+ *
+ * Returns 1 on success, 0 otherwise.
+ */
+int parse_cpu_set(char *cpu_list, cpu_set_t *set)
+{
+ const char *p;
+ int end_cpu;
+ int nr_cpus;
+ int cpu;
+ int i;
+
+ CPU_ZERO(set);
+
+ nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+
+ for (p = cpu_list; *p; ) {
+ cpu = atoi(p);
+ if (cpu < 0 || (!cpu && *p != '0') || cpu >= nr_cpus)
+ goto err;
+
+ while (isdigit(*p))
+ p++;
+ if (*p == '-') {
+ p++;
+ end_cpu = atoi(p);
+ if (end_cpu < cpu || (!end_cpu && *p != '0') || end_cpu >= nr_cpus)
+ goto err;
+ while (isdigit(*p))
+ p++;
+ } else
+ end_cpu = cpu;
+
+ if (cpu == end_cpu) {
+ debug_msg("cpu_set: adding cpu %d\n", cpu);
+ CPU_SET(cpu, set);
+ } else {
+ for (i = cpu; i <= end_cpu; i++) {
+ debug_msg("cpu_set: adding cpu %d\n", i);
+ CPU_SET(i, set);
+ }
+ }
+
+ if (*p == ',')
+ p++;
+ }
+
+ return 0;
+err:
+ debug_msg("Error parsing the cpu set %s\n", cpu_list);
+ return 1;
+}
+
/*
* parse_duration - parse duration with s/m/h/d suffix converting it to seconds
*/
diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h
index 42b6f099d10a..9ab2f0d7bc1c 100644
--- a/tools/tracing/rtla/src/utils.h
+++ b/tools/tracing/rtla/src/utils.h
@@ -1,6 +1,8 @@
// SPDX-License-Identifier: GPL-2.0
+
#include <stdint.h>
#include <time.h>
+#include <sched.h>
/*
* '18446744073709551615\0'
@@ -54,6 +56,7 @@ struct sched_attr {
};
int parse_prio(char *arg, struct sched_attr *sched_param);
+int parse_cpu_set(char *cpu_list, cpu_set_t *set);
int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr);
int set_comm_cgroup(const char *comm_prefix, const char *cgroup);
int set_cpu_dma_latency(int32_t latency);
--
2.38.1
Add the -u/--user-thread option documentation for timerlat top/hist.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
Documentation/tools/rtla/common_timerlat_options.rst | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/Documentation/tools/rtla/common_timerlat_options.rst b/Documentation/tools/rtla/common_timerlat_options.rst
index bacdea6de7a3..88506b397c2d 100644
--- a/Documentation/tools/rtla/common_timerlat_options.rst
+++ b/Documentation/tools/rtla/common_timerlat_options.rst
@@ -26,3 +26,10 @@
Set the /dev/cpu_dma_latency to *us*, aiming to bound exit from idle latencies.
*cyclictest* sets this value to *0* by default, use **--dma-latency** *0* to have
similar results.
+
+**-u**, **--user-threads**
+
+ Set timerlat to run without a workload, and then dispatches user-space workloads
+ to wait on the timerlat_fd. Once the workload is awakes, it goes to sleep again
+ adding so the measurement for the kernel-to-user and user-to-kernel to the tracer
+ output.
--
2.38.1
Group all start tracing after finishing creating all instances.
The tracing instance starts first for the case of hitting a stop
tracing while enabling other instances. The trace instance is the
one with most valuable information.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/osnoise_hist.c | 14 +++++++++++---
tools/tracing/rtla/src/osnoise_top.c | 13 ++++++++++---
tools/tracing/rtla/src/timerlat_hist.c | 17 ++++++++++++-----
tools/tracing/rtla/src/timerlat_top.c | 19 +++++++++++++------
4 files changed, 46 insertions(+), 17 deletions(-)
diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c
index dfbcb5ca7ecb..8f81fa007364 100644
--- a/tools/tracing/rtla/src/osnoise_hist.c
+++ b/tools/tracing/rtla/src/osnoise_hist.c
@@ -870,8 +870,6 @@ int osnoise_hist_main(int argc, char *argv[])
}
}
- trace_instance_start(trace);
-
if (params->trace_output) {
record = osnoise_init_trace_tool("osnoise");
if (!record) {
@@ -885,9 +883,19 @@ int osnoise_hist_main(int argc, char *argv[])
goto out_hist;
}
- trace_instance_start(&record->trace);
}
+ /*
+ * Start the tracer here, after having set all instances.
+ *
+ * Let the trace instance start first for the case of hitting a stop
+ * tracing while enabling other instances. The trace instance is the
+ * one with most valuable information.
+ */
+ if (params->trace_output)
+ trace_instance_start(&record->trace);
+ trace_instance_start(trace);
+
tool->start_time = time(NULL);
osnoise_hist_set_signals(params);
diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c
index 0833537bb2eb..85abba568faf 100644
--- a/tools/tracing/rtla/src/osnoise_top.c
+++ b/tools/tracing/rtla/src/osnoise_top.c
@@ -697,8 +697,6 @@ int osnoise_top_main(int argc, char **argv)
}
}
- trace_instance_start(trace);
-
if (params->trace_output) {
record = osnoise_init_trace_tool("osnoise");
if (!record) {
@@ -711,9 +709,18 @@ int osnoise_top_main(int argc, char **argv)
if (retval)
goto out_top;
}
+ }
+ /*
+ * Start the tracer here, after having set all instances.
+ *
+ * Let the trace instance start first for the case of hitting a stop
+ * tracing while enabling other instances. The trace instance is the
+ * one with most valuable information.
+ */
+ if (params->trace_output)
trace_instance_start(&record->trace);
- }
+ trace_instance_start(trace);
tool->start_time = time(NULL);
osnoise_top_set_signals(params);
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index e720c8908906..f431cf43246c 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -922,8 +922,6 @@ int timerlat_hist_main(int argc, char *argv[])
}
}
- trace_instance_start(trace);
-
if (params->trace_output) {
record = osnoise_init_trace_tool("timerlat");
if (!record) {
@@ -936,8 +934,6 @@ int timerlat_hist_main(int argc, char *argv[])
if (retval)
goto out_hist;
}
-
- trace_instance_start(&record->trace);
}
if (!params->no_aa) {
@@ -956,9 +952,20 @@ int timerlat_hist_main(int argc, char *argv[])
err_msg("Failed to enable timerlat tracer\n");
goto out_hist;
}
+ }
+ /*
+ * Start the tracers here, after having set all instances.
+ *
+ * Let the trace instance start first for the case of hitting a stop
+ * tracing while enabling other instances. The trace instance is the
+ * one with most valuable information.
+ */
+ if (params->trace_output)
+ trace_instance_start(&record->trace);
+ if (!params->no_aa)
trace_instance_start(&aa->trace);
- }
+ trace_instance_start(trace);
tool->start_time = time(NULL);
timerlat_hist_set_signals(params);
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index d6b5a382569e..02cff4948981 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -743,8 +743,6 @@ int timerlat_top_main(int argc, char *argv[])
}
}
- trace_instance_start(trace);
-
if (params->trace_output) {
record = osnoise_init_trace_tool("timerlat");
if (!record) {
@@ -757,8 +755,6 @@ int timerlat_top_main(int argc, char *argv[])
if (retval)
goto out_top;
}
-
- trace_instance_start(&record->trace);
}
if (!params->no_aa) {
@@ -785,11 +781,22 @@ int timerlat_top_main(int argc, char *argv[])
err_msg("Failed to enable timerlat tracer\n");
goto out_top;
}
-
- trace_instance_start(&aa->trace);
}
}
+ /*
+ * Start the tracers here, after having set all instances.
+ *
+ * Let the trace instance start first for the case of hitting a stop
+ * tracing while enabling other instances. The trace instance is the
+ * one with most valuable information.
+ */
+ if (params->trace_output)
+ trace_instance_start(&record->trace);
+ if (!params->no_aa && aa != top)
+ trace_instance_start(&aa->trace);
+ trace_instance_start(trace);
+
top->start_time = time(NULL);
timerlat_top_set_signals(params);
--
2.38.1
Add the support for running timerlat threads in user-space. In this
mode, enabled with -u/--user-threads, timerlat dispatches user-space
processes that will loop in the timerlat_fd, measuring the overhead
for going to user-space and then returning to the kernel - in addition
to the existing measurements.
Here is one example of the tool's output with -u enabled:
$ sudo timerlat top -u -d 600 -q
Timer Latency
0 00:10:01 | IRQ Timer Latency (us) | Thread Timer Latency (us) | Ret user Timer Latency (us)
CPU COUNT | cur min avg max | cur min avg max | cur min avg max
0 #600001 | 0 0 0 3 | 2 1 2 9 | 3 2 3 15
1 #600001 | 0 0 0 2 | 2 1 2 13 | 2 2 3 18
2 #600001 | 0 0 0 10 | 2 1 2 16 | 3 2 3 20
3 #600001 | 0 0 0 7 | 2 1 2 10 | 3 2 3 11
4 #600000 | 0 0 0 16 | 2 1 2 41 | 3 2 3 58
5 #600000 | 0 0 0 3 | 2 1 2 10 | 3 2 3 13
6 #600000 | 0 0 0 5 | 2 1 2 7 | 3 2 3 10
7 #600000 | 0 0 0 1 | 2 1 2 7 | 3 2 3 10
The tuning setup like -p or -C work for the user-space threads as well.
Cc: Steven Rostedt <[email protected]>
Cc: Daniel Bristot de Oliveira <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Jonathan Corbet <[email protected]>
Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
---
tools/tracing/rtla/src/osnoise.c | 65 ++++++++
tools/tracing/rtla/src/osnoise.h | 5 +
tools/tracing/rtla/src/timerlat_top.c | 108 ++++++++++++-
tools/tracing/rtla/src/timerlat_u.c | 224 ++++++++++++++++++++++++++
tools/tracing/rtla/src/timerlat_u.h | 18 +++
tools/tracing/rtla/src/utils.c | 58 +++++++
tools/tracing/rtla/src/utils.h | 2 +
7 files changed, 474 insertions(+), 6 deletions(-)
create mode 100644 tools/tracing/rtla/src/timerlat_u.c
create mode 100644 tools/tracing/rtla/src/timerlat_u.h
diff --git a/tools/tracing/rtla/src/osnoise.c b/tools/tracing/rtla/src/osnoise.c
index 3ca7a3853943..245e9344932b 100644
--- a/tools/tracing/rtla/src/osnoise.c
+++ b/tools/tracing/rtla/src/osnoise.c
@@ -841,6 +841,67 @@ static void osnoise_put_irq_disable(struct osnoise_context *context)
context->orig_opt_irq_disable = OSNOISE_OPTION_INIT_VAL;
}
+static int osnoise_get_workload(struct osnoise_context *context)
+{
+ if (context->opt_workload != OSNOISE_OPTION_INIT_VAL)
+ return context->opt_workload;
+
+ if (context->orig_opt_workload != OSNOISE_OPTION_INIT_VAL)
+ return context->orig_opt_workload;
+
+ context->orig_opt_workload = osnoise_options_get_option("OSNOISE_WORKLOAD");
+
+ return context->orig_opt_workload;
+}
+
+int osnoise_set_workload(struct osnoise_context *context, bool onoff)
+{
+ int opt_workload = osnoise_get_workload(context);
+ int retval;
+
+ if (opt_workload == OSNOISE_OPTION_INIT_VAL)
+ return -1;
+
+ if (opt_workload == onoff)
+ return 0;
+
+ retval = osnoise_options_set_option("OSNOISE_WORKLOAD", onoff);
+ if (retval < 0)
+ return -1;
+
+ context->opt_workload = onoff;
+
+ return 0;
+}
+
+static void osnoise_restore_workload(struct osnoise_context *context)
+{
+ int retval;
+
+ if (context->orig_opt_workload == OSNOISE_OPTION_INIT_VAL)
+ return;
+
+ if (context->orig_opt_workload == context->opt_workload)
+ goto out_done;
+
+ retval = osnoise_options_set_option("OSNOISE_WORKLOAD", context->orig_opt_workload);
+ if (retval < 0)
+ err_msg("Could not restore original OSNOISE_WORKLOAD option\n");
+
+out_done:
+ context->orig_opt_workload = OSNOISE_OPTION_INIT_VAL;
+}
+
+static void osnoise_put_workload(struct osnoise_context *context)
+{
+ osnoise_restore_workload(context);
+
+ if (context->orig_opt_workload == OSNOISE_OPTION_INIT_VAL)
+ return;
+
+ context->orig_opt_workload = OSNOISE_OPTION_INIT_VAL;
+}
+
/*
* enable_osnoise - enable osnoise tracer in the trace_instance
*/
@@ -908,6 +969,9 @@ struct osnoise_context *osnoise_context_alloc(void)
context->orig_opt_irq_disable = OSNOISE_OPTION_INIT_VAL;
context->opt_irq_disable = OSNOISE_OPTION_INIT_VAL;
+ context->orig_opt_workload = OSNOISE_OPTION_INIT_VAL;
+ context->opt_workload = OSNOISE_OPTION_INIT_VAL;
+
osnoise_get_context(context);
return context;
@@ -935,6 +999,7 @@ void osnoise_put_context(struct osnoise_context *context)
osnoise_put_print_stack(context);
osnoise_put_tracing_thresh(context);
osnoise_put_irq_disable(context);
+ osnoise_put_workload(context);
free(context);
}
diff --git a/tools/tracing/rtla/src/osnoise.h b/tools/tracing/rtla/src/osnoise.h
index 4dcf22ccd704..555f4f4903cc 100644
--- a/tools/tracing/rtla/src/osnoise.h
+++ b/tools/tracing/rtla/src/osnoise.h
@@ -42,6 +42,10 @@ struct osnoise_context {
/* -1 as init value because 0 is off */
int orig_opt_irq_disable;
int opt_irq_disable;
+
+ /* -1 as init value because 0 is off */
+ int orig_opt_workload;
+ int opt_workload;
};
/*
@@ -84,6 +88,7 @@ int osnoise_set_print_stack(struct osnoise_context *context,
long long print_stack);
int osnoise_set_irq_disable(struct osnoise_context *context, bool onoff);
+int osnoise_set_workload(struct osnoise_context *context, bool onoff);
/*
* osnoise_tool - osnoise based tool definition.
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 02cff4948981..1640f121baca 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -13,11 +13,13 @@
#include <time.h>
#include <errno.h>
#include <sched.h>
+#include <pthread.h>
#include "utils.h"
#include "osnoise.h"
#include "timerlat.h"
#include "timerlat_aa.h"
+#include "timerlat_u.h"
struct timerlat_top_params {
char *cpus;
@@ -40,6 +42,7 @@ struct timerlat_top_params {
int dump_tasks;
int cgroup;
int hk_cpus;
+ int user_top;
cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
@@ -48,6 +51,7 @@ struct timerlat_top_params {
struct timerlat_top_cpu {
int irq_count;
int thread_count;
+ int user_count;
unsigned long long cur_irq;
unsigned long long min_irq;
@@ -58,6 +62,11 @@ struct timerlat_top_cpu {
unsigned long long min_thread;
unsigned long long sum_thread;
unsigned long long max_thread;
+
+ unsigned long long cur_user;
+ unsigned long long min_user;
+ unsigned long long sum_user;
+ unsigned long long max_user;
};
struct timerlat_top_data {
@@ -98,6 +107,7 @@ static struct timerlat_top_data *timerlat_alloc_top(int nr_cpus)
for (cpu = 0; cpu < nr_cpus; cpu++) {
data->cpu_data[cpu].min_irq = ~0;
data->cpu_data[cpu].min_thread = ~0;
+ data->cpu_data[cpu].min_user = ~0;
}
return data;
@@ -124,12 +134,18 @@ timerlat_top_update(struct osnoise_tool *tool, int cpu,
update_min(&cpu_data->min_irq, &latency);
update_sum(&cpu_data->sum_irq, &latency);
update_max(&cpu_data->max_irq, &latency);
- } else {
+ } else if (thread == 1) {
cpu_data->thread_count++;
cpu_data->cur_thread = latency;
update_min(&cpu_data->min_thread, &latency);
update_sum(&cpu_data->sum_thread, &latency);
update_max(&cpu_data->max_thread, &latency);
+ } else {
+ cpu_data->user_count++;
+ cpu_data->cur_user = latency;
+ update_min(&cpu_data->min_user, &latency);
+ update_sum(&cpu_data->sum_user, &latency);
+ update_max(&cpu_data->max_user, &latency);
}
}
@@ -172,15 +188,25 @@ static void timerlat_top_header(struct osnoise_tool *top)
trace_seq_printf(s, "\033[2;37;40m");
trace_seq_printf(s, " Timer Latency ");
+ if (params->user_top)
+ trace_seq_printf(s, " ");
trace_seq_printf(s, "\033[0;0;0m");
trace_seq_printf(s, "\n");
- trace_seq_printf(s, "%-6s | IRQ Timer Latency (%s) | Thread Timer Latency (%s)\n", duration,
+ trace_seq_printf(s, "%-6s | IRQ Timer Latency (%s) | Thread Timer Latency (%s)", duration,
params->output_divisor == 1 ? "ns" : "us",
params->output_divisor == 1 ? "ns" : "us");
+ if (params->user_top) {
+ trace_seq_printf(s, " | Ret user Timer Latency (%s)",
+ params->output_divisor == 1 ? "ns" : "us");
+ }
+
+ trace_seq_printf(s, "\n");
trace_seq_printf(s, "\033[2;30;47m");
trace_seq_printf(s, "CPU COUNT | cur min avg max | cur min avg max");
+ if (params->user_top)
+ trace_seq_printf(s, " | cur min avg max");
trace_seq_printf(s, "\033[0;0;0m");
trace_seq_printf(s, "\n");
}
@@ -233,7 +259,27 @@ static void timerlat_top_print(struct osnoise_tool *top, int cpu)
trace_seq_printf(s, "%9llu ", cpu_data->min_thread / divisor);
trace_seq_printf(s, "%9llu ",
(cpu_data->sum_thread / cpu_data->thread_count) / divisor);
- trace_seq_printf(s, "%9llu\n", cpu_data->max_thread / divisor);
+ trace_seq_printf(s, "%9llu", cpu_data->max_thread / divisor);
+ }
+
+ if (!params->user_top) {
+ trace_seq_printf(s, "\n");
+ return;
+ }
+
+ trace_seq_printf(s, " |");
+
+ if (!cpu_data->user_count) {
+ trace_seq_printf(s, " - ");
+ trace_seq_printf(s, " - ");
+ trace_seq_printf(s, " - ");
+ trace_seq_printf(s, " -\n");
+ } else {
+ trace_seq_printf(s, "%9llu ", cpu_data->cur_user / divisor);
+ trace_seq_printf(s, "%9llu ", cpu_data->min_user / divisor);
+ trace_seq_printf(s, "%9llu ",
+ (cpu_data->sum_user / cpu_data->user_count) / divisor);
+ trace_seq_printf(s, "%9llu\n", cpu_data->max_user / divisor);
}
}
@@ -288,7 +334,7 @@ static void timerlat_top_usage(char *usage)
"",
" usage: rtla timerlat [top] [-h] [-q] [-a us] [-d s] [-D] [-n] [-p us] [-i us] [-T us] [-s us] \\",
" [[-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
- " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]]",
+ " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]] [-u]",
"",
" -h/--help: print this menu",
" -a/--auto: set automatic trace mode, stopping the session if argument in us latency is hit",
@@ -317,6 +363,7 @@ static void timerlat_top_usage(char *usage)
" f:prio - use SCHED_FIFO with prio",
" d:runtime[us|ms|s]:period[us|ms|s] - use SCHED_DEADLINE with runtime and period",
" in nanoseconds",
+ " -u/--user-threads: use rtla user-space threads instead of in-kernel timerlat threads",
NULL,
};
@@ -371,6 +418,7 @@ static struct timerlat_top_params
{"stack", required_argument, 0, 's'},
{"thread", required_argument, 0, 'T'},
{"trace", optional_argument, 0, 't'},
+ {"user-threads", no_argument, 0, 'u'},
{"trigger", required_argument, 0, '0'},
{"filter", required_argument, 0, '1'},
{"dma-latency", required_argument, 0, '2'},
@@ -383,7 +431,7 @@ static struct timerlat_top_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:0:1:2:345:",
+ c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:u0:1:2:345:",
long_options, &option_index);
/* detect the end of the options. */
@@ -498,6 +546,9 @@ static struct timerlat_top_params
else
params->trace_output = "timerlat_trace.txt";
+ break;
+ case 'u':
+ params->user_top = true;
break;
case '0': /* trigger */
if (params->events) {
@@ -563,6 +614,7 @@ static int
timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *params)
{
int retval;
+ int i;
if (!params->sleep_time)
params->sleep_time = 1;
@@ -573,6 +625,9 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *
err_msg("Failed to apply CPUs config\n");
goto out_err;
}
+ } else {
+ for (i = 0; i < sysconf(_SC_NPROCESSORS_CONF); i++)
+ CPU_SET(i, ¶ms->monitored_cpus);
}
if (params->stop_us) {
@@ -627,6 +682,14 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *
auto_house_keeping(¶ms->monitored_cpus);
}
+ if (params->user_top) {
+ retval = osnoise_set_workload(top->context, 0);
+ if (retval) {
+ err_msg("Failed to set OSNOISE_WORKLOAD option\n");
+ goto out_err;
+ }
+ }
+
return 0;
out_err:
@@ -687,10 +750,12 @@ int timerlat_top_main(int argc, char *argv[])
{
struct timerlat_top_params *params;
struct osnoise_tool *record = NULL;
+ struct timerlat_u_params params_u;
struct osnoise_tool *top = NULL;
struct osnoise_tool *aa = NULL;
struct trace_instance *trace;
int dma_latency_fd = -1;
+ pthread_t timerlat_u;
int return_value = 1;
char *max_lat;
int retval;
@@ -727,7 +792,7 @@ int timerlat_top_main(int argc, char *argv[])
}
}
- if (params->cgroup) {
+ if (params->cgroup && !params->user_top) {
retval = set_comm_cgroup("timerlat/", params->cgroup_name);
if (!retval) {
err_msg("Failed to move threads to cgroup\n");
@@ -800,6 +865,25 @@ int timerlat_top_main(int argc, char *argv[])
top->start_time = time(NULL);
timerlat_top_set_signals(params);
+ if (params->user_top) {
+ /* rtla asked to stop */
+ params_u.should_run = 1;
+ /* all threads left */
+ params_u.stopped_running = 0;
+
+ params_u.set = ¶ms->monitored_cpus;
+ if (params->set_sched)
+ params_u.sched_param = ¶ms->sched_param;
+ else
+ params_u.sched_param = NULL;
+
+ params_u.cgroup_name = params->cgroup_name;
+
+ retval = pthread_create(&timerlat_u, NULL, timerlat_u_dispatcher, ¶ms_u);
+ if (retval)
+ err_msg("Error creating timerlat user-space threads\n");
+ }
+
while (!stop_tracing) {
sleep(params->sleep_time);
@@ -823,6 +907,18 @@ int timerlat_top_main(int argc, char *argv[])
if (trace_is_off(&top->trace, &record->trace))
break;
+ /* is there still any user-threads ? */
+ if (params->user_top) {
+ if (params_u.stopped_running) {
+ debug_msg("timerlat user space threads stopped!\n");
+ break;
+ }
+ }
+ }
+
+ if (params->user_top && !params_u.stopped_running) {
+ params_u.should_run = 0;
+ sleep(1);
}
timerlat_print_stats(params, top);
diff --git a/tools/tracing/rtla/src/timerlat_u.c b/tools/tracing/rtla/src/timerlat_u.c
new file mode 100644
index 000000000000..05e310696dd5
--- /dev/null
+++ b/tools/tracing/rtla/src/timerlat_u.c
@@ -0,0 +1,224 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2023 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+
+#define _GNU_SOURCE
+#include <sched.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <errno.h>
+#include <string.h>
+#include <tracefs.h>
+#include <pthread.h>
+#include <sys/wait.h>
+#include <sys/prctl.h>
+
+#include "utils.h"
+#include "timerlat_u.h"
+
+/*
+ * This is the user-space main for the tool timerlatu/ threads.
+ *
+ * It is as simple as this:
+ * - set affinity
+ * - set priority
+ * - open tracer fd
+ * - spin
+ * - close
+ */
+static int timerlat_u_main(int cpu, struct timerlat_u_params *params)
+{
+ struct sched_param sp = { .sched_priority = 95 };
+ char buffer[1024];
+ int timerlat_fd;
+ cpu_set_t set;
+ int retval;
+
+ /*
+ * This all is only setting up the tool.
+ */
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+
+ retval = sched_setaffinity(gettid(), sizeof(set), &set);
+ if (retval == -1) {
+ err_msg("Error setting user thread affinity\n");
+ exit(1);
+ }
+
+ if (!params->sched_param) {
+ retval = sched_setscheduler(0, SCHED_FIFO, &sp);
+ if (retval < 0) {
+ err_msg("Error setting timerlat u default priority: %s\n", strerror(errno));
+ exit(1);
+ }
+ } else {
+ retval = __set_sched_attr(getpid(), params->sched_param);
+ if (retval) {
+ /* __set_sched_attr prints an error message, so */
+ exit(0);
+ }
+ }
+
+ if (params->cgroup_name) {
+ retval = set_pid_cgroup(gettid(), params->cgroup_name);
+ if (!retval) {
+ err_msg("Error setting timerlat u cgroup pid\n");
+ pthread_exit(&retval);
+ }
+ }
+
+ /*
+ * This is the tool's loop. If you want to use as base for your own tool...
+ * go ahead.
+ */
+ snprintf(buffer, sizeof(buffer), "osnoise/per_cpu/cpu%d/timerlat_fd", cpu);
+
+ timerlat_fd = tracefs_instance_file_open(NULL, buffer, O_RDONLY);
+ if (timerlat_fd < 0) {
+ err_msg("Error opening %s:%s\n", buffer, strerror(errno));
+ exit(1);
+ }
+
+ debug_msg("User-space timerlat pid %d on cpu %d\n", gettid(), cpu);
+
+ /* add should continue with a signal handler */
+ while (true) {
+ retval = read(timerlat_fd, buffer, 1024);
+ if (retval < 0)
+ break;
+ }
+
+ close(timerlat_fd);
+
+ debug_msg("Leaving timerlat pid %d on cpu %d\n", gettid(), cpu);
+ exit(0);
+}
+
+/*
+ * timerlat_u_send_kill - send a kill signal for all processes
+ *
+ * Return the number of processes that received the kill.
+ */
+static int timerlat_u_send_kill(pid_t *procs, int nr_cpus)
+{
+ int killed = 0;
+ int i, retval;
+
+ for (i = 0; i < nr_cpus; i++) {
+ if (!procs[i])
+ continue;
+ retval = kill(procs[i], SIGKILL);
+ if (!retval)
+ killed++;
+ else
+ err_msg("Error killing child process %d\n", procs[i]);
+ }
+
+ return killed;
+}
+
+/**
+ * timerlat_u_dispatcher - dispatch one timerlatu/ process per monitored CPU
+ *
+ * This is a thread main that will fork one new process for each monitored
+ * CPU. It will wait for:
+ *
+ * - rtla to tell to kill the child processes
+ * - some child process to die, and the cleanup all the processes
+ *
+ * whichever comes first.
+ *
+ */
+void *timerlat_u_dispatcher(void *data)
+{
+ int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
+ struct timerlat_u_params *params = data;
+ char proc_name[128];
+ int procs_count = 0;
+ int retval = 1;
+ pid_t *procs;
+ int wstatus;
+ pid_t pid;
+ int i;
+
+ debug_msg("Dispatching timerlat u procs\n");
+
+ procs = calloc(nr_cpus, sizeof(pid_t));
+ if (!procs)
+ pthread_exit(&retval);
+
+ for (i = 0; i < nr_cpus; i++) {
+ if (params->set && !CPU_ISSET(i, params->set))
+ continue;
+
+ pid = fork();
+
+ /* child */
+ if (!pid) {
+
+ /*
+ * rename the process
+ */
+ snprintf(proc_name, sizeof(proc_name), "timerlatu/%d", i);
+ pthread_setname_np(pthread_self(), proc_name);
+ prctl(PR_SET_NAME, (unsigned long)proc_name, 0, 0, 0);
+
+ timerlat_u_main(i, params);
+ /* timerlat_u_main should exit()! Anyways... */
+ pthread_exit(&retval);
+ }
+
+ /* parent */
+ if (pid == -1) {
+ timerlat_u_send_kill(procs, nr_cpus);
+ debug_msg("Failed to create child processes");
+ pthread_exit(&retval);
+ }
+
+ procs_count++;
+ procs[i] = pid;
+ }
+
+ while (params->should_run) {
+ /* check if processes died */
+ pid = waitpid(-1, &wstatus, WNOHANG);
+ if (pid != 0) {
+ for (i = 0; i < nr_cpus; i++) {
+ if (procs[i] == pid) {
+ procs[i] = 0;
+ procs_count--;
+ }
+ }
+ break;
+ }
+
+ sleep(1);
+ }
+
+ timerlat_u_send_kill(procs, nr_cpus);
+
+ while (procs_count) {
+ pid = waitpid(-1, &wstatus, 0);
+ if (pid == -1) {
+ err_msg("Failed to monitor child processes");
+ pthread_exit(&retval);
+ }
+ for (i = 0; i < nr_cpus; i++) {
+ if (procs[i] == pid) {
+ procs[i] = 0;
+ procs_count--;
+ }
+ }
+ }
+
+ params->stopped_running = 1;
+
+ free(procs);
+ retval = 0;
+ pthread_exit(&retval);
+
+}
diff --git a/tools/tracing/rtla/src/timerlat_u.h b/tools/tracing/rtla/src/timerlat_u.h
new file mode 100644
index 000000000000..661511908957
--- /dev/null
+++ b/tools/tracing/rtla/src/timerlat_u.h
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2023 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
+ */
+
+struct timerlat_u_params {
+ /* timerlat -> timerlat_u: user-space threads can keep running */
+ int should_run;
+ /* timerlat_u -> timerlat: all timerlat_u threads left, no reason to continue */
+ int stopped_running;
+
+ /* threads config */
+ cpu_set_t *set;
+ char *cgroup_name;
+ struct sched_attr *sched_param;
+};
+
+void *timerlat_u_dispatcher(void *data);
diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c
index 3e25f0277fb9..623a38908ed5 100644
--- a/tools/tracing/rtla/src/utils.c
+++ b/tools/tracing/rtla/src/utils.c
@@ -626,6 +626,64 @@ static int get_self_cgroup(char *self_cg, int sizeof_self_cg)
return 1;
}
+/*
+ * set_comm_cgroup - Set cgroup to pid_t pid
+ *
+ * If cgroup argument is not NULL, the threads will move to the given cgroup.
+ * Otherwise, the cgroup of the calling, i.e., rtla, thread will be used.
+ *
+ * Supports cgroup v2.
+ *
+ * Returns 1 on success, 0 otherwise.
+ */
+int set_pid_cgroup(pid_t pid, const char *cgroup)
+{
+ char cgroup_path[MAX_PATH - strlen("/cgroup.procs")];
+ char cgroup_procs[MAX_PATH];
+ char pid_str[24];
+ int retval;
+ int cg_fd;
+
+ retval = find_mount("cgroup2", cgroup_path, sizeof(cgroup_path));
+ if (!retval) {
+ err_msg("Did not find cgroupv2 mount point\n");
+ return 0;
+ }
+
+ if (!cgroup) {
+ retval = get_self_cgroup(&cgroup_path[strlen(cgroup_path)],
+ sizeof(cgroup_path) - strlen(cgroup_path));
+ if (!retval) {
+ err_msg("Did not find self cgroup\n");
+ return 0;
+ }
+ } else {
+ snprintf(&cgroup_path[strlen(cgroup_path)],
+ sizeof(cgroup_path) - strlen(cgroup_path), "%s/", cgroup);
+ }
+
+ snprintf(cgroup_procs, MAX_PATH, "%s/cgroup.procs", cgroup_path);
+
+ debug_msg("Using cgroup path at: %s\n", cgroup_procs);
+
+ cg_fd = open(cgroup_procs, O_RDWR);
+ if (cg_fd < 0)
+ return 0;
+
+ snprintf(pid_str, sizeof(pid_str), "%d\n", pid);
+
+ retval = write(cg_fd, pid_str, strlen(pid_str));
+ if (retval < 0)
+ err_msg("Error setting cgroup attributes for pid:%s - %s\n",
+ pid_str, strerror(errno));
+ else
+ debug_msg("Set cgroup attributes for pid:%s\n", pid_str);
+
+ close(cg_fd);
+
+ return (retval >= 0);
+}
+
/**
* set_comm_cgroup - Set cgroup to threads starting with char *comm_prefix
*
diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h
index dec59163cfbc..04ed1e650495 100644
--- a/tools/tracing/rtla/src/utils.h
+++ b/tools/tracing/rtla/src/utils.h
@@ -57,8 +57,10 @@ struct sched_attr {
int parse_prio(char *arg, struct sched_attr *sched_param);
int parse_cpu_set(char *cpu_list, cpu_set_t *set);
+int __set_sched_attr(int pid, struct sched_attr *attr);
int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr);
int set_comm_cgroup(const char *comm_prefix, const char *cgroup);
+int set_pid_cgroup(pid_t pid, const char *cgroup);
int set_cpu_dma_latency(int32_t latency);
int auto_house_keeping(cpu_set_t *monitored_cpus);
--
2.38.1
Hi,
On 06/06/23 18:12, Daniel Bristot de Oliveira wrote:
> This is a series of improvements for rtla, mainly as a result of our
> daily usage of the tool debugging problems at red hat.
>
> The cgroup support and house keeping options are from our usage
> of the tool debugging containers.
>
> The auto-analysis overhead reduction is needed when we go to
> large boxes - but it is really hand in practice, as it gives an idea
> of the problem without having to look at the trace.
>
> Running hwnoise 100 % of CPU time might cause some systems
> to slow down too much. Reduce its utilization to 75% by default to
> avoid problems for people using it for the first time.
>
> Finally, it adds support for running timerlat user-space threads,
> and to collect the additional field via rtla timerlat top/hist.
>
> Changes from V2:
> - Add timerlat hist -u option
> - Link: https://lore.kernel.org/lkml/[email protected]/
> Changes from V1:
> - Add the user-space thread support to rtla timerlat top
> - Link: https://lore.kernel.org/lkml/[email protected]/
>
> Daniel Bristot de Oliveira (11):
> rtla: Add -C cgroup support
> rtla: Add --house-keeping option
> rtla: Change monitored_cpus from char * to cpu_set_t
> rtla: Automatically move rtla to a house-keeping cpu
> rtla/timerlat: Give timerlat auto analysis its own instance
> rtla/timerlat_hist: Add auto-analysis support
> rtla: Start the tracers after creating all instances
> rtla/hwnoise: Reduce runtime to 75%
> rtla: Add timerlat user-space support for timerlat top
> rtla: Add timerlat user-space support for
> Documentation: Add tools/rtla timerlat -u option documentation
>
> Documentation/tools/rtla/common_options.rst | 8 +
> .../tools/rtla/common_timerlat_aa.rst | 7 -
> .../tools/rtla/common_timerlat_options.rst | 7 +
> .../tools/rtla/rtla-timerlat-hist.rst | 7 +-
> .../tools/rtla/rtla-timerlat-top.rst | 7 +
> tools/tracing/rtla/src/osnoise.c | 65 ++++
> tools/tracing/rtla/src/osnoise.h | 5 +
> tools/tracing/rtla/src/osnoise_hist.c | 90 ++++-
> tools/tracing/rtla/src/osnoise_top.c | 83 ++++-
> tools/tracing/rtla/src/timerlat_aa.c | 35 +-
> tools/tracing/rtla/src/timerlat_aa.h | 5 +-
> tools/tracing/rtla/src/timerlat_hist.c | 262 ++++++++++++--
> tools/tracing/rtla/src/timerlat_top.c | 229 +++++++++++--
> tools/tracing/rtla/src/timerlat_u.c | 224 ++++++++++++
> tools/tracing/rtla/src/timerlat_u.h | 18 +
> tools/tracing/rtla/src/utils.c | 324 +++++++++++++++++-
> tools/tracing/rtla/src/utils.h | 7 +
> 17 files changed, 1277 insertions(+), 106 deletions(-)
> create mode 100644 tools/tracing/rtla/src/timerlat_u.c
> create mode 100644 tools/tracing/rtla/src/timerlat_u.h
>
> --
I've been heavily relying on these for debugging various issues with
latency sensitive workloads and they work like a charm.
Tested-by: Juri Lelli <[email protected]>
Best,
Juri
On Tue, 6 Jun 2023 18:12:16 +0200
Daniel Bristot de Oliveira <[email protected]> wrote:
> To avoid having rtla interfering with the measurement threads, add an
> option for the user to set the CPUs in which rtla should run. For
> instance:
>
> # rtla timerlat top -H 0 -C 1-7
Isn't it '-c 1-7' instead of '-C', because -C is for cgroup name?
Thanks,
>
> Will place rtla in the CPU 0, while running the measurement threads in
> the CPU 1-7.
>
> Cc: Steven Rostedt <[email protected]>
> Cc: Daniel Bristot de Oliveira <[email protected]>
> Cc: Masami Hiramatsu <[email protected]>
> Cc: Jonathan Corbet <[email protected]>
> Suggested-by: Juri Lelli <[email protected]>
> Signed-off-by: Daniel Bristot de Oliveira <[email protected]>
> ---
> Documentation/tools/rtla/common_options.rst | 4 ++
> tools/tracing/rtla/src/osnoise_hist.c | 29 +++++++++--
> tools/tracing/rtla/src/osnoise_top.c | 27 +++++++++-
> tools/tracing/rtla/src/timerlat_hist.c | 27 +++++++++-
> tools/tracing/rtla/src/timerlat_top.c | 27 +++++++++-
> tools/tracing/rtla/src/utils.c | 58 +++++++++++++++++++++
> tools/tracing/rtla/src/utils.h | 3 ++
> 7 files changed, 166 insertions(+), 9 deletions(-)
>
> diff --git a/Documentation/tools/rtla/common_options.rst b/Documentation/tools/rtla/common_options.rst
> index ede07359d93c..aeb91ff3bd68 100644
> --- a/Documentation/tools/rtla/common_options.rst
> +++ b/Documentation/tools/rtla/common_options.rst
> @@ -2,6 +2,10 @@
>
> Set the osnoise tracer to run the sample threads in the cpu-list.
>
> +**-H**, **--house-keeping** *cpu-list*
> +
> + Run rtla control threads only on the given cpu-list.
> +
> **-d**, **--duration** *time[s|m|h|d]*
>
> Set the duration of the session.
> diff --git a/tools/tracing/rtla/src/osnoise_hist.c b/tools/tracing/rtla/src/osnoise_hist.c
> index 076f4c6af3dd..d2b68177ffac 100644
> --- a/tools/tracing/rtla/src/osnoise_hist.c
> +++ b/tools/tracing/rtla/src/osnoise_hist.c
> @@ -3,6 +3,7 @@
> * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
> */
>
> +#define _GNU_SOURCE
> #include <getopt.h>
> #include <stdlib.h>
> #include <string.h>
> @@ -11,6 +12,7 @@
> #include <errno.h>
> #include <stdio.h>
> #include <time.h>
> +#include <sched.h>
>
> #include "utils.h"
> #include "osnoise.h"
> @@ -30,6 +32,8 @@ struct osnoise_hist_params {
> int set_sched;
> int output_divisor;
> int cgroup;
> + int hk_cpus;
> + cpu_set_t hk_cpu_set;
> struct sched_attr sched_param;
> struct trace_events *events;
>
> @@ -434,8 +438,8 @@ static void osnoise_hist_usage(char *usage)
> "",
> " usage: rtla osnoise hist [-h] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\",
> " [-T us] [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] \\",
> - " [-c cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] [--no-index] \\",
> - " [--with-zeros] [-C[=cgroup_name]]",
> + " [-c cpu-list] [-H cpu-list] [-P priority] [-b N] [-E N] [--no-header] [--no-summary] \\",
> + " [--no-index] [--with-zeros] [-C[=cgroup_name]]",
> "",
> " -h/--help: print this menu",
> " -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit",
> @@ -445,6 +449,7 @@ static void osnoise_hist_usage(char *usage)
> " -S/--stop-total us: stop trace if the total sample is higher than the argument in us",
> " -T/--threshold us: the minimum delta to be considered a noise",
> " -c/--cpus cpu-list: list of cpus to run osnoise threads",
> + " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
> " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
> " -d/--duration time[s|m|h|d]: duration of the session",
> " -D/--debug: print debug info",
> @@ -507,6 +512,7 @@ static struct osnoise_hist_params
> {"cgroup", optional_argument, 0, 'C'},
> {"debug", no_argument, 0, 'D'},
> {"duration", required_argument, 0, 'd'},
> + {"house-keeping", required_argument, 0, 'H'},
> {"help", no_argument, 0, 'h'},
> {"period", required_argument, 0, 'p'},
> {"priority", required_argument, 0, 'P'},
> @@ -528,7 +534,7 @@ static struct osnoise_hist_params
> /* getopt_long stores the option index here. */
> int option_index = 0;
>
> - c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhp:P:r:s:S:t::T:01234:5:",
> + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:p:P:r:s:S:t::T:01234:5:",
> long_options, &option_index);
>
> /* detect the end of the options. */
> @@ -597,6 +603,14 @@ static struct osnoise_hist_params
> case '?':
> osnoise_hist_usage(NULL);
> break;
> + case 'H':
> + params->hk_cpus = 1;
> + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
> + if (retval) {
> + err_msg("Error parsing house keeping CPUs\n");
> + exit(EXIT_FAILURE);
> + }
> + break;
> case 'p':
> params->period = get_llong_from_str(optarg);
> if (params->period > 10000000)
> @@ -732,6 +746,15 @@ osnoise_hist_apply_config(struct osnoise_tool *tool, struct osnoise_hist_params
> }
> }
>
> + if (params->hk_cpus) {
> + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
> + ¶ms->hk_cpu_set);
> + if (retval == -1) {
> + err_msg("Failed to set rtla to the house keeping CPUs\n");
> + goto out_err;
> + }
> + }
> +
> return 0;
>
> out_err:
> diff --git a/tools/tracing/rtla/src/osnoise_top.c b/tools/tracing/rtla/src/osnoise_top.c
> index 139d8d392540..fcf6c14ce1bc 100644
> --- a/tools/tracing/rtla/src/osnoise_top.c
> +++ b/tools/tracing/rtla/src/osnoise_top.c
> @@ -3,6 +3,7 @@
> * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
> */
>
> +#define _GNU_SOURCE
> #include <getopt.h>
> #include <stdlib.h>
> #include <string.h>
> @@ -10,6 +11,7 @@
> #include <unistd.h>
> #include <stdio.h>
> #include <time.h>
> +#include <sched.h>
>
> #include "osnoise.h"
> #include "utils.h"
> @@ -37,6 +39,8 @@ struct osnoise_top_params {
> int quiet;
> int set_sched;
> int cgroup;
> + int hk_cpus;
> + cpu_set_t hk_cpu_set;
> struct sched_attr sched_param;
> struct trace_events *events;
> enum osnoise_mode mode;
> @@ -278,7 +282,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage)
> static const char * const msg[] = {
> " [-h] [-q] [-D] [-d s] [-a us] [-p us] [-r us] [-s us] [-S us] \\",
> " [-T us] [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] \\",
> - " [-c cpu-list] [-P priority] [-C[=cgroup_name]]",
> + " [-c cpu-list] [-H cpu-list] [-P priority] [-C[=cgroup_name]]",
> "",
> " -h/--help: print this menu",
> " -a/--auto: set automatic trace mode, stopping the session if argument in us sample is hit",
> @@ -288,6 +292,7 @@ static void osnoise_top_usage(struct osnoise_top_params *params, char *usage)
> " -S/--stop-total us: stop trace if the total sample is higher than the argument in us",
> " -T/--threshold us: the minimum delta to be considered a noise",
> " -c/--cpus cpu-list: list of cpus to run osnoise threads",
> + " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
> " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
> " -d/--duration time[s|m|h|d]: duration of the session",
> " -D/--debug: print debug info",
> @@ -354,6 +359,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
> {"debug", no_argument, 0, 'D'},
> {"duration", required_argument, 0, 'd'},
> {"event", required_argument, 0, 'e'},
> + {"house-keeping", required_argument, 0, 'H'},
> {"help", no_argument, 0, 'h'},
> {"period", required_argument, 0, 'p'},
> {"priority", required_argument, 0, 'P'},
> @@ -371,7 +377,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
> /* getopt_long stores the option index here. */
> int option_index = 0;
>
> - c = getopt_long(argc, argv, "a:c:C::d:De:hp:P:qr:s:S:t::T:0:1:",
> + c = getopt_long(argc, argv, "a:c:C::d:De:hH:p:P:qr:s:S:t::T:0:1:",
> long_options, &option_index);
>
> /* Detect the end of the options. */
> @@ -430,6 +436,14 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)
> case '?':
> osnoise_top_usage(params, NULL);
> break;
> + case 'H':
> + params->hk_cpus = 1;
> + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
> + if (retval) {
> + err_msg("Error parsing house keeping CPUs\n");
> + exit(EXIT_FAILURE);
> + }
> + break;
> case 'p':
> params->period = get_llong_from_str(optarg);
> if (params->period > 10000000)
> @@ -561,6 +575,15 @@ osnoise_top_apply_config(struct osnoise_tool *tool, struct osnoise_top_params *p
> }
> }
>
> + if (params->hk_cpus) {
> + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
> + ¶ms->hk_cpu_set);
> + if (retval == -1) {
> + err_msg("Failed to set rtla to the house keeping CPUs\n");
> + goto out_err;
> + }
> + }
> +
> return 0;
>
> out_err:
> diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
> index 459c159923e8..d48c05d238f9 100644
> --- a/tools/tracing/rtla/src/timerlat_hist.c
> +++ b/tools/tracing/rtla/src/timerlat_hist.c
> @@ -3,6 +3,7 @@
> * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
> */
>
> +#define _GNU_SOURCE
> #include <getopt.h>
> #include <stdlib.h>
> #include <string.h>
> @@ -10,6 +11,7 @@
> #include <unistd.h>
> #include <stdio.h>
> #include <time.h>
> +#include <sched.h>
>
> #include "utils.h"
> #include "osnoise.h"
> @@ -31,6 +33,8 @@ struct timerlat_hist_params {
> int set_sched;
> int dma_latency;
> int cgroup;
> + int hk_cpus;
> + cpu_set_t hk_cpu_set;
> struct sched_attr sched_param;
> struct trace_events *events;
> char no_irq;
> @@ -432,7 +436,7 @@ static void timerlat_hist_usage(char *usage)
> char *msg[] = {
> "",
> " usage: [rtla] timerlat hist [-h] [-q] [-d s] [-D] [-n] [-a us] [-p us] [-i us] [-T us] [-s us] \\",
> - " [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] \\",
> + " [-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
> " [-P priority] [-E N] [-b N] [--no-irq] [--no-thread] [--no-header] [--no-summary] \\",
> " [--no-index] [--with-zeros] [--dma-latency us] [-C[=cgroup_name]]",
> "",
> @@ -443,6 +447,7 @@ static void timerlat_hist_usage(char *usage)
> " -T/--thread us: stop trace if the thread latency is higher than the argument in us",
> " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
> " -c/--cpus cpus: run the tracer only on the given cpus",
> + " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
> " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
> " -d/--duration time[m|h|d]: duration of the session in seconds",
> " -D/--debug: print debug info",
> @@ -513,6 +518,7 @@ static struct timerlat_hist_params
> {"debug", no_argument, 0, 'D'},
> {"entries", required_argument, 0, 'E'},
> {"duration", required_argument, 0, 'd'},
> + {"house-keeping", required_argument, 0, 'H'},
> {"help", no_argument, 0, 'h'},
> {"irq", required_argument, 0, 'i'},
> {"nano", no_argument, 0, 'n'},
> @@ -537,7 +543,7 @@ static struct timerlat_hist_params
> /* getopt_long stores the option index here. */
> int option_index = 0;
>
> - c = getopt_long(argc, argv, "a:c:C::b:d:e:E:Dhi:np:P:s:t::T:0123456:7:8:",
> + c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:np:P:s:t::T:0123456:7:8:",
> long_options, &option_index);
>
> /* detect the end of the options. */
> @@ -608,6 +614,14 @@ static struct timerlat_hist_params
> case '?':
> timerlat_hist_usage(NULL);
> break;
> + case 'H':
> + params->hk_cpus = 1;
> + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
> + if (retval) {
> + err_msg("Error parsing house keeping CPUs\n");
> + exit(EXIT_FAILURE);
> + }
> + break;
> case 'i':
> params->stop_us = get_llong_from_str(optarg);
> break;
> @@ -755,6 +769,15 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param
> }
> }
>
> + if (params->hk_cpus) {
> + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
> + ¶ms->hk_cpu_set);
> + if (retval == -1) {
> + err_msg("Failed to set rtla to the house keeping CPUs\n");
> + goto out_err;
> + }
> + }
> +
> return 0;
>
> out_err:
> diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
> index a19cbc2aa1f4..5395d1c5921e 100644
> --- a/tools/tracing/rtla/src/timerlat_top.c
> +++ b/tools/tracing/rtla/src/timerlat_top.c
> @@ -3,6 +3,7 @@
> * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
> */
>
> +#define _GNU_SOURCE
> #include <getopt.h>
> #include <stdlib.h>
> #include <string.h>
> @@ -11,6 +12,7 @@
> #include <stdio.h>
> #include <time.h>
> #include <errno.h>
> +#include <sched.h>
>
> #include "utils.h"
> #include "osnoise.h"
> @@ -37,6 +39,8 @@ struct timerlat_top_params {
> int aa_only;
> int dump_tasks;
> int cgroup;
> + int hk_cpus;
> + cpu_set_t hk_cpu_set;
> struct sched_attr sched_param;
> struct trace_events *events;
> };
> @@ -286,7 +290,7 @@ static void timerlat_top_usage(char *usage)
> static const char *const msg[] = {
> "",
> " usage: rtla timerlat [top] [-h] [-q] [-a us] [-d s] [-D] [-n] [-p us] [-i us] [-T us] [-s us] \\",
> - " [[-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] \\",
> + " [[-t[=file]] [-e sys[:event]] [--filter <filter>] [--trigger <trigger>] [-c cpu-list] [-H cpu-list]\\",
> " [-P priority] [--dma-latency us] [--aa-only us] [-C[=cgroup_name]]",
> "",
> " -h/--help: print this menu",
> @@ -297,6 +301,7 @@ static void timerlat_top_usage(char *usage)
> " -T/--thread us: stop trace if the thread latency is higher than the argument in us",
> " -s/--stack us: save the stack trace at the IRQ if a thread latency is higher than the argument in us",
> " -c/--cpus cpus: run the tracer only on the given cpus",
> + " -H/--house-keeping cpus: run rtla control threads only on the given cpus",
> " -C/--cgroup[=cgroup_name]: set cgroup, if no cgroup_name is passed, the rtla's cgroup will be inherited",
> " -d/--duration time[m|h|d]: duration of the session in seconds",
> " -D/--debug: print debug info",
> @@ -360,6 +365,7 @@ static struct timerlat_top_params
> {"duration", required_argument, 0, 'd'},
> {"event", required_argument, 0, 'e'},
> {"help", no_argument, 0, 'h'},
> + {"house-keeping", required_argument, 0, 'H'},
> {"irq", required_argument, 0, 'i'},
> {"nano", no_argument, 0, 'n'},
> {"period", required_argument, 0, 'p'},
> @@ -380,7 +386,7 @@ static struct timerlat_top_params
> /* getopt_long stores the option index here. */
> int option_index = 0;
>
> - c = getopt_long(argc, argv, "a:c:C::d:De:hi:np:P:qs:t::T:0:1:2:345:",
> + c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:np:P:qs:t::T:0:1:2:345:",
> long_options, &option_index);
>
> /* detect the end of the options. */
> @@ -454,6 +460,14 @@ static struct timerlat_top_params
> case '?':
> timerlat_top_usage(NULL);
> break;
> + case 'H':
> + params->hk_cpus = 1;
> + retval = parse_cpu_set(optarg, ¶ms->hk_cpu_set);
> + if (retval) {
> + err_msg("Error parsing house keeping CPUs\n");
> + exit(EXIT_FAILURE);
> + }
> + break;
> case 'i':
> params->stop_us = get_llong_from_str(optarg);
> break;
> @@ -598,6 +612,15 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *
> }
> }
>
> + if (params->hk_cpus) {
> + retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
> + ¶ms->hk_cpu_set);
> + if (retval == -1) {
> + err_msg("Failed to set rtla to the house keeping CPUs\n");
> + goto out_err;
> + }
> + }
> +
> return 0;
>
> out_err:
> diff --git a/tools/tracing/rtla/src/utils.c b/tools/tracing/rtla/src/utils.c
> index bcc0a9f39cfe..ee6fab09acae 100644
> --- a/tools/tracing/rtla/src/utils.c
> +++ b/tools/tracing/rtla/src/utils.c
> @@ -3,6 +3,7 @@
> * Copyright (C) 2021 Red Hat Inc, Daniel Bristot de Oliveira <[email protected]>
> */
>
> +#define _GNU_SOURCE
> #include <dirent.h>
> #include <stdarg.h>
> #include <stdlib.h>
> @@ -150,6 +151,63 @@ int parse_cpu_list(char *cpu_list, char **monitored_cpus)
> return 1;
> }
>
> +/*
> + * parse_cpu_set - parse a cpu_list filling cpu_set_t argument
> + *
> + * Receives a cpu list, like 1-3,5 (cpus 1, 2, 3, 5), and then set
> + * filling cpu_set_t argument.
> + *
> + * Returns 1 on success, 0 otherwise.
> + */
> +int parse_cpu_set(char *cpu_list, cpu_set_t *set)
> +{
> + const char *p;
> + int end_cpu;
> + int nr_cpus;
> + int cpu;
> + int i;
> +
> + CPU_ZERO(set);
> +
> + nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
> +
> + for (p = cpu_list; *p; ) {
> + cpu = atoi(p);
> + if (cpu < 0 || (!cpu && *p != '0') || cpu >= nr_cpus)
> + goto err;
> +
> + while (isdigit(*p))
> + p++;
> + if (*p == '-') {
> + p++;
> + end_cpu = atoi(p);
> + if (end_cpu < cpu || (!end_cpu && *p != '0') || end_cpu >= nr_cpus)
> + goto err;
> + while (isdigit(*p))
> + p++;
> + } else
> + end_cpu = cpu;
> +
> + if (cpu == end_cpu) {
> + debug_msg("cpu_set: adding cpu %d\n", cpu);
> + CPU_SET(cpu, set);
> + } else {
> + for (i = cpu; i <= end_cpu; i++) {
> + debug_msg("cpu_set: adding cpu %d\n", i);
> + CPU_SET(i, set);
> + }
> + }
> +
> + if (*p == ',')
> + p++;
> + }
> +
> + return 0;
> +err:
> + debug_msg("Error parsing the cpu set %s\n", cpu_list);
> + return 1;
> +}
> +
> /*
> * parse_duration - parse duration with s/m/h/d suffix converting it to seconds
> */
> diff --git a/tools/tracing/rtla/src/utils.h b/tools/tracing/rtla/src/utils.h
> index 42b6f099d10a..9ab2f0d7bc1c 100644
> --- a/tools/tracing/rtla/src/utils.h
> +++ b/tools/tracing/rtla/src/utils.h
> @@ -1,6 +1,8 @@
> // SPDX-License-Identifier: GPL-2.0
> +
> #include <stdint.h>
> #include <time.h>
> +#include <sched.h>
>
> /*
> * '18446744073709551615\0'
> @@ -54,6 +56,7 @@ struct sched_attr {
> };
>
> int parse_prio(char *arg, struct sched_attr *sched_param);
> +int parse_cpu_set(char *cpu_list, cpu_set_t *set);
> int set_comm_sched_attr(const char *comm_prefix, struct sched_attr *attr);
> int set_comm_cgroup(const char *comm_prefix, const char *cgroup);
> int set_cpu_dma_latency(int32_t latency);
> --
> 2.38.1
>
--
Masami Hiramatsu (Google) <[email protected]>
On 6/12/23 16:06, Masami Hiramatsu (Google) wrote:
> On Tue, 6 Jun 2023 18:12:16 +0200
> Daniel Bristot de Oliveira <[email protected]> wrote:
>
>> To avoid having rtla interfering with the measurement threads, add an
>> option for the user to set the CPUs in which rtla should run. For
>> instance:
>>
>> # rtla timerlat top -H 0 -C 1-7
> Isn't it '-c 1-7' instead of '-C', because -C is for cgroup name?
Ooops, you are right.
-- Daniel
> Thanks,
>
On Mon, 12 Jun 2023 16:21:35 +0200
Daniel Bristot de Oliveira <[email protected]> wrote:
> On 6/12/23 16:06, Masami Hiramatsu (Google) wrote:
> > On Tue, 6 Jun 2023 18:12:16 +0200
> > Daniel Bristot de Oliveira <[email protected]> wrote:
> >
> >> To avoid having rtla interfering with the measurement threads, add an
> >> option for the user to set the CPUs in which rtla should run. For
> >> instance:
> >>
> >> # rtla timerlat top -H 0 -C 1-7
> > Isn't it '-c 1-7' instead of '-C', because -C is for cgroup name?
>
> Ooops, you are right.
>
Unless something else comes up, I updated the change log here.
-- Steve