The embedded world, specifically Android mobile SoCs, rely on CPU
hotplugs to manage power and thermal constraints. These hotplugs
can happen at a very rapid pace. Adjacently, they also relies on
many perf event counters for its management. Therefore, there is
a need to preserve these events across hotplugs.
In such a scenario, a perf client (kernel or user-space) can create
events even when the CPU is offline. If the CPU comes online during
the lifetime of the event, the registered event can start counting
spontaneously. As an extension to this, the events' count can also
be preserved across CPU hotplugs. This takes the burden off of the
clients to monitor the state of the CPU.
The tests were conducted on arm64 device.
/* CPU-1 is offline: Event created when CPU1 is offline */
/ # cat /sys/devices/system/cpu/cpu1/online
1
/ # echo 0 > /sys/devices/system/cpu/cpu1/online
Test used for testing
#!/bin/sh
chmod +x *
# Count the cycles events on cpu-1 for every 200 ms
./perf stat -e cycles -I 200 -C 1 &
# Make the CPU-1 offline and online continuously
while true; do
sleep 2
echo 0 > /sys/devices/system/cpu/cpu1/online
sleep 2
echo 1 > /sys/devices/system/cpu/cpu1/online
done
Results:
/ # ./test.sh
# time counts unit events
0.200145885 <not counted> cycles
0.410115208 <not counted> cycles
0.619922551 <not counted> cycles
0.829904635 <not counted> cycles
1.039751614 <not counted> cycles
1.249547603 <not counted> cycles
1.459228280 <not counted> cycles
1.665606561 <not counted> cycles
1.874981926 <not counted> cycles
2.084297811 <not counted> cycles
2.293471249 <not counted> cycles
2.503231561 <not counted> cycles
2.712993332 <not counted> cycles
2.922744478 <not counted> cycles
3.132502186 <not counted> cycles
3.342255050 <not counted> cycles
3.552010102 <not counted> cycles
3.761760363 <not counted> cycles
/* CPU-1 made online: Event started counting */
3.971459269 1925429 cycles
4.181325206 19391145 cycles
4.391074164 113894 cycles
4.599130519 3150152 cycles
4.805564737 487122 cycles
5.015164581 247533 cycles
5.224764529 103622 cycles
# time counts unit events
5.434360831 238179 cycles
5.645293799 238895 cycles
5.854909320 367543 cycles
6.064487966 2383428 cycles
/* CPU-1 made offline: counting stopped
6.274289476 <not counted> cycles
6.483493903 <not counted> cycles
6.693202705 <not counted> cycles
6.902956195 <not counted> cycles
7.112714268 <not counted> cycles
7.322465570 <not counted> cycles
7.532222340 <not counted> cycles
7.741975830 <not counted> cycles
7.951686246 <not counted> cycles
/* CPU-1 made online: Event started counting
8.161469892 22040750 cycles
8.371219528 114977 cycles
8.580979111 259952 cycles
8.790757132 444661 cycles
9.000559215 248512 cycles
9.210385256 246590 cycles
9.420187704 243819 cycles
9.630052287 7102438 cycles
9.839848225 337454 cycles
10.049645048 644072 cycles
10.259476246 1855410 cycles
Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Alexander Shishkin <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Mukesh Ojha (1):
perf: event preserve and create across cpu hotplug
include/linux/perf_event.h | 1 +
kernel/events/core.c | 122 +++++++++++++++++++++++++++++----------------
2 files changed, 79 insertions(+), 44 deletions(-)
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Perf framework doesn't allow preserving CPU events across
CPU hotplugs. The events are scheduled out as and when the
CPU walks offline. Moreover, the framework also doesn't
allow the clients to create events on an offline CPU. As
a result, the clients have to keep on monitoring the CPU
state until it comes back online.
Therefore, introducing the perf framework to support creation
and preserving of (CPU) events for offline CPUs. Through
this, the CPU's online state would be transparent to the
client and it not have to worry about monitoring the CPU's
state. Success would be returned to the client even while
creating the event on an offline CPU. If during the lifetime
of the event the CPU walks offline, the event would be
preserved and would continue to count as soon as (and if) the
CPU comes back online.
Co-authored-by: Peter Zijlstra <[email protected]>
Signed-off-by: Raghavendra Rao Ananta <[email protected]>
Signed-off-by: Mukesh Ojha <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Alexander Shishkin <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
---
Change in V5:
=============
- Rebased it.
Change in V4:
=============
- Released, __get_cpu_context would not be correct way to get the
cpu context of the cpu which is offline, instead use
container_of to get the cpuctx from ctx.
- Changed the goto label name inside event_function_call from
'remove_event_from_context' to 'out'.
Change in V3:
=============
- Jiri has tried perf stat -a and removed one of the cpu from the other
terminal. This resulted in a crash. Crash was because in
event_function_call(), we were passing NULL as cpuctx in
func(event, NULL, ctx, data).Fixed it in this patch.
Change in V2:
=============
As per long back discussion happened at
https://lkml.org/lkml/2018/2/15/1324
Peter.Z. has suggested to do thing in different way and shared
patch as well. This patch fixes the issue seen while trying
to achieve the purpose.
Fixed issue on top of Peter's patch:
===================================
1. Added a NULL check on task to avoid crash in __perf_install_in_context.
2. while trying to add event to context when cpu is offline.
Inside add_event_to_ctx() to make consistent state machine while hotplug.
-event->state += PERF_EVENT_STATE_HOTPLUG_OFFSET;
+event->state = PERF_EVENT_STATE_HOTPLUG_OFFSET;
3. In event_function_call(), added a label 'remove_event_from_ctx' to
delete events from context list while cpu is offline.
include/linux/perf_event.h | 1 +
kernel/events/core.c | 123 ++++++++++++++++++++++++++++-----------------
2 files changed, 79 insertions(+), 45 deletions(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 3dc01cf..52b14b2 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -511,6 +511,7 @@ enum perf_event_state {
PERF_EVENT_STATE_OFF = -1,
PERF_EVENT_STATE_INACTIVE = 0,
PERF_EVENT_STATE_ACTIVE = 1,
+ PERF_EVENT_STATE_HOTPLUG_OFFSET = -32,
};
struct file;
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 118ad1a..82b5106 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -248,6 +248,8 @@ static int event_function(void *info)
static void event_function_call(struct perf_event *event, event_f func, void *data)
{
struct perf_event_context *ctx = event->ctx;
+ struct perf_cpu_context *cpuctx =
+ container_of(ctx, struct perf_cpu_context, ctx);
struct task_struct *task = READ_ONCE(ctx->task); /* verified in event_function */
struct event_function_struct efs = {
.event = event,
@@ -264,17 +266,18 @@ static void event_function_call(struct perf_event *event, event_f func, void *da
lockdep_assert_held(&ctx->mutex);
}
- if (!task) {
- cpu_function_call(event->cpu, event_function, &efs);
- return;
- }
-
if (task == TASK_TOMBSTONE)
return;
again:
- if (!task_function_call(task, event_function, &efs))
- return;
+ if (task) {
+ if (!task_function_call(task, event_function, &efs))
+ return;
+ } else {
+ if (!cpu_function_call(event->cpu, event_function, &efs))
+ return;
+ }
+
raw_spin_lock_irq(&ctx->lock);
/*
@@ -286,11 +289,17 @@ static void event_function_call(struct perf_event *event, event_f func, void *da
raw_spin_unlock_irq(&ctx->lock);
return;
}
+
+ if (!task)
+ goto out;
+
if (ctx->is_active) {
raw_spin_unlock_irq(&ctx->lock);
goto again;
}
- func(event, NULL, ctx, data);
+
+out:
+ func(event, cpuctx, ctx, data);
raw_spin_unlock_irq(&ctx->lock);
}
@@ -2310,7 +2319,7 @@ static void perf_set_shadow_time(struct perf_event *event,
struct perf_event *event, *partial_group = NULL;
struct pmu *pmu = ctx->pmu;
- if (group_event->state == PERF_EVENT_STATE_OFF)
+ if (group_event->state <= PERF_EVENT_STATE_OFF)
return 0;
pmu->start_txn(pmu, PERF_PMU_TXN_ADD);
@@ -2389,6 +2398,14 @@ static int group_can_go_on(struct perf_event *event,
static void add_event_to_ctx(struct perf_event *event,
struct perf_event_context *ctx)
{
+ if (!ctx->task) {
+ struct perf_cpu_context *cpuctx =
+ container_of(ctx, struct perf_cpu_context, ctx);
+
+ if (!cpuctx->online)
+ event->state = PERF_EVENT_STATE_HOTPLUG_OFFSET;
+ }
+
list_add_event(event, ctx);
perf_group_attach(event);
}
@@ -2576,11 +2593,6 @@ static int __perf_install_in_context(void *info)
*/
smp_store_release(&event->ctx, ctx);
- if (!task) {
- cpu_function_call(cpu, __perf_install_in_context, event);
- return;
- }
-
/*
* Should not happen, we validate the ctx is still alive before calling.
*/
@@ -2619,8 +2631,14 @@ static int __perf_install_in_context(void *info)
*/
smp_mb();
again:
- if (!task_function_call(task, __perf_install_in_context, event))
- return;
+
+ if (task) {
+ if (!task_function_call(task, __perf_install_in_context, event))
+ return;
+ } else {
+ if (!cpu_function_call(cpu, __perf_install_in_context, event))
+ return;
+ }
raw_spin_lock_irq(&ctx->lock);
task = ctx->task;
@@ -2637,7 +2655,7 @@ static int __perf_install_in_context(void *info)
* If the task is not running, ctx->lock will avoid it becoming so,
* thus we can safely install the event.
*/
- if (task_curr(task)) {
+ if (task && task_curr(task)) {
raw_spin_unlock_irq(&ctx->lock);
goto again;
}
@@ -11022,16 +11040,7 @@ static int perf_event_set_clock(struct perf_event *event, clockid_t clk_id)
}
if (!task) {
- /*
- * Check if the @cpu we're creating an event for is online.
- *
- * We use the perf_cpu_context::ctx::mutex to serialize against
- * the hotplug notifiers. See perf_event_{init,exit}_cpu().
- */
- struct perf_cpu_context *cpuctx =
- container_of(ctx, struct perf_cpu_context, ctx);
-
- if (!cpuctx->online) {
+ if (!cpu_possible(cpu)) {
err = -ENODEV;
goto err_locked;
}
@@ -11213,15 +11222,7 @@ struct perf_event *
}
if (!task) {
- /*
- * Check if the @cpu we're creating an event for is online.
- *
- * We use the perf_cpu_context::ctx::mutex to serialize against
- * the hotplug notifiers. See perf_event_{init,exit}_cpu().
- */
- struct perf_cpu_context *cpuctx =
- container_of(ctx, struct perf_cpu_context, ctx);
- if (!cpuctx->online) {
+ if (!cpu_possible(cpu)) {
err = -ENODEV;
goto err_unlock;
}
@@ -11949,17 +11950,48 @@ static void perf_swevent_init_cpu(unsigned int cpu)
}
#if defined CONFIG_HOTPLUG_CPU || defined CONFIG_KEXEC_CORE
+static void __perf_event_init_cpu_context(void *__info)
+{
+ struct perf_cpu_context *cpuctx = __info;
+ struct perf_event_context *ctx = &cpuctx->ctx;
+ struct perf_event_context *task_ctx = cpuctx->task_ctx;
+ struct perf_event *event;
+
+ perf_ctx_lock(cpuctx, task_ctx);
+ ctx_sched_out(ctx, cpuctx, EVENT_ALL);
+ if (task_ctx)
+ ctx_sched_out(task_ctx, cpuctx, EVENT_ALL);
+
+ list_for_each_entry_rcu(event, &ctx->event_list, event_entry)
+ perf_event_set_state(event, event->state - PERF_EVENT_STATE_HOTPLUG_OFFSET);
+
+ perf_event_sched_in(cpuctx, task_ctx, current);
+ perf_ctx_unlock(cpuctx, task_ctx);
+}
+
+static void _perf_event_init_cpu_context(int cpu, struct perf_cpu_context *cpuctx)
+{
+ smp_call_function_single(cpu, __perf_event_init_cpu_context, cpuctx, 1);
+}
+
static void __perf_event_exit_context(void *__info)
{
- struct perf_event_context *ctx = __info;
- struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
+ struct perf_cpu_context *cpuctx = __info;
+ struct perf_event_context *ctx = &cpuctx->ctx;
+ struct perf_event_context *task_ctx = cpuctx->task_ctx;
struct perf_event *event;
- raw_spin_lock(&ctx->lock);
- ctx_sched_out(ctx, cpuctx, EVENT_TIME);
- list_for_each_entry(event, &ctx->event_list, event_entry)
- __perf_remove_from_context(event, cpuctx, ctx, (void *)DETACH_GROUP);
- raw_spin_unlock(&ctx->lock);
+ perf_ctx_lock(cpuctx, task_ctx);
+ ctx_sched_out(ctx, cpuctx, EVENT_ALL);
+ if (task_ctx)
+ ctx_sched_out(task_ctx, cpuctx, EVENT_ALL);
+
+ list_for_each_entry_rcu(event, &ctx->event_list, event_entry)
+ perf_event_set_state(event,
+ event->state + PERF_EVENT_STATE_HOTPLUG_OFFSET);
+
+ perf_event_sched_in(cpuctx, task_ctx, current);
+ perf_ctx_unlock(cpuctx, task_ctx);
}
static void perf_event_exit_cpu_context(int cpu)
@@ -11974,7 +12006,7 @@ static void perf_event_exit_cpu_context(int cpu)
ctx = &cpuctx->ctx;
mutex_lock(&ctx->mutex);
- smp_call_function_single(cpu, __perf_event_exit_context, ctx, 1);
+ smp_call_function_single(cpu, __perf_event_exit_context, cpuctx, 1);
cpuctx->online = 0;
mutex_unlock(&ctx->mutex);
}
@@ -11982,7 +12014,7 @@ static void perf_event_exit_cpu_context(int cpu)
mutex_unlock(&pmus_lock);
}
#else
-
+static void _perf_event_init_cpu_context(int cpu, struct perf_cpu_context *cpuctx) { }
static void perf_event_exit_cpu_context(int cpu) { }
#endif
@@ -12003,6 +12035,7 @@ int perf_event_init_cpu(unsigned int cpu)
mutex_lock(&ctx->mutex);
cpuctx->online = 1;
+ _perf_event_init_cpu_context(cpu, cpuctx);
mutex_unlock(&ctx->mutex);
}
mutex_unlock(&pmus_lock);
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
On Fri, Aug 02, 2019 at 12:16:53AM +0530, Mukesh Ojha wrote:
> Perf framework doesn't allow preserving CPU events across
> CPU hotplugs. The events are scheduled out as and when the
> CPU walks offline. Moreover, the framework also doesn't
> allow the clients to create events on an offline CPU. As
> a result, the clients have to keep on monitoring the CPU
> state until it comes back online.
>
> Therefore, introducing the perf framework to support creation
> and preserving of (CPU) events for offline CPUs. Through
> this, the CPU's online state would be transparent to the
> client and it not have to worry about monitoring the CPU's
> state. Success would be returned to the client even while
> creating the event on an offline CPU. If during the lifetime
> of the event the CPU walks offline, the event would be
> preserved and would continue to count as soon as (and if) the
> CPU comes back online.
>
> Co-authored-by: Peter Zijlstra <[email protected]>
> Signed-off-by: Raghavendra Rao Ananta <[email protected]>
> Signed-off-by: Mukesh Ojha <[email protected]>
> Cc: Peter Zijlstra <[email protected]>
> Cc: Ingo Molnar <[email protected]>
> Cc: Arnaldo Carvalho de Melo <[email protected]>
> Cc: Alexander Shishkin <[email protected]>
> Cc: Jiri Olsa <[email protected]>
> Cc: Alexei Starovoitov <[email protected]>
> ---
> Change in V5:
> =============
> - Rebased it.
note that we might need to change how we store cpu topology,
now that it can change during the sampling.. like below it's
the comparison of header data with and without cpu 1
I think some of the report code checks on topology or caches
and it might get confused
perhaps we could watch cpu topology in record and update the
data as we see it changing.. future TODO list ;-)
perf stat is probably fine
jirka
---
-# nrcpus online : 39
+# nrcpus online : 40
# nrcpus avail : 40
# cpudesc : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
# cpuid : GenuineIntel,6,85,4
...
# sibling sockets : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling sockets : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling sockets : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
# sibling dies : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# sibling dies : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# sibling dies : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
# sibling threads : 0,20
+# sibling threads : 1,21
# sibling threads : 2,22
# sibling threads : 3,23
# sibling threads : 4,24
@@ -38,9 +39,8 @@
# sibling threads : 17,37
# sibling threads : 18,38
# sibling threads : 19,39
-# sibling threads : 21
# CPU 0: Core ID 0, Die ID 0, Socket ID 0
-# CPU 1: Core ID -1, Die ID -1, Socket ID -1
+# CPU 1: Core ID 0, Die ID 0, Socket ID 1
# CPU 2: Core ID 4, Die ID 0, Socket ID 0
# CPU 3: Core ID 4, Die ID 0, Socket ID 1
# CPU 4: Core ID 1, Die ID 0, Socket ID 0
@@ -79,14 +79,16 @@
# CPU 37: Core ID 9, Die ID 0, Socket ID 1
# CPU 38: Core ID 10, Die ID 0, Socket ID 0
# CPU 39: Core ID 10, Die ID 0, Socket ID 1
-# node0 meminfo : total = 47391616 kB, free = 46536844 kB
+# node0 meminfo : total = 47391616 kB, free = 46548348 kB
# node0 cpu list : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
-# node1 meminfo : total = 49539612 kB, free = 48908820 kB
-# node1 cpu list : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
+# node1 meminfo : total = 49539612 kB, free = 48897176 kB
+# node1 cpu list : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
# pmu mappings: intel_pt = 8, uncore_cha_1 = 25, uncore_irp_3 = 49, software = 1, uncore_imc_5 = 18, uncore_m3upi_0 = 21, uncore_iio_free_running_5 = 45, uncore_irp_1 = 47, uncore_m2m_1 = 12, uncore_imc_3 = 16, uncore_cha_8 = 32, uncore_iio_free_running_3 = 43, uncore_imc_1 = 14, uncore_upi_1 = 20, power = 10, uncore_cha_6 = 30, uncore_iio_free_running_1 = 41, uncore_iio_4 = 38, uprobe = 7, cpu = 4, uncore_cha_4 = 28, uncore_iio_2 = 36, cstate_core = 53, breakpoint = 5, uncore_cha_2 = 26, uncore_irp_4 = 50, uncore_m3upi_1 = 22, uncore_iio_0 = 34, tracepoint = 2, uncore_cha_0 = 24, uncore_irp_2 = 48, cstate_pkg = 54, uncore_imc_4 = 17, uncore_cha_9 = 33, uncore_iio_free_running_4 = 44, uncore_ubox = 23, uncore_irp_0 = 46, uncore_m2m_0 = 11, uncore_imc_2 = 15, kprobe = 6, uncore_cha_7 = 31, uncore_iio_free_running_2 = 42, uncore_iio_5 = 39, uncore_imc_0 = 13, uncore_upi_0 = 19, uncore_cha_5 = 29, uncore_iio_free_running_0 = 40, uncore_pcu = 52, msr = 9, uncore_iio_3 = 37, uncore_cha_3 = 27, uncore_irp_5 = 51, uncore_iio_1 = 35
# CPU cache info:
# L1 Data 32K [0,20]
# L1 Instruction 32K [0,20]
+# L1 Data 32K [1,21]
+# L1 Instruction 32K [1,21]
# L1 Data 32K [2,22]
# L1 Instruction 32K [2,22]
# L1 Data 32K [3,23]
@@ -123,9 +125,8 @@
# L1 Instruction 32K [18,38]
# L1 Data 32K [19,39]
# L1 Instruction 32K [19,39]
-# L1 Data 32K [21]
-# L1 Instruction 32K [21]
# L2 Unified 1024K [0,20]
+# L2 Unified 1024K [1,21]
# L2 Unified 1024K [2,22]
# L2 Unified 1024K [3,23]
# L2 Unified 1024K [4,24]
@@ -144,12 +145,11 @@
# L2 Unified 1024K [17,37]
# L2 Unified 1024K [18,38]
# L2 Unified 1024K [19,39]
-# L2 Unified 1024K [21]
# L3 Unified 14080K [0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38]
-# L3 Unified 14080K [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39]
...
On 8/12/2019 4:12 PM, Jiri Olsa wrote:
> On Fri, Aug 02, 2019 at 12:16:53AM +0530, Mukesh Ojha wrote:
>> Perf framework doesn't allow preserving CPU events across
>> CPU hotplugs. The events are scheduled out as and when the
>> CPU walks offline. Moreover, the framework also doesn't
>> allow the clients to create events on an offline CPU. As
>> a result, the clients have to keep on monitoring the CPU
>> state until it comes back online.
>>
>> Therefore, introducing the perf framework to support creation
>> and preserving of (CPU) events for offline CPUs. Through
>> this, the CPU's online state would be transparent to the
>> client and it not have to worry about monitoring the CPU's
>> state. Success would be returned to the client even while
>> creating the event on an offline CPU. If during the lifetime
>> of the event the CPU walks offline, the event would be
>> preserved and would continue to count as soon as (and if) the
>> CPU comes back online.
>>
>> Co-authored-by: Peter Zijlstra <[email protected]>
>> Signed-off-by: Raghavendra Rao Ananta <[email protected]>
>> Signed-off-by: Mukesh Ojha <[email protected]>
>> Cc: Peter Zijlstra <[email protected]>
>> Cc: Ingo Molnar <[email protected]>
>> Cc: Arnaldo Carvalho de Melo <[email protected]>
>> Cc: Alexander Shishkin <[email protected]>
>> Cc: Jiri Olsa <[email protected]>
>> Cc: Alexei Starovoitov <[email protected]>
>> ---
>> Change in V5:
>> =============
>> - Rebased it.
> note that we might need to change how we store cpu topology,
> now that it can change during the sampling.. like below it's
> the comparison of header data with and without cpu 1
>
> I think some of the report code checks on topology or caches
> and it might get confused
>
> perhaps we could watch cpu topology in record and update the
> data as we see it changing.. future TODO list ;-)
Thanks Jiri for pointing it out.
I will take a look at perf record header update code.
It would be good, if we get more reviews on this patch as it is core
event framework change.
-Mukesh
>
> perf stat is probably fine
>
> jirka
>
>
> ---
> -# nrcpus online : 39
> +# nrcpus online : 40
> # nrcpus avail : 40
> # cpudesc : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
> # cpuid : GenuineIntel,6,85,4
> ...
> # sibling sockets : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> -# sibling sockets : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> +# sibling sockets : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> # sibling dies : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> -# sibling dies : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> +# sibling dies : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> # sibling threads : 0,20
> +# sibling threads : 1,21
> # sibling threads : 2,22
> # sibling threads : 3,23
> # sibling threads : 4,24
> @@ -38,9 +39,8 @@
> # sibling threads : 17,37
> # sibling threads : 18,38
> # sibling threads : 19,39
> -# sibling threads : 21
> # CPU 0: Core ID 0, Die ID 0, Socket ID 0
> -# CPU 1: Core ID -1, Die ID -1, Socket ID -1
> +# CPU 1: Core ID 0, Die ID 0, Socket ID 1
> # CPU 2: Core ID 4, Die ID 0, Socket ID 0
> # CPU 3: Core ID 4, Die ID 0, Socket ID 1
> # CPU 4: Core ID 1, Die ID 0, Socket ID 0
> @@ -79,14 +79,16 @@
> # CPU 37: Core ID 9, Die ID 0, Socket ID 1
> # CPU 38: Core ID 10, Die ID 0, Socket ID 0
> # CPU 39: Core ID 10, Die ID 0, Socket ID 1
> -# node0 meminfo : total = 47391616 kB, free = 46536844 kB
> +# node0 meminfo : total = 47391616 kB, free = 46548348 kB
> # node0 cpu list : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> -# node1 meminfo : total = 49539612 kB, free = 48908820 kB
> -# node1 cpu list : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> +# node1 meminfo : total = 49539612 kB, free = 48897176 kB
> +# node1 cpu list : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> # pmu mappings: intel_pt = 8, uncore_cha_1 = 25, uncore_irp_3 = 49, software = 1, uncore_imc_5 = 18, uncore_m3upi_0 = 21, uncore_iio_free_running_5 = 45, uncore_irp_1 = 47, uncore_m2m_1 = 12, uncore_imc_3 = 16, uncore_cha_8 = 32, uncore_iio_free_running_3 = 43, uncore_imc_1 = 14, uncore_upi_1 = 20, power = 10, uncore_cha_6 = 30, uncore_iio_free_running_1 = 41, uncore_iio_4 = 38, uprobe = 7, cpu = 4, uncore_cha_4 = 28, uncore_iio_2 = 36, cstate_core = 53, breakpoint = 5, uncore_cha_2 = 26, uncore_irp_4 = 50, uncore_m3upi_1 = 22, uncore_iio_0 = 34, tracepoint = 2, uncore_cha_0 = 24, uncore_irp_2 = 48, cstate_pkg = 54, uncore_imc_4 = 17, uncore_cha_9 = 33, uncore_iio_free_running_4 = 44, uncore_ubox = 23, uncore_irp_0 = 46, uncore_m2m_0 = 11, uncore_imc_2 = 15, kprobe = 6, uncore_cha_7 = 31, uncore_iio_free_running_2 = 42, uncore_iio_5 = 39, uncore_imc_0 = 13, uncore_upi_0 = 19, uncore_cha_5 = 29, uncore_iio_free_running_0 = 40, uncore_pcu = 52, msr = 9, uncore_iio_3 = 37, uncore_cha_3
> = 27, uncore_irp_5 = 51, uncore_iio_1 = 35
> # CPU cache info:
> # L1 Data 32K [0,20]
> # L1 Instruction 32K [0,20]
> +# L1 Data 32K [1,21]
> +# L1 Instruction 32K [1,21]
> # L1 Data 32K [2,22]
> # L1 Instruction 32K [2,22]
> # L1 Data 32K [3,23]
> @@ -123,9 +125,8 @@
> # L1 Instruction 32K [18,38]
> # L1 Data 32K [19,39]
> # L1 Instruction 32K [19,39]
> -# L1 Data 32K [21]
> -# L1 Instruction 32K [21]
> # L2 Unified 1024K [0,20]
> +# L2 Unified 1024K [1,21]
> # L2 Unified 1024K [2,22]
> # L2 Unified 1024K [3,23]
> # L2 Unified 1024K [4,24]
> @@ -144,12 +145,11 @@
> # L2 Unified 1024K [17,37]
> # L2 Unified 1024K [18,38]
> # L2 Unified 1024K [19,39]
> -# L2 Unified 1024K [21]
> # L3 Unified 14080K [0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38]
> -# L3 Unified 14080K [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39]
> ...
On 8/12/2019 4:12 PM, Jiri Olsa wrote:
> On Fri, Aug 02, 2019 at 12:16:53AM +0530, Mukesh Ojha wrote:
>> Perf framework doesn't allow preserving CPU events across
>> CPU hotplugs. The events are scheduled out as and when the
>> CPU walks offline. Moreover, the framework also doesn't
>> allow the clients to create events on an offline CPU. As
>> a result, the clients have to keep on monitoring the CPU
>> state until it comes back online.
>>
>> Therefore, introducing the perf framework to support creation
>> and preserving of (CPU) events for offline CPUs. Through
>> this, the CPU's online state would be transparent to the
>> client and it not have to worry about monitoring the CPU's
>> state. Success would be returned to the client even while
>> creating the event on an offline CPU. If during the lifetime
>> of the event the CPU walks offline, the event would be
>> preserved and would continue to count as soon as (and if) the
>> CPU comes back online.
>>
>> Co-authored-by: Peter Zijlstra <[email protected]>
>> Signed-off-by: Raghavendra Rao Ananta <[email protected]>
>> Signed-off-by: Mukesh Ojha <[email protected]>
>> Cc: Peter Zijlstra <[email protected]>
>> Cc: Ingo Molnar <[email protected]>
>> Cc: Arnaldo Carvalho de Melo <[email protected]>
>> Cc: Alexander Shishkin <[email protected]>
>> Cc: Jiri Olsa <[email protected]>
>> Cc: Alexei Starovoitov <[email protected]>
>> ---
>> Change in V5:
>> =============
>> - Rebased it.
> note that we might need to change how we store cpu topology,
> now that it can change during the sampling.. like below it's
> the comparison of header data with and without cpu 1
>
> I think some of the report code checks on topology or caches
> and it might get confused
>
> perhaps we could watch cpu topology in record and update the
> data as we see it changing.. future TODO list ;-)
Hi Jiri,
Can we do something like below to address issue related to header
update while perf record with offline cpus.
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1432,7 +1432,7 @@ static int __cmd_record(struct record *rec, int
argc, const char **argv)
opts->no_bpf_event = true;
}
- err = record__synthesize(rec, false);
+ err = record__synthesize(rec, true);
if (err < 0)
goto out_child;
@@ -1652,7 +1652,7 @@ static int __cmd_record(struct record *rec, int
argc, const char **argv)
} else
status = err;
- record__synthesize(rec, true);
+ record__synthesize(rec, false);
/* this will be recalculated during process_buildids() */
rec->samples = 0;
Thanks.
Mukesh
>
> perf stat is probably fine
>
> jirka
>
>
> ---
> -# nrcpus online : 39
> +# nrcpus online : 40
> # nrcpus avail : 40
> # cpudesc : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
> # cpuid : GenuineIntel,6,85,4
> ...
> # sibling sockets : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> -# sibling sockets : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> +# sibling sockets : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> # sibling dies : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> -# sibling dies : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> +# sibling dies : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> # sibling threads : 0,20
> +# sibling threads : 1,21
> # sibling threads : 2,22
> # sibling threads : 3,23
> # sibling threads : 4,24
> @@ -38,9 +39,8 @@
> # sibling threads : 17,37
> # sibling threads : 18,38
> # sibling threads : 19,39
> -# sibling threads : 21
> # CPU 0: Core ID 0, Die ID 0, Socket ID 0
> -# CPU 1: Core ID -1, Die ID -1, Socket ID -1
> +# CPU 1: Core ID 0, Die ID 0, Socket ID 1
> # CPU 2: Core ID 4, Die ID 0, Socket ID 0
> # CPU 3: Core ID 4, Die ID 0, Socket ID 1
> # CPU 4: Core ID 1, Die ID 0, Socket ID 0
> @@ -79,14 +79,16 @@
> # CPU 37: Core ID 9, Die ID 0, Socket ID 1
> # CPU 38: Core ID 10, Die ID 0, Socket ID 0
> # CPU 39: Core ID 10, Die ID 0, Socket ID 1
> -# node0 meminfo : total = 47391616 kB, free = 46536844 kB
> +# node0 meminfo : total = 47391616 kB, free = 46548348 kB
> # node0 cpu list : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> -# node1 meminfo : total = 49539612 kB, free = 48908820 kB
> -# node1 cpu list : 3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> +# node1 meminfo : total = 49539612 kB, free = 48897176 kB
> +# node1 cpu list : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
> # pmu mappings: intel_pt = 8, uncore_cha_1 = 25, uncore_irp_3 = 49, software = 1, uncore_imc_5 = 18, uncore_m3upi_0 = 21, uncore_iio_free_running_5 = 45, uncore_irp_1 = 47, uncore_m2m_1 = 12, uncore_imc_3 = 16, uncore_cha_8 = 32, uncore_iio_free_running_3 = 43, uncore_imc_1 = 14, uncore_upi_1 = 20, power = 10, uncore_cha_6 = 30, uncore_iio_free_running_1 = 41, uncore_iio_4 = 38, uprobe = 7, cpu = 4, uncore_cha_4 = 28, uncore_iio_2 = 36, cstate_core = 53, breakpoint = 5, uncore_cha_2 = 26, uncore_irp_4 = 50, uncore_m3upi_1 = 22, uncore_iio_0 = 34, tracepoint = 2, uncore_cha_0 = 24, uncore_irp_2 = 48, cstate_pkg = 54, uncore_imc_4 = 17, uncore_cha_9 = 33, uncore_iio_free_running_4 = 44, uncore_ubox = 23, uncore_irp_0 = 46, uncore_m2m_0 = 11, uncore_imc_2 = 15, kprobe = 6, uncore_cha_7 = 31, uncore_iio_free_running_2 = 42, uncore_iio_5 = 39, uncore_imc_0 = 13, uncore_upi_0 = 19, uncore_cha_5 = 29, uncore_iio_free_running_0 = 40, uncore_pcu = 52, msr = 9, uncore_iio_3 = 37, uncore_cha_3 = 27, uncore_irp_5 = 51, uncore_iio_1 = 35
> # CPU cache info:
> # L1 Data 32K [0,20]
> # L1 Instruction 32K [0,20]
> +# L1 Data 32K [1,21]
> +# L1 Instruction 32K [1,21]
> # L1 Data 32K [2,22]
> # L1 Instruction 32K [2,22]
> # L1 Data 32K [3,23]
> @@ -123,9 +125,8 @@
> # L1 Instruction 32K [18,38]
> # L1 Data 32K [19,39]
> # L1 Instruction 32K [19,39]
> -# L1 Data 32K [21]
> -# L1 Instruction 32K [21]
> # L2 Unified 1024K [0,20]
> +# L2 Unified 1024K [1,21]
> # L2 Unified 1024K [2,22]
> # L2 Unified 1024K [3,23]
> # L2 Unified 1024K [4,24]
> @@ -144,12 +145,11 @@
> # L2 Unified 1024K [17,37]
> # L2 Unified 1024K [18,38]
> # L2 Unified 1024K [19,39]
> -# L2 Unified 1024K [21]
> # L3 Unified 14080K [0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38]
> -# L3 Unified 14080K [3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39]
> ...