2020-06-27 08:31:14

by Chen, Rong A

[permalink] [raw]

Subject: [bpf] will-it-scale.per_thread_ops 2.6% improvement

---

Greeting,

FYI, we noticed a 2.6% improvement of will-it-scale.per_thread_ops due to commit:


commit: 492e639f0c222784e2e0f121966375f641c61b15 ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:

nr_task: 100%
mode: thread
test: pthread_mutex2
cpufreq_governor: performance
ucode: 0x11

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops 2.5% improvement |
| test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
| test parameters | cpufreq_governor=performance |
| | mode=process |
| | nr_task=100% |
| | test=malloc2 |
| | ucode=0x11 |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/pthread_mutex2/will-it-scale/0x11

commit:
b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")

b121b341e5983bdc 492e639f0c222784e2e0f121966
---------------- ---------------------------
%stddev %change %stddev
\ | \
6105969 +2.6% 6265564 will-it-scale.per_thread_ops
9855 +1196.5% 127775 ± 92% will-it-scale.time.maximum_resident_set_size
1.759e+09 +2.6% 1.804e+09 will-it-scale.workload
330.50 ± 13% +71.4% 566.50 ± 21% cpuidle.POLL.usage
149.50 +3.3% 154.50 ± 2% proc-vmstat.nr_anon_transparent_hugepages
24750 +2.5% 25362 proc-vmstat.nr_shmem
22056 ± 6% -21.5% 17321 ± 27% proc-vmstat.numa_hint_faults
22056 ± 6% -21.5% 17321 ± 27% proc-vmstat.numa_hint_faults_local
451.64 ± 11% +27.3% 574.96 ± 21% sched_debug.cfs_rq:/.util_avg.min
0.46 ± 18% -58.7% 0.19 ±105% sched_debug.rt_rq:/.rt_time.avg
133.72 ± 18% -58.7% 55.26 ±105% sched_debug.rt_rq:/.rt_time.max
7.87 ± 18% -58.7% 3.25 ±105% sched_debug.rt_rq:/.rt_time.stddev
123790 ± 14% -13.1% 107602 ± 3% softirqs.CPU100.TIMER
124472 ± 14% -13.0% 108248 ± 5% softirqs.CPU142.TIMER
125239 ± 14% -12.4% 109741 ± 5% softirqs.CPU166.TIMER
130036 ± 11% -15.7% 109609 ± 3% softirqs.CPU170.TIMER
123083 ± 12% -14.6% 105115 ± 3% softirqs.CPU207.TIMER
137740 ± 13% -21.5% 108150 ± 4% softirqs.CPU209.TIMER
126203 ± 11% -11.0% 112261 ± 2% softirqs.CPU213.TIMER
126552 ± 13% -14.7% 107985 ± 2% softirqs.CPU226.TIMER
126084 ± 11% -13.6% 108956 ± 4% softirqs.CPU73.TIMER
3.162e+10 +2.8% 3.252e+10 perf-stat.i.branch-instructions
22.58 +0.3 22.90 perf-stat.i.branch-miss-rate%
7.165e+09 +4.3% 7.47e+09 perf-stat.i.branch-misses
82101720 +1.4% 83214202 perf-stat.i.cache-references
4.361e+11 +2.2% 4.455e+11 perf-stat.i.cpu-cycles
38454 ± 2% +5.2% 40469 ± 2% perf-stat.i.cycles-between-cache-misses
3.53 +0.0 3.57 perf-stat.i.iTLB-load-miss-rate%
3.786e+09 +3.8% 3.931e+09 perf-stat.i.iTLB-load-misses
1.032e+11 +2.4% 1.057e+11 perf-stat.i.iTLB-loads
1.028e+11 +2.7% 1.055e+11 perf-stat.i.instructions
27.45 -1.3% 27.08 perf-stat.i.instructions-per-iTLB-miss
1.51 +2.2% 1.55 perf-stat.i.metric.GHz
468.30 +2.5% 480.18 perf-stat.i.metric.M/sec
22.69 +0.3 23.00 perf-stat.overall.branch-miss-rate%
37831 ± 2% +5.2% 39799 perf-stat.overall.cycles-between-cache-misses
3.54 +0.0 3.59 perf-stat.overall.iTLB-load-miss-rate%
27.14 -1.1% 26.84 perf-stat.overall.instructions-per-iTLB-miss
3.152e+10 +2.8% 3.241e+10 perf-stat.ps.branch-instructions
7.152e+09 +4.2% 7.454e+09 perf-stat.ps.branch-misses
82017742 +1.3% 83120324 perf-stat.ps.cache-references
4.348e+11 +2.1% 4.441e+11 perf-stat.ps.cpu-cycles
3.775e+09 +3.8% 3.919e+09 perf-stat.ps.iTLB-load-misses
1.028e+11 +2.4% 1.053e+11 perf-stat.ps.iTLB-loads
1.025e+11 +2.6% 1.052e+11 perf-stat.ps.instructions
3.122e+13 +2.5% 3.2e+13 perf-stat.total.instructions
10.82 -0.1 10.70 perf-profile.calltrace.cycles-pp.pthread_mutex_unlock
0.72 ± 10% +0.2 0.89 ± 10% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
0.79 ± 11% +0.2 0.96 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt
0.84 ± 11% +0.3 1.09 ± 10% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
0.48 ± 58% +0.3 0.76 ± 9% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.11 ± 11% +0.3 1.39 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
1.34 ± 10% +0.3 1.64 ± 11% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
1.47 ± 10% +0.3 1.78 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.__pthread_mutex_unlock_usercnt
0.14 ±173% +0.5 0.60 ± 9% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.64 ± 12% +0.5 2.15 ± 10% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock
2.17 ± 11% +0.6 2.74 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock
2.62 ± 10% +0.6 3.24 ± 11% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.pthread_mutex_lock
2.89 ± 10% +0.6 3.51 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.pthread_mutex_lock
0.14 ±173% +0.7 0.86 ± 10% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
0.47 ± 59% +1.4 1.83 ± 10% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
0.48 ± 59% +1.4 1.90 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
0.67 ± 17% +1.5 2.19 ± 11% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
0.74 ± 12% -0.6 0.17 ± 10% perf-profile.children.cycles-pp.__remove_hrtimer
0.23 ± 7% -0.2 0.08 ± 20% perf-profile.children.cycles-pp.rb_next
0.29 ± 7% -0.2 0.14 ± 11% perf-profile.children.cycles-pp.timerqueue_del
0.41 ± 10% -0.1 0.32 ± 12% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
8.81 -0.1 8.72 perf-profile.children.cycles-pp.pthread_mutex_unlock
0.11 ± 18% -0.0 0.07 ± 30% perf-profile.children.cycles-pp.rcu_irq_enter
0.11 ± 10% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.timerqueue_add
0.12 ± 11% +0.0 0.15 ± 5% perf-profile.children.cycles-pp.enqueue_hrtimer
0.17 ± 14% +0.0 0.21 ± 12% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.07 ± 15% +0.0 0.12 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.06 ± 20% +0.0 0.11 ± 10% perf-profile.children.cycles-pp.cpuacct_charge
0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.native_sched_clock
0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.sched_clock
0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.__calc_delta
0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.sched_clock_cpu
0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.raise_softirq
0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.cpuacct_account_field
0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.reweight_entity
0.01 ±173% +0.1 0.08 ± 6% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.05 ± 58% +0.1 0.12 ± 20% perf-profile.children.cycles-pp.run_timer_softirq
0.01 ±173% +0.1 0.08 ± 29% perf-profile.children.cycles-pp.run_posix_cpu_timers
0.00 +0.1 0.07 ± 11% perf-profile.children.cycles-pp.hrtimer_active
0.00 +0.1 0.08 ± 24% perf-profile.children.cycles-pp.x86_pmu_disable
0.14 ± 15% +0.1 0.22 ± 8% perf-profile.children.cycles-pp.clockevents_program_event
0.09 ± 15% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.06 ± 60% +0.1 0.14 ± 12% perf-profile.children.cycles-pp.account_user_time
0.09 ± 19% +0.1 0.18 ± 7% perf-profile.children.cycles-pp._raw_spin_lock
0.09 ± 20% +0.1 0.18 ± 10% perf-profile.children.cycles-pp.__update_load_avg_se
0.01 ±173% +0.1 0.10 ± 15% perf-profile.children.cycles-pp.profile_tick
0.14 ± 12% +0.1 0.25 ± 11% perf-profile.children.cycles-pp.ktime_get
0.08 ± 21% +0.1 0.21 ± 13% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.24 ± 17% +0.1 0.37 ± 7% perf-profile.children.cycles-pp.update_curr
0.16 ± 11% +0.1 0.30 ± 18% perf-profile.children.cycles-pp.__softirqentry_text_start
0.01 ±173% +0.2 0.17 ± 15% perf-profile.children.cycles-pp.run_local_timers
0.04 ± 59% +0.2 0.20 ± 20% perf-profile.children.cycles-pp.update_rq_clock
0.26 ± 12% +0.2 0.43 ± 17% perf-profile.children.cycles-pp.irq_exit
0.59 ± 17% +0.5 1.14 ± 8% perf-profile.children.cycles-pp.task_tick_fair
0.88 ± 18% +0.9 1.76 ± 9% perf-profile.children.cycles-pp.scheduler_tick
3.39 ± 12% +1.1 4.47 ± 9% perf-profile.children.cycles-pp.__hrtimer_run_queues
4.47 ± 11% +1.2 5.63 ± 10% perf-profile.children.cycles-pp.hrtimer_interrupt
5.38 ± 10% +1.2 6.62 ± 11% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
5.93 ± 10% +1.3 7.19 ± 11% perf-profile.children.cycles-pp.apic_timer_interrupt
1.19 ± 18% +1.3 2.48 ± 10% perf-profile.children.cycles-pp.update_process_times
1.23 ± 18% +1.4 2.58 ± 10% perf-profile.children.cycles-pp.tick_sched_handle
1.39 ± 17% +1.6 2.96 ± 10% perf-profile.children.cycles-pp.tick_sched_timer
0.22 ± 6% -0.1 0.07 ± 21% perf-profile.self.cycles-pp.rb_next
5.98 -0.1 5.85 perf-profile.self.cycles-pp.pthread_mutex_unlock
0.25 ± 10% -0.1 0.14 ± 15% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
0.11 ± 18% -0.1 0.06 ± 65% perf-profile.self.cycles-pp.rcu_irq_enter
0.04 ± 58% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.prepare_exit_to_usermode
0.07 ± 10% +0.0 0.11 ± 14% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.15 ± 16% +0.0 0.19 ± 6% perf-profile.self.cycles-pp.update_curr
0.06 ± 20% +0.0 0.11 ± 10% perf-profile.self.cycles-pp.cpuacct_charge
0.04 ± 58% +0.1 0.09 ± 19% perf-profile.self.cycles-pp.__softirqentry_text_start
0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.raise_softirq
0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.cpuacct_account_field
0.03 ±100% +0.1 0.09 ± 17% perf-profile.self.cycles-pp.run_timer_softirq
0.01 ±173% +0.1 0.08 ± 21% perf-profile.self.cycles-pp.account_user_time
0.08 ± 17% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.01 ±173% +0.1 0.08 ± 29% perf-profile.self.cycles-pp.run_posix_cpu_timers
0.00 +0.1 0.07 ± 11% perf-profile.self.cycles-pp.hrtimer_active
0.00 +0.1 0.08 ± 24% perf-profile.self.cycles-pp.x86_pmu_disable
0.07 ± 20% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.__update_load_avg_se
0.09 ± 19% +0.1 0.18 ± 7% perf-profile.self.cycles-pp._raw_spin_lock
0.01 ±173% +0.1 0.10 ± 14% perf-profile.self.cycles-pp.profile_tick
0.04 ± 58% +0.1 0.13 ± 12% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.03 ±102% +0.1 0.12 ± 13% perf-profile.self.cycles-pp.tick_sched_timer
0.11 ± 11% +0.1 0.21 ± 12% perf-profile.self.cycles-pp.ktime_get
0.00 +0.1 0.13 ± 16% perf-profile.self.cycles-pp.run_local_timers
0.00 +0.1 0.14 ± 22% perf-profile.self.cycles-pp.update_rq_clock
0.11 ± 20% +0.1 0.25 ± 10% perf-profile.self.cycles-pp.task_tick_fair
3759 +52.3% 5725 ± 33% interrupts.CPU0.NMI:Non-maskable_interrupts
3759 +52.3% 5725 ± 33% interrupts.CPU0.PMI:Performance_monitoring_interrupts
4752 ± 34% +40.9% 6695 ± 24% interrupts.CPU1.NMI:Non-maskable_interrupts
4752 ± 34% +40.9% 6695 ± 24% interrupts.CPU1.PMI:Performance_monitoring_interrupts
607.50 ± 12% +38.6% 842.25 ± 12% interrupts.CPU10.RES:Rescheduling_interrupts
4635 ± 34% +62.9% 7551 interrupts.CPU104.NMI:Non-maskable_interrupts
4635 ± 34% +62.9% 7551 interrupts.CPU104.PMI:Performance_monitoring_interrupts
896.00 ± 40% -45.1% 492.25 ± 79% interrupts.CPU106.RES:Rescheduling_interrupts
28.50 ± 54% +860.5% 273.75 ± 64% interrupts.CPU107.RES:Rescheduling_interrupts
115.00 ± 48% -64.8% 40.50 ± 21% interrupts.CPU119.RES:Rescheduling_interrupts
4633 ± 34% +22.5% 5675 ± 33% interrupts.CPU124.NMI:Non-maskable_interrupts
4633 ± 34% +22.5% 5675 ± 33% interrupts.CPU124.PMI:Performance_monitoring_interrupts
3724 +52.3% 5673 ± 33% interrupts.CPU129.NMI:Non-maskable_interrupts
3724 +52.3% 5673 ± 33% interrupts.CPU129.PMI:Performance_monitoring_interrupts
3705 +54.9% 5741 ± 33% interrupts.CPU13.NMI:Non-maskable_interrupts
3705 +54.9% 5741 ± 33% interrupts.CPU13.PMI:Performance_monitoring_interrupts
537.50 ± 57% +136.8% 1273 ± 53% interrupts.CPU130.RES:Rescheduling_interrupts
4635 ± 34% +22.8% 5692 ± 33% interrupts.CPU131.NMI:Non-maskable_interrupts
4635 ± 34% +22.8% 5692 ± 33% interrupts.CPU131.PMI:Performance_monitoring_interrupts
4648 ± 34% +42.1% 6603 ± 24% interrupts.CPU147.NMI:Non-maskable_interrupts
4648 ± 34% +42.1% 6603 ± 24% interrupts.CPU147.PMI:Performance_monitoring_interrupts
3720 +52.9% 5689 ± 33% interrupts.CPU148.NMI:Non-maskable_interrupts
3720 +52.9% 5689 ± 33% interrupts.CPU148.PMI:Performance_monitoring_interrupts
3725 +52.9% 5695 ± 33% interrupts.CPU149.NMI:Non-maskable_interrupts
3725 +52.9% 5695 ± 33% interrupts.CPU149.PMI:Performance_monitoring_interrupts
4626 ± 34% +23.3% 5704 ± 33% interrupts.CPU152.NMI:Non-maskable_interrupts
4626 ± 34% +23.3% 5704 ± 33% interrupts.CPU152.PMI:Performance_monitoring_interrupts
4657 ± 34% +42.4% 6630 ± 24% interrupts.CPU154.NMI:Non-maskable_interrupts
4657 ± 34% +42.4% 6630 ± 24% interrupts.CPU154.PMI:Performance_monitoring_interrupts
260.00 ± 59% -55.9% 114.75 ± 31% interrupts.CPU156.RES:Rescheduling_interrupts
3729 +77.2% 6608 ± 24% interrupts.CPU163.NMI:Non-maskable_interrupts
3729 +77.2% 6608 ± 24% interrupts.CPU163.PMI:Performance_monitoring_interrupts
4634 ± 34% +22.6% 5681 ± 33% interrupts.CPU164.NMI:Non-maskable_interrupts
4634 ± 34% +22.6% 5681 ± 33% interrupts.CPU164.PMI:Performance_monitoring_interrupts
95.25 ± 48% +415.7% 491.25 ± 64% interrupts.CPU164.RES:Rescheduling_interrupts
7422 -48.8% 3798 interrupts.CPU168.NMI:Non-maskable_interrupts
7422 -48.8% 3798 interrupts.CPU168.PMI:Performance_monitoring_interrupts
3717 +52.6% 5672 ± 33% interrupts.CPU173.NMI:Non-maskable_interrupts
3717 +52.6% 5672 ± 33% interrupts.CPU173.PMI:Performance_monitoring_interrupts
4682 ± 34% +21.4% 5682 ± 33% interrupts.CPU179.NMI:Non-maskable_interrupts
4682 ± 34% +21.4% 5682 ± 33% interrupts.CPU179.PMI:Performance_monitoring_interrupts
4675 ± 33% +23.6% 5778 ± 33% interrupts.CPU19.NMI:Non-maskable_interrupts
4675 ± 33% +23.6% 5778 ± 33% interrupts.CPU19.PMI:Performance_monitoring_interrupts
3718 +52.6% 5675 ± 33% interrupts.CPU197.NMI:Non-maskable_interrupts
3718 +52.6% 5675 ± 33% interrupts.CPU197.PMI:Performance_monitoring_interrupts
3801 +77.3% 6740 ± 24% interrupts.CPU2.NMI:Non-maskable_interrupts
3801 +77.3% 6740 ± 24% interrupts.CPU2.PMI:Performance_monitoring_interrupts
3702 +78.6% 6612 ± 24% interrupts.CPU204.NMI:Non-maskable_interrupts
3702 +78.6% 6612 ± 24% interrupts.CPU204.PMI:Performance_monitoring_interrupts
3701 +52.9% 5660 ± 33% interrupts.CPU205.NMI:Non-maskable_interrupts
3701 +52.9% 5660 ± 33% interrupts.CPU205.PMI:Performance_monitoring_interrupts
156.50 ± 49% +160.7% 408.00 ± 18% interrupts.CPU206.RES:Rescheduling_interrupts
4657 ± 34% +21.6% 5662 ± 33% interrupts.CPU208.NMI:Non-maskable_interrupts
4657 ± 34% +21.6% 5662 ± 33% interrupts.CPU208.PMI:Performance_monitoring_interrupts
3725 +51.9% 5659 ± 33% interrupts.CPU209.NMI:Non-maskable_interrupts
3725 +51.9% 5659 ± 33% interrupts.CPU209.PMI:Performance_monitoring_interrupts
4616 ± 34% +23.2% 5684 ± 33% interrupts.CPU210.NMI:Non-maskable_interrupts
4616 ± 34% +23.2% 5684 ± 33% interrupts.CPU210.PMI:Performance_monitoring_interrupts
4632 ± 34% +43.5% 6645 ± 24% interrupts.CPU211.NMI:Non-maskable_interrupts
4632 ± 34% +43.5% 6645 ± 24% interrupts.CPU211.PMI:Performance_monitoring_interrupts
4570 ± 34% +43.5% 6557 ± 24% interrupts.CPU214.NMI:Non-maskable_interrupts
4570 ± 34% +43.5% 6557 ± 24% interrupts.CPU214.PMI:Performance_monitoring_interrupts
3698 +54.0% 5695 ± 33% interrupts.CPU217.NMI:Non-maskable_interrupts
3698 +54.0% 5695 ± 33% interrupts.CPU217.PMI:Performance_monitoring_interrupts
4659 ± 34% +42.1% 6622 ± 24% interrupts.CPU218.NMI:Non-maskable_interrupts
4659 ± 34% +42.1% 6622 ± 24% interrupts.CPU218.PMI:Performance_monitoring_interrupts
4647 ± 35% +42.8% 6636 ± 24% interrupts.CPU219.NMI:Non-maskable_interrupts
4647 ± 35% +42.8% 6636 ± 24% interrupts.CPU219.PMI:Performance_monitoring_interrupts
46.75 ± 39% +387.2% 227.75 ± 78% interrupts.CPU222.RES:Rescheduling_interrupts
18.00 ±102% +212.5% 56.25 ± 71% interrupts.CPU225.RES:Rescheduling_interrupts
4667 ± 34% +21.4% 5668 ± 33% interrupts.CPU231.NMI:Non-maskable_interrupts
4667 ± 34% +21.4% 5668 ± 33% interrupts.CPU231.PMI:Performance_monitoring_interrupts
4660 ± 34% +21.8% 5674 ± 33% interrupts.CPU232.NMI:Non-maskable_interrupts
4660 ± 34% +21.8% 5674 ± 33% interrupts.CPU232.PMI:Performance_monitoring_interrupts
4633 ± 34% +22.6% 5679 ± 33% interrupts.CPU237.NMI:Non-maskable_interrupts
4633 ± 34% +22.6% 5679 ± 33% interrupts.CPU237.PMI:Performance_monitoring_interrupts
4646 ± 34% +21.3% 5638 ± 33% interrupts.CPU238.NMI:Non-maskable_interrupts
4646 ± 34% +21.3% 5638 ± 33% interrupts.CPU238.PMI:Performance_monitoring_interrupts
3719 +52.1% 5657 ± 33% interrupts.CPU245.NMI:Non-maskable_interrupts
3719 +52.1% 5657 ± 33% interrupts.CPU245.PMI:Performance_monitoring_interrupts
4635 ± 34% +22.1% 5660 ± 32% interrupts.CPU255.NMI:Non-maskable_interrupts
4635 ± 34% +22.1% 5660 ± 32% interrupts.CPU255.PMI:Performance_monitoring_interrupts
54.50 ± 28% +242.2% 186.50 ± 54% interrupts.CPU256.RES:Rescheduling_interrupts
39.50 ± 19% +162.0% 103.50 ± 45% interrupts.CPU260.RES:Rescheduling_interrupts
4662 ± 34% +22.9% 5730 ± 33% interrupts.CPU267.NMI:Non-maskable_interrupts
4662 ± 34% +22.9% 5730 ± 33% interrupts.CPU267.PMI:Performance_monitoring_interrupts
4613 ± 34% +24.1% 5724 ± 33% interrupts.CPU269.NMI:Non-maskable_interrupts
4613 ± 34% +24.1% 5724 ± 33% interrupts.CPU269.PMI:Performance_monitoring_interrupts
4669 ± 34% +22.6% 5723 ± 33% interrupts.CPU27.NMI:Non-maskable_interrupts
4669 ± 34% +22.6% 5723 ± 33% interrupts.CPU27.PMI:Performance_monitoring_interrupts
4616 ± 34% +24.1% 5727 ± 33% interrupts.CPU274.NMI:Non-maskable_interrupts
4616 ± 34% +24.1% 5727 ± 33% interrupts.CPU274.PMI:Performance_monitoring_interrupts
3732 +52.4% 5689 ± 33% interrupts.CPU276.NMI:Non-maskable_interrupts
3732 +52.4% 5689 ± 33% interrupts.CPU276.PMI:Performance_monitoring_interrupts
3718 +53.7% 5713 ± 33% interrupts.CPU277.NMI:Non-maskable_interrupts
3718 +53.7% 5713 ± 33% interrupts.CPU277.PMI:Performance_monitoring_interrupts
4639 ± 34% +44.0% 6682 ± 24% interrupts.CPU278.NMI:Non-maskable_interrupts
4639 ± 34% +44.0% 6682 ± 24% interrupts.CPU278.PMI:Performance_monitoring_interrupts
3707 +79.7% 6663 ± 24% interrupts.CPU279.NMI:Non-maskable_interrupts
3707 +79.7% 6663 ± 24% interrupts.CPU279.PMI:Performance_monitoring_interrupts
133.50 ± 44% -80.1% 26.50 ± 36% interrupts.CPU279.RES:Rescheduling_interrupts
3722 +79.1% 6668 ± 24% interrupts.CPU281.NMI:Non-maskable_interrupts
3722 +79.1% 6668 ± 24% interrupts.CPU281.PMI:Performance_monitoring_interrupts
3726 +53.5% 5719 ± 33% interrupts.CPU283.NMI:Non-maskable_interrupts
3726 +53.5% 5719 ± 33% interrupts.CPU283.PMI:Performance_monitoring_interrupts
4642 ± 34% +43.8% 6678 ± 24% interrupts.CPU285.NMI:Non-maskable_interrupts
4642 ± 34% +43.8% 6678 ± 24% interrupts.CPU285.PMI:Performance_monitoring_interrupts
3775 +77.7% 6709 ± 24% interrupts.CPU32.NMI:Non-maskable_interrupts
3775 +77.7% 6709 ± 24% interrupts.CPU32.PMI:Performance_monitoring_interrupts
4681 ± 33% +22.9% 5755 ± 33% interrupts.CPU34.NMI:Non-maskable_interrupts
4681 ± 33% +22.9% 5755 ± 33% interrupts.CPU34.PMI:Performance_monitoring_interrupts
354.25 ± 9% +30.3% 461.75 ± 20% interrupts.CPU36.RES:Rescheduling_interrupts
337.75 ± 4% +30.6% 441.25 ± 19% interrupts.CPU38.RES:Rescheduling_interrupts
4633 ± 34% +23.7% 5729 ± 33% interrupts.CPU48.NMI:Non-maskable_interrupts
4633 ± 34% +23.7% 5729 ± 33% interrupts.CPU48.PMI:Performance_monitoring_interrupts
233.50 ± 18% +47.9% 345.25 ± 7% interrupts.CPU54.RES:Rescheduling_interrupts
4682 ± 34% +22.9% 5756 ± 33% interrupts.CPU6.NMI:Non-maskable_interrupts
4682 ± 34% +22.9% 5756 ± 33% interrupts.CPU6.PMI:Performance_monitoring_interrupts
656.50 ± 17% +54.4% 1013 ± 16% interrupts.CPU6.RES:Rescheduling_interrupts
3726 +53.9% 5736 ± 33% interrupts.CPU64.NMI:Non-maskable_interrupts
3726 +53.9% 5736 ± 33% interrupts.CPU64.PMI:Performance_monitoring_interrupts
3724 +53.3% 5708 ± 33% interrupts.CPU65.NMI:Non-maskable_interrupts
3724 +53.3% 5708 ± 33% interrupts.CPU65.PMI:Performance_monitoring_interrupts
4543 ± 33% +23.7% 5618 ± 33% interrupts.CPU71.NMI:Non-maskable_interrupts
4543 ± 33% +23.7% 5618 ± 33% interrupts.CPU71.PMI:Performance_monitoring_interrupts
902.50 ± 69% -68.4% 285.00 ± 27% interrupts.CPU72.RES:Rescheduling_interrupts
4647 ± 34% +22.2% 5677 ± 33% interrupts.CPU73.NMI:Non-maskable_interrupts
4647 ± 34% +22.2% 5677 ± 33% interrupts.CPU73.PMI:Performance_monitoring_interrupts
3736 +77.2% 6622 ± 24% interrupts.CPU79.NMI:Non-maskable_interrupts
3736 +77.2% 6622 ± 24% interrupts.CPU79.PMI:Performance_monitoring_interrupts
4705 ± 34% +22.2% 5750 ± 33% interrupts.CPU8.NMI:Non-maskable_interrupts
4705 ± 34% +22.2% 5750 ± 33% interrupts.CPU8.PMI:Performance_monitoring_interrupts
4666 ± 34% +41.7% 6613 ± 24% interrupts.CPU83.NMI:Non-maskable_interrupts
4666 ± 34% +41.7% 6613 ± 24% interrupts.CPU83.PMI:Performance_monitoring_interrupts
4644 ± 34% +22.3% 5680 ± 33% interrupts.CPU84.NMI:Non-maskable_interrupts
4644 ± 34% +22.3% 5680 ± 33% interrupts.CPU84.PMI:Performance_monitoring_interrupts
4653 ± 35% +22.1% 5682 ± 33% interrupts.CPU86.NMI:Non-maskable_interrupts
4653 ± 35% +22.1% 5682 ± 33% interrupts.CPU86.PMI:Performance_monitoring_interrupts
3712 +52.5% 5660 ± 33% interrupts.CPU87.NMI:Non-maskable_interrupts
3712 +52.5% 5660 ± 33% interrupts.CPU87.PMI:Performance_monitoring_interrupts
3741 +77.5% 6640 ± 24% interrupts.CPU88.NMI:Non-maskable_interrupts
3741 +77.5% 6640 ± 24% interrupts.CPU88.PMI:Performance_monitoring_interrupts
1278 ± 17% -49.1% 650.50 ± 51% interrupts.CPU88.RES:Rescheduling_interrupts
3716 +77.9% 6611 ± 24% interrupts.CPU89.NMI:Non-maskable_interrupts
3716 +77.9% 6611 ± 24% interrupts.CPU89.PMI:Performance_monitoring_interrupts
511.25 ± 14% +63.9% 837.75 ± 33% interrupts.CPU9.RES:Rescheduling_interrupts
4648 ± 34% +42.8% 6639 ± 24% interrupts.CPU91.NMI:Non-maskable_interrupts
4648 ± 34% +42.8% 6639 ± 24% interrupts.CPU91.PMI:Performance_monitoring_interrupts
4648 ± 34% +22.4% 5689 ± 33% interrupts.CPU92.NMI:Non-maskable_interrupts
4648 ± 34% +22.4% 5689 ± 33% interrupts.CPU92.PMI:Performance_monitoring_interrupts
3716 +103.5% 7565 interrupts.CPU95.NMI:Non-maskable_interrupts
3716 +103.5% 7565 interrupts.CPU95.PMI:Performance_monitoring_interrupts
4639 ± 34% +22.1% 5664 ± 33% interrupts.CPU96.NMI:Non-maskable_interrupts
4639 ± 34% +22.1% 5664 ± 33% interrupts.CPU96.PMI:Performance_monitoring_interrupts
3722 +53.1% 5699 ± 33% interrupts.CPU98.NMI:Non-maskable_interrupts
3722 +53.1% 5699 ± 33% interrupts.CPU98.PMI:Performance_monitoring_interrupts



will-it-scale.per_thread_ops

7e+06 +-------------------------------------------------------------------+
| O O O O O O O O O O O O O O O O O O O |
6e+06 |.+..+.+.+ +.+..+.+.+..+.+.+..+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.|
| : : |
5e+06 |-+ : : |
| : : |
4e+06 |-+ : : |
| : : |
3e+06 |-+ : : |
| : : |
2e+06 |-+ : : |
| : : |
1e+06 |-+ : |
| : |
0 +-------------------------------------------------------------------+


will-it-scale.workload

2e+09 +-----------------------------------------------------------------+
1.8e+09 |-O O O O O O O O O O O O O O O O O O O O |
|.+..+.+.+ +.+.+.+..+.+.+..+.+.+.+..+.+.+..+.+.+.+..+.+.+.+..+.|
1.6e+09 |-+ : : |
1.4e+09 |-+ : : |
| : : |
1.2e+09 |-+ : : |
1e+09 |-+ : : |
8e+08 |-+ : : |
| : : |
6e+08 |-+ : : |
4e+08 |-+ :: |
| :: |
2e+08 |-+ : |
0 +-----------------------------------------------------------------+


[*] bisect-good sample
[O] bisect-bad sample

***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/malloc2/will-it-scale/0x11

commit:
b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")

b121b341e5983bdc 492e639f0c222784e2e0f121966
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
4:4 -100% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
%stddev %change %stddev
\ | \
969613 +2.5% 993613 will-it-scale.per_process_ops
9816 +1.7% 9984 will-it-scale.time.maximum_resident_set_size
2.792e+08 +2.5% 2.862e+08 will-it-scale.workload
329.25 ± 13% +43.7% 473.00 cpuidle.POLL.usage
7397888 ± 5% +17.6% 8699904 meminfo.DirectMap2M
2259 ± 2% +7.5% 2429 vmstat.system.cs
5771 ± 5% -7.8% 5323 numa-vmstat.node1.nr_slab_reclaimable
3003 ± 13% +34.0% 4024 numa-vmstat.node1.nr_slab_unreclaimable
23875 +3.1% 24612 proc-vmstat.nr_shmem
1959 +3.3% 2024 proc-vmstat.pgactivate
23087 ± 5% -7.8% 21292 numa-meminfo.node1.KReclaimable
23087 ± 5% -7.8% 21292 numa-meminfo.node1.SReclaimable
12013 ± 13% +34.0% 16096 numa-meminfo.node1.SUnreclaim
1094 ± 9% +20.5% 1319 slabinfo.dmaengine-unmap-16.active_objs
1094 ± 9% +20.5% 1319 slabinfo.dmaengine-unmap-16.num_objs
1573 ± 4% -18.9% 1275 slabinfo.file_lock_cache.active_objs
1573 ± 4% -18.9% 1275 slabinfo.file_lock_cache.num_objs
20496 ± 2% -10.1% 18432 slabinfo.skbuff_head_cache.active_objs
20496 ± 2% -10.1% 18432 slabinfo.skbuff_head_cache.num_objs
4.371e+10 +2.6% 4.486e+10 perf-stat.i.branch-instructions
4.12 -0.1 4.05 perf-stat.i.branch-miss-rate%
2.36 -0.0 2.31 perf-stat.i.cache-miss-rate%
8.193e+08 +2.1% 8.365e+08 perf-stat.i.cache-references
2224 ± 2% +7.9% 2400 perf-stat.i.context-switches
4.33e+11 +2.3% 4.428e+11 perf-stat.i.cpu-cycles
279.15 -1.5% 275.01 perf-stat.i.cpu-migrations
9.822e+08 +1.7% 9.991e+08 perf-stat.i.iTLB-load-misses
1.56e+11 +2.3% 1.596e+11 perf-stat.i.iTLB-loads
1.554e+11 +2.6% 1.594e+11 perf-stat.i.instructions
1.52 +2.2% 1.55 perf-stat.i.metric.GHz
0.03 +3.7% 0.03 perf-stat.i.metric.K/sec
701.66 +2.3% 717.66 perf-stat.i.metric.M/sec
2740 +2.8% 2816 perf-stat.i.minor-faults
2740 +2.8% 2816 perf-stat.i.page-faults
4.10 -0.1 4.04 perf-stat.overall.branch-miss-rate%
2.24 -0.1 2.19 perf-stat.overall.cache-miss-rate%
4.36e+10 +2.6% 4.473e+10 perf-stat.ps.branch-instructions
8.182e+08 +2.1% 8.35e+08 perf-stat.ps.cache-references
2185 ± 2% +7.9% 2359 perf-stat.ps.context-switches
4.32e+11 +2.3% 4.418e+11 perf-stat.ps.cpu-cycles
269.08 -1.6% 264.85 perf-stat.ps.cpu-migrations
9.792e+08 +1.7% 9.958e+08 perf-stat.ps.iTLB-load-misses
1.556e+11 +2.3% 1.591e+11 perf-stat.ps.iTLB-loads
1.55e+11 +2.5% 1.59e+11 perf-stat.ps.instructions
2694 +2.5% 2760 perf-stat.ps.minor-faults
2694 +2.5% 2760 perf-stat.ps.page-faults
4.754e+13 +2.7% 4.882e+13 perf-stat.total.instructions
3888 ± 8% -14.7% 3317 sched_debug.cfs_rq:/.load.avg
137829 ± 62% -80.9% 26264 sched_debug.cfs_rq:/.load.max
2980 -19.8% 2390 sched_debug.cfs_rq:/.load.min
9697 ± 53% -75.3% 2397 sched_debug.cfs_rq:/.load.stddev
3.52 ± 9% -15.9% 2.97 sched_debug.cfs_rq:/.load_avg.avg
105.71 ± 63% -72.1% 29.50 sched_debug.cfs_rq:/.load_avg.max
8.21 ± 50% -66.4% 2.76 sched_debug.cfs_rq:/.load_avg.stddev
1.30 -13.4% 1.12 sched_debug.cfs_rq:/.nr_spread_over.avg
407305 ± 5% +17.4% 478222 sched_debug.cfs_rq:/.spread0.max
-5404888 -3.2% -5231353 sched_debug.cfs_rq:/.spread0.min
0.83 +160.0% 2.17 sched_debug.cfs_rq:/.util_est_enqueued.min
7576 -9.1% 6888 sched_debug.cpu.curr->pid.max
335.97 ± 5% +11.0% 373.05 sched_debug.cpu.curr->pid.stddev
23888 ± 52% +214.7% 75185 sched_debug.cpu.nr_switches.max
878.67 ± 17% -25.1% 658.33 sched_debug.cpu.nr_switches.min
2016 ± 30% +129.0% 4618 sched_debug.cpu.nr_switches.stddev
0.00 ± 20% -100.0% 0.00 sched_debug.cpu.nr_uninterruptible.avg
16513 ± 86% +329.7% 70964 sched_debug.cpu.sched_count.max
1557 ± 42% +176.4% 4305 sched_debug.cpu.sched_count.stddev
2278 +29.0% 2939 sched_debug.cpu.sched_goidle.max
202.41 +11.1% 224.95 sched_debug.cpu.sched_goidle.stddev
6738 ± 44% +230.4% 22261 sched_debug.cpu.ttwu_count.max
673.86 ± 23% +112.5% 1431 sched_debug.cpu.ttwu_count.stddev
4944 ± 77% +343.2% 21913 sched_debug.cpu.ttwu_local.max
463.98 ± 43% +189.1% 1341 sched_debug.cpu.ttwu_local.stddev
0.20 ± 74% -100.0% 0.00 sched_debug.rt_rq:/.rt_time.avg
57.79 ± 74% -100.0% 0.00 sched_debug.rt_rq:/.rt_time.max
3.40 ± 74% -100.0% 0.00 sched_debug.rt_rq:/.rt_time.stddev
50.90 ± 10% -11.0 39.91 perf-profile.calltrace.cycles-pp.malloc
17.95 ± 10% -3.5 14.47 perf-profile.calltrace.cycles-pp.free
2.32 ± 30% -1.5 0.84 perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
2.02 ± 14% -1.0 1.02 perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.05 ± 28% -0.3 0.75 perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
0.65 ± 17% +0.3 0.97 perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
0.13 ±173% +0.4 0.55 perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.89 ± 13% +0.4 2.34 perf-profile.calltrace.cycles-pp.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop.prepare_exit_to_usermode
1.89 ± 13% +0.5 2.34 perf-profile.calltrace.cycles-pp.change_protection.change_prot_numa.task_numa_work.task_work_run.exit_to_usermode_loop
0.30 ±101% +0.5 0.75 perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.92 ± 13% +0.5 2.38 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
1.90 ± 13% +0.5 2.36 perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
0.00 +0.5 0.51 perf-profile.calltrace.cycles-pp.__acct_update_integrals.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
0.00 +0.6 0.55 perf-profile.calltrace.cycles-pp.__run_timers.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
0.00 +0.6 0.60 perf-profile.calltrace.cycles-pp.cpuacct_charge.update_curr.task_tick_fair.scheduler_tick.update_process_times
0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp._raw_spin_lock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
0.13 ±173% +0.7 0.84 perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.task_tick_fair.scheduler_tick.update_process_times
0.13 ±173% +0.8 0.91 perf-profile.calltrace.cycles-pp.__update_load_avg_se.update_load_avg.task_tick_fair.scheduler_tick.update_process_times
0.00 +0.9 0.89 perf-profile.calltrace.cycles-pp.run_local_timers.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
0.00 +0.9 0.95 perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
0.00 +1.0 0.98 perf-profile.calltrace.cycles-pp.account_user_time.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
1.02 ± 22% +1.0 2.01 perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
0.00 +1.0 0.99 perf-profile.calltrace.cycles-pp.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
0.00 +1.0 1.00 perf-profile.calltrace.cycles-pp.ksoftirqd_running.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
2.62 ± 16% +1.0 3.63 perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.00 +1.1 1.11 perf-profile.calltrace.cycles-pp.run_posix_cpu_timers.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
0.00 +1.1 1.15 perf-profile.calltrace.cycles-pp.update_rq_clock.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
1.15 ± 23% +1.6 2.78 perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
0.71 ± 36% +1.8 2.54 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
1.20 ± 30% +2.8 4.00 perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
2.75 ± 24% +3.7 6.47 perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
4.15 ± 25% +6.5 10.63 perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
6.49 ± 18% +8.8 15.25 perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
14.71 ± 23% +8.8 23.52 perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
19.68 ± 24% +9.5 29.17 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
6.81 ± 19% +9.8 16.62 perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
7.62 ± 19% +10.9 18.52 perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
26.15 ± 25% +13.0 39.11 perf-profile.calltrace.cycles-pp.apic_timer_interrupt
23.59 ± 25% +13.2 36.78 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
51.00 ± 10% -11.0 40.04 perf-profile.children.cycles-pp.malloc
16.65 ± 10% -3.3 13.31 perf-profile.children.cycles-pp.free
2.71 ± 25% -1.8 0.92 perf-profile.children.cycles-pp.__remove_hrtimer
2.36 ± 11% -1.2 1.15 perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
1.86 ± 12% -0.4 1.43 perf-profile.children.cycles-pp.malloc@plt
1.26 ± 23% -0.4 0.84 perf-profile.children.cycles-pp.timerqueue_del
0.88 ± 29% -0.4 0.48 perf-profile.children.cycles-pp.rb_next
0.67 ± 28% -0.3 0.36 perf-profile.children.cycles-pp.rcu_irq_enter
0.17 ± 12% -0.1 0.05 perf-profile.children.cycles-pp.__intel_pmu_enable_all
0.37 ± 12% -0.0 0.32 perf-profile.children.cycles-pp.native_write_msr
0.25 ± 9% -0.0 0.21 perf-profile.children.cycles-pp.native_apic_msr_eoi_write
0.14 ± 5% -0.0 0.12 perf-profile.children.cycles-pp.irq_work_tick
0.07 ± 13% +0.0 0.10 perf-profile.children.cycles-pp.irq_work_run_list
0.00 +0.1 0.05 perf-profile.children.cycles-pp.account_entity_enqueue
0.00 +0.1 0.05 perf-profile.children.cycles-pp.rebalance_domains
0.00 +0.1 0.05 perf-profile.children.cycles-pp.account_process_tick
0.14 ± 19% +0.1 0.19 perf-profile.children.cycles-pp.ret_from_intr
0.30 ± 7% +0.1 0.36 perf-profile.children.cycles-pp.vm_normal_page
0.18 ± 13% +0.1 0.24 perf-profile.children.cycles-pp.fpregs_assert_state_consistent
0.00 +0.1 0.06 perf-profile.children.cycles-pp.rcu_qs
0.17 ± 9% +0.1 0.24 perf-profile.children.cycles-pp.x86_pmu_disable
0.01 ±173% +0.1 0.08 perf-profile.children.cycles-pp.nohz_balance_exit_idle
0.00 +0.1 0.07 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.00 +0.1 0.07 perf-profile.children.cycles-pp.__note_gp_changes
0.23 ± 19% +0.1 0.30 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
0.01 ±173% +0.1 0.10 perf-profile.children.cycles-pp.update_min_vruntime
0.03 ±102% +0.1 0.13 perf-profile.children.cycles-pp.account_entity_dequeue
0.11 ± 17% +0.1 0.22 perf-profile.children.cycles-pp.update_cfs_group
0.10 ± 15% +0.1 0.20 perf-profile.children.cycles-pp.native_read_msr
0.39 ± 11% +0.1 0.50 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.28 ± 18% +0.1 0.39 perf-profile.children.cycles-pp.lapic_next_deadline
0.09 ± 19% +0.1 0.20 perf-profile.children.cycles-pp.__accumulate_pelt_segments
0.39 ± 9% +0.1 0.50 perf-profile.children.cycles-pp.timerqueue_add
0.48 ± 12% +0.1 0.61 perf-profile.children.cycles-pp.enqueue_hrtimer
0.11 ± 23% +0.1 0.25 perf-profile.children.cycles-pp.__calc_delta
0.05 ± 67% +0.2 0.21 perf-profile.children.cycles-pp.note_gp_changes
0.14 ± 27% +0.2 0.31 perf-profile.children.cycles-pp.trigger_load_balance
0.02 ±173% +0.2 0.19 perf-profile.children.cycles-pp.calc_global_load_tick
0.09 ± 17% +0.2 0.27 perf-profile.children.cycles-pp.native_sched_clock
0.07 ± 6% +0.2 0.25 perf-profile.children.cycles-pp.hrtimer_run_queues
0.18 ± 18% +0.2 0.37 perf-profile.children.cycles-pp.arch_scale_freq_tick
0.10 ± 17% +0.2 0.29 perf-profile.children.cycles-pp.sched_clock
0.16 ± 23% +0.2 0.39 perf-profile.children.cycles-pp.reweight_entity
0.68 ± 13% +0.2 0.91 perf-profile.children.cycles-pp.ktime_get
0.11 ± 17% +0.2 0.34 perf-profile.children.cycles-pp.sched_clock_cpu
0.61 ± 17% +0.2 0.85 perf-profile.children.cycles-pp.clockevents_program_event
0.21 ± 7% +0.3 0.50 perf-profile.children.cycles-pp.cpuacct_account_field
0.09 ± 19% +0.3 0.42 perf-profile.children.cycles-pp.hrtimer_active
2.15 ± 8% +0.3 2.48 perf-profile.children.cycles-pp.change_p4d_range
2.13 ± 8% +0.3 2.46 perf-profile.children.cycles-pp.change_pte_range
0.81 ± 14% +0.3 1.15 perf-profile.children.cycles-pp.ktime_get_update_offsets_now
2.15 ± 8% +0.3 2.49 perf-profile.children.cycles-pp.change_protection
2.15 ± 8% +0.3 2.49 perf-profile.children.cycles-pp.change_prot_numa
0.15 ± 13% +0.3 0.49 perf-profile.children.cycles-pp.raise_softirq
0.23 ± 9% +0.3 0.57 perf-profile.children.cycles-pp.__acct_update_integrals
2.16 ± 8% +0.3 2.51 perf-profile.children.cycles-pp.task_work_run
2.16 ± 8% +0.3 2.51 perf-profile.children.cycles-pp.task_numa_work
0.15 ± 16% +0.3 0.50 perf-profile.children.cycles-pp.profile_tick
2.19 ± 8% +0.3 2.54 perf-profile.children.cycles-pp.exit_to_usermode_loop
0.31 ± 9% +0.4 0.67 perf-profile.children.cycles-pp.cpuacct_charge
0.37 ± 15% +0.4 0.75 perf-profile.children.cycles-pp._raw_spin_lock_irq
0.55 ± 10% +0.4 0.94 perf-profile.children.cycles-pp._raw_spin_lock
0.14 ± 34% +0.4 0.58 perf-profile.children.cycles-pp.rcu_core
2.58 ± 9% +0.5 3.04 perf-profile.children.cycles-pp.prepare_exit_to_usermode
0.46 ± 18% +0.5 0.95 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.44 ± 16% +0.6 1.02 perf-profile.children.cycles-pp.__update_load_avg_se
0.24 ± 30% +0.6 0.84 perf-profile.children.cycles-pp.__run_timers
0.47 ± 3% +0.6 1.08 perf-profile.children.cycles-pp.account_user_time
0.45 ± 14% +0.6 1.07 perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.30 ± 7% +0.7 0.99 perf-profile.children.cycles-pp.run_local_timers
0.28 ± 31% +0.8 1.10 perf-profile.children.cycles-pp.run_timer_softirq
0.17 ± 44% +0.9 1.11 perf-profile.children.cycles-pp.ksoftirqd_running
2.99 ± 11% +0.9 3.94 perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.26 ± 45% +1.0 1.23 perf-profile.children.cycles-pp.run_posix_cpu_timers
0.25 ± 23% +1.0 1.27 perf-profile.children.cycles-pp.update_rq_clock
1.23 ± 19% +1.1 2.31 perf-profile.children.cycles-pp.update_curr
1.35 ± 19% +1.8 3.10 perf-profile.children.cycles-pp.update_load_avg
0.83 ± 31% +2.0 2.85 perf-profile.children.cycles-pp.__softirqentry_text_start
1.40 ± 27% +3.1 4.46 perf-profile.children.cycles-pp.irq_exit
3.20 ± 20% +4.0 7.18 perf-profile.children.cycles-pp.task_tick_fair
4.84 ± 21% +6.1 10.97 perf-profile.children.cycles-pp.scheduler_tick
6.83 ± 17% +8.9 15.73 perf-profile.children.cycles-pp.update_process_times
17.23 ± 19% +9.0 26.21 perf-profile.children.cycles-pp.__hrtimer_run_queues
22.90 ± 19% +9.5 32.43 perf-profile.children.cycles-pp.hrtimer_interrupt
7.16 ± 18% +10.0 17.12 perf-profile.children.cycles-pp.tick_sched_handle
8.02 ± 18% +11.1 19.09 perf-profile.children.cycles-pp.tick_sched_timer
29.66 ± 21% +13.1 42.74 perf-profile.children.cycles-pp.apic_timer_interrupt
27.48 ± 21% +13.2 40.69 perf-profile.children.cycles-pp.smp_apic_timer_interrupt
46.00 ± 10% -10.4 35.62 perf-profile.self.cycles-pp.malloc
13.39 ± 11% -2.9 10.47 perf-profile.self.cycles-pp.free
1.45 ± 27% -1.4 0.09 perf-profile.self.cycles-pp.__remove_hrtimer
1.41 ± 12% -0.9 0.55 perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
1.75 ± 12% -0.4 1.31 perf-profile.self.cycles-pp.malloc@plt
0.84 ± 29% -0.4 0.47 perf-profile.self.cycles-pp.rb_next
0.66 ± 29% -0.3 0.36 perf-profile.self.cycles-pp.rcu_irq_enter
0.25 ± 14% -0.0 0.20 perf-profile.self.cycles-pp.timerqueue_del
0.24 ± 9% -0.0 0.20 perf-profile.self.cycles-pp.native_apic_msr_eoi_write
0.36 ± 12% -0.0 0.32 perf-profile.self.cycles-pp.native_write_msr
0.14 ± 5% -0.0 0.11 perf-profile.self.cycles-pp.irq_work_tick
0.10 ± 17% +0.0 0.13 perf-profile.self.cycles-pp.lapic_next_deadline
0.07 ± 7% +0.0 0.11 perf-profile.self.cycles-pp.update_process_times
0.26 ± 9% +0.0 0.31 perf-profile.self.cycles-pp.vm_normal_page
0.00 +0.1 0.05 perf-profile.self.cycles-pp.__note_gp_changes
0.00 +0.1 0.05 perf-profile.self.cycles-pp.account_entity_enqueue
0.00 +0.1 0.06 perf-profile.self.cycles-pp.rcu_qs
0.11 ± 14% +0.1 0.17 perf-profile.self.cycles-pp.clockevents_program_event
0.01 ±173% +0.1 0.08 perf-profile.self.cycles-pp.irq_work_run_list
0.01 ±173% +0.1 0.08 perf-profile.self.cycles-pp.nohz_balance_exit_idle
0.00 +0.1 0.07 perf-profile.self.cycles-pp.sched_clock_cpu
0.00 +0.1 0.07 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.17 ± 12% +0.1 0.24 perf-profile.self.cycles-pp.x86_pmu_disable
0.01 ±173% +0.1 0.09 perf-profile.self.cycles-pp.update_min_vruntime
0.21 ± 18% +0.1 0.29 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.09 ± 21% +0.1 0.17 perf-profile.self.cycles-pp.arch_scale_freq_tick
0.23 ± 13% +0.1 0.32 perf-profile.self.cycles-pp.prepare_exit_to_usermode
0.06 ± 20% +0.1 0.15 perf-profile.self.cycles-pp.tick_sched_handle
0.00 +0.1 0.09 perf-profile.self.cycles-pp.note_gp_changes
0.03 ±100% +0.1 0.12 perf-profile.self.cycles-pp.account_entity_dequeue
0.30 ± 9% +0.1 0.39 perf-profile.self.cycles-pp.irq_exit
0.11 ± 19% +0.1 0.21 perf-profile.self.cycles-pp.update_cfs_group
0.39 ± 11% +0.1 0.49 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.08 ± 26% +0.1 0.19 perf-profile.self.cycles-pp.reweight_entity
0.09 ± 17% +0.1 0.20 perf-profile.self.cycles-pp.native_read_msr
0.09 ± 24% +0.1 0.20 perf-profile.self.cycles-pp.__accumulate_pelt_segments
0.11 ± 26% +0.1 0.23 perf-profile.self.cycles-pp.trigger_load_balance
0.31 ± 7% +0.1 0.43 perf-profile.self.cycles-pp.timerqueue_add
0.11 ± 19% +0.1 0.25 perf-profile.self.cycles-pp.__calc_delta
0.02 ±173% +0.2 0.18 perf-profile.self.cycles-pp.rcu_core
0.02 ±173% +0.2 0.18 perf-profile.self.cycles-pp.calc_global_load_tick
0.09 ± 19% +0.2 0.26 perf-profile.self.cycles-pp.native_sched_clock
0.07 ± 7% +0.2 0.25 perf-profile.self.cycles-pp.hrtimer_run_queues
0.53 ± 14% +0.2 0.72 perf-profile.self.cycles-pp.ktime_get
0.21 ± 7% +0.3 0.50 perf-profile.self.cycles-pp.cpuacct_account_field
0.27 +0.3 0.58 perf-profile.self.cycles-pp.account_user_time
0.09 ± 15% +0.3 0.42 perf-profile.self.cycles-pp.hrtimer_active
0.23 ± 9% +0.3 0.57 perf-profile.self.cycles-pp.__acct_update_integrals
0.15 ± 14% +0.3 0.49 perf-profile.self.cycles-pp.raise_softirq
0.15 ± 16% +0.3 0.50 perf-profile.self.cycles-pp.profile_tick
0.67 ± 12% +0.4 1.03 perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.30 ± 9% +0.4 0.67 perf-profile.self.cycles-pp.cpuacct_charge
0.12 ± 29% +0.4 0.49 perf-profile.self.cycles-pp.run_timer_softirq
0.37 ± 14% +0.4 0.75 perf-profile.self.cycles-pp._raw_spin_lock_irq
0.54 ± 9% +0.4 0.93 perf-profile.self.cycles-pp._raw_spin_lock
0.16 ± 32% +0.4 0.59 perf-profile.self.cycles-pp.__run_timers
0.42 ± 17% +0.5 0.87 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.23 ± 11% +0.5 0.69 perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.27 ± 19% +0.5 0.75 perf-profile.self.cycles-pp.tick_sched_timer
0.42 ± 25% +0.5 0.90 perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
0.23 ± 8% +0.5 0.73 perf-profile.self.cycles-pp.run_local_timers
0.39 ± 16% +0.5 0.90 perf-profile.self.cycles-pp.__update_load_avg_se
0.78 ± 24% +0.5 1.29 perf-profile.self.cycles-pp.update_curr
0.28 ± 29% +0.5 0.80 perf-profile.self.cycles-pp.task_tick_fair
0.27 ± 33% +0.5 0.81 perf-profile.self.cycles-pp.__softirqentry_text_start
0.45 ± 24% +0.7 1.14 perf-profile.self.cycles-pp.update_load_avg
0.14 ± 31% +0.8 0.93 perf-profile.self.cycles-pp.update_rq_clock
0.17 ± 44% +0.9 1.10 perf-profile.self.cycles-pp.ksoftirqd_running
0.26 ± 45% +1.0 1.22 perf-profile.self.cycles-pp.run_posix_cpu_timers
1251 ±134% +6253.9% 79487 interrupts.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
1867 ± 12% -27.6% 1351 interrupts.CPU0.RES:Rescheduling_interrupts
551.25 ± 53% -45.2% 302.00 interrupts.CPU10.RES:Rescheduling_interrupts
140.00 ± 87% -92.9% 10.00 interrupts.CPU101.RES:Rescheduling_interrupts
1121 ± 88% -79.4% 231.00 interrupts.CPU102.RES:Rescheduling_interrupts
118.50 ± 31% -68.8% 37.00 interrupts.CPU103.RES:Rescheduling_interrupts
387.25 ± 43% +187.4% 1113 interrupts.CPU104.RES:Rescheduling_interrupts
1326 ± 84% -78.1% 291.00 interrupts.CPU106.RES:Rescheduling_interrupts
86.50 ± 19% -52.6% 41.00 interrupts.CPU109.RES:Rescheduling_interrupts
589.75 ± 18% -54.6% 268.00 interrupts.CPU11.RES:Rescheduling_interrupts
1711 ± 39% -85.9% 242.00 interrupts.CPU110.RES:Rescheduling_interrupts
97.75 ± 62% -83.6% 16.00 interrupts.CPU113.RES:Rescheduling_interrupts
897.75 ± 44% +145.7% 2206 interrupts.CPU120.RES:Rescheduling_interrupts
1079 ± 71% +310.7% 4433 interrupts.CPU122.RES:Rescheduling_interrupts
4608 ± 34% +63.4% 7532 interrupts.CPU123.NMI:Non-maskable_interrupts
4608 ± 34% +63.4% 7532 interrupts.CPU123.PMI:Performance_monitoring_interrupts
240.75 ±105% -93.8% 15.00 interrupts.CPU125.RES:Rescheduling_interrupts
134.00 ± 62% -95.5% 6.00 interrupts.CPU129.RES:Rescheduling_interrupts
1251 ±134% +6253.9% 79487 interrupts.CPU13.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
607.50 ± 34% -58.7% 251.00 interrupts.CPU13.RES:Rescheduling_interrupts
3690 +103.5% 7510 interrupts.CPU130.NMI:Non-maskable_interrupts
3690 +103.5% 7510 interrupts.CPU130.PMI:Performance_monitoring_interrupts
3693 +103.6% 7521 interrupts.CPU131.NMI:Non-maskable_interrupts
3693 +103.6% 7521 interrupts.CPU131.PMI:Performance_monitoring_interrupts
4588 ± 33% +63.9% 7522 interrupts.CPU132.NMI:Non-maskable_interrupts
4588 ± 33% +63.9% 7522 interrupts.CPU132.PMI:Performance_monitoring_interrupts
1734 ± 91% -79.5% 356.00 interrupts.CPU132.RES:Rescheduling_interrupts
1104 ± 87% -85.2% 163.00 interrupts.CPU134.RES:Rescheduling_interrupts
3664 +105.1% 7515 interrupts.CPU136.NMI:Non-maskable_interrupts
3664 +105.1% 7515 interrupts.CPU136.PMI:Performance_monitoring_interrupts
685.00 ± 73% +233.7% 2286 interrupts.CPU138.RES:Rescheduling_interrupts
481.75 ± 25% -35.4% 311.00 interrupts.CPU14.RES:Rescheduling_interrupts
110.25 ± 27% -69.2% 34.00 interrupts.CPU143.RES:Rescheduling_interrupts
248.25 ± 46% +102.2% 502.00 interrupts.CPU144.RES:Rescheduling_interrupts
4612 ± 35% +62.9% 7514 interrupts.CPU145.NMI:Non-maskable_interrupts
4612 ± 35% +62.9% 7514 interrupts.CPU145.PMI:Performance_monitoring_interrupts
180.00 ± 75% -79.4% 37.00 interrupts.CPU145.RES:Rescheduling_interrupts
4620 ± 34% +62.4% 7506 interrupts.CPU149.NMI:Non-maskable_interrupts
4620 ± 34% +62.4% 7506 interrupts.CPU149.PMI:Performance_monitoring_interrupts
276.75 ±116% -97.8% 6.00 interrupts.CPU149.RES:Rescheduling_interrupts
4688 ± 35% +62.8% 7635 interrupts.CPU15.NMI:Non-maskable_interrupts
4688 ± 35% +62.8% 7635 interrupts.CPU15.PMI:Performance_monitoring_interrupts
495.25 ± 12% -47.7% 259.00 interrupts.CPU15.RES:Rescheduling_interrupts
409.25 ± 30% +97.2% 807.00 interrupts.CPU150.RES:Rescheduling_interrupts
4628 ± 34% +62.7% 7530 interrupts.CPU152.NMI:Non-maskable_interrupts
4628 ± 34% +62.7% 7530 interrupts.CPU152.PMI:Performance_monitoring_interrupts
4614 ± 34% +64.1% 7573 interrupts.CPU153.NMI:Non-maskable_interrupts
4614 ± 34% +64.1% 7573 interrupts.CPU153.PMI:Performance_monitoring_interrupts
212.00 ± 84% +230.2% 700.00 interrupts.CPU156.RES:Rescheduling_interrupts
4621 ± 34% +62.5% 7512 interrupts.CPU159.NMI:Non-maskable_interrupts
4621 ± 34% +62.5% 7512 interrupts.CPU159.PMI:Performance_monitoring_interrupts
4633 ± 34% +64.9% 7640 interrupts.CPU16.NMI:Non-maskable_interrupts
4633 ± 34% +64.9% 7640 interrupts.CPU16.PMI:Performance_monitoring_interrupts
4623 ± 35% +62.8% 7525 interrupts.CPU163.NMI:Non-maskable_interrupts
4623 ± 35% +62.8% 7525 interrupts.CPU163.PMI:Performance_monitoring_interrupts
4623 ± 35% +62.4% 7509 interrupts.CPU164.NMI:Non-maskable_interrupts
4623 ± 35% +62.4% 7509 interrupts.CPU164.PMI:Performance_monitoring_interrupts
4649 ± 34% +61.9% 7527 interrupts.CPU165.NMI:Non-maskable_interrupts
4649 ± 34% +61.9% 7527 interrupts.CPU165.PMI:Performance_monitoring_interrupts
226.75 ± 63% -78.4% 49.00 interrupts.CPU166.RES:Rescheduling_interrupts
100.75 ± 69% -94.0% 6.00 interrupts.CPU169.RES:Rescheduling_interrupts
425.25 ± 20% -42.4% 245.00 interrupts.CPU17.RES:Rescheduling_interrupts
4603 ± 35% +63.7% 7537 interrupts.CPU170.NMI:Non-maskable_interrupts
4603 ± 35% +63.7% 7537 interrupts.CPU170.PMI:Performance_monitoring_interrupts
4635 ± 34% +62.0% 7508 interrupts.CPU171.NMI:Non-maskable_interrupts
4635 ± 34% +62.0% 7508 interrupts.CPU171.PMI:Performance_monitoring_interrupts
165.00 ± 50% +269.1% 609.00 interrupts.CPU174.RES:Rescheduling_interrupts
219.50 ± 55% +134.2% 514.00 interrupts.CPU176.RES:Rescheduling_interrupts
3689 +104.2% 7535 interrupts.CPU177.NMI:Non-maskable_interrupts
3689 +104.2% 7535 interrupts.CPU177.PMI:Performance_monitoring_interrupts
250.50 ± 35% +108.8% 523.00 interrupts.CPU178.RES:Rescheduling_interrupts
534.50 ± 7% -44.1% 299.00 interrupts.CPU18.RES:Rescheduling_interrupts
1195 -48.7% 613.00 interrupts.CPU181.CAL:Function_call_interrupts
3684 +104.3% 7525 interrupts.CPU184.NMI:Non-maskable_interrupts
3684 +104.3% 7525 interrupts.CPU184.PMI:Performance_monitoring_interrupts
243.75 ± 19% +35.4% 330.00 interrupts.CPU190.RES:Rescheduling_interrupts
61.25 ± 59% -91.8% 5.00 interrupts.CPU191.RES:Rescheduling_interrupts
114.00 ±120% -94.7% 6.00 interrupts.CPU195.RES:Rescheduling_interrupts
538.75 ± 84% -77.4% 122.00 interrupts.CPU196.RES:Rescheduling_interrupts
193.50 ± 10% -95.3% 9.00 interrupts.CPU197.RES:Rescheduling_interrupts
454.00 ± 22% -55.5% 202.00 interrupts.CPU198.RES:Rescheduling_interrupts
764.75 ± 56% -58.9% 314.00 interrupts.CPU2.RES:Rescheduling_interrupts
463.50 ± 9% -23.0% 357.00 interrupts.CPU20.RES:Rescheduling_interrupts
243.50 ± 48% -96.3% 9.00 interrupts.CPU200.RES:Rescheduling_interrupts
82.25 ± 49% -83.0% 14.00 interrupts.CPU205.RES:Rescheduling_interrupts
349.75 ± 12% +120.4% 771.00 interrupts.CPU206.RES:Rescheduling_interrupts
222.25 ± 56% -80.7% 43.00 interrupts.CPU208.RES:Rescheduling_interrupts
4685 ± 35% +61.5% 7565 interrupts.CPU21.NMI:Non-maskable_interrupts
4685 ± 35% +61.5% 7565 interrupts.CPU21.PMI:Performance_monitoring_interrupts
394.50 ± 48% -92.1% 31.00 interrupts.CPU210.RES:Rescheduling_interrupts
457.50 ± 74% -78.6% 98.00 interrupts.CPU212.RES:Rescheduling_interrupts
107.50 ± 51% -86.0% 15.00 interrupts.CPU213.RES:Rescheduling_interrupts
79.75 ± 53% -88.7% 9.00 interrupts.CPU215.RES:Rescheduling_interrupts
137.25 ± 24% -46.8% 73.00 interrupts.CPU218.RES:Rescheduling_interrupts
84.25 ± 29% -90.5% 8.00 interrupts.CPU219.RES:Rescheduling_interrupts
3707 +105.5% 7619 interrupts.CPU22.NMI:Non-maskable_interrupts
3707 +105.5% 7619 interrupts.CPU22.PMI:Performance_monitoring_interrupts
512.75 ± 29% -43.6% 289.00 interrupts.CPU22.RES:Rescheduling_interrupts
157.75 ± 42% -77.8% 35.00 interrupts.CPU220.RES:Rescheduling_interrupts
93.00 ±103% -92.5% 7.00 interrupts.CPU221.RES:Rescheduling_interrupts
73.75 ± 22% -67.5% 24.00 interrupts.CPU222.RES:Rescheduling_interrupts
4615 ± 35% +63.0% 7521 interrupts.CPU225.NMI:Non-maskable_interrupts
4615 ± 35% +63.0% 7521 interrupts.CPU225.PMI:Performance_monitoring_interrupts
75.00 ± 44% -93.3% 5.00 interrupts.CPU225.RES:Rescheduling_interrupts
144.75 ± 46% -76.5% 34.00 interrupts.CPU226.RES:Rescheduling_interrupts
3740 +104.5% 7648 interrupts.CPU23.NMI:Non-maskable_interrupts
3740 +104.5% 7648 interrupts.CPU23.PMI:Performance_monitoring_interrupts
404.75 ± 23% -38.7% 248.00 interrupts.CPU23.RES:Rescheduling_interrupts
154.75 ± 62% +122.9% 345.00 interrupts.CPU232.RES:Rescheduling_interrupts
69.25 ± 32% -85.6% 10.00 interrupts.CPU233.RES:Rescheduling_interrupts
121.50 ± 53% -76.1% 29.00 interrupts.CPU236.RES:Rescheduling_interrupts
3772 +102.4% 7636 interrupts.CPU24.NMI:Non-maskable_interrupts
3772 +102.4% 7636 interrupts.CPU24.PMI:Performance_monitoring_interrupts
683.25 ± 43% -58.4% 284.00 interrupts.CPU24.RES:Rescheduling_interrupts
218.25 ±101% -78.9% 46.00 interrupts.CPU240.RES:Rescheduling_interrupts
4616 ± 34% +62.7% 7509 interrupts.CPU242.NMI:Non-maskable_interrupts
4616 ± 34% +62.7% 7509 interrupts.CPU242.PMI:Performance_monitoring_interrupts
108.50 ± 30% -39.2% 66.00 interrupts.CPU242.RES:Rescheduling_interrupts
113.25 ± 44% -93.8% 7.00 interrupts.CPU243.RES:Rescheduling_interrupts
60.50 ± 83% -88.4% 7.00 interrupts.CPU247.RES:Rescheduling_interrupts
161.75 ±125% -96.9% 5.00 interrupts.CPU249.RES:Rescheduling_interrupts
239.50 ± 65% -80.0% 48.00 interrupts.CPU250.RES:Rescheduling_interrupts
105.25 ± 51% +286.7% 407.00 interrupts.CPU252.RES:Rescheduling_interrupts
4622 ± 35% +62.5% 7510 interrupts.CPU253.NMI:Non-maskable_interrupts
4622 ± 35% +62.5% 7510 interrupts.CPU253.PMI:Performance_monitoring_interrupts
151.50 ± 36% -48.5% 78.00 interrupts.CPU256.RES:Rescheduling_interrupts
3669 +105.7% 7547 interrupts.CPU257.NMI:Non-maskable_interrupts
3669 +105.7% 7547 interrupts.CPU257.PMI:Performance_monitoring_interrupts
63.75 ± 52% -90.6% 6.00 interrupts.CPU257.RES:Rescheduling_interrupts
168.00 ± 56% -64.9% 59.00 interrupts.CPU258.RES:Rescheduling_interrupts
90.75 ± 86% -91.2% 8.00 interrupts.CPU259.RES:Rescheduling_interrupts
1164 ± 63% -75.5% 285.00 interrupts.CPU26.RES:Rescheduling_interrupts
4592 ± 34% +63.5% 7507 interrupts.CPU260.NMI:Non-maskable_interrupts
4592 ± 34% +63.5% 7507 interrupts.CPU260.PMI:Performance_monitoring_interrupts
119.25 ± 28% -74.0% 31.00 interrupts.CPU260.RES:Rescheduling_interrupts
4595 ± 35% +63.3% 7505 interrupts.CPU262.NMI:Non-maskable_interrupts
4595 ± 35% +63.3% 7505 interrupts.CPU262.PMI:Performance_monitoring_interrupts
4590 ± 34% +63.7% 7514 interrupts.CPU263.NMI:Non-maskable_interrupts
4590 ± 34% +63.7% 7514 interrupts.CPU263.PMI:Performance_monitoring_interrupts
62.25 ± 56% -92.0% 5.00 interrupts.CPU263.RES:Rescheduling_interrupts
98.50 ± 36% +383.2% 476.00 interrupts.CPU264.RES:Rescheduling_interrupts
51.25 ± 80% -86.3% 7.00 interrupts.CPU265.RES:Rescheduling_interrupts
244.75 ± 84% -86.1% 34.00 interrupts.CPU266.RES:Rescheduling_interrupts
114.50 ± 13% -79.0% 24.00 interrupts.CPU268.RES:Rescheduling_interrupts
4610 ± 35% +64.0% 7559 interrupts.CPU269.NMI:Non-maskable_interrupts
4610 ± 35% +64.0% 7559 interrupts.CPU269.PMI:Performance_monitoring_interrupts
441.50 ± 27% -36.6% 280.00 interrupts.CPU27.RES:Rescheduling_interrupts
129.00 ± 62% +20168.2% 26146 interrupts.CPU272.RES:Rescheduling_interrupts
109.25 ± 65% -93.6% 7.00 interrupts.CPU279.RES:Rescheduling_interrupts
3728 +104.8% 7636 interrupts.CPU28.NMI:Non-maskable_interrupts
3728 +104.8% 7636 interrupts.CPU28.PMI:Performance_monitoring_interrupts
556.75 ± 23% -47.9% 290.00 interrupts.CPU28.RES:Rescheduling_interrupts
358.50 ± 79% -86.1% 50.00 interrupts.CPU282.RES:Rescheduling_interrupts
202.75 ± 54% -83.2% 34.00 interrupts.CPU284.RES:Rescheduling_interrupts
242.25 ± 69% -88.0% 29.00 interrupts.CPU287.RES:Rescheduling_interrupts
3718 +105.0% 7623 interrupts.CPU29.NMI:Non-maskable_interrupts
3718 +105.0% 7623 interrupts.CPU29.PMI:Performance_monitoring_interrupts
389.25 ± 14% -32.2% 264.00 interrupts.CPU29.RES:Rescheduling_interrupts
3757 +102.6% 7612 interrupts.CPU30.NMI:Non-maskable_interrupts
3757 +102.6% 7612 interrupts.CPU30.PMI:Performance_monitoring_interrupts
941.25 ± 19% -63.3% 345.00 interrupts.CPU30.RES:Rescheduling_interrupts
3730 +104.4% 7625 interrupts.CPU31.NMI:Non-maskable_interrupts
3730 +104.4% 7625 interrupts.CPU31.PMI:Performance_monitoring_interrupts
3744 +103.8% 7632 interrupts.CPU32.NMI:Non-maskable_interrupts
3744 +103.8% 7632 interrupts.CPU32.PMI:Performance_monitoring_interrupts
637.75 ± 41% -56.6% 277.00 interrupts.CPU32.RES:Rescheduling_interrupts
4649 ± 34% +64.3% 7637 interrupts.CPU34.NMI:Non-maskable_interrupts
4649 ± 34% +64.3% 7637 interrupts.CPU34.PMI:Performance_monitoring_interrupts
416.25 ± 16% -30.3% 290.00 interrupts.CPU34.RES:Rescheduling_interrupts
3725 +104.1% 7602 interrupts.CPU35.NMI:Non-maskable_interrupts
3725 +104.1% 7602 interrupts.CPU35.PMI:Performance_monitoring_interrupts
3695 +104.7% 7562 interrupts.CPU36.NMI:Non-maskable_interrupts
3695 +104.7% 7562 interrupts.CPU36.PMI:Performance_monitoring_interrupts
3748 +104.3% 7657 interrupts.CPU37.NMI:Non-maskable_interrupts
3748 +104.3% 7657 interrupts.CPU37.PMI:Performance_monitoring_interrupts
324.00 ± 9% -18.8% 263.00 interrupts.CPU37.RES:Rescheduling_interrupts
436.00 ± 5% -27.8% 315.00 interrupts.CPU38.RES:Rescheduling_interrupts
4671 ± 34% +61.0% 7521 interrupts.CPU39.NMI:Non-maskable_interrupts
4671 ± 34% +61.0% 7521 interrupts.CPU39.PMI:Performance_monitoring_interrupts
349.25 ± 14% -30.7% 242.00 interrupts.CPU39.RES:Rescheduling_interrupts
710.50 ± 32% -52.9% 335.00 interrupts.CPU4.RES:Rescheduling_interrupts
3707 +103.9% 7560 interrupts.CPU42.NMI:Non-maskable_interrupts
3707 +103.9% 7560 interrupts.CPU42.PMI:Performance_monitoring_interrupts
4597 ± 35% +63.4% 7513 interrupts.CPU43.NMI:Non-maskable_interrupts
4597 ± 35% +63.4% 7513 interrupts.CPU43.PMI:Performance_monitoring_interrupts
4610 ± 35% +64.1% 7568 interrupts.CPU44.NMI:Non-maskable_interrupts
4610 ± 35% +64.1% 7568 interrupts.CPU44.PMI:Performance_monitoring_interrupts
887.25 ± 95% -67.7% 287.00 interrupts.CPU44.RES:Rescheduling_interrupts
3669 +106.6% 7582 interrupts.CPU45.NMI:Non-maskable_interrupts
3669 +106.6% 7582 interrupts.CPU45.PMI:Performance_monitoring_interrupts
604.50 ± 66% -55.0% 272.00 interrupts.CPU47.RES:Rescheduling_interrupts
3717 +103.6% 7570 interrupts.CPU49.NMI:Non-maskable_interrupts
3717 +103.6% 7570 interrupts.CPU49.PMI:Performance_monitoring_interrupts
160.50 ± 20% +63.9% 263.00 interrupts.CPU49.RES:Rescheduling_interrupts
4601 ± 34% +64.8% 7585 interrupts.CPU50.NMI:Non-maskable_interrupts
4601 ± 34% +64.8% 7585 interrupts.CPU50.PMI:Performance_monitoring_interrupts
170.50 ± 36% +67.2% 285.00 interrupts.CPU50.RES:Rescheduling_interrupts
4590 ± 34% +65.5% 7595 interrupts.CPU51.NMI:Non-maskable_interrupts
4590 ± 34% +65.5% 7595 interrupts.CPU51.PMI:Performance_monitoring_interrupts
146.75 ± 43% -78.9% 31.00 interrupts.CPU51.RES:Rescheduling_interrupts
4627 ± 35% +62.4% 7514 interrupts.CPU52.NMI:Non-maskable_interrupts
4627 ± 35% +62.4% 7514 interrupts.CPU52.PMI:Performance_monitoring_interrupts
266.25 ± 57% -58.3% 111.00 interrupts.CPU54.RES:Rescheduling_interrupts
204.00 ± 71% -87.7% 25.00 interrupts.CPU55.RES:Rescheduling_interrupts
3703 +105.1% 7596 interrupts.CPU58.NMI:Non-maskable_interrupts
3703 +105.1% 7596 interrupts.CPU58.PMI:Performance_monitoring_interrupts
589.00 ± 41% -45.8% 319.00 interrupts.CPU6.RES:Rescheduling_interrupts
255.25 ± 57% -63.6% 93.00 interrupts.CPU60.RES:Rescheduling_interrupts
174.25 ± 19% -62.1% 66.00 interrupts.CPU62.RES:Rescheduling_interrupts
207.25 ± 50% -65.3% 72.00 interrupts.CPU66.RES:Rescheduling_interrupts
169.00 ± 54% -65.1% 59.00 interrupts.CPU68.RES:Rescheduling_interrupts
103.25 ± 66% -84.5% 16.00 interrupts.CPU69.RES:Rescheduling_interrupts
501.75 ± 24% -43.4% 284.00 interrupts.CPU7.RES:Rescheduling_interrupts
286.50 ± 53% +388.3% 1399 interrupts.CPU72.RES:Rescheduling_interrupts
634.25 ± 71% +406.0% 3209 interrupts.CPU74.RES:Rescheduling_interrupts
484.00 ± 55% +147.3% 1197 interrupts.CPU76.RES:Rescheduling_interrupts
85.75 ± 53% -63.8% 31.00 interrupts.CPU77.RES:Rescheduling_interrupts
539.25 ± 18% +73.9% 938.00 interrupts.CPU8.RES:Rescheduling_interrupts
755.25 ± 91% -77.0% 174.00 interrupts.CPU80.RES:Rescheduling_interrupts
63.25 ± 53% +119.8% 139.00 interrupts.CPU81.RES:Rescheduling_interrupts
519.25 ± 33% -65.1% 181.00 interrupts.CPU82.RES:Rescheduling_interrupts
517.25 ± 28% -60.8% 203.00 interrupts.CPU84.RES:Rescheduling_interrupts
132.25 ± 50% -61.4% 51.00 interrupts.CPU85.RES:Rescheduling_interrupts
382.00 ± 35% +77.2% 677.00 interrupts.CPU86.RES:Rescheduling_interrupts
62.50 ± 62% -74.4% 16.00 interrupts.CPU87.RES:Rescheduling_interrupts
92.00 ± 32% -66.3% 31.00 interrupts.CPU89.RES:Rescheduling_interrupts
436.75 ± 19% -39.1% 266.00 interrupts.CPU9.RES:Rescheduling_interrupts
1072 ± 76% -73.7% 282.00 interrupts.CPU92.RES:Rescheduling_interrupts
167.00 ± 75% -82.6% 29.00 interrupts.CPU93.RES:Rescheduling_interrupts
952.75 ± 66% -68.1% 304.00 interrupts.CPU96.RES:Rescheduling_interrupts
96.50 ± 25% +217.1% 306.00 interrupts.CPU97.RES:Rescheduling_interrupts
54.25 ± 22% +351.6% 245.00 interrupts.CPU99.RES:Rescheduling_interrupts
39164 ± 11% +56.5% 61282 softirqs.CPU0.RCU
119211 ± 14% +35.1% 161073 softirqs.CPU0.TIMER
39132 ± 15% +54.9% 60633 softirqs.CPU1.RCU
106485 ± 10% +39.0% 148019 softirqs.CPU1.TIMER
40469 ± 13% +46.2% 59148 softirqs.CPU10.RCU
33205 ± 16% +58.8% 52724 softirqs.CPU100.RCU
33017 ± 16% +58.1% 52202 softirqs.CPU101.RCU
32946 ± 16% +59.9% 52688 softirqs.CPU102.RCU
32954 ± 16% +58.7% 52303 softirqs.CPU103.RCU
32712 ± 16% +59.2% 52093 softirqs.CPU104.RCU
32374 ± 16% +62.4% 52576 softirqs.CPU105.RCU
32420 ± 16% +63.4% 52971 softirqs.CPU106.RCU
32316 ± 15% +62.9% 52648 softirqs.CPU107.RCU
32685 ± 15% +62.1% 52968 softirqs.CPU108.RCU
132926 ± 9% +18.9% 157999 softirqs.CPU108.TIMER
32436 ± 16% +61.6% 52414 softirqs.CPU109.RCU
40413 ± 14% +60.2% 64751 softirqs.CPU11.RCU
115232 ± 12% +21.9% 140473 softirqs.CPU11.TIMER
32751 ± 16% +60.0% 52403 softirqs.CPU110.RCU
32850 ± 16% +60.6% 52742 softirqs.CPU111.RCU
32532 ± 17% +60.8% 52319 softirqs.CPU112.RCU
32501 ± 16% +60.8% 52267 softirqs.CPU113.RCU
31957 ± 16% +64.6% 52601 softirqs.CPU114.RCU
32115 ± 15% +62.2% 52104 softirqs.CPU115.RCU
32767 ± 16% +59.3% 52200 softirqs.CPU116.RCU
32625 ± 15% +61.7% 52741 softirqs.CPU117.RCU
32319 ± 16% +61.3% 52142 softirqs.CPU118.RCU
32227 ± 15% +61.7% 52110 softirqs.CPU119.RCU
43084 ± 21% +41.9% 61121 softirqs.CPU12.RCU
31813 ± 17% +64.6% 52360 softirqs.CPU120.RCU
31512 ± 15% +66.8% 52559 softirqs.CPU121.RCU
32569 ± 15% +61.6% 52619 softirqs.CPU122.RCU
32337 ± 16% +61.7% 52276 softirqs.CPU123.RCU
32442 ± 15% +61.9% 52524 softirqs.CPU124.RCU
32258 ± 15% +63.3% 52680 softirqs.CPU125.RCU
32480 ± 16% +61.6% 52500 softirqs.CPU126.RCU
32535 ± 17% +60.8% 52330 softirqs.CPU127.RCU
32059 ± 16% +60.4% 51435 softirqs.CPU128.RCU
32312 ± 16% +61.8% 52278 softirqs.CPU129.RCU
1056 ±137% +3671.0% 39822 softirqs.CPU13.NET_RX
40034 ± 11% +51.2% 60514 softirqs.CPU13.RCU
108802 ± 10% +29.6% 141012 softirqs.CPU13.TIMER
32215 ± 16% +62.1% 52205 softirqs.CPU130.RCU
32315 ± 16% +61.1% 52075 softirqs.CPU131.RCU
33006 ± 15% +59.0% 52476 softirqs.CPU132.RCU
32266 ± 17% +61.9% 52250 softirqs.CPU133.RCU
32626 ± 16% +62.9% 53136 softirqs.CPU134.RCU
32433 ± 18% +58.8% 51494 softirqs.CPU135.RCU
32294 ± 14% +58.7% 51235 softirqs.CPU136.RCU
31820 ± 16% +63.5% 52021 softirqs.CPU137.RCU
32892 ± 16% +58.8% 52227 softirqs.CPU138.RCU
32976 ± 15% +57.9% 52060 softirqs.CPU139.RCU
40138 ± 12% +60.0% 64221 softirqs.CPU14.RCU
117850 ± 9% +15.9% 136643 softirqs.CPU14.TIMER
32440 ± 16% +59.4% 51714 softirqs.CPU140.RCU
32239 ± 16% +65.8% 53461 softirqs.CPU141.RCU
32639 ± 16% +59.7% 52137 softirqs.CPU142.RCU
32350 ± 16% +62.3% 52492 softirqs.CPU143.RCU
32203 ± 16% +61.8% 52101 softirqs.CPU144.RCU
32170 ± 17% +60.4% 51609 softirqs.CPU145.RCU
32892 ± 16% +65.1% 54293 softirqs.CPU146.RCU
32355 ± 16% +61.3% 52182 softirqs.CPU147.RCU
32385 ± 16% +60.7% 52047 softirqs.CPU148.RCU
32356 ± 17% +62.5% 52581 softirqs.CPU149.RCU
38976 ± 16% +61.0% 62748 softirqs.CPU15.RCU
110967 ± 12% +24.7% 138390 softirqs.CPU15.TIMER
32399 ± 16% +63.3% 52903 softirqs.CPU150.RCU
32238 ± 15% +57.9% 50889 softirqs.CPU151.RCU
32360 ± 16% +62.0% 52413 softirqs.CPU152.RCU
32475 ± 16% +60.2% 52015 softirqs.CPU153.RCU
32209 ± 16% +61.4% 51998 softirqs.CPU154.RCU
32373 ± 16% +61.2% 52181 softirqs.CPU155.RCU
32295 ± 15% +65.9% 53590 softirqs.CPU156.RCU
32127 ± 16% +57.1% 50483 softirqs.CPU157.RCU
32514 ± 16% +60.6% 52204 softirqs.CPU158.RCU
32857 ± 16% +58.2% 51977 softirqs.CPU159.RCU
39215 ± 11% +54.0% 60374 softirqs.CPU16.RCU
116903 ± 13% +28.9% 150659 softirqs.CPU16.TIMER
32327 ± 16% +61.5% 52199 softirqs.CPU160.RCU
32410 ± 16% +60.4% 51970 softirqs.CPU161.RCU
32024 ± 16% +60.8% 51503 softirqs.CPU162.RCU
31125 ± 14% +67.1% 52016 softirqs.CPU163.RCU
32266 ± 16% +60.0% 51633 softirqs.CPU164.RCU
32209 ± 17% +61.9% 52135 softirqs.CPU165.RCU
32200 ± 16% +61.4% 51956 softirqs.CPU166.RCU
31951 ± 16% +62.5% 51932 softirqs.CPU167.RCU
31914 ± 16% +62.4% 51836 softirqs.CPU168.RCU
32081 ± 16% +62.3% 52066 softirqs.CPU169.RCU
39400 ± 11% +52.2% 59956 softirqs.CPU17.RCU
107863 ± 7% +37.2% 147984 softirqs.CPU17.TIMER
32016 ± 17% +63.9% 52460 softirqs.CPU170.RCU
31961 ± 16% +62.7% 51996 softirqs.CPU171.RCU
32978 ± 15% +56.1% 51468 softirqs.CPU172.RCU
32233 ± 16% +62.1% 52246 softirqs.CPU173.RCU
32152 ± 16% +63.4% 52543 softirqs.CPU174.RCU
32075 ± 16% +62.0% 51952 softirqs.CPU175.RCU
32196 ± 16% +60.3% 51620 softirqs.CPU176.RCU
32269 ± 17% +59.5% 51462 softirqs.CPU177.RCU
32096 ± 15% +61.7% 51886 softirqs.CPU178.RCU
32182 ± 16% +60.2% 51570 softirqs.CPU179.RCU
38279 ± 15% +55.2% 59426 softirqs.CPU18.RCU
117158 ± 13% +31.0% 153465 softirqs.CPU18.TIMER
31735 ± 16% +62.2% 51480 softirqs.CPU180.RCU
29335 ± 5% +75.2% 51409 softirqs.CPU181.RCU
31981 ± 16% +60.1% 51205 softirqs.CPU182.RCU
31948 ± 15% +57.6% 50365 softirqs.CPU183.RCU
32189 ± 17% +59.1% 51205 softirqs.CPU184.RCU
31957 ± 16% +60.7% 51344 softirqs.CPU185.RCU
32298 ± 17% +58.2% 51080 softirqs.CPU186.RCU
32335 ± 16% +58.8% 51344 softirqs.CPU187.RCU
32287 ± 16% +59.6% 51540 softirqs.CPU188.RCU
31965 ± 16% +59.1% 50857 softirqs.CPU189.RCU
39495 ± 14% +53.9% 60800 softirqs.CPU19.RCU
114131 ± 9% +20.9% 138034 softirqs.CPU19.TIMER
31213 ± 17% +65.4% 51631 softirqs.CPU190.RCU
31471 ± 17% +64.1% 51645 softirqs.CPU191.RCU
31760 ± 17% +62.9% 51728 softirqs.CPU192.RCU
32147 ± 16% +61.8% 52016 softirqs.CPU193.RCU
32079 ± 17% +62.9% 52256 softirqs.CPU194.RCU
31755 ± 16% +63.2% 51815 softirqs.CPU195.RCU
32240 ± 16% +62.6% 52417 softirqs.CPU196.RCU
32033 ± 16% +60.6% 51435 softirqs.CPU197.RCU
32041 ± 17% +62.4% 52022 softirqs.CPU198.RCU
31849 ± 17% +64.2% 52294 softirqs.CPU199.RCU
39674 ± 10% +54.8% 61436 softirqs.CPU2.RCU
40657 ± 17% +48.6% 60422 softirqs.CPU20.RCU
113465 ± 13% +32.5% 150390 softirqs.CPU20.TIMER
31959 ± 16% +62.8% 52023 softirqs.CPU200.RCU
31710 ± 16% +62.6% 51548 softirqs.CPU201.RCU
32021 ± 17% +61.8% 51801 softirqs.CPU202.RCU
32269 ± 16% +59.5% 51463 softirqs.CPU203.RCU
31922 ± 15% +60.0% 51087 softirqs.CPU204.RCU
31672 ± 17% +63.4% 51740 softirqs.CPU205.RCU
31875 ± 16% +63.2% 52006 softirqs.CPU206.RCU
31746 ± 16% +63.5% 51898 softirqs.CPU207.RCU
32414 ± 16% +61.0% 52186 softirqs.CPU208.RCU
32251 ± 16% +60.8% 51875 softirqs.CPU209.RCU
40977 ± 13% +46.2% 59928 softirqs.CPU21.RCU
109618 ± 13% +23.9% 135789 softirqs.CPU21.TIMER
32763 ± 15% +58.2% 51825 softirqs.CPU210.RCU
32054 ± 15% +62.8% 52195 softirqs.CPU211.RCU
32462 ± 16% +61.6% 52468 softirqs.CPU212.RCU
32438 ± 16% +61.8% 52499 softirqs.CPU213.RCU
31689 ± 17% +62.5% 51498 softirqs.CPU214.RCU
32181 ± 16% +63.6% 52657 softirqs.CPU215.RCU
32243 ± 16% +60.5% 51745 softirqs.CPU216.RCU
32122 ± 15% +59.8% 51345 softirqs.CPU217.RCU
32194 ± 16% +62.6% 52340 softirqs.CPU218.RCU
33401 ± 15% +58.2% 52834 softirqs.CPU219.RCU
39309 ± 19% +53.5% 60334 softirqs.CPU22.RCU
119248 ± 16% +23.3% 147000 softirqs.CPU22.TIMER
31767 ± 16% +63.9% 52051 softirqs.CPU220.RCU
32483 ± 15% +60.4% 52113 softirqs.CPU221.RCU
32330 ± 16% +61.1% 52088 softirqs.CPU222.RCU
32645 ± 15% +59.9% 52206 softirqs.CPU223.RCU
32122 ± 17% +59.0% 51071 softirqs.CPU224.RCU
32307 ± 16% +60.0% 51695 softirqs.CPU225.RCU
32176 ± 16% +61.2% 51868 softirqs.CPU226.RCU
32405 ± 16% +60.4% 51994 softirqs.CPU227.RCU
32186 ± 20% +61.2% 51874 softirqs.CPU228.RCU
31808 ± 16% +63.5% 52008 softirqs.CPU229.RCU
40782 ± 17% +48.6% 60604 softirqs.CPU23.RCU
32003 ± 16% +58.8% 50807 softirqs.CPU230.RCU
32424 ± 16% +58.7% 51456 softirqs.CPU231.RCU
32316 ± 16% +60.7% 51924 softirqs.CPU232.RCU
32725 ± 15% +59.2% 52098 softirqs.CPU233.RCU
31237 ± 15% +65.4% 51666 softirqs.CPU234.RCU
32479 ± 15% +60.3% 52063 softirqs.CPU235.RCU
133723 ± 7% +15.4% 154367 softirqs.CPU235.TIMER
32028 ± 16% +62.2% 51963 softirqs.CPU236.RCU
32136 ± 16% +62.4% 52192 softirqs.CPU237.RCU
32023 ± 16% +60.5% 51408 softirqs.CPU238.RCU
32055 ± 16% +60.7% 51524 softirqs.CPU239.RCU
39888 ± 17% +53.7% 61300 softirqs.CPU24.RCU
112430 ± 15% +27.1% 142949 softirqs.CPU24.TIMER
31745 ± 17% +63.8% 52007 softirqs.CPU240.RCU
32474 ± 16% +60.9% 52242 softirqs.CPU241.RCU
31904 ± 16% +62.7% 51899 softirqs.CPU242.RCU
32032 ± 16% +61.9% 51868 softirqs.CPU243.RCU
31357 ± 18% +64.8% 51685 softirqs.CPU244.RCU
132812 ± 8% +15.3% 153157 softirqs.CPU244.TIMER
31549 ± 16% +64.3% 51841 softirqs.CPU245.RCU
32027 ± 16% +60.8% 51504 softirqs.CPU246.RCU
32033 ± 16% +77.1% 56745 softirqs.CPU247.RCU
141833 ± 5% +19.5% 169427 softirqs.CPU247.TIMER
31859 ± 17% +65.0% 52580 softirqs.CPU248.RCU
32090 ± 16% +62.2% 52055 softirqs.CPU249.RCU
39971 ± 13% +52.6% 61007 softirqs.CPU25.RCU
114330 ± 12% +28.9% 147423 softirqs.CPU25.TIMER
31491 ± 16% +64.9% 51918 softirqs.CPU250.RCU
31766 ± 15% +64.4% 52234 softirqs.CPU251.RCU
32019 ± 16% +61.0% 51544 softirqs.CPU252.RCU
31922 ± 17% +59.7% 50971 softirqs.CPU253.RCU
31514 ± 16% +64.1% 51705 softirqs.CPU254.RCU
31765 ± 17% +64.6% 52276 softirqs.CPU255.RCU
31738 ± 16% +60.0% 50780 softirqs.CPU256.RCU
33985 ± 11% +48.7% 50523 softirqs.CPU257.RCU
31187 ± 17% +57.1% 49000 softirqs.CPU258.RCU
32031 ± 14% +50.2% 48097 softirqs.CPU259.RCU
39939 ± 11% +53.3% 61220 softirqs.CPU26.RCU
115081 ± 12% +24.6% 143406 softirqs.CPU26.TIMER
31401 ± 16% +60.2% 50314 softirqs.CPU260.RCU
30452 ± 16% +62.8% 49574 softirqs.CPU261.RCU
30819 ± 16% +62.0% 49913 softirqs.CPU262.RCU
31668 ± 22% +57.4% 49847 softirqs.CPU263.RCU
32298 ± 16% +58.0% 51038 softirqs.CPU264.RCU
31601 ± 13% +57.3% 49722 softirqs.CPU265.RCU
123387 ± 6% +17.4% 144849 softirqs.CPU265.TIMER
31810 ± 16% +60.5% 51049 softirqs.CPU266.RCU
32986 ± 14% +51.1% 49844 softirqs.CPU267.RCU
32309 ± 16% +58.0% 51060 softirqs.CPU268.RCU
32278 ± 16% +59.2% 51371 softirqs.CPU269.RCU
40744 ± 17% +58.9% 64726 softirqs.CPU27.RCU
108716 ± 13% +41.4% 153739 softirqs.CPU27.TIMER
32431 ± 16% +56.5% 50753 softirqs.CPU270.RCU
32242 ± 16% +55.4% 50109 softirqs.CPU271.RCU
33991 ± 15% -25.3% 25388 softirqs.CPU272.RCU
122760 ± 13% +26.1% 154830 softirqs.CPU272.TIMER
33868 ± 16% +45.9% 49423 softirqs.CPU273.RCU
125322 ± 12% +22.9% 154051 softirqs.CPU273.TIMER
33902 ± 15% +57.9% 53546 softirqs.CPU274.RCU
33622 ± 16% +53.0% 51450 softirqs.CPU275.RCU
133137 ± 8% +16.5% 155075 softirqs.CPU275.TIMER
33881 ± 16% +59.1% 53894 softirqs.CPU276.RCU
33983 ± 16% +60.2% 54449 softirqs.CPU277.RCU
33462 ± 15% +57.7% 52778 softirqs.CPU278.RCU
126550 ± 9% +18.6% 150046 softirqs.CPU278.TIMER
34169 ± 16% +58.3% 54100 softirqs.CPU279.RCU
40134 ± 12% +54.1% 61856 softirqs.CPU28.RCU
106399 ± 7% +36.8% 145511 softirqs.CPU28.TIMER
34216 ± 15% +54.1% 52717 softirqs.CPU280.RCU
34772 ± 14% +49.9% 52127 softirqs.CPU281.RCU
34184 ± 16% +55.3% 53074 softirqs.CPU282.RCU
33411 ± 17% +57.4% 52591 softirqs.CPU283.RCU
33482 ± 17% +59.2% 53316 softirqs.CPU284.RCU
124921 ± 7% +18.4% 147846 softirqs.CPU284.TIMER
34232 ± 16% +59.9% 54735 softirqs.CPU285.RCU
33099 ± 16% +61.5% 53454 softirqs.CPU286.RCU
128032 ± 6% +15.8% 148252 softirqs.CPU286.TIMER
33660 ± 14% +60.2% 53931 softirqs.CPU287.RCU
39735 ± 14% +52.4% 60570 softirqs.CPU29.RCU
109107 ± 10% +31.4% 143385 softirqs.CPU29.TIMER
38978 ± 17% +58.8% 61916 softirqs.CPU3.RCU
114894 ± 13% +24.5% 143095 softirqs.CPU3.TIMER
39122 ± 13% +57.3% 61555 softirqs.CPU30.RCU
112105 ± 11% +29.4% 145113 softirqs.CPU30.TIMER
40231 ± 15% +58.2% 63642 softirqs.CPU31.RCU
106758 ± 9% +28.5% 137237 softirqs.CPU31.TIMER
36607 ± 12% +49.8% 54850 softirqs.CPU32.RCU
107614 ± 10% +30.6% 140576 softirqs.CPU32.TIMER
38637 ± 11% +41.5% 54675 softirqs.CPU33.RCU
111703 ± 10% +25.3% 139965 softirqs.CPU33.TIMER
37654 ± 14% +48.0% 55715 softirqs.CPU34.RCU
108130 ± 10% +29.2% 139690 softirqs.CPU34.TIMER
37742 ± 14% +48.5% 56044 softirqs.CPU35.RCU
111809 ± 13% +35.7% 151748 softirqs.CPU35.TIMER
37760 ± 13% +50.7% 56894 softirqs.CPU36.RCU
104801 ± 6% +41.3% 148074 softirqs.CPU36.TIMER
38759 ± 10% +44.3% 55925 softirqs.CPU37.RCU
110484 ± 10% +31.8% 145580 softirqs.CPU37.TIMER
37881 ± 13% +52.6% 57806 softirqs.CPU38.RCU
111036 ± 14% +28.4% 142517 softirqs.CPU38.TIMER
37751 ± 13% +53.5% 57963 softirqs.CPU39.RCU
109975 ± 12% +29.8% 142703 softirqs.CPU39.TIMER
40700 ± 15% +55.2% 63155 softirqs.CPU4.RCU
33795 ± 15% +53.5% 51887 softirqs.CPU40.RCU
32637 ± 15% +67.7% 54725 softirqs.CPU41.RCU
33955 ± 15% +66.4% 56488 softirqs.CPU42.RCU
128703 ± 11% +31.3% 168987 softirqs.CPU42.TIMER
34606 ± 14% +54.9% 53594 softirqs.CPU43.RCU
124225 ± 11% +22.8% 152583 softirqs.CPU43.TIMER
34280 ± 14% +61.1% 55211 softirqs.CPU44.RCU
33700 ± 16% +64.4% 55416 softirqs.CPU45.RCU
122489 ± 7% +23.4% 151156 softirqs.CPU45.TIMER
34421 ± 14% +60.7% 55327 softirqs.CPU46.RCU
124335 ± 12% +22.7% 152608 softirqs.CPU46.TIMER
32622 ± 17% +63.8% 53430 softirqs.CPU47.RCU
34127 ± 16% +59.8% 54523 softirqs.CPU48.RCU
35165 ± 16% +58.9% 55872 softirqs.CPU49.RCU
40442 ± 11% +56.7% 63356 softirqs.CPU5.RCU
109994 ± 13% +22.7% 135002 softirqs.CPU5.TIMER
34820 ± 16% +61.1% 56093 softirqs.CPU50.RCU
35464 ± 15% +59.4% 56524 softirqs.CPU51.RCU
125051 ± 11% +24.2% 155354 softirqs.CPU51.TIMER
36036 ± 18% +53.7% 55396 softirqs.CPU52.RCU
35654 ± 16% +52.6% 54416 softirqs.CPU53.RCU
35691 ± 11% +52.4% 54411 softirqs.CPU54.RCU
34660 ± 13% +53.3% 53123 softirqs.CPU55.RCU
126943 ± 9% +21.1% 153725 softirqs.CPU55.TIMER
34264 ± 15% +62.4% 55653 softirqs.CPU56.RCU
125229 ± 7% +20.9% 151461 softirqs.CPU56.TIMER
34576 ± 16% +62.1% 56057 softirqs.CPU57.RCU
34536 ± 16% +61.4% 55726 softirqs.CPU58.RCU
129172 ± 5% +16.5% 150459 softirqs.CPU58.TIMER
35042 ± 16% +57.1% 55050 softirqs.CPU59.RCU
40606 ± 12% +60.2% 65069 softirqs.CPU6.RCU
109977 ± 9% +23.9% 136257 softirqs.CPU6.TIMER
35058 ± 16% +57.7% 55277 softirqs.CPU60.RCU
34951 ± 15% +52.2% 53188 softirqs.CPU61.RCU
35594 ± 14% +54.4% 54950 softirqs.CPU62.RCU
35217 ± 15% +52.7% 53790 softirqs.CPU63.RCU
33270 ± 15% +59.3% 52990 softirqs.CPU64.RCU
33326 ± 16% +55.1% 51696 softirqs.CPU65.RCU
33573 ± 15% +56.8% 52631 softirqs.CPU66.RCU
33582 ± 15% +53.8% 51642 softirqs.CPU67.RCU
32917 ± 15% +59.9% 52644 softirqs.CPU68.RCU
33037 ± 15% +63.0% 53849 softirqs.CPU69.RCU
39522 ± 10% +60.3% 63357 softirqs.CPU7.RCU
33226 ± 16% +59.2% 52908 softirqs.CPU70.RCU
32781 ± 15% +58.6% 51989 softirqs.CPU71.RCU
33064 ± 13% +80.6% 59702 softirqs.CPU72.RCU
32784 ± 14% +56.8% 51422 softirqs.CPU73.RCU
32988 ± 17% +60.6% 52968 softirqs.CPU74.RCU
32197 ± 15% +61.3% 51922 softirqs.CPU75.RCU
32210 ± 15% +55.6% 50127 softirqs.CPU76.RCU
31828 ± 15% +57.7% 50193 softirqs.CPU77.RCU
33704 ± 14% +52.1% 51273 softirqs.CPU78.RCU
33146 ± 17% +53.4% 50850 softirqs.CPU79.RCU
40019 ± 14% +53.5% 61416 softirqs.CPU8.RCU
122353 ± 9% +21.5% 148650 softirqs.CPU8.TIMER
32597 ± 15% +64.3% 53549 softirqs.CPU80.RCU
134840 ± 5% +15.2% 155292 softirqs.CPU80.TIMER
32221 ± 15% +64.6% 53052 softirqs.CPU81.RCU
33039 ± 15% +59.6% 52743 softirqs.CPU82.RCU
32787 ± 16% +59.1% 52164 softirqs.CPU83.RCU
32695 ± 15% +58.8% 51926 softirqs.CPU84.RCU
32467 ± 16% +64.0% 53258 softirqs.CPU85.RCU
136153 ± 7% +14.7% 156116 softirqs.CPU85.TIMER
32705 ± 16% +62.3% 53086 softirqs.CPU86.RCU
32601 ± 16% +62.1% 52854 softirqs.CPU87.RCU
32995 ± 15% +59.9% 52757 softirqs.CPU88.RCU
32822 ± 16% +59.3% 52298 softirqs.CPU89.RCU
40364 ± 14% +52.0% 61352 softirqs.CPU9.RCU
112908 ± 14% +33.1% 150306 softirqs.CPU9.TIMER
32824 ± 16% +62.8% 53450 softirqs.CPU90.RCU
32906 ± 16% +64.4% 54093 softirqs.CPU91.RCU
32596 ± 15% +61.9% 52772 softirqs.CPU92.RCU
32472 ± 16% +66.6% 54103 softirqs.CPU93.RCU
32897 ± 16% +60.2% 52703 softirqs.CPU94.RCU
32672 ± 16% +62.5% 53086 softirqs.CPU95.RCU
32311 ± 15% +57.7% 50960 softirqs.CPU96.RCU
32505 ± 15% +58.6% 51551 softirqs.CPU97.RCU
32357 ± 13% +64.3% 53148 softirqs.CPU98.RCU
32434 ± 16% +63.6% 53073 softirqs.CPU99.RCU
9259 ±127% +353.8% 42022 softirqs.NET_RX
9669747 ± 15% +59.0% 15377610 softirqs.RCU



***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/context_switch1/will-it-scale/0x11

commit:
b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")

b121b341e5983bdc 492e639f0c222784e2e0f121966
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
2:2 -100% :2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
2:2 -100% :2 dmesg.WARNING:stack_recursion





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen

---

Attachments:

(No filename) (107.52 kB)
config-5.7.0-rc2-01182-g492e639f0c222 (205.74 kB)
job-script (7.49 kB)
job.yaml (5.18 kB)
reproduce (356.00 B)
Download all attachments

2020-06-28 06:08:35

by Yonghong Song

[permalink] [raw]

Subject: Re: [bpf] will-it-scale.per_thread_ops 2.6% improvement

---

On 6/27/20 1:22 AM, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a 2.6% improvement of will-it-scale.per_thread_ops due to commit:

Thanks. The improvement is always good although I do not know
what will-it-scale.per_thread_ops exactly running.

But I think the commit
492e639f0c222784e2e0f121966375f641c61b15 ("bpf: Add bpf_seq_printf
and bpf_seq_write helpers")
probably not the reason for the improvement. This patch just add bpf
helpers and they should not be called in normal user codes.

If you want to identify which commit is responsible for the improvement,
you probably want to double check.

>
>
> commit: 492e639f0c222784e2e0f121966375f641c61b15 ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: will-it-scale
> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
> with following parameters:
>
> nr_task: 100%
> mode: thread
> test: pthread_mutex2
> cpufreq_governor: performance
> ucode: 0x11
>
> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> test-url: https://github.com/antonblanchard/will-it-scale
>
> In addition to that, the commit also has significant impact on the following tests:
>
> +------------------+----------------------------------------------------------------------+
> | testcase: change | will-it-scale: will-it-scale.per_process_ops 2.5% improvement |
> | test machine | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory |
> | test parameters | cpufreq_governor=performance |
> | | mode=process |
> | | nr_task=100% |
> | | test=malloc2 |
> | | ucode=0x11 |
> +------------------+----------------------------------------------------------------------+
>
>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
> gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/pthread_mutex2/will-it-scale/0x11
>
> commit:
> b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
> 492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")
>
> b121b341e5983bdc 492e639f0c222784e2e0f121966
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 6105969 +2.6% 6265564 will-it-scale.per_thread_ops
> 9855 +1196.5% 127775 ± 92% will-it-scale.time.maximum_resident_set_size
> 1.759e+09 +2.6% 1.804e+09 will-it-scale.workload
[...]
>
>
> ***************************************************************************************************
> lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
> =========================================================================================
> compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
> gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-knm01/context_switch1/will-it-scale/0x11
>
> commit:
> b121b341e5 ("bpf: Add PTR_TO_BTF_ID_OR_NULL support")
> 492e639f0c ("bpf: Add bpf_seq_printf and bpf_seq_write helpers")
>
> b121b341e5983bdc 492e639f0c222784e2e0f121966
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> 2:2 -100% :2 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
> 2:2 -100% :2 dmesg.WARNING:stack_recursion
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>