2020-10-12 07:00:37

by Xing Zhengjun

Subject: Re: [LKP] [mm] 5ef64cc898: vm-scalability.throughput -20.6% regression

Hi Linus,

Do you have time to look at this? Thanks. I re-test it in v5.9-rc8,
the regression still existed.


On 9/21/2020 5:23 PM, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -20.6% regression of vm-scalability.throughput due to commit:
>
>
> commit: 5ef64cc8987a9211d3f3667331ba3411a94ddc79 ("mm: allow a controlled amount of unfairness in the page lock")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
>
> in testcase: vm-scalability
> on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
> with following parameters:
>
> runtime: 300s
> size: 2T
> test: shm-xread-seq
> cpufreq_governor: performance
> ucode: 0xd6
>
> test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
> test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/
>
> In addition to that, the commit also has significant impact on the following tests:
>
> +------------------+---------------------------------------------------------------------------+
> | testcase: change | unixbench: unixbench.score 5.4% improvement |
> | test machine | 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory |
> | test parameters | cpufreq_governor=performance |
> | | nr_task=30% |
> | | runtime=300s |
> | | test=shell8 |
> | | ucode=0xd6 |
> +------------------+---------------------------------------------------------------------------+
> | testcase: change | will-it-scale: will-it-scale.per_thread_ops 688.3% improvement |
> | test machine | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
> | test parameters | cpufreq_governor=performance |
> | | mode=thread |
> | | nr_task=50% |
> | | test=readseek2 |
> | | ucode=0x5002f01 |
> +------------------+---------------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.timerfd.ops_per_sec -3.2% regression |
> | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | testtime=30s |
> | | ucode=0x5002f01 |
> +------------------+---------------------------------------------------------------------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <[email protected]>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
> gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/300s/2T/lkp-cfl-e1/shm-xread-seq/vm-scalability/0xd6
>
> commit:
> 5925fa68fe (" perf tools fixes for v5.9: 3rd batch")
> 5ef64cc898 ("mm: allow a controlled amount of unfairness in the page lock")
>
> 5925fa68fe824465 5ef64cc8987a9211d3f3667331b
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> 3:4 -34% 1:4 perf-profile.calltrace.cycles-pp.sync_regs.error_entry
> 3:4 -24% 2:4 perf-profile.calltrace.cycles-pp.error_entry
> 5:4 -30% 4:4 perf-profile.children.cycles-pp.error_entry
> 0:4 -4% 0:4 perf-profile.self.cycles-pp.error_entry
> %stddev %change %stddev
> \ | \
> 985306 -20.6% 782124 vm-scalability.median
> 5.42 ? 4% -2.3 3.15 ? 3% vm-scalability.median_stddev%
> 15764904 -20.6% 12513939 vm-scalability.throughput
> 173.16 +22.0% 211.25 vm-scalability.time.elapsed_time
> 173.16 +22.0% 211.25 vm-scalability.time.elapsed_time.max
> 15477 ? 4% +129.1% 35465 ? 3% vm-scalability.time.involuntary_context_switches
> 2.569e+08 +56.2% 4.013e+08 vm-scalability.time.minor_page_faults
> 1283 +12.8% 1447 vm-scalability.time.percent_of_cpu_this_job_got
> 971.71 +87.0% 1816 vm-scalability.time.system_time
> 56042863 -10.0% 50440135 vm-scalability.time.voluntary_context_switches
> 197.11 +19.3% 235.15 uptime.boot
> 1052993 -19.6% 846362 meminfo.Active
> 1052899 -19.6% 846266 meminfo.Active(anon)
> 105575 -17.9% 86662 meminfo.max_used_kB
> 20.13 -12.0 8.10 mpstat.cpu.all.idle%
> 0.61 ? 3% -0.6 0.01 ? 3% mpstat.cpu.all.iowait%
> 0.95 +0.6 1.53 mpstat.cpu.all.irq%
> 34.10 +19.4 53.49 mpstat.cpu.all.sys%
> 44.17 -7.4 36.81 mpstat.cpu.all.usr%
> 20.75 ? 2% -60.2% 8.25 ? 5% vmstat.cpu.id
> 34.25 +57.7% 54.00 vmstat.cpu.sy
> 43.25 -16.8% 36.00 vmstat.cpu.us
> 11.00 +27.3% 14.00 vmstat.procs.r
> 637460 -26.0% 471762 vmstat.system.cs
> 68148 +88.2% 128254 vmstat.system.in
> 41584830 ? 3% -86.3% 5680589 cpuidle.C1.time
> 7034552 -52.7% 3328990 cpuidle.C1.usage
> 166290 ? 42% -52.8% 78434 ? 30% cpuidle.C10.time
> 1.039e+08 -87.4% 13042277 ? 3% cpuidle.C1E.time
> 9612513 ? 2% -96.9% 294433 ? 3% cpuidle.C1E.usage
> 2.713e+08 -59.5% 1.1e+08 cpuidle.POLL.time
> 48132156 -40.9% 28439987 cpuidle.POLL.usage
> 264158 -19.1% 213736 proc-vmstat.nr_active_anon
> 2482625 -1.7% 2441126 proc-vmstat.nr_file_pages
> 2240838 -1.9% 2199269 proc-vmstat.nr_shmem
> 264158 -19.1% 213736 proc-vmstat.nr_zone_active_anon
> 40104796 +19.0% 47743573 proc-vmstat.numa_hit
> 40104796 +19.0% 47743573 proc-vmstat.numa_local
> 40189131 +19.0% 47826525 proc-vmstat.pgalloc_normal
> 2.572e+08 +56.2% 4.016e+08 proc-vmstat.pgfault
> 39751695 +17.5% 46709736 ? 2% proc-vmstat.pgfree
> 19586 ? 2% +25.4% 24554 proc-vmstat.pgreuse
> 12821 ? 11% +70.1% 21804 ? 7% softirqs.CPU0.RCU
> 55373 +58.2% 87598 softirqs.CPU0.SCHED
> 13132 ? 12% +56.6% 20564 ? 2% softirqs.CPU1.RCU
> 54173 ? 2% +63.0% 88299 softirqs.CPU1.SCHED
> 12063 ? 11% +56.0% 18815 ? 8% softirqs.CPU10.RCU
> 52997 ? 2% +63.7% 86776 softirqs.CPU10.SCHED
> 11146 ? 12% +73.5% 19343 ? 6% softirqs.CPU11.RCU
> 52858 +62.8% 86060 softirqs.CPU11.SCHED
> 12394 ? 20% +57.9% 19564 ? 6% softirqs.CPU12.RCU
> 52153 +66.2% 86659 softirqs.CPU12.SCHED
> 11714 ? 11% +58.9% 18619 ? 9% softirqs.CPU13.RCU
> 51630 +69.1% 87317 softirqs.CPU13.SCHED
> 11541 ? 15% +71.1% 19751 ? 10% softirqs.CPU14.RCU
> 52171 ? 2% +67.7% 87471 softirqs.CPU14.SCHED
> 12034 ? 13% +60.5% 19317 ? 8% softirqs.CPU15.RCU
> 52657 +65.2% 86972 softirqs.CPU15.SCHED
> 11964 ? 13% +75.0% 20934 ? 6% softirqs.CPU2.RCU
> 53736 +62.2% 87150 softirqs.CPU2.SCHED
> 11927 ? 11% +61.5% 19267 ? 9% softirqs.CPU3.RCU
> 52931 ? 2% +63.2% 86386 softirqs.CPU3.SCHED
> 12353 ? 15% +58.0% 19513 ? 6% softirqs.CPU4.RCU
> 51881 +68.0% 87174 softirqs.CPU4.SCHED
> 11154 ? 16% +72.4% 19234 ? 6% softirqs.CPU5.RCU
> 51954 +67.9% 87209 softirqs.CPU5.SCHED
> 11700 ? 12% +79.5% 21004 ? 11% softirqs.CPU6.RCU
> 51722 +68.9% 87358 softirqs.CPU6.SCHED
> 11795 ? 11% +79.5% 21169 ? 13% softirqs.CPU7.RCU
> 52777 +64.8% 86985 softirqs.CPU7.SCHED
> 11352 ? 16% +70.3% 19336 ? 7% softirqs.CPU8.RCU
> 53939 +61.1% 86897 softirqs.CPU8.SCHED
> 13223 ? 21% +54.5% 20423 ? 10% softirqs.CPU9.RCU
> 53570 ? 2% +61.4% 86435 softirqs.CPU9.SCHED
> 192323 ? 12% +65.7% 318668 ? 7% softirqs.RCU
> 846531 +64.5% 1392755 softirqs.SCHED
> 13340 ? 3% +28.3% 17120 ? 3% softirqs.TIMER
> 5616584 +255.3% 19953996 interrupts.CAL:Function_call_interrupts
> 403342 ? 4% +225.2% 1311736 interrupts.CPU0.CAL:Function_call_interrupts
> 348468 +21.9% 424744 interrupts.CPU0.LOC:Local_timer_interrupts
> 28665 ? 4% -21.7% 22431 ? 2% interrupts.CPU0.RES:Rescheduling_interrupts
> 385359 +233.7% 1285782 interrupts.CPU1.CAL:Function_call_interrupts
> 348549 +21.8% 424706 interrupts.CPU1.LOC:Local_timer_interrupts
> 28794 ? 3% -17.9% 23633 interrupts.CPU1.RES:Rescheduling_interrupts
> 369509 ? 3% +240.9% 1259663 interrupts.CPU10.CAL:Function_call_interrupts
> 348480 +21.9% 424783 interrupts.CPU10.LOC:Local_timer_interrupts
> 30068 ? 3% -18.6% 24464 ? 2% interrupts.CPU10.RES:Rescheduling_interrupts
> 353201 ? 2% +246.8% 1225041 interrupts.CPU11.CAL:Function_call_interrupts
> 348488 +21.9% 424702 interrupts.CPU11.LOC:Local_timer_interrupts
> 30935 ? 3% -22.9% 23861 ? 5% interrupts.CPU11.RES:Rescheduling_interrupts
> 323460 ? 2% +276.6% 1218231 interrupts.CPU12.CAL:Function_call_interrupts
> 348410 +21.9% 424849 interrupts.CPU12.LOC:Local_timer_interrupts
> 31301 ? 3% -24.0% 23799 ? 4% interrupts.CPU12.RES:Rescheduling_interrupts
> 306249 +291.8% 1199851 interrupts.CPU13.CAL:Function_call_interrupts
> 348436 +21.9% 424721 interrupts.CPU13.LOC:Local_timer_interrupts
> 31048 ? 3% -19.3% 25040 ? 3% interrupts.CPU13.RES:Rescheduling_interrupts
> 311947 ? 2% +291.1% 1220172 interrupts.CPU14.CAL:Function_call_interrupts
> 348554 +21.8% 424574 interrupts.CPU14.LOC:Local_timer_interrupts
> 30893 ? 4% -18.7% 25116 ? 4% interrupts.CPU14.RES:Rescheduling_interrupts
> 339980 ? 2% +259.0% 1220678 interrupts.CPU15.CAL:Function_call_interrupts
> 348418 +21.9% 424803 interrupts.CPU15.LOC:Local_timer_interrupts
> 29624 ? 3% -16.7% 24674 interrupts.CPU15.RES:Rescheduling_interrupts
> 369199 ? 2% +241.6% 1261224 interrupts.CPU2.CAL:Function_call_interrupts
> 348518 +21.9% 424863 interrupts.CPU2.LOC:Local_timer_interrupts
> 29702 ? 2% -21.3% 23377 ? 4% interrupts.CPU2.RES:Rescheduling_interrupts
> 352942 +253.9% 1249080 interrupts.CPU3.CAL:Function_call_interrupts
> 348266 +21.9% 424684 interrupts.CPU3.LOC:Local_timer_interrupts
> 31038 -25.3% 23178 ? 4% interrupts.CPU3.RES:Rescheduling_interrupts
> 327561 +272.0% 1218661 interrupts.CPU4.CAL:Function_call_interrupts
> 348447 +21.9% 424700 interrupts.CPU4.LOC:Local_timer_interrupts
> 30527 -22.9% 23534 ? 3% interrupts.CPU4.RES:Rescheduling_interrupts
> 311428 ? 2% +286.3% 1202941 interrupts.CPU5.CAL:Function_call_interrupts
> 348533 +21.9% 424787 interrupts.CPU5.LOC:Local_timer_interrupts
> 30825 ? 4% -13.7% 26600 ? 5% interrupts.CPU5.RES:Rescheduling_interrupts
> 319963 ? 2% +287.0% 1238272 interrupts.CPU6.CAL:Function_call_interrupts
> 348246 +22.0% 424716 interrupts.CPU6.LOC:Local_timer_interrupts
> 29411 ? 4% -20.2% 23471 ? 2% interrupts.CPU6.RES:Rescheduling_interrupts
> 344529 ? 3% +263.6% 1252800 interrupts.CPU7.CAL:Function_call_interrupts
> 348501 +21.8% 424469 interrupts.CPU7.LOC:Local_timer_interrupts
> 30725 ? 3% -20.7% 24366 ? 5% interrupts.CPU7.RES:Rescheduling_interrupts
> 408919 ? 5% +219.7% 1307498 interrupts.CPU8.CAL:Function_call_interrupts
> 348306 +22.0% 424854 interrupts.CPU8.LOC:Local_timer_interrupts
> 27910 ? 4% -19.9% 22361 ? 2% interrupts.CPU8.RES:Rescheduling_interrupts
> 388993 ? 3% +229.7% 1282361 interrupts.CPU9.CAL:Function_call_interrupts
> 348529 +21.9% 424872 interrupts.CPU9.LOC:Local_timer_interrupts
> 28499 ? 2% -13.0% 24791 ? 4% interrupts.CPU9.RES:Rescheduling_interrupts
> 5575155 +21.9% 6795831 interrupts.LOC:Local_timer_interrupts
> 479969 ? 2% -19.8% 384701 interrupts.RES:Rescheduling_interrupts
> 239.00 ? 4% -44.9% 131.75 ? 11% interrupts.TLB:TLB_shootdowns
> 48085 +69.4% 81451 sched_debug.cfs_rq:/.exec_clock.avg
> 49258 +67.3% 82414 sched_debug.cfs_rq:/.exec_clock.max
> 46356 +73.3% 80327 sched_debug.cfs_rq:/.exec_clock.min
> 80499 ? 81% -60.4% 31857 ? 3% sched_debug.cfs_rq:/.load.stddev
> 127.01 ? 9% -16.2% 106.49 ? 13% sched_debug.cfs_rq:/.load_avg.avg
> 642.42 ? 23% -30.5% 446.56 ? 3% sched_debug.cfs_rq:/.load_avg.max
> 169.53 ? 17% -29.5% 119.49 ? 12% sched_debug.cfs_rq:/.load_avg.stddev
> 588910 +87.0% 1101108 sched_debug.cfs_rq:/.min_vruntime.avg
> 606363 +84.1% 1116408 sched_debug.cfs_rq:/.min_vruntime.max
> 561578 ? 2% +91.7% 1076386 sched_debug.cfs_rq:/.min_vruntime.min
> 0.71 ? 8% +24.8% 0.89 sched_debug.cfs_rq:/.nr_running.avg
> 0.45 ? 9% -47.2% 0.24 sched_debug.cfs_rq:/.nr_running.stddev
> 0.73 ? 18% +59.6% 1.16 ? 11% sched_debug.cfs_rq:/.nr_spread_over.avg
> 1559 ? 5% +9.0% 1699 ? 5% sched_debug.cfs_rq:/.runnable_avg.max
> 381.23 ? 4% +62.0% 617.48 ? 5% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 1129 ? 8% +20.4% 1359 ? 6% sched_debug.cfs_rq:/.util_est_enqueued.max
> 77869 ? 6% -37.4% 48731 ? 8% sched_debug.cpu.avg_idle.avg
> 399988 ? 12% -30.9% 276240 ? 23% sched_debug.cpu.avg_idle.max
> 96550 ? 12% -34.3% 63478 ? 24% sched_debug.cpu.avg_idle.stddev
> 83560 +35.7% 113424 sched_debug.cpu.clock.avg
> 83561 +35.7% 113425 sched_debug.cpu.clock.max
> 83558 +35.7% 113423 sched_debug.cpu.clock.min
> 0.78 ? 9% -35.6% 0.50 ? 4% sched_debug.cpu.clock.stddev
> 82827 +35.1% 111904 sched_debug.cpu.clock_task.avg
> 82999 +35.0% 112089 sched_debug.cpu.clock_task.max
> 82262 +35.2% 111189 sched_debug.cpu.clock_task.min
> 1618 ? 6% +48.8% 2408 ? 2% sched_debug.cpu.curr->pid.avg
> 2493 +16.5% 2905 sched_debug.cpu.curr->pid.max
> 985.88 ? 8% -28.7% 703.01 ? 14% sched_debug.cpu.curr->pid.stddev
> 0.79 ? 5% +24.7% 0.98 sched_debug.cpu.nr_running.avg
> 0.54 ? 11% -19.9% 0.44 ? 3% sched_debug.cpu.nr_running.stddev
> 2444639 +11.6% 2728125 sched_debug.cpu.nr_switches.avg
> 2466248 +18.3% 2917708 sched_debug.cpu.nr_switches.max
> 13988 ? 27% +642.0% 103791 ? 3% sched_debug.cpu.nr_switches.stddev
> 0.21 ? 22% -75.6% 0.05 ? 76% sched_debug.cpu.nr_uninterruptible.avg
> 2440164 +11.6% 2723801 sched_debug.cpu.sched_count.avg
> 2457113 +18.4% 2909974 sched_debug.cpu.sched_count.max
> 12769 ? 28% +704.0% 102671 ? 3% sched_debug.cpu.sched_count.stddev
> 1219314 +11.5% 1359077 sched_debug.cpu.sched_goidle.avg
> 1227676 +18.3% 1452278 sched_debug.cpu.sched_goidle.max
> 6331 ? 28% +710.4% 51313 ? 3% sched_debug.cpu.sched_goidle.stddev
> 1220257 +11.7% 1363456 sched_debug.cpu.ttwu_count.avg
> 1251629 +11.3% 1392991 sched_debug.cpu.ttwu_count.max
> 1196282 +10.0% 1316458 sched_debug.cpu.ttwu_count.min
> 17387 ? 8% +16.8% 20308 ? 10% sched_debug.cpu.ttwu_count.stddev
> 790.29 ? 3% +163.0% 2078 ? 5% sched_debug.cpu.ttwu_local.avg
> 1612 ? 20% +87.9% 3028 ? 8% sched_debug.cpu.ttwu_local.max
> 531.25 ? 6% +148.4% 1319 ? 18% sched_debug.cpu.ttwu_local.min
> 283.96 ? 23% +49.2% 423.75 ? 12% sched_debug.cpu.ttwu_local.stddev
> 83560 +35.7% 113423 sched_debug.cpu_clk
> 82663 +36.2% 112557 sched_debug.ktime
> 84851 +35.2% 114688 sched_debug.sched_clk
> 32.08 +6.1% 34.03 perf-stat.i.MPKI
> 1.672e+10 -16.1% 1.403e+10 perf-stat.i.branch-instructions
> 0.26 +0.0 0.31 perf-stat.i.branch-miss-rate%
> 33778666 +9.6% 37021638 perf-stat.i.branch-misses
> 1.77 ? 2% -0.4 1.35 perf-stat.i.cache-miss-rate%
> 20527502 ? 2% -19.2% 16588407 perf-stat.i.cache-misses
> 649397 -26.3% 478422 perf-stat.i.context-switches
> 1.28 +13.5% 1.46 perf-stat.i.cpi
> 5.822e+10 +5.2% 6.125e+10 perf-stat.i.cpu-cycles
> 209.75 ? 6% +22.7% 257.35 ? 3% perf-stat.i.cpu-migrations
> 3159 +21.6% 3843 perf-stat.i.cycles-between-cache-misses
> 0.05 ? 5% +0.0 0.07 ? 6% perf-stat.i.dTLB-load-miss-rate%
> 8766755 ? 5% +14.0% 9996512 ? 5% perf-stat.i.dTLB-load-misses
> 1.572e+10 -15.2% 1.333e+10 perf-stat.i.dTLB-loads
> 0.05 +0.0 0.07 perf-stat.i.dTLB-store-miss-rate%
> 1534360 +27.2% 1951224 perf-stat.i.dTLB-store-misses
> 3.172e+09 -10.9% 2.826e+09 perf-stat.i.dTLB-stores
> 78.19 +15.6 93.74 perf-stat.i.iTLB-load-miss-rate%
> 6626547 -23.1% 5098140 perf-stat.i.iTLB-load-misses
> 1936185 ? 3% -84.8% 294324 perf-stat.i.iTLB-loads
> 5.451e+10 -15.1% 4.625e+10 perf-stat.i.instructions
> 0.94 -19.5% 0.76 perf-stat.i.ipc
> 3.64 +5.2% 3.83 perf-stat.i.metric.GHz
> 0.07 ? 8% -25.7% 0.06 ? 14% perf-stat.i.metric.K/sec
> 2310 -14.7% 1971 perf-stat.i.metric.M/sec
> 1488490 +27.9% 1903399 perf-stat.i.minor-faults
> 195100 ? 2% -26.6% 143181 perf-stat.i.node-loads
> 13149630 -18.3% 10746838 perf-stat.i.node-stores
> 1488491 +27.9% 1903399 perf-stat.i.page-faults
> 24.51 +17.8% 28.88 perf-stat.overall.MPKI
> 0.20 +0.1 0.26 perf-stat.overall.branch-miss-rate%
> 1.54 ? 2% -0.3 1.24 perf-stat.overall.cache-miss-rate%
> 1.07 +24.0% 1.32 perf-stat.overall.cpi
> 2837 ? 2% +30.1% 3691 perf-stat.overall.cycles-between-cache-misses
> 0.06 ? 5% +0.0 0.07 ? 5% perf-stat.overall.dTLB-load-miss-rate%
> 0.05 +0.0 0.07 perf-stat.overall.dTLB-store-miss-rate%
> 77.39 +17.1 94.54 perf-stat.overall.iTLB-load-miss-rate%
> 8229 ? 2% +10.3% 9073 perf-stat.overall.instructions-per-iTLB-miss
> 0.94 -19.3% 0.76 perf-stat.overall.ipc
> 3984 +3.7% 4129 perf-stat.overall.path-length
> 1.662e+10 -16.0% 1.396e+10 perf-stat.ps.branch-instructions
> 33589028 +9.7% 36852570 perf-stat.ps.branch-misses
> 20412439 ? 2% -19.1% 16512315 perf-stat.ps.cache-misses
> 645542 -26.3% 476080 perf-stat.ps.context-switches
> 5.788e+10 +5.3% 6.096e+10 perf-stat.ps.cpu-cycles
> 208.60 ? 5% +23.0% 256.52 ? 3% perf-stat.ps.cpu-migrations
> 8715455 ? 5% +14.1% 9947519 ? 5% perf-stat.ps.dTLB-load-misses
> 1.563e+10 -15.1% 1.327e+10 perf-stat.ps.dTLB-loads
> 1525309 +27.3% 1941693 perf-stat.ps.dTLB-store-misses
> 3.153e+09 -10.8% 2.812e+09 perf-stat.ps.dTLB-stores
> 6587329 -23.0% 5073245 perf-stat.ps.iTLB-load-misses
> 1924701 ? 3% -84.8% 292898 perf-stat.ps.iTLB-loads
> 5.419e+10 -15.1% 4.603e+10 perf-stat.ps.instructions
> 1479701 +28.0% 1894095 perf-stat.ps.minor-faults
> 194078 ? 2% -26.5% 142593 perf-stat.ps.node-loads
> 13072694 -18.2% 10694520 perf-stat.ps.node-stores
> 1479702 +28.0% 1894095 perf-stat.ps.page-faults
> 9.416e+12 +3.7% 9.76e+12 perf-stat.total.instructions
> 28.81 ? 37% -21.6 7.25 ? 2% perf-profile.calltrace.cycles-pp.secondary_startup_64
> 27.38 ? 37% -20.6 6.82 ? 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 27.38 ? 37% -20.6 6.82 ? 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 27.34 ? 37% -20.5 6.79 ? 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 23.22 ? 37% -18.7 4.48 ? 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 23.20 ? 37% -18.7 4.47 ? 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> 15.78 ? 39% -12.4 3.42 ? 4% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 7.97 ? 37% -7.1 0.87 ? 15% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 5.85 ? 40% -5.0 0.82 ? 13% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_page_function.__wake_up_common.wake_up_page_bit.do_fault
> 6.11 ? 40% -4.9 1.19 ? 12% perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.do_fault.__handle_mm_fault.handle_mm_fault
> 6.02 ? 41% -4.9 1.12 ? 12% perf-profile.calltrace.cycles-pp.wake_page_function.__wake_up_common.wake_up_page_bit.do_fault.__handle_mm_fault
> 4.33 ? 40% -4.0 0.28 ?100% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.wake_page_function.__wake_up_common.wake_up_page_bit
> 4.31 ? 40% -4.0 0.28 ?100% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.wake_page_function.__wake_up_common
> 4.14 ? 40% -3.9 0.27 ?100% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.wake_page_function
> 2.32 ? 42% -1.6 0.74 ? 12% perf-profile.calltrace.cycles-pp.schedule.io_schedule.__lock_page.find_lock_entry.shmem_getpage_gfp
> 2.27 ? 42% -1.5 0.73 ? 12% perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.__lock_page.find_lock_entry
> 2.37 ? 42% -1.5 0.89 ? 18% perf-profile.calltrace.cycles-pp.io_schedule.__lock_page.find_lock_entry.shmem_getpage_gfp.shmem_fault
> 0.28 ?100% +0.4 0.63 ? 4% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.asm_call_on_stack.sysvec_call_function_single
> 0.27 ?100% +0.4 0.62 ? 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.asm_call_on_stack
> 0.30 ?100% +0.4 0.68 ? 4% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.asm_call_on_stack.sysvec_call_function_single.asm_sysvec_call_function_single
> 0.32 ?100% +0.4 0.74 ? 5% perf-profile.calltrace.cycles-pp.asm_call_on_stack.sysvec_call_function_single.asm_sysvec_call_function_single.finish_task_switch.__schedule
> 0.32 ?100% +0.4 0.74 ? 5% perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.asm_call_on_stack.sysvec_call_function_single.asm_sysvec_call_function_single.finish_task_switch
> 0.14 ?173% +0.5 0.59 ? 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single
> 0.35 ?100% +0.5 0.81 ? 5% perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.finish_task_switch.__schedule.schedule_idle
> 0.40 ?100% +0.5 0.90 ? 6% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
> 0.14 ?173% +0.5 0.65 ? 4% perf-profile.calltrace.cycles-pp.release_pages.pagevec_lru_move_fn.activate_page.mark_page_accessed.zap_pte_range
> 0.38 ?100% +0.5 0.89 ? 5% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.finish_task_switch.__schedule.schedule_idle.do_idle
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlinkat
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.unlinkat
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlinkat
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.shmem_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.shmem_truncate_range.shmem_evict_inode.evict.do_unlinkat.do_syscall_64
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.calltrace.cycles-pp.shmem_undo_range.shmem_truncate_range.shmem_evict_inode.evict.do_unlinkat
> 0.26 ?173% +0.9 1.13 ? 5% perf-profile.calltrace.cycles-pp.__activate_page.pagevec_lru_move_fn.activate_page.mark_page_accessed.zap_pte_range
> 0.28 ?173% +1.2 1.50 ? 4% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.zap_pte_range.unmap_page_range.unmap_vmas
> 0.37 ?173% +1.5 1.90 ? 5% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap
> 0.73 ?173% +2.2 2.89 ? 3% perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap
> 0.76 ?173% +3.0 3.73 ? 3% perf-profile.calltrace.cycles-pp.workingset_age_nonresident.workingset_activation.mark_page_accessed.zap_pte_range.unmap_page_range
> 0.91 ?173% +3.4 4.34 ? 3% perf-profile.calltrace.cycles-pp.workingset_activation.mark_page_accessed.zap_pte_range.unmap_page_range.unmap_vmas
> 0.00 +4.7 4.65 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.wake_up_page_bit.do_fault.__handle_mm_fault
> 0.00 +4.8 4.78 ? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.wake_up_page_bit.do_fault.__handle_mm_fault.handle_mm_fault
> 1.56 ?173% +5.9 7.47 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.pagevec_lru_move_fn.activate_page.mark_page_accessed
> 1.57 ?173% +6.0 7.53 ? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.pagevec_lru_move_fn.activate_page.mark_page_accessed.zap_pte_range
> 0.00 +7.3 7.34 ? 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.wake_up_page_bit.filemap_map_pages.do_fault
> 0.00 +7.5 7.46 ? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.wake_up_page_bit.filemap_map_pages.do_fault.__handle_mm_fault
> 2.07 ?173% +7.7 9.74 ? 3% perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.activate_page.mark_page_accessed.zap_pte_range.unmap_page_range
> 0.37 ?100% +8.2 8.52 ? 2% perf-profile.calltrace.cycles-pp.wake_up_page_bit.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault
> 2.41 ?173% +8.7 11.09 ? 3% perf-profile.calltrace.cycles-pp.activate_page.mark_page_accessed.zap_pte_range.unmap_page_range.unmap_vmas
> 18.70 ? 39% +11.2 29.95 ? 2% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
> 4.12 ?173% +14.6 18.74 ? 3% perf-profile.calltrace.cycles-pp.mark_page_accessed.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap
> 5.98 ?173% +20.5 26.51 ? 3% perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap.mmput
> 6.00 ?173% +20.6 26.57 ? 3% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
> 6.00 ?173% +20.6 26.57 ? 3% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
> 6.00 ?173% +20.6 26.60 ? 3% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 6.00 ?173% +20.6 26.60 ? 3% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 28.81 ? 37% -21.6 7.25 ? 2% perf-profile.children.cycles-pp.secondary_startup_64
> 28.81 ? 37% -21.6 7.25 ? 2% perf-profile.children.cycles-pp.cpu_startup_entry
> 28.77 ? 37% -21.6 7.22 ? 2% perf-profile.children.cycles-pp.do_idle
> 27.38 ? 37% -20.6 6.82 ? 2% perf-profile.children.cycles-pp.start_secondary
> 24.40 ? 37% -19.6 4.76 ? 3% perf-profile.children.cycles-pp.cpuidle_enter
> 24.38 ? 37% -19.6 4.76 ? 3% perf-profile.children.cycles-pp.cpuidle_enter_state
> 15.86 ? 38% -12.2 3.63 ? 3% perf-profile.children.cycles-pp.poll_idle
> 8.17 ? 36% -7.3 0.92 ? 13% perf-profile.children.cycles-pp.intel_idle
> 6.35 ? 38% -4.3 2.06 ? 4% perf-profile.children.cycles-pp.wake_page_function
> 6.45 ? 38% -4.3 2.20 ? 4% perf-profile.children.cycles-pp.__wake_up_common
> 6.17 ? 38% -4.2 1.95 ? 4% perf-profile.children.cycles-pp.try_to_wake_up
> 5.04 ? 38% -3.0 2.06 ? 3% perf-profile.children.cycles-pp.ttwu_do_activate
> 5.02 ? 38% -3.0 2.04 ? 3% perf-profile.children.cycles-pp.enqueue_task_fair
> 4.84 ? 38% -2.9 1.97 ? 3% perf-profile.children.cycles-pp.enqueue_entity
> 4.02 ? 38% -2.4 1.62 ? 3% perf-profile.children.cycles-pp.__account_scheduler_latency
> 3.40 ? 38% -2.0 1.36 ? 2% perf-profile.children.cycles-pp.stack_trace_save_tsk
> 3.17 ? 39% -1.9 1.27 perf-profile.children.cycles-pp.arch_stack_walk
> 2.06 ? 38% -1.3 0.79 perf-profile.children.cycles-pp.unwind_next_frame
> 2.40 ? 40% -1.2 1.17 perf-profile.children.cycles-pp.io_schedule
> 2.36 ? 40% -1.2 1.15 perf-profile.children.cycles-pp.schedule
> 1.42 ? 52% -1.0 0.42 ? 15% perf-profile.children.cycles-pp.start_kernel
> 1.34 ? 38% -1.0 0.38 ? 4% perf-profile.children.cycles-pp.menu_select
> 1.22 ? 40% -0.7 0.57 perf-profile.children.cycles-pp.dequeue_task_fair
> 1.08 ? 41% -0.6 0.50 perf-profile.children.cycles-pp.dequeue_entity
> 0.95 ? 38% -0.5 0.40 ? 3% perf-profile.children.cycles-pp.pick_next_task_fair
> 0.80 ? 37% -0.4 0.39 ? 2% perf-profile.children.cycles-pp.update_load_avg
> 0.62 ? 39% -0.4 0.24 ? 6% perf-profile.children.cycles-pp.set_next_entity
> 0.50 ? 38% -0.3 0.16 ? 7% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
> 0.48 ? 38% -0.3 0.21 ? 7% perf-profile.children.cycles-pp.__orc_find
> 0.49 ? 44% -0.3 0.22 ? 3% perf-profile.children.cycles-pp.update_curr
> 0.44 ? 42% -0.3 0.18 ? 4% perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
> 0.36 ? 38% -0.2 0.13 ? 8% perf-profile.children.cycles-pp.select_task_rq_fair
> 0.44 ? 39% -0.2 0.21 ? 6% perf-profile.children.cycles-pp.unwind_get_return_address
> 0.37 ? 39% -0.2 0.17 ? 7% perf-profile.children.cycles-pp.__kernel_text_address
> 0.34 ? 36% -0.2 0.14 ? 5% perf-profile.children.cycles-pp.orc_find
> 0.33 ? 40% -0.2 0.17 ? 3% perf-profile.children.cycles-pp.update_rq_clock
> 0.26 ? 33% -0.2 0.09 ? 12% perf-profile.children.cycles-pp.tick_nohz_next_event
> 0.29 ? 36% -0.2 0.12 ? 4% perf-profile.children.cycles-pp.__switch_to_asm
> 0.30 ? 41% -0.2 0.14 ? 10% perf-profile.children.cycles-pp.kernel_text_address
> 0.28 ? 35% -0.1 0.14 ? 9% perf-profile.children.cycles-pp.sched_clock_cpu
> 0.24 ? 36% -0.1 0.10 ? 4% perf-profile.children.cycles-pp.ktime_get
> 0.26 ? 36% -0.1 0.12 ? 8% perf-profile.children.cycles-pp.sched_clock
> 0.25 ? 35% -0.1 0.12 ? 11% perf-profile.children.cycles-pp.native_sched_clock
> 0.25 ? 41% -0.1 0.12 ? 5% perf-profile.children.cycles-pp.__switch_to
> 0.18 ? 35% -0.1 0.06 ? 11% perf-profile.children.cycles-pp.tick_nohz_idle_exit
> 0.18 ? 40% -0.1 0.06 ? 7% perf-profile.children.cycles-pp.hrtimer_next_event_without
> 0.19 ? 43% -0.1 0.07 perf-profile.children.cycles-pp.in_sched_functions
> 0.16 ? 42% -0.1 0.04 ? 60% perf-profile.children.cycles-pp.update_cfs_rq_h_load
> 0.17 ? 39% -0.1 0.05 ? 58% perf-profile.children.cycles-pp.get_next_timer_interrupt
> 0.22 ? 39% -0.1 0.11 ? 7% perf-profile.children.cycles-pp.__update_load_avg_se
> 0.18 ? 39% -0.1 0.07 ? 14% perf-profile.children.cycles-pp.ttwu_do_wakeup
> 0.16 ? 41% -0.1 0.07 ? 7% perf-profile.children.cycles-pp.check_preempt_curr
> 0.11 ? 39% -0.1 0.03 ?100% perf-profile.children.cycles-pp.switch_mm_irqs_off
> 0.14 ? 40% -0.1 0.05 ? 8% perf-profile.children.cycles-pp.stack_access_ok
> 0.14 ? 40% -0.1 0.05 ? 9% perf-profile.children.cycles-pp.__unwind_start
> 0.12 ? 42% -0.1 0.04 ? 57% perf-profile.children.cycles-pp.reweight_entity
> 0.15 ? 39% -0.1 0.07 ? 11% perf-profile.children.cycles-pp.read_tsc
> 0.15 ? 32% -0.1 0.08 ? 16% perf-profile.children.cycles-pp.delayacct_end
> 0.10 ? 39% -0.1 0.04 ? 58% perf-profile.children.cycles-pp.account_entity_dequeue
> 0.03 ?100% +0.0 0.07 ? 7% perf-profile.children.cycles-pp.native_apic_msr_eoi_write
> 0.01 ?173% +0.0 0.06 ? 7% perf-profile.children.cycles-pp.xas_init_marks
> 0.03 ?100% +0.0 0.07 ? 11% perf-profile.children.cycles-pp.__unlock_page_memcg
> 0.00 +0.1 0.06 ? 9% perf-profile.children.cycles-pp.rcu_all_qs
> 0.06 ? 61% +0.1 0.12 ? 5% perf-profile.children.cycles-pp.__mod_memcg_state
> 0.07 ? 41% +0.1 0.13 ? 17% perf-profile.children.cycles-pp.xas_store
> 0.05 ? 64% +0.1 0.11 ? 4% perf-profile.children.cycles-pp.__mod_node_page_state
> 0.02 ?173% +0.1 0.09 perf-profile.children.cycles-pp.unaccount_page_cache_page
> 0.02 ?173% +0.1 0.09 ? 7% perf-profile.children.cycles-pp.__tlb_remove_page_size
> 0.07 ? 68% +0.1 0.17 ? 9% perf-profile.children.cycles-pp.__mod_lruvec_state
> 0.07 ? 50% +0.1 0.18 ? 10% perf-profile.children.cycles-pp.serial8250_console_write
> 0.02 ?173% +0.1 0.12 ? 12% perf-profile.children.cycles-pp.devkmsg_write.cold
> 0.02 ?173% +0.1 0.12 ? 12% perf-profile.children.cycles-pp.devkmsg_emit
> 0.03 ?173% +0.1 0.14 ? 11% perf-profile.children.cycles-pp.io_serial_in
> 0.04 ?118% +0.1 0.15 ? 9% perf-profile.children.cycles-pp.serial8250_console_putchar
> 0.03 ?102% +0.1 0.14 ? 13% perf-profile.children.cycles-pp.ksys_write
> 0.03 ?102% +0.1 0.14 ? 13% perf-profile.children.cycles-pp.vfs_write
> 0.03 ?102% +0.1 0.14 ? 13% perf-profile.children.cycles-pp.new_sync_write
> 0.04 ?118% +0.1 0.16 ? 11% perf-profile.children.cycles-pp.wait_for_xmitr
> 0.02 ?173% +0.1 0.13 ? 11% perf-profile.children.cycles-pp.write
> 0.05 ?114% +0.1 0.17 ? 11% perf-profile.children.cycles-pp.uart_console_write
> 0.04 ?173% +0.1 0.15 ? 16% perf-profile.children.cycles-pp.process_one_work
> 0.03 ?173% +0.1 0.15 ? 16% perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
> 0.04 ?173% +0.1 0.16 ? 17% perf-profile.children.cycles-pp.worker_thread
> 0.03 ?173% +0.1 0.16 ? 14% perf-profile.children.cycles-pp.memcpy_erms
> 0.13 ? 37% +0.1 0.26 ? 4% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
> 0.04 ?173% +0.1 0.16 ? 15% perf-profile.children.cycles-pp.ret_from_fork
> 0.09 ? 67% +0.1 0.21 ? 12% perf-profile.children.cycles-pp.vprintk_emit
> 0.04 ?173% +0.1 0.16 ? 15% perf-profile.children.cycles-pp.kthread
> 0.03 ?173% +0.1 0.16 ? 8% perf-profile.children.cycles-pp.free_pcppages_bulk
> 0.08 ? 61% +0.1 0.21 ? 12% perf-profile.children.cycles-pp.console_unlock
> 0.06 ?129% +0.2 0.23 ? 6% perf-profile.children.cycles-pp.free_unref_page_list
> 0.07 ?118% +0.2 0.26 ? 7% perf-profile.children.cycles-pp.__delete_from_page_cache
> 0.09 ?127% +0.2 0.33 ? 8% perf-profile.children.cycles-pp.__pagevec_release
> 0.10 ?120% +0.2 0.34 ? 9% perf-profile.children.cycles-pp.delete_from_page_cache
> 0.06 ?173% +0.2 0.31 ? 9% perf-profile.children.cycles-pp.mem_cgroup_page_lruvec
> 0.10 ?120% +0.3 0.38 ? 7% perf-profile.children.cycles-pp.truncate_inode_page
> 0.08 ?173% +0.3 0.39 ? 8% perf-profile.children.cycles-pp.free_pages_and_swap_cache
> 0.34 ? 51% +0.3 0.69 ? 5% perf-profile.children.cycles-pp.lock_page_memcg
> 0.54 ? 46% +0.4 0.91 ? 3% perf-profile.children.cycles-pp.sched_ttwu_pending
> 0.25 ? 76% +0.4 0.62 ? 4% perf-profile.children.cycles-pp.unlock_page_memcg
> 0.55 ? 43% +0.4 0.93 ? 3% perf-profile.children.cycles-pp.find_get_entry
> 0.55 ? 45% +0.4 0.93 ? 3% perf-profile.children.cycles-pp.__sysvec_call_function_single
> 0.60 ? 44% +0.4 1.01 ? 4% perf-profile.children.cycles-pp.sysvec_call_function_single
> 0.86 ? 35% +0.4 1.28 ? 3% perf-profile.children.cycles-pp.asm_call_on_stack
> 0.66 ? 44% +0.5 1.12 ? 3% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
> 0.75 ? 30% +0.5 1.30 ? 4% perf-profile.children.cycles-pp.PageHuge
> 0.23 ?123% +0.6 0.83 ? 7% perf-profile.children.cycles-pp.do_unlinkat
> 0.23 ?123% +0.6 0.83 ? 7% perf-profile.children.cycles-pp.evict
> 0.23 ?123% +0.6 0.83 ? 7% perf-profile.children.cycles-pp.shmem_evict_inode
> 0.23 ?123% +0.6 0.83 ? 7% perf-profile.children.cycles-pp.shmem_truncate_range
> 0.23 ?123% +0.6 0.83 ? 7% perf-profile.children.cycles-pp.shmem_undo_range
> 0.18 ?173% +0.7 0.83 ? 7% perf-profile.children.cycles-pp.unlinkat
> 0.26 ?173% +0.9 1.14 ? 5% perf-profile.children.cycles-pp.__activate_page
> 0.37 ?173% +1.6 1.93 ? 6% perf-profile.children.cycles-pp.tlb_flush_mmu
> 0.57 ?147% +2.0 2.54 ? 4% perf-profile.children.cycles-pp.release_pages
> 0.74 ?173% +2.2 2.90 ? 3% perf-profile.children.cycles-pp.page_remove_rmap
> 0.76 ?173% +3.0 3.74 ? 3% perf-profile.children.cycles-pp.workingset_age_nonresident
> 0.91 ?173% +3.4 4.34 ? 3% perf-profile.children.cycles-pp.workingset_activation
> 2.37 ?148% +7.6 9.94 ? 3% perf-profile.children.cycles-pp.pagevec_lru_move_fn
> 7.14 ? 38% +7.8 14.90 ? 3% perf-profile.children.cycles-pp.wake_up_page_bit
> 2.41 ?173% +8.7 11.09 ? 3% perf-profile.children.cycles-pp.activate_page
> 18.80 ? 39% +11.2 30.01 ? 2% perf-profile.children.cycles-pp.filemap_map_pages
> 4.13 ?173% +14.6 18.76 ? 3% perf-profile.children.cycles-pp.mark_page_accessed
> 4.09 ? 52% +16.8 20.87 ? 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 2.63 ? 93% +17.4 20.05 ? 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 6.00 ?173% +20.6 26.56 ? 3% perf-profile.children.cycles-pp.zap_pte_range
> 6.00 ?173% +20.6 26.57 ? 3% perf-profile.children.cycles-pp.unmap_vmas
> 6.00 ?173% +20.6 26.57 ? 3% perf-profile.children.cycles-pp.unmap_page_range
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.children.cycles-pp.mmput
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.children.cycles-pp.exit_mmap
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.children.cycles-pp.__x64_sys_exit_group
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.children.cycles-pp.do_group_exit
> 6.00 ?173% +20.6 26.61 ? 3% perf-profile.children.cycles-pp.do_exit
> 6.33 ?168% +21.3 27.62 ? 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 6.32 ?168% +21.3 27.62 ? 3% perf-profile.children.cycles-pp.do_syscall_64
> 15.70 ? 38% -12.1 3.56 ? 3% perf-profile.self.cycles-pp.poll_idle
> 8.17 ? 36% -7.3 0.92 ? 13% perf-profile.self.cycles-pp.intel_idle
> 1.06 ? 38% -0.7 0.38 ? 2% perf-profile.self.cycles-pp.unwind_next_frame
> 0.67 ? 38% -0.5 0.16 ? 9% perf-profile.self.cycles-pp.menu_select
> 0.60 ? 41% -0.3 0.30 ? 5% perf-profile.self.cycles-pp.__schedule
> 0.48 ? 38% -0.3 0.21 ? 7% perf-profile.self.cycles-pp.__orc_find
> 0.40 ? 40% -0.3 0.13 ? 9% perf-profile.self.cycles-pp.set_next_entity
> 0.46 ? 38% -0.3 0.20 ? 4% perf-profile.self.cycles-pp.__account_scheduler_latency
> 0.38 ? 38% -0.2 0.16 ? 8% perf-profile.self.cycles-pp.update_load_avg
> 0.36 ? 38% -0.2 0.17 ? 2% perf-profile.self.cycles-pp.try_to_wake_up
> 0.33 ? 37% -0.2 0.14 ? 7% perf-profile.self.cycles-pp.orc_find
> 0.33 ? 46% -0.2 0.15 ? 10% perf-profile.self.cycles-pp.update_curr
> 0.28 ? 39% -0.2 0.09 ? 8% perf-profile.self.cycles-pp.do_idle
> 0.29 ? 36% -0.2 0.12 ? 4% perf-profile.self.cycles-pp.__switch_to_asm
> 0.25 ? 42% -0.1 0.11 ? 6% perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
> 0.20 ? 37% -0.1 0.07 ? 15% perf-profile.self.cycles-pp.stack_trace_save_tsk
> 0.24 ? 36% -0.1 0.12 ? 7% perf-profile.self.cycles-pp.native_sched_clock
> 0.24 ? 39% -0.1 0.12 ? 7% perf-profile.self.cycles-pp.__switch_to
> 0.16 ? 42% -0.1 0.04 ? 60% perf-profile.self.cycles-pp.update_cfs_rq_h_load
> 0.22 ? 41% -0.1 0.11 ? 7% perf-profile.self.cycles-pp.__update_load_avg_se
> 0.20 ? 42% -0.1 0.09 ? 4% perf-profile.self.cycles-pp.update_rq_clock
> 0.17 ? 39% -0.1 0.07 perf-profile.self.cycles-pp.enqueue_task_fair
> 0.16 ? 37% -0.1 0.06 ? 6% perf-profile.self.cycles-pp.enqueue_entity
> 0.12 ? 43% -0.1 0.03 ?100% perf-profile.self.cycles-pp.__unwind_start
> 0.15 ? 39% -0.1 0.07 ? 6% perf-profile.self.cycles-pp.read_tsc
> 0.19 ? 32% -0.1 0.11 ? 7% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
> 0.11 ? 36% -0.1 0.03 ?100% perf-profile.self.cycles-pp.switch_mm_irqs_off
> 0.16 ? 38% -0.1 0.08 ? 8% perf-profile.self.cycles-pp.update_cfs_group
> 0.13 ? 40% -0.1 0.05 perf-profile.self.cycles-pp.stack_access_ok
> 0.10 ? 36% -0.1 0.03 ?100% perf-profile.self.cycles-pp.account_entity_dequeue
> 0.03 ?100% +0.0 0.07 ? 7% perf-profile.self.cycles-pp.native_apic_msr_eoi_write
> 0.03 ?100% +0.0 0.07 ? 10% perf-profile.self.cycles-pp.__unlock_page_memcg
> 0.04 ? 59% +0.0 0.09 ? 4% perf-profile.self.cycles-pp.__mod_node_page_state
> 0.00 +0.1 0.05 perf-profile.self.cycles-pp.__mod_zone_page_state
> 0.01 ?173% +0.1 0.07 ? 22% perf-profile.self.cycles-pp.__mod_lruvec_state
> 0.06 ? 63% +0.1 0.12 ? 7% perf-profile.self.cycles-pp.__mod_memcg_state
> 0.02 ?173% +0.1 0.08 ? 10% perf-profile.self.cycles-pp.vm_normal_page
> 0.01 ?173% +0.1 0.08 ? 6% perf-profile.self.cycles-pp.__tlb_remove_page_size
> 0.02 ?173% +0.1 0.10 ? 10% perf-profile.self.cycles-pp.free_pcppages_bulk
> 0.06 ? 71% +0.1 0.14 ? 9% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
> 0.03 ?173% +0.1 0.14 ? 11% perf-profile.self.cycles-pp.io_serial_in
> 0.03 ?173% +0.1 0.15 ? 16% perf-profile.self.cycles-pp.memcpy_erms
> 0.21 ? 42% +0.2 0.42 ? 5% perf-profile.self.cycles-pp.find_lock_entry
> 0.07 ?173% +0.2 0.30 ? 6% perf-profile.self.cycles-pp.pagevec_lru_move_fn
> 0.07 ? 64% +0.2 0.30 ? 2% perf-profile.self.cycles-pp.wake_up_page_bit
> 0.06 ?173% +0.2 0.30 ? 9% perf-profile.self.cycles-pp.mem_cgroup_page_lruvec
> 0.07 ?173% +0.3 0.36 ? 8% perf-profile.self.cycles-pp.free_pages_and_swap_cache
> 0.35 ? 43% +0.3 0.66 ? 3% perf-profile.self.cycles-pp.find_get_entry
> 0.33 ? 53% +0.3 0.68 ? 5% perf-profile.self.cycles-pp.lock_page_memcg
> 0.25 ? 76% +0.4 0.62 ? 3% perf-profile.self.cycles-pp.unlock_page_memcg
> 0.16 ?173% +0.5 0.67 ? 5% perf-profile.self.cycles-pp.__activate_page
> 0.75 ? 30% +0.5 1.28 ? 4% perf-profile.self.cycles-pp.PageHuge
> 0.49 ?150% +1.7 2.21 ? 4% perf-profile.self.cycles-pp.release_pages
> 0.57 ?173% +1.8 2.33 ? 3% perf-profile.self.cycles-pp.zap_pte_range
> 0.81 ?173% +2.5 3.32 ? 4% perf-profile.self.cycles-pp.mark_page_accessed
> 0.76 ?173% +3.0 3.72 ? 3% perf-profile.self.cycles-pp.workingset_age_nonresident
> 4.08 ? 52% +16.8 20.86 ? 3% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
> vm-scalability.time.system_time
>
> 1900 +--------------------------------------------------------------------+
> 1800 |-O O O O O O O O O O O O O O O O O O O O O O |
> | O O |
> 1700 |-+ |
> 1600 |-+ |
> | |
> 1500 |-+ |
> 1400 |-+ |
> 1300 |-+ |
> | |
> 1200 |-+ |
> 1100 |-+ |
> | |
> 1000 |.+..+.+..+.+.+..+.+.+..+.+..+.+.+..+.+.+..+.+..+.+.+..+.+.+..+.+..+.|
> 900 +--------------------------------------------------------------------+
>
>
> vm-scalability.time.percent_of_cpu_this_job_got
>
> 1460 +--------------------------------------------------------------------+
> 1440 |-O O O O O O O O O O O O O O O O O O O O O O O O |
> | |
> 1420 |-+ |
> 1400 |-+ |
> | |
> 1380 |-+ |
> 1360 |-+ |
> 1340 |-+ |
> | |
> 1320 |-+ |
> 1300 |-+ |
> | .+. .+.. .+.. .+. .+..+.+. .+.+..+.|
> 1280 |.+..+.+. +.+..+.+ +.+..+.+ +.+.+..+.+. + +. |
> 1260 +--------------------------------------------------------------------+
>
>
> vm-scalability.time.elapsed_time
>
> 215 +---------------------------------------------------------------------+
> | O O O O O O O O O O O O O O O O O O |
> 210 |-+ O O O O O O |
> 205 |-+ |
> | |
> 200 |-+ |
> 195 |-+ |
> | |
> 190 |-+ |
> 185 |-+ |
> | |
> 180 |-+ |
> 175 |-+ +. +.. + .+. |
> | .+.+.. + +..+. .+.+.. + .+..+. .. + .+.+.+. +..+.|
> 170 +---------------------------------------------------------------------+
>
>
> vm-scalability.time.elapsed_time.max
>
> 215 +---------------------------------------------------------------------+
> | O O O O O O O O O O O O O O O O O O |
> 210 |-+ O O O O O O |
> 205 |-+ |
> | |
> 200 |-+ |
> 195 |-+ |
> | |
> 190 |-+ |
> 185 |-+ |
> | |
> 180 |-+ |
> 175 |-+ +. +.. + .+. |
> | .+.+.. + +..+. .+.+.. + .+..+. .. + .+.+.+. +..+.|
> 170 +---------------------------------------------------------------------+
>
>
> vm-scalability.time.minor_page_faults
>
> 4.2e+08 +-----------------------------------------------------------------+
> | O O O O O O O O |
> 4e+08 |-O O O O O O O O O O O O O O O O |
> 3.8e+08 |-+ |
> | |
> 3.6e+08 |-+ |
> 3.4e+08 |-+ |
> | |
> 3.2e+08 |-+ |
> 3e+08 |-+ |
> | |
> 2.8e+08 |-+ |
> 2.6e+08 |-+ .+.. .+. .+. .+. |
> |.+..+.+.+ +.+.+.+..+.+.+..+.+ +..+.+.+..+ +.+..+.+ +..+.|
> 2.4e+08 +-----------------------------------------------------------------+
>
>
> vm-scalability.time.voluntary_context_switches
>
> 5.7e+07 +-----------------------------------------------------------------+
> | + + + .+. .+ |
> 5.6e+07 |-+ :+ +. :+ :+ .+ +. +|
> 5.5e+07 |.+ .+ : + + +. .+. .+ : +.. .+.. : +.+..+ |
> | +..+ + : + +. +.+. + : +.+ : |
> 5.4e+07 |-+ + + + |
> | |
> 5.3e+07 |-+ |
> | |
> 5.2e+07 |-+ |
> 5.1e+07 |-+ O |
> | O O O O O |
> 5e+07 |-+ O O O |
> | O O O O O O O O O O O |
> 4.9e+07 +-----------------------------------------------------------------+
>
>
> vm-scalability.time.involuntary_context_switches
>
> 45000 +-------------------------------------------------------------------+
> | O |
> 40000 |-+ O O |
> | O O O |
> 35000 |-+ O O O O O O O |
> | O O O O O O O O |
> 30000 |-+ O O O |
> | |
> 25000 |-+ |
> | |
> 20000 |-+ .+.. .+ .+ |
> |.+..+.+ .+. .+. .+.+.+. + .+.+..+ + .+. .+. |
> 15000 |-+ + +..+ +. +..+ +. +.+..+ +..+.|
> | |
> 10000 +-------------------------------------------------------------------+
>
>
> vm-scalability.throughput
>
> 1.65e+07 +----------------------------------------------------------------+
> |.+. .+. .+. |
> 1.6e+07 |-+ +..+.+. .+..+.+.+.+. +.+.+.. .+ +..+.+. .+..+.+.+. .+.+.|
> 1.55e+07 |-+ + + + +. |
> | |
> 1.5e+07 |-+ |
> 1.45e+07 |-+ |
> | |
> 1.4e+07 |-+ |
> 1.35e+07 |-+ |
> | |
> 1.3e+07 |-+ |
> 1.25e+07 |-O O O O O O O O O O O O O O O O O O O O O O |
> | O O |
> 1.2e+07 +----------------------------------------------------------------+
>
>
> vm-scalability.median
>
> 1.05e+06 +----------------------------------------------------------------+
> | |
> 1e+06 |.+. .+ .+.+.+..+. .+.. +.+. .+ +.. |
> | +..+ + .+..+ +.+ + +..+ + + +.+.+. .+.+.|
> | + + + +. |
> 950000 |-+ |
> | |
> 900000 |-+ |
> | |
> 850000 |-+ |
> | |
> | |
> 800000 |-O O O O O O O O O O O O O O O O O |
> | O O O O O O O |
> 750000 +----------------------------------------------------------------+
>
>
> vm-scalability.median_stddev_
>
> 6 +---------------------------------------------------------------------+
> | +.. +..+. +. +. +.. .+. .+ |
> 5.5 |:+ + + +..+ + + : + + + +. : +.|
> |: : +.+ + + : +.+.. : + +..+ : .. |
> 5 |-+ : .. + : .. + + + + |
> | + + + |
> 4.5 |-+ |
> | |
> 4 |-+ |
> | |
> 3.5 |-+ O |
> | O O O O O |
> 3 |-+ O O O O O O O O O O O O O O O |
> | O O O |
> 2.5 +---------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
> ***************************************************************************************************
> lkp-cfl-e1: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
> =========================================================================================
> compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
> gcc-9/performance/x86_64-rhel-8.3/30%/debian-10.4-x86_64-20200603.cgz/300s/lkp-cfl-e1/shell8/unixbench/0xd6
>
> commit:
> 5925fa68fe (" perf tools fixes for v5.9: 3rd batch")
> 5ef64cc898 ("mm: allow a controlled amount of unfairness in the page lock")
>
> 5925fa68fe824465 5ef64cc8987a9211d3f3667331b
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 21570 +5.4% 22733 unixbench.score
> 1054019 ? 4% +25.8% 1325439 ? 3% unixbench.time.involuntary_context_switches
> 299493 ? 2% -87.4% 37651 ? 3% unixbench.time.major_page_faults
> 1.034e+08 +5.6% 1.092e+08 unixbench.time.minor_page_faults
> 1286 +7.3% 1379 unixbench.time.percent_of_cpu_this_job_got
> 334.45 +11.0% 371.24 unixbench.time.system_time
> 477.46 +4.8% 500.52 unixbench.time.user_time
> 3401111 -7.4% 3148880 unixbench.time.voluntary_context_switches
> 815347 +5.4% 859335 unixbench.workload
> 5.09 ? 3% -5.1 0.00 ?100% mpstat.cpu.all.iowait%
> 1752 ? 16% -21.0% 1384 ? 13% slabinfo.skbuff_head_cache.active_objs
> 1752 ? 16% -19.2% 1416 ? 11% slabinfo.skbuff_head_cache.num_objs
> 10037 -16.6% 8374 ? 7% slabinfo.vmap_area.active_objs
> 10355 -11.8% 9132 ? 6% slabinfo.vmap_area.num_objs
> 48.00 +5.7% 50.75 vmstat.cpu.sy
> 31.25 +9.6% 34.25 vmstat.cpu.us
> 3.50 ? 24% -100.0% 0.00 vmstat.procs.b
> 18.75 ? 4% +17.3% 22.00 ? 3% vmstat.procs.r
> 111395 -9.5% 100851 vmstat.system.cs
> 36967 +3.2% 38149 vmstat.system.in
> 52268017 -42.8% 29897553 ? 3% cpuidle.C1.time
> 1117760 -25.5% 833099 cpuidle.C1.usage
> 50982561 ? 2% -61.2% 19764268 ? 6% cpuidle.C1E.time
> 769266 -43.4% 435332 ? 3% cpuidle.C1E.usage
> 74333 ? 3% -31.2% 51121 ? 16% cpuidle.C3.usage
> 5602742 ? 4% -97.7% 129968 ? 28% cpuidle.POLL.time
> 697760 ? 2% -98.4% 11332 ? 12% cpuidle.POLL.usage
> 16352 ? 3% -8.3% 14993 ? 2% softirqs.CPU1.SCHED
> 15561 -11.7% 13743 softirqs.CPU10.SCHED
> 15622 -10.2% 14029 softirqs.CPU13.SCHED
> 15418 -8.0% 14188 ? 2% softirqs.CPU14.SCHED
> 15598 -8.0% 14355 ? 2% softirqs.CPU15.SCHED
> 15749 -10.1% 14155 softirqs.CPU3.SCHED
> 15847 -10.0% 14262 softirqs.CPU4.SCHED
> 970.25 +2.3% 993.00 proc-vmstat.nr_page_table_pages
> 13460 +5.3% 14177 ? 3% proc-vmstat.nr_slab_unreclaimable
> 70057163 +4.5% 73199359 proc-vmstat.numa_hit
> 70057163 +4.5% 73199359 proc-vmstat.numa_local
> 100067 +21.2% 121296 proc-vmstat.pgactivate
> 74525416 +4.7% 78002806 proc-vmstat.pgalloc_normal
> 1.041e+08 +5.1% 1.094e+08 proc-vmstat.pgfault
> 74511080 +4.7% 77987815 proc-vmstat.pgfree
> 5126035 +8.5% 5561192 proc-vmstat.pgreuse
> 3425 +7.0% 3664 proc-vmstat.thp_fault_alloc
> 1450159 +5.4% 1528363 proc-vmstat.unevictable_pgs_culled
> 15.80 ?100% +200.5% 47.50 ? 32% sched_debug.cfs_rq:/.removed.load_avg.avg
> 61.21 ?100% +137.4% 145.31 ? 14% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 1.85 ?130% +649.4% 13.88 ? 30% sched_debug.cfs_rq:/.removed.runnable_avg.avg
> 29.62 ?130% +512.2% 181.38 ? 14% sched_debug.cfs_rq:/.removed.runnable_avg.max
> 7.17 ?130% +531.8% 45.30 ? 17% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
> 1.48 ?170% +815.3% 13.59 ? 29% sched_debug.cfs_rq:/.removed.util_avg.avg
> 23.75 ?170% +649.5% 178.00 ? 13% sched_debug.cfs_rq:/.removed.util_avg.max
> 5.75 ?170% +673.5% 44.47 ? 16% sched_debug.cfs_rq:/.removed.util_avg.stddev
> 124782 ? 12% +29.5% 161579 ? 9% sched_debug.cpu.avg_idle.avg
> 5321 ? 44% +152.1% 13417 ? 11% sched_debug.cpu.curr->pid.avg
> 7329 ? 36% +104.2% 14969 ? 8% sched_debug.cpu.curr->pid.max
> 0.94 ? 16% +33.3% 1.25 ? 4% sched_debug.cpu.nr_running.avg
> 213716 -9.1% 194283 sched_debug.cpu.nr_switches.min
> 219.88 ? 16% -69.1% 67.88 ? 25% sched_debug.cpu.nr_uninterruptible.max
> -212.88 -64.7% -75.25 sched_debug.cpu.nr_uninterruptible.min
> 116.88 ? 11% -68.1% 37.27 ? 11% sched_debug.cpu.nr_uninterruptible.stddev
> 215276 -9.2% 195388 sched_debug.cpu.sched_count.avg
> 219021 -9.4% 198334 sched_debug.cpu.sched_count.max
> 212018 -9.6% 191589 sched_debug.cpu.sched_count.min
> 63854 -33.3% 42613 ? 2% sched_debug.cpu.sched_goidle.avg
> 65292 -33.2% 43641 ? 2% sched_debug.cpu.sched_goidle.max
> 62644 -33.7% 41522 ? 2% sched_debug.cpu.sched_goidle.min
> 36980 ? 4% +15.4% 42667 ? 4% sched_debug.cpu.ttwu_local.avg
> 37905 ? 2% +14.8% 43522 ? 2% sched_debug.cpu.ttwu_local.max
> 36245 ? 5% +15.1% 41734 ? 5% sched_debug.cpu.ttwu_local.min
> 479.50 ?122% +208.1% 1477 ? 66% interrupts.132:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
> 21616 +25.2% 27062 interrupts.CPU0.RES:Rescheduling_interrupts
> 1431 -65.8% 489.00 ? 4% interrupts.CPU0.TLB:TLB_shootdowns
> 479.50 ?122% +208.1% 1477 ? 66% interrupts.CPU1.132:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
> 1687 ? 7% -12.9% 1469 ? 4% interrupts.CPU1.CAL:Function_call_interrupts
> 21784 +23.9% 26998 ? 2% interrupts.CPU1.RES:Rescheduling_interrupts
> 1399 ? 2% -64.9% 491.75 ? 7% interrupts.CPU1.TLB:TLB_shootdowns
> 21969 +22.0% 26811 interrupts.CPU10.RES:Rescheduling_interrupts
> 1388 ? 2% -64.0% 500.50 interrupts.CPU10.TLB:TLB_shootdowns
> 21807 +24.8% 27212 interrupts.CPU11.RES:Rescheduling_interrupts
> 1367 ? 2% -64.4% 487.25 ? 5% interrupts.CPU11.TLB:TLB_shootdowns
> 21738 ? 2% +26.4% 27485 interrupts.CPU12.RES:Rescheduling_interrupts
> 1376 -62.3% 518.50 ? 9% interrupts.CPU12.TLB:TLB_shootdowns
> 21766 +25.0% 27217 interrupts.CPU13.RES:Rescheduling_interrupts
> 1425 -65.6% 490.75 ? 9% interrupts.CPU13.TLB:TLB_shootdowns
> 1557 ? 4% -9.7% 1406 ? 6% interrupts.CPU14.CAL:Function_call_interrupts
> 21777 +26.0% 27447 interrupts.CPU14.RES:Rescheduling_interrupts
> 1426 ? 2% -64.9% 500.25 ? 7% interrupts.CPU14.TLB:TLB_shootdowns
> 21639 ? 2% +27.8% 27653 interrupts.CPU15.RES:Rescheduling_interrupts
> 1441 ? 3% -64.6% 510.75 ? 6% interrupts.CPU15.TLB:TLB_shootdowns
> 21908 +24.0% 27161 interrupts.CPU2.RES:Rescheduling_interrupts
> 1415 -64.5% 503.00 ? 2% interrupts.CPU2.TLB:TLB_shootdowns
> 21904 +22.4% 26804 interrupts.CPU3.RES:Rescheduling_interrupts
> 1474 -65.6% 507.25 ? 8% interrupts.CPU3.TLB:TLB_shootdowns
> 21534 ? 2% +26.2% 27181 interrupts.CPU4.RES:Rescheduling_interrupts
> 1356 -62.2% 513.00 ? 6% interrupts.CPU4.TLB:TLB_shootdowns
> 21836 +23.9% 27045 interrupts.CPU5.RES:Rescheduling_interrupts
> 1407 ? 2% -63.9% 508.50 ? 2% interrupts.CPU5.TLB:TLB_shootdowns
> 21999 +23.3% 27125 interrupts.CPU6.RES:Rescheduling_interrupts
> 1403 ? 4% -62.7% 524.00 ? 9% interrupts.CPU6.TLB:TLB_shootdowns
> 1557 ? 4% -11.0% 1385 ? 4% interrupts.CPU7.CAL:Function_call_interrupts
> 21508 ? 2% +26.1% 27111 interrupts.CPU7.RES:Rescheduling_interrupts
> 1417 ? 5% -64.1% 509.50 ? 6% interrupts.CPU7.TLB:TLB_shootdowns
> 21905 +25.2% 27429 interrupts.CPU8.RES:Rescheduling_interrupts
> 1411 ? 2% -65.4% 488.50 ? 8% interrupts.CPU8.TLB:TLB_shootdowns
> 21904 +24.7% 27307 interrupts.CPU9.RES:Rescheduling_interrupts
> 1388 ? 2% -63.7% 503.50 interrupts.CPU9.TLB:TLB_shootdowns
> 348600 +24.8% 435055 interrupts.RES:Rescheduling_interrupts
> 22530 -64.3% 8046 ? 4% interrupts.TLB:TLB_shootdowns
> 48.16 -1.7% 47.32 perf-stat.i.MPKI
> 9.687e+09 +5.1% 1.019e+10 perf-stat.i.branch-instructions
> 2.126e+08 +4.6% 2.224e+08 perf-stat.i.branch-misses
> 4.44 +0.2 4.60 perf-stat.i.cache-miss-rate%
> 94624278 +7.1% 1.013e+08 perf-stat.i.cache-misses
> 2.382e+09 +3.3% 2.461e+09 perf-stat.i.cache-references
> 114789 -9.1% 104363 perf-stat.i.context-switches
> 5.294e+10 +5.8% 5.601e+10 perf-stat.i.cpu-cycles
> 19853 -22.2% 15448 perf-stat.i.cpu-migrations
> 569.61 -1.9% 558.71 perf-stat.i.cycles-between-cache-misses
> 1.223e+10 +5.1% 1.285e+10 perf-stat.i.dTLB-loads
> 3655243 +4.1% 3805877 perf-stat.i.dTLB-store-misses
> 7.024e+09 +5.1% 7.383e+09 perf-stat.i.dTLB-stores
> 10344903 +3.7% 10732085 perf-stat.i.iTLB-load-misses
> 8031239 +1.9% 8185952 perf-stat.i.iTLB-loads
> 4.759e+10 +5.2% 5.006e+10 perf-stat.i.instructions
> 5236 +1.2% 5299 perf-stat.i.instructions-per-iTLB-miss
> 4684 ? 2% -87.4% 589.00 ? 3% perf-stat.i.major-faults
> 3.31 +5.8% 3.50 perf-stat.i.metric.GHz
> 1960 +5.0% 2058 perf-stat.i.metric.M/sec
> 1588100 +5.6% 1677762 perf-stat.i.minor-faults
> 5071078 +8.5% 5500804 perf-stat.i.node-loads
> 31239619 +7.2% 33482613 perf-stat.i.node-stores
> 1592784 +5.4% 1678351 perf-stat.i.page-faults
> 50.05 -1.8% 49.15 perf-stat.overall.MPKI
> 3.97 +0.1 4.12 perf-stat.overall.cache-miss-rate%
> 559.44 -1.2% 552.70 perf-stat.overall.cycles-between-cache-misses
> 4600 +1.4% 4664 perf-stat.overall.instructions-per-iTLB-miss
> 9.533e+09 +5.1% 1.002e+10 perf-stat.ps.branch-instructions
> 2.093e+08 +4.6% 2.189e+08 perf-stat.ps.branch-misses
> 93122368 +7.1% 99714732 perf-stat.ps.cache-misses
> 2.344e+09 +3.3% 2.421e+09 perf-stat.ps.cache-references
> 112962 -9.1% 102685 perf-stat.ps.context-switches
> 5.21e+10 +5.8% 5.511e+10 perf-stat.ps.cpu-cycles
> 19537 -22.2% 15200 perf-stat.ps.cpu-migrations
> 1.203e+10 +5.1% 1.264e+10 perf-stat.ps.dTLB-loads
> 3597096 +4.1% 3744712 perf-stat.ps.dTLB-store-misses
> 6.913e+09 +5.1% 7.264e+09 perf-stat.ps.dTLB-stores
> 10180397 +3.7% 10559643 perf-stat.ps.iTLB-load-misses
> 7903463 +1.9% 8054357 perf-stat.ps.iTLB-loads
> 4.684e+10 +5.2% 4.926e+10 perf-stat.ps.instructions
> 4609 ? 2% -87.4% 579.53 ? 3% perf-stat.ps.major-faults
> 1562827 +5.6% 1650780 perf-stat.ps.minor-faults
> 4990492 +8.5% 5412486 perf-stat.ps.node-loads
> 30742642 +7.2% 32944203 perf-stat.ps.node-stores
> 1567436 +5.4% 1651360 perf-stat.ps.page-faults
> 3.004e+12 +5.2% 3.159e+12 perf-stat.total.instructions
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.calltrace.cycles-pp.write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.calltrace.cycles-pp.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write.do_syscall_64
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.calltrace.cycles-pp.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write.ksys_write
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.calltrace.cycles-pp.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write.vfs_write
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.cold.new_sync_write
> 25.68 ? 57% -14.9 10.79 ?173% perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit.devkmsg_write.cold
> 24.54 ? 57% -14.0 10.51 ?173% perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.devkmsg_emit
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal.exit_to_user_mode_prepare.syscall_exit_to_user_mode
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal.exit_to_user_mode_prepare
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.calltrace.cycles-pp.arch_do_signal.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
> 20.19 ? 58% -11.1 9.09 ?173% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock
> 20.19 ? 58% -11.1 9.09 ?173% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
> 13.48 ? 48% -10.9 2.58 ?109% perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
> 13.48 ? 48% -10.9 2.58 ?109% perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
> 16.11 ? 40% -10.4 5.73 ?114% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 13.48 ? 48% -9.9 3.59 ?106% perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal
> 13.48 ? 48% -9.9 3.59 ?106% perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
> 18.44 ? 58% -9.6 8.81 ?173% perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
> 4.35 ? 76% -3.5 0.85 ?173% perf-profile.calltrace.cycles-pp.io_serial_out.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
> 47.23 ? 27% -26.2 21.02 ? 78% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.children.cycles-pp.write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.children.cycles-pp.ksys_write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.children.cycles-pp.vfs_write
> 29.27 ? 57% -17.3 11.93 ?173% perf-profile.children.cycles-pp.new_sync_write
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.children.cycles-pp.devkmsg_write.cold
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.children.cycles-pp.devkmsg_emit
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.children.cycles-pp.vprintk_emit
> 29.09 ? 57% -17.2 11.93 ?173% perf-profile.children.cycles-pp.console_unlock
> 25.68 ? 57% -14.9 10.79 ?173% perf-profile.children.cycles-pp.serial8250_console_write
> 24.54 ? 57% -14.3 10.23 ?173% perf-profile.children.cycles-pp.uart_console_write
> 15.19 ? 44% -11.6 3.59 ?106% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
> 15.19 ? 44% -11.6 3.59 ?106% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
> 15.74 ? 41% -11.6 4.15 ?112% perf-profile.children.cycles-pp.do_group_exit
> 15.74 ? 41% -11.6 4.15 ?112% perf-profile.children.cycles-pp.do_exit
> 21.15 ? 58% -11.5 9.66 ?173% perf-profile.children.cycles-pp.wait_for_xmitr
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.children.cycles-pp.arch_do_signal
> 15.00 ? 46% -11.4 3.59 ?106% perf-profile.children.cycles-pp.get_signal
> 20.19 ? 58% -11.1 9.09 ?173% perf-profile.children.cycles-pp.serial8250_console_putchar
> 13.48 ? 48% -10.9 2.58 ?109% perf-profile.children.cycles-pp.perf_release
> 13.48 ? 48% -10.9 2.58 ?109% perf-profile.children.cycles-pp.perf_event_release_kernel
> 19.40 ? 58% -10.0 9.38 ?173% perf-profile.children.cycles-pp.io_serial_in
> 13.48 ? 48% -9.9 3.59 ?106% perf-profile.children.cycles-pp.task_work_run
> 13.48 ? 48% -9.9 3.59 ?106% perf-profile.children.cycles-pp.__fput
> 4.54 ? 79% -3.4 1.14 ?173% perf-profile.children.cycles-pp.io_serial_out
> 19.40 ? 58% -10.0 9.38 ?173% perf-profile.self.cycles-pp.io_serial_in
> 4.54 ? 79% -3.4 1.14 ?173% perf-profile.self.cycles-pp.io_serial_out
>
>
>
> ***************************************************************************************************
> lkp-csl-2ap3: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
> =========================================================================================
> compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
> gcc-9/performance/x86_64-rhel-8.3/thread/50%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap3/readseek2/will-it-scale/0x5002f01
>
> commit:
> 5925fa68fe (" perf tools fixes for v5.9: 3rd batch")
> 5ef64cc898 ("mm: allow a controlled amount of unfairness in the page lock")
>
> 5925fa68fe824465 5ef64cc8987a9211d3f3667331b
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 1266 +688.3% 9979 ? 3% will-it-scale.per_thread_ops
> 121587 +688.0% 958052 ? 3% will-it-scale.workload
> 71.50 ? 5% -28.3% 51.25 vmstat.cpu.id
> 25.00 ? 18% -100.0% 0.00 vmstat.cpu.wa
> 1.00 +8975.0% 90.75 vmstat.procs.r
> 1.225e+09 ? 41% -63.7% 4.441e+08 ? 69% cpuidle.C1.time
> 1.272e+08 ? 18% -43.3% 72207404 ? 7% cpuidle.C1E.usage
> 26464356 ? 10% +69.2% 44765776 ? 10% cpuidle.POLL.time
> 2008351 ? 16% +641.3% 14887271 ? 10% cpuidle.POLL.usage
> 7829 ? 6% +1248.0% 105533 ? 5% meminfo.Active
> 7825 ? 6% +1248.6% 105529 ? 5% meminfo.Active(anon)
> 36955 +11.8% 41313 meminfo.Mapped
> 27224 +368.3% 127479 ? 4% meminfo.Shmem
> 31081 -59.8% 12502 ?104% numa-numastat.node1.other_node
> 220546 ? 7% -28.7% 157349 ? 3% numa-numastat.node2.local_node
> 251604 ? 6% -28.2% 180707 ? 7% numa-numastat.node2.numa_hit
> 7880 ?170% +294.3% 31072 numa-numastat.node3.other_node
> 71.78 ? 6% -20.4 51.40 mpstat.cpu.all.idle%
> 25.99 ? 16% -26.0 0.03 ?105% mpstat.cpu.all.iowait%
> 1.53 ? 3% -1.0 0.52 ? 2% mpstat.cpu.all.irq%
> 0.11 ? 2% -0.0 0.07 ? 3% mpstat.cpu.all.soft%
> 0.53 ? 2% +47.4 47.91 mpstat.cpu.all.sys%
> 0.05 +0.0 0.09 ? 22% mpstat.cpu.all.usr%
> 1988 ? 18% +31.3% 2610 ? 20% numa-vmstat.node0.nr_mapped
> 77149 ? 37% -63.9% 27823 ? 25% numa-vmstat.node0.numa_other
> 179.00 ?130% +232.8% 595.75 ? 56% numa-vmstat.node1.nr_active_anon
> 179.00 ?130% +232.8% 595.75 ? 56% numa-vmstat.node1.nr_zone_active_anon
> 1260 ? 23% +1900.9% 25226 ? 5% numa-vmstat.node3.nr_active_anon
> 71809 +34.9% 96880 ? 3% numa-vmstat.node3.nr_file_pages
> 1845 ? 34% +1367.1% 27068 ? 11% numa-vmstat.node3.nr_shmem
> 9936 ? 9% -42.1% 5758 ? 4% numa-vmstat.node3.nr_slab_reclaimable
> 1260 ? 23% +1900.8% 25225 ? 5% numa-vmstat.node3.nr_zone_active_anon
> 65717 ? 43% +87.3% 123087 numa-vmstat.node3.numa_other
> 1181 ? 13% +28.6% 1520 ? 5% slabinfo.dmaengine-unmap-16.active_objs
> 1181 ? 13% +28.6% 1520 ? 5% slabinfo.dmaengine-unmap-16.num_objs
> 6785 ? 9% +24.7% 8464 ? 2% slabinfo.files_cache.active_objs
> 6785 ? 9% +24.7% 8464 ? 2% slabinfo.files_cache.num_objs
> 3748 ? 5% +13.8% 4267 ? 2% slabinfo.sighand_cache.active_objs
> 3793 ? 5% +13.4% 4300 ? 2% slabinfo.sighand_cache.num_objs
> 829.50 ? 14% +23.8% 1027 ? 5% slabinfo.skbuff_fclone_cache.active_objs
> 829.50 ? 14% +23.8% 1027 ? 5% slabinfo.skbuff_fclone_cache.num_objs
> 25690 -17.0% 21326 slabinfo.vmap_area.active_objs
> 25696 -17.0% 21331 slabinfo.vmap_area.num_objs
> 7835 ? 19% +30.5% 10227 ? 22% numa-meminfo.node0.Mapped
> 718.00 ?129% +232.6% 2387 ? 56% numa-meminfo.node1.Active
> 718.00 ?129% +232.1% 2384 ? 56% numa-meminfo.node1.Active(anon)
> 5038 ? 23% +1904.1% 100980 ? 5% numa-meminfo.node3.Active
> 5035 ? 23% +1905.3% 100980 ? 5% numa-meminfo.node3.Active(anon)
> 287239 +34.9% 387535 ? 3% numa-meminfo.node3.FilePages
> 39748 ? 9% -42.0% 23046 ? 4% numa-meminfo.node3.KReclaimable
> 789152 ? 4% +7.8% 850631 ? 6% numa-meminfo.node3.MemUsed
> 39748 ? 9% -42.0% 23046 ? 4% numa-meminfo.node3.SReclaimable
> 7378 ? 34% +1367.6% 108288 ? 11% numa-meminfo.node3.Shmem
> 112975 ? 7% -22.4% 87674 ? 10% numa-meminfo.node3.Slab
> 1955 ? 6% +1249.2% 26376 ? 5% proc-vmstat.nr_active_anon
> 292548 +8.6% 317683 proc-vmstat.nr_file_pages
> 9365 +12.1% 10501 proc-vmstat.nr_mapped
> 6805 +368.3% 31870 ? 4% proc-vmstat.nr_shmem
> 31824 +1.8% 32389 proc-vmstat.nr_slab_reclaimable
> 1955 ? 6% +1249.2% 26376 ? 5% proc-vmstat.nr_zone_active_anon
> 2404 ? 12% +107.0% 4976 ? 49% proc-vmstat.numa_hint_faults_local
> 996995 +3.3% 1030379 proc-vmstat.numa_hit
> 903561 +3.7% 936976 proc-vmstat.numa_local
> 92254 ? 13% +44.3% 133088 ? 10% proc-vmstat.numa_pte_updates
> 1519 ? 24% +1609.0% 25963 ? 6% proc-vmstat.pgactivate
> 1084057 +3.1% 1117729 proc-vmstat.pgalloc_normal
> 1156577 +4.0% 1202453 proc-vmstat.pgfault
> 170.02 ? 57% +8564.4% 14731 ? 41% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 16736 ? 45% +11393.3% 1923555 ? 36% sched_debug.cfs_rq:/.MIN_vruntime.max
> 1596 ? 49% +10399.2% 167614 ? 39% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 1242 ? 10% +5367.8% 67950 ? 8% sched_debug.cfs_rq:/.exec_clock.avg
> 3496 ? 16% +3299.1% 118860 ? 6% sched_debug.cfs_rq:/.exec_clock.max
> 307.99 ? 22% +5682.7% 17809 ? 31% sched_debug.cfs_rq:/.exec_clock.min
> 443.83 ? 8% +2973.4% 13640 ? 4% sched_debug.cfs_rq:/.exec_clock.stddev
> 466502 ? 18% -85.2% 69202 ?129% sched_debug.cfs_rq:/.load.max
> 56674 ? 17% -82.3% 10016 ? 67% sched_debug.cfs_rq:/.load.stddev
> 553.96 ? 7% -54.4% 252.48 ? 39% sched_debug.cfs_rq:/.load_avg.max
> 53.50 ? 10% -43.5% 30.25 ? 38% sched_debug.cfs_rq:/.load_avg.stddev
> 170.02 ? 57% +8564.4% 14731 ? 41% sched_debug.cfs_rq:/.max_vruntime.avg
> 16736 ? 45% +11393.3% 1923555 ? 36% sched_debug.cfs_rq:/.max_vruntime.max
> 1596 ? 49% +10399.2% 167614 ? 39% sched_debug.cfs_rq:/.max_vruntime.stddev
> 33466 ? 25% +17200.5% 5789901 ? 8% sched_debug.cfs_rq:/.min_vruntime.avg
> 60233 ? 19% +16685.1% 10110254 ? 6% sched_debug.cfs_rq:/.min_vruntime.max
> 20331 ? 26% +7420.2% 1528965 ? 32% sched_debug.cfs_rq:/.min_vruntime.min
> 4797 ? 11% +24047.1% 1158377 ? 4% sched_debug.cfs_rq:/.min_vruntime.stddev
> 0.03 ? 15% +1586.0% 0.53 ? 6% sched_debug.cfs_rq:/.nr_running.avg
> 0.17 ? 8% +155.9% 0.43 ? 2% sched_debug.cfs_rq:/.nr_running.stddev
> 0.13 ?103% +263.7% 0.49 ? 18% sched_debug.cfs_rq:/.nr_spread_over.avg
> 1.13 ? 63% +237.5% 3.82 ? 15% sched_debug.cfs_rq:/.nr_spread_over.max
> 0.17 ? 89% +337.2% 0.76 ? 11% sched_debug.cfs_rq:/.nr_spread_over.stddev
> 31.86 ? 14% +1420.5% 484.49 ? 7% sched_debug.cfs_rq:/.runnable_avg.avg
> 702.53 ? 4% +33.6% 938.62 ? 5% sched_debug.cfs_rq:/.runnable_avg.max
> 86.88 ? 7% +323.3% 367.76 ? 2% sched_debug.cfs_rq:/.runnable_avg.stddev
> -1332 -1.7e+05% 2293744 ? 23% sched_debug.cfs_rq:/.spread0.avg
> 25416 ? 45% +25922.7% 6614096 ? 10% sched_debug.cfs_rq:/.spread0.max
> -14452 +13512.9% -1967432 sched_debug.cfs_rq:/.spread0.min
> 4799 ? 11% +24038.1% 1158452 ? 4% sched_debug.cfs_rq:/.spread0.stddev
> 31.73 ? 14% +1425.9% 484.20 ? 7% sched_debug.cfs_rq:/.util_avg.avg
> 702.49 ? 4% +33.6% 938.58 ? 5% sched_debug.cfs_rq:/.util_avg.max
> 86.82 ? 7% +323.6% 367.72 ? 2% sched_debug.cfs_rq:/.util_avg.stddev
> 3.25 ? 12% +13255.4% 434.48 ? 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 259.20 ? 5% +242.9% 888.80 sched_debug.cfs_rq:/.util_est_enqueued.max
> 24.35 ? 6% +1360.2% 355.60 sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 853642 -37.8% 530714 sched_debug.cpu.avg_idle.avg
> 75902 ? 12% -80.9% 14497 ? 29% sched_debug.cpu.avg_idle.min
> 177509 ? 4% +147.6% 439550 sched_debug.cpu.avg_idle.stddev
> 96.07 ? 9% +1468.1% 1506 ? 2% sched_debug.cpu.curr->pid.avg
> 720.78 ? 4% +132.6% 1676 sched_debug.cpu.curr->pid.stddev
> 0.00 ? 20% +322.0% 0.00 ? 27% sched_debug.cpu.next_balance.stddev
> 0.02 ? 8% +1657.9% 0.41 sched_debug.cpu.nr_running.avg
> 0.15 ? 4% +207.4% 0.45 sched_debug.cpu.nr_running.stddev
> 49550 ? 6% -31.8% 33787 ? 10% sched_debug.cpu.nr_switches.stddev
> 0.40 -94.3% 0.02 ? 42% sched_debug.cpu.nr_uninterruptible.avg
> 49442 ? 6% -31.9% 33686 ? 10% sched_debug.cpu.sched_count.stddev
> 24716 ? 6% -31.9% 16830 ? 10% sched_debug.cpu.sched_goidle.stddev
> 24764 ? 6% -32.3% 16770 ? 10% sched_debug.cpu.ttwu_count.stddev
> 4432 ? 40% -60.8% 1735 ? 26% sched_debug.cpu.ttwu_local.max
> 337.31 ? 35% -54.2% 154.44 ? 23% sched_debug.cpu.ttwu_local.stddev
> 17.43 -85.9% 2.46 ? 7% perf-stat.i.MPKI
> 7.452e+08 +805.2% 6.745e+09 perf-stat.i.branch-instructions
> 1.69 -1.3 0.34 ? 3% perf-stat.i.branch-miss-rate%
> 13281209 +71.2% 22738819 ? 2% perf-stat.i.branch-misses
> 13.68 ? 6% +40.3 54.02 ? 2% perf-stat.i.cache-miss-rate%
> 8061015 ? 8% +357.2% 36853851 ? 9% perf-stat.i.cache-misses
> 59267552 ? 2% +15.1% 68218277 ? 7% perf-stat.i.cache-references
> 3.22 ? 2% +221.8% 10.36 perf-stat.i.cpi
> 1.103e+10 ? 2% +2512.2% 2.882e+11 perf-stat.i.cpu-cycles
> 223.55 +11.5% 249.23 perf-stat.i.cpu-migrations
> 1409 ? 5% +459.4% 7886 ? 10% perf-stat.i.cycles-between-cache-misses
> 0.00 ? 3% -0.0 0.00 ? 4% perf-stat.i.dTLB-load-miss-rate%
> 42216 ? 6% +374.4% 200286 ? 4% perf-stat.i.dTLB-load-misses
> 9.242e+08 +667.5% 7.093e+09 perf-stat.i.dTLB-loads
> 0.00 -0.0 0.00 ? 3% perf-stat.i.dTLB-store-miss-rate%
> 13269 ? 2% +23.8% 16421 perf-stat.i.dTLB-store-misses
> 5.132e+08 +64.4% 8.439e+08 ? 2% perf-stat.i.dTLB-stores
> 57.39 +14.0 71.43 perf-stat.i.iTLB-load-miss-rate%
> 7475200 ? 2% +51.8% 11347314 ? 4% perf-stat.i.iTLB-load-misses
> 5545950 -18.1% 4541031 ? 7% perf-stat.i.iTLB-loads
> 3.499e+09 +694.7% 2.781e+10 perf-stat.i.instructions
> 470.84 +421.6% 2455 ? 4% perf-stat.i.instructions-per-iTLB-miss
> 0.31 ? 2% -68.9% 0.10 perf-stat.i.ipc
> 0.69 ? 3% +55.3% 1.07 ? 5% perf-stat.i.major-faults
> 0.06 ? 2% +2510.5% 1.50 perf-stat.i.metric.GHz
> 0.44 ? 14% +128.0% 1.01 ? 2% perf-stat.i.metric.K/sec
> 11.72 +556.3% 76.89 perf-stat.i.metric.M/sec
> 3716 +4.5% 3884 perf-stat.i.minor-faults
> 96.24 +2.0 98.23 perf-stat.i.node-load-miss-rate%
> 1506224 ? 8% +563.7% 9996147 perf-stat.i.node-load-misses
> 59182 ? 20% +202.2% 178865 ? 2% perf-stat.i.node-loads
> 97.87 +1.9 99.75 perf-stat.i.node-store-miss-rate%
> 406316 ? 6% +1532.3% 6632290 perf-stat.i.node-store-misses
> 9574 ? 16% +26.7% 12131 ? 8% perf-stat.i.node-stores
> 3717 +4.5% 3885 perf-stat.i.page-faults
> 16.94 -85.5% 2.45 ? 7% perf-stat.overall.MPKI
> 1.78 -1.4 0.34 ? 3% perf-stat.overall.branch-miss-rate%
> 13.59 ? 6% +40.3 53.94 ? 2% perf-stat.overall.cache-miss-rate%
> 3.15 ? 2% +228.8% 10.37 perf-stat.overall.cpi
> 1374 ? 5% +474.4% 7895 ? 10% perf-stat.overall.cycles-between-cache-misses
> 0.00 ? 6% -0.0 0.00 ? 4% perf-stat.overall.dTLB-load-miss-rate%
> 0.00 -0.0 0.00 ? 3% perf-stat.overall.dTLB-store-miss-rate%
> 57.40 +14.1 71.46 perf-stat.overall.iTLB-load-miss-rate%
> 468.36 ? 2% +424.0% 2454 ? 4% perf-stat.overall.instructions-per-iTLB-miss
> 0.32 ? 2% -69.6% 0.10 perf-stat.overall.ipc
> 96.24 +2.0 98.24 perf-stat.overall.node-load-miss-rate%
> 97.70 +2.1 99.82 perf-stat.overall.node-store-miss-rate%
> 7.428e+08 +804.5% 6.719e+09 perf-stat.ps.branch-instructions
> 13241794 +70.8% 22612139 ? 2% perf-stat.ps.branch-misses
> 8035056 ? 8% +356.9% 36710582 ? 9% perf-stat.ps.cache-misses
> 59071874 ? 2% +15.0% 67947551 ? 7% perf-stat.ps.cache-references
> 1.1e+10 ? 2% +2510.8% 2.871e+11 perf-stat.ps.cpu-cycles
> 222.96 +11.4% 248.34 perf-stat.ps.cpu-migrations
> 42122 ? 6% +373.7% 199521 ? 4% perf-stat.ps.dTLB-load-misses
> 9.212e+08 +667.0% 7.065e+09 perf-stat.ps.dTLB-loads
> 13235 ? 2% +23.4% 16333 perf-stat.ps.dTLB-store-misses
> 5.116e+08 +64.3% 8.404e+08 ? 2% perf-stat.ps.dTLB-stores
> 7450273 ? 2% +51.7% 11304013 ? 4% perf-stat.ps.iTLB-load-misses
> 5527433 -18.2% 4522978 ? 7% perf-stat.ps.iTLB-loads
> 3.488e+09 +694.1% 2.77e+10 perf-stat.ps.instructions
> 0.69 ? 3% +53.2% 1.05 ? 5% perf-stat.ps.major-faults
> 3705 +4.3% 3862 perf-stat.ps.minor-faults
> 1501438 ? 8% +563.2% 9958261 perf-stat.ps.node-load-misses
> 59037 ? 20% +202.0% 178306 ? 2% perf-stat.ps.node-loads
> 405024 ? 6% +1531.3% 6607029 perf-stat.ps.node-store-misses
> 9563 ? 16% +26.0% 12047 ? 8% perf-stat.ps.node-stores
> 3705 +4.3% 3863 perf-stat.ps.page-faults
> 1.053e+12 +693.1% 8.355e+12 perf-stat.total.instructions
> 89.08 -56.7 32.42 ? 20% perf-profile.calltrace.cycles-pp.secondary_startup_64
> 88.49 -56.3 32.22 ? 20% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 88.48 -56.3 32.22 ? 20% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 88.43 -56.2 32.22 ? 20% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 71.64 -39.7 31.95 ? 20% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 67.10 -35.2 31.95 ? 20% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> 25.60 ? 4% -25.6 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 14.64 ? 7% -14.6 0.00 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
> 45.20 -13.4 31.77 ? 20% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 12.89 ? 2% -12.9 0.00 perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 9.12 ? 6% -9.1 0.00 perf-profile.calltrace.cycles-pp.asm_call_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
> 9.08 ? 6% -9.1 0.00 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
> 8.88 ? 6% -8.9 0.00 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 2.17 +25.8 27.93 ? 10% perf-profile.calltrace.cycles-pp.__lock_page.find_lock_entry.shmem_getpage_gfp.shmem_file_read_iter.new_sync_read
> 2.46 +26.5 28.95 ? 10% perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_read
> 2.42 +26.5 28.91 ? 10% perf-profile.calltrace.cycles-pp.find_lock_entry.shmem_getpage_gfp.shmem_file_read_iter.new_sync_read.vfs_read
> 0.00 +27.4 27.39 ? 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__lock_page.find_lock_entry.shmem_getpage_gfp
> 0.00 +27.5 27.48 ? 10% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__lock_page.find_lock_entry.shmem_getpage_gfp.shmem_file_read_iter
> 4.69 ? 4% +32.8 37.48 ? 9% perf-profile.calltrace.cycles-pp.wake_up_page_bit.shmem_file_read_iter.new_sync_read.vfs_read.ksys_read
> 0.00 +36.9 36.91 ? 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.wake_up_page_bit.shmem_file_read_iter.new_sync_read
> 0.00 +37.0 36.98 ? 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.wake_up_page_bit.shmem_file_read_iter.new_sync_read.vfs_read
> 9.25 ? 2% +58.2 67.42 ? 9% perf-profile.calltrace.cycles-pp.__libc_read
> 8.75 ? 2% +58.6 67.37 ? 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
> 8.61 ? 2% +58.8 67.36 ? 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
> 8.53 ? 2% +58.8 67.35 ? 9% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
> 8.33 ? 2% +59.0 67.30 ? 9% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
> 8.04 ? 2% +59.2 67.27 ? 9% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 7.99 ? 2% +59.3 67.26 ? 9% perf-profile.calltrace.cycles-pp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
> 89.08 -56.7 32.42 ? 20% perf-profile.children.cycles-pp.secondary_startup_64
> 89.08 -56.7 32.42 ? 20% perf-profile.children.cycles-pp.cpu_startup_entry
> 89.06 -56.6 32.42 ? 20% perf-profile.children.cycles-pp.do_idle
> 88.49 -56.3 32.22 ? 20% perf-profile.children.cycles-pp.start_secondary
> 72.17 -40.0 32.15 ? 20% perf-profile.children.cycles-pp.cpuidle_enter
> 72.13 -40.0 32.15 ? 20% perf-profile.children.cycles-pp.cpuidle_enter_state
> 20.61 ? 2% -20.3 0.29 ? 17% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
> 14.94 ? 7% -14.7 0.23 ? 19% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
> 45.50 -13.5 31.96 ? 20% perf-profile.children.cycles-pp.intel_idle
> 12.98 ? 2% -12.9 0.04 ? 59% perf-profile.children.cycles-pp.menu_select
> 11.71 ? 6% -11.5 0.22 ? 17% perf-profile.children.cycles-pp.asm_call_on_stack
> 9.28 ? 6% -9.1 0.16 ? 15% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
> 9.08 ? 6% -8.9 0.16 ? 15% perf-profile.children.cycles-pp.hrtimer_interrupt
> 4.91 ? 4% -4.8 0.11 ? 11% perf-profile.children.cycles-pp.__hrtimer_run_queues
> 4.24 ? 4% -4.1 0.18 ? 6% perf-profile.children.cycles-pp.__wake_up_common
> 4.10 ? 4% -4.0 0.15 ? 5% perf-profile.children.cycles-pp.wake_page_function
> 3.90 ? 4% -3.8 0.11 ? 4% perf-profile.children.cycles-pp.try_to_wake_up
> 3.39 ? 3% -3.3 0.13 ? 6% perf-profile.children.cycles-pp.ttwu_do_activate
> 3.36 ? 3% -3.2 0.13 ? 6% perf-profile.children.cycles-pp.enqueue_task_fair
> 3.19 ? 3% -3.1 0.13 ? 5% perf-profile.children.cycles-pp.enqueue_entity
> 3.09 ? 6% -3.0 0.08 ? 19% perf-profile.children.cycles-pp.tick_sched_timer
> 2.83 ? 2% -2.7 0.14 ? 5% perf-profile.children.cycles-pp.__schedule
> 2.67 ? 5% -2.6 0.03 ?105% perf-profile.children.cycles-pp.irq_exit_rcu
> 2.42 ? 6% -2.3 0.07 ? 15% perf-profile.children.cycles-pp.tick_sched_handle
> 2.40 ? 6% -2.3 0.07 ? 15% perf-profile.children.cycles-pp.update_process_times
> 2.33 ? 4% -2.2 0.10 ? 4% perf-profile.children.cycles-pp.__account_scheduler_latency
> 2.05 ? 5% -2.0 0.08 ? 5% perf-profile.children.cycles-pp.stack_trace_save_tsk
> 1.85 ? 4% -1.8 0.08 ? 10% perf-profile.children.cycles-pp.arch_stack_walk
> 1.74 -1.6 0.10 ? 4% perf-profile.children.cycles-pp.schedule
> 1.71 -1.6 0.10 ? 7% perf-profile.children.cycles-pp.io_schedule
> 1.20 ? 4% -1.1 0.06 ? 7% perf-profile.children.cycles-pp.unwind_next_frame
> 1.15 ? 6% -1.1 0.04 ? 58% perf-profile.children.cycles-pp.scheduler_tick
> 1.19 ? 18% -1.1 0.13 ? 6% perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
> 1.07 ? 18% -0.9 0.12 ? 6% perf-profile.children.cycles-pp.sched_ttwu_pending
> 0.65 ? 10% -0.5 0.14 ? 11% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.55 ? 17% -0.5 0.10 ? 16% perf-profile.children.cycles-pp.poll_idle
> 0.53 ? 7% -0.4 0.10 ? 10% perf-profile.children.cycles-pp.__libc_lseek64
> 0.59 ? 29% -0.4 0.20 ? 20% perf-profile.children.cycles-pp.start_kernel
> 0.50 ? 2% -0.4 0.12 ? 10% perf-profile.children.cycles-pp.copy_page_to_iter
> 0.25 ? 13% -0.2 0.03 ?100% perf-profile.children.cycles-pp.ksys_lseek
> 0.30 ? 6% -0.2 0.09 ? 9% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
> 0.29 ? 6% -0.2 0.09 ? 9% perf-profile.children.cycles-pp.copyout
> 0.25 ? 26% -0.2 0.05 ? 8% perf-profile.children.cycles-pp.__fdget_pos
> 0.00 +0.1 0.14 ? 10% perf-profile.children.cycles-pp.mark_page_accessed
> 0.07 ? 17% +0.5 0.54 ? 11% perf-profile.children.cycles-pp.find_get_entry
> 2.18 +25.8 27.95 ? 10% perf-profile.children.cycles-pp.__lock_page
> 2.46 +26.5 28.95 ? 10% perf-profile.children.cycles-pp.shmem_getpage_gfp
> 2.43 +26.5 28.92 ? 10% perf-profile.children.cycles-pp.find_lock_entry
> 0.23 ? 13% +27.3 27.49 ? 10% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 4.69 ? 4% +32.8 37.48 ? 9% perf-profile.children.cycles-pp.wake_up_page_bit
> 1.40 ? 5% +35.6 37.03 ? 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 9.53 ? 2% +57.9 67.46 ? 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 9.32 ? 2% +58.1 67.43 ? 9% perf-profile.children.cycles-pp.__libc_read
> 9.33 ? 2% +58.1 67.44 ? 9% perf-profile.children.cycles-pp.do_syscall_64
> 8.61 ? 2% +58.7 67.35 ? 9% perf-profile.children.cycles-pp.ksys_read
> 8.42 ? 2% +58.9 67.30 ? 9% perf-profile.children.cycles-pp.vfs_read
> 8.05 ? 2% +59.2 67.27 ? 9% perf-profile.children.cycles-pp.new_sync_read
> 7.99 ? 2% +59.3 67.26 ? 9% perf-profile.children.cycles-pp.shmem_file_read_iter
> 0.34 ? 7% +64.0 64.31 ? 9% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 45.49 -13.5 31.96 ? 20% perf-profile.self.cycles-pp.intel_idle
> 9.33 ? 5% -9.3 0.00 perf-profile.self.cycles-pp.menu_select
> 9.05 ? 14% -9.1 0.00 perf-profile.self.cycles-pp.cpuidle_enter_state
> 1.15 ? 5% -1.0 0.11 ? 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.49 ? 16% -0.4 0.10 ? 16% perf-profile.self.cycles-pp.poll_idle
> 0.29 ? 5% -0.2 0.09 ? 9% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
> 0.00 +0.1 0.14 ? 11% perf-profile.self.cycles-pp.mark_page_accessed
> 0.00 +0.2 0.18 ? 11% perf-profile.self.cycles-pp.wake_up_page_bit
> 0.19 ? 14% +0.3 0.47 ? 12% perf-profile.self.cycles-pp.shmem_file_read_iter
> 0.01 ?173% +0.3 0.33 ? 11% perf-profile.self.cycles-pp.find_lock_entry
> 0.06 ? 14% +0.5 0.51 ? 11% perf-profile.self.cycles-pp.find_get_entry
> 0.34 ? 7% +64.0 64.31 ? 9% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
> 40631 +54.3% 62702 ? 5% softirqs.CPU0.SCHED
> 39633 ? 5% +94.6% 77112 ? 7% softirqs.CPU1.SCHED
> 38496 +106.7% 79558 ? 4% softirqs.CPU10.SCHED
> 38753 +88.1% 72913 ? 7% softirqs.CPU100.SCHED
> 38787 +84.0% 71383 ? 12% softirqs.CPU101.SCHED
> 38756 +75.9% 68184 ? 7% softirqs.CPU102.SCHED
> 38730 +78.4% 69103 ? 6% softirqs.CPU103.SCHED
> 38745 +75.2% 67870 ? 5% softirqs.CPU104.SCHED
> 38709 +94.0% 75083 ? 8% softirqs.CPU105.SCHED
> 38983 +69.0% 65884 ? 10% softirqs.CPU106.SCHED
> 38712 +73.6% 67205 ? 6% softirqs.CPU107.SCHED
> 38657 +79.1% 69224 ? 6% softirqs.CPU108.SCHED
> 38755 +82.1% 70592 ? 9% softirqs.CPU109.SCHED
> 38376 +103.8% 78211 ? 7% softirqs.CPU11.SCHED
> 38830 +73.5% 67362 ? 8% softirqs.CPU110.SCHED
> 38711 +78.9% 69245 ? 12% softirqs.CPU111.SCHED
> 38752 +74.7% 67717 ? 6% softirqs.CPU112.SCHED
> 38878 +82.8% 71085 ? 8% softirqs.CPU113.SCHED
> 38720 +79.7% 69581 ? 9% softirqs.CPU114.SCHED
> 38626 +74.7% 67473 ? 7% softirqs.CPU115.SCHED
> 34760 ? 20% +102.9% 70520 ? 8% softirqs.CPU116.SCHED
> 38677 +76.2% 68131 ? 7% softirqs.CPU117.SCHED
> 35465 ? 21% +91.8% 68005 ? 7% softirqs.CPU118.SCHED
> 34519 ? 20% +103.1% 70108 ? 7% softirqs.CPU119.SCHED
> 38170 +100.7% 76597 ? 9% softirqs.CPU12.SCHED
> 39018 +83.8% 71712 ? 8% softirqs.CPU120.SCHED
> 38857 +75.0% 67994 ? 9% softirqs.CPU121.SCHED
> 38733 +76.0% 68183 ? 8% softirqs.CPU122.SCHED
> 38784 +78.3% 69142 ? 6% softirqs.CPU123.SCHED
> 38778 +74.8% 67770 ? 3% softirqs.CPU124.SCHED
> 39019 +74.5% 68088 ? 6% softirqs.CPU125.SCHED
> 38781 +74.9% 67826 ? 7% softirqs.CPU126.SCHED
> 38770 +71.9% 66651 ? 8% softirqs.CPU127.SCHED
> 38841 +81.5% 70505 ? 8% softirqs.CPU128.SCHED
> 38690 +75.1% 67746 ? 4% softirqs.CPU129.SCHED
> 38239 ? 2% +96.2% 75045 ? 6% softirqs.CPU13.SCHED
> 38875 +79.2% 69655 ? 8% softirqs.CPU130.SCHED
> 38866 +73.9% 67575 ? 7% softirqs.CPU131.SCHED
> 38723 +75.5% 67944 ? 7% softirqs.CPU132.SCHED
> 38643 +78.7% 69042 ? 9% softirqs.CPU133.SCHED
> 38669 +78.9% 69174 ? 9% softirqs.CPU134.SCHED
> 38658 +78.7% 69077 ? 9% softirqs.CPU135.SCHED
> 38795 +75.0% 67873 ? 12% softirqs.CPU136.SCHED
> 38817 +70.2% 66079 ? 8% softirqs.CPU137.SCHED
> 39032 +75.8% 68606 ? 6% softirqs.CPU138.SCHED
> 38864 +72.8% 67154 ? 10% softirqs.CPU139.SCHED
> 38779 ? 3% +102.2% 78423 ? 5% softirqs.CPU14.SCHED
> 38948 +81.9% 70830 ? 10% softirqs.CPU140.SCHED
> 38662 +83.5% 70926 ? 12% softirqs.CPU141.SCHED
> 35132 ? 19% +98.2% 69627 ? 8% softirqs.CPU142.SCHED
> 38825 +69.5% 65816 ? 5% softirqs.CPU143.SCHED
> 35007 ? 20% +99.3% 69785 ? 8% softirqs.CPU144.SCHED
> 38938 +76.3% 68648 ? 10% softirqs.CPU145.SCHED
> 38990 +67.6% 65363 ? 3% softirqs.CPU146.SCHED
> 39037 +74.1% 67972 ? 2% softirqs.CPU147.SCHED
> 38810 +76.8% 68600 ? 4% softirqs.CPU148.SCHED
> 38929 +76.2% 68578 ? 10% softirqs.CPU149.SCHED
> 38071 +101.6% 76734 ? 6% softirqs.CPU15.SCHED
> 38904 +81.0% 70427 ? 14% softirqs.CPU150.SCHED
> 38908 +72.7% 67194 ? 10% softirqs.CPU151.SCHED
> 38827 +75.7% 68201 ? 8% softirqs.CPU152.SCHED
> 38848 +70.5% 66256 ? 4% softirqs.CPU153.SCHED
> 38894 +75.4% 68216 ? 9% softirqs.CPU154.SCHED
> 38881 +79.8% 69913 ? 4% softirqs.CPU155.SCHED
> 38886 +73.5% 67482 ? 9% softirqs.CPU156.SCHED
> 38777 +73.2% 67168 ? 6% softirqs.CPU157.SCHED
> 38833 +76.4% 68520 ? 10% softirqs.CPU158.SCHED
> 38874 +71.4% 66620 ? 9% softirqs.CPU159.SCHED
> 38189 +104.4% 78046 ? 6% softirqs.CPU16.SCHED
> 38889 +71.9% 66832 ? 7% softirqs.CPU160.SCHED
> 38982 +80.9% 70533 ? 7% softirqs.CPU161.SCHED
> 38937 +71.4% 66745 ? 4% softirqs.CPU162.SCHED
> 34815 ? 20% +96.1% 68256 ? 6% softirqs.CPU163.SCHED
> 38870 +72.7% 67148 ? 9% softirqs.CPU164.SCHED
> 38827 +76.0% 68355 ? 6% softirqs.CPU165.SCHED
> 38850 +82.1% 70729 ? 10% softirqs.CPU166.SCHED
> 35197 ? 21% +100.0% 70389 ? 10% softirqs.CPU167.SCHED
> 39361 ? 2% +82.8% 71946 ? 6% softirqs.CPU168.SCHED
> 38813 +82.6% 70862 ? 7% softirqs.CPU169.SCHED
> 38121 +95.0% 74331 ? 7% softirqs.CPU17.SCHED
> 38956 +80.2% 70181 ? 6% softirqs.CPU170.SCHED
> 38916 +79.7% 69914 ? 6% softirqs.CPU171.SCHED
> 38859 +76.2% 68479 softirqs.CPU172.SCHED
> 39012 +78.8% 69749 ? 9% softirqs.CPU173.SCHED
> 38863 +71.7% 66736 ? 4% softirqs.CPU174.SCHED
> 38841 +81.3% 70431 ? 7% softirqs.CPU175.SCHED
> 38806 +78.9% 69422 ? 10% softirqs.CPU176.SCHED
> 38885 +71.6% 66722 ? 6% softirqs.CPU177.SCHED
> 38896 +79.5% 69803 ? 9% softirqs.CPU178.SCHED
> 38940 +72.3% 67090 ? 12% softirqs.CPU179.SCHED
> 37988 +101.2% 76416 ? 6% softirqs.CPU18.SCHED
> 38844 +74.8% 67888 ? 4% softirqs.CPU180.SCHED
> 38949 +70.0% 66213 ? 5% softirqs.CPU181.SCHED
> 39087 +70.4% 66587 ? 5% softirqs.CPU182.SCHED
> 38848 +70.4% 66185 ? 5% softirqs.CPU183.SCHED
> 38838 +78.3% 69242 ? 7% softirqs.CPU184.SCHED
> 38993 +72.7% 67340 ? 10% softirqs.CPU185.SCHED
> 38797 +67.6% 65035 ? 7% softirqs.CPU186.SCHED
> 38794 +69.4% 65713 ? 6% softirqs.CPU187.SCHED
> 38914 +62.2% 63112 ? 6% softirqs.CPU188.SCHED
> 38898 +71.8% 66822 ? 4% softirqs.CPU189.SCHED
> 38250 +102.0% 77270 ? 7% softirqs.CPU19.SCHED
> 38851 +63.0% 63339 ? 4% softirqs.CPU190.SCHED
> 38794 +23.5% 47899 ? 3% softirqs.CPU191.SCHED
> 38426 +97.9% 76043 ? 11% softirqs.CPU2.SCHED
> 38106 +97.4% 75204 ? 5% softirqs.CPU20.SCHED
> 39713 ? 7% +95.0% 77425 ? 10% softirqs.CPU21.SCHED
> 38374 +102.0% 77519 ? 5% softirqs.CPU22.SCHED
> 38131 +98.3% 75617 ? 6% softirqs.CPU23.SCHED
> 39241 ? 3% +83.5% 72016 ? 5% softirqs.CPU24.SCHED
> 30118 ? 27% +151.0% 75608 ? 6% softirqs.CPU25.SCHED
> 38278 +97.2% 75482 ? 6% softirqs.CPU26.SCHED
> 38217 +96.1% 74939 ? 6% softirqs.CPU27.SCHED
> 38037 +99.1% 75732 ? 9% softirqs.CPU28.SCHED
> 38193 +98.2% 75710 ? 6% softirqs.CPU29.SCHED
> 38287 +102.3% 77439 ? 6% softirqs.CPU3.SCHED
> 38154 +99.0% 75924 ? 7% softirqs.CPU30.SCHED
> 38073 +102.0% 76913 ? 6% softirqs.CPU31.SCHED
> 38084 +92.9% 73483 ? 6% softirqs.CPU32.SCHED
> 38042 +99.3% 75828 ? 10% softirqs.CPU33.SCHED
> 38193 +94.2% 74188 ? 5% softirqs.CPU34.SCHED
> 38243 +98.8% 76013 ? 5% softirqs.CPU35.SCHED
> 37961 +99.4% 75711 ? 5% softirqs.CPU36.SCHED
> 38135 +94.9% 74336 ? 6% softirqs.CPU37.SCHED
> 38335 +94.2% 74461 ? 3% softirqs.CPU38.SCHED
> 38342 +94.4% 74533 ? 8% softirqs.CPU39.SCHED
> 37627 ? 2% +94.6% 73233 ? 6% softirqs.CPU4.SCHED
> 38084 +98.8% 75711 ? 2% softirqs.CPU40.SCHED
> 38257 +102.8% 77575 ? 5% softirqs.CPU41.SCHED
> 37955 +97.2% 74843 ? 9% softirqs.CPU42.SCHED
> 38071 +100.0% 76133 ? 4% softirqs.CPU43.SCHED
> 38082 +92.2% 73183 ? 5% softirqs.CPU44.SCHED
> 38148 +90.7% 72759 ? 7% softirqs.CPU45.SCHED
> 34308 ? 20% +115.3% 73863 ? 7% softirqs.CPU46.SCHED
> 38238 +103.4% 77791 ? 9% softirqs.CPU47.SCHED
> 38665 +94.8% 75310 ? 5% softirqs.CPU48.SCHED
> 38415 +99.2% 76524 ? 5% softirqs.CPU49.SCHED
> 38158 +94.4% 74167 ? 6% softirqs.CPU5.SCHED
> 38443 +104.8% 78724 ? 7% softirqs.CPU50.SCHED
> 34425 ? 20% +125.0% 77470 ? 10% softirqs.CPU51.SCHED
> 38267 +99.8% 76461 ? 7% softirqs.CPU52.SCHED
> 38349 +99.2% 76392 ? 4% softirqs.CPU53.SCHED
> 38403 +94.9% 74848 ? 5% softirqs.CPU54.SCHED
> 38395 +101.7% 77431 ? 5% softirqs.CPU55.SCHED
> 38356 +100.3% 76846 ? 5% softirqs.CPU56.SCHED
> 38445 +105.2% 78902 ? 11% softirqs.CPU57.SCHED
> 38351 +100.3% 76801 ? 4% softirqs.CPU58.SCHED
> 38463 +96.3% 75505 ? 9% softirqs.CPU59.SCHED
> 38189 +102.9% 77480 ? 6% softirqs.CPU6.SCHED
> 38471 +101.3% 77442 ? 6% softirqs.CPU60.SCHED
> 38361 +102.9% 77834 ? 10% softirqs.CPU61.SCHED
> 38143 +98.5% 75709 ? 7% softirqs.CPU62.SCHED
> 38955 ? 2% +102.1% 78740 ? 6% softirqs.CPU63.SCHED
> 38370 +103.9% 78255 ? 6% softirqs.CPU64.SCHED
> 38401 +94.2% 74557 ? 5% softirqs.CPU65.SCHED
> 38433 +104.2% 78462 ? 7% softirqs.CPU66.SCHED
> 38339 +100.5% 76880 ? 6% softirqs.CPU67.SCHED
> 38398 +103.3% 78048 ? 4% softirqs.CPU68.SCHED
> 38319 +100.5% 76811 ? 8% softirqs.CPU69.SCHED
> 34179 ? 21% +123.9% 76523 ? 7% softirqs.CPU7.SCHED
> 38552 +93.5% 74604 ? 4% softirqs.CPU70.SCHED
> 38506 +93.4% 74456 ? 6% softirqs.CPU71.SCHED
> 38504 +92.3% 74035 ? 5% softirqs.CPU72.SCHED
> 38134 +96.9% 75089 ? 7% softirqs.CPU73.SCHED
> 38196 +99.0% 76000 ? 6% softirqs.CPU74.SCHED
> 38306 +99.1% 76276 ? 6% softirqs.CPU75.SCHED
> 38705 +100.5% 77595 ? 10% softirqs.CPU76.SCHED
> 38366 +99.0% 76349 ? 5% softirqs.CPU77.SCHED
> 38450 +106.6% 79426 ? 8% softirqs.CPU78.SCHED
> 38155 +98.8% 75834 ? 4% softirqs.CPU79.SCHED
> 37097 ? 2% +111.0% 78291 ? 8% softirqs.CPU8.SCHED
> 38264 +100.0% 76540 ? 3% softirqs.CPU80.SCHED
> 34260 ? 20% +130.9% 79108 ? 7% softirqs.CPU81.SCHED
> 38212 +99.3% 76155 ? 4% softirqs.CPU82.SCHED
> 38116 +106.6% 78750 softirqs.CPU83.SCHED
> 38164 +104.8% 78146 ? 7% softirqs.CPU84.SCHED
> 38154 +109.1% 79799 ? 7% softirqs.CPU85.SCHED
> 38363 +107.0% 79418 ? 9% softirqs.CPU86.SCHED
> 38224 +108.8% 79830 ? 7% softirqs.CPU87.SCHED
> 38587 +99.1% 76832 ? 6% softirqs.CPU88.SCHED
> 38495 +104.7% 78813 ? 3% softirqs.CPU89.SCHED
> 38367 ? 2% +83.6% 70426 ? 4% softirqs.CPU9.SCHED
> 34134 ? 19% +137.1% 80916 ? 10% softirqs.CPU90.SCHED
> 38192 +109.9% 80161 ? 7% softirqs.CPU91.SCHED
> 38336 +116.7% 83080 ? 7% softirqs.CPU92.SCHED
> 38193 +107.3% 79164 ? 8% softirqs.CPU93.SCHED
> 27437 ? 12% -25.5% 20438 ? 13% softirqs.CPU94.RCU
> 38299 +115.9% 82699 ? 8% softirqs.CPU94.SCHED
> 18235 ? 4% +202.2% 55116 ? 33% softirqs.CPU95.SCHED
> 36482 ? 13% +135.6% 85946 ? 9% softirqs.CPU96.SCHED
> 38587 +80.8% 69767 ? 4% softirqs.CPU97.SCHED
> 38930 +78.6% 69537 ? 4% softirqs.CPU98.SCHED
> 32564 ? 32% +111.7% 68941 ? 7% softirqs.CPU99.SCHED
> 7325109 +89.6% 13891305 ? 6% softirqs.SCHED
> 236548 ? 4% +1218.3% 3118323 ? 8% interrupts.CAL:Function_call_interrupts
> 1429 ? 10% +585.9% 9804 ? 24% interrupts.CPU0.CAL:Function_call_interrupts
> 231.75 ? 33% +2494.9% 6013 ? 13% interrupts.CPU0.NMI:Non-maskable_interrupts
> 231.75 ? 33% +2494.9% 6013 ? 13% interrupts.CPU0.PMI:Performance_monitoring_interrupts
> 6.00 ? 69% +6658.3% 405.50 ? 23% interrupts.CPU0.TLB:TLB_shootdowns
> 1364 ? 16% +1245.8% 18364 ? 4% interrupts.CPU1.CAL:Function_call_interrupts
> 203.00 +3411.8% 7129 ? 4% interrupts.CPU1.NMI:Non-maskable_interrupts
> 203.00 +3411.8% 7129 ? 4% interrupts.CPU1.PMI:Performance_monitoring_interrupts
> 1143 ? 15% -66.3% 385.25 ? 8% interrupts.CPU1.RES:Rescheduling_interrupts
> 21.00 ? 83% +3652.4% 788.00 ? 7% interrupts.CPU1.TLB:TLB_shootdowns
> 1110 ? 5% +1759.6% 20646 ? 12% interrupts.CPU10.CAL:Function_call_interrupts
> 208.00 +2277.9% 4946 ? 29% interrupts.CPU10.NMI:Non-maskable_interrupts
> 208.00 +2277.9% 4946 ? 29% interrupts.CPU10.PMI:Performance_monitoring_interrupts
> 830.50 ? 28% -51.5% 402.50 ? 13% interrupts.CPU10.RES:Rescheduling_interrupts
> 29.00 ?143% +2773.3% 833.25 ? 13% interrupts.CPU10.TLB:TLB_shootdowns
> 1131 ? 18% +1366.0% 16591 ? 7% interrupts.CPU100.CAL:Function_call_interrupts
> 204.00 ? 3% +2103.3% 4494 ? 20% interrupts.CPU100.NMI:Non-maskable_interrupts
> 204.00 ? 3% +2103.3% 4494 ? 20% interrupts.CPU100.PMI:Performance_monitoring_interrupts
> 1286 ? 8% -74.4% 329.50 ? 9% interrupts.CPU100.RES:Rescheduling_interrupts
> 9.00 ? 65% +8488.9% 773.00 ? 14% interrupts.CPU100.TLB:TLB_shootdowns
> 1189 ? 23% +1270.2% 16298 ? 12% interrupts.CPU101.CAL:Function_call_interrupts
> 202.00 ? 3% +3234.9% 6736 ? 14% interrupts.CPU101.NMI:Non-maskable_interrupts
> 202.00 ? 3% +3234.9% 6736 ? 14% interrupts.CPU101.PMI:Performance_monitoring_interrupts
> 1486 ? 20% -80.0% 297.50 ? 11% interrupts.CPU101.RES:Rescheduling_interrupts
> 17.75 ? 97% +3942.3% 717.50 ? 10% interrupts.CPU101.TLB:TLB_shootdowns
> 1066 ? 9% +1193.7% 13800 ? 7% interrupts.CPU102.CAL:Function_call_interrupts
> 201.75 ? 3% +2311.2% 4864 ? 29% interrupts.CPU102.NMI:Non-maskable_interrupts
> 201.75 ? 3% +2311.2% 4864 ? 29% interrupts.CPU102.PMI:Performance_monitoring_interrupts
> 1124 ? 17% -75.4% 276.25 ? 10% interrupts.CPU102.RES:Rescheduling_interrupts
> 8.00 ? 30% +7431.2% 602.50 ? 11% interrupts.CPU102.TLB:TLB_shootdowns
> 1151 ? 12% +1149.8% 14391 ? 5% interrupts.CPU103.CAL:Function_call_interrupts
> 180.75 ? 26% +2754.5% 5159 ? 23% interrupts.CPU103.NMI:Non-maskable_interrupts
> 180.75 ? 26% +2754.5% 5159 ? 23% interrupts.CPU103.PMI:Performance_monitoring_interrupts
> 1452 ? 9% -79.3% 301.25 ? 16% interrupts.CPU103.RES:Rescheduling_interrupts
> 22.00 ? 69% +2867.0% 652.75 ? 17% interrupts.CPU103.TLB:TLB_shootdowns
> 1373 ? 17% +954.6% 14487 ? 8% interrupts.CPU104.CAL:Function_call_interrupts
> 176.00 ? 25% +2905.3% 5289 ? 23% interrupts.CPU104.NMI:Non-maskable_interrupts
> 176.00 ? 25% +2905.3% 5289 ? 23% interrupts.CPU104.PMI:Performance_monitoring_interrupts
> 1366 ? 24% -80.1% 271.75 ? 18% interrupts.CPU104.RES:Rescheduling_interrupts
> 9.25 ? 15% +6373.0% 598.75 ? 14% interrupts.CPU104.TLB:TLB_shootdowns
> 1337 ? 8% +1221.1% 17663 ? 12% interrupts.CPU105.CAL:Function_call_interrupts
> 205.25 +2922.3% 6203 ? 26% interrupts.CPU105.NMI:Non-maskable_interrupts
> 205.25 +2922.3% 6203 ? 26% interrupts.CPU105.PMI:Performance_monitoring_interrupts
> 1466 ? 13% -77.0% 337.50 ? 11% interrupts.CPU105.RES:Rescheduling_interrupts
> 29.50 ? 66% +2657.6% 813.50 ? 9% interrupts.CPU105.TLB:TLB_shootdowns
> 1314 ? 10% +937.2% 13633 ? 17% interrupts.CPU106.CAL:Function_call_interrupts
> 179.00 ? 24% +2685.2% 4985 ? 31% interrupts.CPU106.NMI:Non-maskable_interrupts
> 179.00 ? 24% +2685.2% 4985 ? 31% interrupts.CPU106.PMI:Performance_monitoring_interrupts
> 1635 ? 9% -84.8% 248.00 ? 4% interrupts.CPU106.RES:Rescheduling_interrupts
> 11.25 ? 21% +5428.9% 622.00 ? 18% interrupts.CPU106.TLB:TLB_shootdowns
> 1110 ? 6% +1134.1% 13701 ? 2% interrupts.CPU107.CAL:Function_call_interrupts
> 174.75 ? 24% +2917.6% 5273 ? 28% interrupts.CPU107.NMI:Non-maskable_interrupts
> 174.75 ? 24% +2917.6% 5273 ? 28% interrupts.CPU107.PMI:Performance_monitoring_interrupts
> 1395 ? 27% -82.0% 250.75 ? 16% interrupts.CPU107.RES:Rescheduling_interrupts
> 8.75 ? 35% +7760.0% 687.75 ? 19% interrupts.CPU107.TLB:TLB_shootdowns
> 1461 ? 9% +997.1% 16036 ? 18% interrupts.CPU108.CAL:Function_call_interrupts
> 177.25 ? 25% +3711.1% 6755 ? 15% interrupts.CPU108.NMI:Non-maskable_interrupts
> 177.25 ? 25% +3711.1% 6755 ? 15% interrupts.CPU108.PMI:Performance_monitoring_interrupts
> 1525 ? 9% -81.6% 280.25 ? 19% interrupts.CPU108.RES:Rescheduling_interrupts
> 22.00 ? 60% +2913.6% 663.00 ? 14% interrupts.CPU108.TLB:TLB_shootdowns
> 1167 ? 10% +1162.5% 14736 ? 14% interrupts.CPU109.CAL:Function_call_interrupts
> 151.50 ? 31% +3887.3% 6040 ? 33% interrupts.CPU109.NMI:Non-maskable_interrupts
> 151.50 ? 31% +3887.3% 6040 ? 33% interrupts.CPU109.PMI:Performance_monitoring_interrupts
> 1331 ? 28% -76.7% 310.50 ? 14% interrupts.CPU109.RES:Rescheduling_interrupts
> 8.00 ? 39% +8687.5% 703.00 ? 14% interrupts.CPU109.TLB:TLB_shootdowns
> 1125 ? 13% +1642.3% 19605 ? 14% interrupts.CPU11.CAL:Function_call_interrupts
> 176.25 ? 23% +2172.6% 4005 ? 44% interrupts.CPU11.NMI:Non-maskable_interrupts
> 176.25 ? 23% +2172.6% 4005 ? 44% interrupts.CPU11.PMI:Performance_monitoring_interrupts
> 1199 ? 31% -67.0% 395.50 ? 13% interrupts.CPU11.RES:Rescheduling_interrupts
> 18.00 ? 74% +4216.7% 777.00 ? 15% interrupts.CPU11.TLB:TLB_shootdowns
> 1108 ? 10% +1073.9% 13009 ? 5% interrupts.CPU110.CAL:Function_call_interrupts
> 149.50 ? 31% +2692.1% 4174 ? 41% interrupts.CPU110.NMI:Non-maskable_interrupts
> 149.50 ? 31% +2692.1% 4174 ? 41% interrupts.CPU110.PMI:Performance_monitoring_interrupts
> 1303 ? 9% -80.7% 251.50 ? 12% interrupts.CPU110.RES:Rescheduling_interrupts
> 18.25 ? 79% +3406.8% 640.00 ? 17% interrupts.CPU110.TLB:TLB_shootdowns
> 1189 ? 21% +1172.3% 15130 ? 19% interrupts.CPU111.CAL:Function_call_interrupts
> 147.75 ? 33% +2802.4% 4288 ? 42% interrupts.CPU111.NMI:Non-maskable_interrupts
> 147.75 ? 33% +2802.4% 4288 ? 42% interrupts.CPU111.PMI:Performance_monitoring_interrupts
> 1547 ? 28% -82.1% 277.25 ? 21% interrupts.CPU111.RES:Rescheduling_interrupts
> 28.50 ? 99% +2150.9% 641.50 ? 29% interrupts.CPU111.TLB:TLB_shootdowns
> 1135 ? 12% +1154.3% 14236 ? 8% interrupts.CPU112.CAL:Function_call_interrupts
> 148.75 ? 32% +2586.7% 3996 ? 48% interrupts.CPU112.NMI:Non-maskable_interrupts
> 148.75 ? 32% +2586.7% 3996 ? 48% interrupts.CPU112.PMI:Performance_monitoring_interrupts
> 1316 ? 12% -79.5% 270.00 ? 13% interrupts.CPU112.RES:Rescheduling_interrupts
> 7.50 ? 46% +7680.0% 583.50 ? 12% interrupts.CPU112.TLB:TLB_shootdowns
> 1193 ? 16% +1226.6% 15833 ? 21% interrupts.CPU113.CAL:Function_call_interrupts
> 149.75 ? 32% +4233.9% 6490 ? 10% interrupts.CPU113.NMI:Non-maskable_interrupts
> 149.75 ? 32% +4233.9% 6490 ? 10% interrupts.CPU113.PMI:Performance_monitoring_interrupts
> 1572 ? 33% -81.0% 298.00 ? 11% interrupts.CPU113.RES:Rescheduling_interrupts
> 9.75 ? 61% +7625.6% 753.25 ? 10% interrupts.CPU113.TLB:TLB_shootdowns
> 1397 ? 15% +992.7% 15273 ? 9% interrupts.CPU114.CAL:Function_call_interrupts
> 148.00 ? 32% +3691.9% 5612 ? 39% interrupts.CPU114.NMI:Non-maskable_interrupts
> 148.00 ? 32% +3691.9% 5612 ? 39% interrupts.CPU114.PMI:Performance_monitoring_interrupts
> 1313 ? 13% -79.0% 276.00 ? 8% interrupts.CPU114.RES:Rescheduling_interrupts
> 28.75 ?101% +2879.1% 856.50 ? 18% interrupts.CPU114.TLB:TLB_shootdowns
> 1152 ? 11% +1054.8% 13309 ? 5% interrupts.CPU115.CAL:Function_call_interrupts
> 176.75 ? 25% +2775.8% 5083 ? 26% interrupts.CPU115.NMI:Non-maskable_interrupts
> 176.75 ? 25% +2775.8% 5083 ? 26% interrupts.CPU115.PMI:Performance_monitoring_interrupts
> 1278 ? 22% -78.1% 280.00 ? 12% interrupts.CPU115.RES:Rescheduling_interrupts
> 8.75 ? 31% +7614.3% 675.00 ? 11% interrupts.CPU115.TLB:TLB_shootdowns
> 1427 ? 24% +934.8% 14774 ? 15% interrupts.CPU116.CAL:Function_call_interrupts
> 203.00 ? 3% +2693.3% 5670 ? 25% interrupts.CPU116.NMI:Non-maskable_interrupts
> 203.00 ? 3% +2693.3% 5670 ? 25% interrupts.CPU116.PMI:Performance_monitoring_interrupts
> 1451 ? 35% -77.1% 331.75 ? 13% interrupts.CPU116.RES:Rescheduling_interrupts
> 11.50 ? 29% +4582.6% 538.50 ? 14% interrupts.CPU116.TLB:TLB_shootdowns
> 1225 ? 16% +1001.5% 13499 ? 21% interrupts.CPU117.CAL:Function_call_interrupts
> 176.50 ? 25% +2546.0% 4670 ? 23% interrupts.CPU117.NMI:Non-maskable_interrupts
> 176.50 ? 25% +2546.0% 4670 ? 23% interrupts.CPU117.PMI:Performance_monitoring_interrupts
> 1317 ? 28% -78.7% 281.25 ? 30% interrupts.CPU117.RES:Rescheduling_interrupts
> 8.25 ? 71% +7412.1% 619.75 ? 30% interrupts.CPU117.TLB:TLB_shootdowns
> 1216 ? 12% +1026.3% 13704 ? 6% interrupts.CPU118.CAL:Function_call_interrupts
> 204.75 ? 4% +2645.8% 5622 ? 29% interrupts.CPU118.NMI:Non-maskable_interrupts
> 204.75 ? 4% +2645.8% 5622 ? 29% interrupts.CPU118.PMI:Performance_monitoring_interrupts
> 1314 ? 20% -78.5% 283.00 ? 9% interrupts.CPU118.RES:Rescheduling_interrupts
> 19.25 ? 81% +3127.3% 621.25 ? 17% interrupts.CPU118.TLB:TLB_shootdowns
> 1140 ? 8% +1211.5% 14961 ? 4% interrupts.CPU119.CAL:Function_call_interrupts
> 202.75 ? 4% +2414.2% 5097 ? 29% interrupts.CPU119.NMI:Non-maskable_interrupts
> 202.75 ? 4% +2414.2% 5097 ? 29% interrupts.CPU119.PMI:Performance_monitoring_interrupts
> 1249 ? 11% -76.3% 296.00 ? 18% interrupts.CPU119.RES:Rescheduling_interrupts
> 10.00 ? 27% +6227.5% 632.75 ? 11% interrupts.CPU119.TLB:TLB_shootdowns
> 1262 ? 10% +1468.3% 19799 ? 8% interrupts.CPU12.CAL:Function_call_interrupts
> 207.25 ? 2% +1845.1% 4031 ? 23% interrupts.CPU12.NMI:Non-maskable_interrupts
> 207.25 ? 2% +1845.1% 4031 ? 23% interrupts.CPU12.PMI:Performance_monitoring_interrupts
> 810.50 ? 27% -52.0% 389.00 ? 10% interrupts.CPU12.RES:Rescheduling_interrupts
> 15.75 ? 90% +5019.0% 806.25 ? 8% interrupts.CPU12.TLB:TLB_shootdowns
> 1266 ? 4% +1199.9% 16462 ? 7% interrupts.CPU120.CAL:Function_call_interrupts
> 203.75 ? 3% +2367.0% 5026 ? 32% interrupts.CPU120.NMI:Non-maskable_interrupts
> 203.75 ? 3% +2367.0% 5026 ? 32% interrupts.CPU120.PMI:Performance_monitoring_interrupts
> 1049 ? 15% -66.2% 354.75 ? 17% interrupts.CPU120.RES:Rescheduling_interrupts
> 18.75 ? 84% +3682.7% 709.25 ? 12% interrupts.CPU120.TLB:TLB_shootdowns
> 1206 ? 15% +1076.3% 14194 ? 7% interrupts.CPU121.CAL:Function_call_interrupts
> 199.50 ? 4% +2815.2% 5815 ? 17% interrupts.CPU121.NMI:Non-maskable_interrupts
> 199.50 ? 4% +2815.2% 5815 ? 17% interrupts.CPU121.PMI:Performance_monitoring_interrupts
> 1507 ? 28% -81.1% 284.50 ? 18% interrupts.CPU121.RES:Rescheduling_interrupts
> 11.00 ? 22% +5811.4% 650.25 ? 20% interrupts.CPU121.TLB:TLB_shootdowns
> 1016 ? 5% +1261.5% 13839 ? 7% interrupts.CPU122.CAL:Function_call_interrupts
> 198.75 ? 4% +3276.7% 6711 ? 4% interrupts.CPU122.NMI:Non-maskable_interrupts
> 198.75 ? 4% +3276.7% 6711 ? 4% interrupts.CPU122.PMI:Performance_monitoring_interrupts
> 1169 ? 30% -73.5% 310.00 ? 2% interrupts.CPU122.RES:Rescheduling_interrupts
> 12.50 ? 32% +5146.0% 655.75 ? 20% interrupts.CPU122.TLB:TLB_shootdowns
> 1141 ? 5% +1093.9% 13625 ? 10% interrupts.CPU123.CAL:Function_call_interrupts
> 178.75 ? 27% +3893.4% 7138 ? 11% interrupts.CPU123.NMI:Non-maskable_interrupts
> 178.75 ? 27% +3893.4% 7138 ? 11% interrupts.CPU123.PMI:Performance_monitoring_interrupts
> 1197 ? 52% -73.1% 321.50 ? 16% interrupts.CPU123.RES:Rescheduling_interrupts
> 8.75 ? 75% +7505.7% 665.50 ? 24% interrupts.CPU123.TLB:TLB_shootdowns
> 1079 ? 8% +1259.3% 14673 ? 12% interrupts.CPU124.CAL:Function_call_interrupts
> 198.50 ? 4% +2452.9% 5067 ? 28% interrupts.CPU124.NMI:Non-maskable_interrupts
> 198.50 ? 4% +2452.9% 5067 ? 28% interrupts.CPU124.PMI:Performance_monitoring_interrupts
> 1326 ? 10% -78.6% 284.25 ? 27% interrupts.CPU124.RES:Rescheduling_interrupts
> 11.25 ? 14% +6237.8% 713.00 ? 14% interrupts.CPU124.TLB:TLB_shootdowns
> 1146 ? 8% +1212.8% 15051 ? 7% interrupts.CPU125.CAL:Function_call_interrupts
> 200.50 ? 3% +2483.8% 5180 ? 28% interrupts.CPU125.NMI:Non-maskable_interrupts
> 200.50 ? 3% +2483.8% 5180 ? 28% interrupts.CPU125.PMI:Performance_monitoring_interrupts
> 1121 ? 12% -75.7% 272.25 ? 11% interrupts.CPU125.RES:Rescheduling_interrupts
> 8.00 ? 46% +8106.2% 656.50 ? 6% interrupts.CPU125.TLB:TLB_shootdowns
> 1054 ? 6% +1175.3% 13451 ? 12% interrupts.CPU126.CAL:Function_call_interrupts
> 196.00 ? 3% +2189.5% 4487 ? 37% interrupts.CPU126.NMI:Non-maskable_interrupts
> 196.00 ? 3% +2189.5% 4487 ? 37% interrupts.CPU126.PMI:Performance_monitoring_interrupts
> 1300 ? 6% -77.5% 293.25 ? 13% interrupts.CPU126.RES:Rescheduling_interrupts
> 10.50 ? 44% +6485.7% 691.50 ? 8% interrupts.CPU126.TLB:TLB_shootdowns
> 1201 ? 16% +1013.0% 13375 ? 7% interrupts.CPU127.CAL:Function_call_interrupts
> 175.50 ? 25% +3272.9% 5919 ? 33% interrupts.CPU127.NMI:Non-maskable_interrupts
> 175.50 ? 25% +3272.9% 5919 ? 33% interrupts.CPU127.PMI:Performance_monitoring_interrupts
> 1281 ? 10% -75.4% 315.75 ? 19% interrupts.CPU127.RES:Rescheduling_interrupts
> 33.75 ? 81% +1794.8% 639.50 ? 3% interrupts.CPU127.TLB:TLB_shootdowns
> 1316 ? 10% +941.9% 13716 ? 22% interrupts.CPU128.CAL:Function_call_interrupts
> 171.50 ? 23% +2881.6% 5113 ? 31% interrupts.CPU128.NMI:Non-maskable_interrupts
> 171.50 ? 23% +2881.6% 5113 ? 31% interrupts.CPU128.PMI:Performance_monitoring_interrupts
> 1202 ? 9% -73.0% 324.25 ? 13% interrupts.CPU128.RES:Rescheduling_interrupts
> 14.50 ? 28% +4265.5% 633.00 ? 22% interrupts.CPU128.TLB:TLB_shootdowns
> 1198 ? 10% +1072.9% 14057 ? 19% interrupts.CPU129.CAL:Function_call_interrupts
> 173.00 ? 25% +2972.8% 5316 ? 41% interrupts.CPU129.NMI:Non-maskable_interrupts
> 173.00 ? 25% +2972.8% 5316 ? 41% interrupts.CPU129.PMI:Performance_monitoring_interrupts
> 1219 ? 38% -74.2% 314.75 ? 22% interrupts.CPU129.RES:Rescheduling_interrupts
> 15.25 ? 43% +4042.6% 631.75 ? 15% interrupts.CPU129.TLB:TLB_shootdowns
> 1569 ? 50% +997.0% 17214 ? 9% interrupts.CPU13.CAL:Function_call_interrupts
> 204.75 ? 5% +2106.8% 4518 ? 53% interrupts.CPU13.NMI:Non-maskable_interrupts
> 204.75 ? 5% +2106.8% 4518 ? 53% interrupts.CPU13.PMI:Performance_monitoring_interrupts
> 1204 ? 29% -71.1% 348.50 ? 27% interrupts.CPU13.RES:Rescheduling_interrupts
> 16.75 ? 87% +4329.9% 742.00 ? 14% interrupts.CPU13.TLB:TLB_shootdowns
> 1000 ? 4% +1382.7% 14830 ? 12% interrupts.CPU130.CAL:Function_call_interrupts
> 172.00 ? 24% +3391.7% 6005 ? 23% interrupts.CPU130.NMI:Non-maskable_interrupts
> 172.00 ? 24% +3391.7% 6005 ? 23% interrupts.CPU130.PMI:Performance_monitoring_interrupts
> 1174 ? 14% -73.3% 313.25 ? 19% interrupts.CPU130.RES:Rescheduling_interrupts
> 10.25 ? 32% +6546.3% 681.25 ? 4% interrupts.CPU130.TLB:TLB_shootdowns
> 1156 ? 9% +1093.9% 13804 ? 14% interrupts.CPU131.CAL:Function_call_interrupts
> 176.00 ? 28% +2972.2% 5407 ? 28% interrupts.CPU131.NMI:Non-maskable_interrupts
> 176.00 ? 28% +2972.2% 5407 ? 28% interrupts.CPU131.PMI:Performance_monitoring_interrupts
> 1370 ? 22% -76.7% 319.25 ? 14% interrupts.CPU131.RES:Rescheduling_interrupts
> 30.00 ? 52% +1986.7% 626.00 ? 19% interrupts.CPU131.TLB:TLB_shootdowns
> 1151 ? 9% +1135.4% 14222 ? 13% interrupts.CPU132.CAL:Function_call_interrupts
> 168.00 ? 23% +2673.5% 4659 ? 29% interrupts.CPU132.NMI:Non-maskable_interrupts
> 168.00 ? 23% +2673.5% 4659 ? 29% interrupts.CPU132.PMI:Performance_monitoring_interrupts
> 1643 ? 13% -79.8% 332.75 ? 16% interrupts.CPU132.RES:Rescheduling_interrupts
> 14.50 ? 36% +5208.6% 769.75 ? 11% interrupts.CPU132.TLB:TLB_shootdowns
> 1150 ? 9% +1148.8% 14370 ? 9% interrupts.CPU133.CAL:Function_call_interrupts
> 144.25 ? 34% +3515.1% 5214 ? 25% interrupts.CPU133.NMI:Non-maskable_interrupts
> 144.25 ? 34% +3515.1% 5214 ? 25% interrupts.CPU133.PMI:Performance_monitoring_interrupts
> 1200 ? 27% -74.7% 303.50 ? 14% interrupts.CPU133.RES:Rescheduling_interrupts
> 27.75 ? 77% +2288.3% 662.75 ? 13% interrupts.CPU133.TLB:TLB_shootdowns
> 1041 ? 11% +1120.7% 12707 ? 6% interrupts.CPU134.CAL:Function_call_interrupts
> 172.75 ? 27% +2867.6% 5126 ? 25% interrupts.CPU134.NMI:Non-maskable_interrupts
> 172.75 ? 27% +2867.6% 5126 ? 25% interrupts.CPU134.PMI:Performance_monitoring_interrupts
> 1147 ? 32% -70.3% 340.50 ? 22% interrupts.CPU134.RES:Rescheduling_interrupts
> 8.50 ? 37% +8305.9% 714.50 ? 23% interrupts.CPU134.TLB:TLB_shootdowns
> 1169 ? 8% +1200.7% 15208 ? 16% interrupts.CPU135.CAL:Function_call_interrupts
> 171.50 ? 27% +2944.2% 5220 ? 35% interrupts.CPU135.NMI:Non-maskable_interrupts
> 171.50 ? 27% +2944.2% 5220 ? 35% interrupts.CPU135.PMI:Performance_monitoring_interrupts
> 1110 ? 16% -68.3% 351.50 ? 21% interrupts.CPU135.RES:Rescheduling_interrupts
> 12.00 ? 33% +5641.7% 689.00 ? 21% interrupts.CPU135.TLB:TLB_shootdowns
> 1118 ? 7% +1085.7% 13265 ? 12% interrupts.CPU136.CAL:Function_call_interrupts
> 174.25 ? 27% +2698.6% 4876 ? 24% interrupts.CPU136.NMI:Non-maskable_interrupts
> 174.25 ? 27% +2698.6% 4876 ? 24% interrupts.CPU136.PMI:Performance_monitoring_interrupts
> 1392 ? 7% -79.5% 285.25 ? 12% interrupts.CPU136.RES:Rescheduling_interrupts
> 58.25 ?108% +987.1% 633.25 ? 17% interrupts.CPU136.TLB:TLB_shootdowns
> 1250 ? 29% +980.4% 13513 ? 14% interrupts.CPU137.CAL:Function_call_interrupts
> 199.25 ? 3% +2614.4% 5408 ? 24% interrupts.CPU137.NMI:Non-maskable_interrupts
> 199.25 ? 3% +2614.4% 5408 ? 24% interrupts.CPU137.PMI:Performance_monitoring_interrupts
> 1281 ? 10% -77.5% 288.75 ? 31% interrupts.CPU137.RES:Rescheduling_interrupts
> 21.75 ? 76% +2344.8% 531.75 ? 15% interrupts.CPU137.TLB:TLB_shootdowns
> 1161 ? 9% +1142.5% 14431 ? 10% interrupts.CPU138.CAL:Function_call_interrupts
> 204.75 ? 11% +2804.8% 5947 ? 19% interrupts.CPU138.NMI:Non-maskable_interrupts
> 204.75 ? 11% +2804.8% 5947 ? 19% interrupts.CPU138.PMI:Performance_monitoring_interrupts
> 1324 ? 19% -75.6% 322.75 ? 23% interrupts.CPU138.RES:Rescheduling_interrupts
> 12.25 ? 21% +5142.9% 642.25 ? 11% interrupts.CPU138.TLB:TLB_shootdowns
> 1089 ? 11% +1073.7% 12781 ? 9% interrupts.CPU139.CAL:Function_call_interrupts
> 174.00 ? 24% +3120.3% 5603 ? 31% interrupts.CPU139.NMI:Non-maskable_interrupts
> 174.00 ? 24% +3120.3% 5603 ? 31% interrupts.CPU139.PMI:Performance_monitoring_interrupts
> 1434 ? 19% -82.0% 258.00 ? 8% interrupts.CPU139.RES:Rescheduling_interrupts
> 30.75 ?119% +2056.9% 663.25 ? 14% interrupts.CPU139.TLB:TLB_shootdowns
> 1220 ? 14% +1438.2% 18769 ? 8% interrupts.CPU14.CAL:Function_call_interrupts
> 206.75 ? 4% +2813.7% 6024 ? 25% interrupts.CPU14.NMI:Non-maskable_interrupts
> 206.75 ? 4% +2813.7% 6024 ? 25% interrupts.CPU14.PMI:Performance_monitoring_interrupts
> 1255 ? 29% -67.5% 408.00 ? 23% interrupts.CPU14.RES:Rescheduling_interrupts
> 11.25 ? 22% +7417.8% 845.75 ? 11% interrupts.CPU14.TLB:TLB_shootdowns
> 1172 ? 9% +1141.5% 14556 ? 10% interrupts.CPU140.CAL:Function_call_interrupts
> 172.00 ? 23% +2840.4% 5057 ? 26% interrupts.CPU140.NMI:Non-maskable_interrupts
> 172.00 ? 23% +2840.4% 5057 ? 26% interrupts.CPU140.PMI:Performance_monitoring_interrupts
> 1538 ? 33% -77.5% 346.00 ? 19% interrupts.CPU140.RES:Rescheduling_interrupts
> 13.00 ? 56% +4734.6% 628.50 ? 25% interrupts.CPU140.TLB:TLB_shootdowns
> 1131 ? 11% +1224.3% 14981 ? 4% interrupts.CPU141.CAL:Function_call_interrupts
> 174.00 ? 25% +3593.5% 6426 ? 16% interrupts.CPU141.NMI:Non-maskable_interrupts
> 174.00 ? 25% +3593.5% 6426 ? 16% interrupts.CPU141.PMI:Performance_monitoring_interrupts
> 1099 ? 4% -73.7% 289.00 ? 12% interrupts.CPU141.RES:Rescheduling_interrupts
> 10.25 ? 17% +6936.6% 721.25 ? 21% interrupts.CPU141.TLB:TLB_shootdowns
> 1101 ? 13% +1300.5% 15422 ? 16% interrupts.CPU142.CAL:Function_call_interrupts
> 171.25 ? 23% +2942.3% 5210 ? 19% interrupts.CPU142.NMI:Non-maskable_interrupts
> 171.25 ? 23% +2942.3% 5210 ? 19% interrupts.CPU142.PMI:Performance_monitoring_interrupts
> 1238 ? 3% -75.2% 307.25 ? 19% interrupts.CPU142.RES:Rescheduling_interrupts
> 13.75 ? 17% +5021.8% 704.25 ? 21% interrupts.CPU142.TLB:TLB_shootdowns
> 1014 ? 12% +1164.9% 12826 ? 14% interrupts.CPU143.CAL:Function_call_interrupts
> 171.50 ? 24% +3326.4% 5876 ? 10% interrupts.CPU143.NMI:Non-maskable_interrupts
> 171.50 ? 24% +3326.4% 5876 ? 10% interrupts.CPU143.PMI:Performance_monitoring_interrupts
> 1299 ? 32% -75.4% 319.00 ? 21% interrupts.CPU143.RES:Rescheduling_interrupts
> 10.00 ? 37% +7027.5% 712.75 ? 15% interrupts.CPU143.TLB:TLB_shootdowns
> 1149 ? 4% +1225.5% 15239 ? 11% interrupts.CPU144.CAL:Function_call_interrupts
> 199.00 ? 31% +2607.5% 5388 ? 38% interrupts.CPU144.NMI:Non-maskable_interrupts
> 199.00 ? 31% +2607.5% 5388 ? 38% interrupts.CPU144.PMI:Performance_monitoring_interrupts
> 901.00 ? 17% -67.5% 292.75 ? 7% interrupts.CPU144.RES:Rescheduling_interrupts
> 29.75 ? 64% +2062.2% 643.25 ? 18% interrupts.CPU144.TLB:TLB_shootdowns
> 1240 ? 5% +1016.7% 13849 ? 6% interrupts.CPU145.CAL:Function_call_interrupts
> 170.25 ? 25% +3281.4% 5756 ? 7% interrupts.CPU145.NMI:Non-maskable_interrupts
> 170.25 ? 25% +3281.4% 5756 ? 7% interrupts.CPU145.PMI:Performance_monitoring_interrupts
> 1430 ? 17% -81.6% 263.75 ? 22% interrupts.CPU145.RES:Rescheduling_interrupts
> 14.75 ? 10% +4771.2% 718.50 ? 16% interrupts.CPU145.TLB:TLB_shootdowns
> 1360 ? 33% +901.6% 13626 ? 13% interrupts.CPU146.CAL:Function_call_interrupts
> 196.75 ? 2% +2932.3% 5966 ? 27% interrupts.CPU146.NMI:Non-maskable_interrupts
> 196.75 ? 2% +2932.3% 5966 ? 27% interrupts.CPU146.PMI:Performance_monitoring_interrupts
> 1173 ? 14% -77.6% 263.00 ? 21% interrupts.CPU146.RES:Rescheduling_interrupts
> 19.00 ? 82% +2542.1% 502.00 ? 35% interrupts.CPU146.TLB:TLB_shootdowns
> 1161 ? 27% +1038.3% 13224 ? 21% interrupts.CPU147.CAL:Function_call_interrupts
> 197.25 ? 2% +2361.7% 4855 ? 39% interrupts.CPU147.NMI:Non-maskable_interrupts
> 197.25 ? 2% +2361.7% 4855 ? 39% interrupts.CPU147.PMI:Performance_monitoring_interrupts
> 1253 ? 27% -76.8% 291.25 ? 23% interrupts.CPU147.RES:Rescheduling_interrupts
> 12.25 ? 63% +4467.3% 559.50 ? 29% interrupts.CPU147.TLB:TLB_shootdowns
> 1127 ? 13% +1099.5% 13527 ? 8% interrupts.CPU148.CAL:Function_call_interrupts
> 200.75 ? 3% +2706.2% 5633 ? 37% interrupts.CPU148.NMI:Non-maskable_interrupts
> 200.75 ? 3% +2706.2% 5633 ? 37% interrupts.CPU148.PMI:Performance_monitoring_interrupts
> 1375 ? 15% -79.6% 280.25 ? 12% interrupts.CPU148.RES:Rescheduling_interrupts
> 16.50 ? 29% +3775.8% 639.50 ? 10% interrupts.CPU148.TLB:TLB_shootdowns
> 1132 ? 13% +1194.9% 14664 ? 19% interrupts.CPU149.CAL:Function_call_interrupts
> 197.50 ? 2% +2371.9% 4882 ? 33% interrupts.CPU149.NMI:Non-maskable_interrupts
> 197.50 ? 2% +2371.9% 4882 ? 33% interrupts.CPU149.PMI:Performance_monitoring_interrupts
> 1546 ? 22% -81.7% 283.50 ? 12% interrupts.CPU149.RES:Rescheduling_interrupts
> 34.75 ? 42% +1830.2% 670.75 ? 17% interrupts.CPU149.TLB:TLB_shootdowns
> 1273 ? 32% +1440.2% 19606 ? 13% interrupts.CPU15.CAL:Function_call_interrupts
> 201.00 ? 2% +2902.0% 6034 ? 26% interrupts.CPU15.NMI:Non-maskable_interrupts
> 201.00 ? 2% +2902.0% 6034 ? 26% interrupts.CPU15.PMI:Performance_monitoring_interrupts
> 7.25 ? 54% +11358.6% 830.75 ? 21% interrupts.CPU15.TLB:TLB_shootdowns
> 1135 ? 14% +1192.5% 14676 ? 21% interrupts.CPU150.CAL:Function_call_interrupts
> 172.25 ? 25% +2381.3% 4274 ? 34% interrupts.CPU150.NMI:Non-maskable_interrupts
> 172.25 ? 25% +2381.3% 4274 ? 34% interrupts.CPU150.PMI:Performance_monitoring_interrupts
> 1234 ? 23% -76.3% 292.00 ? 14% interrupts.CPU150.RES:Rescheduling_interrupts
> 12.25 ? 38% +5240.8% 654.25 ? 23% interrupts.CPU150.TLB:TLB_shootdowns
> 1328 ? 22% +941.4% 13832 ? 15% interrupts.CPU151.CAL:Function_call_interrupts
> 171.25 ? 25% +2801.0% 4968 ? 33% interrupts.CPU151.NMI:Non-maskable_interrupts
> 171.25 ? 25% +2801.0% 4968 ? 33% interrupts.CPU151.PMI:Performance_monitoring_interrupts
> 1492 ? 11% -82.1% 266.75 ? 15% interrupts.CPU151.RES:Rescheduling_interrupts
> 18.25 ? 35% +2826.0% 534.00 ? 30% interrupts.CPU151.TLB:TLB_shootdowns
> 1203 ? 6% +1112.1% 14588 ? 18% interrupts.CPU152.CAL:Function_call_interrupts
> 171.50 ? 25% +2989.7% 5298 ? 26% interrupts.CPU152.NMI:Non-maskable_interrupts
> 171.50 ? 25% +2989.7% 5298 ? 26% interrupts.CPU152.PMI:Performance_monitoring_interrupts
> 1402 ? 26% -79.6% 285.50 ? 12% interrupts.CPU152.RES:Rescheduling_interrupts
> 12.50 ? 47% +4642.0% 592.75 ? 15% interrupts.CPU152.TLB:TLB_shootdowns
> 1075 ? 5% +1003.9% 11870 ? 26% interrupts.CPU153.CAL:Function_call_interrupts
> 178.50 ? 27% +3121.7% 5750 ? 17% interrupts.CPU153.NMI:Non-maskable_interrupts
> 178.50 ? 27% +3121.7% 5750 ? 17% interrupts.CPU153.PMI:Performance_monitoring_interrupts
> 1309 ? 26% -79.7% 265.25 ? 24% interrupts.CPU153.RES:Rescheduling_interrupts
> 33.00 ?119% +1569.7% 551.00 ? 32% interrupts.CPU153.TLB:TLB_shootdowns
> 1275 ? 18% +953.4% 13433 ? 13% interrupts.CPU154.CAL:Function_call_interrupts
> 174.25 ? 25% +3646.2% 6527 ? 22% interrupts.CPU154.NMI:Non-maskable_interrupts
> 174.25 ? 25% +3646.2% 6527 ? 22% interrupts.CPU154.PMI:Performance_monitoring_interrupts
> 1363 ? 24% -79.2% 283.75 ? 12% interrupts.CPU154.RES:Rescheduling_interrupts
> 19.75 ? 27% +2900.0% 592.50 ? 15% interrupts.CPU154.TLB:TLB_shootdowns
> 1179 ? 10% +1137.7% 14592 ? 20% interrupts.CPU155.CAL:Function_call_interrupts
> 172.50 ? 25% +4136.8% 7308 ? 15% interrupts.CPU155.NMI:Non-maskable_interrupts
> 172.50 ? 25% +4136.8% 7308 ? 15% interrupts.CPU155.PMI:Performance_monitoring_interrupts
> 1368 ? 17% -78.3% 297.25 ? 18% interrupts.CPU155.RES:Rescheduling_interrupts
> 17.00 ? 39% +3847.1% 671.00 ? 8% interrupts.CPU155.TLB:TLB_shootdowns
> 1130 ? 15% +1116.7% 13754 ? 8% interrupts.CPU156.CAL:Function_call_interrupts
> 171.00 ? 23% +3322.1% 5851 ? 16% interrupts.CPU156.NMI:Non-maskable_interrupts
> 171.00 ? 23% +3322.1% 5851 ? 16% interrupts.CPU156.PMI:Performance_monitoring_interrupts
> 1229 ? 24% -78.5% 264.25 ? 18% interrupts.CPU156.RES:Rescheduling_interrupts
> 19.00 ? 69% +3373.7% 660.00 ? 10% interrupts.CPU156.TLB:TLB_shootdowns
> 1025 ? 7% +1280.2% 14154 ? 26% interrupts.CPU157.CAL:Function_call_interrupts
> 170.50 ? 25% +3323.3% 5836 ? 12% interrupts.CPU157.NMI:Non-maskable_interrupts
> 170.50 ? 25% +3323.3% 5836 ? 12% interrupts.CPU157.PMI:Performance_monitoring_interrupts
> 1303 ? 12% -79.6% 265.50 ? 27% interrupts.CPU157.RES:Rescheduling_interrupts
> 20.50 ?100% +2854.9% 605.75 ? 5% interrupts.CPU157.TLB:TLB_shootdowns
> 1248 ? 13% +1136.5% 15432 ? 6% interrupts.CPU158.CAL:Function_call_interrupts
> 146.25 ? 35% +3346.5% 5040 ? 26% interrupts.CPU158.NMI:Non-maskable_interrupts
> 146.25 ? 35% +3346.5% 5040 ? 26% interrupts.CPU158.PMI:Performance_monitoring_interrupts
> 1532 ? 23% -78.8% 324.75 ? 18% interrupts.CPU158.RES:Rescheduling_interrupts
> 15.00 ? 37% +4298.3% 659.75 ? 34% interrupts.CPU158.TLB:TLB_shootdowns
> 1242 ? 7% +942.0% 12942 ? 12% interrupts.CPU159.CAL:Function_call_interrupts
> 148.50 ? 36% +3184.8% 4878 ? 21% interrupts.CPU159.NMI:Non-maskable_interrupts
> 148.50 ? 36% +3184.8% 4878 ? 21% interrupts.CPU159.PMI:Performance_monitoring_interrupts
> 1268 ? 21% -79.8% 256.75 ? 20% interrupts.CPU159.RES:Rescheduling_interrupts
> 32.50 ? 90% +1569.2% 542.50 ? 16% interrupts.CPU159.TLB:TLB_shootdowns
> 1106 ? 8% +1676.8% 19659 ? 12% interrupts.CPU16.CAL:Function_call_interrupts
> 205.75 ? 3% +2565.1% 5483 ? 36% interrupts.CPU16.NMI:Non-maskable_interrupts
> 205.75 ? 3% +2565.1% 5483 ? 36% interrupts.CPU16.PMI:Performance_monitoring_interrupts
> 1281 ? 30% -67.6% 414.50 ? 13% interrupts.CPU16.RES:Rescheduling_interrupts
> 21.50 ? 55% +4008.1% 883.25 ? 6% interrupts.CPU16.TLB:TLB_shootdowns
> 1168 ? 12% +1036.9% 13284 ? 14% interrupts.CPU160.CAL:Function_call_interrupts
> 144.25 ? 34% +4039.7% 5971 ? 30% interrupts.CPU160.NMI:Non-maskable_interrupts
> 144.25 ? 34% +4039.7% 5971 ? 30% interrupts.CPU160.PMI:Performance_monitoring_interrupts
> 1048 ? 27% -76.3% 249.00 ? 16% interrupts.CPU160.RES:Rescheduling_interrupts
> 11.00 ? 38% +6231.8% 696.50 ? 11% interrupts.CPU160.TLB:TLB_shootdowns
> 1106 ? 14% +1200.8% 14396 ? 18% interrupts.CPU161.CAL:Function_call_interrupts
> 147.25 ? 33% +3577.9% 5415 ? 14% interrupts.CPU161.NMI:Non-maskable_interrupts
> 147.25 ? 33% +3577.9% 5415 ? 14% interrupts.CPU161.PMI:Performance_monitoring_interrupts
> 1495 ? 15% -80.4% 293.25 ? 17% interrupts.CPU161.RES:Rescheduling_interrupts
> 18.25 ? 5% +3780.8% 708.25 ? 8% interrupts.CPU161.TLB:TLB_shootdowns
> 1217 ? 23% +998.1% 13369 ? 14% interrupts.CPU162.CAL:Function_call_interrupts
> 153.25 ? 33% +3249.8% 5133 ? 37% interrupts.CPU162.NMI:Non-maskable_interrupts
> 153.25 ? 33% +3249.8% 5133 ? 37% interrupts.CPU162.PMI:Performance_monitoring_interrupts
> 1321 ? 29% -81.0% 251.50 ? 14% interrupts.CPU162.RES:Rescheduling_interrupts
> 30.50 ? 62% +1946.7% 624.25 ? 8% interrupts.CPU162.TLB:TLB_shootdowns
> 1071 ? 4% +1210.4% 14044 ? 12% interrupts.CPU163.CAL:Function_call_interrupts
> 146.50 ? 34% +3657.0% 5504 ? 31% interrupts.CPU163.NMI:Non-maskable_interrupts
> 146.50 ? 34% +3657.0% 5504 ? 31% interrupts.CPU163.PMI:Performance_monitoring_interrupts
> 1582 ? 14% -82.1% 283.00 ? 11% interrupts.CPU163.RES:Rescheduling_interrupts
> 11.75 ? 21% +5280.9% 632.25 ? 14% interrupts.CPU163.TLB:TLB_shootdowns
> 1089 ? 10% +1103.2% 13105 ? 10% interrupts.CPU164.CAL:Function_call_interrupts
> 147.75 ? 36% +3304.1% 5029 ? 51% interrupts.CPU164.NMI:Non-maskable_interrupts
> 147.75 ? 36% +3304.1% 5029 ? 51% interrupts.CPU164.PMI:Performance_monitoring_interrupts
> 1245 ? 28% -79.0% 261.25 ? 13% interrupts.CPU164.RES:Rescheduling_interrupts
> 24.00 ? 80% +2418.8% 604.50 ? 14% interrupts.CPU164.TLB:TLB_shootdowns
> 1067 ? 6% +1151.9% 13361 ? 13% interrupts.CPU165.CAL:Function_call_interrupts
> 147.50 ? 34% +4051.4% 6123 ? 28% interrupts.CPU165.NMI:Non-maskable_interrupts
> 147.50 ? 34% +4051.4% 6123 ? 28% interrupts.CPU165.PMI:Performance_monitoring_interrupts
> 1398 ? 30% -79.5% 286.25 ? 18% interrupts.CPU165.RES:Rescheduling_interrupts
> 22.00 ? 73% +2510.2% 574.25 ? 24% interrupts.CPU165.TLB:TLB_shootdowns
> 1009 ? 8% +1390.8% 15053 ? 11% interrupts.CPU166.CAL:Function_call_interrupts
> 152.00 ? 36% +3428.5% 5363 ? 26% interrupts.CPU166.NMI:Non-maskable_interrupts
> 152.00 ? 36% +3428.5% 5363 ? 26% interrupts.CPU166.PMI:Performance_monitoring_interrupts
> 1299 ? 12% -76.7% 302.75 ? 12% interrupts.CPU166.RES:Rescheduling_interrupts
> 9.50 ? 39% +6428.9% 620.25 ? 13% interrupts.CPU166.TLB:TLB_shootdowns
> 1049 ? 12% +1315.0% 14850 ? 8% interrupts.CPU167.CAL:Function_call_interrupts
> 157.00 ? 40% +3769.9% 6075 ? 41% interrupts.CPU167.NMI:Non-maskable_interrupts
> 157.00 ? 40% +3769.9% 6075 ? 41% interrupts.CPU167.PMI:Performance_monitoring_interrupts
> 1629 ? 6% -81.5% 302.00 ? 19% interrupts.CPU167.RES:Rescheduling_interrupts
> 14.50 ? 25% +4098.3% 608.75 ? 20% interrupts.CPU167.TLB:TLB_shootdowns
> 1799 ? 40% +813.1% 16428 ? 3% interrupts.CPU168.CAL:Function_call_interrupts
> 181.00 ? 24% +3204.0% 5980 ? 34% interrupts.CPU168.NMI:Non-maskable_interrupts
> 181.00 ? 24% +3204.0% 5980 ? 34% interrupts.CPU168.PMI:Performance_monitoring_interrupts
> 987.50 ? 10% -70.2% 294.50 ? 12% interrupts.CPU168.RES:Rescheduling_interrupts
> 20.75 ? 81% +3185.5% 681.75 ? 19% interrupts.CPU168.TLB:TLB_shootdowns
> 1475 ? 21% +927.6% 15162 ? 10% interrupts.CPU169.CAL:Function_call_interrupts
> 178.25 ? 22% +3114.6% 5730 ? 22% interrupts.CPU169.NMI:Non-maskable_interrupts
> 178.25 ? 22% +3114.6% 5730 ? 22% interrupts.CPU169.PMI:Performance_monitoring_interrupts
> 1203 ? 19% -75.2% 298.00 ? 19% interrupts.CPU169.RES:Rescheduling_interrupts
> 46.00 ? 95% +1379.9% 680.75 ? 19% interrupts.CPU169.TLB:TLB_shootdowns
> 1073 ? 12% +1504.9% 17232 ? 11% interrupts.CPU17.CAL:Function_call_interrupts
> 203.75 ? 2% +1949.0% 4174 ? 15% interrupts.CPU17.NMI:Non-maskable_interrupts
> 203.75 ? 2% +1949.0% 4174 ? 15% interrupts.CPU17.PMI:Performance_monitoring_interrupts
> 1024 ? 30% -66.9% 339.25 ? 16% interrupts.CPU17.RES:Rescheduling_interrupts
> 16.75 ?105% +4129.9% 708.50 ? 12% interrupts.CPU17.TLB:TLB_shootdowns
> 1363 ? 13% +986.6% 14813 ? 10% interrupts.CPU170.CAL:Function_call_interrupts
> 175.50 ? 26% +3093.4% 5604 ? 28% interrupts.CPU170.NMI:Non-maskable_interrupts
> 175.50 ? 26% +3093.4% 5604 ? 28% interrupts.CPU170.PMI:Performance_monitoring_interrupts
> 1143 ? 37% -76.2% 272.75 ? 21% interrupts.CPU170.RES:Rescheduling_interrupts
> 23.25 ?111% +2730.1% 658.00 ? 8% interrupts.CPU170.TLB:TLB_shootdowns
> 1430 ? 9% +994.8% 15658 ? 9% interrupts.CPU171.CAL:Function_call_interrupts
> 188.50 ? 30% +2864.6% 5588 ? 24% interrupts.CPU171.NMI:Non-maskable_interrupts
> 188.50 ? 30% +2864.6% 5588 ? 24% interrupts.CPU171.PMI:Performance_monitoring_interrupts
> 1123 ? 33% -76.3% 266.25 ? 12% interrupts.CPU171.RES:Rescheduling_interrupts
> 10.25 ? 24% +6714.6% 698.50 ? 12% interrupts.CPU171.TLB:TLB_shootdowns
> 1565 ? 49% +811.5% 14268 ? 15% interrupts.CPU172.CAL:Function_call_interrupts
> 208.25 ? 5% +2480.3% 5373 ? 31% interrupts.CPU172.NMI:Non-maskable_interrupts
> 208.25 ? 5% +2480.3% 5373 ? 31% interrupts.CPU172.PMI:Performance_monitoring_interrupts
> 35.25 ?128% +1641.1% 613.75 ? 11% interrupts.CPU172.TLB:TLB_shootdowns
> 1228 ? 15% +1209.1% 16085 ? 12% interrupts.CPU173.CAL:Function_call_interrupts
> 203.00 ? 4% +3154.1% 6605 ? 13% interrupts.CPU173.NMI:Non-maskable_interrupts
> 203.00 ? 4% +3154.1% 6605 ? 13% interrupts.CPU173.PMI:Performance_monitoring_interrupts
> 1248 ? 26% -78.0% 274.50 ? 21% interrupts.CPU173.RES:Rescheduling_interrupts
> 33.50 ? 62% +1862.7% 657.50 ? 27% interrupts.CPU173.TLB:TLB_shootdowns
> 1070 ? 8% +1239.8% 14338 ? 11% interrupts.CPU174.CAL:Function_call_interrupts
> 203.50 ? 4% +2676.9% 5651 ? 18% interrupts.CPU174.NMI:Non-maskable_interrupts
> 203.50 ? 4% +2676.9% 5651 ? 18% interrupts.CPU174.PMI:Performance_monitoring_interrupts
> 19.50 ? 70% +3064.1% 617.00 ? 9% interrupts.CPU174.TLB:TLB_shootdowns
> 1468 ? 19% +1011.8% 16329 ? 3% interrupts.CPU175.CAL:Function_call_interrupts
> 200.50 ? 3% +2875.7% 5966 ? 16% interrupts.CPU175.NMI:Non-maskable_interrupts
> 200.50 ? 3% +2875.7% 5966 ? 16% interrupts.CPU175.PMI:Performance_monitoring_interrupts
> 1106 ? 31% -75.1% 275.25 ? 7% interrupts.CPU175.RES:Rescheduling_interrupts
> 28.75 ? 60% +2479.1% 741.50 ? 14% interrupts.CPU175.TLB:TLB_shootdowns
> 1393 ? 8% +876.9% 13611 ? 8% interrupts.CPU176.CAL:Function_call_interrupts
> 200.00 ? 3% +3190.5% 6581 ? 13% interrupts.CPU176.NMI:Non-maskable_interrupts
> 200.00 ? 3% +3190.5% 6581 ? 13% interrupts.CPU176.PMI:Performance_monitoring_interrupts
> 971.50 ? 11% -71.1% 280.75 ? 7% interrupts.CPU176.RES:Rescheduling_interrupts
> 14.00 ? 49% +5275.0% 752.50 ? 6% interrupts.CPU176.TLB:TLB_shootdowns
> 1394 ? 12% +803.9% 12600 ? 6% interrupts.CPU177.CAL:Function_call_interrupts
> 202.25 ? 2% +2877.6% 6022 ? 13% interrupts.CPU177.NMI:Non-maskable_interrupts
> 202.25 ? 2% +2877.6% 6022 ? 13% interrupts.CPU177.PMI:Performance_monitoring_interrupts
> 1117 ? 18% -76.0% 268.00 ? 21% interrupts.CPU177.RES:Rescheduling_interrupts
> 14.00 ? 26% +4117.9% 590.50 ? 14% interrupts.CPU177.TLB:TLB_shootdowns
> 1430 ? 15% +968.2% 15281 ? 4% interrupts.CPU178.CAL:Function_call_interrupts
> 207.25 ? 9% +2723.2% 5851 ? 21% interrupts.CPU178.NMI:Non-maskable_interrupts
> 207.25 ? 9% +2723.2% 5851 ? 21% interrupts.CPU178.PMI:Performance_monitoring_interrupts
> 916.50 ? 21% -74.1% 237.25 ? 6% interrupts.CPU178.RES:Rescheduling_interrupts
> 20.50 ? 69% +2967.1% 628.75 ? 12% interrupts.CPU178.TLB:TLB_shootdowns
> 1350 ? 12% +866.1% 13042 ? 16% interrupts.CPU179.CAL:Function_call_interrupts
> 203.25 ? 3% +2323.0% 4924 ? 28% interrupts.CPU179.NMI:Non-maskable_interrupts
> 203.25 ? 3% +2323.0% 4924 ? 28% interrupts.CPU179.PMI:Performance_monitoring_interrupts
> 1495 ? 14% -83.6% 245.75 ? 12% interrupts.CPU179.RES:Rescheduling_interrupts
> 23.75 ? 76% +2553.7% 630.25 ? 17% interrupts.CPU179.TLB:TLB_shootdowns
> 1155 ? 9% +1583.0% 19451 ? 15% interrupts.CPU18.CAL:Function_call_interrupts
> 202.50 ? 3% +2122.7% 4501 ? 44% interrupts.CPU18.NMI:Non-maskable_interrupts
> 202.50 ? 3% +2122.7% 4501 ? 44% interrupts.CPU18.PMI:Performance_monitoring_interrupts
> 1107 ? 20% -67.0% 365.50 ? 24% interrupts.CPU18.RES:Rescheduling_interrupts
> 5.50 ? 32% +14081.8% 780.00 ? 15% interrupts.CPU18.TLB:TLB_shootdowns
> 1327 ? 4% +1054.4% 15324 ? 6% interrupts.CPU180.CAL:Function_call_interrupts
> 179.25 ? 26% +3402.6% 6278 ? 12% interrupts.CPU180.NMI:Non-maskable_interrupts
> 179.25 ? 26% +3402.6% 6278 ? 12% interrupts.CPU180.PMI:Performance_monitoring_interrupts
> 1134 ? 19% -78.7% 241.25 ? 21% interrupts.CPU180.RES:Rescheduling_interrupts
> 19.75 ? 59% +3427.8% 696.75 ? 11% interrupts.CPU180.TLB:TLB_shootdowns
> 1279 ? 5% +947.7% 13403 ? 8% interrupts.CPU181.CAL:Function_call_interrupts
> 184.50 ? 25% +3133.9% 5966 ? 15% interrupts.CPU181.NMI:Non-maskable_interrupts
> 184.50 ? 25% +3133.9% 5966 ? 15% interrupts.CPU181.PMI:Performance_monitoring_interrupts
> 1399 ? 8% -83.7% 228.75 ? 21% interrupts.CPU181.RES:Rescheduling_interrupts
> 17.25 ? 49% +3360.9% 597.00 ? 14% interrupts.CPU181.TLB:TLB_shootdowns
> 1176 ? 11% +971.9% 12610 ? 12% interrupts.CPU182.CAL:Function_call_interrupts
> 179.75 ? 27% +2885.3% 5366 ? 25% interrupts.CPU182.NMI:Non-maskable_interrupts
> 179.75 ? 27% +2885.3% 5366 ? 25% interrupts.CPU182.PMI:Performance_monitoring_interrupts
> 813.25 ? 28% -70.0% 244.25 ? 24% interrupts.CPU182.RES:Rescheduling_interrupts
> 9.25 ? 55% +7583.8% 710.75 ? 18% interrupts.CPU182.TLB:TLB_shootdowns
> 1458 ? 24% +788.6% 12958 ? 14% interrupts.CPU183.CAL:Function_call_interrupts
> 177.00 ? 27% +3059.6% 5592 ? 8% interrupts.CPU183.NMI:Non-maskable_interrupts
> 177.00 ? 27% +3059.6% 5592 ? 8% interrupts.CPU183.PMI:Performance_monitoring_interrupts
> 1085 ? 28% -78.8% 229.75 ? 12% interrupts.CPU183.RES:Rescheduling_interrupts
> 22.00 ? 71% +2365.9% 542.50 ? 23% interrupts.CPU183.TLB:TLB_shootdowns
> 1143 ? 6% +1166.9% 14480 ? 6% interrupts.CPU184.CAL:Function_call_interrupts
> 178.00 ? 27% +3791.3% 6926 ? 14% interrupts.CPU184.NMI:Non-maskable_interrupts
> 178.00 ? 27% +3791.3% 6926 ? 14% interrupts.CPU184.PMI:Performance_monitoring_interrupts
> 1061 ? 17% -73.8% 278.25 ? 19% interrupts.CPU184.RES:Rescheduling_interrupts
> 11.00 ? 43% +5834.1% 652.75 ? 26% interrupts.CPU184.TLB:TLB_shootdowns
> 1274 ? 11% +969.3% 13628 ? 4% interrupts.CPU185.CAL:Function_call_interrupts
> 181.75 ? 27% +3123.4% 5858 ? 34% interrupts.CPU185.NMI:Non-maskable_interrupts
> 181.75 ? 27% +3123.4% 5858 ? 34% interrupts.CPU185.PMI:Performance_monitoring_interrupts
> 1332 ? 19% -82.9% 227.50 ? 5% interrupts.CPU185.RES:Rescheduling_interrupts
> 15.00 ? 30% +3793.3% 584.00 ? 10% interrupts.CPU185.TLB:TLB_shootdowns
> 1452 ? 24% +830.7% 13514 ? 13% interrupts.CPU186.CAL:Function_call_interrupts
> 175.75 ? 26% +2956.8% 5372 ? 35% interrupts.CPU186.NMI:Non-maskable_interrupts
> 175.75 ? 26% +2956.8% 5372 ? 35% interrupts.CPU186.PMI:Performance_monitoring_interrupts
> 1349 ? 18% -83.2% 226.75 ? 14% interrupts.CPU186.RES:Rescheduling_interrupts
> 21.50 ? 44% +2820.9% 628.00 ? 15% interrupts.CPU186.TLB:TLB_shootdowns
> 1241 ? 14% +969.0% 13269 ? 6% interrupts.CPU187.CAL:Function_call_interrupts
> 177.25 ? 26% +2643.6% 4863 ? 33% interrupts.CPU187.NMI:Non-maskable_interrupts
> 177.25 ? 26% +2643.6% 4863 ? 33% interrupts.CPU187.PMI:Performance_monitoring_interrupts
> 1196 ? 46% -79.0% 251.25 ? 23% interrupts.CPU187.RES:Rescheduling_interrupts
> 21.25 ? 42% +2525.9% 558.00 ? 23% interrupts.CPU187.TLB:TLB_shootdowns
> 1401 ? 16% +720.2% 11493 ? 12% interrupts.CPU188.CAL:Function_call_interrupts
> 182.50 ? 27% +2534.8% 4808 ? 31% interrupts.CPU188.NMI:Non-maskable_interrupts
> 182.50 ? 27% +2534.8% 4808 ? 31% interrupts.CPU188.PMI:Performance_monitoring_interrupts
> 1079 ? 23% -80.9% 206.50 ? 17% interrupts.CPU188.RES:Rescheduling_interrupts
> 13.75 ? 13% +3150.9% 447.00 ? 23% interrupts.CPU188.TLB:TLB_shootdowns
> 1352 ? 11% +881.9% 13277 ? 16% interrupts.CPU189.CAL:Function_call_interrupts
> 179.50 ? 26% +2773.8% 5158 ? 37% interrupts.CPU189.NMI:Non-maskable_interrupts
> 179.50 ? 26% +2773.8% 5158 ? 37% interrupts.CPU189.PMI:Performance_monitoring_interrupts
> 1375 ? 28% -82.5% 240.25 ? 17% interrupts.CPU189.RES:Rescheduling_interrupts
> 35.25 ? 48% +1895.7% 703.50 ? 19% interrupts.CPU189.TLB:TLB_shootdowns
> 1156 ? 4% +1495.6% 18453 ? 14% interrupts.CPU19.CAL:Function_call_interrupts
> 204.50 +2211.9% 4727 ? 23% interrupts.CPU19.NMI:Non-maskable_interrupts
> 204.50 +2211.9% 4727 ? 23% interrupts.CPU19.PMI:Performance_monitoring_interrupts
> 1344 ? 21% -69.5% 410.50 ? 15% interrupts.CPU19.RES:Rescheduling_interrupts
> 12.25 ? 15% +6300.0% 784.00 ? 7% interrupts.CPU19.TLB:TLB_shootdowns
> 1154 ? 6% +947.2% 12092 ? 11% interrupts.CPU190.CAL:Function_call_interrupts
> 176.75 ? 26% +2962.0% 5412 ? 25% interrupts.CPU190.NMI:Non-maskable_interrupts
> 176.75 ? 26% +2962.0% 5412 ? 25% interrupts.CPU190.PMI:Performance_monitoring_interrupts
> 1021 ? 17% -79.4% 210.25 ? 23% interrupts.CPU190.RES:Rescheduling_interrupts
> 18.00 ? 25% +2488.9% 466.00 ? 17% interrupts.CPU190.TLB:TLB_shootdowns
> 1060 ? 12% +354.3% 4815 ? 10% interrupts.CPU191.CAL:Function_call_interrupts
> 183.25 ? 25% +2279.8% 4361 ? 29% interrupts.CPU191.NMI:Non-maskable_interrupts
> 183.25 ? 25% +2279.8% 4361 ? 29% interrupts.CPU191.PMI:Performance_monitoring_interrupts
> 642.50 ? 26% -85.0% 96.50 ? 11% interrupts.CPU191.RES:Rescheduling_interrupts
> 1533 ? 33% +1016.4% 17117 ? 4% interrupts.CPU2.CAL:Function_call_interrupts
> 211.75 ? 8% +3240.4% 7073 ? 16% interrupts.CPU2.NMI:Non-maskable_interrupts
> 211.75 ? 8% +3240.4% 7073 ? 16% interrupts.CPU2.PMI:Performance_monitoring_interrupts
> 1065 ? 7% -64.9% 374.25 ? 10% interrupts.CPU2.RES:Rescheduling_interrupts
> 9.75 ? 48% +8120.5% 801.50 ? 17% interrupts.CPU2.TLB:TLB_shootdowns
> 1158 ? 9% +1451.8% 17977 ? 18% interrupts.CPU20.CAL:Function_call_interrupts
> 206.50 ? 4% +1652.7% 3619 ? 28% interrupts.CPU20.NMI:Non-maskable_interrupts
> 206.50 ? 4% +1652.7% 3619 ? 28% interrupts.CPU20.PMI:Performance_monitoring_interrupts
> 1099 ? 49% -65.9% 375.00 ? 16% interrupts.CPU20.RES:Rescheduling_interrupts
> 20.75 ? 86% +4350.6% 923.50 ? 7% interrupts.CPU20.TLB:TLB_shootdowns
> 4622 ?128% +288.3% 17949 ? 16% interrupts.CPU21.CAL:Function_call_interrupts
> 175.25 ? 24% +2803.9% 5089 ? 22% interrupts.CPU21.NMI:Non-maskable_interrupts
> 175.25 ? 24% +2803.9% 5089 ? 22% interrupts.CPU21.PMI:Performance_monitoring_interrupts
> 1170 ? 33% -65.3% 405.50 ? 16% interrupts.CPU21.RES:Rescheduling_interrupts
> 9.75 ? 8% +8459.0% 834.50 ? 23% interrupts.CPU21.TLB:TLB_shootdowns
> 1130 ? 8% +1540.3% 18539 ? 19% interrupts.CPU22.CAL:Function_call_interrupts
> 181.75 ? 26% +2270.2% 4307 ? 15% interrupts.CPU22.NMI:Non-maskable_interrupts
> 181.75 ? 26% +2270.2% 4307 ? 15% interrupts.CPU22.PMI:Performance_monitoring_interrupts
> 1330 ? 25% -70.1% 398.00 ? 20% interrupts.CPU22.RES:Rescheduling_interrupts
> 10.25 ? 54% +8246.3% 855.50 ? 15% interrupts.CPU22.TLB:TLB_shootdowns
> 1130 ? 12% +1531.3% 18446 ? 13% interrupts.CPU23.CAL:Function_call_interrupts
> 177.50 ? 23% +2556.3% 4715 ? 31% interrupts.CPU23.NMI:Non-maskable_interrupts
> 177.50 ? 23% +2556.3% 4715 ? 31% interrupts.CPU23.PMI:Performance_monitoring_interrupts
> 1240 ? 6% -68.6% 389.75 ? 15% interrupts.CPU23.RES:Rescheduling_interrupts
> 33.00 ? 84% +2371.2% 815.50 ? 6% interrupts.CPU23.TLB:TLB_shootdowns
> 1247 ? 11% +1214.2% 16394 ? 13% interrupts.CPU24.CAL:Function_call_interrupts
> 180.25 ? 27% +3176.8% 5906 ? 31% interrupts.CPU24.NMI:Non-maskable_interrupts
> 180.25 ? 27% +3176.8% 5906 ? 31% interrupts.CPU24.PMI:Performance_monitoring_interrupts
> 934.00 ? 6% -59.9% 374.25 ? 17% interrupts.CPU24.RES:Rescheduling_interrupts
> 12.50 ? 36% +6030.0% 766.25 ? 12% interrupts.CPU24.TLB:TLB_shootdowns
> 1104 ? 13% +1607.5% 18855 ? 21% interrupts.CPU25.CAL:Function_call_interrupts
> 199.75 ? 3% +2961.6% 6115 ? 38% interrupts.CPU25.NMI:Non-maskable_interrupts
> 199.75 ? 3% +2961.6% 6115 ? 38% interrupts.CPU25.PMI:Performance_monitoring_interrupts
> 1100 ? 41% -59.5% 445.25 ? 14% interrupts.CPU25.RES:Rescheduling_interrupts
> 9.00 ? 47% +9119.4% 829.75 ? 13% interrupts.CPU25.TLB:TLB_shootdowns
> 1463 ? 17% +1091.0% 17433 ? 4% interrupts.CPU26.CAL:Function_call_interrupts
> 199.25 ? 4% +2527.7% 5235 ? 37% interrupts.CPU26.NMI:Non-maskable_interrupts
> 199.25 ? 4% +2527.7% 5235 ? 37% interrupts.CPU26.PMI:Performance_monitoring_interrupts
> 1474 ? 18% -73.8% 386.25 ? 25% interrupts.CPU26.RES:Rescheduling_interrupts
> 11.75 ? 38% +6944.7% 827.75 ? 14% interrupts.CPU26.TLB:TLB_shootdowns
> 1148 ? 17% +1236.0% 15346 ? 6% interrupts.CPU27.CAL:Function_call_interrupts
> 204.50 ? 4% +2298.5% 4905 ? 40% interrupts.CPU27.NMI:Non-maskable_interrupts
> 204.50 ? 4% +2298.5% 4905 ? 40% interrupts.CPU27.PMI:Performance_monitoring_interrupts
> 1265 ? 31% -69.6% 384.50 ? 20% interrupts.CPU27.RES:Rescheduling_interrupts
> 24.00 ? 68% +3257.3% 805.75 ? 20% interrupts.CPU27.TLB:TLB_shootdowns
> 1024 ? 5% +1668.9% 18118 ? 7% interrupts.CPU28.CAL:Function_call_interrupts
> 198.25 ? 3% +3294.2% 6729 ? 16% interrupts.CPU28.NMI:Non-maskable_interrupts
> 198.25 ? 3% +3294.2% 6729 ? 16% interrupts.CPU28.PMI:Performance_monitoring_interrupts
> 1301 ? 8% -69.6% 395.00 ? 6% interrupts.CPU28.RES:Rescheduling_interrupts
> 32.25 ? 61% +2263.6% 762.25 ? 11% interrupts.CPU28.TLB:TLB_shootdowns
> 1072 ? 7% +1619.3% 18435 ? 11% interrupts.CPU29.CAL:Function_call_interrupts
> 200.00 ? 3% +3180.0% 6560 ? 13% interrupts.CPU29.NMI:Non-maskable_interrupts
> 200.00 ? 3% +3180.0% 6560 ? 13% interrupts.CPU29.PMI:Performance_monitoring_interrupts
> 1335 ? 15% -68.9% 415.75 ? 11% interrupts.CPU29.RES:Rescheduling_interrupts
> 23.50 ? 56% +3357.4% 812.50 ? 7% interrupts.CPU29.TLB:TLB_shootdowns
> 1328 ? 33% +1287.1% 18424 ? 17% interrupts.CPU3.CAL:Function_call_interrupts
> 204.00 ? 3% +3221.1% 6775 ? 15% interrupts.CPU3.NMI:Non-maskable_interrupts
> 204.00 ? 3% +3221.1% 6775 ? 15% interrupts.CPU3.PMI:Performance_monitoring_interrupts
> 1256 ? 9% -70.6% 369.25 ? 13% interrupts.CPU3.RES:Rescheduling_interrupts
> 11.75 ? 46% +6491.5% 774.50 ? 10% interrupts.CPU3.TLB:TLB_shootdowns
> 1028 ? 3% +1513.6% 16588 ? 12% interrupts.CPU30.CAL:Function_call_interrupts
> 194.50 ? 2% +3653.2% 7300 ? 20% interrupts.CPU30.NMI:Non-maskable_interrupts
> 194.50 ? 2% +3653.2% 7300 ? 20% interrupts.CPU30.PMI:Performance_monitoring_interrupts
> 1336 ? 11% -67.4% 435.25 ? 23% interrupts.CPU30.RES:Rescheduling_interrupts
> 40.50 ?122% +1838.9% 785.25 ? 7% interrupts.CPU30.TLB:TLB_shootdowns
> 1140 ? 6% +1545.2% 18768 ? 8% interrupts.CPU31.CAL:Function_call_interrupts
> 200.25 ? 4% +2843.4% 5894 ? 32% interrupts.CPU31.NMI:Non-maskable_interrupts
> 200.25 ? 4% +2843.4% 5894 ? 32% interrupts.CPU31.PMI:Performance_monitoring_interrupts
> 1193 ? 27% -65.4% 413.25 ? 11% interrupts.CPU31.RES:Rescheduling_interrupts
> 34.50 ? 85% +2294.9% 826.25 ? 5% interrupts.CPU31.TLB:TLB_shootdowns
> 1205 ? 9% +1171.7% 15324 ? 11% interrupts.CPU32.CAL:Function_call_interrupts
> 197.75 ? 5% +3269.5% 6663 ? 14% interrupts.CPU32.NMI:Non-maskable_interrupts
> 197.75 ? 5% +3269.5% 6663 ? 14% interrupts.CPU32.PMI:Performance_monitoring_interrupts
> 1052 ? 15% -63.8% 380.75 ? 14% interrupts.CPU32.RES:Rescheduling_interrupts
> 7.50 ? 39% +11026.7% 834.50 ? 17% interrupts.CPU32.TLB:TLB_shootdowns
> 1199 ? 15% +1305.8% 16855 ? 6% interrupts.CPU33.CAL:Function_call_interrupts
> 176.00 ? 26% +3479.7% 6300 ? 32% interrupts.CPU33.NMI:Non-maskable_interrupts
> 176.00 ? 26% +3479.7% 6300 ? 32% interrupts.CPU33.PMI:Performance_monitoring_interrupts
> 1174 ? 39% -66.5% 393.25 ? 11% interrupts.CPU33.RES:Rescheduling_interrupts
> 21.75 ? 89% +3767.8% 841.25 ? 11% interrupts.CPU33.TLB:TLB_shootdowns
> 1103 ? 5% +1436.9% 16955 ? 11% interrupts.CPU34.CAL:Function_call_interrupts
> 170.75 ? 24% +3065.3% 5404 ? 23% interrupts.CPU34.NMI:Non-maskable_interrupts
> 170.75 ? 24% +3065.3% 5404 ? 23% interrupts.CPU34.PMI:Performance_monitoring_interrupts
> 1522 ? 14% -76.6% 355.75 ? 14% interrupts.CPU34.RES:Rescheduling_interrupts
> 41.50 ?119% +1819.3% 796.50 ? 4% interrupts.CPU34.TLB:TLB_shootdowns
> 1107 ? 12% +1574.7% 18539 ? 15% interrupts.CPU35.CAL:Function_call_interrupts
> 200.75 ? 6% +2967.6% 6158 ? 23% interrupts.CPU35.NMI:Non-maskable_interrupts
> 200.75 ? 6% +2967.6% 6158 ? 23% interrupts.CPU35.PMI:Performance_monitoring_interrupts
> 1132 ? 20% -63.5% 413.00 ? 8% interrupts.CPU35.RES:Rescheduling_interrupts
> 9.00 ? 30% +9308.3% 846.75 ? 16% interrupts.CPU35.TLB:TLB_shootdowns
> 1011 ? 5% +1621.5% 17417 ? 8% interrupts.CPU36.CAL:Function_call_interrupts
> 194.75 ? 3% +3467.9% 6948 ? 7% interrupts.CPU36.NMI:Non-maskable_interrupts
> 194.75 ? 3% +3467.9% 6948 ? 7% interrupts.CPU36.PMI:Performance_monitoring_interrupts
> 983.00 ? 32% -58.4% 409.25 ? 22% interrupts.CPU36.RES:Rescheduling_interrupts
> 21.75 ? 75% +3134.5% 703.50 ? 10% interrupts.CPU36.TLB:TLB_shootdowns
> 1136 ? 9% +1481.5% 17970 ? 15% interrupts.CPU37.CAL:Function_call_interrupts
> 192.75 ? 3% +3192.0% 6345 ? 12% interrupts.CPU37.NMI:Non-maskable_interrupts
> 192.75 ? 3% +3192.0% 6345 ? 12% interrupts.CPU37.PMI:Performance_monitoring_interrupts
> 1269 ? 40% -70.2% 379.00 ? 22% interrupts.CPU37.RES:Rescheduling_interrupts
> 16.50 ? 21% +5768.2% 968.25 ? 32% interrupts.CPU37.TLB:TLB_shootdowns
> 1168 ? 11% +1204.1% 15238 ? 13% interrupts.CPU38.CAL:Function_call_interrupts
> 195.25 ? 3% +3163.8% 6372 ? 13% interrupts.CPU38.NMI:Non-maskable_interrupts
> 195.25 ? 3% +3163.8% 6372 ? 13% interrupts.CPU38.PMI:Performance_monitoring_interrupts
> 1412 ? 26% -71.1% 408.00 ? 21% interrupts.CPU38.RES:Rescheduling_interrupts
> 27.00 ?104% +2736.1% 765.75 ? 20% interrupts.CPU38.TLB:TLB_shootdowns
> 1159 ? 2% +1433.1% 17769 ? 19% interrupts.CPU39.CAL:Function_call_interrupts
> 170.50 ? 26% +3568.5% 6254 ? 25% interrupts.CPU39.NMI:Non-maskable_interrupts
> 170.50 ? 26% +3568.5% 6254 ? 25% interrupts.CPU39.PMI:Performance_monitoring_interrupts
> 1395 ? 22% -72.4% 385.50 ? 17% interrupts.CPU39.RES:Rescheduling_interrupts
> 33.00 ? 79% +2272.0% 782.75 ? 16% interrupts.CPU39.TLB:TLB_shootdowns
> 1093 ? 8% +1469.4% 17154 ? 11% interrupts.CPU4.CAL:Function_call_interrupts
> 205.50 ? 3% +2958.8% 6285 ? 27% interrupts.CPU4.NMI:Non-maskable_interrupts
> 205.50 ? 3% +2958.8% 6285 ? 27% interrupts.CPU4.PMI:Performance_monitoring_interrupts
> 1344 ? 24% -73.6% 354.75 ? 11% interrupts.CPU4.RES:Rescheduling_interrupts
> 21.00 ? 71% +3188.1% 690.50 ? 14% interrupts.CPU4.TLB:TLB_shootdowns
> 1058 ? 9% +1555.4% 17518 ? 16% interrupts.CPU40.CAL:Function_call_interrupts
> 147.00 ? 34% +4442.0% 6676 ? 10% interrupts.CPU40.NMI:Non-maskable_interrupts
> 147.00 ? 34% +4442.0% 6676 ? 10% interrupts.CPU40.PMI:Performance_monitoring_interrupts
> 1115 ? 11% -63.5% 407.75 ? 18% interrupts.CPU40.RES:Rescheduling_interrupts
> 33.50 ? 74% +2391.8% 834.75 ? 16% interrupts.CPU40.TLB:TLB_shootdowns
> 1037 ? 9% +1763.6% 19334 ? 8% interrupts.CPU41.CAL:Function_call_interrupts
> 148.00 ? 34% +4094.8% 6208 ? 6% interrupts.CPU41.NMI:Non-maskable_interrupts
> 148.00 ? 34% +4094.8% 6208 ? 6% interrupts.CPU41.PMI:Performance_monitoring_interrupts
> 1286 ? 16% -69.1% 397.25 ? 11% interrupts.CPU41.RES:Rescheduling_interrupts
> 8.00 ? 23% +11725.0% 946.00 ? 6% interrupts.CPU41.TLB:TLB_shootdowns
> 1202 ? 7% +1323.9% 17118 ? 16% interrupts.CPU42.CAL:Function_call_interrupts
> 170.75 ? 25% +3722.4% 6526 ? 13% interrupts.CPU42.NMI:Non-maskable_interrupts
> 170.75 ? 25% +3722.4% 6526 ? 13% interrupts.CPU42.PMI:Performance_monitoring_interrupts
> 1201 ? 21% -68.9% 373.00 ? 8% interrupts.CPU42.RES:Rescheduling_interrupts
> 27.50 ?115% +2939.1% 835.75 ? 5% interrupts.CPU42.TLB:TLB_shootdowns
> 1114 ? 8% +1456.3% 17337 ? 15% interrupts.CPU43.CAL:Function_call_interrupts
> 151.00 ? 37% +4446.0% 6864 ? 11% interrupts.CPU43.NMI:Non-maskable_interrupts
> 151.00 ? 37% +4446.0% 6864 ? 11% interrupts.CPU43.PMI:Performance_monitoring_interrupts
> 1182 ? 22% -65.1% 412.50 ? 12% interrupts.CPU43.RES:Rescheduling_interrupts
> 20.50 ? 72% +3782.9% 796.00 ? 7% interrupts.CPU43.TLB:TLB_shootdowns
> 1072 ? 13% +1418.1% 16273 ? 19% interrupts.CPU44.CAL:Function_call_interrupts
> 147.00 ? 33% +4449.7% 6688 ? 13% interrupts.CPU44.NMI:Non-maskable_interrupts
> 147.00 ? 33% +4449.7% 6688 ? 13% interrupts.CPU44.PMI:Performance_monitoring_interrupts
> 1036 ? 47% -62.0% 393.75 ? 23% interrupts.CPU44.RES:Rescheduling_interrupts
> 19.00 ? 78% +4377.6% 850.75 ? 16% interrupts.CPU44.TLB:TLB_shootdowns
> 1324 ? 20% +1200.4% 17217 ? 25% interrupts.CPU45.CAL:Function_call_interrupts
> 146.50 ? 34% +4031.6% 6052 ? 9% interrupts.CPU45.NMI:Non-maskable_interrupts
> 146.50 ? 34% +4031.6% 6052 ? 9% interrupts.CPU45.PMI:Performance_monitoring_interrupts
> 1421 ? 17% -69.5% 433.50 ? 23% interrupts.CPU45.RES:Rescheduling_interrupts
> 12.75 ? 32% +5831.4% 756.25 ? 18% interrupts.CPU45.TLB:TLB_shootdowns
> 1117 ? 8% +1500.1% 17885 ? 6% interrupts.CPU46.CAL:Function_call_interrupts
> 147.00 ? 35% +4182.7% 6295 ? 26% interrupts.CPU46.NMI:Non-maskable_interrupts
> 147.00 ? 35% +4182.7% 6295 ? 26% interrupts.CPU46.PMI:Performance_monitoring_interrupts
> 1382 ? 19% -72.6% 378.75 ? 18% interrupts.CPU46.RES:Rescheduling_interrupts
> 17.25 ? 82% +4456.5% 786.00 ? 14% interrupts.CPU46.TLB:TLB_shootdowns
> 991.00 ? 3% +1739.2% 18226 ? 21% interrupts.CPU47.CAL:Function_call_interrupts
> 146.50 ? 34% +4033.8% 6056 ? 38% interrupts.CPU47.NMI:Non-maskable_interrupts
> 146.50 ? 34% +4033.8% 6056 ? 38% interrupts.CPU47.PMI:Performance_monitoring_interrupts
> 1355 ? 27% -71.1% 392.25 ? 7% interrupts.CPU47.RES:Rescheduling_interrupts
> 9.75 ? 36% +7759.0% 766.25 ? 13% interrupts.CPU47.TLB:TLB_shootdowns
> 1255 ? 4% +1345.2% 18141 ? 12% interrupts.CPU48.CAL:Function_call_interrupts
> 185.25 ? 28% +3158.0% 6035 ? 20% interrupts.CPU48.NMI:Non-maskable_interrupts
> 185.25 ? 28% +3158.0% 6035 ? 20% interrupts.CPU48.PMI:Performance_monitoring_interrupts
> 1187 ? 9% -63.4% 434.50 ? 11% interrupts.CPU48.RES:Rescheduling_interrupts
> 10.75 ? 27% +7479.1% 814.75 ? 14% interrupts.CPU48.TLB:TLB_shootdowns
> 1049 +1531.7% 17116 ? 14% interrupts.CPU49.CAL:Function_call_interrupts
> 193.50 ? 3% +2870.8% 5748 ? 24% interrupts.CPU49.NMI:Non-maskable_interrupts
> 193.50 ? 3% +2870.8% 5748 ? 24% interrupts.CPU49.PMI:Performance_monitoring_interrupts
> 1169 ? 24% -65.7% 400.75 ? 15% interrupts.CPU49.RES:Rescheduling_interrupts
> 12.50 ? 41% +5830.0% 741.25 ? 15% interrupts.CPU49.TLB:TLB_shootdowns
> 1230 ? 22% +1391.4% 18352 ? 14% interrupts.CPU5.CAL:Function_call_interrupts
> 179.00 ? 26% +2636.3% 4898 ? 21% interrupts.CPU5.NMI:Non-maskable_interrupts
> 179.00 ? 26% +2636.3% 4898 ? 21% interrupts.CPU5.PMI:Performance_monitoring_interrupts
> 1262 ? 25% -72.4% 349.00 ? 27% interrupts.CPU5.RES:Rescheduling_interrupts
> 8.50 ? 39% +8964.7% 770.50 ? 6% interrupts.CPU5.TLB:TLB_shootdowns
> 1181 ? 2% +1606.9% 20171 ? 8% interrupts.CPU50.CAL:Function_call_interrupts
> 195.50 ? 3% +2789.4% 5648 ? 24% interrupts.CPU50.NMI:Non-maskable_interrupts
> 195.50 ? 3% +2789.4% 5648 ? 24% interrupts.CPU50.PMI:Performance_monitoring_interrupts
> 1315 ? 15% -68.4% 416.25 ? 14% interrupts.CPU50.RES:Rescheduling_interrupts
> 25.50 ? 78% +3609.8% 946.00 ? 19% interrupts.CPU50.TLB:TLB_shootdowns
> 1153 ? 16% +1384.8% 17131 ? 4% interrupts.CPU51.CAL:Function_call_interrupts
> 197.00 ? 2% +2581.5% 5282 ? 26% interrupts.CPU51.NMI:Non-maskable_interrupts
> 197.00 ? 2% +2581.5% 5282 ? 26% interrupts.CPU51.PMI:Performance_monitoring_interrupts
> 1266 ? 14% -65.7% 433.75 ? 9% interrupts.CPU51.RES:Rescheduling_interrupts
> 14.00 ? 39% +6333.9% 900.75 ? 19% interrupts.CPU51.TLB:TLB_shootdowns
> 1022 ? 4% +1549.3% 16868 ? 11% interrupts.CPU52.CAL:Function_call_interrupts
> 199.75 ? 3% +2957.8% 6108 ? 27% interrupts.CPU52.NMI:Non-maskable_interrupts
> 199.75 ? 3% +2957.8% 6108 ? 27% interrupts.CPU52.PMI:Performance_monitoring_interrupts
> 1245 ? 30% -69.1% 385.00 ? 10% interrupts.CPU52.RES:Rescheduling_interrupts
> 12.00 ? 45% +6639.6% 808.75 ? 4% interrupts.CPU52.TLB:TLB_shootdowns
> 1071 ? 10% +1656.0% 18811 ? 15% interrupts.CPU53.CAL:Function_call_interrupts
> 198.75 ? 3% +2980.1% 6121 ? 29% interrupts.CPU53.NMI:Non-maskable_interrupts
> 198.75 ? 3% +2980.1% 6121 ? 29% interrupts.CPU53.PMI:Performance_monitoring_interrupts
> 1074 ? 34% -63.5% 392.50 ? 13% interrupts.CPU53.RES:Rescheduling_interrupts
> 15.00 ?105% +5255.0% 803.25 ? 14% interrupts.CPU53.TLB:TLB_shootdowns
> 1127 ? 21% +1410.0% 17017 ? 21% interrupts.CPU54.CAL:Function_call_interrupts
> 169.50 ? 24% +3912.4% 6801 ? 27% interrupts.CPU54.NMI:Non-maskable_interrupts
> 169.50 ? 24% +3912.4% 6801 ? 27% interrupts.CPU54.PMI:Performance_monitoring_interrupts
> 1311 ? 26% -71.2% 377.25 ? 23% interrupts.CPU54.RES:Rescheduling_interrupts
> 13.00 ? 21% +6096.2% 805.50 ? 18% interrupts.CPU54.TLB:TLB_shootdowns
> 1322 ? 31% +1308.5% 18620 ? 15% interrupts.CPU55.CAL:Function_call_interrupts
> 147.00 ? 35% +3924.8% 5916 ? 29% interrupts.CPU55.NMI:Non-maskable_interrupts
> 147.00 ? 35% +3924.8% 5916 ? 29% interrupts.CPU55.PMI:Performance_monitoring_interrupts
> 1107 ? 16% -65.8% 379.00 ? 19% interrupts.CPU55.RES:Rescheduling_interrupts
> 24.25 ?114% +3718.6% 926.00 ? 19% interrupts.CPU55.TLB:TLB_shootdowns
> 1116 ? 4% +1521.7% 18110 ? 14% interrupts.CPU56.CAL:Function_call_interrupts
> 173.25 ? 26% +3283.4% 5861 ? 39% interrupts.CPU56.NMI:Non-maskable_interrupts
> 173.25 ? 26% +3283.4% 5861 ? 39% interrupts.CPU56.PMI:Performance_monitoring_interrupts
> 1150 ? 23% -68.1% 366.50 ? 20% interrupts.CPU56.RES:Rescheduling_interrupts
> 10.50 ? 39% +8031.0% 853.75 ? 7% interrupts.CPU56.TLB:TLB_shootdowns
> 1202 ? 21% +1357.6% 17531 ? 21% interrupts.CPU57.CAL:Function_call_interrupts
> 188.50 ? 31% +3120.7% 6071 ? 34% interrupts.CPU57.NMI:Non-maskable_interrupts
> 188.50 ? 31% +3120.7% 6071 ? 34% interrupts.CPU57.PMI:Performance_monitoring_interrupts
> 1343 ? 16% -70.7% 393.50 ? 10% interrupts.CPU57.RES:Rescheduling_interrupts
> 12.75 ? 24% +7007.8% 906.25 ? 19% interrupts.CPU57.TLB:TLB_shootdowns
> 1054 ? 9% +1572.7% 17643 ? 15% interrupts.CPU58.CAL:Function_call_interrupts
> 173.00 ? 26% +2821.1% 5053 ? 23% interrupts.CPU58.NMI:Non-maskable_interrupts
> 173.00 ? 26% +2821.1% 5053 ? 23% interrupts.CPU58.PMI:Performance_monitoring_interrupts
> 1287 ? 19% -68.4% 407.00 ? 17% interrupts.CPU58.RES:Rescheduling_interrupts
> 16.00 ? 98% +5365.6% 874.50 ? 11% interrupts.CPU58.TLB:TLB_shootdowns
> 1035 ? 11% +1525.8% 16831 ? 19% interrupts.CPU59.CAL:Function_call_interrupts
> 171.25 ? 26% +2705.4% 4804 ? 56% interrupts.CPU59.NMI:Non-maskable_interrupts
> 171.25 ? 26% +2705.4% 4804 ? 56% interrupts.CPU59.PMI:Performance_monitoring_interrupts
> 1214 ? 30% -71.0% 352.50 ? 5% interrupts.CPU59.RES:Rescheduling_interrupts
> 27.50 ?116% +3414.5% 966.50 ? 27% interrupts.CPU59.TLB:TLB_shootdowns
> 1158 ? 11% +1497.6% 18500 ? 7% interrupts.CPU6.CAL:Function_call_interrupts
> 176.75 ? 25% +3246.7% 5915 ? 28% interrupts.CPU6.NMI:Non-maskable_interrupts
> 176.75 ? 25% +3246.7% 5915 ? 28% interrupts.CPU6.PMI:Performance_monitoring_interrupts
> 1568 ? 19% -75.1% 391.00 ? 17% interrupts.CPU6.RES:Rescheduling_interrupts
> 9.00 ? 40% +9402.8% 855.25 ? 6% interrupts.CPU6.TLB:TLB_shootdowns
> 1036 ? 7% +1762.4% 19294 ? 24% interrupts.CPU60.CAL:Function_call_interrupts
> 172.25 ? 26% +3278.2% 5819 ? 27% interrupts.CPU60.NMI:Non-maskable_interrupts
> 172.25 ? 26% +3278.2% 5819 ? 27% interrupts.CPU60.PMI:Performance_monitoring_interrupts
> 1405 ? 24% -71.3% 404.00 ? 14% interrupts.CPU60.RES:Rescheduling_interrupts
> 24.75 ? 80% +3187.9% 813.75 ? 8% interrupts.CPU60.TLB:TLB_shootdowns
> 1071 ? 7% +1622.3% 18449 ? 12% interrupts.CPU61.CAL:Function_call_interrupts
> 170.75 ? 27% +3293.9% 5795 ? 30% interrupts.CPU61.NMI:Non-maskable_interrupts
> 170.75 ? 27% +3293.9% 5795 ? 30% interrupts.CPU61.PMI:Performance_monitoring_interrupts
> 1413 ? 13% -71.7% 400.00 ? 5% interrupts.CPU61.RES:Rescheduling_interrupts
> 13.75 ? 22% +6080.0% 849.75 ? 4% interrupts.CPU61.TLB:TLB_shootdowns
> 1145 ? 13% +1577.6% 19212 ? 26% interrupts.CPU62.CAL:Function_call_interrupts
> 170.75 ? 27% +3811.3% 6678 ? 32% interrupts.CPU62.NMI:Non-maskable_interrupts
> 170.75 ? 27% +3811.3% 6678 ? 32% interrupts.CPU62.PMI:Performance_monitoring_interrupts
> 1050 ? 33% -64.0% 378.25 ? 18% interrupts.CPU62.RES:Rescheduling_interrupts
> 9.25 ? 59% +8451.4% 791.00 ? 30% interrupts.CPU62.TLB:TLB_shootdowns
> 2207 ? 90% +761.8% 19021 ? 12% interrupts.CPU63.CAL:Function_call_interrupts
> 172.25 ? 27% +3726.7% 6591 ? 23% interrupts.CPU63.NMI:Non-maskable_interrupts
> 172.25 ? 27% +3726.7% 6591 ? 23% interrupts.CPU63.PMI:Performance_monitoring_interrupts
> 1324 ? 27% -70.2% 395.00 ? 18% interrupts.CPU63.RES:Rescheduling_interrupts
> 22.00 ? 50% +4025.0% 907.50 ? 10% interrupts.CPU63.TLB:TLB_shootdowns
> 1163 ? 5% +1450.6% 18045 ? 10% interrupts.CPU64.CAL:Function_call_interrupts
> 192.50 ? 2% +2521.7% 5046 ? 33% interrupts.CPU64.NMI:Non-maskable_interrupts
> 192.50 ? 2% +2521.7% 5046 ? 33% interrupts.CPU64.PMI:Performance_monitoring_interrupts
> 1376 ? 28% -71.2% 395.75 ? 12% interrupts.CPU64.RES:Rescheduling_interrupts
> 21.00 ? 27% +3470.2% 749.75 ? 8% interrupts.CPU64.TLB:TLB_shootdowns
> 1069 ? 9% +1462.5% 16710 ? 11% interrupts.CPU65.CAL:Function_call_interrupts
> 193.00 ? 2% +3553.5% 7051 ? 13% interrupts.CPU65.NMI:Non-maskable_interrupts
> 193.00 ? 2% +3553.5% 7051 ? 13% interrupts.CPU65.PMI:Performance_monitoring_interrupts
> 1199 ? 24% -68.8% 374.00 ? 12% interrupts.CPU65.RES:Rescheduling_interrupts
> 11.75 ? 62% +6238.3% 744.75 ? 9% interrupts.CPU65.TLB:TLB_shootdowns
> 1055 ? 7% +1710.1% 19105 ? 13% interrupts.CPU66.CAL:Function_call_interrupts
> 204.25 ? 7% +3271.5% 6886 ? 16% interrupts.CPU66.NMI:Non-maskable_interrupts
> 204.25 ? 7% +3271.5% 6886 ? 16% interrupts.CPU66.PMI:Performance_monitoring_interrupts
> 1186 ? 43% -65.7% 406.75 ? 5% interrupts.CPU66.RES:Rescheduling_interrupts
> 17.00 ? 89% +4783.8% 830.25 ? 8% interrupts.CPU66.TLB:TLB_shootdowns
> 1190 ? 16% +1446.7% 18409 ? 12% interrupts.CPU67.CAL:Function_call_interrupts
> 193.50 ? 3% +3171.6% 6330 ? 15% interrupts.CPU67.NMI:Non-maskable_interrupts
> 193.50 ? 3% +3171.6% 6330 ? 15% interrupts.CPU67.PMI:Performance_monitoring_interrupts
> 1108 ? 17% -64.6% 392.00 ? 14% interrupts.CPU67.RES:Rescheduling_interrupts
> 16.75 ? 68% +4800.0% 820.75 ? 10% interrupts.CPU67.TLB:TLB_shootdowns
> 1058 ? 3% +1616.3% 18158 ? 16% interrupts.CPU68.CAL:Function_call_interrupts
> 194.75 ? 3% +3460.2% 6933 ? 13% interrupts.CPU68.NMI:Non-maskable_interrupts
> 194.75 ? 3% +3460.2% 6933 ? 13% interrupts.CPU68.PMI:Performance_monitoring_interrupts
> 1436 ? 22% -72.7% 392.50 ? 17% interrupts.CPU68.RES:Rescheduling_interrupts
> 15.50 ? 39% +5325.8% 841.00 ? 11% interrupts.CPU68.TLB:TLB_shootdowns
> 1025 ? 7% +1608.1% 17512 ? 20% interrupts.CPU69.CAL:Function_call_interrupts
> 193.25 ? 3% +2644.6% 5304 ? 28% interrupts.CPU69.NMI:Non-maskable_interrupts
> 193.25 ? 3% +2644.6% 5304 ? 28% interrupts.CPU69.PMI:Performance_monitoring_interrupts
> 1345 ? 30% -72.5% 369.75 ? 15% interrupts.CPU69.RES:Rescheduling_interrupts
> 20.25 ? 41% +4295.1% 890.00 ? 15% interrupts.CPU69.TLB:TLB_shootdowns
> 1217 ? 11% +1464.7% 19046 ? 14% interrupts.CPU7.CAL:Function_call_interrupts
> 208.50 ? 6% +2593.2% 5615 ? 29% interrupts.CPU7.NMI:Non-maskable_interrupts
> 208.50 ? 6% +2593.2% 5615 ? 29% interrupts.CPU7.PMI:Performance_monitoring_interrupts
> 1075 ? 19% -64.9% 377.00 ? 9% interrupts.CPU7.RES:Rescheduling_interrupts
> 7.75 ? 27% +10390.3% 813.00 ? 16% interrupts.CPU7.TLB:TLB_shootdowns
> 1022 ? 7% +1610.7% 17483 ? 8% interrupts.CPU70.CAL:Function_call_interrupts
> 199.25 ? 3% +3039.6% 6255 ? 16% interrupts.CPU70.NMI:Non-maskable_interrupts
> 199.25 ? 3% +3039.6% 6255 ? 16% interrupts.CPU70.PMI:Performance_monitoring_interrupts
> 1400 ? 10% -71.7% 396.75 ? 17% interrupts.CPU70.RES:Rescheduling_interrupts
> 9.75 ? 24% +8520.5% 840.50 ? 10% interrupts.CPU70.TLB:TLB_shootdowns
> 946.25 ? 2% +1795.8% 17939 ? 20% interrupts.CPU71.CAL:Function_call_interrupts
> 199.50 ? 8% +2880.2% 5945 ? 28% interrupts.CPU71.NMI:Non-maskable_interrupts
> 199.50 ? 8% +2880.2% 5945 ? 28% interrupts.CPU71.PMI:Performance_monitoring_interrupts
> 1114 ? 24% -68.6% 350.00 ? 22% interrupts.CPU71.RES:Rescheduling_interrupts
> 20.75 ?115% +3991.6% 849.00 ? 14% interrupts.CPU71.TLB:TLB_shootdowns
> 1275 ? 6% +1258.0% 17321 ? 5% interrupts.CPU72.CAL:Function_call_interrupts
> 217.50 ? 9% +2579.8% 5828 ? 19% interrupts.CPU72.NMI:Non-maskable_interrupts
> 217.50 ? 9% +2579.8% 5828 ? 19% interrupts.CPU72.PMI:Performance_monitoring_interrupts
> 989.25 ? 7% -62.1% 375.00 ? 17% interrupts.CPU72.RES:Rescheduling_interrupts
> 12.25 ? 40% +6334.7% 788.25 ? 18% interrupts.CPU72.TLB:TLB_shootdowns
> 947.00 ? 14% +1776.1% 17766 ? 12% interrupts.CPU73.CAL:Function_call_interrupts
> 210.00 ? 7% +2687.9% 5854 ? 21% interrupts.CPU73.NMI:Non-maskable_interrupts
> 210.00 ? 7% +2687.9% 5854 ? 21% interrupts.CPU73.PMI:Performance_monitoring_interrupts
> 972.75 ? 14% -61.3% 376.00 ? 18% interrupts.CPU73.RES:Rescheduling_interrupts
> 13.00 ? 47% +5917.3% 782.25 ? 13% interrupts.CPU73.TLB:TLB_shootdowns
> 1414 ? 15% +1142.3% 17569 ? 8% interrupts.CPU74.CAL:Function_call_interrupts
> 199.75 ? 3% +2971.5% 6135 ? 14% interrupts.CPU74.NMI:Non-maskable_interrupts
> 199.75 ? 3% +2971.5% 6135 ? 14% interrupts.CPU74.PMI:Performance_monitoring_interrupts
> 1032 ? 22% -64.1% 370.50 ? 12% interrupts.CPU74.RES:Rescheduling_interrupts
> 12.75 ? 15% +6227.5% 806.75 ? 10% interrupts.CPU74.TLB:TLB_shootdowns
> 1299 ? 9% +1348.1% 18810 ? 13% interrupts.CPU75.CAL:Function_call_interrupts
> 203.50 ? 3% +2811.8% 5925 ? 22% interrupts.CPU75.NMI:Non-maskable_interrupts
> 203.50 ? 3% +2811.8% 5925 ? 22% interrupts.CPU75.PMI:Performance_monitoring_interrupts
> 1076 ? 11% -65.2% 374.00 ? 15% interrupts.CPU75.RES:Rescheduling_interrupts
> 10.50 ? 39% +7202.4% 766.75 ? 14% interrupts.CPU75.TLB:TLB_shootdowns
> 1345 ? 17% +1239.1% 18014 ? 10% interrupts.CPU76.CAL:Function_call_interrupts
> 206.50 ? 7% +2614.3% 5605 ? 24% interrupts.CPU76.NMI:Non-maskable_interrupts
> 206.50 ? 7% +2614.3% 5605 ? 24% interrupts.CPU76.PMI:Performance_monitoring_interrupts
> 1611 ? 33% -77.8% 357.75 ? 6% interrupts.CPU76.RES:Rescheduling_interrupts
> 28.25 ? 59% +2906.2% 849.25 ? 6% interrupts.CPU76.TLB:TLB_shootdowns
> 1213 ? 13% +1514.4% 19595 ? 17% interrupts.CPU77.CAL:Function_call_interrupts
> 201.50 ? 4% +2808.8% 5861 ? 16% interrupts.CPU77.NMI:Non-maskable_interrupts
> 201.50 ? 4% +2808.8% 5861 ? 16% interrupts.CPU77.PMI:Performance_monitoring_interrupts
> 1088 ? 13% -67.2% 356.50 ? 21% interrupts.CPU77.RES:Rescheduling_interrupts
> 20.00 ? 82% +3843.8% 788.75 ? 21% interrupts.CPU77.TLB:TLB_shootdowns
> 1274 ? 15% +1467.5% 19973 ? 15% interrupts.CPU78.CAL:Function_call_interrupts
> 203.00 ? 6% +2826.2% 5940 ? 26% interrupts.CPU78.NMI:Non-maskable_interrupts
> 203.00 ? 6% +2826.2% 5940 ? 26% interrupts.CPU78.PMI:Performance_monitoring_interrupts
> 1368 ? 23% -72.7% 374.00 ? 12% interrupts.CPU78.RES:Rescheduling_interrupts
> 15.00 ? 35% +5416.7% 827.50 ? 8% interrupts.CPU78.TLB:TLB_shootdowns
> 1218 ? 10% +1509.9% 19608 ? 11% interrupts.CPU79.CAL:Function_call_interrupts
> 198.00 ? 3% +2778.9% 5700 ? 34% interrupts.CPU79.NMI:Non-maskable_interrupts
> 198.00 ? 3% +2778.9% 5700 ? 34% interrupts.CPU79.PMI:Performance_monitoring_interrupts
> 994.00 ? 53% -63.5% 362.50 ? 16% interrupts.CPU79.RES:Rescheduling_interrupts
> 12.50 ? 37% +5876.0% 747.00 ? 19% interrupts.CPU79.TLB:TLB_shootdowns
> 1216 ? 12% +1528.0% 19804 ? 9% interrupts.CPU8.CAL:Function_call_interrupts
> 204.25 ? 2% +2584.2% 5482 ? 23% interrupts.CPU8.NMI:Non-maskable_interrupts
> 204.25 ? 2% +2584.2% 5482 ? 23% interrupts.CPU8.PMI:Performance_monitoring_interrupts
> 908.00 ? 25% -57.7% 384.50 ? 15% interrupts.CPU8.RES:Rescheduling_interrupts
> 17.75 ? 98% +4738.0% 858.75 ? 7% interrupts.CPU8.TLB:TLB_shootdowns
> 1431 ? 5% +1096.4% 17127 ? 19% interrupts.CPU80.CAL:Function_call_interrupts
> 198.50 ? 3% +2388.4% 4939 ? 36% interrupts.CPU80.NMI:Non-maskable_interrupts
> 198.50 ? 3% +2388.4% 4939 ? 36% interrupts.CPU80.PMI:Performance_monitoring_interrupts
> 990.75 ? 20% -62.3% 373.25 ? 17% interrupts.CPU80.RES:Rescheduling_interrupts
> 15.00 ? 12% +4636.7% 710.50 ? 9% interrupts.CPU80.TLB:TLB_shootdowns
> 1297 ? 6% +1303.6% 18211 ? 13% interrupts.CPU81.CAL:Function_call_interrupts
> 199.75 ? 3% +2650.4% 5494 ? 30% interrupts.CPU81.NMI:Non-maskable_interrupts
> 199.75 ? 3% +2650.4% 5494 ? 30% interrupts.CPU81.PMI:Performance_monitoring_interrupts
> 1206 ? 21% -67.1% 397.00 ? 15% interrupts.CPU81.RES:Rescheduling_interrupts
> 9.00 ? 45% +9580.6% 871.25 ? 10% interrupts.CPU81.TLB:TLB_shootdowns
> 1290 ? 16% +1374.2% 19024 ? 17% interrupts.CPU82.CAL:Function_call_interrupts
> 213.75 ? 16% +2576.6% 5721 ? 24% interrupts.CPU82.NMI:Non-maskable_interrupts
> 213.75 ? 16% +2576.6% 5721 ? 24% interrupts.CPU82.PMI:Performance_monitoring_interrupts
> 1347 ? 19% -73.0% 363.75 ? 19% interrupts.CPU82.RES:Rescheduling_interrupts
> 16.50 ? 17% +4842.4% 815.50 ? 13% interrupts.CPU82.TLB:TLB_shootdowns
> 1438 ? 49% +1202.3% 18734 ? 11% interrupts.CPU83.CAL:Function_call_interrupts
> 202.00 ? 2% +3182.2% 6630 ? 27% interrupts.CPU83.NMI:Non-maskable_interrupts
> 202.00 ? 2% +3182.2% 6630 ? 27% interrupts.CPU83.PMI:Performance_monitoring_interrupts
> 972.00 ? 34% -56.0% 427.50 ? 28% interrupts.CPU83.RES:Rescheduling_interrupts
> 10.25 ? 4% +8290.2% 860.00 ? 9% interrupts.CPU83.TLB:TLB_shootdowns
> 1247 ? 8% +1491.5% 19857 ? 3% interrupts.CPU84.CAL:Function_call_interrupts
> 204.25 ? 6% +2474.4% 5258 ? 20% interrupts.CPU84.NMI:Non-maskable_interrupts
> 204.25 ? 6% +2474.4% 5258 ? 20% interrupts.CPU84.PMI:Performance_monitoring_interrupts
> 1083 ? 20% -64.8% 381.00 ? 11% interrupts.CPU84.RES:Rescheduling_interrupts
> 28.75 ? 70% +2530.4% 756.25 ? 8% interrupts.CPU84.TLB:TLB_shootdowns
> 1190 ? 9% +1535.3% 19468 ? 10% interrupts.CPU85.CAL:Function_call_interrupts
> 204.25 ? 5% +2661.3% 5640 ? 21% interrupts.CPU85.NMI:Non-maskable_interrupts
> 204.25 ? 5% +2661.3% 5640 ? 21% interrupts.CPU85.PMI:Performance_monitoring_interrupts
> 879.50 ? 5% -56.1% 386.25 ? 11% interrupts.CPU85.RES:Rescheduling_interrupts
> 11.75 ? 38% +7053.2% 840.50 ? 10% interrupts.CPU85.TLB:TLB_shootdowns
> 1306 ? 11% +1246.6% 17593 ? 11% interrupts.CPU86.CAL:Function_call_interrupts
> 205.50 ? 6% +2968.5% 6305 ? 20% interrupts.CPU86.NMI:Non-maskable_interrupts
> 205.50 ? 6% +2968.5% 6305 ? 20% interrupts.CPU86.PMI:Performance_monitoring_interrupts
> 1465 ? 22% -72.2% 406.75 ? 14% interrupts.CPU86.RES:Rescheduling_interrupts
> 21.00 ? 22% +3439.3% 743.25 ? 14% interrupts.CPU86.TLB:TLB_shootdowns
> 1271 ? 16% +1446.3% 19653 ? 14% interrupts.CPU87.CAL:Function_call_interrupts
> 170.00 ? 21% +3408.2% 5964 ? 28% interrupts.CPU87.NMI:Non-maskable_interrupts
> 170.00 ? 21% +3408.2% 5964 ? 28% interrupts.CPU87.PMI:Performance_monitoring_interrupts
> 1211 ? 33% -66.4% 407.00 ? 15% interrupts.CPU87.RES:Rescheduling_interrupts
> 36.75 ? 66% +2429.3% 929.50 ? 15% interrupts.CPU87.TLB:TLB_shootdowns
> 1240 ? 8% +1407.9% 18705 ? 8% interrupts.CPU88.CAL:Function_call_interrupts
> 171.25 ? 22% +2680.7% 4762 ? 26% interrupts.CPU88.NMI:Non-maskable_interrupts
> 171.25 ? 22% +2680.7% 4762 ? 26% interrupts.CPU88.PMI:Performance_monitoring_interrupts
> 1250 ? 19% -71.0% 363.00 ? 13% interrupts.CPU88.RES:Rescheduling_interrupts
> 19.00 ? 59% +4122.4% 802.25 ? 18% interrupts.CPU88.TLB:TLB_shootdowns
> 1401 ? 39% +1330.7% 20051 ? 20% interrupts.CPU89.CAL:Function_call_interrupts
> 174.50 ? 19% +2893.8% 5224 ? 34% interrupts.CPU89.NMI:Non-maskable_interrupts
> 174.50 ? 19% +2893.8% 5224 ? 34% interrupts.CPU89.PMI:Performance_monitoring_interrupts
> 1102 ? 28% -63.8% 399.50 ? 18% interrupts.CPU89.RES:Rescheduling_interrupts
> 14.50 ? 44% +6006.9% 885.50 ? 10% interrupts.CPU89.TLB:TLB_shootdowns
> 1137 ? 6% +1300.8% 15931 ? 9% interrupts.CPU9.CAL:Function_call_interrupts
> 209.25 +2083.9% 4569 ? 34% interrupts.CPU9.NMI:Non-maskable_interrupts
> 209.25 +2083.9% 4569 ? 34% interrupts.CPU9.PMI:Performance_monitoring_interrupts
> 868.25 ? 9% -64.5% 308.25 ? 22% interrupts.CPU9.RES:Rescheduling_interrupts
> 16.25 ? 99% +3958.5% 659.50 ? 12% interrupts.CPU9.TLB:TLB_shootdowns
> 1735 ? 24% +1116.8% 21117 ? 12% interrupts.CPU90.CAL:Function_call_interrupts
> 172.25 ? 22% +3146.7% 5592 ? 22% interrupts.CPU90.NMI:Non-maskable_interrupts
> 172.25 ? 22% +3146.7% 5592 ? 22% interrupts.CPU90.PMI:Performance_monitoring_interrupts
> 11.00 ? 51% +7452.3% 830.75 ? 12% interrupts.CPU90.TLB:TLB_shootdowns
> 1216 ? 9% +1593.4% 20592 ? 9% interrupts.CPU91.CAL:Function_call_interrupts
> 171.50 ? 24% +2807.9% 4987 ? 37% interrupts.CPU91.NMI:Non-maskable_interrupts
> 171.50 ? 24% +2807.9% 4987 ? 37% interrupts.CPU91.PMI:Performance_monitoring_interrupts
> 10.00 ? 33% +9285.0% 938.50 ? 3% interrupts.CPU91.TLB:TLB_shootdowns
> 1601 ? 31% +1199.7% 20807 ? 5% interrupts.CPU92.CAL:Function_call_interrupts
> 176.25 ? 22% +2839.7% 5181 ? 26% interrupts.CPU92.NMI:Non-maskable_interrupts
> 176.25 ? 22% +2839.7% 5181 ? 26% interrupts.CPU92.PMI:Performance_monitoring_interrupts
> 980.00 ? 29% -53.0% 461.00 ? 16% interrupts.CPU92.RES:Rescheduling_interrupts
> 27.50 ? 89% +3560.9% 1006 ? 10% interrupts.CPU92.TLB:TLB_shootdowns
> 1250 ? 11% +1394.2% 18678 ? 11% interrupts.CPU93.CAL:Function_call_interrupts
> 173.00 ? 23% +3162.4% 5644 ? 41% interrupts.CPU93.NMI:Non-maskable_interrupts
> 173.00 ? 23% +3162.4% 5644 ? 41% interrupts.CPU93.PMI:Performance_monitoring_interrupts
> 860.00 ? 44% -54.3% 393.00 ? 7% interrupts.CPU93.RES:Rescheduling_interrupts
> 5.25 ? 47% +14252.4% 753.50 ? 16% interrupts.CPU93.TLB:TLB_shootdowns
> 1246 ? 10% +1684.5% 22235 ? 11% interrupts.CPU94.CAL:Function_call_interrupts
> 171.25 ? 22% +2938.8% 5204 ? 34% interrupts.CPU94.NMI:Non-maskable_interrupts
> 171.25 ? 22% +2938.8% 5204 ? 34% interrupts.CPU94.PMI:Performance_monitoring_interrupts
> 1248 ? 28% -65.6% 429.50 ? 11% interrupts.CPU94.RES:Rescheduling_interrupts
> 15.50 ? 18% +6246.8% 983.75 ? 7% interrupts.CPU94.TLB:TLB_shootdowns
> 1366 ? 11% +1942.5% 27906 ? 11% interrupts.CPU95.CAL:Function_call_interrupts
> 178.50 ? 21% +3917.6% 7171 ? 24% interrupts.CPU95.NMI:Non-maskable_interrupts
> 178.50 ? 21% +3917.6% 7171 ? 24% interrupts.CPU95.PMI:Performance_monitoring_interrupts
> 1552 ? 14% -65.5% 535.75 ? 12% interrupts.CPU95.RES:Rescheduling_interrupts
> 24.75 ? 11% +4877.8% 1232 ? 4% interrupts.CPU95.TLB:TLB_shootdowns
> 1383 ? 18% +1511.1% 22285 ? 7% interrupts.CPU96.CAL:Function_call_interrupts
> 189.75 ? 24% +2856.4% 5609 ? 22% interrupts.CPU96.NMI:Non-maskable_interrupts
> 189.75 ? 24% +2856.4% 5609 ? 22% interrupts.CPU96.PMI:Performance_monitoring_interrupts
> 2279 ? 13% -80.7% 439.25 ? 13% interrupts.CPU96.RES:Rescheduling_interrupts
> 18.25 ? 19% +5732.9% 1064 ? 8% interrupts.CPU96.TLB:TLB_shootdowns
> 1344 ? 21% +988.4% 14633 ? 11% interrupts.CPU97.CAL:Function_call_interrupts
> 175.25 ? 24% +2505.1% 4565 ? 24% interrupts.CPU97.NMI:Non-maskable_interrupts
> 175.25 ? 24% +2505.1% 4565 ? 24% interrupts.CPU97.PMI:Performance_monitoring_interrupts
> 1252 ? 35% -75.2% 311.25 ? 7% interrupts.CPU97.RES:Rescheduling_interrupts
> 23.25 ?106% +2841.9% 684.00 ? 11% interrupts.CPU97.TLB:TLB_shootdowns
> 1460 ? 19% +897.0% 14556 ? 16% interrupts.CPU98.CAL:Function_call_interrupts
> 208.00 ? 5% +2507.9% 5424 ? 33% interrupts.CPU98.NMI:Non-maskable_interrupts
> 208.00 ? 5% +2507.9% 5424 ? 33% interrupts.CPU98.PMI:Performance_monitoring_interrupts
> 1478 ? 21% -80.3% 291.25 ? 24% interrupts.CPU98.RES:Rescheduling_interrupts
> 14.00 ? 44% +4632.1% 662.50 ? 24% interrupts.CPU98.TLB:TLB_shootdowns
> 1310 ? 32% +1008.5% 14523 ? 11% interrupts.CPU99.CAL:Function_call_interrupts
> 203.00 ? 3% +2390.0% 5054 ? 28% interrupts.CPU99.NMI:Non-maskable_interrupts
> 203.00 ? 3% +2390.0% 5054 ? 28% interrupts.CPU99.PMI:Performance_monitoring_interrupts
> 1333 ? 13% -77.4% 300.75 ? 17% interrupts.CPU99.RES:Rescheduling_interrupts
> 15.25 ? 37% +4483.6% 699.00 ? 12% interrupts.CPU99.TLB:TLB_shootdowns
> 3.25 ? 13% +6584.6% 217.25 ? 8% interrupts.IWI:IRQ_work_interrupts
> 35365 ? 2% +2967.1% 1084710 ? 7% interrupts.NMI:Non-maskable_interrupts
> 35365 ? 2% +2967.1% 1084710 ? 7% interrupts.PMI:Performance_monitoring_interrupts
> 237203 ? 2% -72.7% 64704 ? 12% interrupts.RES:Rescheduling_interrupts
> 3697 ? 12% +3721.5% 141290 interrupts.TLB:TLB_shootdowns
>
>
>
> ***************************************************************************************************
> lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
> interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/stress-ng/30s/0x5002f01
>
> commit:
> 5925fa68fe (" perf tools fixes for v5.9: 3rd batch")
> 5ef64cc898 ("mm: allow a controlled amount of unfairness in the page lock")
>
> 5925fa68fe824465 5ef64cc8987a9211d3f3667331b
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :4 50% 2:4 kmsg.BTRFS_error(device_sda5):bdev/dev/sda5_errs:wr#,rd#,flush#,corrupt#,gen
> %stddev %change %stddev
> \ | \
> 1543392 ? 2% -62.7% 575755 ? 90% stress-ng.sigchld.ops
> 51445 ? 2% -62.7% 19191 ? 90% stress-ng.sigchld.ops_per_sec
> 2.397e+08 ? 46% -90.9% 21848257 ? 2% stress-ng.sigrt.ops
> 7970038 ? 46% -90.9% 727138 ? 2% stress-ng.sigrt.ops_per_sec
> 6635 +3.0% 6831 stress-ng.time.percent_of_cpu_this_job_got
> 42339 +4.7% 44340 stress-ng.time.system_time
> 4468 ? 2% -15.0% 3800 ? 7% stress-ng.time.user_time
> 1.629e+09 -3.2% 1.576e+09 ? 2% stress-ng.timerfd.ops
> 54286374 -3.2% 52528279 ? 2% stress-ng.timerfd.ops_per_sec
> 0.75 ?173% +33333.3% 250.75 ?149% interrupts.104:PCI-MSI.31981637-edge.i40e-eth0-TxRx-68
> 0.75 ?173% +33266.7% 250.25 ?150% interrupts.CPU68.104:PCI-MSI.31981637-edge.i40e-eth0-TxRx-68
> 15.59 ? 2% -1.8 13.80 ? 3% mpstat.cpu.all.idle%
> 3.17 ? 4% -0.7 2.48 ? 13% mpstat.cpu.all.iowait%
> 46863 ? 29% -21.0% 37005 ? 30% proc-vmstat.nr_kernel_stack
> 6181473 ? 3% -60.5% 2442698 ? 88% proc-vmstat.pgreuse
> 6.755e+08 ? 8% -27.7% 4.883e+08 ? 16% cpuidle.C1.time
> 1.461e+08 ? 42% -50.0% 73023084 ? 7% cpuidle.POLL.time
> 12133129 ? 45% -87.8% 1477722 ? 8% cpuidle.POLL.usage
> 15.92 ? 2% -11.7% 14.06 ? 3% iostat.cpu.idle
> 3.18 ? 4% -22.0% 2.48 ? 12% iostat.cpu.iowait
> 76.37 +3.6% 79.12 iostat.cpu.system
> 15.25 ? 2% -11.5% 13.50 ? 3% vmstat.cpu.id
> 76.25 +3.0% 78.50 vmstat.cpu.sy
> 129.50 ? 59% -96.9% 4.00 ? 25% vmstat.procs.b
> 77813 ? 2% -10.5% 69667 ? 7% softirqs.CPU25.SCHED
> 81392 ? 7% -15.5% 68815 ? 8% softirqs.CPU41.SCHED
> 78946 ? 2% -10.3% 70795 ? 6% softirqs.CPU76.SCHED
> 78599 ? 2% -12.4% 68835 ? 8% softirqs.CPU77.SCHED
> 81487 ? 6% -14.0% 70042 ? 9% softirqs.CPU83.SCHED
> 67827 ? 16% -31.1% 46745 ? 2% slabinfo.Acpi-State.active_objs
> 1370 ? 17% -32.8% 920.25 ? 2% slabinfo.Acpi-State.active_slabs
> 69916 ? 17% -32.8% 46969 ? 2% slabinfo.Acpi-State.num_objs
> 1370 ? 17% -32.8% 920.25 ? 2% slabinfo.Acpi-State.num_slabs
> 810.00 ? 6% +17.8% 954.00 ? 3% slabinfo.kmem_cache_node.active_objs
> 847.50 ? 6% +17.0% 991.50 ? 3% slabinfo.kmem_cache_node.num_objs
> 5228 -16.6% 4359 ? 9% slabinfo.sighand_cache.active_objs
> 5314 -15.0% 4516 ? 8% slabinfo.sighand_cache.num_objs
> 50134 ? 42% -70.9% 14578 ? 46% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 2621056 ? 41% -46.6% 1399499 ? 46% sched_debug.cfs_rq:/.MIN_vruntime.max
> 345309 ? 38% -58.9% 142092 ? 46% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 50134 ? 42% -70.9% 14578 ? 46% sched_debug.cfs_rq:/.max_vruntime.avg
> 2621056 ? 41% -46.6% 1399499 ? 46% sched_debug.cfs_rq:/.max_vruntime.max
> 345309 ? 38% -58.9% 142092 ? 46% sched_debug.cfs_rq:/.max_vruntime.stddev
> 139637 ? 19% +35.2% 188810 ? 14% sched_debug.cpu.avg_idle.min
> 5696 ? 33% -57.4% 2425 ? 58% sched_debug.cpu.max_idle_balance_cost.stddev
> 575.75 -83.2% 96.71 ? 16% sched_debug.cpu.nr_uninterruptible.max
> -567.52 -73.4% -150.73 sched_debug.cpu.nr_uninterruptible.min
> 304.20 ? 8% -88.5% 35.11 ? 18% sched_debug.cpu.nr_uninterruptible.stddev
> 286124 ? 3% -13.0% 248853 ? 9% sched_debug.cpu.sched_goidle.avg
> 250037 ? 2% -14.0% 214914 ? 8% sched_debug.cpu.sched_goidle.min
> 1.051e+08 -6.7% 98127607 ? 4% perf-stat.i.branch-misses
> 63185666 ? 7% -16.5% 52728923 ? 15% perf-stat.i.cache-misses
> 2.133e+11 +3.0% 2.196e+11 perf-stat.i.cpu-cycles
> 5687177 ? 2% -5.4% 5377921 ? 3% perf-stat.i.dTLB-store-misses
> 8.732e+09 -4.0% 8.379e+09 perf-stat.i.dTLB-stores
> 73450472 -4.3% 70273743 perf-stat.i.iTLB-load-misses
> 0.26 -5.7% 0.25 ? 2% perf-stat.i.ipc
> 2.21 +3.0% 2.28 perf-stat.i.metric.GHz
> 532333 -9.8% 480107 ? 6% perf-stat.i.page-faults
> 5.10 ? 4% -13.4% 4.42 ? 11% perf-stat.overall.MPKI
> 0.94 -0.0 0.92 perf-stat.overall.branch-miss-rate%
> 70.77 -6.0 64.73 ? 4% perf-stat.overall.node-load-miss-rate%
> 83.74 -1.1 82.62 perf-stat.overall.node-store-miss-rate%
> 1.018e+08 -6.6% 95063940 ? 4% perf-stat.ps.branch-misses
> 71426753 ? 6% -13.6% 61704660 ? 12% perf-stat.ps.cache-misses
> 2.156e+11 +2.8% 2.216e+11 perf-stat.ps.cpu-cycles
> 8.315e+09 -4.1% 7.972e+09 ? 2% perf-stat.ps.dTLB-stores
> 69988088 -4.5% 66854914 perf-stat.ps.iTLB-load-misses
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
>
> _______________________________________________
> LKP mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>

--
Zhengjun Xing

2020-10-12 16:36:27

by Linus Torvalds

Subject: Re: [LKP] [mm] 5ef64cc898: vm-scalability.throughput -20.6% regression

On Sun, Oct 11, 2020 at 11:57 PM Xing Zhengjun
<[email protected]> wrote:
>
> Hi Linus,
>
> Do you have time to look at this? Thanks. I re-test it in v5.9-rc8,
> the regression still existed.

This is one of the series vm-scalability tests that got a huge
improvement (up to 160%) when I did the complete page fairness patch
(ie commit 2a9127fcf229).

But since that fairness thing caused regressions elsewhere, it got
mostly limited, so we are likely back in the same ballpark as before
(although hopefully without some of the absolute _worst_ latency
peaks, who knows).

All these vm-scalability tests seem to be very noisy and unreliable,
and the exact details of the page locking can cause huge differences
for almost random reasons.

I think the main issue is just that bad timing luck and the fact that
the page lock is too contended under some loads makes for test cases
that can show fairly bi-modal behavior.

Linus