LinuxLists.cc - [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

2020-06-22 01:13:27

Subject: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

Greeting,

FYI, we

commit: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
href="https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/">https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/
the issue, kindly add following tag
kernel test robot <[email protected]>
--------------------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
bin/lkp install job.yaml # job file is attached in this email
========================================================================
governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
ce/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-knm01/anon-cow-seq-hugetlb/vm-scalability/0x11
("mm/memblock.c: remove redundant assignment to variable max_addr")
("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization")
c0d0381ade79885c04a04c30328
---------------------------
%reproduction fail:runs
| |
-25% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
25% 1:4 dmesg.WARNING:at_ip__mutex_lock/0x
25% 1:4 dmesg.WARNING:at_ip_perf_event_mmap_output/0x
%change %stddev
| \
-29.5% 26549 ± 4% vm-scalability.median
+9.3 18.48 ± 21% vm-scalability.median_stddev%
-33.4% 8429873 ± 4% vm-scalability.throughput
-19.5% 1369683 ± 5% vm-scalability.time.minor_page_faults
-19.3% 10848 vm-scalability.time.percent_of_cpu_this_job_got
-10.5% 23830 vm-scalability.time.system_time
-35.3% 9255 ± 4% vm-scalability.time.user_time
+621.3% 904348 ± 4% vm-scalability.time.voluntary_context_switches
-19.6% 1.984e+09 ± 5% vm-scalability.workload
+20.9% 4.605e+10 ± 2% cpuidle.C1.time
-17.3% 7873 ± 4% numa-vmstat.node0.nr_mapped
-12.4% 2184278 ± 3% numa-numastat.node0.local_node
-12.4% 2184223 ± 3% numa-numastat.node0.numa_hit
+9.4 58.62 mpstat.cpu.all.idle%
-3.2 29.70 mpstat.cpu.all.sys%
-6.2 11.64 ± 4% mpstat.cpu.all.usr%
+47.6% 7169 numa-meminfo.node0.HugePages_Surp
+47.6% 7169 numa-meminfo.node0.HugePages_Total
-17.7% 31054 ± 4% numa-meminfo.node0.Mapped
+38.3% 16981169 numa-meminfo.node0.MemUsed
+18.9% 58.25 vmstat.cpu.id
-33.8% 11.25 ± 3% vmstat.cpu.us
-19.5% 111.25 ± 2% vmstat.procs.r
+88.0% 10969 ± 3% vmstat.system.cs
+48.0% 7186 meminfo.HugePages_Surp
+48.0% 7186 meminfo.HugePages_Total
+48.0% 14718559 meminfo.Hugetlb
-13.9% 41859 ± 3% meminfo.Mapped
+36.8% 17619364 meminfo.Memused
+29.1% 105324 meminfo.max_used_kB
+285.4% 11139 ± 3% perf-stat.i.context-switches
+243.7% 1025 ± 2% perf-stat.i.cpu-migrations
+119.3% 0.24 perf-stat.i.ipc
+129.7% 0.10 ± 4% perf-stat.i.metric.K/sec
+122.1% 0.24 perf-stat.overall.ipc
+137.3% 7372 ± 3% perf-stat.overall.path-length
+284.4% 10902 ± 3% perf-stat.ps.context-switches
+257.9% 984.31 ± 2% perf-stat.ps.cpu-migrations
-11.7% 98924 ± 2% slabinfo.anon_vma.active_objs
-11.7% 2150 ± 2% slabinfo.anon_vma.active_slabs
-11.7% 98924 ± 2% slabinfo.anon_vma.num_objs
-11.7% 2150 ± 2% slabinfo.anon_vma.num_slabs
-9.1% 235038 ± 2% slabinfo.anon_vma_chain.active_objs
-9.1% 3672 ± 2% slabinfo.anon_vma_chain.active_slabs
-9.1% 235049 ± 2% slabinfo.anon_vma_chain.num_objs
-9.1% 3672 ± 2% slabinfo.anon_vma_chain.num_slabs
-17.8% 1232 ± 5% slabinfo.hugetlbfs_inode_cache.active_objs
-17.8% 1232 ± 5% slabinfo.hugetlbfs_inode_cache.num_objs
+15.9% 790.50 ± 6% slabinfo.numa_policy.active_objs
+15.9% 790.50 ± 6% slabinfo.numa_policy.num_objs
-19.6% 869890 ± 5% proc-vmstat.htlb_buddy_alloc_success
-6.8% 1604364 proc-vmstat.nr_dirty_background_threshold
-6.8% 3212652 proc-vmstat.nr_dirty_threshold
-6.8% 16197198 proc-vmstat.nr_free_pages
+3.8% 364.25 proc-vmstat.nr_inactive_file
+1.4% 40269 proc-vmstat.nr_kernel_stack
-13.7% 10570 ± 3% proc-vmstat.nr_mapped
+8.8% 2863 proc-vmstat.nr_page_table_pages
-1.6% 97341 proc-vmstat.nr_slab_unreclaimable
+3.8% 364.25 proc-vmstat.nr_zone_inactive_file
-90.8% 577.50 ± 47% proc-vmstat.numa_hint_faults
-90.8% 577.50 ± 47% proc-vmstat.numa_hint_faults_local
-12.3% 2205532 ± 3% proc-vmstat.numa_hit
-12.3% 2205531 ± 3% proc-vmstat.numa_local
-19.6% 4.469e+08 ± 5% proc-vmstat.pgalloc_normal
-12.9% 2187221 ± 3% proc-vmstat.pgfault
-19.5% 4.468e+08 ± 5% proc-vmstat.pgfree
-19.7% 48269 ± 17% sched_debug.cfs_rq:/.exec_clock.avg
-23.4% 54794 ± 16% sched_debug.cfs_rq:/.exec_clock.max
-16.9% 36954 ± 18% sched_debug.cfs_rq:/.exec_clock.min
-47.8% 2412 ± 11% sched_debug.cfs_rq:/.exec_clock.stddev
+371.9% 2897 ±124% sched_debug.cfs_rq:/.load_avg.max
+321.1% 208.79 ±123% sched_debug.cfs_rq:/.load_avg.stddev
-37.6% 7189313 ± 17% sched_debug.cfs_rq:/.min_vruntime.avg
-40.0% 8039878 ± 15% sched_debug.cfs_rq:/.min_vruntime.max
-37.2% 5294770 ± 18% sched_debug.cfs_rq:/.min_vruntime.min
-56.4% 343805 ± 11% sched_debug.cfs_rq:/.min_vruntime.stddev
-40.0% 0.22 ± 10% sched_debug.cfs_rq:/.nr_running.avg
+134.9% 49.72 ± 21% sched_debug.cfs_rq:/.nr_spread_over.avg
+60.7% 90.25 ± 19% sched_debug.cfs_rq:/.nr_spread_over.max
+289.4% 27.06 ± 23% sched_debug.cfs_rq:/.nr_spread_over.min
+39.4% 10.20 ± 19% sched_debug.cfs_rq:/.nr_spread_over.stddev
-27.5% 299.85 ± 14% sched_debug.cfs_rq:/.runnable_avg.avg
-65.7% 286.48 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev
-29.7% 1845657 ± 13% sched_debug.cfs_rq:/.spread0.avg
-39.8% 2699533 ± 11% sched_debug.cfs_rq:/.spread0.max
-94.3% -26940 sched_debug.cfs_rq:/.spread0.min
-56.8% 345835 ± 11% sched_debug.cfs_rq:/.spread0.stddev
-22.5% 288.10 ± 14% sched_debug.cfs_rq:/.util_avg.avg
-32.5% 258.09 ± 4% sched_debug.cfs_rq:/.util_avg.stddev
-54.8% 149.57 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.avg
-27.6% 259.25 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev
+17.6% 4760305 ± 4% sched_debug.cpu.avg_idle.avg
+116.4% 1215479 ± 7% sched_debug.cpu.avg_idle.min
-38.6% 449.23 ± 8% sched_debug.cpu.clock.stddev
-38.6% 449.23 ± 8% sched_debug.cpu.clock_task.stddev
-43.0% 2702 ± 18% sched_debug.cpu.curr->pid.avg
-15.8% 4650 ± 11% sched_debug.cpu.curr->pid.stddev
+16.3% 2465873 ± 4% sched_debug.cpu.max_idle_balance_cost.avg
+24.6% 1380330 ± 3% sched_debug.cpu.max_idle_balance_cost.min
+11.1% 646020 ± 5% sched_debug.cpu.max_idle_balance_cost.stddev
-30.8% 0.00 ± 11% sched_debug.cpu.next_balance.stddev
-32.6% 0.21 ± 10% sched_debug.cpu.nr_running.avg
+46.1% 7477 ± 11% sched_debug.cpu.nr_switches.avg
+85.0% 4257 ± 22% sched_debug.cpu.nr_switches.min
+300.2% 0.34 ± 17% sched_debug.cpu.nr_uninterruptible.avg
+24.1% 367.95 ± 8% sched_debug.cpu.nr_uninterruptible.max
+38.8% -69.98 sched_debug.cpu.nr_uninterruptible.min
+53.3% 35.32 ± 12% sched_debug.cpu.nr_uninterruptible.stddev
+98.2% 4646 ± 18% sched_debug.cpu.sched_count.avg
+193.0% 3252 ± 23% sched_debug.cpu.sched_count.min
+125.6% 2060 ± 18% sched_debug.cpu.sched_goidle.avg
+296.7% 1416 ± 25% sched_debug.cpu.sched_goidle.min
+102.6% 2326 ± 18% sched_debug.cpu.ttwu_count.avg
+80.8% 922.26 ± 21% sched_debug.cpu.ttwu_count.min
-11.3% 131.52 ± 6% sched_debug.cpu.ttwu_local.stddev
-10.0 11.99 ± 11% perf-profile.calltrace.cycles-pp.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault
-8.1 10.96 ± 11% perf-profile.calltrace.cycles-pp.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault
-8.0 10.76 ± 11% perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault
-7.6 10.17 ± 10% perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault
-1.5 0.68 ± 23% perf-profile.calltrace.cycles-pp.alloc_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault
-0.6 0.40 ±102% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.do_access
-0.6 0.32 ±103% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.do_access
-0.2 0.60 ± 14% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
-0.2 0.60 ± 14% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit
-0.2 0.60 ± 14% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput
-0.2 0.59 ± 15% perf-profile.calltrace.cycles-pp.__free_huge_page.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap
+0.6 2.00 ± 28% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
+1.5 2.16 ± 33% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock
+1.6 2.67 ± 32% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock.__mutex_lock
+2.0 3.29 ± 26% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock.__mutex_lock.hugetlb_fault
+2.3 3.66 ± 25% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.osq_lock.__mutex_lock.hugetlb_fault.handle_mm_fault
+20.1 49.83 ± 13% perf-profile.calltrace.cycles-pp.__mutex_lock.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault
+23.2 38.88 ± 18% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.hugetlb_fault.handle_mm_fault.do_page_fault
-10.0 12.00 ± 11% perf-profile.children.cycles-pp.hugetlb_cow
-8.2 11.11 ± 11% perf-profile.children.cycles-pp.copy_user_huge_page
-8.0 10.77 ± 11% perf-profile.children.cycles-pp.copy_subpage
-7.8 10.54 ± 11% perf-profile.children.cycles-pp.copy_page
-7.0 10.00 ± 53% perf-profile.children.cycles-pp.do_rw_once
-1.7 0.80 ± 14% perf-profile.children.cycles-pp.alloc_huge_page
-1.6 1.48 ± 4% perf-profile.children.cycles-pp._raw_spin_lock
-1.5 1.10 ± 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
-1.0 0.57 ± 14% perf-profile.children.cycles-pp.alloc_surplus_huge_page
-0.5 1.34 ± 4% perf-profile.children.cycles-pp.do_syscall_64
-0.5 1.35 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
-0.4 0.38 ± 12% perf-profile.children.cycles-pp.alloc_fresh_huge_page
-0.4 0.85 ± 11% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
-0.3 0.09 ± 20% perf-profile.children.cycles-pp.prep_new_huge_page
-0.2 0.60 ± 14% perf-profile.children.cycles-pp.tlb_finish_mmu
-0.2 0.60 ± 14% perf-profile.children.cycles-pp.tlb_flush_mmu
-0.2 0.60 ± 15% perf-profile.children.cycles-pp.release_pages
-0.2 0.59 ± 15% perf-profile.children.cycles-pp.__free_huge_page
-0.2 0.26 ± 14% perf-profile.children.cycles-pp.___might_sleep
-0.1 0.06 ± 6% perf-profile.children.cycles-pp.update_and_free_page
-0.1 0.21 ± 6% perf-profile.children.cycles-pp.ksys_read
-0.1 0.18 ± 10% perf-profile.children.cycles-pp.read
-0.1 0.20 ± 8% perf-profile.children.cycles-pp.vfs_read
-0.1 0.09 ± 8% perf-profile.children.cycles-pp._cond_resched
-0.1 0.09 ± 17% perf-profile.children.cycles-pp.__libc_start_main
-0.1 0.08 ± 17% perf-profile.children.cycles-pp.main
-0.1 0.12 ± 17% perf-profile.children.cycles-pp.x86_pmu_disable
-0.1 0.19 ± 6% perf-profile.children.cycles-pp.ksys_write
-0.1 0.19 ± 8% perf-profile.children.cycles-pp.vfs_write
-0.1 0.09 ± 20% perf-profile.children.cycles-pp.new_sync_read
-0.1 0.09 ± 17% perf-profile.children.cycles-pp.pipe_read
-0.1 0.18 ± 7% perf-profile.children.cycles-pp.new_sync_write
-0.1 0.03 ±100% perf-profile.children.cycles-pp.__perf_sw_event
-0.0 0.06 ± 14% perf-profile.children.cycles-pp.__wake_up_common
-0.0 0.08 ± 10% perf-profile.children.cycles-pp.schedule_timeout
-0.0 0.07 ± 17% perf-profile.children.cycles-pp.__wake_up_common_lock
-0.0 0.06 ± 15% perf-profile.children.cycles-pp.autoremove_wake_function
-0.0 0.07 ± 10% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
-0.0 0.22 ± 5% perf-profile.children.cycles-pp.tick_program_event
-0.0 0.05 ± 8% perf-profile.children.cycles-pp.prepare_exit_to_usermode
-0.0 0.06 ± 13% perf-profile.children.cycles-pp.rcu_gp_kthread
-0.0 0.06 ± 14% perf-profile.children.cycles-pp.cmd_stat
+0.0 0.06 ± 11% perf-profile.children.cycles-pp.select_task_rq_fair
+0.0 0.16 ± 9% perf-profile.children.cycles-pp.finish_task_switch
+0.1 0.06 ± 14% perf-profile.children.cycles-pp.rwsem_wake
+0.1 0.06 ± 7% perf-profile.children.cycles-pp.arch_stack_walk
+0.1 0.06 ± 13% perf-profile.children.cycles-pp.dequeue_task_fair
+0.1 0.07 ± 13% perf-profile.children.cycles-pp.stack_trace_save_tsk
+0.1 0.21 ± 10% perf-profile.children.cycles-pp.schedule_idle
+0.1 0.14 ± 10% perf-profile.children.cycles-pp.enqueue_entity
+0.1 0.11 ± 7% perf-profile.children.cycles-pp.update_load_avg
+0.1 0.09 ± 12% perf-profile.children.cycles-pp.__account_scheduler_latency
+0.1 0.16 ± 11% perf-profile.children.cycles-pp.activate_task
+0.1 0.16 ± 11% perf-profile.children.cycles-pp.ttwu_do_activate
+0.1 0.16 ± 11% perf-profile.children.cycles-pp.enqueue_task_fair
+0.1 0.09 ± 8% perf-profile.children.cycles-pp.wake_up_q
+0.1 0.27 ± 13% perf-profile.children.cycles-pp.schedule_preempt_disabled
+0.1 0.14 ± 15% perf-profile.children.cycles-pp.sched_ttwu_pending
+0.4 1.56 ± 8% perf-profile.children.cycles-pp.load_balance
+0.4 1.41 ± 9% perf-profile.children.cycles-pp.find_busiest_group
+0.4 1.38 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats
+0.5 1.20 ± 11% perf-profile.children.cycles-pp.newidle_balance
+0.6 1.11 ± 12% perf-profile.children.cycles-pp.pick_next_task_fair
+0.6 1.35 ± 10% perf-profile.children.cycles-pp.schedule
+0.7 0.68 ± 17% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
+0.7 1.58 ± 9% perf-profile.children.cycles-pp.__schedule
+10.7 62.86 ± 9% perf-profile.children.cycles-pp.page_fault
+10.7 62.83 ± 9% perf-profile.children.cycles-pp.do_page_fault
+10.8 62.74 ± 9% perf-profile.children.cycles-pp.handle_mm_fault
+10.8 62.67 ± 9% perf-profile.children.cycles-pp.hugetlb_fault
+20.1 49.84 ± 13% perf-profile.children.cycles-pp.__mutex_lock
+23.4 39.18 ± 18% perf-profile.children.cycles-pp.osq_lock
-7.1 9.48 ± 9% perf-profile.self.cycles-pp.copy_page
-6.4 8.90 ± 53% perf-profile.self.cycles-pp.do_rw_once
-1.4 1.02 ± 7% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
-0.2 0.48 ± 14% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
-0.2 0.21 ± 11% perf-profile.self.cycles-pp.copy_subpage
-0.2 0.17 ± 11% perf-profile.self.cycles-pp.copy_user_huge_page
-0.1 0.05 ± 8% perf-profile.self.cycles-pp.update_and_free_page
-0.1 0.11 ± 18% perf-profile.self.cycles-pp.x86_pmu_disable
-0.1 0.12 ± 10% perf-profile.self.cycles-pp.___might_sleep
-0.1 0.28 ± 8% perf-profile.self.cycles-pp.run_timer_softirq
-0.1 0.12 ± 6% perf-profile.self.cycles-pp.get_page_from_freelist
-0.0 0.22 ± 5% perf-profile.self.cycles-pp.tick_program_event
-0.0 0.06 ± 7% perf-profile.self.cycles-pp.try_to_wake_up
+0.1 0.06 ± 9% perf-profile.self.cycles-pp.update_load_avg
+0.3 1.06 ± 8% perf-profile.self.cycles-pp.update_sd_lb_stats
+21.2 35.60 ± 18% perf-profile.self.cycles-pp.osq_lock
+11.6% 25420 ± 2% softirqs.CPU0.SCHED
+24.5% 19248 ± 5% softirqs.CPU1.SCHED
+27.1% 10590 ± 11% softirqs.CPU10.RCU
+16.0% 17145 ± 4% softirqs.CPU10.SCHED
+15.6% 16462 ± 2% softirqs.CPU100.SCHED
+18.2% 16683 ± 3% softirqs.CPU101.SCHED
+18.4% 16472 ± 5% softirqs.CPU102.SCHED
+18.0% 16742 ± 3% softirqs.CPU103.SCHED
+19.8% 16528 softirqs.CPU104.SCHED
+17.7% 16859 softirqs.CPU105.SCHED
+19.4% 16530 ± 2% softirqs.CPU106.SCHED
+20.0% 16983 ± 2% softirqs.CPU107.SCHED
+18.6% 16492 ± 2% softirqs.CPU108.SCHED
+16.1% 16726 ± 2% softirqs.CPU109.SCHED
+19.0% 17085 ± 2% softirqs.CPU11.SCHED
+21.8% 16667 ± 3% softirqs.CPU110.SCHED
+25.6% 16907 ± 3% softirqs.CPU111.SCHED
+23.2% 16925 softirqs.CPU112.SCHED
+20.5% 17032 softirqs.CPU113.SCHED
+20.8% 17491 ± 4% softirqs.CPU114.SCHED
-7.8% 113426 ± 2% softirqs.CPU114.TIMER
+23.4% 17299 softirqs.CPU115.SCHED
+24.4% 16824 ± 2% softirqs.CPU116.SCHED
+24.5% 16840 ± 2% softirqs.CPU117.SCHED
+16.1% 16557 ± 2% softirqs.CPU119.SCHED
+22.1% 9648 ± 6% softirqs.CPU12.RCU
+19.5% 17130 ± 4% softirqs.CPU12.SCHED
+26.5% 17047 softirqs.CPU120.SCHED
+20.7% 17183 softirqs.CPU121.SCHED
+27.0% 16708 ± 2% softirqs.CPU122.SCHED
+28.8% 16786 softirqs.CPU123.SCHED
+16.5% 16577 ± 3% softirqs.CPU124.SCHED
+15.4% 16652 softirqs.CPU125.SCHED
+16.3% 16771 softirqs.CPU126.SCHED
+39.3% 9305 ± 22% softirqs.CPU127.RCU
+23.7% 16890 ± 3% softirqs.CPU128.SCHED
+17.9% 16979 ± 3% softirqs.CPU129.SCHED
+20.2% 17048 ± 2% softirqs.CPU13.SCHED
+35.8% 9097 ± 17% softirqs.CPU130.RCU
+22.2% 17051 ± 4% softirqs.CPU130.SCHED
+21.8% 17379 ± 2% softirqs.CPU131.SCHED
-7.0% 120290 softirqs.CPU131.TIMER
+16.0% 16396 ± 2% softirqs.CPU132.SCHED
+22.5% 16617 ± 3% softirqs.CPU133.SCHED
+20.1% 16712 ± 2% softirqs.CPU134.SCHED
+18.9% 17016 ± 2% softirqs.CPU135.SCHED
+20.3% 16402 ± 5% softirqs.CPU136.SCHED
+19.3% 16518 ± 2% softirqs.CPU137.SCHED
+19.2% 16750 ± 2% softirqs.CPU138.SCHED
+18.1% 16816 softirqs.CPU139.SCHED
+22.1% 17271 ± 2% softirqs.CPU14.SCHED
+22.9% 16546 softirqs.CPU140.SCHED
+21.7% 16601 ± 2% softirqs.CPU141.SCHED
+25.5% 16661 ± 3% softirqs.CPU142.SCHED
+29.2% 16136 ± 2% softirqs.CPU143.SCHED
+22.8% 16502 softirqs.CPU144.SCHED
-8.1% 115836 softirqs.CPU144.TIMER
+21.2% 16966 ± 3% softirqs.CPU145.SCHED
+14.8% 15883 ± 6% softirqs.CPU146.SCHED
+17.1% 16735 softirqs.CPU147.SCHED
+15.2% 16711 softirqs.CPU148.SCHED
+17.5% 16945 softirqs.CPU149.SCHED
+24.4% 9977 ± 7% softirqs.CPU15.RCU
+28.2% 17737 ± 5% softirqs.CPU15.SCHED
+15.0% 16723 ± 2% softirqs.CPU150.SCHED
+9.5% 16526 ± 4% softirqs.CPU152.SCHED
+15.6% 16804 softirqs.CPU153.SCHED
+14.8% 16554 ± 2% softirqs.CPU154.SCHED
+15.4% 16833 ± 2% softirqs.CPU155.SCHED
+11.8% 16598 ± 2% softirqs.CPU156.SCHED
+35.2% 9124 ± 19% softirqs.CPU158.RCU
+25.9% 17429 ± 4% softirqs.CPU158.SCHED
+11.8% 16785 ± 2% softirqs.CPU159.SCHED
+25.8% 16945 ± 2% softirqs.CPU16.SCHED
-8.1% 117855 softirqs.CPU160.TIMER
+13.7% 17184 softirqs.CPU162.SCHED
+12.8% 16685 ± 3% softirqs.CPU163.SCHED
+21.5% 16889 softirqs.CPU164.SCHED
+13.5% 16510 ± 2% softirqs.CPU165.SCHED
+13.3% 16354 ± 3% softirqs.CPU166.SCHED
+14.1% 16494 ± 2% softirqs.CPU167.SCHED
-6.9% 122396 ± 2% softirqs.CPU167.TIMER
+23.0% 16824 ± 3% softirqs.CPU168.SCHED
+18.1% 17085 ± 3% softirqs.CPU169.SCHED
+22.5% 9437 ± 4% softirqs.CPU17.RCU
+19.8% 17237 softirqs.CPU17.SCHED
+16.9% 17038 ± 3% softirqs.CPU170.SCHED
+19.3% 16905 softirqs.CPU171.SCHED
+17.9% 16709 softirqs.CPU172.SCHED
-8.4% 116497 ± 2% softirqs.CPU172.TIMER
+11.1% 16698 ± 2% softirqs.CPU174.SCHED
+17.0% 17458 ± 5% softirqs.CPU175.SCHED
+18.8% 16833 ± 2% softirqs.CPU176.SCHED
-8.4% 116378 ± 2% softirqs.CPU176.TIMER
+16.9% 16788 ± 4% softirqs.CPU177.SCHED
+18.7% 17169 softirqs.CPU178.SCHED
+15.1% 16837 softirqs.CPU179.SCHED
-8.2% 113392 ± 2% softirqs.CPU179.TIMER
+27.6% 9794 ± 2% softirqs.CPU18.RCU
+21.6% 17233 softirqs.CPU18.SCHED
+15.8% 16707 softirqs.CPU180.SCHED
+12.4% 17084 softirqs.CPU181.SCHED
+20.2% 16723 ± 3% softirqs.CPU182.SCHED
+15.1% 16786 ± 3% softirqs.CPU183.SCHED
+18.0% 16827 ± 2% softirqs.CPU184.SCHED
+16.3% 17053 softirqs.CPU185.SCHED
+21.0% 17202 ± 3% softirqs.CPU186.SCHED
+13.0% 17022 ± 3% softirqs.CPU187.SCHED
-7.6% 115887 ± 2% softirqs.CPU187.TIMER
+25.0% 16801 ± 3% softirqs.CPU188.SCHED
+28.8% 17297 ± 2% softirqs.CPU189.SCHED
+27.0% 9806 ± 9% softirqs.CPU19.RCU
+20.7% 16986 softirqs.CPU19.SCHED
+20.4% 16536 ± 2% softirqs.CPU190.SCHED
+19.8% 16494 softirqs.CPU192.SCHED
+17.4% 16327 ± 2% softirqs.CPU193.SCHED
+34.3% 8616 ± 17% softirqs.CPU194.RCU
+17.6% 16468 ± 8% softirqs.CPU194.SCHED
+19.9% 16905 softirqs.CPU195.SCHED
+16.1% 16698 ± 5% softirqs.CPU196.SCHED
+23.7% 16952 softirqs.CPU198.SCHED
+23.7% 10157 ± 7% softirqs.CPU2.RCU
+17.4% 16946 ± 4% softirqs.CPU2.SCHED
+30.4% 10135 ± 8% softirqs.CPU20.RCU
+23.7% 17320 softirqs.CPU20.SCHED
+12.7% 16463 ± 3% softirqs.CPU200.SCHED
+15.5% 17199 softirqs.CPU201.SCHED
+21.7% 17167 ± 2% softirqs.CPU202.SCHED
-7.1% 118685 softirqs.CPU202.TIMER
+18.6% 17048 ± 2% softirqs.CPU203.SCHED
-8.8% 115539 softirqs.CPU203.TIMER
+13.1% 16356 ± 3% softirqs.CPU205.SCHED
+26.6% 16634 ± 3% softirqs.CPU206.SCHED
+11.8% 16550 ± 2% softirqs.CPU207.SCHED
+21.0% 16624 ± 4% softirqs.CPU208.SCHED
+18.7% 16661 ± 2% softirqs.CPU209.SCHED
-8.2% 116998 softirqs.CPU209.TIMER
+30.5% 10122 ± 12% softirqs.CPU21.RCU
+25.2% 17185 ± 2% softirqs.CPU21.SCHED
+16.1% 17123 ± 4% softirqs.CPU211.SCHED
+18.5% 16501 ± 2% softirqs.CPU212.SCHED
+16.1% 16765 ± 3% softirqs.CPU213.SCHED
+21.6% 16844 ± 2% softirqs.CPU214.SCHED
+16.5% 15635 ± 2% softirqs.CPU215.SCHED
-7.4% 118224 softirqs.CPU215.TIMER
+21.7% 16686 ± 3% softirqs.CPU216.SCHED
-7.8% 116544 softirqs.CPU216.TIMER
+18.2% 16416 softirqs.CPU217.SCHED
+20.5% 16875 softirqs.CPU218.SCHED
+23.9% 17210 ± 4% softirqs.CPU22.SCHED
+12.0% 16694 ± 2% softirqs.CPU220.SCHED
+17.3% 17153 softirqs.CPU221.SCHED
+13.9% 16643 ± 2% softirqs.CPU224.SCHED
+12.9% 16643 softirqs.CPU225.SCHED
-8.3% 115717 softirqs.CPU225.TIMER
+17.2% 16753 softirqs.CPU226.SCHED
+10.3% 16800 ± 3% softirqs.CPU228.SCHED
+15.4% 16479 ± 3% softirqs.CPU23.SCHED
+24.2% 17126 ± 4% softirqs.CPU230.SCHED
-9.4% 114889 ± 3% softirqs.CPU231.TIMER
+15.0% 16958 ± 3% softirqs.CPU232.SCHED
+11.9% 16893 ± 2% softirqs.CPU233.SCHED
+12.9% 16738 softirqs.CPU234.SCHED
+18.7% 16945 ± 2% softirqs.CPU236.SCHED
+10.9% 16846 softirqs.CPU237.SCHED
+11.1% 16955 ± 3% softirqs.CPU239.SCHED
+16.3% 16466 ± 3% softirqs.CPU24.SCHED
+13.9% 16711 ± 3% softirqs.CPU240.SCHED
-7.6% 117651 ± 2% softirqs.CPU240.TIMER
+20.2% 17097 softirqs.CPU241.SCHED
+15.2% 16771 softirqs.CPU242.SCHED
+19.9% 16785 softirqs.CPU243.SCHED
+21.5% 16724 ± 2% softirqs.CPU244.SCHED
+16.4% 16781 softirqs.CPU245.SCHED
+14.8% 16940 ± 4% softirqs.CPU246.SCHED
+12.3% 16722 ± 4% softirqs.CPU249.SCHED
+24.1% 17148 softirqs.CPU25.SCHED
+18.1% 16774 ± 3% softirqs.CPU250.SCHED
+18.5% 17331 softirqs.CPU251.SCHED
+11.3% 16564 ± 2% softirqs.CPU252.SCHED
+16.3% 16877 ± 3% softirqs.CPU254.SCHED
+18.0% 16858 ± 4% softirqs.CPU255.SCHED
-7.5% 112473 softirqs.CPU255.TIMER
+14.8% 16736 softirqs.CPU256.SCHED
+22.8% 17259 softirqs.CPU257.SCHED
+13.4% 17097 ± 3% softirqs.CPU258.SCHED
+20.2% 17251 ± 3% softirqs.CPU259.SCHED
+20.1% 16806 ± 2% softirqs.CPU26.SCHED
+19.0% 16534 ± 5% softirqs.CPU260.SCHED
-7.4% 118758 softirqs.CPU260.TIMER
+20.7% 17251 softirqs.CPU261.SCHED
+15.1% 16644 softirqs.CPU262.SCHED
+17.8% 16624 ± 4% softirqs.CPU263.SCHED
+25.0% 16657 softirqs.CPU264.SCHED
+25.1% 16676 ± 4% softirqs.CPU266.SCHED
+19.5% 17165 ± 2% softirqs.CPU267.SCHED
-7.6% 113293 ± 2% softirqs.CPU268.TIMER
+25.2% 16940 softirqs.CPU27.SCHED
+19.8% 17187 softirqs.CPU270.SCHED
+14.3% 16403 ± 5% softirqs.CPU271.SCHED
+14.4% 17152 ± 4% softirqs.CPU272.SCHED
+13.4% 17012 softirqs.CPU273.SCHED
+17.0% 17157 ± 4% softirqs.CPU274.SCHED
+17.2% 16749 ± 4% softirqs.CPU277.SCHED
+41.8% 8938 ± 16% softirqs.CPU278.RCU
+24.6% 17424 ± 2% softirqs.CPU278.SCHED
+16.0% 17119 softirqs.CPU279.SCHED
+23.9% 16950 softirqs.CPU28.SCHED
+17.0% 16617 ± 3% softirqs.CPU280.SCHED
+14.4% 16240 ± 2% softirqs.CPU281.SCHED
+16.3% 17013 ± 3% softirqs.CPU283.SCHED
+22.7% 16795 softirqs.CPU284.SCHED
+17.5% 16679 ± 3% softirqs.CPU285.SCHED
-8.1% 113741 ± 3% softirqs.CPU285.TIMER
+15.5% 17050 softirqs.CPU286.SCHED
+17.2% 15894 softirqs.CPU287.SCHED
+19.9% 17020 ± 2% softirqs.CPU29.SCHED
+20.5% 17292 softirqs.CPU3.SCHED
+20.9% 16814 ± 4% softirqs.CPU30.SCHED
+41.8% 10759 ± 23% softirqs.CPU31.RCU
+26.6% 16967 ± 2% softirqs.CPU31.SCHED
+24.6% 16937 softirqs.CPU32.SCHED
+29.0% 16989 softirqs.CPU33.SCHED
+25.8% 16974 ± 3% softirqs.CPU34.SCHED
+25.9% 17018 ± 2% softirqs.CPU35.SCHED
+22.1% 16763 ± 2% softirqs.CPU36.SCHED
+27.1% 17091 softirqs.CPU37.SCHED
+40.0% 10368 ± 17% softirqs.CPU38.RCU
+22.4% 17509 ± 3% softirqs.CPU38.SCHED
+18.6% 16918 softirqs.CPU39.SCHED
+23.6% 9862 ± 5% softirqs.CPU4.RCU
+23.4% 17349 ± 3% softirqs.CPU4.SCHED
+19.4% 17030 ± 2% softirqs.CPU40.SCHED
+24.1% 17090 softirqs.CPU41.SCHED
+17.7% 16976 softirqs.CPU42.SCHED
+21.2% 17345 ± 2% softirqs.CPU43.SCHED
+27.7% 16954 softirqs.CPU44.SCHED
-15.7% 114264 ± 3% softirqs.CPU44.TIMER
+20.9% 16829 softirqs.CPU45.SCHED
+27.9% 16834 ± 2% softirqs.CPU46.SCHED
+22.0% 16614 softirqs.CPU47.SCHED
+25.7% 17054 ± 2% softirqs.CPU48.SCHED
+21.6% 16769 softirqs.CPU49.SCHED
+20.1% 17126 ± 4% softirqs.CPU5.SCHED
+25.3% 16934 softirqs.CPU50.SCHED
+30.3% 16669 softirqs.CPU51.SCHED
+19.9% 16774 ± 3% softirqs.CPU52.SCHED
+17.1% 16722 ± 2% softirqs.CPU53.SCHED
+36.9% 9940 ± 21% softirqs.CPU54.RCU
+20.1% 16918 softirqs.CPU54.SCHED
+23.1% 16629 softirqs.CPU55.SCHED
-9.4% 118014 ± 7% softirqs.CPU55.TIMER
+27.9% 17030 ± 2% softirqs.CPU56.SCHED
+33.2% 9803 ± 12% softirqs.CPU57.RCU
+21.3% 16768 ± 2% softirqs.CPU57.SCHED
+27.1% 9274 ± 7% softirqs.CPU58.RCU
+23.4% 17181 ± 2% softirqs.CPU58.SCHED
+30.3% 9476 ± 11% softirqs.CPU59.RCU
+19.7% 16956 ± 2% softirqs.CPU59.SCHED
+16.5% 9524 ± 3% softirqs.CPU6.RCU
+25.2% 17132 ± 5% softirqs.CPU6.SCHED
+23.3% 16825 softirqs.CPU60.SCHED
+24.9% 16786 ± 2% softirqs.CPU61.SCHED
+25.7% 16959 softirqs.CPU62.SCHED
+25.5% 17170 softirqs.CPU63.SCHED
+22.3% 16695 ± 3% softirqs.CPU64.SCHED
+22.7% 16819 softirqs.CPU65.SCHED
+23.0% 16826 softirqs.CPU66.SCHED
+19.3% 16797 ± 2% softirqs.CPU67.SCHED
+18.2% 16215 ± 2% softirqs.CPU68.SCHED
+25.1% 16754 ± 2% softirqs.CPU69.SCHED
+21.0% 9730 ± 7% softirqs.CPU7.RCU
+27.3% 17320 ± 5% softirqs.CPU7.SCHED
+29.2% 16828 ± 2% softirqs.CPU70.SCHED
+32.1% 16255 softirqs.CPU71.SCHED
+37.3% 10612 ± 22% softirqs.CPU72.RCU
+28.0% 17050 ± 2% softirqs.CPU72.SCHED
+22.0% 16883 ± 3% softirqs.CPU73.SCHED
+14.4% 16707 ± 4% softirqs.CPU74.SCHED
+19.1% 16892 ± 2% softirqs.CPU75.SCHED
+19.0% 16830 softirqs.CPU76.SCHED
+28.1% 9090 ± 12% softirqs.CPU77.RCU
+28.0% 16741 softirqs.CPU77.SCHED
-10.8% 112765 ± 2% softirqs.CPU78.TIMER
+12.6% 16576 ± 3% softirqs.CPU79.SCHED
+27.1% 10119 ± 10% softirqs.CPU8.RCU
+25.1% 17232 ± 5% softirqs.CPU8.SCHED
+13.6% 16355 ± 2% softirqs.CPU80.SCHED
+20.7% 16896 ± 2% softirqs.CPU81.SCHED
+18.2% 16706 softirqs.CPU82.SCHED
+17.6% 16385 ± 3% softirqs.CPU83.SCHED
+16.1% 16776 ± 2% softirqs.CPU84.SCHED
+16.4% 16799 softirqs.CPU86.SCHED
+28.2% 9036 ± 10% softirqs.CPU88.RCU
+20.1% 16697 ± 2% softirqs.CPU88.SCHED
-7.3% 118017 softirqs.CPU88.TIMER
+17.4% 16855 ± 2% softirqs.CPU89.SCHED
+14.7% 17004 softirqs.CPU9.SCHED
+18.4% 16908 softirqs.CPU90.SCHED
+18.8% 16894 ± 2% softirqs.CPU91.SCHED
+26.7% 8987 ± 10% softirqs.CPU92.RCU
+24.0% 17014 ± 3% softirqs.CPU92.SCHED
+20.6% 16939 softirqs.CPU93.SCHED
+17.9% 16649 ± 3% softirqs.CPU94.SCHED
-8.2% 119400 softirqs.CPU95.TIMER
+23.5% 16847 ± 2% softirqs.CPU96.SCHED
+29.4% 17001 ± 3% softirqs.CPU97.SCHED
+25.7% 16791 ± 2% softirqs.CPU98.SCHED
+20.0% 16938 softirqs.CPU99.SCHED
+21.8% 2426642 ± 3% softirqs.RCU
+18.4% 4855612 softirqs.SCHED
+675.8% 3522 ±139% interrupts.30:IR-PCI-MSI.2097153-edge.eth0-TxRx-0
-1.6% 378485 interrupts.CAL:Function_call_interrupts
-46.7% 496.00 ± 10% interrupts.CPU0.TLB:TLB_shootdowns
-44.8% 418.75 ± 11% interrupts.CPU1.TLB:TLB_shootdowns
+675.8% 3522 ±139% interrupts.CPU10.30:IR-PCI-MSI.2097153-edge.eth0-TxRx-0
-39.4% 537.00 ± 3% interrupts.CPU10.TLB:TLB_shootdowns
+67.0% 283.00 ± 9% interrupts.CPU100.TLB:TLB_shootdowns
+44.0% 257.00 ± 9% interrupts.CPU101.TLB:TLB_shootdowns
+62.7% 275.00 ± 20% interrupts.CPU104.TLB:TLB_shootdowns
+88.1% 2753 ± 22% interrupts.CPU105.NMI:Non-maskable_interrupts
+88.1% 2753 ± 22% interrupts.CPU105.PMI:Performance_monitoring_interrupts
+73.4% 275.25 ± 4% interrupts.CPU106.TLB:TLB_shootdowns
+67.3% 263.50 ± 13% interrupts.CPU109.TLB:TLB_shootdowns
-43.7% 1625 ± 6% interrupts.CPU11.NMI:Non-maskable_interrupts
-43.7% 1625 ± 6% interrupts.CPU11.PMI:Performance_monitoring_interrupts
-38.8% 436.50 ± 13% interrupts.CPU11.TLB:TLB_shootdowns
+79.3% 261.75 ± 14% interrupts.CPU110.TLB:TLB_shootdowns
+82.6% 295.75 ± 12% interrupts.CPU112.TLB:TLB_shootdowns
+170.1% 1204 ± 91% interrupts.CPU114.RES:Rescheduling_interrupts
+70.8% 273.75 ± 14% interrupts.CPU114.TLB:TLB_shootdowns
+66.0% 2352 ± 34% interrupts.CPU115.NMI:Non-maskable_interrupts
+66.0% 2352 ± 34% interrupts.CPU115.PMI:Performance_monitoring_interrupts
+37.9% 525.00 ± 15% interrupts.CPU116.RES:Rescheduling_interrupts
+48.2% 271.25 ± 14% interrupts.CPU116.TLB:TLB_shootdowns
+74.5% 297.50 ± 5% interrupts.CPU118.TLB:TLB_shootdowns
+251.3% 1596 ±103% interrupts.CPU12.RES:Rescheduling_interrupts
-36.9% 534.25 ± 10% interrupts.CPU12.TLB:TLB_shootdowns
+65.7% 287.00 ± 13% interrupts.CPU122.TLB:TLB_shootdowns
+80.5% 759.00 ± 37% interrupts.CPU123.RES:Rescheduling_interrupts
+75.9% 435.75 ± 14% interrupts.CPU124.RES:Rescheduling_interrupts
+58.7% 288.50 ± 15% interrupts.CPU126.TLB:TLB_shootdowns
+39.1% 507.50 ± 11% interrupts.CPU128.RES:Rescheduling_interrupts
+248.5% 1177 ± 74% interrupts.CPU13.RES:Rescheduling_interrupts
-47.8% 409.00 ± 3% interrupts.CPU13.TLB:TLB_shootdowns
+84.9% 512.25 ± 10% interrupts.CPU131.RES:Rescheduling_interrupts
+71.7% 267.50 ± 21% interrupts.CPU132.TLB:TLB_shootdowns
+50.4% 262.00 ± 19% interrupts.CPU133.TLB:TLB_shootdowns
+116.4% 296.50 ± 11% interrupts.CPU134.TLB:TLB_shootdowns
+86.3% 283.25 ± 17% interrupts.CPU138.TLB:TLB_shootdowns
-32.5% 553.75 ± 3% interrupts.CPU14.TLB:TLB_shootdowns
+81.1% 503.00 ± 7% interrupts.CPU140.RES:Rescheduling_interrupts
+59.3% 475.00 ± 15% interrupts.CPU142.RES:Rescheduling_interrupts
+38.6% 266.75 ± 4% interrupts.CPU143.TLB:TLB_shootdowns
+103.4% 590.00 ± 18% interrupts.CPU145.RES:Rescheduling_interrupts
+90.4% 199.00 ± 16% interrupts.CPU145.TLB:TLB_shootdowns
+96.9% 446.00 ± 15% interrupts.CPU146.RES:Rescheduling_interrupts
+145.4% 235.00 ± 12% interrupts.CPU146.TLB:TLB_shootdowns
+65.2% 239.50 ± 16% interrupts.CPU147.TLB:TLB_shootdowns
+212.2% 281.75 ± 19% interrupts.CPU148.TLB:TLB_shootdowns
+142.7% 253.00 ± 11% interrupts.CPU149.TLB:TLB_shootdowns
-35.6% 1860 ± 30% interrupts.CPU15.NMI:Non-maskable_interrupts
-35.6% 1860 ± 30% interrupts.CPU15.PMI:Performance_monitoring_interrupts
+275.2% 1771 ±106% interrupts.CPU15.RES:Rescheduling_interrupts
-50.0% 408.50 ± 11% interrupts.CPU15.TLB:TLB_shootdowns
+207.4% 260.50 ± 10% interrupts.CPU150.TLB:TLB_shootdowns
+108.6% 278.50 ± 17% interrupts.CPU151.TLB:TLB_shootdowns
+132.0% 240.75 ± 8% interrupts.CPU152.TLB:TLB_shootdowns
+155.0% 267.75 ± 7% interrupts.CPU153.TLB:TLB_shootdowns
+133.1% 234.25 ± 8% interrupts.CPU154.TLB:TLB_shootdowns
+220.4% 235.50 ± 11% interrupts.CPU155.TLB:TLB_shootdowns
+182.1% 647.50 ± 8% interrupts.CPU156.RES:Rescheduling_interrupts
+110.9% 241.50 ± 29% interrupts.CPU156.TLB:TLB_shootdowns
+79.7% 226.00 ± 8% interrupts.CPU157.TLB:TLB_shootdowns
+143.1% 237.00 ± 16% interrupts.CPU158.TLB:TLB_shootdowns
+136.4% 224.00 ± 21% interrupts.CPU159.TLB:TLB_shootdowns
+89.2% 855.25 ± 39% interrupts.CPU16.RES:Rescheduling_interrupts
-54.3% 438.25 ± 8% interrupts.CPU16.TLB:TLB_shootdowns
+145.0% 237.00 ± 9% interrupts.CPU161.TLB:TLB_shootdowns
+169.5% 254.00 ± 10% interrupts.CPU162.TLB:TLB_shootdowns
+141.8% 243.00 ± 14% interrupts.CPU163.TLB:TLB_shootdowns
+186.5% 249.25 ± 8% interrupts.CPU164.TLB:TLB_shootdowns
+145.7% 229.75 ± 9% interrupts.CPU165.TLB:TLB_shootdowns
+165.4% 293.25 ± 15% interrupts.CPU166.TLB:TLB_shootdowns
+105.7% 665.00 ± 39% interrupts.CPU167.RES:Rescheduling_interrupts
+140.3% 232.50 ± 14% interrupts.CPU167.TLB:TLB_shootdowns
+140.8% 539.50 ± 40% interrupts.CPU168.RES:Rescheduling_interrupts
+151.1% 219.75 ± 14% interrupts.CPU168.TLB:TLB_shootdowns
+79.9% 2015 ± 27% interrupts.CPU169.NMI:Non-maskable_interrupts
+79.9% 2015 ± 27% interrupts.CPU169.PMI:Performance_monitoring_interrupts
+117.5% 218.00 ± 9% interrupts.CPU169.TLB:TLB_shootdowns
-30.3% 465.50 ± 14% interrupts.CPU17.TLB:TLB_shootdowns
+252.7% 264.50 ± 22% interrupts.CPU170.TLB:TLB_shootdowns
+126.0% 213.00 ± 13% interrupts.CPU171.TLB:TLB_shootdowns
+53.3% 488.00 ± 18% interrupts.CPU172.RES:Rescheduling_interrupts
+221.1% 273.75 ± 19% interrupts.CPU172.TLB:TLB_shootdowns
+130.7% 189.75 ± 20% interrupts.CPU173.TLB:TLB_shootdowns
+261.6% 245.00 ± 12% interrupts.CPU174.TLB:TLB_shootdowns
+588.6% 1568 ± 95% interrupts.CPU175.RES:Rescheduling_interrupts
+150.8% 229.50 ± 10% interrupts.CPU175.TLB:TLB_shootdowns
+147.2% 246.00 ± 22% interrupts.CPU176.TLB:TLB_shootdowns
+200.0% 237.75 ± 27% interrupts.CPU177.TLB:TLB_shootdowns
+129.7% 201.00 ± 4% interrupts.CPU178.TLB:TLB_shootdowns
+82.7% 232.50 ± 15% interrupts.CPU179.TLB:TLB_shootdowns
+137.6% 2758 ± 27% interrupts.CPU18.NMI:Non-maskable_interrupts
+137.6% 2758 ± 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts
+149.5% 1062 ± 64% interrupts.CPU18.RES:Rescheduling_interrupts
-50.1% 492.25 ± 13% interrupts.CPU18.TLB:TLB_shootdowns
+120.7% 247.75 ± 8% interrupts.CPU180.TLB:TLB_shootdowns
+96.0% 247.50 ± 9% interrupts.CPU181.TLB:TLB_shootdowns
+74.6% 400.25 ± 7% interrupts.CPU182.RES:Rescheduling_interrupts
+173.4% 280.25 ± 4% interrupts.CPU182.TLB:TLB_shootdowns
+52.6% 386.50 ± 8% interrupts.CPU183.RES:Rescheduling_interrupts
+153.4% 245.75 ± 4% interrupts.CPU183.TLB:TLB_shootdowns
+143.7% 241.25 ± 18% interrupts.CPU184.TLB:TLB_shootdowns
+57.0% 451.25 ± 32% interrupts.CPU185.RES:Rescheduling_interrupts
+139.2% 221.25 ± 5% interrupts.CPU185.TLB:TLB_shootdowns
+37.4% 3162 ± 5% interrupts.CPU186.NMI:Non-maskable_interrupts
+37.4% 3162 ± 5% interrupts.CPU186.PMI:Performance_monitoring_interrupts
+204.0% 266.75 ± 20% interrupts.CPU186.TLB:TLB_shootdowns
+157.0% 249.25 ± 10% interrupts.CPU187.TLB:TLB_shootdowns
+54.2% 403.25 ± 15% interrupts.CPU188.RES:Rescheduling_interrupts
+103.7% 221.00 ± 26% interrupts.CPU188.TLB:TLB_shootdowns
+88.4% 421.50 ± 18% interrupts.CPU189.RES:Rescheduling_interrupts
+108.4% 248.50 ± 25% interrupts.CPU189.TLB:TLB_shootdowns
+117.4% 870.25 ± 52% interrupts.CPU19.RES:Rescheduling_interrupts
-31.8% 476.75 ± 10% interrupts.CPU19.TLB:TLB_shootdowns
+148.6% 248.00 ± 12% interrupts.CPU190.TLB:TLB_shootdowns
+146.6% 242.25 ± 17% interrupts.CPU191.TLB:TLB_shootdowns
+145.4% 258.25 ± 19% interrupts.CPU192.TLB:TLB_shootdowns
+191.3% 260.75 ± 17% interrupts.CPU193.TLB:TLB_shootdowns
+285.7% 255.50 ± 17% interrupts.CPU194.TLB:TLB_shootdowns
+84.9% 515.50 ± 32% interrupts.CPU195.RES:Rescheduling_interrupts
+247.8% 278.25 ± 14% interrupts.CPU195.TLB:TLB_shootdowns
+246.3% 243.25 ± 11% interrupts.CPU196.TLB:TLB_shootdowns
+147.0% 258.75 ± 23% interrupts.CPU197.TLB:TLB_shootdowns
+188.2% 250.00 ± 20% interrupts.CPU198.TLB:TLB_shootdowns
+150.1% 235.75 ± 19% interrupts.CPU199.TLB:TLB_shootdowns
-52.9% 1453 ± 6% interrupts.CPU2.NMI:Non-maskable_interrupts
-52.9% 1453 ± 6% interrupts.CPU2.PMI:Performance_monitoring_interrupts
+179.6% 1555 ± 83% interrupts.CPU2.RES:Rescheduling_interrupts
-42.6% 511.50 ± 13% interrupts.CPU2.TLB:TLB_shootdowns
+193.9% 1072 ± 40% interrupts.CPU20.RES:Rescheduling_interrupts
-35.7% 530.75 ± 6% interrupts.CPU20.TLB:TLB_shootdowns
+83.0% 236.50 ± 7% interrupts.CPU200.TLB:TLB_shootdowns
+200.0% 265.50 ± 12% interrupts.CPU201.TLB:TLB_shootdowns
+99.8% 233.25 ± 24% interrupts.CPU202.TLB:TLB_shootdowns
+121.8% 564.00 ± 14% interrupts.CPU203.RES:Rescheduling_interrupts
+203.0% 252.25 ± 18% interrupts.CPU203.TLB:TLB_shootdowns
+195.2% 325.50 ± 35% interrupts.CPU204.TLB:TLB_shootdowns
+60.5% 532.75 ± 15% interrupts.CPU205.RES:Rescheduling_interrupts
+165.4% 236.25 ± 11% interrupts.CPU205.TLB:TLB_shootdowns
+112.8% 502.75 ± 10% interrupts.CPU206.RES:Rescheduling_interrupts
+195.2% 244.25 ± 13% interrupts.CPU206.TLB:TLB_shootdowns
+86.6% 226.25 ± 18% interrupts.CPU207.TLB:TLB_shootdowns
+170.7% 226.75 ± 5% interrupts.CPU208.TLB:TLB_shootdowns
+131.5% 764.50 ± 44% interrupts.CPU209.RES:Rescheduling_interrupts
+217.8% 240.75 ± 9% interrupts.CPU209.TLB:TLB_shootdowns
-39.2% 473.00 ± 34% interrupts.CPU21.TLB:TLB_shootdowns
+178.4% 748.25 ± 37% interrupts.CPU210.RES:Rescheduling_interrupts
+100.0% 229.50 ± 13% interrupts.CPU210.TLB:TLB_shootdowns
+101.1% 225.25 ± 15% interrupts.CPU211.TLB:TLB_shootdowns
+159.3% 251.50 ± 21% interrupts.CPU212.TLB:TLB_shootdowns
+80.3% 475.50 ± 15% interrupts.CPU213.RES:Rescheduling_interrupts
+98.5% 201.00 ± 28% interrupts.CPU215.TLB:TLB_shootdowns
+70.6% 2177 ± 30% interrupts.CPU216.NMI:Non-maskable_interrupts
+70.6% 2177 ± 30% interrupts.CPU216.PMI:Performance_monitoring_interrupts
+48.7% 414.50 ± 14% interrupts.CPU216.RES:Rescheduling_interrupts
+118.3% 208.50 ± 11% interrupts.CPU216.TLB:TLB_shootdowns
+54.2% 2146 ± 33% interrupts.CPU217.NMI:Non-maskable_interrupts
+54.2% 2146 ± 33% interrupts.CPU217.PMI:Performance_monitoring_interrupts
+84.9% 211.75 ± 16% interrupts.CPU217.TLB:TLB_shootdowns
+308.4% 314.50 ± 55% interrupts.CPU218.TLB:TLB_shootdowns
+259.0% 254.00 ± 21% interrupts.CPU219.TLB:TLB_shootdowns
-36.6% 561.25 ± 12% interrupts.CPU22.TLB:TLB_shootdowns
+315.9% 242.25 ± 13% interrupts.CPU220.TLB:TLB_shootdowns
+103.9% 470.00 ± 18% interrupts.CPU221.RES:Rescheduling_interrupts
+254.1% 216.00 ± 3% interrupts.CPU221.TLB:TLB_shootdowns
+102.6% 232.50 ± 22% interrupts.CPU223.TLB:TLB_shootdowns
+146.9% 232.75 ± 15% interrupts.CPU224.TLB:TLB_shootdowns
+142.6% 225.00 ± 12% interrupts.CPU225.TLB:TLB_shootdowns
+182.0% 239.00 ± 4% interrupts.CPU226.TLB:TLB_shootdowns
+164.8% 241.00 ± 20% interrupts.CPU227.TLB:TLB_shootdowns
-40.0% 1407 ± 17% interrupts.CPU228.NMI:Non-maskable_interrupts
-40.0% 1407 ± 17% interrupts.CPU228.PMI:Performance_monitoring_interrupts
+232.0% 205.00 ± 10% interrupts.CPU228.TLB:TLB_shootdowns
+188.7% 216.50 ± 14% interrupts.CPU229.TLB:TLB_shootdowns
-38.5% 434.00 ± 19% interrupts.CPU23.TLB:TLB_shootdowns
+208.6% 242.25 ± 11% interrupts.CPU230.TLB:TLB_shootdowns
+214.4% 212.25 ± 20% interrupts.CPU231.TLB:TLB_shootdowns
+141.9% 494.00 ± 11% interrupts.CPU232.RES:Rescheduling_interrupts
+254.0% 231.00 ± 13% interrupts.CPU232.TLB:TLB_shootdowns
+106.9% 216.75 ± 26% interrupts.CPU233.TLB:TLB_shootdowns
+226.9% 212.50 ± 8% interrupts.CPU234.TLB:TLB_shootdowns
+186.8% 206.50 ± 20% interrupts.CPU235.TLB:TLB_shootdowns
+75.3% 460.25 ± 13% interrupts.CPU236.RES:Rescheduling_interrupts
+195.5% 215.00 ± 7% interrupts.CPU236.TLB:TLB_shootdowns
+187.2% 224.75 ± 32% interrupts.CPU237.TLB:TLB_shootdowns
+166.9% 246.25 ± 17% interrupts.CPU238.TLB:TLB_shootdowns
+118.5% 263.25 ± 19% interrupts.CPU239.TLB:TLB_shootdowns
-42.3% 502.75 ± 12% interrupts.CPU24.TLB:TLB_shootdowns
+271.0% 236.50 ± 6% interrupts.CPU240.TLB:TLB_shootdowns
+207.4% 237.50 ± 15% interrupts.CPU241.TLB:TLB_shootdowns
+310.6% 204.25 ± 13% interrupts.CPU242.TLB:TLB_shootdowns
+194.7% 224.00 ± 17% interrupts.CPU243.TLB:TLB_shootdowns
+183.6% 229.00 ± 17% interrupts.CPU244.TLB:TLB_shootdowns
+171.8% 235.75 ± 8% interrupts.CPU245.TLB:TLB_shootdowns
+344.2% 251.00 ± 13% interrupts.CPU246.TLB:TLB_shootdowns
+153.1% 684.75 ± 34% interrupts.CPU247.RES:Rescheduling_interrupts
+180.2% 247.25 ± 18% interrupts.CPU247.TLB:TLB_shootdowns
+167.1% 217.00 ± 20% interrupts.CPU248.TLB:TLB_shootdowns
+216.4% 212.00 ± 11% interrupts.CPU249.TLB:TLB_shootdowns
-36.9% 467.25 ± 7% interrupts.CPU25.TLB:TLB_shootdowns
+247.0% 234.25 ± 11% interrupts.CPU250.TLB:TLB_shootdowns
+111.1% 194.25 ± 24% interrupts.CPU251.TLB:TLB_shootdowns
+223.5% 233.75 ± 16% interrupts.CPU252.TLB:TLB_shootdowns
+178.7% 219.50 ± 12% interrupts.CPU253.TLB:TLB_shootdowns
+258.4% 230.25 ± 11% interrupts.CPU254.TLB:TLB_shootdowns
+44.3% 385.25 ± 13% interrupts.CPU255.RES:Rescheduling_interrupts
+106.1% 235.50 ± 12% interrupts.CPU255.TLB:TLB_shootdowns
+213.9% 232.25 ± 13% interrupts.CPU256.TLB:TLB_shootdowns
+158.4% 222.25 ± 19% interrupts.CPU257.TLB:TLB_shootdowns
+54.4% 2417 ± 27% interrupts.CPU258.NMI:Non-maskable_interrupts
+54.4% 2417 ± 27% interrupts.CPU258.PMI:Performance_monitoring_interrupts
+118.6% 550.75 ± 54% interrupts.CPU258.RES:Rescheduling_interrupts
+192.7% 239.25 ± 9% interrupts.CPU258.TLB:TLB_shootdowns
+109.1% 223.25 ± 20% interrupts.CPU259.TLB:TLB_shootdowns
-42.9% 514.00 ± 6% interrupts.CPU26.TLB:TLB_shootdowns
+291.8% 263.50 ± 22% interrupts.CPU260.TLB:TLB_shootdowns
+289.0% 230.50 ± 8% interrupts.CPU261.TLB:TLB_shootdowns
+61.0% 471.25 ± 16% interrupts.CPU262.RES:Rescheduling_interrupts
+400.0% 248.75 ± 14% interrupts.CPU262.TLB:TLB_shootdowns
+52.1% 494.25 ± 27% interrupts.CPU263.RES:Rescheduling_interrupts
+143.1% 211.50 ± 11% interrupts.CPU263.TLB:TLB_shootdowns
+71.7% 374.75 ± 5% interrupts.CPU264.RES:Rescheduling_interrupts
+188.8% 218.75 ± 9% interrupts.CPU264.TLB:TLB_shootdowns
+306.2% 228.50 ± 20% interrupts.CPU265.TLB:TLB_shootdowns
+221.1% 259.25 ± 12% interrupts.CPU266.TLB:TLB_shootdowns
+56.2% 423.75 ± 22% interrupts.CPU267.RES:Rescheduling_interrupts
+352.7% 272.75 ± 16% interrupts.CPU267.TLB:TLB_shootdowns
+295.5% 221.50 ± 20% interrupts.CPU268.TLB:TLB_shootdowns
+151.2% 244.25 ± 17% interrupts.CPU269.TLB:TLB_shootdowns
-32.2% 466.25 ± 11% interrupts.CPU27.TLB:TLB_shootdowns
+93.6% 402.75 ± 8% interrupts.CPU270.RES:Rescheduling_interrupts
+368.3% 269.25 ± 19% interrupts.CPU270.TLB:TLB_shootdowns
+288.5% 286.50 ± 14% interrupts.CPU271.TLB:TLB_shootdowns
+37.3% 383.50 ± 10% interrupts.CPU272.RES:Rescheduling_interrupts
+189.6% 229.50 ± 28% interrupts.CPU272.TLB:TLB_shootdowns
+201.8% 653.50 ± 66% interrupts.CPU273.RES:Rescheduling_interrupts
+256.1% 233.25 ± 11% interrupts.CPU273.TLB:TLB_shootdowns
+198.3% 223.00 ± 17% interrupts.CPU274.TLB:TLB_shootdowns
+204.4% 208.50 ± 10% interrupts.CPU275.TLB:TLB_shootdowns
+226.8% 195.25 ± 13% interrupts.CPU276.TLB:TLB_shootdowns
+80.2% 526.75 ± 32% interrupts.CPU277.RES:Rescheduling_interrupts
+233.8% 244.50 ± 14% interrupts.CPU277.TLB:TLB_shootdowns
+248.0% 214.00 ± 9% interrupts.CPU278.TLB:TLB_shootdowns
+188.1% 470.25 ± 18% interrupts.CPU279.RES:Rescheduling_interrupts
+142.0% 197.25 ± 8% interrupts.CPU279.TLB:TLB_shootdowns
+139.3% 892.00 ± 59% interrupts.CPU28.RES:Rescheduling_interrupts
-41.6% 520.50 ± 10% interrupts.CPU28.TLB:TLB_shootdowns
+223.8% 245.25 ± 12% interrupts.CPU280.TLB:TLB_shootdowns
+234.0% 221.25 ± 12% interrupts.CPU281.TLB:TLB_shootdowns
+90.9% 456.75 ± 15% interrupts.CPU282.RES:Rescheduling_interrupts
+281.9% 242.50 ± 11% interrupts.CPU282.TLB:TLB_shootdowns
+215.2% 238.00 ± 10% interrupts.CPU283.TLB:TLB_shootdowns
+72.2% 464.50 ± 20% interrupts.CPU284.RES:Rescheduling_interrupts
+251.7% 233.00 ± 22% interrupts.CPU284.TLB:TLB_shootdowns
+117.3% 225.50 ± 6% interrupts.CPU285.TLB:TLB_shootdowns
+42.6% 391.75 ± 9% interrupts.CPU286.RES:Rescheduling_interrupts
+106.1% 211.25 ± 3% interrupts.CPU286.TLB:TLB_shootdowns
-17.7% 1320 interrupts.CPU287.CAL:Function_call_interrupts
-26.0% 1376 ± 16% interrupts.CPU287.NMI:Non-maskable_interrupts
-26.0% 1376 ± 16% interrupts.CPU287.PMI:Performance_monitoring_interrupts
+54.3% 378.75 ± 8% interrupts.CPU287.RES:Rescheduling_interrupts
-35.5% 441.75 ± 4% interrupts.CPU29.TLB:TLB_shootdowns
-32.4% 466.00 ± 6% interrupts.CPU3.TLB:TLB_shootdowns
+149.8% 998.75 ± 69% interrupts.CPU30.RES:Rescheduling_interrupts
-36.1% 506.75 ± 7% interrupts.CPU30.TLB:TLB_shootdowns
-56.9% 1372 ± 6% interrupts.CPU31.NMI:Non-maskable_interrupts
-56.9% 1372 ± 6% interrupts.CPU31.PMI:Performance_monitoring_interrupts
+134.5% 663.50 ± 31% interrupts.CPU31.RES:Rescheduling_interrupts
-43.8% 440.25 ± 3% interrupts.CPU31.TLB:TLB_shootdowns
-39.7% 546.25 ± 19% interrupts.CPU32.TLB:TLB_shootdowns
-35.0% 461.00 ± 14% interrupts.CPU33.TLB:TLB_shootdowns
+92.7% 914.50 ± 50% interrupts.CPU34.RES:Rescheduling_interrupts
-43.0% 501.50 ± 9% interrupts.CPU34.TLB:TLB_shootdowns
-42.5% 431.25 ± 10% interrupts.CPU35.TLB:TLB_shootdowns
-45.4% 485.25 ± 6% interrupts.CPU36.TLB:TLB_shootdowns
+34.3% 557.75 ± 9% interrupts.CPU37.RES:Rescheduling_interrupts
-45.1% 449.25 ± 14% interrupts.CPU37.TLB:TLB_shootdowns
+23.7% 561.00 ± 11% interrupts.CPU38.RES:Rescheduling_interrupts
-37.8% 558.75 ± 7% interrupts.CPU38.TLB:TLB_shootdowns
-41.3% 441.25 ± 11% interrupts.CPU39.TLB:TLB_shootdowns
+48.7% 2234 ± 24% interrupts.CPU4.NMI:Non-maskable_interrupts
+48.7% 2234 ± 24% interrupts.CPU4.PMI:Performance_monitoring_interrupts
-47.3% 534.00 ± 6% interrupts.CPU4.TLB:TLB_shootdowns
-41.4% 517.00 ± 17% interrupts.CPU40.TLB:TLB_shootdowns
-35.8% 489.25 ± 4% interrupts.CPU41.TLB:TLB_shootdowns
-49.2% 506.00 ± 8% interrupts.CPU42.TLB:TLB_shootdowns
-35.4% 421.25 ± 11% interrupts.CPU43.TLB:TLB_shootdowns
+102.1% 851.25 ± 47% interrupts.CPU44.RES:Rescheduling_interrupts
-39.3% 497.25 ± 7% interrupts.CPU44.TLB:TLB_shootdowns
+79.1% 660.75 ± 22% interrupts.CPU45.RES:Rescheduling_interrupts
-41.6% 467.75 ± 8% interrupts.CPU45.TLB:TLB_shootdowns
-38.5% 483.75 ± 12% interrupts.CPU46.TLB:TLB_shootdowns
-35.4% 463.25 ± 11% interrupts.CPU47.TLB:TLB_shootdowns
-34.6% 531.75 ± 6% interrupts.CPU48.TLB:TLB_shootdowns
-26.2% 486.25 ± 6% interrupts.CPU49.TLB:TLB_shootdowns
-36.6% 466.00 ± 18% interrupts.CPU5.TLB:TLB_shootdowns
-43.0% 485.75 ± 9% interrupts.CPU50.TLB:TLB_shootdowns
-40.3% 487.25 ± 12% interrupts.CPU51.TLB:TLB_shootdowns
-42.5% 517.50 ± 10% interrupts.CPU52.TLB:TLB_shootdowns
-33.8% 433.00 ± 19% interrupts.CPU53.TLB:TLB_shootdowns
-40.9% 543.50 ± 6% interrupts.CPU54.TLB:TLB_shootdowns
-31.5% 470.75 ± 16% interrupts.CPU55.TLB:TLB_shootdowns
-41.5% 1442 ± 13% interrupts.CPU56.NMI:Non-maskable_interrupts
-41.5% 1442 ± 13% interrupts.CPU56.PMI:Performance_monitoring_interrupts
+106.1% 705.00 ± 30% interrupts.CPU56.RES:Rescheduling_interrupts
-45.2% 485.25 ± 8% interrupts.CPU56.TLB:TLB_shootdowns
-43.0% 440.00 ± 11% interrupts.CPU57.TLB:TLB_shootdowns
-46.3% 506.00 ± 10% interrupts.CPU58.TLB:TLB_shootdowns
-38.0% 467.50 interrupts.CPU59.TLB:TLB_shootdowns
+311.4% 1619 ±109% interrupts.CPU6.RES:Rescheduling_interrupts
-46.6% 459.00 ± 7% interrupts.CPU6.TLB:TLB_shootdowns
-43.4% 506.00 ± 13% interrupts.CPU60.TLB:TLB_shootdowns
+78.1% 2999 ± 14% interrupts.CPU61.NMI:Non-maskable_interrupts
+78.1% 2999 ± 14% interrupts.CPU61.PMI:Performance_monitoring_interrupts
-34.9% 451.75 ± 9% interrupts.CPU61.TLB:TLB_shootdowns
-49.3% 474.75 ± 8% interrupts.CPU62.TLB:TLB_shootdowns
+112.9% 897.00 ± 11% interrupts.CPU63.RES:Rescheduling_interrupts
-41.1% 431.00 ± 7% interrupts.CPU63.TLB:TLB_shootdowns
-43.7% 496.75 ± 7% interrupts.CPU64.TLB:TLB_shootdowns
-36.8% 465.50 ± 5% interrupts.CPU65.TLB:TLB_shootdowns
-42.6% 505.25 ± 8% interrupts.CPU66.TLB:TLB_shootdowns
-40.5% 561.75 ± 10% interrupts.CPU68.TLB:TLB_shootdowns
+100.5% 788.50 ± 26% interrupts.CPU69.RES:Rescheduling_interrupts
-38.3% 487.50 ± 8% interrupts.CPU69.TLB:TLB_shootdowns
+407.0% 2106 ±123% interrupts.CPU7.RES:Rescheduling_interrupts
-36.8% 429.25 interrupts.CPU7.TLB:TLB_shootdowns
+92.1% 672.75 ± 12% interrupts.CPU70.RES:Rescheduling_interrupts
-41.8% 453.50 ± 11% interrupts.CPU70.TLB:TLB_shootdowns
-39.8% 405.75 ± 11% interrupts.CPU71.TLB:TLB_shootdowns
+95.7% 901.75 ± 43% interrupts.CPU74.RES:Rescheduling_interrupts
+82.1% 292.75 ± 15% interrupts.CPU76.TLB:TLB_shootdowns
+45.8% 288.75 ± 9% interrupts.CPU77.TLB:TLB_shootdowns
-35.5% 1486 ± 12% interrupts.CPU79.NMI:Non-maskable_interrupts
-35.5% 1486 ± 12% interrupts.CPU79.PMI:Performance_monitoring_interrupts
+72.9% 268.50 ± 8% interrupts.CPU79.TLB:TLB_shootdowns
-45.4% 484.50 ± 10% interrupts.CPU8.TLB:TLB_shootdowns
+32.9% 255.25 ± 12% interrupts.CPU80.TLB:TLB_shootdowns
+95.1% 309.75 ± 12% interrupts.CPU82.TLB:TLB_shootdowns
+120.8% 586.75 ± 30% interrupts.CPU84.RES:Rescheduling_interrupts
+80.2% 264.50 ± 20% interrupts.CPU84.TLB:TLB_shootdowns
+56.0% 290.25 ± 9% interrupts.CPU86.TLB:TLB_shootdowns
+66.5% 281.75 ± 2% interrupts.CPU88.TLB:TLB_shootdowns
-47.5% 462.00 ± 4% interrupts.CPU9.TLB:TLB_shootdowns
+75.8% 283.00 ± 5% interrupts.CPU90.TLB:TLB_shootdowns
+50.3% 292.75 ± 11% interrupts.CPU91.TLB:TLB_shootdowns
-37.8% 1692 ± 6% interrupts.CPU92.NMI:Non-maskable_interrupts
-37.8% 1692 ± 6% interrupts.CPU92.PMI:Performance_monitoring_interrupts
+36.3% 260.00 ± 22% interrupts.CPU92.TLB:TLB_shootdowns
+57.0% 295.25 ± 10% interrupts.CPU94.TLB:TLB_shootdowns
+59.6% 319.50 ± 16% interrupts.CPU95.TLB:TLB_shootdowns
+105.1% 648.50 ± 19% interrupts.CPU96.RES:Rescheduling_interrupts
+91.0% 274.50 ± 8% interrupts.CPU96.TLB:TLB_shootdowns
-34.2% 1586 ± 46% interrupts.CPU97.NMI:Non-maskable_interrupts
-34.2% 1586 ± 46% interrupts.CPU97.PMI:Performance_monitoring_interrupts
+56.0% 290.25 ± 16% interrupts.CPU98.TLB:TLB_shootdowns

vm-scalability.time.user_time

+-------------------------------------------------------------------+
+ |
:+ + + + |
.++. +. + + +.+ :+ .++ ++. +.++ + + :+ + +|
++ + + + ++.+++ +.+++.++.+ ++ + +.+ + :+ |
+ |
|
|
|
|
|
OO O O |
OO O OO O OO O |
O O OO |
+-------------------------------------------------------------------+

vm-scalability.time.voluntary_context_switches

+------------------------------------------------------------------+
|
O OOO OOO O O OO |
|
|
|
|
|
|
|
|
|
|
+.+++.++.+++.+++.+++.+++. .+ +. |
+------------------------------------------------------------------+

vm-scalability.throughput

+-----------------------------------------------------------------+
|
+ + |
+. + + +. + + |
| +++ +.+ +.+++ .++ +++ +.++ +++ +.+ +. |
+ +.+++ +++.++ +++.+++.+|
|
|
|
|
|
O |
O OO O OO OO O |
O O |
+-----------------------------------------------------------------+

vm-scalability.median

+-------------------------------------------------------------------+
+ + |
: : :: |
.++. : + .+++. : +. + |
+ +.+ ++.+++ +++ + ++.+ + +. .+++. +. ++. |
+++ + + +|
|
|
|
|
|
O |
O O |
O OO OOO O O O |
+-------------------------------------------------------------------+

have been estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
configuration may affect actual performance.

Attachments:

(No filename) (81.22 kB)
config-5.6.0-05750-gc0d0381ade798 (208.58 kB)
job-script (7.68 kB)
job.yaml (5.29 kB)
reproduce (4.81 kB)
Download all attachments

2020-06-22 22:06:23

by Mike Kravetz

[permalink] [raw]

Subject: Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 6/21/20 5:55 PM, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -33.4% regression of vm-scalability.throughput due to commit:
>
>
> commit: c0d0381ade79885c04a04c303284b040616b116e ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: vm-scalability
> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
> with following parameters:
>
> runtime: 300s
> size: 8T
> test: anon-cow-seq-hugetlb
> cpufreq_governor: performance
> ucode: 0x11
>

Some performance regression is not surprising as the change includes acquiring
and holding the i_mmap_rwsem (in read mode) during hugetlb page faults. 33.4%
seems a bit high. But, the test is primarily exercising the hugetlb page
fault path and little else.

The reason for taking the i_mmap_rwsem is to prevent PMD unsharing from
invalidating the pmd we are operating on. This specific test case is operating
on anonymous private mappings. So, PMD sharing is not possible and we can
eliminate acquiring the mutex in this case. In fact, we should check all
mappings (even sharable) for the possibly of PMD sharing and only take the
mutex if necessary. It will make the code a bit uglier, but will take care
of some of these regressions. We still need to take the mutex in the case
of PMD sharing. I'm afraid a regression is unavoidable in that case.

I'll put together a patch.
--
Mike Kravetz

2020-06-25 23:40:39

by Mike Kravetz

[permalink] [raw]

Subject: Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 6/22/20 3:01 PM, Mike Kravetz wrote:
> On 6/21/20 5:55 PM, kernel test robot wrote:
>> Greeting,
>>
>> FYI, we noticed a -33.4% regression of vm-scalability.throughput due to commit:
>>
>>
>> commit: c0d0381ade79885c04a04c303284b040616b116e ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization")
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>
>> in testcase: vm-scalability
>> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
>> with following parameters:
>>
>> runtime: 300s
>> size: 8T
>> test: anon-cow-seq-hugetlb
>> cpufreq_governor: performance
>> ucode: 0x11
>>
>
> Some performance regression is not surprising as the change includes acquiring
> and holding the i_mmap_rwsem (in read mode) during hugetlb page faults. 33.4%
> seems a bit high. But, the test is primarily exercising the hugetlb page
> fault path and little else.
>
> The reason for taking the i_mmap_rwsem is to prevent PMD unsharing from
> invalidating the pmd we are operating on. This specific test case is operating
> on anonymous private mappings. So, PMD sharing is not possible and we can
> eliminate acquiring the mutex in this case. In fact, we should check all
> mappings (even sharable) for the possibly of PMD sharing and only take the
> mutex if necessary. It will make the code a bit uglier, but will take care
> of some of these regressions. We still need to take the mutex in the case
> of PMD sharing. I'm afraid a regression is unavoidable in that case.
>
> I'll put together a patch.

Not acquiring the mutex on faults when sharing is not possible is quite
straight forward. We can even use the existing routine vma_shareable()
to easily check. However, the next patch in the series 87bf91d39bb5
"hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race" depends
on always acquiring the mutex. If we break this assumption, then the
code to back out hugetlb reservations needs to be written. A high level
view of what needs to be done is in the commit message for 87bf91d39bb5.

I'm working on the code to back out reservations.
--
Mike Kravetz

2020-07-06 20:30:26

by Mike Kravetz

[permalink] [raw]

Subject: [RFC PATCH 0/3] hugetlbfs: address fault time regression

Commits c0d0381ade79 and 87bf91d39bb5 changed the way huegtlb locking
was performed to address BUGs. One specific change was to always take
the i_mmap_rwsem in read mode during fault processing. One result of
this change was a 33% regression for anon non-shared page faults [1].

Technically, i_mmap_rwsem only needs to be taken during page faults
if the pmd can potentially be shared. pmd sharing is not possible for
anon non-shared mappings (as in the reported regression), therefore the
code can be modified to not acquire the semaphore in this case.

Unfortunately, commit 87bf91d39bb5 depends on i_mmap_rwsem always being
taken in the fault path to prevent fault/truncation races. So, that
approach is no longer appropriate. Rather, the code now detects races
and backs out operations.

This code "works" in that it only takes i_mmap_rwsem when necessary and
addresses the original BUGs. However, I am sending as an RFC because:
- I am unsure if the added complexity is worth performance benefit.
- There needs to be a better way/location to make a decison about taking
the semaphore. See FIXME's in the code.

Comments and suggestions would be appreciated.

[1] https://lore.kernel.org/lkml/20200622005551.GK5535@shao2-debian

Mike Kravetz (3):
Revert: "hugetlbfs: Use i_mmap_rwsem to address page fault/truncate
race"
hugetlbfs: Only take i_mmap_rwsem when sharing is possible
huegtlbfs: handle page fault/truncate races

fs/hugetlbfs/inode.c | 69 +++++++++-----------
mm/hugetlb.c | 150 ++++++++++++++++++++++++++++++-------------
2 files changed, 137 insertions(+), 82 deletions(-)

--
2.25.4

2020-08-21 08:40:43

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 6/26/2020 5:33 AM, Mike Kravetz wrote:
> On 6/22/20 3:01 PM, Mike Kravetz wrote:
>> On 6/21/20 5:55 PM, kernel test robot wrote:
>>> Greeting,
>>>
>>> FYI, we noticed a -33.4% regression of vm-scalability.throughput due to commit:
>>>
>>>
>>> commit: c0d0381ade79885c04a04c303284b040616b116e ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization")
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>>
>>> in testcase: vm-scalability
>>> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
>>> with following parameters:
>>>
>>> runtime: 300s
>>> size: 8T
>>> test: anon-cow-seq-hugetlb
>>> cpufreq_governor: performance
>>> ucode: 0x11
>>>
>>
>> Some performance regression is not surprising as the change includes acquiring
>> and holding the i_mmap_rwsem (in read mode) during hugetlb page faults. 33.4%
>> seems a bit high. But, the test is primarily exercising the hugetlb page
>> fault path and little else.
>>
>> The reason for taking the i_mmap_rwsem is to prevent PMD unsharing from
>> invalidating the pmd we are operating on. This specific test case is operating
>> on anonymous private mappings. So, PMD sharing is not possible and we can
>> eliminate acquiring the mutex in this case. In fact, we should check all
>> mappings (even sharable) for the possibly of PMD sharing and only take the
>> mutex if necessary. It will make the code a bit uglier, but will take care
>> of some of these regressions. We still need to take the mutex in the case
>> of PMD sharing. I'm afraid a regression is unavoidable in that case.
>>
>> I'll put together a patch.
>
> Not acquiring the mutex on faults when sharing is not possible is quite
> straight forward. We can even use the existing routine vma_shareable()
> to easily check. However, the next patch in the series 87bf91d39bb5
> "hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race" depends
> on always acquiring the mutex. If we break this assumption, then the
> code to back out hugetlb reservations needs to be written. A high level
> view of what needs to be done is in the commit message for 87bf91d39bb5.
>
> I'm working on the code to back out reservations.
>

I find that 34ae204f18519f0920bd50a644abd6fefc8dbfcf(hugetlbfs: remove
call to huge_pte_alloc without i_mmap_rwsem) fixed this regression, I
test with the patch, the regression reduced to 10.1%, do you have plan
to continue to improve it? Thanks.

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/runtime/size/test/cpufreq_governor/ucode:

lkp-knm01/vm-scalability/debian-x86_64-20191114.cgz/x86_64-rhel-7.6/gcc-7/300s/8T/anon-cow-seq-hugetlb/performance/0x11

commit:
49aef7175cc6eb703a9280a7b830e675fe8f2704
c0d0381ade79885c04a04c303284b040616b116e
v5.8
34ae204f18519f0920bd50a644abd6fefc8dbfcf
v5.9-rc1

49aef7175cc6eb70 c0d0381ade79885c04a04c30328 v5.8
34ae204f18519f0920bd50a644a v5.9-rc1
---------------- --------------------------- ---------------------------
--------------------------- ---------------------------
%stddev %change %stddev %change
%stddev %change %stddev %change %stddev
\ | \ | \
| \ | \
38084 -31.1% 26231 ± 2% -26.6% 27944 ±
5% -7.0% 35405 -7.5% 35244
vm-scalability.median
9.92 ± 9% +12.0 21.95 ± 4% +3.9 13.87 ±
30% -5.3 4.66 ± 9% -6.6 3.36 ± 7%
vm-scalability.median_stddev%
12827311 -35.0% 8340256 ± 2% -30.9% 8865669 ±
5% -10.1% 11532087 -10.2% 11513595 ± 2%
vm-scalability.throughput
2.507e+09 -22.7% 1.938e+09 -15.3% 2.122e+09 ±
6% +8.0% 2.707e+09 +8.0% 2.707e+09 ± 2%
vm-scalability.workload

--
Zhengjun Xing

2020-08-21 21:06:30

by Mike Kravetz

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 8/21/20 1:39 AM, Xing Zhengjun wrote:
>
>
> On 6/26/2020 5:33 AM, Mike Kravetz wrote:
>> On 6/22/20 3:01 PM, Mike Kravetz wrote:
>>> On 6/21/20 5:55 PM, kernel test robot wrote:
>>>> Greeting,
>>>>
>>>> FYI, we noticed a -33.4% regression of vm-scalability.throughput due to commit:
>>>>
>>>>
>>>> commit: c0d0381ade79885c04a04c303284b040616b116e ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization")
>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>>>
>>>> in testcase: vm-scalability
>>>> on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
>>>> with following parameters:
>>>>
>>>> runtime: 300s
>>>> size: 8T
>>>> test: anon-cow-seq-hugetlb
>>>> cpufreq_governor: performance
>>>> ucode: 0x11
>>>>
>>>
>>> Some performance regression is not surprising as the change includes acquiring
>>> and holding the i_mmap_rwsem (in read mode) during hugetlb page faults. 33.4%
>>> seems a bit high. But, the test is primarily exercising the hugetlb page
>>> fault path and little else.
>>>
>>> The reason for taking the i_mmap_rwsem is to prevent PMD unsharing from
>>> invalidating the pmd we are operating on. This specific test case is operating
>>> on anonymous private mappings. So, PMD sharing is not possible and we can
>>> eliminate acquiring the mutex in this case. In fact, we should check all
>>> mappings (even sharable) for the possibly of PMD sharing and only take the
>>> mutex if necessary. It will make the code a bit uglier, but will take care
>>> of some of these regressions. We still need to take the mutex in the case
>>> of PMD sharing. I'm afraid a regression is unavoidable in that case.
>>>
>>> I'll put together a patch.
>>
>> Not acquiring the mutex on faults when sharing is not possible is quite
>> straight forward. We can even use the existing routine vma_shareable()
>> to easily check. However, the next patch in the series 87bf91d39bb5
>> "hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race" depends
>> on always acquiring the mutex. If we break this assumption, then the
>> code to back out hugetlb reservations needs to be written. A high level
>> view of what needs to be done is in the commit message for 87bf91d39bb5.
>>
>> I'm working on the code to back out reservations.
>>
>
> I find that 34ae204f18519f0920bd50a644abd6fefc8dbfcf(hugetlbfs: remove call to huge_pte_alloc without i_mmap_rwsem) fixed this regression, I test with the patch, the regression reduced to 10.1%, do you have plan to continue to improve it? Thanks.

Thank you for testing!

Commit 34ae204f1851 was not created to address performance. Rather it was
created more for the sake of correctness.

IIRC, this test is going to stress the page fault path by continuing to produce
faults for the duration of the test. Commit 34ae204f1851 removed an unneeded,
somewhat dangerous and redundant call to huge_pte_alloc in the fault path.
Your testing showed that removing this call helped address a good amount of the
performance regression. The series proposed in this thread attempts to
eliminate more potentially unnecessary code in the hugetlb fault path.
Specifically, it will only acquire the i_mmap_rwsem when necessary.

Would you be willing to test this series on top of 34ae204f1851? I will need
to rebase the series to take the changes made by 34ae204f1851 into account.

BTW - I believe that shared hugetlb mappings are the most common use case.
In the shared case, acquiring i_mmap_rwsem is necessary if pmd sharing is
possible. There is little we can do to eliminate this. The test suite does
not appear to test shared mappings.
--
Mike Kravetz

2020-08-21 23:40:19

by Mike Kravetz

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 8/21/20 2:02 PM, Mike Kravetz wrote:
> Would you be willing to test this series on top of 34ae204f1851? I will need
> to rebase the series to take the changes made by 34ae204f1851 into account.

Actually, the series in this thread will apply/run cleanly on top of
34ae204f1851. No need to rebase or port. If we decide to move forward more
work is required. See a few FIXME's in the patches.
--
Mike Kravetz

2020-10-12 06:18:03

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

Hi Mike,

I re-test it in v5.9-rc8, the regression still existed. It is almost
the same as 34ae204f1851. Do you have time to look at it? Thanks.

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/runtime/size/test/cpufreq_governor/ucode:

lkp-knm01/vm-scalability/debian-x86_64-20191114.cgz/x86_64-rhel-7.6/gcc-7/300s/8T/anon-cow-seq-hugetlb/performance/0x11

commit:
49aef7175cc6eb703a9280a7b830e675fe8f2704
c0d0381ade79885c04a04c303284b040616b116e
v5.8
34ae204f18519f0920bd50a644abd6fefc8dbfcf
v5.9-rc1
v5.9-rc8

49aef7175cc6eb70 c0d0381ade79885c04a04c30328 v5.8
34ae204f18519f0920bd50a644a v5.9-rc1
v5.9-rc8
---------------- --------------------------- ---------------------------
--------------------------- ---------------------------
---------------------------
%stddev %change %stddev %change
%stddev %change %stddev %change %stddev
%change %stddev
\ | \ | \
| \ | \ |
\
38043 ± 3% -30.2% 26560 ± 4% -29.5% 26815 ±
6% -7.4% 35209 ± 2% -7.4% 35244 -8.8%
34704 vm-scalability.median
7.86 ± 19% +9.7 17.54 ± 21% +10.4 18.23 ±
34% -3.1 4.75 ± 7% -4.5 3.36 ± 7% -4.0
3.82 ± 15% vm-scalability.median_stddev%
12822071 ± 3% -34.1% 8450822 ± 4% -33.6% 8517252 ±
6% -10.7% 11453675 ± 2% -10.2% 11513595 ± 2% -11.6%
11331657 vm-scalability.throughput
2.523e+09 ± 3% -20.7% 2.001e+09 ± 5% -19.9% 2.021e+09 ±
7% +6.8% 2.694e+09 ± 2% +7.3% 2.707e+09 ± 2% +5.4%
2.661e+09 vm-scalability.workload

On 8/22/2020 7:36 AM, Mike Kravetz wrote:
> On 8/21/20 2:02 PM, Mike Kravetz wrote:
>> Would you be willing to test this series on top of 34ae204f1851? I will need
>> to rebase the series to take the changes made by 34ae204f1851 into account.
>
> Actually, the series in this thread will apply/run cleanly on top of
> 34ae204f1851. No need to rebase or port. If we decide to move forward more
> work is required. See a few FIXME's in the patches.
>

--
Zhengjun Xing

2020-10-12 17:46:32

by Mike Kravetz

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 10/11/20 10:29 PM, Xing Zhengjun wrote:
> Hi Mike,
>
> I re-test it in v5.9-rc8, the regression still existed. It is almost the same as 34ae204f1851. Do you have time to look at it? Thanks.
>

Thank you for testing.

Just curious, did you apply the series in this thread or just test v5.9-rc8?
If just testing v5.9-rc8, no changes to this code were added after 34ae204f1851,
so results being the same are expected.

There are some functional issues with this new hugetlb locking model that
are currently being worked. It is likely to result in significantly different
code. The performance issues discovered here will be taken into account with
the new code. However, as previously mentioned additional synchronization
is required for functional correctness. As a result, there will be some
regression in this code.

--
Mike Kravetz

> =========================================================================================
> tbox_group/testcase/rootfs/kconfig/compiler/runtime/size/test/cpufreq_governor/ucode:
>
> lkp-knm01/vm-scalability/debian-x86_64-20191114.cgz/x86_64-rhel-7.6/gcc-7/300s/8T/anon-cow-seq-hugetlb/performance/0x11
>
> commit:
> 49aef7175cc6eb703a9280a7b830e675fe8f2704
> c0d0381ade79885c04a04c303284b040616b116e
> v5.8
> 34ae204f18519f0920bd50a644abd6fefc8dbfcf
> v5.9-rc1
> v5.9-rc8
>
> 49aef7175cc6eb70 c0d0381ade79885c04a04c30328 v5.8 34ae204f18519f0920bd50a644a v5.9-rc1 v5.9-rc8
> ---------------- --------------------------- --------------------------- --------------------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev %change %stddev %change %stddev %change %stddev
> \ | \ | \ | \ | \ | \
> 38043 ± 3% -30.2% 26560 ± 4% -29.5% 26815 ± 6% -7.4% 35209 ± 2% -7.4% 35244 -8.8% 34704 vm-scalability.median
> 7.86 ± 19% +9.7 17.54 ± 21% +10.4 18.23 ± 34% -3.1 4.75 ± 7% -4.5 3.36 ± 7% -4.0 3.82 ± 15% vm-scalability.median_stddev%
> 12822071 ± 3% -34.1% 8450822 ± 4% -33.6% 8517252 ± 6% -10.7% 11453675 ± 2% -10.2% 11513595 ± 2% -11.6% 11331657 vm-scalability.throughput
> 2.523e+09 ± 3% -20.7% 2.001e+09 ± 5% -19.9% 2.021e+09 ± 7% +6.8% 2.694e+09 ± 2% +7.3% 2.707e+09 ± 2% +5.4% 2.661e+09 vm-scalability.workload
>
>
> On 8/22/2020 7:36 AM, Mike Kravetz wrote:
>> On 8/21/20 2:02 PM, Mike Kravetz wrote:
>>> Would you be willing to test this series on top of 34ae204f1851? I will need
>>> to rebase the series to take the changes made by 34ae204f1851 into account.
>>
>> Actually, the series in this thread will apply/run cleanly on top of
>> 34ae204f1851. No need to rebase or port. If we decide to move forward more
>> work is required. See a few FIXME's in the patches.
>>

2020-10-13 11:21:24

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 10/13/2020 1:40 AM, Mike Kravetz wrote:
> On 10/11/20 10:29 PM, Xing Zhengjun wrote:
>> Hi Mike,
>>
>> I re-test it in v5.9-rc8, the regression still existed. It is almost the same as 34ae204f1851. Do you have time to look at it? Thanks.
>>
>
> Thank you for testing.
>
> Just curious, did you apply the series in this thread or just test v5.9-rc8?
> If just testing v5.9-rc8, no changes to this code were added after 34ae204f1851,
> so results being the same are expected.
>

I just test v5.9-rc8. Where can I find the series patches you mentioned
here? Or just wait for the next mainline release?

> There are some functional issues with this new hugetlb locking model that
> are currently being worked. It is likely to result in significantly different
> code. The performance issues discovered here will be taken into account with
> the new code. However, as previously mentioned additional synchronization
> is required for functional correctness. As a result, there will be some
> regression in this code.
>

--
Zhengjun Xing

2020-10-13 11:23:10

by Mike Kravetz

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 10/12/20 6:59 PM, Xing Zhengjun wrote:
>
>
> On 10/13/2020 1:40 AM, Mike Kravetz wrote:
>> On 10/11/20 10:29 PM, Xing Zhengjun wrote:
>>> Hi Mike,
>>>
>>> I re-test it in v5.9-rc8, the regression still existed. It is almost the same as 34ae204f1851. Do you have time to look at it? Thanks.
>>>
>>
>> Thank you for testing.
>>
>> Just curious, did you apply the series in this thread or just test v5.9-rc8?
>> If just testing v5.9-rc8, no changes to this code were added after 34ae204f1851,
>> so results being the same are expected.
>>
>
> I just test v5.9-rc8. Where can I find the series patches you mentioned here? Or just wait for the next mainline release?
>

My apologies. I missed that you were not cc'ed on this thred:
https://lore.kernel.org/linux-mm/[email protected]/

As mentioned, there will likely be another revision to the way locking is
handled. The new scheme will try to consider performance as is done in
the above link. I suggest you wait for next revision. If you do not mind,
I will cc you when the new code is posted.
--
Mike Kravetz

2020-10-13 11:37:17

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] Re: [hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

On 10/13/2020 11:01 AM, Mike Kravetz wrote:
> On 10/12/20 6:59 PM, Xing Zhengjun wrote:
>>
>>
>> On 10/13/2020 1:40 AM, Mike Kravetz wrote:
>>> On 10/11/20 10:29 PM, Xing Zhengjun wrote:
>>>> Hi Mike,
>>>>
>>>> I re-test it in v5.9-rc8, the regression still existed. It is almost the same as 34ae204f1851. Do you have time to look at it? Thanks.
>>>>
>>>
>>> Thank you for testing.
>>>
>>> Just curious, did you apply the series in this thread or just test v5.9-rc8?
>>> If just testing v5.9-rc8, no changes to this code were added after 34ae204f1851,
>>> so results being the same are expected.
>>>
>>
>> I just test v5.9-rc8. Where can I find the series patches you mentioned here? Or just wait for the next mainline release?
>>
>
> My apologies. I missed that you were not cc'ed on this thred:
> https://lore.kernel.org/linux-mm/[email protected]/
>
> As mentioned, there will likely be another revision to the way locking is
> handled. The new scheme will try to consider performance as is done in
> the above link. I suggest you wait for next revision. If you do not mind,
> I will cc you when the new code is posted.
>

OK. I will wait for the next revision.

--
Zhengjun Xing