Hello,
kernel test robot noticed a -11.2% regression of stress-ng.file-ioctl.ops_per_sec on:
commit: dfad37051ade6ac0d404ef4913f3bd01954ee51c ("remap_range: move permission hooks out of do_clone_file_range()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
testcase: stress-ng
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:
nr_threads: 10%
disk: 1HDD
testtime: 60s
fs: btrfs
test: file-ioctl
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240131/[email protected]
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/file-ioctl/stress-ng/60s
commit:
d53471ba6f ("splice: remove permission hook from iter_file_splice_write()")
dfad37051a ("remap_range: move permission hooks out of do_clone_file_range()")
d53471ba6f7ae97a dfad37051ade6ac0d404ef4913f
---------------- ---------------------------
%stddev %change %stddev
\ | \
2.57 -0.3 2.27 mpstat.cpu.all.usr%
7.40 +3.4% 7.65 iostat.cpu.system
2.50 -11.5% 2.22 iostat.cpu.user
95739218 -11.2% 84990543 ? 2% stress-ng.file-ioctl.ops
1595650 -11.2% 1416506 ? 2% stress-ng.file-ioctl.ops_per_sec
267.41 +4.2% 278.66 stress-ng.time.system_time
90.19 -12.5% 78.96 stress-ng.time.user_time
0.12 ? 9% +37.6% 0.16 ? 3% perf-stat.i.MPKI
5.619e+09 -4.9% 5.346e+09 perf-stat.i.branch-instructions
25.26 ? 12% +5.4 30.67 ? 2% perf-stat.i.cache-miss-rate%
3226271 ? 8% +32.3% 4268159 ? 2% perf-stat.i.cache-misses
13880671 ? 2% +7.6% 14934433 perf-stat.i.cache-references
0.83 +3.9% 0.86 perf-stat.i.cpi
7405 ? 8% -26.1% 5473 ? 2% perf-stat.i.cycles-between-cache-misses
5.186e+09 -6.0% 4.873e+09 perf-stat.i.dTLB-stores
2.807e+10 -3.9% 2.696e+10 perf-stat.i.instructions
1.21 -3.7% 1.17 perf-stat.i.ipc
257.16 +12.9% 290.46 perf-stat.i.metric.K/sec
290.80 -4.2% 278.45 perf-stat.i.metric.M/sec
1580051 ? 11% +38.0% 2180479 ? 5% perf-stat.i.node-load-misses
228848 ? 22% +116.2% 494834 ? 27% perf-stat.i.node-loads
0.11 ? 9% +37.7% 0.16 ? 3% perf-stat.overall.MPKI
23.29 ? 11% +5.3 28.58 ? 2% perf-stat.overall.cache-miss-rate%
0.82 +3.9% 0.86 perf-stat.overall.cpi
7231 ? 8% -25.1% 5416 ? 2% perf-stat.overall.cycles-between-cache-misses
1.21 -3.7% 1.17 perf-stat.overall.ipc
5.524e+09 -4.8% 5.257e+09 perf-stat.ps.branch-instructions
3170718 ? 8% +32.4% 4196610 ? 2% perf-stat.ps.cache-misses
13646445 ? 2% +7.6% 14686495 ? 2% perf-stat.ps.cache-references
5.099e+09 -6.0% 4.792e+09 perf-stat.ps.dTLB-stores
2.759e+10 -3.9% 2.651e+10 perf-stat.ps.instructions
1553350 ? 11% +38.1% 2144498 ? 5% perf-stat.ps.node-load-misses
224907 ? 22% +116.2% 486304 ? 27% perf-stat.ps.node-loads
1.668e+12 -3.4% 1.611e+12 ? 2% perf-stat.total.instructions
5.57 ? 3% -0.7 4.85 ? 2% perf-profile.calltrace.cycles-pp.__fget_light.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
0.89 ? 23% -0.4 0.45 ? 44% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
2.30 ? 2% -0.3 2.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
1.69 ? 3% -0.3 1.39 ? 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
1.99 ? 2% -0.3 1.72 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.16 ? 3% -0.2 1.00 ? 3% perf-profile.calltrace.cycles-pp.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.60 ? 4% -0.2 0.44 ? 45% perf-profile.calltrace.cycles-pp.__fget_light.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +1.5 1.52 ? 2% perf-profile.calltrace.cycles-pp.__fsnotify_parent.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
0.00 +6.9 6.94 ? 6% perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl
0.00 +7.4 7.41 ? 6% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
21.11 +7.4 28.53 perf-profile.calltrace.cycles-pp.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
3.18 ? 2% +8.7 11.87 ? 3% perf-profile.calltrace.cycles-pp.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.46 ? 9% +8.9 10.36 ? 4% perf-profile.calltrace.cycles-pp.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64
10.70 -1.3 9.39 ? 3% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
11.31 -1.1 10.24 ? 2% perf-profile.children.cycles-pp.entry_SYSCALL_64
7.87 ? 3% -1.0 6.90 perf-profile.children.cycles-pp.__fget_light
5.13 -0.7 4.46 ? 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.89 -0.4 0.46 ? 5% perf-profile.children.cycles-pp.do_clone_file_range
3.45 ? 2% -0.4 3.10 perf-profile.children.cycles-pp.llseek
1.80 ? 4% -0.3 1.49 ? 3% perf-profile.children.cycles-pp.stress_file_ioctl
1.83 -0.2 1.63 ? 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
1.53 ? 3% -0.2 1.34 ? 4% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
2.32 ? 3% -0.2 2.13 perf-profile.children.cycles-pp.syscall_return_via_sysret
1.58 ? 2% -0.2 1.40 perf-profile.children.cycles-pp.memdup_user
1.81 -0.2 1.62 perf-profile.children.cycles-pp.__get_user_4
1.26 ? 3% -0.2 1.08 ? 3% perf-profile.children.cycles-pp.__x64_sys_fcntl
1.32 ? 2% -0.2 1.14 ? 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
2.06 ? 2% -0.2 1.90 ? 3% perf-profile.children.cycles-pp.syscall_enter_from_user_mode
1.12 ? 3% -0.1 0.99 ? 2% perf-profile.children.cycles-pp.security_file_ioctl
0.84 ? 3% -0.1 0.73 ? 3% perf-profile.children.cycles-pp.ksys_lseek
0.29 ? 4% -0.1 0.18 ? 4% perf-profile.children.cycles-pp.generic_file_rw_checks
0.76 ? 3% -0.1 0.68 perf-profile.children.cycles-pp.amd_clear_divider
0.84 ? 3% -0.1 0.75 ? 3% perf-profile.children.cycles-pp.__put_user_4
0.86 ? 4% -0.1 0.78 ? 3% perf-profile.children.cycles-pp._raw_spin_lock
0.53 ? 3% -0.1 0.46 ? 4% perf-profile.children.cycles-pp.__fdget_pos
0.19 ? 11% -0.1 0.12 ? 10% perf-profile.children.cycles-pp.stress_mwc8
0.54 ? 5% -0.1 0.48 ? 6% perf-profile.children.cycles-pp.__check_object_size
0.73 ? 2% -0.1 0.67 ? 5% perf-profile.children.cycles-pp.__fdget
0.49 ? 2% -0.1 0.43 ? 3% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.51 ? 4% -0.1 0.45 ? 5% perf-profile.children.cycles-pp.ioctl@plt
0.58 ? 3% -0.0 0.54 ? 4% perf-profile.children.cycles-pp.__get_user_2
0.38 ? 3% -0.0 0.33 ? 4% perf-profile.children.cycles-pp.__kmem_cache_alloc_node
0.44 ? 3% -0.0 0.40 ? 3% perf-profile.children.cycles-pp.__libc_fcntl64
0.24 ? 6% -0.0 0.20 ? 7% perf-profile.children.cycles-pp.do_fcntl
0.48 ? 3% -0.0 0.44 ? 2% perf-profile.children.cycles-pp.set_close_on_exec
0.16 ? 8% -0.0 0.14 ? 8% perf-profile.children.cycles-pp.__check_heap_object
0.00 +0.2 0.25 ? 4% perf-profile.children.cycles-pp.fsnotify_perm
0.57 +0.6 1.15 ? 3% perf-profile.children.cycles-pp.aa_file_perm
85.52 +1.4 86.91 perf-profile.children.cycles-pp.ioctl
0.00 +1.6 1.55 perf-profile.children.cycles-pp.__fsnotify_parent
62.60 +4.0 66.55 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
59.77 +4.3 64.05 perf-profile.children.cycles-pp.do_syscall_64
47.98 +5.7 53.66 perf-profile.children.cycles-pp.__x64_sys_ioctl
21.64 +7.3 28.98 perf-profile.children.cycles-pp.do_vfs_ioctl
8.29 ? 4% +7.4 15.74 ? 6% perf-profile.children.cycles-pp.apparmor_file_permission
8.78 ? 4% +7.9 16.64 ? 5% perf-profile.children.cycles-pp.security_file_permission
3.30 ? 2% +8.7 11.96 ? 3% perf-profile.children.cycles-pp.ioctl_file_clone
1.68 +8.9 10.55 ? 3% perf-profile.children.cycles-pp.vfs_clone_file_range
10.33 -1.3 9.02 ? 3% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
11.15 -1.2 9.92 ? 2% perf-profile.self.cycles-pp.ioctl
7.55 ? 3% -0.9 6.61 perf-profile.self.cycles-pp.__fget_light
3.16 ? 4% -0.5 2.69 ? 2% perf-profile.self.cycles-pp.do_vfs_ioctl
2.95 ? 2% -0.4 2.55 ? 2% perf-profile.self.cycles-pp.__x64_sys_ioctl
3.32 -0.4 2.93 ? 2% perf-profile.self.cycles-pp.do_syscall_64
3.08 ? 2% -0.4 2.72 ? 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
3.13 -0.4 2.78 ? 2% perf-profile.self.cycles-pp.entry_SYSCALL_64
2.39 ? 2% -0.3 2.10 ? 2% perf-profile.self.cycles-pp.ioctl_preallocate
0.57 ? 2% -0.3 0.31 ? 9% perf-profile.self.cycles-pp.do_clone_file_range
2.02 ? 2% -0.3 1.77 ? 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
1.54 ? 4% -0.2 1.29 ? 3% perf-profile.self.cycles-pp.stress_file_ioctl
1.83 -0.2 1.62 ? 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
2.32 ? 3% -0.2 2.13 perf-profile.self.cycles-pp.syscall_return_via_sysret
1.77 -0.2 1.58 perf-profile.self.cycles-pp.__get_user_4
1.28 ? 2% -0.2 1.11 ? 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
1.76 ? 2% -0.1 1.62 ? 3% perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.25 ? 6% -0.1 0.12 ? 8% perf-profile.self.cycles-pp.generic_file_rw_checks
0.48 ? 2% -0.1 0.38 ? 4% perf-profile.self.cycles-pp.ioctl_file_clone
0.79 ? 3% -0.1 0.70 ? 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
0.81 ? 3% -0.1 0.73 ? 4% perf-profile.self.cycles-pp.__put_user_4
0.81 ? 5% -0.1 0.73 ? 3% perf-profile.self.cycles-pp._raw_spin_lock
0.52 ? 4% -0.1 0.44 ? 3% perf-profile.self.cycles-pp.amd_clear_divider
0.17 ? 11% -0.1 0.12 ? 10% perf-profile.self.cycles-pp.stress_mwc8
0.57 ? 3% -0.0 0.52 ? 4% perf-profile.self.cycles-pp.__get_user_2
0.42 ? 4% -0.0 0.38 ? 3% perf-profile.self.cycles-pp.__libc_fcntl64
0.30 ? 3% -0.0 0.26 ? 5% perf-profile.self.cycles-pp.__x64_sys_fcntl
0.22 ? 5% -0.0 0.18 ? 6% perf-profile.self.cycles-pp.do_fcntl
0.28 ? 3% -0.0 0.24 ? 2% perf-profile.self.cycles-pp.__kmem_cache_alloc_node
0.00 +0.2 0.22 ? 4% perf-profile.self.cycles-pp.fsnotify_perm
0.49 ? 3% +0.4 0.92 ? 2% perf-profile.self.cycles-pp.security_file_permission
0.46 ? 2% +0.5 0.96 ? 2% perf-profile.self.cycles-pp.aa_file_perm
0.00 +1.5 1.52 ? 2% perf-profile.self.cycles-pp.__fsnotify_parent
7.75 ? 4% +6.8 14.58 ? 7% perf-profile.self.cycles-pp.apparmor_file_permission
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
On Wed, Jan 31, 2024 at 4:13 PM kenel test robot <oliver.sang@intelcom> wrote:
>
>
>
> Hello,
>
> kernel test robot noticed a -11.2% regression of stress-ng.file-ioctl.ops_per_sec on:
>
>
> commit: dfad37051ade6ac0d404ef4913f3bd01954ee51c ("remap_range: move permission hooks out of do_clone_file_range()")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
Can you please try this fix:
7d4213664bda remap_range: move sanity checks out of do_clone_file_range()
from:
https://github.com/amir73il/linux ovl-fixes
Thanks,
Amir.
On Wed, Jan 31, 2024 at 5:47 PM Amir Goldstein <[email protected]> wrote:
>
> On Wed, Jan 31, 2024 at 4:13 PM kenel test robot <[email protected]> wrote:
> >
> >
> >
> > Hello,
> >
> > kernel test robot noticed a -11.2% regression of stress-ng.file-ioctl.ops_per_sec on:
> >
> >
> > commit: dfad37051ade6ac0d404ef4913f3bd01954ee51c ("remap_range: move permission hooks out of do_clone_file_range()")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >
>
> Can you please try this fix:
>
> 7d4213664bda remap_range: move sanity checks out of do_clone_file_range()
>
> from:
>
> https://github.com/amir73il/linux ovl-fixes
>
Sorry, Oliver, this was a buggy commit.
I pushed this fixes version to ovl-fixes branch:
1c5e7db8e1b2 remap_range: merge do_clone_file_range() into
vfs_clone_file_range()
Can you please test.
Thanks,
Amir.
hi, Amir,
On Fri, Feb 02, 2024 at 11:13:56AM +0200, Amir Goldstein wrote:
> On Wed, Jan 31, 2024 at 5:47 PM Amir Goldstein <[email protected]> wrote:
> >
> > On Wed, Jan 31, 2024 at 4:13 PM kenel test robot <[email protected]> wrote:
> > >
> > >
> > >
> > > Hello,
> > >
> > > kernel test robot noticed a -11.2% regression of stress-ng.file-ioctl.ops_per_sec on:
> > >
> > >
> > > commit: dfad37051ade6ac0d404ef4913f3bd01954ee51c ("remap_range: move permission hooks out of do_clone_file_range()")
> > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > >
> >
> > Can you please try this fix:
> >
> > 7d4213664bda remap_range: move sanity checks out of do_clone_file_range()
> >
> > from:
> >
> > https://github.com/amir73il/linux ovl-fixes
> >
>
> Sorry, Oliver, this was a buggy commit.
> I pushed this fixes version to ovl-fixes branch:
>
> 1c5e7db8e1b2 remap_range: merge do_clone_file_range() into
> vfs_clone_file_range()
>
> Can you please test.
the regression disappeared by above commit in our tests.
I noticed this branch is based on v6.8-rc2, so I directly tested upon it and its
parent (3f01e53bf6). I found 3f01e53bf6 has same data as dfad37051a we reported.
and on 1c5e7db8e1b2, the performance back to the same level before dfad37051a.
below is the summary:
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/file-ioctl/stress-ng/60s
commit:
d53471ba6f ("splice: remove permission hook from iter_file_splice_write()")
dfad37051a ("remap_range: move permission hooks out of do_clone_file_range()")
3f01e53bf6 ("MAINTAINERS: update overlayfs git tree")
1c5e7db8e1 ("remap_range: merge do_clone_file_range() into vfs_clone_file_range()")
d53471ba6f7ae97a dfad37051ade6ac0d404ef4913f 3f01e53bf658495e01cab85d82a 1c5e7db8e1b25b9ef86a9026862
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
95739218 -11.2% 84990543 ± 2% -11.3% 84951004 +0.7% 96455282 stress-ng.file-ioctl.ops
1595650 -11.2% 1416506 ± 2% -11.3% 1415846 +0.7% 1607584 stress-ng.file-ioctl.ops_per_sec
below is the details FYI:
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/file-ioctl/stress-ng/60s
commit:
d53471ba6f ("splice: remove permission hook from iter_file_splice_write()")
dfad37051a ("remap_range: move permission hooks out of do_clone_file_range()")
3f01e53bf6 ("MAINTAINERS: update overlayfs git tree")
1c5e7db8e1 ("remap_range: merge do_clone_file_range() into vfs_clone_file_range()")
d53471ba6f7ae97a dfad37051ade6ac0d404ef4913f 3f01e53bf658495e01cab85d82a 1c5e7db8e1b25b9ef86a9026862
---------------- --------------------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev %change %stddev
\ | \ | \ | \
2.57 -0.3 2.27 -0.3 2.25 -0.0 2.52 mpstat.cpu.all.usr%
7.40 +3.4% 7.65 +4.1% 7.71 +0.4% 7.43 iostat.cpu.system
2.50 -11.5% 2.22 -12.5% 2.19 -1.9% 2.46 iostat.cpu.user
49702 ± 6% -3.4% 48023 ± 12% +13.4% 56347 ± 8% +20.0% 59637 ± 2% meminfo.AnonHugePages
74632 -0.2% 74463 -57.7% 31584 -57.6% 31669 meminfo.Percpu
87960 ± 2% +1.6% 89400 ± 5% +12.2% 98666 ± 7% +4.3% 91739 ± 9% numa-meminfo.node0.SUnreclaim
69529 ± 3% -1.9% 68208 ± 7% -16.2% 58254 ± 13% -5.5% 65677 ± 14% numa-meminfo.node1.SUnreclaim
21990 ± 2% +1.6% 22350 ± 5% +12.2% 24666 ± 7% +4.3% 22934 ± 9% numa-vmstat.node0.nr_slab_unreclaimable
17382 ± 3% -1.9% 17052 ± 7% -16.2% 14563 ± 13% -5.5% 16419 ± 14% numa-vmstat.node1.nr_slab_unreclaimable
267.41 +4.2% 278.66 +4.7% 280.04 +0.7% 269.27 time.system_time
90.19 -12.5% 78.96 -14.0% 77.59 -2.0% 88.37 time.user_time
34.41 +0.5% 34.57 +3.7% 35.69 +3.2% 35.51 boot-time.boot
23.41 +0.5% 23.52 +5.5% 24.69 +4.5% 24.47 boot-time.dhcp
1991 +0.6% 2002 +4.0% 2071 +3.4% 2059 boot-time.idle
1434 ± 10% -18.2% 1172 ± 18% -57.1% 615.00 ± 9% -63.1% 529.50 ± 11% perf-c2c.DRAM.remote
1117 ± 9% -7.0% 1039 ± 9% -52.8% 527.67 ± 11% -57.1% 480.00 ± 13% perf-c2c.HITM.local
167.67 ± 14% -5.5% 158.50 ± 31% -41.0% 99.00 ± 14% -57.6% 71.17 ± 22% perf-c2c.HITM.remote
95739218 -11.2% 84990543 ± 2% -11.3% 84951004 +0.7% 96455282 stress-ng.file-ioctl.ops
1595650 -11.2% 1416506 ± 2% -11.3% 1415846 +0.7% 1607584 stress-ng.file-ioctl.ops_per_sec
267.41 +4.2% 278.66 +4.7% 280.04 +0.7% 269.27 stress-ng.time.system_time
90.19 -12.5% 78.96 -14.0% 77.59 -2.0% 88.37 stress-ng.time.user_time
44.89 ± 16% +1.5% 45.57 ± 19% -100.0% 0.00 -100.0% 0.00 sched_debug.cfs_rq:/.util_est_enqueued.avg
515.17 ± 2% +5.4% 542.92 ± 9% -100.0% 0.00 -100.0% 0.00 sched_debug.cfs_rq:/.util_est_enqueued.max
135.06 ± 8% +2.3% 138.12 ± 10% -100.0% 0.00 -100.0% 0.00 sched_debug.cfs_rq:/.util_est_enqueued.stddev
891.47 -0.4% 887.93 +15.8% 1031 +14.9% 1024 sched_debug.cpu.clock_task.stddev
12529207 +0.0% 12529207 -50.2% 6237751 -50.2% 6237751 sched_debug.sysctl_sched.sysctl_sched_features
86772 -0.4% 86400 +4.9% 91010 +4.7% 90835 proc-vmstat.nr_anon_pages
105949 -0.4% 105517 +5.1% 111368 +4.8% 111042 proc-vmstat.nr_inactive_anon
105949 -0.4% 105517 +5.1% 111368 +4.8% 111042 proc-vmstat.nr_zone_inactive_anon
425778 +0.8% 429132 +1.3% 431364 +2.4% 435972 proc-vmstat.pgalloc_normal
415867 +0.5% 417794 +1.2% 420781 +1.9% 423729 proc-vmstat.pgfree
696576 ± 2% -3.8% 670080 ± 7% -100.0% 0.00 -100.0% 0.00 proc-vmstat.unevictable_pgs_scanned
0.00 ± 17% +0.0% 0.00 ± 17% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.tpm_try_transmit.tpm_transmit
0.00 ± 20% +9.1% 0.00 ± 28% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.wait_for_tpm_stat.tpm_tis_send_data
0.00 ± 31% +0.0% 0.00 ± 17% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.hwrng_fillfn.kthread.ret_from_fork
0.01 ± 28% -28.3% 0.01 ± 54% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.00 ± 30% +13.3% 0.00 ± 13% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.tpm_try_transmit.tpm_transmit
0.00 ± 39% -4.8% 0.00 ± 41% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.wait_for_tpm_stat.tpm_tis_send_data
0.00 ± 31% +0.0% 0.00 ± 17% -100.0% 0.00 -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.hwrng_fillfn.kthread.ret_from_fork
564.55 ± 37% -48.8% 288.96 ± 73% +10.0% 621.20 ± 62% -90.2% 55.05 ±223% perf-sched.wait_and_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.02 ± 27% -17.6% 0.01 ± 24% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
4.00 ± 54% +8.3% 4.33 ±126% +29.2% 5.17 ± 51% -87.5% 0.50 ±223% perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
51.83 ± 20% -13.5% 44.83 ± 34% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
1013 -1.2% 1001 ±100% +48.2% 1502 ± 63% -83.8% 164.00 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.06 ± 41% -23.9% 0.04 ± 60% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
564.54 ± 37% -44.3% 314.30 ± 57% +10.2% 622.40 ± 62% -72.6% 154.89 ± 80% perf-sched.wait_time.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.01 ± 61% +29.4% 0.02 ± 16% +1.2% 0.01 ± 9% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
0.02 ± 29% -19.2% 0.01 ± 21% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.02 ± 27% -17.6% 0.01 ± 24% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.71 +0.0% 0.71 -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.tpm_try_transmit.tpm_transmit
0.45 +0.1% 0.45 -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.usleep_range_state.wait_for_tpm_stat.tpm_tis_send_data
0.99 -0.1% 0.99 -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.hwrng_fillfn.kthread.ret_from_fork
0.00 ±145% +160.0% 0.00 ±114% +19320.0% 0.16 ±217% +20700.0% 0.17 ±202% perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
0.03 ± 64% +14.2% 0.03 ± 63% -13.7% 0.03 ± 18% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
0.04 ± 59% -5.6% 0.03 ± 57% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt
0.06 ± 41% -23.9% 0.04 ± 60% -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.98 -0.1% 0.98 -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.tpm_try_transmit.tpm_transmit
0.99 +0.1% 0.99 -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.usleep_range_state.wait_for_tpm_stat.tpm_tis_send_data
0.99 -0.1% 0.99 -100.0% 0.00 -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.hwrng_fillfn.kthread.ret_from_fork
0.00 ±145% +160.0% 0.00 ±114% +19320.0% 0.16 ±217% +20700.0% 0.17 ±202% perf-sched.wait_time.max.ms.schedule_timeout.khugepaged_wait_work.khugepaged.kthread
0.12 ± 9% +37.6% 0.16 ± 3% +46.3% 0.17 ± 2% +0.9% 0.12 ± 8% perf-stat.i.MPKI
5.619e+09 -4.9% 5.346e+09 -10.1% 5.053e+09 -8.3% 5.154e+09 perf-stat.i.branch-instructions
0.13 +0.0 0.13 ± 5% +0.0 0.13 ± 2% +0.0 0.14 perf-stat.i.branch-miss-rate%
8104366 -3.2% 7841290 ± 5% -2.2% 7928723 ± 2% +0.4% 8134981 perf-stat.i.branch-misses
25.26 ± 12% +5.4 30.67 ± 2% +6.1 31.39 ± 2% -2.8 22.49 ± 12% perf-stat.i.cache-miss-rate%
3226271 ± 8% +32.3% 4268159 ± 2% +34.1% 4327362 ± 2% -6.7% 3008704 ± 7% perf-stat.i.cache-misses
13880671 ± 2% +7.6% 14934433 +7.0% 14856536 +4.2% 14464224 ± 3% perf-stat.i.cache-references
0.83 +3.9% 0.86 +8.9% 0.90 +8.2% 0.89 perf-stat.i.cpi
7405 ± 8% -26.1% 5473 ± 2% -27.2% 5395 ± 2% +7.5% 7963 ± 7% perf-stat.i.cycles-between-cache-misses
0.02 ±210% +0.0 0.03 ±217% -0.0 0.00 ± 7% -0.0 0.00 ± 6% perf-stat.i.dTLB-load-miss-rate%
1198124 ±210% +87.7% 2248507 ±217% -95.7% 51944 ± 7% -95.8% 50495 ± 6% perf-stat.i.dTLB-load-misses
7.817e+09 -2.7% 7.61e+09 -5.8% 7.364e+09 -6.8% 7.285e+09 perf-stat.i.dTLB-loads
0.00 ± 4% +0.0 0.00 ± 3% +0.0 0.00 ± 3% +0.0 0.00 ± 2% perf-stat.i.dTLB-store-miss-rate%
26775 ± 3% -2.5% 26108 ± 2% -7.7% 24702 ± 3% -1.4% 26389 ± 2% perf-stat.i.dTLB-store-misses
5.186e+09 -6.0% 4.873e+09 -10.8% 4.624e+09 -8.4% 4.749e+09 perf-stat.i.dTLB-stores
2.807e+10 -3.9% 2.696e+10 -8.3% 2.575e+10 -7.5% 2.597e+10 perf-stat.i.instructions
1.21 -3.7% 1.17 -8.1% 1.11 -7.6% 1.12 perf-stat.i.ipc
257.16 +12.9% 290.46 +12.7% 289.89 +2.2% 262.78 ± 2% perf-stat.i.metric.K/sec
290.80 -4.2% 278.45 -8.5% 266.14 -7.7% 268.43 perf-stat.i.metric.M/sec
1580051 ± 11% +38.0% 2180479 ± 5% +41.9% 2242249 ± 3% -7.4% 1463122 ± 12% perf-stat.i.node-load-misses
228848 ± 22% +116.2% 494834 ± 27% +83.2% 419274 ± 22% -18.7% 186032 ± 32% perf-stat.i.node-loads
739626 ± 15% +28.2% 948465 ± 11% +36.7% 1011333 ± 8% -8.4% 677284 ± 4% perf-stat.i.node-store-misses
0.11 ± 9% +37.7% 0.16 ± 3% +46.1% 0.17 ± 2% +0.8% 0.12 ± 8% perf-stat.overall.MPKI
0.14 +0.0 0.15 ± 5% +0.0 0.16 ± 2% +0.0 0.16 perf-stat.overall.branch-miss-rate%
23.29 ± 11% +5.3 28.58 ± 2% +5.8 29.13 ± 2% -2.4 20.89 ± 11% perf-stat.overall.cache-miss-rate%
0.82 +3.9% 0.86 +8.8% 0.90 +8.1% 0.89 perf-stat.overall.cpi
7231 ± 8% -25.1% 5416 ± 2% -26.1% 5343 ± 2% +7.0% 7740 ± 6% perf-stat.overall.cycles-between-cache-misses
0.02 ±210% +0.0 0.03 ±217% -0.0 0.00 ± 7% -0.0 0.00 ± 6% perf-stat.overall.dTLB-load-miss-rate%
0.00 ± 3% +0.0 0.00 ± 3% +0.0 0.00 ± 2% +0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate%
1.21 -3.7% 1.17 -8.1% 1.11 -7.5% 1.12 perf-stat.overall.ipc
5.524e+09 -4.8% 5.257e+09 -10.1% 4.967e+09 -8.3% 5.068e+09 perf-stat.ps.branch-instructions
7962517 -3.1% 7713102 ± 5% -2.3% 7781058 +0.5% 8006027 perf-stat.ps.branch-misses
3170718 ± 8% +32.4% 4196610 ± 2% +34.1% 4253192 ± 2% -6.7% 2957362 ± 7% perf-stat.ps.cache-misses
13646445 ± 2% +7.6% 14686495 ± 2% +7.0% 14601960 +4.2% 14219304 ± 3% perf-stat.ps.cache-references
1178079 ±210% +87.7% 2210990 ±217% -95.7% 51043 ± 7% -95.8% 49643 ± 6% perf-stat.ps.dTLB-load-misses
7.685e+09 -2.6% 7.483e+09 -5.8% 7.24e+09 -6.8% 7.163e+09 perf-stat.ps.dTLB-loads
26301 ± 3% -2.5% 25656 ± 2% -7.8% 24251 ± 3% -1.5% 25913 ± 2% perf-stat.ps.dTLB-store-misses
5.099e+09 -6.0% 4.792e+09 -10.8% 4.546e+09 -8.4% 4.67e+09 perf-stat.ps.dTLB-stores
2.759e+10 -3.9% 2.651e+10 -8.3% 2.531e+10 -7.5% 2.553e+10 perf-stat.ps.instructions
1553350 ± 11% +38.1% 2144498 ± 5% +41.9% 2204343 ± 3% -7.4% 1438512 ± 12% perf-stat.ps.node-load-misses
224907 ± 22% +116.2% 486304 ± 27% +83.2% 412125 ± 22% -18.7% 182868 ± 32% perf-stat.ps.node-loads
727127 ± 15% +28.3% 932767 ± 11% +36.7% 994262 ± 8% -8.4% 665892 ± 4% perf-stat.ps.node-store-misses
1.668e+12 -3.4% 1.611e+12 ± 2% -8.6% 1.524e+12 -7.5% 1.544e+12 perf-stat.total.instructions
5.57 ± 3% -0.7 4.85 ± 2% -5.6 0.00 -5.6 0.00 perf-profile.calltrace.cycles-pp.__fget_light.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
0.89 ± 23% -0.4 0.45 ± 44% -0.9 0.00 -0.9 0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
4.28 ± 5% -0.3 3.94 ± 9% -0.5 3.79 +0.0 4.29 perf-profile.calltrace.cycles-pp._copy_from_user.ioctl_preallocate.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.30 ± 2% -0.3 2.00 -0.3 1.99 +0.0 2.32 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
1.69 ± 3% -0.3 1.39 ± 4% -0.4 1.26 ± 2% -0.2 1.48 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
1.99 ± 2% -0.3 1.72 -0.2 1.74 +0.0 2.02 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.28 ±101% -0.2 0.08 ±223% +0.3 0.58 ± 9% +0.5 0.79 ± 27% perf-profile.calltrace.cycles-pp.security_file_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
2.27 -0.2 2.09 ± 5% -0.2 2.06 ± 2% +0.1 2.32 perf-profile.calltrace.cycles-pp._copy_from_user.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.16 ± 3% -0.2 1.00 ± 3% -0.1 1.01 +0.0 1.19 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.60 ± 4% -0.2 0.44 ± 45% -0.6 0.00 -0.6 0.00 perf-profile.calltrace.cycles-pp.__fget_light.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.47 ± 11% -0.1 1.36 -0.5 0.95 ± 23% -0.2 1.31 ± 8% perf-profile.calltrace.cycles-pp.memdup_user.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +0.0 0.00 +0.5 0.52 ± 3% perf-profile.calltrace.cycles-pp.__fdget_raw.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +0.5 0.53 ± 46% +0.8 0.83 ± 26% perf-profile.calltrace.cycles-pp.__fdget.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64
0.00 +0.0 0.00 +5.6 5.59 +6.5 6.50 ± 3% perf-profile.calltrace.cycles-pp.__fdget.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
0.00 +0.0 0.00 +7.3 7.28 ± 3% +0.0 0.00 perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.remap_verify_area.vfs_clone_file_range.ioctl_file_clone
0.00 +0.0 0.00 +7.6 7.64 ± 2% +0.0 0.00 perf-profile.calltrace.cycles-pp.security_file_permission.remap_verify_area.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl
0.00 +0.0 0.00 +8.2 8.20 ± 2% +0.0 0.00 perf-profile.calltrace.cycles-pp.remap_verify_area.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
0.00 +1.5 1.52 ± 2% +1.3 1.33 ± 15% +0.0 0.00 perf-profile.calltrace.cycles-pp.__fsnotify_parent.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
0.00 +6.9 6.94 ± 6% +0.0 0.00 +0.0 0.00 perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl
0.00 +7.4 7.41 ± 6% +0.0 0.00 +0.0 0.00 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl
21.11 +7.4 28.53 +7.6 28.73 -0.8 20.32 perf-profile.calltrace.cycles-pp.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
3.18 ± 2% +8.7 11.87 ± 3% +9.0 12.22 -1.1 2.04 ± 8% perf-profile.calltrace.cycles-pp.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.46 ± 9% +8.9 10.36 ± 4% +9.3 10.77 -1.5 0.00 perf-profile.calltrace.cycles-pp.vfs_clone_file_range.ioctl_file_clone.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64
10.70 -1.3 9.39 ± 3% -1.6 9.11 +0.0 10.73 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
11.31 -1.1 10.24 ± 2% -1.5 9.76 -0.2 11.11 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64
7.87 ± 3% -1.0 6.90 -7.9 0.00 -7.9 0.00 perf-profile.children.cycles-pp.__fget_light
5.13 -0.7 4.46 ± 2% -1.3 3.82 -0.8 4.33 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
7.74 ± 3% -0.6 7.09 ± 5% -0.8 6.90 +0.1 7.80 perf-profile.children.cycles-pp._copy_from_user
0.89 -0.4 0.46 ± 5% -0.5 0.40 ± 5% -0.9 0.00 perf-profile.children.cycles-pp.do_clone_file_range
3.45 ± 2% -0.4 3.10 -0.4 3.08 +0.0 3.48 perf-profile.children.cycles-pp.llseek
1.80 ± 4% -0.3 1.49 ± 3% -0.2 1.60 ± 2% +0.0 1.80 ± 3% perf-profile.children.cycles-pp.stress_file_ioctl
1.83 -0.2 1.63 ± 4% -0.2 1.63 ± 3% -0.0 1.83 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
1.53 ± 3% -0.2 1.34 ± 4% -1.5 0.00 -1.5 0.00 perf-profile.children.cycles-pp.exit_to_user_mode_prepare
2.32 ± 3% -0.2 2.13 -0.3 2.03 ± 2% +0.0 2.34 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret
1.58 ± 2% -0.2 1.40 -0.3 1.26 ± 3% -0.2 1.40 perf-profile.children.cycles-pp.memdup_user
1.81 -0.2 1.62 -0.2 1.57 ± 2% -0.0 1.80 ± 4% perf-profile.children.cycles-pp.__get_user_4
1.26 ± 3% -0.2 1.08 ± 3% -0.1 1.12 ± 2% +0.0 1.31 ± 3% perf-profile.children.cycles-pp.__x64_sys_fcntl
1.32 ± 2% -0.2 1.14 ± 2% -0.4 0.90 ± 4% -0.3 1.04 perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
2.06 ± 2% -0.2 1.90 ± 3% -2.1 0.00 -2.1 0.00 perf-profile.children.cycles-pp.syscall_enter_from_user_mode
1.12 ± 3% -0.1 0.99 ± 2% +0.0 1.13 +0.2 1.27 ± 2% perf-profile.children.cycles-pp.security_file_ioctl
0.84 ± 3% -0.1 0.73 ± 3% -0.1 0.77 ± 2% +0.1 0.92 ± 3% perf-profile.children.cycles-pp.ksys_lseek
0.29 ± 4% -0.1 0.18 ± 4% -0.1 0.16 ± 5% -0.1 0.17 ± 8% perf-profile.children.cycles-pp.generic_file_rw_checks
0.76 ± 3% -0.1 0.68 -0.1 0.68 -0.0 0.75 ± 4% perf-profile.children.cycles-pp.amd_clear_divider
0.84 ± 3% -0.1 0.75 ± 3% -0.1 0.77 ± 2% +0.1 0.89 ± 2% perf-profile.children.cycles-pp.__put_user_4
0.86 ± 4% -0.1 0.78 ± 3% -0.1 0.78 ± 3% +0.0 0.89 ± 2% perf-profile.children.cycles-pp._raw_spin_lock
0.53 ± 3% -0.1 0.46 ± 4% -0.0 0.50 ± 3% +0.1 0.60 ± 6% perf-profile.children.cycles-pp.__fdget_pos
0.19 ± 11% -0.1 0.12 ± 10% -0.1 0.12 ± 9% +0.0 0.22 ± 6% perf-profile.children.cycles-pp.stress_mwc8
0.54 ± 5% -0.1 0.48 ± 6% -0.1 0.45 ± 5% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.__check_object_size
0.73 ± 2% -0.1 0.67 ± 5% +6.0 6.78 +7.1 7.84 ± 2% perf-profile.children.cycles-pp.__fdget
0.49 ± 2% -0.1 0.43 ± 3% -0.2 0.34 ± 3% -0.1 0.40 ± 3% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.51 ± 4% -0.1 0.45 ± 5% -0.0 0.48 ± 18% -0.0 0.49 perf-profile.children.cycles-pp.ioctl@plt
0.58 ± 3% -0.0 0.54 ± 4% -0.1 0.53 ± 4% +0.0 0.59 ± 3% perf-profile.children.cycles-pp.__get_user_2
0.38 ± 3% -0.0 0.33 ± 4% -0.4 0.00 -0.4 0.00 perf-profile.children.cycles-pp.__kmem_cache_alloc_node
0.44 ± 3% -0.0 0.40 ± 3% -0.1 0.39 ± 5% +0.0 0.45 ± 5% perf-profile.children.cycles-pp.__libc_fcntl64
0.24 ± 6% -0.0 0.20 ± 7% -0.0 0.21 ± 5% -0.0 0.23 ± 5% perf-profile.children.cycles-pp.do_fcntl
0.48 ± 3% -0.0 0.44 ± 2% -0.0 0.44 ± 3% +0.0 0.49 ± 2% perf-profile.children.cycles-pp.set_close_on_exec
0.38 ± 6% -0.0 0.36 ± 3% -0.0 0.38 ± 4% +0.1 0.44 ± 4% perf-profile.children.cycles-pp.check_flag
0.26 ± 5% -0.0 0.24 ± 11% -0.1 0.20 ± 9% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.check_heap_object
0.16 ± 8% -0.0 0.14 ± 8% -0.0 0.15 ± 9% +0.0 0.17 ± 9% perf-profile.children.cycles-pp.__check_heap_object
0.10 ± 13% -0.0 0.08 ± 11% +0.0 0.11 ± 10% +0.0 0.13 ± 10% perf-profile.children.cycles-pp.security_file_fcntl
0.20 ± 7% -0.0 0.18 ± 5% -0.0 0.17 ± 5% +0.0 0.21 ± 12% perf-profile.children.cycles-pp.inode_get_bytes
0.13 ± 7% -0.0 0.12 ± 15% -0.0 0.11 ± 9% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.__virt_addr_valid
0.08 ± 11% -0.0 0.08 ± 6% +0.2 0.26 ± 3% +0.2 0.30 ± 6% perf-profile.children.cycles-pp.kfree
0.08 ± 14% -0.0 0.08 ± 6% +0.0 0.10 ± 10% +0.0 0.10 ± 9% perf-profile.children.cycles-pp.__errno_location
0.06 ± 11% +0.0 0.06 ± 11% +0.4 0.46 ± 5% +0.5 0.54 ± 3% perf-profile.children.cycles-pp.__fdget_raw
0.00 +0.0 0.00 +8.3 8.30 ± 2% +0.0 0.00 perf-profile.children.cycles-pp.remap_verify_area
0.28 ± 3% +0.0 0.30 ± 7% -0.0 0.26 ± 11% -0.2 0.06 ± 14% perf-profile.children.cycles-pp.__cond_resched
0.00 +0.2 0.25 ± 4% +0.0 0.00 +0.0 0.00 perf-profile.children.cycles-pp.fsnotify_perm
0.57 +0.6 1.15 ± 3% +0.6 1.13 ± 2% +0.0 0.60 ± 3% perf-profile.children.cycles-pp.aa_file_perm
85.52 +1.4 86.91 +1.3 86.85 -0.0 85.51 perf-profile.children.cycles-pp.ioctl
0.00 +1.6 1.55 +1.5 1.52 +0.0 0.00 perf-profile.children.cycles-pp.__fsnotify_parent
62.60 +4.0 66.55 +4.5 67.05 +0.1 62.67 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
59.77 +4.3 64.05 +4.8 64.53 +0.0 59.82 perf-profile.children.cycles-pp.do_syscall_64
47.98 +5.7 53.66 +6.2 54.22 +0.1 48.10 perf-profile.children.cycles-pp.__x64_sys_ioctl
21.64 +7.3 28.98 +7.5 29.19 -0.8 20.85 perf-profile.children.cycles-pp.do_vfs_ioctl
8.29 ± 4% +7.4 15.74 ± 6% +7.9 16.18 ± 3% +0.3 8.60 ± 9% perf-profile.children.cycles-pp.apparmor_file_permission
8.78 ± 4% +7.9 16.64 ± 5% +8.2 17.03 ± 3% +0.3 9.10 ± 9% perf-profile.children.cycles-pp.security_file_permission
3.30 ± 2% +8.7 11.96 ± 3% +9.0 12.30 -1.1 2.22 ± 3% perf-profile.children.cycles-pp.ioctl_file_clone
1.68 +8.9 10.55 ± 3% +9.2 10.92 -1.1 0.60 ± 7% perf-profile.children.cycles-pp.vfs_clone_file_range
10.33 -1.3 9.02 ± 3% -1.5 8.80 +0.1 10.38 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
11.15 -1.2 9.92 ± 2% -1.4 9.77 -0.1 11.07 ± 2% perf-profile.self.cycles-pp.ioctl
7.55 ± 3% -0.9 6.61 -7.6 0.00 -7.6 0.00 perf-profile.self.cycles-pp.__fget_light
7.54 ± 3% -0.6 6.92 ± 6% -0.8 6.73 +0.1 7.59 perf-profile.self.cycles-pp._copy_from_user
3.16 ± 4% -0.5 2.69 ± 2% -0.5 2.68 -0.1 3.11 ± 2% perf-profile.self.cycles-pp.do_vfs_ioctl
2.95 ± 2% -0.4 2.55 ± 2% -0.3 2.64 +0.1 3.04 ± 2% perf-profile.self.cycles-pp.__x64_sys_ioctl
3.32 -0.4 2.93 ± 2% +1.7 5.00 +2.3 5.66 ± 2% perf-profile.self.cycles-pp.do_syscall_64
3.08 ± 2% -0.4 2.72 ± 3% -0.3 2.74 ± 2% +0.0 3.10 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
3.13 -0.4 2.78 ± 2% -0.4 2.73 ± 2% +0.0 3.15 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64
2.39 ± 2% -0.3 2.10 ± 2% -0.3 2.09 ± 2% -0.0 2.38 perf-profile.self.cycles-pp.ioctl_preallocate
0.57 ± 2% -0.3 0.31 ± 9% -0.3 0.26 ± 5% -0.6 0.00 perf-profile.self.cycles-pp.do_clone_file_range
2.02 ± 2% -0.3 1.77 ± 3% +0.2 2.26 ± 2% +0.5 2.54 ± 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
1.54 ± 4% -0.2 1.29 ± 3% -0.2 1.37 ± 3% +0.0 1.55 ± 4% perf-profile.self.cycles-pp.stress_file_ioctl
1.83 -0.2 1.62 ± 4% -0.2 1.62 ± 3% -0.0 1.83 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
2.32 ± 3% -0.2 2.13 -0.3 2.03 ± 2% +0.0 2.33 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret
1.77 -0.2 1.58 -0.2 1.54 ± 2% -0.0 1.75 ± 4% perf-profile.self.cycles-pp.__get_user_4
1.28 ± 2% -0.2 1.11 ± 4% -1.3 0.00 -1.3 0.00 perf-profile.self.cycles-pp.exit_to_user_mode_prepare
1.76 ± 2% -0.1 1.62 ± 3% -1.8 0.00 -1.8 0.00 perf-profile.self.cycles-pp.syscall_enter_from_user_mode
0.25 ± 6% -0.1 0.12 ± 8% -0.2 0.10 ± 9% -0.1 0.16 ± 6% perf-profile.self.cycles-pp.generic_file_rw_checks
0.48 ± 2% -0.1 0.38 ± 4% -0.1 0.34 ± 6% -0.0 0.47 ± 3% perf-profile.self.cycles-pp.ioctl_file_clone
0.79 ± 3% -0.1 0.70 ± 2% -0.1 0.67 ± 4% -0.0 0.78 ± 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
0.81 ± 3% -0.1 0.73 ± 4% -0.1 0.75 ± 2% +0.1 0.87 ± 2% perf-profile.self.cycles-pp.__put_user_4
0.90 ± 4% -0.1 0.82 ± 5% -0.2 0.75 ± 5% -0.1 0.85 ± 2% perf-profile.self.cycles-pp.vfs_fallocate
0.81 ± 5% -0.1 0.73 ± 3% -0.1 0.74 ± 3% +0.0 0.84 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
0.52 ± 4% -0.1 0.44 ± 3% -0.1 0.46 ± 3% -0.0 0.51 ± 6% perf-profile.self.cycles-pp.amd_clear_divider
0.17 ± 11% -0.1 0.12 ± 10% -0.1 0.11 ± 9% +0.0 0.21 ± 6% perf-profile.self.cycles-pp.stress_mwc8
0.57 ± 3% -0.0 0.52 ± 4% -0.0 0.52 ± 4% +0.0 0.58 ± 2% perf-profile.self.cycles-pp.__get_user_2
0.42 ± 4% -0.0 0.38 ± 3% -0.0 0.37 ± 6% +0.0 0.42 ± 6% perf-profile.self.cycles-pp.__libc_fcntl64
0.30 ± 3% -0.0 0.26 ± 5% +0.1 0.39 ± 5% +0.2 0.47 ± 5% perf-profile.self.cycles-pp.__x64_sys_fcntl
0.22 ± 5% -0.0 0.18 ± 6% -0.0 0.19 ± 6% -0.0 0.21 ± 4% perf-profile.self.cycles-pp.do_fcntl
0.28 ± 3% -0.0 0.24 ± 2% -0.3 0.00 -0.3 0.00 perf-profile.self.cycles-pp.__kmem_cache_alloc_node
0.27 ± 4% -0.0 0.24 ± 8% +6.1 6.32 +7.1 7.33 ± 2% perf-profile.self.cycles-pp.__fdget
0.14 ± 10% -0.0 0.12 ± 8% +0.3 0.49 ± 4% +0.4 0.59 ± 6% perf-profile.self.cycles-pp.__fdget_pos
0.19 ± 4% -0.0 0.17 ± 8% +0.0 0.20 ± 39% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.ioctl@plt
0.22 ± 6% -0.0 0.21 ± 4% +0.0 0.23 ± 6% +0.0 0.26 ± 5% perf-profile.self.cycles-pp.check_flag
0.07 ± 10% -0.0 0.06 ± 13% +0.2 0.28 ± 3% +0.3 0.32 ± 5% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.12 ± 7% -0.0 0.12 ± 16% -0.0 0.10 ± 7% -0.0 0.09 ± 12% perf-profile.self.cycles-pp.__virt_addr_valid
0.07 ± 13% -0.0 0.06 ± 11% +0.2 0.24 ± 3% +0.2 0.29 ± 5% perf-profile.self.cycles-pp.kfree
0.46 ± 5% -0.0 0.46 ± 3% -0.1 0.37 ± 6% -0.0 0.42 ± 6% perf-profile.self.cycles-pp.llseek
0.10 ± 13% -0.0 0.09 ± 9% -0.0 0.07 ± 12% -0.0 0.08 ± 13% perf-profile.self.cycles-pp.check_heap_object
0.00 +0.0 0.00 +0.4 0.42 ± 5% +0.5 0.50 ± 3% perf-profile.self.cycles-pp.__fdget_raw
0.00 +0.0 0.00 +0.5 0.46 ± 4% +0.0 0.00 perf-profile.self.cycles-pp.remap_verify_area
0.05 ± 45% +0.0 0.06 ± 8% +0.0 0.08 ± 9% +0.0 0.08 ± 13% perf-profile.self.cycles-pp.__errno_location
0.66 ± 2% +0.0 0.68 ± 3% -0.0 0.64 ± 3% -0.2 0.42 ± 8% perf-profile.self.cycles-pp.vfs_clone_file_range
0.00 +0.2 0.22 ± 4% +0.0 0.00 +0.0 0.00 perf-profile.self.cycles-pp.fsnotify_perm
0.49 ± 3% +0.4 0.92 ± 2% +0.4 0.94 +0.0 0.51 ± 3% perf-profile.self.cycles-pp.security_file_permission
0.46 ± 2% +0.5 0.96 ± 2% +0.5 0.94 ± 2% +0.0 0.47 ± 3% perf-profile.self.cycles-pp.aa_file_perm
0.00 +1.5 1.52 ± 2% +1.5 1.49 ± 2% +0.0 0.00 perf-profile.self.cycles-pp.__fsnotify_parent
7.75 ± 4% +6.8 14.58 ± 7% +7.3 15.02 ± 4% +0.3 8.03 ± 10% perf-profile.self.cycles-pp.apparmor_file_permission
>
> Thanks,
> Amir.
On Sun, Feb 4, 2024 at 8:33 AM Oliver Sang <[email protected]> wrote:
>
> hi, Amir,
>
> On Fri, Feb 02, 2024 at 11:13:56AM +0200, Amir Goldstein wrote:
> > On Wed, Jan 31, 2024 at 5:47 PM Amir Goldstein <[email protected]> wrote:
> > >
> > > On Wed, Jan 31, 2024 at 4:13 PM kenel test robot <[email protected]> wrote:
> > > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > kernel test robot noticed a -11.2% regression of stress-ng.file-ioctl.ops_per_sec on:
> > > >
> > > >
> > > > commit: dfad37051ade6ac0d404ef4913f3bd01954ee51c ("remap_range: move permission hooks out of do_clone_file_range()")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > >
> > >
> > > Can you please try this fix:
> > >
> > > 7d4213664bda remap_range: move sanity checks out of do_clone_file_range()
> > >
> > > from:
> > >
> > > https://github.com/amir73il/linux ovl-fixes
> > >
> >
> > Sorry, Oliver, this was a buggy commit.
> > I pushed this fixes version to ovl-fixes branch:
> >
> > 1c5e7db8e1b2 remap_range: merge do_clone_file_range() into
> > vfs_clone_file_range()
> >
> > Can you please test.
>
> the regression disappeared by above commit in our tests.
>
> I noticed this branch is based on v6.8-rc2, so I directly tested upon it and its
> parent (3f01e53bf6). I found 3f01e53bf6 has same data as dfad37051a we reported.
>
> and on 1c5e7db8e1b2, the performance back to the same level before dfad37051a.
>
Thanks for testing!
Christian, can you please amend the fix commit to
Reported-and-tested-by: kernel test robot <[email protected]>
Thanks,
Amir.
> Christian, can you please amend the fix commit to
> Reported-and-tested-by: kernel test robot <[email protected]>
Ok, done.