2021-04-22 14:26:32

by kernel test robot

[permalink] [raw]
Subject: [sched,debug] 3b87f136f8: stress-ng.procfs.ops_per_sec -31.7% regression



Greeting,

FYI, we noticed a -31.7% regression of stress-ng.procfs.ops_per_sec due to commit:


commit: 3b87f136f8fccddf7da016ab7d04bb3cf9b180f0 ("sched,debug: Convert sysctl sched_domains to debugfs")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core


in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

nr_threads: 10%
disk: 1HDD
testtime: 60s
fs: ext4
class: os
test: procfs
cpufreq_governor: performance
ucode: 0x5003006




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml
bin/lkp run compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
os/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp5/procfs/stress-ng/60s/0x5003006

commit:
9af0440ec8 ("debugfs: Implement debugfs_create_str()")
3b87f136f8 ("sched,debug: Convert sysctl sched_domains to debugfs")

9af0440ec86ebdab 3b87f136f8fccddf7da016ab7d0
---------------- ---------------------------
%stddev %change %stddev
\ | \
16865 ? 4% -31.7% 11516 ? 6% stress-ng.procfs.ops
280.96 ? 4% -31.7% 191.86 ? 6% stress-ng.procfs.ops_per_sec
138362 ? 3% -21.1% 109140 ? 4% stress-ng.time.minor_page_faults
22050 ? 20% -31.6% 15085 ? 10% stress-ng.time.voluntary_context_switches
369981 -8.0% 340222 proc-vmstat.pgfault
16660 ?132% -87.9% 2014 ?139% softirqs.NET_RX
2736 ? 4% -8.1% 2513 ? 2% vmstat.system.cs
10101 ? 3% -11.7% 8915 ? 3% slabinfo.kmalloc-256.active_objs
10159 ? 3% -11.6% 8977 ? 3% slabinfo.kmalloc-256.num_objs
0.32 ? 30% -0.2 0.11 ? 23% perf-profile.children.cycles-pp._raw_spin_lock
0.21 ? 31% -0.2 0.06 ? 48% perf-profile.children.cycles-pp.do_sys_open
0.21 ? 31% -0.2 0.06 ? 48% perf-profile.children.cycles-pp.do_sys_openat2
0.21 ? 34% -0.2 0.06 ? 47% perf-profile.children.cycles-pp.do_filp_open
0.21 ? 34% -0.2 0.06 ? 47% perf-profile.children.cycles-pp.path_openat
5483 -8.0% 5042 perf-stat.i.minor-faults
5495 -8.0% 5055 perf-stat.i.page-faults
2520 ? 5% -7.8% 2323 ? 2% perf-stat.ps.context-switches
5344 -8.1% 4911 perf-stat.ps.minor-faults
5355 -8.1% 4922 perf-stat.ps.page-faults
550269 ? 4% -20.6% 436852 ? 4% interrupts.CAL:Function_call_interrupts
12293 ? 98% -80.9% 2349 ? 44% interrupts.CPU5.CAL:Function_call_interrupts
14886 ?104% -85.1% 2221 ? 60% interrupts.CPU5.TLB:TLB_shootdowns
17907 ? 68% -85.2% 2647 ? 56% interrupts.CPU53.CAL:Function_call_interrupts
21317 ? 71% -88.1% 2532 ? 74% interrupts.CPU53.TLB:TLB_shootdowns
10259 ? 67% -59.9% 4115 ? 71% interrupts.CPU62.CAL:Function_call_interrupts
12377 ? 72% -64.3% 4420 ? 86% interrupts.CPU62.TLB:TLB_shootdowns
615020 ? 5% -22.8% 474888 ? 4% interrupts.TLB:TLB_shootdowns



stress-ng.procfs.ops_per_sec

360 +---------------------------------------------------------------------+
340 |-+ + |
| :: |
320 |-+ : : |
300 |-+ : : |
|.+.+.+.+.+.+ +..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+ +.+.+ |
280 |-+ + .. |
260 |-+ + |
240 |-+ |
| |
220 |-+ O O |
200 |-O O O O O O O O O O O O O O O O O O O O O O O O O |
| O O |
180 |-+ |
160 +---------------------------------------------------------------------+


[*] bisect-good sample
[O] bisect-bad sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Thanks,
Oliver Sang


Attachments:
(No filename) (5.92 kB)
config-5.12.0-rc2-00059-g3b87f136f8fc (175.55 kB)
job-script (8.40 kB)
job.yaml (5.63 kB)
reproduce (551.00 B)
Download all attachments

2021-04-22 16:25:12

by Peter Zijlstra

[permalink] [raw]
Subject: Re: [sched,debug] 3b87f136f8: stress-ng.procfs.ops_per_sec -31.7% regression

On Thu, Apr 22, 2021 at 10:42:58PM +0800, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed a -31.7% regression of stress-ng.procfs.ops_per_sec due to commit:

> commit: 3b87f136f8fccddf7da016ab7d04bb3cf9b180f0 ("sched,debug: Convert sysctl sched_domains to debugfs")

> on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory

> 16865 ? 4% -31.7% 11516 ? 6% stress-ng.procfs.ops

The patch in question removes a ton of procfs files, so this is just
about expected. If this also lowers the procfs ops/s measure, this could
be because all these procfs files were trivial.

Some procfs files are expensive to read and collect lots of data, these
files were trivial and fast, by removing them, the average time to read
a procfs file might very well have increased.

Over-all: -EDONTCARE.