LinuxLists.cc - [bpf] b89fbfbb85: netperf.Throughput

2021-08-20 05:34:07

Subject: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression

Greeting,

FYI, we noticed a

commit: b89fbfbb854c9afc30

If you fix the issue, Reported-by: kernel

Details are as below:
--------------------------

To reproduce:

git clone bin/lkp install bin/lkp split-job bin/lkp run
========================== cluster/compiler/cpufreq_g cs-localhost/gcc-9/perfo
commit:
652c1b17b8 ("bpf: b89fbfbb85 ("bpf:
652c1b17b85b9c19 ---------------- %stddev %change \ | 382126 23882 -21.3% 873.33 2593 -4.4% 71.41 ? 6% -28.3% 2.26e+08 1.146e+08 0.26 ? 3% -0.0 74155969 ? 4% 2329526 83788 ? 32% -40.1% 83788 ? 32% -40.1% 21155 ? 32% -40.0% 21155 ? 32% -40.0% 408960 ? 4% 408960 ? 4% 423396 ? 5% 163889 2566 -11.2% 164274 2566 -11.2% 1046207 1672962 102319 ? 4% -12.5% 106662 ? 5% -15.8% 1077486 1046207 102319 ? 4% -12.5% 4584675 4351098 513996 ? 6% 4595758 1896705 2847632 ? 2% 857.83 ? 9% +78.0% 857.83 ? 9% +78.0% 1744 ? 12% +40.8% 1527 ? 15% +32.8% 29210 ? 29% -37.1% 3329 ? 12% -24.2% 9987 ? 9% +96.8% 2163 ? 10% +29.7% 2448 ? 14% +24.7% 2229 ? 10% +25.0% 5662 ? 11% -34.8% 10902 ? 27% -32.8% 5596 ? 6% -23.0% 6154 ? 10% -22.7% 5253 ? 11% -38.4% 4555 ? 12% -29.9% 3191 ? 11% -31.7% 17323 ? 45% -51.6% 1950 ? 19% -42.3% 1950 ? 19% -42.3% 2412 ? 42% -56.3% 2412 ? 42% -56.3% 9938 ? 14% -40.5% 11392 ? 21% +48.0% 1587 ? 11% +26.5% 0.00 -29.2% 0.00 -25.0% 0.00 -33.3% 0.01 ? 13% +51.6% 0.00 -25.0% 0.01 ? 13% +51.6% 58.25 ? 7% +44.8% 0.01 +42.9% 98.49 ? 5% -10.8% 159.17 ? 7% 3326971 1767 -12.2% 6.75 ? 17% -25.1% 0.12 ?117% -84.7% 0.02 ? 4% +33.1% 0.01 ? 14% +337.8% 0.02 ? 7% -11.6% 0.02 ? 3% +27.6% 0.02 ? 5% +18.2% 0.00 ? 14% +75.0% 1.35 ? 49% -70.1% 58.25 ? 7% +44.8% 0.01 +60.0% 98.49 ? 5% -10.8% 0.30 ?176% -94.0% 0.03 ? 8% +1134.8% 0.03 ? 12% +699.5% 0.03 ? 10% +1738.9% 0.03 ? 13% +1386.6% 0.03 ? 12% +23.2% 0.01 ? 81% +1207.4% 0.03 ? 6% +31.8% 2.70 ? 28% -64.2% 11.77 ? 2% +31.6% 1.004e+10 1.559e+08 11.20 ? 3% +7.0 65819504 ? 3% 5.901e+08 2356585 1.55 +19.1% 7.708e+10 295.82 1235 ? 4% 1.52e+10 242730 ? 15% 8.874e+09 89.72 -4.5 1.073e+08 11647157 5.082e+10 486.39 0.66 -15.6% 0.40 -3.0% 214.40 180.75 13387607 ? 3% 1252745 ? 3% 10872702 550301 11.61 +29.0% 1.55 -0.0 11.15 ? 3% +7.1 1.52 +18.5% 1172 ? 3% 0.00 ? 14% -0.0 90.21 -4.7 473.55 0.66 -15.6% 134988 1.001e+10 1.554e+08 65603770 ? 3% 5.882e+08 2348732 7.683e+10 294.87 1.515e+10 241927 ? 15% 8.846e+09 1.07e+08 11608473 5.066e+10 13343927 ? 3% 1248713 ? 3% 10837421 549221 1.547e+13 4014884 ? 13% 258765 ? 12% 1176900 ? 14% 2200920 ? 9% 160791 ? 8% 2325401 ? 9% 162209 ? 7% 2439752 ? 15% 172191 ? 11% 1973419 ? 10% 145853 ? 7% 155145 ? 8% 2183325 ? 16% 151893 ? 11% 141163 ? 16% 2137865 ? 10% 2167225 ? 15% 151688 ? 11% 2304600 ? 7% 160072 ? 3% 2196758 ? 18% 155180 ? 11% 2069783 ? 14% 149326 ? 13% 1847061 ? 4% 134926 ? 5% 1951796 ? 13% 142225 ? 9% 1921345 ? 12% 2248959 ? 19% 160768 ? 13% 1862307 ? 12% 1357517 ? 18% 2393344 ? 10% 165076 ? 8% 2296035 ? 16% 160311 ? 11% 2232836 ? 19% 159350 ? 15% 2270802 ? 15% 158247 ? 12% 1974994 ? 8% 145079 ? 7% 2023137 ? 17% 140636 ? 11% 142098 ? 7% 3597896 ? 14% 234587 ? 13% 297635 ? 11% 3962994 ? 10% 264305 ? 10% 3147476 ? 13% 213457 ? 11% 2805154 ? 15% 2175695 ? 13% 1903308 ? 12% 145385 ? 9% 200826 ? 14% 1520428 ? 13% 1379814 ? 8% 110710 ? 6% -19.1% 1090621 ? 14% 91279 ? 10% -15.5% 1240050 ? 8% 100298 ? 5% -19.7% 1215230 ? 19% 98790 ? 12% -23.3% 1010316 ? 23% 88547 ? 14% -18.0% 1192624 ? 26% 5245236 ? 6% 306421 ? 4% 4267374 ? 8% 271765 ? 8% 3409429 ? 13% 225978 ? 12% 3060626 ? 11% 209935 ? 10% 2490473 ? 14% 174568 ? 12% 2038572 ? 14% 149102 ? 10% 1806889 ? 19% 135107 ? 12% -29.8% 1724482 ? 13% 130437 ? 10% -26.9% 1571110 ? 10% 119409 ? 8% -19.4% 1211612 ? 13% 98706 ? 7% -15.2% 1286365 ? 12% 102717 ? 9% -25.6% 1235222 ? 15% 105331 ? 7% -19.0% 3750481 ? 10% 241578 ? 9% 2995431 ? 11% 203087 ? 9% 2630704 ? 5% 182284 ? 5% 1676900 ? 13% 2085902 ? 8% 148876 ? 8% 2137018 ? 5% 150723 ? 4% 3.505e+08 25635284 239503 5.94 ? 5% -1.2 5.91 ? 5% -1.2 5.31 ? 5% -1.1 5.28 ? 5% -1.1 3.92 ? 6% -0.7 2.97 ? 6% -0.6 2.95 ? 6% -0.6 2.92 ? 6% -0.6 4.37 ? 4% -0.6 1.78 ? 4% -0.5 1.78 ? 4% -0.5 1.73 ? 4% -0.5 1.73 ? 4% -0.5 1.75 ? 4% -0.5 1.72 ? 4% -0.5 1.71 ? 4% -0.5 1.63 ? 4% -0.5 1.45 ? 4% -0.4 1.41 ? 4% -0.4 1.52 ? 7% -0.4 0.61 ? 5% -0.4 1.08 ? 4% -0.3 1.11 ? 6% -0.3 0.61 ? 6% -0.3 1.04 ? 6% -0.3 1.21 ? 7% -0.3 0.97 ? 6% -0.3 0.95 ? 6% -0.2 0.68 ? 6% -0.2 0.66 ? 6% -0.2 0.58 ? 4% -0.2 0.69 ? 5% -0.2 1.10 ? 5% -0.2 0.95 ? 5% -0.2 1.04 ? 4% -0.2 0.92 ? 6% -0.2 0.96 ? 5% -0.2 0.97 ? 5% -0.2 0.64 ? 6% -0.2 0.94 ? 5% -0.2 0.75 ? 6% -0.2 0.98 ? 6% -0.2 0.84 ? 5% -0.2 0.80 ? 5% -0.2 0.83 ? 5% -0.2 1.21 ? 4% -0.2 0.78 ? 6% -0.1 0.78 ? 5% -0.1 0.62 ? 3% -0.1 0.92 ? 5% -0.1 0.90 ? 5% -0.1 0.58 ? 4% +0.1 0.58 ? 6% +0.2 0.73 ? 6% +0.2 1.08 ? 5% +0.2 1.12 ? 5% +0.2 1.03 ? 5% +0.2 0.57 ? 4% +0.2 0.81 ? 5% +0.3 1.72 ? 5% +0.4 1.95 ? 5% +0.4 1.60 ? 5% +0.5 0.00 +0.6 0.00 +0.6 3.18 ? 4% +0.8 3.15 ? 4% +0.8 3.52 ? 4% +0.9 3.75 ? 4% +1.0 5.98 ? 4% +1.1 6.00 ? 4% +1.1 5.93 ? 4% +1.1 5.78 ? 4% +1.1 4.54 ? 4% +1.2 5.54 ? 4% +1.5 11.94 ? 4% +2.6 11.92 ? 4% +2.6 11.72 ? 4% +2.6 11.67 ? 4% +2.6 5.95 ? 5% -1.2 5.91 ? 5% -1.2 5.31 ? 5% -1.1 5.29 ? 5% -1.1 3.99 ? 5% -0.7 1.80 ? 6% -0.7 2.97 ? 6% -0.6 3.18 ? 4% -0.6 2.95 ? 6% -0.6 2.92 ? 6% -0.6 4.38 ? 4% -0.6 2.45 ? 4% -0.5 2.49 ? 4% -0.5 1.47 ? 6% -0.5 1.22 ? 6% -0.5 2.45 ? 4% -0.5 2.40 ? 4% -0.5 1.78 ? 4% -0.5 1.78 ? 4% -0.5 1.73 ? 4% -0.5 1.75 ? 4% -0.5 1.83 ? 5% -0.4 2.21 ? 5% -0.4 1.88 ? 4% -0.4 1.52 ? 7% -0.4 1.05 ? 6% -0.3 1.50 ? 6% -0.3 1.61 ? 6% -0.3 1.60 ? 6% -0.3 1.22 ? 5% -0.3 1.21 ? 6% -0.3 0.62 ? 5% -0.2 1.06 ? 5% -0.2 0.60 ? 6% -0.2 0.83 ? 4% -0.2 1.25 ? 6% -0.2 0.96 ? 5% -0.2 1.14 ? 6% -0.2 1.15 ? 6% -0.2 1.12 ? 6% -0.2 0.36 ? 7% -0.2 0.78 ? 5% -0.2 0.90 ? 6% -0.2 0.88 ? 5% -0.2 0.83 ? 5% -0.2 1.23 ? 4% -0.2 0.73 ? 6% -0.2 0.72 ? 6% -0.2 1.02 ? 5% -0.2 0.68 ? 4% -0.2 0.79 ? 6% -0.1 0.93 ? 5% -0.1 0.70 ? 5% -0.1 0.68 ? 5% -0.1 0.90 ? 5% -0.1 0.61 ? 6% -0.1 0.59 ? 5% -0.1 0.78 ? 5% -0.1 0.61 ? 6% -0.1 0.46 ? 5% -0.1 0.59 ? 5% -0.1 0.61 ? 6% -0.1 0.50 ? 6% -0.1 0.45 ? 3% -0.1 0.71 ? 5% -0.1 0.44 ? 7% -0.1 0.50 ? 6% -0.1 0.48 ? 6% -0.1 0.39 ? 7% -0.1 0.48 ? 4% -0.1 0.46 ? 7% -0.1 0.32 ? 7% -0.1 0.33 ? 6% -0.1 0.40 ? 5% -0.1 0.30 ? 4% -0.1 0.42 ? 5% -0.1 0.37 ? 5% -0.1 0.25 ? 6% -0.1 0.40 ? 5% -0.1 0.32 ? 8% -0.1 0.37 ? 5% -0.1 0.37 ? 5% -0.1 0.26 ? 5% -0.1 0.27 ? 5% -0.1 0.23 ? 4% -0.1 0.38 ? 5% -0.1 0.22 ? 6% -0.1 0.30 ? 8% -0.1 0.46 ? 4% -0.1 0.32 ? 6% -0.1 0.22 ? 8% -0.1 0.32 ? 5% -0.1 0.26 ? 7% -0.1 0.18 ? 4% -0.1 0.14 ? 9% -0.1 0.28 ? 6% -0.1 0.25 ? 7% -0.1 0.20 ? 7% -0.0 0.32 ? 3% -0.0 0.22 ? 3% -0.0 0.25 ? 6% -0.0 0.24 ? 5% -0.0 0.12 ? 10% -0.0 0.29 ? 5% -0.0 0.19 ? 5% -0.0 0.14 ? 7% -0.0 0.19 ? 5% -0.0 0.17 ? 7% -0.0 0.09 ? 8% -0.0 0.15 ? 5% -0.0 0.07 ? 10% -0.0 0.12 ? 4% -0.0 0.15 ? 9% -0.0 0.18 ? 5% -0.0 0.20 ? 6% -0.0 0.18 ? 3% -0.0 0.11 ? 6% -0.0 0.15 ? 6% -0.0 0.11 ? 8% -0.0 0.08 ? 11% -0.0 0.15 ? 2% -0.0 0.16 ? 7% -0.0 0.09 ? 10% -0.0 0.09 ? 12% -0.0 0.17 ? 4% -0.0 0.12 ? 6% -0.0 0.14 ? 5% -0.0 0.09 ? 10% -0.0 0.14 ? 7% -0.0 0.14 ? 3% -0.0 0.14 ? 7% -0.0 0.08 ? 8% -0.0 0.16 ? 4% -0.0 0.11 ? 8% -0.0 0.07 ? 7% -0.0 0.11 ? 6% -0.0 0.10 ? 9% -0.0 0.09 ? 5% -0.0 0.08 ? 8% -0.0 0.08 ? 8% -0.0 0.07 ? 10% -0.0 0.08 ? 4% -0.0 0.06 ? 7% -0.0 0.08 ? 6% -0.0 0.17 ? 5% +0.0 0.10 ? 8% +0.0 0.06 ? 11% +0.0 0.14 ? 5% +0.0 0.23 ? 8% +0.1 0.06 ? 7% +0.1 0.00 +0.1 0.06 ? 9% +0.1 0.03 ?105% +0.1 0.08 ? 20% +0.1 0.13 ? 6% +0.1 0.06 ? 7% +0.1 0.07 ? 5% +0.1 0.00 +0.1 0.00 +0.1 0.42 ? 7% +0.1 0.07 ? 12% +0.1 0.07 ? 10% +0.1 0.02 ? 99% +0.1 0.08 ? 7% +0.1 0.11 ? 12% +0.1 0.00 +0.1 0.14 ? 7% +0.1 0.14 ? 4% +0.1 0.36 ? 4% +0.1 0.17 ? 7% +0.1 0.24 ? 6% +0.2 0.09 ? 8% +0.2 0.00 +0.2 0.34 ? 6% +0.2 0.22 ? 5% +0.2 0.33 ? 6% +0.2 0.59 ? 6% +0.2 0.14 ? 7% +0.2 0.11 ? 8% +0.2 0.07 ? 8% +0.2 0.20 ? 4% +0.2 0.18 ? 6% +0.2 0.04 ? 71% +0.2 0.64 ? 4% +0.2 0.11 ? 7% +0.2 0.11 ? 6% +0.3 0.10 ? 6% +0.3 0.21 ? 7% +0.3 0.14 ? 5% +0.3 1.73 ? 5% +0.3 0.17 ? 5% +0.4 1.95 ? 5% +0.4 1.61 ? 5% +0.5 0.44 ? 6% +0.6 2.07 ? 5% +0.7 0.27 ? 3% +0.8 0.62 ? 6% +0.8 0.58 ? 5% +0.8 0.26 ? 4% +1.0 5.78 ? 4% +1.1 4.54 ? 4% +1.2 5.54 ? 4% +1.5 11.94 ? 4% +2.6 11.92 ? 4% +2.6 11.72 ? 4% +2.6 11.67 ? 4% +2.6 3.85 ? 5% -0.7 1.21 ? 6% -0.5 1.87 ? 4% -0.4 1.70 ? 5% -0.3 1.04 ? 6% -0.3 1.14 ? 6% -0.3 0.62 ? 4% -0.2 0.50 ? 9% -0.2 1.04 ? 7% -0.2 0.82 ? 4% -0.2 0.87 ? 6% -0.2 0.36 ? 6% -0.1 0.59 ? 5% -0.1 0.41 ? 7% -0.1 0.52 ? 9% -0.1 0.60 ? 6% -0.1 0.69 ? 5% -0.1 0.34 ? 7% -0.1 0.40 ? 5% -0.1 0.39 ? 5% -0.1 0.36 ? 4% -0.1 0.40 ? 5% -0.1 0.29 ? 4% -0.1 0.28 ? 5% -0.1 0.45 ? 4% -0.1 0.21 ? 7% -0.1 0.32 ? 5% -0.1 0.26 ? 8% -0.1 0.17 ? 7% -0.1 0.18 ? 5% -0.1 0.12 ? 8% -0.1 0.24 ? 6% -0.1 0.19 ? 6% -0.0 0.18 ? 5% -0.0 0.34 ? 5% -0.0 0.07 ? 8% -0.0 0.12 ? 10% -0.0 0.12 ? 4% -0.0 0.23 ? 4% -0.0 0.19 ? 5% -0.0 0.16 ? 4% -0.0 0.15 ? 9% -0.0 0.22 ? 4% -0.0 0.08 ? 10% -0.0 0.07 ? 10% -0.0 0.15 ? 6% -0.0 0.21 ? 5% -0.0 0.09 ? 5% -0.0 0.20 ? 6% -0.0 0.16 ? 5% -0.0 0.12 ? 6% -0.0 0.10 ? 6% -0.0 0.14 ? 8% -0.0 0.14 ? 4% -0.0 0.08 ? 6% -0.0 0.08 ? 11% -0.0 0.12 ? 6% -0.0 0.10 ? 4% -0.0 0.14 ? 10% -0.0 0.12 ? 6% -0.0 0.11 ? 8% -0.0 0.16 ? 4% -0.0 0.09 ? 4% -0.0 0.07 ? 6% -0.0 0.08 ? 10% -0.0 0.14 ? 5% -0.0 0.08 ? 8% -0.0 0.11 ? 6% -0.0 0.09 ? 4% -0.0 0.06 ? 7% -0.0 0.08 ? 4% -0.0 0.08 ? 7% -0.0 0.10 ? 4% -0.0 0.07 ? 10% -0.0 0.08 ? 9% -0.0 0.06 ? 13% +0.0 0.06 ? 11% +0.0 0.09 ? 9% +0.0 0.06 ? 13% +0.0 0.07 ? 7% +0.0 0.00 +0.1 0.11 ? 9% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.08 ? 12% +0.1 0.00 +0.1 0.02 ?146% +0.1 0.12 ? 6% +0.1 0.07 ? 7% +0.1 0.06 ? 9% +0.1 0.00 +0.1 0.06 ? 11% +0.1 0.07 +0.1 0.00 +0.1 0.04 ? 45% +0.1 0.05 +0.1 0.07 ? 9% +0.1 0.00 +0.1 0.04 ? 45% +0.1 0.02 ? 99% +0.1 0.06 ? 13% +0.1 0.09 ? 16% +0.1 0.05 ? 49% +0.1 0.00 +0.1 0.06 ? 7% +0.1 0.12 ? 8% +0.1 0.13 ? 7% +0.1 0.14 ? 4% +0.1 0.01 ?223% +0.1 0.45 ? 7% +0.1 0.00 +0.1 0.47 ? 4% +0.2 0.06 ? 45% +0.2 0.08 ? 8% +0.2 0.10 ? 7% +0.2 0.16 ? 4% +0.2 0.07 ? 8% +0.2 0.02 ?141% +0.2 0.03 ?100% +0.2 0.10 ? 5% +0.3 0.28 ? 6% +0.3 0.14 ? 7% +0.3 0.12 ? 7% +0.4 0.17 ? 5% +0.4 0.77 ? 4% +0.4 0.09 ? 4% +0.5 0.26 ? 7% +0.6 0.25 ? 5% +0.6 0.25 ? 3% +1.0

26000 +----------------- | 25000 |-+ 24000 |+.++.++.++.++.+ | 23000 |-+ | 22000 |-+ | 21000 |-+ 20000 |-+ |O OO OO OO OO O 19000 |-+ | 18000 +----------------- 420000 +---------------- | 400000 |-+ |+.++.++.++.++.+ 380000 |-+ | 360000 |-+ | 340000 |-+ | 320000 |O+O OO O O | O OO O 300000 |-+ | 280000 +---------------- 1.25e+08 +-------------- | 1.2e+08 |-+ + 1.15e+08 |+.++.+ +.+ ++ | 1.1e+08 |-+ | 1.05e+08 |-+ | 1e+08 |-+ 9.5e+07 |-+ |O OO OOO OO OO 9e+07 |-+ | 8.5e+07 +-------------- 85 +-------------------- | 80 |-+ | 75 |-+ | 70 |.++.++.+ +. .+. +.+ | +.+ ++ + 65 |-+ | 60 |-+ | 55 |-+O O O OO | O OO O O O OO OO 50 +-------------------- 2620 +------------------ | +.++. + 2600 |-+ + + +.+ + +. : 2580 |-+ + + + | 2560 |-+ | 2540 |-+ | 2520 |-+O O O OO 2500 |-O OO O O O O OO | O O 2480 |-+ | 2460 +------------------ 890 +------------------- | 880 |.++.++. | ++. +.+ .++. : 870 |-+ + + + | 860 |-+ | 850 |-+ | 840 |-OO OO OO OO OO O OO | OO O 830 |-+ | 820 +------------------- 2.4e+08 +--------------- | 2.3e+08 |+.++.++.+++.++ | 2.2e+08 |-+ | 2.1e+08 |-+ | 2e+08 |-+ | 1.9e+08 |-+ |O OO OO OOO OO 1.8e+08 |-+ | 1.7e+08 +--------------- 3.6e+06 +--------------- 3.5e+06 |-+ | 3.4e+06 |+.+ +.+++.++.++.+ 3.3e+06 |-+ +.+ | 3.2e+06 |-+ 3.1e+06 |-+ 3e+06 |-+ | 2.9e+06 |-+ 2.8e+06 |O+OO OO OOO OO | 2.7e+06 |-+ 2.6e+06 +--------------- [*] bisect-good sample
[O] bisect-bad sample

Disclaimer:
Results have been for informational design or configuration

---
0DAY/LKP+ Test Infrastructure https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
Netperf is a benchmark that can be use to measure various aspect of networking performance.
/www.netperf.org/netperf/">http://www.netperf.org/netperf/
kindly add following tag
test robot <[email protected]>
------------------------------------------------------------------------>
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
job.yaml # job file is attached in this email
--compatible job.yaml # generate the yaml file for lkp run
generated-yaml-file
===============================================================
overnor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
rmance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003006
Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
Implement minimal BPF perf link")
b89fbfbb854c9afc3047e8273cc
---------------------------
%stddev
\
-21.3% 300769 netperf.Throughput_total_tps
18798 netperf.Throughput_tps
-5.1% 828.83 netperf.time.percent_of_cpu_this_job_got
2478 netperf.time.system_time
51.23 netperf.time.user_time
-22.8% 1.744e+08 netperf.time.voluntary_context_switches
-21.3% 90230742 netperf.workload
0.21 mpstat.cpu.all.usr%
+31.2% 97319249 ? 7% turbostat.C1
-8.6% 2130170 vmstat.system.cs
50201 ? 33% numa-meminfo.node3.Inactive
50201 ? 33% numa-meminfo.node3.Inactive(anon)
12694 ? 32% numa-vmstat.node3.nr_inactive_anon
12694 ? 32% numa-vmstat.node3.nr_zone_inactive_anon
-12.5% 357892 meminfo.Inactive
-12.5% 357892 meminfo.Inactive(anon)
-15.8% 356441 ? 4% meminfo.Mapped
-11.1% 145655 slabinfo.filp.active_objs
2278 slabinfo.filp.active_slabs
-11.2% 145861 slabinfo.filp.num_objs
2278 slabinfo.filp.num_slabs
-6.0% 983836 proc-vmstat.nr_active_anon
-4.5% 1598068 proc-vmstat.nr_file_pages
89519 proc-vmstat.nr_inactive_anon
89786 ? 4% proc-vmstat.nr_mapped
-7.0% 1002582 proc-vmstat.nr_shmem
-6.0% 983836 proc-vmstat.nr_zone_active_anon
89519 proc-vmstat.nr_zone_inactive_anon
-13.1% 3982852 proc-vmstat.numa_hit
-14.4% 3722478 proc-vmstat.numa_local
-29.2% 363996 ? 3% proc-vmstat.pgactivate
-13.1% 3993935 proc-vmstat.pgalloc_normal
-21.7% 1485852 proc-vmstat.pgfree
-9.6% 2572843 ? 9% interrupts.CAL:Function_call_interrupts
1526 ? 15% interrupts.CPU115.NMI:Non-maskable_interrupts
1526 ? 15% interrupts.CPU115.PMI:Performance_monitoring_interrupts
2456 ? 11% interrupts.CPU118.RES:Rescheduling_interrupts
2028 ? 16% interrupts.CPU119.RES:Rescheduling_interrupts
18383 ? 28% interrupts.CPU120.CAL:Function_call_interrupts
2523 ? 18% interrupts.CPU170.RES:Rescheduling_interrupts
19656 ? 15% interrupts.CPU176.CAL:Function_call_interrupts
2804 ? 11% interrupts.CPU176.RES:Rescheduling_interrupts
3054 ? 11% interrupts.CPU187.RES:Rescheduling_interrupts
2787 ? 11% interrupts.CPU188.RES:Rescheduling_interrupts
3693 ? 16% interrupts.CPU24.RES:Rescheduling_interrupts
7331 ? 35% interrupts.CPU37.CAL:Function_call_interrupts
4306 ? 17% interrupts.CPU48.RES:Rescheduling_interrupts
4760 ? 13% interrupts.CPU49.RES:Rescheduling_interrupts
3233 ? 21% interrupts.CPU50.RES:Rescheduling_interrupts
3193 ? 19% interrupts.CPU51.RES:Rescheduling_interrupts
2179 ? 19% interrupts.CPU53.RES:Rescheduling_interrupts
8384 ? 31% interrupts.CPU57.CAL:Function_call_interrupts
1126 ? 36% interrupts.CPU57.NMI:Non-maskable_interrupts
1126 ? 36% interrupts.CPU57.PMI:Performance_monitoring_interrupts
1055 ? 55% interrupts.CPU59.NMI:Non-maskable_interrupts
1055 ? 55% interrupts.CPU59.PMI:Performance_monitoring_interrupts
5909 ? 28% interrupts.CPU7.CAL:Function_call_interrupts
16859 ? 14% interrupts.CPU77.CAL:Function_call_interrupts
2007 ? 12% interrupts.CPU91.RES:Rescheduling_interrupts
0.00 ? 13% perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.tcp_close
0.00 perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_accept.__sys_accept4_file
0.00 perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
0.01 ? 25% perf-sched.sch_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__unix_dgram_recvmsg.__sys_recvfrom
0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
0.01 ? 25% perf-sched.sch_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__unix_dgram_recvmsg.__sys_recvfrom
84.33 ? 3% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.01 perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
87.89 ? 2% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
-31.6% 108.83 ? 3% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
-19.2% 2688283 perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
1551 ? 2% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
5.06 ? 7% perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
0.02 ?156% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.03 ? 16% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.inet_accept
0.03 ? 34% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.sk_wait_data
0.02 ? 4% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.tcp_sendmsg
0.03 ? 3% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.lock_sock_nested.inet_accept.__sys_accept4_file
0.03 ? 6% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.lock_sock_nested.inet_csk_accept.inet_accept
0.01 ? 55% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.lock_sock_nested.sk_wait_data.tcp_recvmsg_locked
0.40 ? 93% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_write_begin
84.33 ? 3% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
0.01 perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
87.89 ? 2% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
0.02 ?156% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.38 ?200% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
0.27 ?188% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.inet_accept
0.57 ?145% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.sk_wait_data
0.46 ?201% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.sock_alloc_inode.alloc_inode
0.04 ? 5% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.lock_sock_nested.inet_csk_accept.inet_accept
0.18 ?176% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.lock_sock_nested.sk_wait_data.tcp_recvmsg_locked
0.04 ? 6% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.lock_sock_nested.tcp_sendmsg.sock_sendmsg
0.97 ? 76% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_write_begin
15.48 ? 3% perf-stat.i.MPKI
-17.9% 8.241e+09 perf-stat.i.branch-instructions
-19.3% 1.258e+08 perf-stat.i.branch-misses
18.19 perf-stat.i.cache-miss-rate%
+72.8% 1.138e+08 perf-stat.i.cache-misses
+5.5% 6.228e+08 perf-stat.i.cache-references
-8.5% 2155457 perf-stat.i.context-switches
1.84 perf-stat.i.cpi
-3.0% 7.474e+10 perf-stat.i.cpu-cycles
-3.6% 285.22 perf-stat.i.cpu-migrations
-44.4% 687.56 ? 3% perf-stat.i.cycles-between-cache-misses
-18.5% 1.239e+10 perf-stat.i.dTLB-loads
-48.0% 126135 ? 20% perf-stat.i.dTLB-store-misses
-18.4% 7.24e+09 perf-stat.i.dTLB-stores
85.18 perf-stat.i.iTLB-load-miss-rate%
-21.8% 83946753 perf-stat.i.iTLB-load-misses
+22.3% 14248926 ? 2% perf-stat.i.iTLB-loads
-18.2% 4.158e+10 perf-stat.i.instructions
+4.4% 507.91 perf-stat.i.instructions-per-iTLB-miss
0.55 perf-stat.i.ipc
0.39 perf-stat.i.metric.GHz
+21.3% 260.08 perf-stat.i.metric.K/sec
-17.9% 148.39 perf-stat.i.metric.M/sec
+57.6% 21095835 perf-stat.i.node-load-misses
+66.1% 2081130 perf-stat.i.node-loads
-21.9% 8486959 perf-stat.i.node-store-misses
-28.0% 396340 ? 2% perf-stat.i.node-stores
14.98 perf-stat.overall.MPKI
1.53 perf-stat.overall.branch-miss-rate%
18.27 perf-stat.overall.cache-miss-rate%
1.80 perf-stat.overall.cpi
-44.0% 657.12 perf-stat.overall.cycles-between-cache-misses
0.00 ? 20% perf-stat.overall.dTLB-store-miss-rate%
85.49 perf-stat.overall.iTLB-load-miss-rate%
+4.6% 495.37 perf-stat.overall.instructions-per-iTLB-miss
0.56 perf-stat.overall.ipc
+3.9% 140279 perf-stat.overall.path-length
-17.9% 8.214e+09 perf-stat.ps.branch-instructions
-19.3% 1.254e+08 perf-stat.ps.branch-misses
+72.8% 1.134e+08 perf-stat.ps.cache-misses
+5.5% 6.207e+08 perf-stat.ps.cache-references
-8.5% 2148196 perf-stat.ps.context-switches
-3.0% 7.449e+10 perf-stat.ps.cpu-cycles
-3.6% 284.33 perf-stat.ps.cpu-migrations
-18.5% 1.235e+10 perf-stat.ps.dTLB-loads
-48.0% 125735 ? 20% perf-stat.ps.dTLB-store-misses
-18.4% 7.217e+09 perf-stat.ps.dTLB-stores
-21.8% 83668098 perf-stat.ps.iTLB-load-misses
+22.3% 14200885 ? 2% perf-stat.ps.iTLB-loads
-18.2% 4.145e+10 perf-stat.ps.instructions
+57.6% 21026557 perf-stat.ps.node-load-misses
+66.1% 2074254 perf-stat.ps.node-loads
-21.9% 8459398 perf-stat.ps.node-store-misses
-28.0% 395635 ? 2% perf-stat.ps.node-stores
-18.2% 1.266e+13 perf-stat.total.instructions
-27.1% 2926304 ? 12% softirqs.CPU1.NET_RX
-24.7% 194744 ? 10% softirqs.CPU1.RCU
-22.9% 907366 ? 18% softirqs.CPU10.NET_RX
-27.9% 1587061 ? 16% softirqs.CPU121.NET_RX
-19.3% 129762 ? 11% softirqs.CPU121.RCU
-36.6% 1475333 ? 14% softirqs.CPU122.NET_RX
-26.0% 120056 ? 10% softirqs.CPU122.RCU
-39.7% 1470207 ? 13% softirqs.CPU123.NET_RX
-31.2% 118529 ? 8% softirqs.CPU123.RCU
-15.6% 1666511 ? 10% softirqs.CPU126.NET_RX
-11.8% 128598 ? 6% softirqs.CPU126.RCU
-18.8% 125917 ? 12% softirqs.CPU127.RCU
-32.6% 1472632 ? 13% softirqs.CPU128.NET_RX
-24.8% 114269 ? 11% softirqs.CPU128.RCU
-19.9% 113100 ? 11% softirqs.CPU133.RCU
-21.8% 1672211 ? 15% softirqs.CPU137.NET_RX
-29.4% 1531034 ? 12% softirqs.CPU139.NET_RX
-22.5% 117531 ? 8% softirqs.CPU139.RCU
-28.0% 1660132 ? 24% softirqs.CPU145.NET_RX
-19.5% 128924 ? 18% softirqs.CPU145.RCU
-34.1% 1447425 ? 14% softirqs.CPU149.NET_RX
-26.6% 113886 ? 8% softirqs.CPU149.RCU
-32.2% 1402561 ? 14% softirqs.CPU150.NET_RX
-26.7% 109469 ? 11% softirqs.CPU150.RCU
-21.0% 1459388 ? 9% softirqs.CPU151.NET_RX
-16.1% 113270 ? 7% softirqs.CPU151.RCU
-29.6% 1373752 ? 20% softirqs.CPU156.NET_RX
-23.2% 109249 ? 14% softirqs.CPU156.RCU
-22.2% 1495303 ? 10% softirqs.CPU159.NET_RX
-30.2% 1569394 ? 16% softirqs.CPU161.NET_RX
-22.5% 124600 ? 12% softirqs.CPU161.RCU
-22.1% 1449974 ? 15% softirqs.CPU165.NET_RX
-31.9% 923915 ? 26% softirqs.CPU167.NET_RX
-40.6% 1421882 ? 22% softirqs.CPU169.NET_RX
-29.3% 116649 ? 17% softirqs.CPU169.RCU
-35.8% 1474994 ? 12% softirqs.CPU170.NET_RX
-29.1% 113713 ? 8% softirqs.CPU170.RCU
-28.7% 1592693 ? 18% softirqs.CPU171.NET_RX
-23.6% 121777 ? 14% softirqs.CPU171.RCU
-27.7% 1642884 ? 14% softirqs.CPU172.NET_RX
-22.4% 122791 ? 10% softirqs.CPU172.RCU
-17.4% 1631109 ? 10% softirqs.CPU173.NET_RX
-13.5% 125486 ? 4% softirqs.CPU177.RCU
-24.1% 1535725 ? 7% softirqs.CPU179.NET_RX
-16.7% 117143 ? 5% softirqs.CPU179.RCU
-17.0% 117946 ? 7% softirqs.CPU180.RCU
-34.5% 2358266 ? 19% softirqs.CPU2.NET_RX
-29.1% 166321 ? 14% softirqs.CPU2.RCU
-27.7% 215326 ? 13% softirqs.CPU24.RCU
-33.7% 2628479 ? 15% softirqs.CPU25.NET_RX
-29.8% 185617 ? 12% softirqs.CPU25.RCU
-30.7% 2180384 ? 23% softirqs.CPU26.NET_RX
-26.9% 155950 ? 17% softirqs.CPU26.RCU
-35.9% 1798759 ? 23% softirqs.CPU27.NET_RX
-34.2% 1430966 ? 19% softirqs.CPU28.NET_RX
-37.4% 1192171 ? 23% softirqs.CPU29.NET_RX
-28.3% 104230 ? 15% softirqs.CPU29.RCU
-28.9% 142832 ? 11% softirqs.CPU3.RCU
-31.1% 1048096 ? 18% softirqs.CPU30.NET_RX
-30.9% 952779 ? 15% softirqs.CPU31.NET_RX
89561 ? 9% softirqs.CPU31.RCU
-24.6% 821788 ? 20% softirqs.CPU34.NET_RX
77091 ? 11% softirqs.CPU34.RCU
-30.7% 859760 ? 18% softirqs.CPU38.NET_RX
80576 ? 10% softirqs.CPU38.RCU
-37.2% 763371 ? 24% softirqs.CPU39.NET_RX
75800 ? 12% softirqs.CPU39.RCU
-29.0% 716849 ? 12% softirqs.CPU43.NET_RX
72601 ? 6% softirqs.CPU43.RCU
-30.5% 829456 ? 24% softirqs.CPU44.NET_RX
-28.4% 3754659 ? 15% softirqs.CPU48.NET_RX
-26.8% 224181 ? 10% softirqs.CPU48.RCU
-38.2% 2638707 ? 9% softirqs.CPU49.NET_RX
-33.0% 181951 ? 8% softirqs.CPU49.RCU
-41.4% 1997125 ? 22% softirqs.CPU50.NET_RX
-37.1% 142118 ? 18% softirqs.CPU50.RCU
-44.7% 1691740 ? 13% softirqs.CPU51.NET_RX
-38.0% 130118 ? 10% softirqs.CPU51.RCU
-44.3% 1386146 ? 17% softirqs.CPU52.NET_RX
-36.1% 111632 ? 11% softirqs.CPU52.RCU
-39.3% 1237845 ? 15% softirqs.CPU53.NET_RX
-30.6% 103549 ? 9% softirqs.CPU53.RCU
-39.7% 1089972 ? 12% softirqs.CPU54.NET_RX
94798 ? 8% softirqs.CPU54.RCU
-35.0% 1120764 ? 17% softirqs.CPU55.NET_RX
95317 ? 10% softirqs.CPU55.RCU
-24.8% 1181396 ? 15% softirqs.CPU56.NET_RX
96251 ? 9% softirqs.CPU56.RCU
-26.1% 894990 ? 10% softirqs.CPU62.NET_RX
83741 ? 5% softirqs.CPU62.RCU
-38.1% 796328 ? 10% softirqs.CPU63.NET_RX
76430 ? 6% softirqs.CPU63.RCU
-28.2% 886801 ? 17% softirqs.CPU64.NET_RX
85343 ? 8% softirqs.CPU64.RCU
-20.5% 2981716 ? 10% softirqs.CPU73.NET_RX
-18.3% 197334 ? 8% softirqs.CPU73.RCU
-17.0% 2485879 ? 6% softirqs.CPU74.NET_RX
-15.3% 172016 ? 6% softirqs.CPU74.RCU
-22.1% 2049366 ? 10% softirqs.CPU75.NET_RX
-18.3% 148875 ? 7% softirqs.CPU75.RCU
-22.3% 1302832 ? 12% softirqs.CPU78.NET_RX
-33.4% 1389393 ? 24% softirqs.CPU97.NET_RX
-22.3% 115711 ? 20% softirqs.CPU97.RCU
-31.5% 1464045 ? 12% softirqs.CPU98.NET_RX
-23.5% 115253 ? 8% softirqs.CPU98.RCU
-21.7% 2.744e+08 softirqs.NET_RX
-16.0% 21527846 softirqs.RCU
+83.7% 440010 softirqs.TIMER
4.76 ? 6% perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.73 ? 6% perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.22 ? 6% perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.20 ? 6% perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
3.18 ? 6% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
2.32 ? 5% perf-profile.calltrace.cycles-pp.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.31 ? 5% perf-profile.calltrace.cycles-pp.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.29 ? 5% perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.78 ? 6% perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
1.30 ? 9% perf-profile.calltrace.cycles-pp.ret_from_fork
1.30 ? 9% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
1.26 ? 9% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.26 ? 10% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.28 ? 9% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
1.24 ? 9% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
1.24 ? 10% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
1.18 ? 10% perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
1.04 ? 9% perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
1.01 ? 10% perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core
1.15 ? 4% perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
0.26 ?100% perf-profile.calltrace.cycles-pp.apparmor_file_free_security.security_file_free.__fput.task_work_run.exit_to_user_mode_prepare
0.79 ? 11% perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch
0.83 ? 7% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
0.34 ? 70% perf-profile.calltrace.cycles-pp.security_file_free.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
0.78 ? 7% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
0.95 ? 3% perf-profile.calltrace.cycles-pp.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept
0.72 ? 7% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
0.70 ? 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
0.44 ? 44% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
0.43 ? 44% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.inet_csk_accept.inet_accept
0.36 ? 70% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.48 ? 44% perf-profile.calltrace.cycles-pp.__dentry_kill.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
0.89 ? 4% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.74 ? 8% perf-profile.calltrace.cycles-pp.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv
0.84 ? 4% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu
0.72 ? 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established
0.77 ? 4% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process
0.78 ? 4% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv
0.45 ? 44% perf-profile.calltrace.cycles-pp.inet_csk_destroy_sock.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv
0.76 ? 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
0.57 ? 3% perf-profile.calltrace.cycles-pp.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4
0.80 ? 8% perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
0.67 ? 8% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue
0.63 ? 8% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_wakeup.tcp_fin
0.67 ? 5% perf-profile.calltrace.cycles-pp.tcp_send_fin.__tcp_close.tcp_close.inet_release.__sock_release
1.06 ? 9% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
0.63 ? 7% perf-profile.calltrace.cycles-pp.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
0.63 ? 7% perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept
0.48 ? 44% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.79 ? 5% perf-profile.calltrace.cycles-pp.inet_csk_clone_lock.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv
0.77 ? 5% perf-profile.calltrace.cycles-pp.sk_clone_lock.inet_csk_clone_lock.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req
0.72 ? 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
0.78 ? 5% perf-profile.calltrace.cycles-pp.tcp_v4_send_synack.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
0.94 ? 6% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
1.30 ? 6% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
1.35 ? 6% perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
1.26 ? 6% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
0.81 ? 6% perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.12 ? 5% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
2.07 ? 5% perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
2.35 ? 6% perf-profile.calltrace.cycles-pp.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
2.14 ? 6% perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
0.57 ? 4% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
0.62 ? 6% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
3.96 ? 6% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
3.96 ? 6% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
4.44 ? 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
4.75 ? 5% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
7.08 ? 5% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
7.11 ? 6% perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
7.04 ? 6% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
6.91 ? 6% perf-profile.calltrace.cycles-pp.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
5.74 ? 5% perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
7.04 ? 5% perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
14.51 ? 5% perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
14.50 ? 5% perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
14.31 ? 5% perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
14.27 ? 5% perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
4.76 ? 6% perf-profile.children.cycles-pp.__x64_sys_recvfrom
4.74 ? 6% perf-profile.children.cycles-pp.__sys_recvfrom
4.22 ? 6% perf-profile.children.cycles-pp.inet_recvmsg
4.21 ? 6% perf-profile.children.cycles-pp.tcp_recvmsg
3.25 ? 5% perf-profile.children.cycles-pp.poll_idle
1.14 ? 6% perf-profile.children.cycles-pp._raw_spin_lock_bh
2.32 ? 5% perf-profile.children.cycles-pp.__x64_sys_accept
2.54 ? 8% perf-profile.children.cycles-pp.kmem_cache_free
2.31 ? 5% perf-profile.children.cycles-pp.__sys_accept4
2.29 ? 5% perf-profile.children.cycles-pp.__sys_accept4_file
3.79 ? 6% perf-profile.children.cycles-pp.tcp_recvmsg_locked
1.92 ? 9% perf-profile.children.cycles-pp.rcu_do_batch
1.96 ? 9% perf-profile.children.cycles-pp.rcu_core
0.96 ? 5% perf-profile.children.cycles-pp.lock_sock_nested
0.71 ? 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.95 ? 9% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
1.91 ? 9% perf-profile.children.cycles-pp.page_counter_uncharge
1.30 ? 9% perf-profile.children.cycles-pp.ret_from_fork
1.30 ? 9% perf-profile.children.cycles-pp.kthread
1.26 ? 10% perf-profile.children.cycles-pp.run_ksoftirqd
1.28 ? 9% perf-profile.children.cycles-pp.smpboot_thread_fn
1.43 ? 6% perf-profile.children.cycles-pp.__dev_queue_xmit
1.82 ? 6% perf-profile.children.cycles-pp._raw_spin_lock
1.50 ? 9% perf-profile.children.cycles-pp.page_counter_cancel
1.15 ? 4% perf-profile.children.cycles-pp.inet_accept
0.76 ? 5% perf-profile.children.cycles-pp.percpu_counter_add_batch
1.21 ? 7% perf-profile.children.cycles-pp.kmem_cache_alloc
1.33 ? 6% perf-profile.children.cycles-pp.sock_alloc_file
1.32 ? 6% perf-profile.children.cycles-pp.alloc_file_pseudo
0.95 ? 6% perf-profile.children.cycles-pp.__alloc_skb
0.95 ? 3% perf-profile.children.cycles-pp.inet_csk_accept
0.38 ? 10% perf-profile.children.cycles-pp.__tcp_get_metrics
0.83 ? 8% perf-profile.children.cycles-pp.sock_def_wakeup
0.38 ? 4% perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
0.63 ? 4% perf-profile.children.cycles-pp.sk_forced_mem_schedule
1.05 ? 5% perf-profile.children.cycles-pp.inet_csk_destroy_sock
0.77 ? 4% perf-profile.children.cycles-pp.autoremove_wake_function
0.94 ? 6% perf-profile.children.cycles-pp.alloc_empty_file
0.96 ? 6% perf-profile.children.cycles-pp.alloc_file
0.93 ? 6% perf-profile.children.cycles-pp.__alloc_file
0.19 ? 3% perf-profile.children.cycles-pp.ip_finish_output
0.61 ? 6% perf-profile.children.cycles-pp.sk_stream_alloc_skb
0.73 ? 5% perf-profile.children.cycles-pp.dev_hard_start_xmit
0.72 ? 6% perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
0.67 ? 5% perf-profile.children.cycles-pp.tcp_send_fin
1.07 ? 9% perf-profile.children.cycles-pp.menu_select
0.57 ? 7% perf-profile.children.cycles-pp.sock_alloc
0.56 ? 7% perf-profile.children.cycles-pp.new_inode_pseudo
0.86 ? 6% perf-profile.children.cycles-pp.__kfree_skb
0.53 ? 8% perf-profile.children.cycles-pp.tcp_get_metrics
0.65 ? 4% perf-profile.children.cycles-pp.loopback_xmit
0.79 ? 5% perf-profile.children.cycles-pp.inet_csk_clone_lock
0.56 ? 6% perf-profile.children.cycles-pp.__dentry_kill
0.54 ? 6% perf-profile.children.cycles-pp.sk_reset_timer
0.77 ? 6% perf-profile.children.cycles-pp.sk_clone_lock
0.49 ? 8% perf-profile.children.cycles-pp.alloc_inode
0.47 ? 9% perf-profile.children.cycles-pp.propagate_protected_usage
0.66 ? 7% perf-profile.children.cycles-pp.security_file_alloc
0.49 ? 5% perf-profile.children.cycles-pp.apparmor_file_free_security
0.35 ? 5% perf-profile.children.cycles-pp.lock_timer_base
0.48 ? 6% perf-profile.children.cycles-pp.mod_timer
0.50 ? 5% perf-profile.children.cycles-pp.security_file_free
0.39 ? 7% perf-profile.children.cycles-pp.sock_alloc_inode
0.35 ? 6% perf-profile.children.cycles-pp.sk_filter_trim_cap
0.61 ? 6% perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.34 ? 8% perf-profile.children.cycles-pp.kmalloc_reserve
0.40 ? 6% perf-profile.children.cycles-pp.skb_copy_datagram_iter
0.40 ? 6% perf-profile.children.cycles-pp.__skb_datagram_iter
0.31 ? 9% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.40 ? 6% perf-profile.children.cycles-pp.sched_clock_cpu
0.38 ? 6% perf-profile.children.cycles-pp.sock_recvmsg
0.24 ? 5% perf-profile.children.cycles-pp.irqtime_account_irq
0.26 ? 6% perf-profile.children.cycles-pp.tcp_write_queue_purge
0.33 ? 9% perf-profile.children.cycles-pp.raw_local_deliver
0.22 ? 6% perf-profile.children.cycles-pp.__ksize
0.34 ? 5% perf-profile.children.cycles-pp.native_sched_clock
0.30 ? 6% perf-profile.children.cycles-pp.apparmor_sk_clone_security
0.18 ? 8% perf-profile.children.cycles-pp.ip_local_out
0.34 ? 5% perf-profile.children.cycles-pp.skb_release_all
0.25 ? 6% perf-profile.children.cycles-pp.netif_rx
0.30 ? 6% perf-profile.children.cycles-pp.security_sk_clone
0.30 ? 5% perf-profile.children.cycles-pp.__tcp_send_ack
0.19 ? 5% perf-profile.children.cycles-pp.__skb_clone
0.20 ? 10% perf-profile.children.cycles-pp.validate_xmit_skb
0.16 ? 9% perf-profile.children.cycles-pp.__ip_local_out
0.32 ? 4% perf-profile.children.cycles-pp.skb_release_head_state
0.15 ? 7% perf-profile.children.cycles-pp.__check_object_size
0.24 ? 5% perf-profile.children.cycles-pp.netif_rx_internal
0.40 ? 5% perf-profile.children.cycles-pp.read_tsc
0.27 ? 4% perf-profile.children.cycles-pp.tcp_mstamp_refresh
0.16 ? 11% perf-profile.children.cycles-pp.mod_objcg_state
0.26 ? 6% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.21 ? 5% perf-profile.children.cycles-pp.enqueue_to_backlog
0.13 ? 9% perf-profile.children.cycles-pp.ip_send_check
0.08 ? 4% perf-profile.children.cycles-pp.migrate_enable
0.22 ? 5% perf-profile.children.cycles-pp.sk_stop_timer
0.20 ? 4% perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
0.16 ? 6% perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.28 ? 6% perf-profile.children.cycles-pp.select_task_rq_fair
0.18 ? 8% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.20 ? 8% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.19 ? 9% perf-profile.children.cycles-pp.check_preempt_curr
0.08 ? 6% perf-profile.children.cycles-pp.sock_put
0.24 ? 6% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.15 ? 7% perf-profile.children.cycles-pp.del_timer
0.10 ? 14% perf-profile.children.cycles-pp.netif_skb_features
0.15 ? 6% perf-profile.children.cycles-pp.__entry_text_start
0.13 ? 6% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.05 ? 45% perf-profile.children.cycles-pp.netdev_core_pick_tx
0.11 ? 6% perf-profile.children.cycles-pp.tcp_rcv_space_adjust
0.04 ? 71% perf-profile.children.cycles-pp.minmax_running_min
0.08 ? 8% perf-profile.children.cycles-pp.tcp_queue_rcv
0.11 ? 7% perf-profile.children.cycles-pp.simple_copy_to_iter
0.14 ? 8% perf-profile.children.cycles-pp._copy_to_iter
0.17 ? 9% perf-profile.children.cycles-pp.resched_curr
0.15 ? 2% perf-profile.children.cycles-pp.sock_getsockopt
0.08 ? 7% perf-profile.children.cycles-pp.destroy_inode
0.12 ? 8% perf-profile.children.cycles-pp.__netif_receive_skb_core
0.08 ? 10% perf-profile.children.cycles-pp.__destroy_inode
0.05 ? 8% perf-profile.children.cycles-pp.nf_hook_slow
0.12 ? 10% perf-profile.children.cycles-pp.copyout
0.13 ? 5% perf-profile.children.cycles-pp.__cond_resched
0.06 ? 11% perf-profile.children.cycles-pp.inet_reqsk_alloc
0.06 ? 9% perf-profile.children.cycles-pp.inet_csk_complete_hashdance
0.14 ? 8% perf-profile.children.cycles-pp.__fget_light
0.09 ? 10% perf-profile.children.cycles-pp.cpus_share_cache
0.12 ? 10% perf-profile.children.cycles-pp.tcp_wfree
0.06 ? 9% perf-profile.children.cycles-pp.migrate_disable
0.12 ? 6% perf-profile.children.cycles-pp.sk_setup_caps
0.11 ? 5% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.11 ? 5% perf-profile.children.cycles-pp.d_instantiate
0.06 ? 8% perf-profile.children.cycles-pp.tcp_options_write
0.13 ? 4% perf-profile.children.cycles-pp.evict
0.08 ? 5% perf-profile.children.cycles-pp.tcp_validate_incoming
0.04 ? 45% perf-profile.children.cycles-pp.__d_instantiate
0.09 ? 6% perf-profile.children.cycles-pp.__might_fault
0.08 ? 5% perf-profile.children.cycles-pp.___perf_sw_event
0.06 ? 11% perf-profile.children.cycles-pp.security_sock_rcv_skb
0.06 perf-profile.children.cycles-pp.kmem_cache_alloc_trace
0.06 ? 9% perf-profile.children.cycles-pp.move_addr_to_user
0.06 ? 13% perf-profile.children.cycles-pp.set_next_task_idle
0.06 ? 11% perf-profile.children.cycles-pp.pick_next_task_idle
0.05 ? 7% perf-profile.children.cycles-pp.prepare_to_wait_exclusive
0.06 ? 6% perf-profile.children.cycles-pp.add_wait_queue
0.20 ? 9% perf-profile.children.cycles-pp.inet_twsk_deschedule_put
0.13 ? 9% perf-profile.children.cycles-pp.inet_twsk_alloc
0.10 ? 9% perf-profile.children.cycles-pp.tcp_mtup_init
0.18 ? 9% perf-profile.children.cycles-pp.inet_twsk_kill
0.29 ? 6% perf-profile.children.cycles-pp.tcp_make_synack
0.12 ? 6% perf-profile.children.cycles-pp.tcp_synack_rtt_meas
0.06 ? 9% perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
0.12 ? 5% perf-profile.children.cycles-pp.tcp_ca_openreq_child
0.10 ? 10% perf-profile.children.cycles-pp.inet_lhash2_lookup
0.15 ? 7% perf-profile.children.cycles-pp.__inet_lookup_listener
0.20 ? 3% perf-profile.children.cycles-pp.__ip_finish_output
0.14 ? 8% perf-profile.children.cycles-pp.__ip_dev_find
0.15 ? 8% perf-profile.children.cycles-pp.tcp_sync_mss
0.08 ? 5% perf-profile.children.cycles-pp.sock_prot_inuse_add
0.09 ? 5% perf-profile.children.cycles-pp.tcp_push
0.50 ? 6% perf-profile.children.cycles-pp.tcp_time_wait
0.17 ? 5% perf-profile.children.cycles-pp.secure_tcp_ts_off
0.17 ? 5% perf-profile.children.cycles-pp.tcp_select_initial_window
0.13 ? 7% perf-profile.children.cycles-pp.inet_lookup_ifaddr_rcu
0.20 ? 8% perf-profile.children.cycles-pp.skb_entail
0.23 ? 6% perf-profile.children.cycles-pp.tcp_openreq_init_rwin
0.13 ? 8% perf-profile.children.cycles-pp.tcp_stream_memory_free
0.27 ? 3% perf-profile.children.cycles-pp.tcp_update_pacing_rate
0.28 ? 10% perf-profile.children.cycles-pp.tcp_parse_options
0.50 ? 7% perf-profile.children.cycles-pp.tcp_finish_connect
0.31 ? 4% perf-profile.children.cycles-pp.tcp_connect_init
0.41 ? 7% perf-profile.children.cycles-pp.fib_table_lookup
0.26 ? 5% perf-profile.children.cycles-pp.tcp_assign_congestion_control
0.17 ? 9% perf-profile.children.cycles-pp.__sk_free
0.53 ? 5% perf-profile.children.cycles-pp.tcp_v4_init_sock
0.41 ? 6% perf-profile.children.cycles-pp.tcp_schedule_loss_probe
0.52 ? 5% perf-profile.children.cycles-pp.tcp_init_sock
0.78 ? 5% perf-profile.children.cycles-pp.tcp_v4_send_synack
0.34 ? 8% perf-profile.children.cycles-pp.ip_rcv_finish
0.31 ? 8% perf-profile.children.cycles-pp.ip_rcv_finish_core
0.29 ? 7% perf-profile.children.cycles-pp.fib_lookup_good_nhc
0.42 ? 8% perf-profile.children.cycles-pp.inet_csk_route_child_sock
0.41 ? 8% perf-profile.children.cycles-pp.inet_csk_route_req
0.27 ? 8% perf-profile.children.cycles-pp.tcp_newly_delivered
0.88 ? 5% perf-profile.children.cycles-pp.tcp_init_transfer
0.36 ? 6% perf-profile.children.cycles-pp.inet_sk_rebuild_header
0.38 ? 5% perf-profile.children.cycles-pp.inet_sock_destruct
0.39 ? 6% perf-profile.children.cycles-pp.__tcp_select_window
0.54 ? 6% perf-profile.children.cycles-pp.tcp_ack_update_rtt
0.49 ? 6% perf-profile.children.cycles-pp.inet_ehashfn
2.08 ? 5% perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
0.55 ? 6% perf-profile.children.cycles-pp.ip_rcv_core
2.36 ? 6% perf-profile.children.cycles-pp.tcp_check_req
2.14 ? 6% perf-profile.children.cycles-pp.tcp_conn_request
1.03 ? 7% perf-profile.children.cycles-pp.ip_route_output_flow
2.77 ? 6% perf-profile.children.cycles-pp.tcp_ack
1.06 ? 4% perf-profile.children.cycles-pp.__sk_dst_check
1.42 ? 7% perf-profile.children.cycles-pp.ip_route_output_key_hash
1.38 ? 6% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
1.22 ? 4% perf-profile.children.cycles-pp.ipv4_dst_check
6.92 ? 6% perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
5.74 ? 5% perf-profile.children.cycles-pp.tcp_connect
7.04 ? 5% perf-profile.children.cycles-pp.tcp_v4_connect
14.51 ? 5% perf-profile.children.cycles-pp.__x64_sys_connect
14.50 ? 5% perf-profile.children.cycles-pp.__sys_connect
14.31 ? 5% perf-profile.children.cycles-pp.inet_stream_connect
14.27 ? 5% perf-profile.children.cycles-pp.__inet_stream_connect
3.14 ? 5% perf-profile.self.cycles-pp.poll_idle
0.70 ? 7% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
1.49 ? 9% perf-profile.self.cycles-pp.page_counter_cancel
1.40 ? 6% perf-profile.self.cycles-pp._raw_spin_lock
0.74 ? 5% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.88 ? 7% perf-profile.self.cycles-pp.__inet_lookup_established
0.37 ? 11% perf-profile.self.cycles-pp.__tcp_get_metrics
0.27 ? 10% perf-profile.self.cycles-pp.__sk_destruct
0.82 ? 7% perf-profile.self.cycles-pp._raw_spin_lock_bh
0.62 ? 5% perf-profile.self.cycles-pp.sk_forced_mem_schedule
0.71 ? 6% perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
0.23 ? 4% perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
0.46 ? 8% perf-profile.self.cycles-pp.propagate_protected_usage
0.29 ? 7% perf-profile.self.cycles-pp.tcp_recvmsg_locked
0.40 ? 8% perf-profile.self.cycles-pp.__dev_queue_xmit
0.49 ? 4% perf-profile.self.cycles-pp.apparmor_file_free_security
0.59 ? 6% perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.24 ? 4% perf-profile.self.cycles-pp.tcp_clean_rtx_queue
0.33 ? 5% perf-profile.self.cycles-pp.native_sched_clock
0.32 ? 6% perf-profile.self.cycles-pp.kmem_cache_free
0.30 ? 6% perf-profile.self.cycles-pp.apparmor_sk_clone_security
0.33 ? 10% perf-profile.self.cycles-pp.raw_local_deliver
0.22 ? 6% perf-profile.self.cycles-pp.__ksize
0.22 ? 6% perf-profile.self.cycles-pp.ip_finish_output2
0.38 ? 5% perf-profile.self.cycles-pp.read_tsc
0.15 ? 6% perf-profile.self.cycles-pp.__skb_clone
0.26 ? 5% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.21 ? 6% perf-profile.self.cycles-pp.tcp_sendmsg_locked
0.12 ? 7% perf-profile.self.cycles-pp.fib_table_lookup
0.13 ? 9% perf-profile.self.cycles-pp.ip_send_check
0.07 ? 12% perf-profile.self.cycles-pp.migrate_enable
0.20 ? 6% perf-profile.self.cycles-pp.__alloc_skb
0.14 ? 6% perf-profile.self.cycles-pp.process_backlog
0.13 ? 5% perf-profile.self.cycles-pp.net_rx_action
0.30 ? 4% perf-profile.self.cycles-pp.enqueue_entity
0.03 ?100% perf-profile.self.cycles-pp.minmax_running_min
0.08 ? 6% perf-profile.self.cycles-pp.sock_put
0.08 ? 8% perf-profile.self.cycles-pp.tcp_queue_rcv
0.19 ? 10% perf-profile.self.cycles-pp.__softirqentry_text_start
0.15 ? 6% perf-profile.self.cycles-pp.__entry_text_start
0.12 ? 10% perf-profile.self.cycles-pp.mod_objcg_state
0.11 ? 6% perf-profile.self.cycles-pp.do_syscall_64
0.18 ? 7% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.04 ? 44% perf-profile.self.cycles-pp.netdev_core_pick_tx
0.04 ? 71% perf-profile.self.cycles-pp.inet_reqsk_alloc
0.11 ? 8% perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.17 ? 5% perf-profile.self.cycles-pp.__local_bh_enable_ip
0.05 ? 8% perf-profile.self.cycles-pp.tcp_data_queue
0.17 ? 9% perf-profile.self.cycles-pp.resched_curr
0.13 ? 5% perf-profile.self.cycles-pp.__tcp_close
0.08 ? 8% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.07 ? 8% perf-profile.self.cycles-pp.irqtime_account_irq
0.12 ? 11% perf-profile.self.cycles-pp.sk_reset_timer
0.11 ? 9% perf-profile.self.cycles-pp.__netif_receive_skb_core
0.05 ? 45% perf-profile.self.cycles-pp.tcp_rcv_synsent_state_process
0.06 ? 9% perf-profile.self.cycles-pp.migrate_disable
0.09 ? 10% perf-profile.self.cycles-pp.cpus_share_cache
0.07 ? 11% perf-profile.self.cycles-pp.tcp_rcv_space_adjust
0.11 ? 5% perf-profile.self.cycles-pp.enqueue_to_backlog
0.10 ? 4% perf-profile.self.cycles-pp.validate_xmit_skb
0.08 ? 8% perf-profile.self.cycles-pp.tcp_validate_incoming
0.14 ? 9% perf-profile.self.cycles-pp.__fget_light
0.06 ? 11% perf-profile.self.cycles-pp.__netif_receive_skb_one_core
0.05 perf-profile.self.cycles-pp.tcp_options_write
0.06 ? 15% perf-profile.self.cycles-pp.netif_skb_features
0.12 ? 6% perf-profile.self.cycles-pp.sk_setup_caps
0.06 ? 11% perf-profile.self.cycles-pp.tcp_cleanup_rbuf
0.09 ? 4% perf-profile.self.cycles-pp.do_softirq
0.07 ? 9% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.04 ? 45% perf-profile.self.cycles-pp.tcp_established_options
0.06 ? 11% perf-profile.self.cycles-pp.sk_filter_trim_cap
0.06 ? 7% perf-profile.self.cycles-pp.__list_del_entry_valid
0.08 ? 8% perf-profile.self.cycles-pp.dequeue_task_fair
0.05 ? 8% perf-profile.self.cycles-pp.__alloc_file
0.06 ? 7% perf-profile.self.cycles-pp.schedule
0.08 ? 8% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
0.09 ? 7% perf-profile.self.cycles-pp.tcp_init_transfer
0.12 ? 7% perf-profile.self.cycles-pp.inet_twsk_alloc
0.09 ? 7% perf-profile.self.cycles-pp.tcp_mtup_init
0.10 ? 7% perf-profile.self.cycles-pp.inet_csk_get_port
0.06 ? 9% perf-profile.self.cycles-pp.__inet_inherit_port
0.17 ? 6% perf-profile.self.cycles-pp.sk_clone_lock
0.06 ? 8% perf-profile.self.cycles-pp.inet_put_port
0.06 ? 8% perf-profile.self.cycles-pp.__inet_lookup_listener
0.06 ? 9% perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
0.14 ? 6% perf-profile.self.cycles-pp.tcp_connect
0.07 ? 7% perf-profile.self.cycles-pp.inet_twsk_kill
0.10 ? 10% perf-profile.self.cycles-pp.inet_lhash2_lookup
0.20 ? 4% perf-profile.self.cycles-pp.__ip_finish_output
0.15 ? 8% perf-profile.self.cycles-pp.tcp_sync_mss
0.14 ? 7% perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
0.08 ? 4% perf-profile.self.cycles-pp.sock_prot_inuse_add
0.15 ? 7% perf-profile.self.cycles-pp.tcp_make_synack
0.16 ? 6% perf-profile.self.cycles-pp.__inet_bind
0.09 ? 5% perf-profile.self.cycles-pp.tcp_push
0.14 ? 6% perf-profile.self.cycles-pp.tcp_openreq_init_rwin
0.15 ? 6% perf-profile.self.cycles-pp.tcp_create_openreq_child
0.17 ? 6% perf-profile.self.cycles-pp.tcp_select_initial_window
0.10 ? 7% perf-profile.self.cycles-pp.tcp_v4_send_synack
0.15 ? 6% perf-profile.self.cycles-pp.tcp_get_metrics
0.13 ? 7% perf-profile.self.cycles-pp.inet_lookup_ifaddr_rcu
0.18 ? 6% perf-profile.self.cycles-pp.tcp_connect_init
0.20 ? 7% perf-profile.self.cycles-pp.tcp_update_metrics
0.18 ? 7% perf-profile.self.cycles-pp.tcp_init_metrics
0.12 ? 11% perf-profile.self.cycles-pp.tcp_stream_memory_free
0.19 ? 8% perf-profile.self.cycles-pp.skb_entail
0.24 ? 5% perf-profile.self.cycles-pp.tcp_event_new_data_sent
0.27 ? 3% perf-profile.self.cycles-pp.tcp_update_pacing_rate
0.28 ? 9% perf-profile.self.cycles-pp.tcp_parse_options
0.14 ? 5% perf-profile.self.cycles-pp.secure_tcp_ts_off
0.60 ? 6% perf-profile.self.cycles-pp.tcp_v4_rcv
0.15 ? 10% perf-profile.self.cycles-pp.__sk_free
0.62 ? 8% perf-profile.self.cycles-pp.tcp_ack
0.22 ? 7% perf-profile.self.cycles-pp.tcp_set_state
0.25 ? 6% perf-profile.self.cycles-pp.tcp_assign_congestion_control
0.31 ? 7% perf-profile.self.cycles-pp.ip_rcv_finish_core
0.37 ? 7% perf-profile.self.cycles-pp.tcp_schedule_loss_probe
0.28 ? 8% perf-profile.self.cycles-pp.fib_lookup_good_nhc
0.23 ? 4% perf-profile.self.cycles-pp.tcp_init_sock
0.26 ? 7% perf-profile.self.cycles-pp.tcp_newly_delivered
0.38 ? 6% perf-profile.self.cycles-pp.__tcp_select_window
0.60 ? 7% perf-profile.self.cycles-pp.tcp_write_xmit
0.48 ? 6% perf-profile.self.cycles-pp.inet_ehashfn
0.48 ? 6% perf-profile.self.cycles-pp.tcp_ack_update_rtt
0.55 ? 7% perf-profile.self.cycles-pp.ip_rcv_core
1.20 ? 6% perf-profile.self.cycles-pp.__tcp_transmit_skb
0.59 ? 9% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
0.82 ? 6% perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
0.88 ? 6% perf-profile.self.cycles-pp.ip_output
1.21 ? 4% perf-profile.self.cycles-pp.ipv4_dst_check

netperf.Throughput_tps

--------------------------------------------------+
|
+.++.++.++.++.++.+ |
++.+ +.++. |
+.++.++.++.++.++.+ ++.++.+|
|
|
|
|
|
|
|
O OO OO O O |
O OO O O |
--------------------------------------------------+

netperf.Throughput_total_tps

--------------------------------------------------+
|
++.++.++.++.++.++. |
+++. .++. |
++.++.++.++.+++.++ ++.++.+|
|
|
|
|
|
|
|
OO OO OO OO OO OO |
|
--------------------------------------------------+

netperf.workload

--------------------------------------------------+
|
+. .+++.++.+++.++.++. |
+++. .++.+ |
++.+++.++.+++.++ ++.++.+|
|
|
|
|
|
|
|
OOO OO OOO OO OO |
|
--------------------------------------------------+

netperf.time.user_time

--------------------------------------------------+
++ |
:: |
: : |
: : |
+.+ +. .+. : : |
+.+ ++ ++.+ +.+.+ .++ ++.++.|
+.++.+ +.+ + |
+.+ |
|
|
|
O |
O O |
--------------------------------------------------+

netperf.time.system_time

--------------------------------------------------+
+ +.++.++. +.+ .+ ++.+ + |
+ +.++ +.+ .+ + + +.++.+|
+.++ +.++ + |
|
|
|
|
|
|
|
|
OO OO |
OO |
--------------------------------------------------+

netperf.time.percent_of_cpu_this_job_got

--------------------------------------------------+
|
+.++.+ +. .++.++.+ |
+.+ ++.+.++. +. + +. .++.|
+ ++.++.++.+ ++ |
|
|
|
|
|
|
|
O OO OO |
O |
--------------------------------------------------+

netperf.time.voluntary_context_switches

--------------------------------------------------+
.++.++.+++.++.++.+ |
: |
++.++.++.+++.++.++.++.+++.++.++.+|
|
|
|
|
|
|
|
|
O |
OO OO OO OO OO O |
--------------------------------------------------+

--------------------------------------------------+
+.++ .+ +. |
: + +.+ +++.+ |
+.++. +.++.++.+++. |
+++.+ ++.++.+|
|
|
|
|
|
|
|
OO OO OO O |
O O OO O |
--------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.
Open Source Technology Center
.org/hyperkitty/list/lkp@lists.01.org">https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Attachments:

(No filename) (79.54 kB)
config-5.14.0-rc5-01158-gb89fbfbb854c (178.24 kB)
job-script (8.00 kB)
job.yaml (5.46 kB)
reproduce (1.09 kB)
Download all attachments

2021-08-20 06:32:33

by Yonghong Song

[permalink] [raw]

Subject: Re: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression

On 8/19/21 10:50 PM, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed a -21.3% regression of netperf.Throughput_tps due to commit:
>
>
> commit: b89fbfbb854c9afc3047e8273cc3a694650b802e ("bpf: Implement minimal BPF perf link")
> https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

Could you do a test again to double check?
The above patch is related to bpf perf_event area.
It should not have any impact with netperf.

>
>
> in testcase: netperf
> on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
> with following parameters:
>
> ip: ipv4
> runtime: 300s
> nr_threads: 16
> cluster: cs-localhost
> test: TCP_CRR
> cpufreq_governor: performance
> ucode: 0x5003006
>
> test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
> test-url: http://www.netperf.org/netperf/
>
>
>
[...]

2021-09-02 08:38:43

by Carel Si

[permalink] [raw]

Subject: RE: [LKP] Re: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression

Hi Yonghong,

> -----Original Message-----
> From: Yonghong Song <[email protected]>
> Sent: Friday, August 20, 2021 2:29 PM
> To: Sang, Oliver <[email protected]>; Andrii Nakryiko <[email protected]>
> Cc: Daniel Borkmann <[email protected]>; Peter Zijlstra
> <[email protected]>; LKML <[email protected]>; Linux Memory
> Management List <[email protected]>; [email protected]; lkp <[email protected]>
> Subject: [LKP] Re: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression
>
>
>
> On 8/19/21 10:50 PM, kernel test robot wrote:
> >
> >
> > Greeting,
> >
> > FYI, we noticed a -21.3% regression of netperf.Throughput_tps due to commit:
> >
> >
> > commit: b89fbfbb854c9afc3047e8273cc3a694650b802e ("bpf: Implement
> > minimal BPF perf link")
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git
> > master
>
> Could you do a test again to double check?
> The above patch is related to bpf perf_event area.
> It should not have any impact with netperf.

We first tested again and confirmed the regression still exists (-21.4%), the result is stable [1], more details are in [2].

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
382134 -21.4% 300300 netperf.Throughput_total_tps
23958 -21.4% 18828 netperf.Throughput_tps

Then we tried another platform, Ice Lake, it has 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G, but we found the regression is still existing (-5.4%), the result is stable [3], more details are in [4].

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
522472 -5.3% 494763 netperf.Throughput_total_tps
32654 -5.3% 30922 netperf.Throughput_tps

[1]

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18860.396249999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18808.879375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18905.214375000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18750.746874999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18803.823125000006,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 19201.846,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18819.47,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18979.38,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18837.719375000004,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18809.4725,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18776.799375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18773.092500000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18743.978125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18794.814374999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18810.783124999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18821.65625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18680.149375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18747.203125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18819.728750000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18830.020625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18656.501874999998
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_total_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301766.33999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300942.07,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 302483.43000000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300011.94999999995,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300861.1700000001,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 288027.69,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301111.52,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 303670.08,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301403.51000000007,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300951.56,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300428.79,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300369.48000000004,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 299903.65,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300717.02999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300972.52999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301146.5,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 298882.39,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 299955.25,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301115.66000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301280.33,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 298504.02999999997
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],

[2]

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003006

commit:
652c1b17b8 ("bpf: Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
b89fbfbb85 ("bpf: Implement minimal BPF perf link")

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
381418 -21.3% 300214 netperf.Throughput_total_tps
23906 -21.3% 18820 netperf.Throughput_tps
872.95 -5.2% 827.95 netperf.time.percent_of_cpu_this_job_got
2586 -4.3% 2474 netperf.time.system_time
2.25e+08 -22.8% 1.736e+08 netperf.time.voluntary_context_switches
1.144e+08 -21.3% 90064356 netperf.workload
2346775 -8.7% 2143411 vmstat.system.cs
872.95 -5.2% 827.95 time.percent_of_cpu_this_job_got
2586 -4.3% 2474 time.system_time
2.25e+08 -22.8% 1.736e+08 time.voluntary_context_switches
0.01 ± 7% +68.4% 0.01 ± 7% perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
3491773 ± 2% -22.6% 2702148 perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
4512682 ± 4% +21.9% 5501355 ± 2% perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
0.01 ± 8% +86.5% 0.01 ± 8% perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
0.03 ± 8% +171.3% 0.08 ±231% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.apparmor_file_alloc_security.security_file_alloc.__alloc_file
113308 -2.8% 110156 proc-vmstat.nr_slab_unreclaimable
4057280 ± 2% -8.6% 3709088 ± 2% proc-vmstat.numa_hit
3847678 ± 2% -8.9% 3506977 ± 2% proc-vmstat.numa_local
4055392 ± 2% -8.5% 3709694 ± 2% proc-vmstat.pgalloc_normal
1770294 ± 2% -23.5% 1355122 ± 2% proc-vmstat.pgfree
2456040 ± 16% -34.0% 1620559 ± 20% softirqs.CPU121.NET_RX
1737990 ± 24% -36.1% 1109923 ± 27% softirqs.CPU163.NET_RX
3.519e+08 -22.1% 2.742e+08 softirqs.NET_RX
26128869 ± 2% -15.7% 22015818 softirqs.RCU
237092 ± 2% +84.0% 436186 softirqs.TIMER
170292 -12.0% 149884 slabinfo.filp.active_objs
2663 -12.0% 2343 slabinfo.filp.active_slabs
170459 -12.0% 150036 slabinfo.filp.num_objs
2663 -12.0% 2343 slabinfo.filp.num_slabs
49886 ± 5% -11.6% 44079 ± 4% slabinfo.kmalloc-512.active_objs
50080 ± 5% -11.6% 44269 ± 4% slabinfo.kmalloc-512.num_objs
12.06 ± 4% +26.2% 15.22 ± 2% perf-stat.i.MPKI
1.004e+10 -18.0% 8.233e+09 perf-stat.i.branch-instructions
1.562e+08 -19.6% 1.256e+08 perf-stat.i.branch-misses
10.92 +7.2 18.09 perf-stat.i.cache-miss-rate%
64015758 ± 2% +75.1% 1.121e+08 perf-stat.i.cache-misses
5.888e+08 +5.2% 6.196e+08 perf-stat.i.cache-references
2379887 -8.7% 2173629 perf-stat.i.context-switches
1.57 ± 2% +17.2% 1.84 perf-stat.i.cpi
7.739e+10 -3.5% 7.467e+10 perf-stat.i.cpu-cycles
1272 ± 2% -43.8% 715.53 ± 3% perf-stat.i.cycles-between-cache-misses
903463 ± 18% -27.4% 655855 ± 5% perf-stat.i.dTLB-load-misses
1.518e+10 -18.3% 1.24e+10 perf-stat.i.dTLB-loads
228225 ± 31% -52.8% 107778 ± 23% perf-stat.i.dTLB-store-misses
8.879e+09 -18.5% 7.234e+09 perf-stat.i.dTLB-stores
89.92 -4.7 85.18 perf-stat.i.iTLB-load-miss-rate%
1.075e+08 -21.1% 84797404 perf-stat.i.iTLB-load-misses
11549994 ± 4% +23.9% 14312532 ± 2% perf-stat.i.iTLB-loads
5.079e+10 -18.2% 4.155e+10 perf-stat.i.instructions
0.65 -15.2% 0.55 perf-stat.i.ipc
0.40 -3.5% 0.39 perf-stat.i.metric.GHz
180.81 -17.8% 148.59 perf-stat.i.metric.M/sec
13211450 +59.2% 21027780 perf-stat.i.node-load-misses
1191177 ± 5% +68.5% 2006794 ± 9% perf-stat.i.node-loads
10841458 -22.2% 8430280 perf-stat.i.node-store-misses
11.59 +28.7% 14.91 perf-stat.overall.MPKI
1.56 -0.0 1.53 perf-stat.overall.branch-miss-rate%
10.87 +7.2 18.09 perf-stat.overall.cache-miss-rate%
1.52 +18.0% 1.80 perf-stat.overall.cpi
1209 ± 2% -44.9% 666.15 perf-stat.overall.cycles-between-cache-misses
90.30 -4.7 85.56 perf-stat.overall.iTLB-load-miss-rate%
472.39 +3.8% 490.15 perf-stat.overall.instructions-per-iTLB-miss
0.66 -15.2% 0.56 perf-stat.overall.ipc
1e+10 -18.0% 8.205e+09 perf-stat.ps.branch-instructions
1.557e+08 -19.6% 1.251e+08 perf-stat.ps.branch-misses
63792201 ± 2% +75.1% 1.117e+08 perf-stat.ps.cache-misses
5.867e+08 +5.2% 6.174e+08 perf-stat.ps.cache-references
2371522 -8.7% 2165977 perf-stat.ps.context-switches
7.712e+10 -3.5% 7.441e+10 perf-stat.ps.cpu-cycles
900712 ± 18% -27.4% 653896 ± 5% perf-stat.ps.dTLB-load-misses
1.513e+10 -18.3% 1.236e+10 perf-stat.ps.dTLB-loads
227485 ± 31% -52.8% 107457 ± 23% perf-stat.ps.dTLB-store-misses
8.848e+09 -18.5% 7.208e+09 perf-stat.ps.dTLB-stores
1.072e+08 -21.1% 84499344 perf-stat.ps.iTLB-load-misses
11509852 ± 4% +23.9% 14262519 ± 2% perf-stat.ps.iTLB-loads
5.062e+10 -18.2% 4.14e+10 perf-stat.ps.instructions
13165220 +59.2% 20953789 perf-stat.ps.node-load-misses
1187095 ± 5% +68.5% 1999816 ± 9% perf-stat.ps.node-loads
10803432 -22.2% 8400688 perf-stat.ps.node-store-misses
1.543e+13 -18.2% 1.262e+13 perf-stat.total.instructions
5.24 ± 7% -1.2 4.05 ± 7% perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.22 ± 7% -1.2 4.03 ± 7% perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
4.13 ± 21% -1.0 3.17 ± 6% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
2.75 ± 6% -0.6 2.15 ± 8% perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.calltrace.cycles-pp.ret_from_fork
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
1.67 ± 9% -0.5 1.22 ± 9% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.66 ± 9% -0.5 1.21 ± 9% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
1.67 ± 9% -0.5 1.22 ± 9% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.68 ± 9% -0.5 1.23 ± 9% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
1.65 ± 9% -0.5 1.20 ± 9% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
1.58 ± 9% -0.4 1.14 ± 9% perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
1.40 ± 9% -0.4 1.02 ± 9% perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
1.36 ± 9% -0.4 0.99 ± 9% perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core
1.39 ± 8% -0.3 1.09 ± 9% perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
1.07 ± 10% -0.3 0.78 ± 9% perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch
1.10 ± 6% -0.3 0.83 ± 7% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.03 ± 6% -0.3 0.78 ± 7% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu
1.04 ± 8% -0.2 0.79 ± 8% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
0.96 ± 6% -0.2 0.72 ± 7% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv
0.95 ± 6% -0.2 0.71 ± 7% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process
0.97 ± 8% -0.2 0.74 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
0.92 ± 6% -0.2 0.68 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
0.90 ± 8% -0.2 0.68 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
0.88 ± 8% -0.2 0.67 ± 8% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
0.75 ± 7% -0.2 0.57 ± 23% perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept
0.67 ± 7% -0.2 0.49 ± 41% perf-profile.calltrace.cycles-pp.inet_csk_get_port.__inet_bind.inet_bind.__sys_bind.__x64_sys_bind
0.55 ± 22% +0.2 0.72 ± 7% perf-profile.calltrace.cycles-pp.tcp_v4_send_synack.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
0.49 ± 40% +0.3 0.75 ± 8% perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.65 ± 6% +0.3 1.97 ± 7% perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.87 ± 6% +0.4 2.24 ± 7% perf-profile.calltrace.cycles-pp.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
1.56 ± 6% +0.5 2.02 ± 6% perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
3.08 ± 6% +0.7 3.74 ± 7% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
3.05 ± 6% +0.7 3.73 ± 7% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
3.42 ± 6% +0.8 4.18 ± 7% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
3.64 ± 6% +0.9 4.50 ± 6% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
4.42 ± 6% +1.0 5.44 ± 7% perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
5.36 ± 6% +1.3 6.68 ± 7% perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
11.59 ± 5% +2.1 13.65 ± 6% perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.54 ± 5% +2.1 13.61 ± 6% perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
5.94 ± 6% -1.3 4.60 ± 6% perf-profile.children.cycles-pp.__x64_sys_recvfrom
5.90 ± 6% -1.3 4.58 ± 6% perf-profile.children.cycles-pp.__sys_recvfrom
5.32 ± 6% -1.2 4.08 ± 6% perf-profile.children.cycles-pp.inet_recvmsg
5.30 ± 6% -1.2 4.07 ± 6% perf-profile.children.cycles-pp.tcp_recvmsg
4.17 ± 22% -1.0 3.19 ± 6% perf-profile.children.cycles-pp.poll_idle
4.48 ± 6% -0.8 3.71 ± 6% perf-profile.children.cycles-pp.tcp_recvmsg_locked
3.06 ± 8% -0.7 2.39 ± 7% perf-profile.children.cycles-pp.kmem_cache_free
1.66 ± 6% -0.6 1.06 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_bh
2.80 ± 6% -0.6 2.21 ± 7% perf-profile.children.cycles-pp.__x64_sys_accept
2.78 ± 6% -0.6 2.20 ± 7% perf-profile.children.cycles-pp.__sys_accept4
2.75 ± 6% -0.6 2.17 ± 7% perf-profile.children.cycles-pp.__sys_accept4_file
2.37 ± 9% -0.5 1.85 ± 8% perf-profile.children.cycles-pp.rcu_core
2.33 ± 9% -0.5 1.81 ± 8% perf-profile.children.cycles-pp.rcu_do_batch
2.35 ± 8% -0.5 1.84 ± 8% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
2.31 ± 8% -0.5 1.81 ± 8% perf-profile.children.cycles-pp.page_counter_uncharge
1.12 ± 7% -0.5 0.63 ± 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.37 ± 6% -0.5 0.89 ± 6% perf-profile.children.cycles-pp.lock_sock_nested
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.children.cycles-pp.ret_from_fork
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.children.cycles-pp.kthread
1.67 ± 9% -0.5 1.22 ± 9% perf-profile.children.cycles-pp.run_ksoftirqd
1.68 ± 9% -0.5 1.23 ± 9% perf-profile.children.cycles-pp.smpboot_thread_fn
2.84 ± 6% -0.4 2.43 ± 6% perf-profile.children.cycles-pp.sk_wait_data
1.78 ± 6% -0.4 1.36 ± 6% perf-profile.children.cycles-pp.__dev_queue_xmit
1.83 ± 9% -0.4 1.43 ± 8% perf-profile.children.cycles-pp.page_counter_cancel
2.14 ± 6% -0.4 1.75 ± 6% perf-profile.children.cycles-pp._raw_spin_lock
1.44 ± 7% -0.3 1.13 ± 6% perf-profile.children.cycles-pp.kmem_cache_alloc
1.40 ± 7% -0.3 1.10 ± 8% perf-profile.children.cycles-pp.inet_accept
1.56 ± 6% -0.3 1.26 ± 6% perf-profile.children.cycles-pp.alloc_file_pseudo
1.56 ± 6% -0.3 1.27 ± 6% perf-profile.children.cycles-pp.sock_alloc_file
1.03 ± 6% -0.3 0.73 ± 7% perf-profile.children.cycles-pp.percpu_counter_add_batch
1.20 ± 6% -0.3 0.92 ± 6% perf-profile.children.cycles-pp.__alloc_skb
0.62 ± 6% -0.3 0.37 ± 9% perf-profile.children.cycles-pp.__tcp_get_metrics
0.97 ± 5% -0.2 0.73 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function
1.15 ± 6% -0.2 0.92 ± 8% perf-profile.children.cycles-pp.inet_csk_accept
0.59 ± 5% -0.2 0.36 ± 7% perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
0.80 ± 6% -0.2 0.58 ± 6% perf-profile.children.cycles-pp.sk_forced_mem_schedule
1.12 ± 6% -0.2 0.90 ± 6% perf-profile.children.cycles-pp.alloc_file
1.10 ± 6% -0.2 0.89 ± 6% perf-profile.children.cycles-pp.alloc_empty_file
1.03 ± 6% -0.2 0.82 ± 7% perf-profile.children.cycles-pp.sock_def_wakeup
1.08 ± 6% -0.2 0.88 ± 6% perf-profile.children.cycles-pp.__alloc_file
1.21 ± 6% -0.2 1.01 ± 6% perf-profile.children.cycles-pp.inet_csk_destroy_sock
0.78 ± 6% -0.2 0.58 ± 6% perf-profile.children.cycles-pp.sk_stream_alloc_skb
0.89 ± 6% -0.2 0.70 ± 7% perf-profile.children.cycles-pp.dev_hard_start_xmit
0.36 ± 6% -0.2 0.18 ± 6% perf-profile.children.cycles-pp.ip_finish_output
0.85 ± 6% -0.2 0.68 ± 7% perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
0.78 ± 6% -0.2 0.61 ± 7% perf-profile.children.cycles-pp.loopback_xmit
1.00 ± 6% -0.2 0.83 ± 6% perf-profile.children.cycles-pp.__kfree_skb
1.15 ± 6% -0.2 0.98 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.68 ± 6% -0.2 0.51 ± 7% perf-profile.children.cycles-pp.tcp_get_metrics
0.79 ± 6% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.tcp_send_fin
0.92 ± 6% -0.2 0.76 ± 6% perf-profile.children.cycles-pp.inet_csk_clone_lock
0.67 ± 6% -0.2 0.51 ± 8% perf-profile.children.cycles-pp.sk_reset_timer
0.68 ± 6% -0.2 0.52 ± 7% perf-profile.children.cycles-pp.sock_alloc
0.67 ± 6% -0.2 0.52 ± 7% perf-profile.children.cycles-pp.new_inode_pseudo
0.89 ± 5% -0.1 0.74 ± 6% perf-profile.children.cycles-pp.sk_clone_lock
0.77 ± 7% -0.1 0.63 ± 7% perf-profile.children.cycles-pp.tcp_done
0.76 ± 6% -0.1 0.62 ± 6% perf-profile.children.cycles-pp.security_file_alloc
0.58 ± 7% -0.1 0.45 ± 7% perf-profile.children.cycles-pp.alloc_inode
0.59 ± 6% -0.1 0.46 ± 7% perf-profile.children.cycles-pp.mod_timer
0.67 ± 6% -0.1 0.54 ± 6% perf-profile.children.cycles-pp.__dentry_kill
0.69 ± 6% -0.1 0.57 ± 6% perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.45 ± 6% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.lock_timer_base
0.58 ± 7% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.apparmor_file_free_security
0.59 ± 7% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.security_file_free
0.68 ± 7% -0.1 0.56 ± 7% perf-profile.children.cycles-pp.inet_csk_get_port
0.44 ± 5% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.sk_filter_trim_cap
0.47 ± 7% -0.1 0.36 ± 7% perf-profile.children.cycles-pp.sock_alloc_inode
0.54 ± 8% -0.1 0.44 ± 10% perf-profile.children.cycles-pp.propagate_protected_usage
0.68 ± 6% -0.1 0.58 ± 6% perf-profile.children.cycles-pp.skb_release_data
0.43 ± 6% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.kmalloc_reserve
0.45 ± 10% -0.1 0.36 ± 10% perf-profile.children.cycles-pp.inet_csk_bind_conflict
0.38 ± 6% -0.1 0.30 ± 7% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.40 ± 6% -0.1 0.32 ± 7% perf-profile.children.cycles-pp.raw_local_deliver
0.46 ± 6% -0.1 0.38 ± 7% perf-profile.children.cycles-pp.read_tsc
0.40 ± 6% -0.1 0.32 ± 5% perf-profile.children.cycles-pp.skb_release_all
0.44 ± 6% -0.1 0.36 ± 7% perf-profile.children.cycles-pp.sock_recvmsg
0.37 ± 7% -0.1 0.29 ± 8% perf-profile.children.cycles-pp.__tcp_send_ack
0.46 ± 7% -0.1 0.38 ± 7% perf-profile.children.cycles-pp.__skb_datagram_iter
0.48 ± 6% -0.1 0.40 ± 7% perf-profile.children.cycles-pp.___might_sleep
0.43 ± 7% -0.1 0.36 ± 8% perf-profile.children.cycles-pp.security_socket_recvmsg
0.38 ± 7% -0.1 0.30 ± 5% perf-profile.children.cycles-pp.skb_release_head_state
0.37 ± 7% -0.1 0.29 ± 7% perf-profile.children.cycles-pp.apparmor_sk_clone_security
0.37 ± 7% -0.1 0.30 ± 6% perf-profile.children.cycles-pp.security_sk_clone
0.29 ± 6% -0.1 0.22 ± 8% perf-profile.children.cycles-pp.__ksize
0.31 ± 6% -0.1 0.24 ± 9% perf-profile.children.cycles-pp.netif_rx
0.37 ± 7% -0.1 0.29 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair
0.32 ± 6% -0.1 0.25 ± 8% perf-profile.children.cycles-pp.tcp_write_queue_purge
0.30 ± 7% -0.1 0.23 ± 9% perf-profile.children.cycles-pp.netif_rx_internal
0.24 ± 5% -0.1 0.17 ± 7% perf-profile.children.cycles-pp.ip_local_out
0.30 ± 6% -0.1 0.23 ± 8% perf-profile.children.cycles-pp.irqtime_account_irq
0.40 ± 6% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.apparmor_socket_post_create
0.40 ± 6% -0.1 0.33 ± 6% perf-profile.children.cycles-pp.security_socket_post_create
0.25 ± 8% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.__skb_clone
0.32 ± 6% -0.1 0.25 ± 7% perf-profile.children.cycles-pp.tcp_mstamp_refresh
0.22 ± 4% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.__ip_local_out
0.26 ± 7% -0.1 0.20 ± 9% perf-profile.children.cycles-pp.validate_xmit_skb
0.27 ± 7% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.sk_stop_timer
0.26 ± 6% -0.1 0.20 ± 7% perf-profile.children.cycles-pp.enqueue_to_backlog
0.21 ± 8% -0.1 0.15 ± 8% perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.14 ± 7% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.migrate_enable
0.20 ± 7% -0.1 0.15 ± 8% perf-profile.children.cycles-pp.__check_object_size
0.18 ± 6% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.ip_send_check
0.24 ± 8% -0.1 0.19 ± 7% perf-profile.children.cycles-pp.__slab_free
0.24 ± 6% -0.1 0.19 ± 6% perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
0.26 ± 8% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.d_alloc_pseudo
0.21 ± 6% -0.0 0.17 ± 9% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.12 ± 8% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.sock_put
0.28 ± 6% -0.0 0.24 ± 7% perf-profile.children.cycles-pp.__switch_to
0.18 ± 7% -0.0 0.14 ± 7% perf-profile.children.cycles-pp.del_timer
0.21 ± 5% -0.0 0.17 ± 9% perf-profile.children.cycles-pp.kfree
0.15 ± 9% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.12 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.tcp_queue_rcv
0.08 ± 9% -0.0 0.05 ± 41% perf-profile.children.cycles-pp.netdev_core_pick_tx
0.13 ± 7% -0.0 0.10 ± 11% perf-profile.children.cycles-pp.netif_skb_features
0.08 ± 9% -0.0 0.05 ± 34% perf-profile.children.cycles-pp.nf_hook_slow
0.22 ± 6% -0.0 0.18 ± 9% perf-profile.children.cycles-pp.__might_sleep
0.10 ± 9% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.__destroy_inode
0.14 ± 7% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.tcp_rcv_space_adjust
0.10 ± 8% -0.0 0.07 ± 9% perf-profile.children.cycles-pp.destroy_inode
0.09 ± 8% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.inet_csk_complete_hashdance
0.14 ± 7% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.copyout
0.22 ± 7% -0.0 0.19 ± 8% perf-profile.children.cycles-pp.__update_load_avg_se
0.18 ± 6% -0.0 0.15 ± 8% perf-profile.children.cycles-pp.__fget_light
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.children.cycles-pp.simple_copy_to_iter
0.15 ± 8% -0.0 0.13 ± 7% perf-profile.children.cycles-pp.__cond_resched
0.09 ± 9% -0.0 0.06 ± 10% perf-profile.children.cycles-pp.inet_reqsk_alloc
0.13 ± 6% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.d_instantiate
0.09 ± 12% -0.0 0.06 ± 10% perf-profile.children.cycles-pp.security_sock_rcv_skb
0.08 ± 11% -0.0 0.05 ± 24% perf-profile.children.cycles-pp.migrate_disable
0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.tcp_validate_incoming
0.14 ± 8% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.tcp_wfree
0.12 ± 7% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.tcp_add_backlog
0.07 ± 10% -0.0 0.05 ± 33% perf-profile.children.cycles-pp.fsnotify_destroy_marks
0.09 ± 8% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.sock_setsockopt
0.10 ± 7% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.tcp_init_xmit_timers
0.09 ± 9% +0.0 0.13 ± 9% perf-profile.children.cycles-pp.inet_twsk_alloc
0.06 ± 8% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.tcp_mtup_init
0.23 ± 6% +0.0 0.27 ± 5% perf-profile.children.cycles-pp.tcp_make_synack
0.06 ± 9% +0.1 0.12 ± 8% perf-profile.children.cycles-pp.tcp_synack_rtt_meas
0.06 ± 10% +0.1 0.11 ± 8% perf-profile.children.cycles-pp.tcp_ca_openreq_child
0.00 +0.1 0.06 ± 10% perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
0.07 ± 9% +0.1 0.14 ± 9% perf-profile.children.cycles-pp.__inet_lookup_listener
0.12 ± 7% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.__ip_finish_output
0.06 ± 13% +0.1 0.14 ± 8% perf-profile.children.cycles-pp.__ip_dev_find
0.01 ±212% +0.1 0.09 ± 8% perf-profile.children.cycles-pp.inet_lhash2_lookup
0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.sock_prot_inuse_add
0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.tcp_push
0.07 ± 10% +0.1 0.16 ± 8% perf-profile.children.cycles-pp.tcp_select_initial_window
0.07 ± 10% +0.1 0.16 ± 7% perf-profile.children.cycles-pp.tcp_sync_mss
0.07 ± 9% +0.1 0.16 ± 8% perf-profile.children.cycles-pp.secure_tcp_ts_off
0.36 ± 6% +0.1 0.47 ± 6% perf-profile.children.cycles-pp.tcp_finish_connect
0.02 ±120% +0.1 0.13 ± 10% perf-profile.children.cycles-pp.inet_lookup_ifaddr_rcu
0.08 ± 11% +0.1 0.19 ± 8% perf-profile.children.cycles-pp.skb_entail
0.17 ± 5% +0.1 0.29 ± 7% perf-profile.children.cycles-pp.tcp_connect_init
0.10 ± 9% +0.1 0.21 ± 7% perf-profile.children.cycles-pp.tcp_openreq_init_rwin
0.00 +0.1 0.13 ± 9% perf-profile.children.cycles-pp.tcp_stream_memory_free
0.14 ± 6% +0.1 0.26 ± 7% perf-profile.children.cycles-pp.tcp_parse_options
0.13 ± 8% +0.1 0.26 ± 8% perf-profile.children.cycles-pp.tcp_update_pacing_rate
0.58 ± 5% +0.2 0.73 ± 5% perf-profile.children.cycles-pp.tcp_v4_send_synack
0.08 ± 7% +0.2 0.24 ± 6% perf-profile.children.cycles-pp.tcp_assign_congestion_control
0.34 ± 7% +0.2 0.50 ± 6% perf-profile.children.cycles-pp.tcp_v4_init_sock
0.33 ± 6% +0.2 0.49 ± 6% perf-profile.children.cycles-pp.tcp_init_sock
0.23 ± 6% +0.2 0.40 ± 6% perf-profile.children.cycles-pp.fib_table_lookup
0.00 +0.2 0.17 ± 8% perf-profile.children.cycles-pp.__sk_free
0.22 ± 7% +0.2 0.41 ± 6% perf-profile.children.cycles-pp.tcp_schedule_loss_probe
0.14 ± 7% +0.2 0.33 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish
0.63 ± 5% +0.2 0.83 ± 7% perf-profile.children.cycles-pp.tcp_init_transfer
0.10 ± 8% +0.2 0.31 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish_core
0.04 ± 40% +0.2 0.25 ± 6% perf-profile.children.cycles-pp.tcp_newly_delivered
0.07 ± 10% +0.2 0.28 ± 7% perf-profile.children.cycles-pp.fib_lookup_good_nhc
0.19 ± 7% +0.2 0.41 ± 6% perf-profile.children.cycles-pp.inet_csk_route_child_sock
0.18 ± 6% +0.2 0.40 ± 6% perf-profile.children.cycles-pp.inet_csk_route_req
0.11 ± 7% +0.2 0.34 ± 8% perf-profile.children.cycles-pp.inet_sk_rebuild_header
0.11 ± 8% +0.2 0.35 ± 7% perf-profile.children.cycles-pp.inet_sock_destruct
0.10 ± 9% +0.3 0.37 ± 8% perf-profile.children.cycles-pp.__tcp_select_window
1.69 ± 5% +0.3 2.00 ± 6% perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
0.20 ± 6% +0.3 0.52 ± 8% perf-profile.children.cycles-pp.tcp_ack_update_rtt
0.15 ± 8% +0.3 0.49 ± 7% perf-profile.children.cycles-pp.inet_ehashfn
1.91 ± 5% +0.4 2.27 ± 6% perf-profile.children.cycles-pp.tcp_check_req
0.17 ± 6% +0.4 0.54 ± 7% perf-profile.children.cycles-pp.ip_rcv_core
1.59 ± 5% +0.5 2.05 ± 6% perf-profile.children.cycles-pp.tcp_conn_request
0.43 ± 5% +0.6 0.99 ± 5% perf-profile.children.cycles-pp.ip_route_output_flow
2.02 ± 6% +0.6 2.67 ± 6% perf-profile.children.cycles-pp.tcp_ack
0.26 ± 7% +0.8 1.02 ± 7% perf-profile.children.cycles-pp.__sk_dst_check
0.60 ± 5% +0.8 1.37 ± 6% perf-profile.children.cycles-pp.ip_route_output_key_hash
0.55 ± 5% +0.8 1.33 ± 6% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
5.73 ± 5% +0.9 6.60 ± 6% perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
0.24 ± 6% +0.9 1.19 ± 7% perf-profile.children.cycles-pp.ipv4_dst_check
4.49 ± 5% +1.0 5.47 ± 6% perf-profile.children.cycles-pp.tcp_connect
5.45 ± 5% +1.3 6.72 ± 6% perf-profile.children.cycles-pp.tcp_v4_connect
11.81 ± 5% +2.0 13.85 ± 6% perf-profile.children.cycles-pp.__x64_sys_connect
11.79 ± 5% +2.1 13.84 ± 6% perf-profile.children.cycles-pp.__sys_connect
11.59 ± 5% +2.1 13.65 ± 6% perf-profile.children.cycles-pp.inet_stream_connect
11.54 ± 5% +2.1 13.61 ± 6% perf-profile.children.cycles-pp.__inet_stream_connect
4.00 ± 20% -0.9 3.08 ± 6% perf-profile.self.cycles-pp.poll_idle
1.12 ± 7% -0.5 0.63 ± 7% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
1.82 ± 9% -0.4 1.42 ± 8% perf-profile.self.cycles-pp.page_counter_cancel
1.01 ± 6% -0.3 0.72 ± 6% perf-profile.self.cycles-pp.percpu_counter_add_batch
1.65 ± 6% -0.3 1.36 ± 5% perf-profile.self.cycles-pp._raw_spin_lock
0.62 ± 6% -0.3 0.36 ± 8% perf-profile.self.cycles-pp.__tcp_get_metrics
1.09 ± 6% -0.3 0.84 ± 6% perf-profile.self.cycles-pp.__inet_lookup_established
0.47 ± 9% -0.2 0.25 ± 7% perf-profile.self.cycles-pp.__sk_destruct
0.79 ± 6% -0.2 0.57 ± 6% perf-profile.self.cycles-pp.sk_forced_mem_schedule
0.99 ± 6% -0.2 0.78 ± 7% perf-profile.self.cycles-pp._raw_spin_lock_bh
0.84 ± 6% -0.2 0.67 ± 7% perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
1.14 ± 6% -0.2 0.97 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.36 ± 6% -0.1 0.22 ± 8% perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
0.67 ± 6% -0.1 0.55 ± 6% perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.40 ± 7% -0.1 0.28 ± 7% perf-profile.self.cycles-pp.tcp_recvmsg_locked
0.58 ± 7% -0.1 0.46 ± 5% perf-profile.self.cycles-pp.apparmor_file_free_security
0.50 ± 6% -0.1 0.38 ± 7% perf-profile.self.cycles-pp.__dev_queue_xmit
0.54 ± 9% -0.1 0.43 ± 10% perf-profile.self.cycles-pp.propagate_protected_usage
0.40 ± 6% -0.1 0.30 ± 7% perf-profile.self.cycles-pp.kmem_cache_free
0.33 ± 6% -0.1 0.24 ± 7% perf-profile.self.cycles-pp.tcp_clean_rtx_queue
0.45 ± 10% -0.1 0.35 ± 10% perf-profile.self.cycles-pp.inet_csk_bind_conflict
0.28 ± 8% -0.1 0.20 ± 9% perf-profile.self.cycles-pp.ip_finish_output2
0.39 ± 6% -0.1 0.31 ± 7% perf-profile.self.cycles-pp.raw_local_deliver
0.45 ± 6% -0.1 0.37 ± 6% perf-profile.self.cycles-pp.read_tsc
0.47 ± 6% -0.1 0.39 ± 7% perf-profile.self.cycles-pp.___might_sleep
0.37 ± 7% -0.1 0.29 ± 7% perf-profile.self.cycles-pp.apparmor_sk_clone_security
0.26 ± 7% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.tcp_sendmsg_locked
0.28 ± 6% -0.1 0.21 ± 7% perf-profile.self.cycles-pp.__ksize
0.39 ± 6% -0.1 0.32 ± 7% perf-profile.self.cycles-pp.do_idle
0.39 ± 6% -0.1 0.33 ± 7% perf-profile.self.cycles-pp.apparmor_socket_post_create
0.20 ± 8% -0.1 0.14 ± 7% perf-profile.self.cycles-pp.__skb_clone
0.23 ± 8% -0.1 0.17 ± 9% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.13 ± 7% -0.1 0.07 ± 11% perf-profile.self.cycles-pp.migrate_enable
0.17 ± 6% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.ip_send_check
0.24 ± 8% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.__slab_free
0.24 ± 7% -0.1 0.19 ± 7% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.24 ± 7% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.__alloc_skb
0.17 ± 7% -0.0 0.13 ± 7% perf-profile.self.cycles-pp.net_rx_action
0.18 ± 7% -0.0 0.13 ± 10% perf-profile.self.cycles-pp.process_backlog
0.16 ± 7% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.fib_table_lookup
0.12 ± 9% -0.0 0.07 ± 8% perf-profile.self.cycles-pp.sock_put
0.27 ± 5% -0.0 0.23 ± 8% perf-profile.self.cycles-pp.__switch_to
0.14 ± 11% -0.0 0.10 ± 11% perf-profile.self.cycles-pp.do_syscall_64
0.08 ± 13% -0.0 0.04 ± 49% perf-profile.self.cycles-pp.select_task_rq_fair
0.15 ± 9% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.21 ± 5% -0.0 0.17 ± 9% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.21 ± 7% -0.0 0.17 ± 9% perf-profile.self.cycles-pp.__local_bh_enable_ip
0.20 ± 5% -0.0 0.17 ± 9% perf-profile.self.cycles-pp.kfree
0.08 ± 7% -0.0 0.04 ± 64% perf-profile.self.cycles-pp.netdev_core_pick_tx
0.15 ± 6% -0.0 0.12 ± 9% perf-profile.self.cycles-pp.__tcp_close
0.11 ± 9% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.tcp_queue_rcv
0.07 ± 9% -0.0 0.04 ± 56% perf-profile.self.cycles-pp.migrate_disable
0.07 ± 8% -0.0 0.04 ± 64% perf-profile.self.cycles-pp.inet_reqsk_alloc
0.09 ± 9% -0.0 0.05 ± 10% perf-profile.self.cycles-pp.tcp_data_queue
0.18 ± 8% -0.0 0.15 ± 6% perf-profile.self.cycles-pp.__entry_text_start
0.18 ± 6% -0.0 0.14 ± 8% perf-profile.self.cycles-pp.__fget_light
0.11 ± 9% -0.0 0.08 ± 7% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.__check_object_size
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.enqueue_to_backlog
0.14 ± 9% -0.0 0.11 ± 10% perf-profile.self.cycles-pp.sk_reset_timer
0.12 ± 9% -0.0 0.09 ± 8% perf-profile.self.cycles-pp.validate_xmit_skb
0.14 ± 9% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.sk_setup_caps
0.09 ± 7% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.irqtime_account_irq
0.11 ± 10% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.tcp_validate_incoming
0.14 ± 8% -0.0 0.11 ± 8% perf-profile.self.cycles-pp.tcp_wfree
0.07 ± 10% -0.0 0.05 ± 23% perf-profile.self.cycles-pp.tcp_rcv_synsent_state_process
0.08 ± 9% -0.0 0.06 ± 10% perf-profile.self.cycles-pp.sk_filter_trim_cap
0.08 ± 6% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.__netif_receive_skb_one_core
0.07 ± 8% -0.0 0.05 ± 23% perf-profile.self.cycles-pp.tcp_options_write
0.08 ± 9% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.netif_skb_features
0.08 ± 9% -0.0 0.06 ± 9% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.06 ± 10% +0.0 0.08 ± 8% perf-profile.self.cycles-pp.tcp_init_transfer
0.09 ± 9% +0.0 0.12 ± 9% perf-profile.self.cycles-pp.inet_twsk_alloc
0.07 ± 8% +0.0 0.10 ± 7% perf-profile.self.cycles-pp.inet_csk_get_port
0.06 ± 8% +0.0 0.09 ± 8% perf-profile.self.cycles-pp.tcp_mtup_init
0.11 ± 7% +0.0 0.16 ± 7% perf-profile.self.cycles-pp.sk_clone_lock
0.00 +0.1 0.06 ± 10% perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
0.00 +0.1 0.06 ± 12% perf-profile.self.cycles-pp.inet_twsk_kill
0.07 ± 7% +0.1 0.13 ± 8% perf-profile.self.cycles-pp.tcp_connect
0.07 ± 7% +0.1 0.14 ± 7% perf-profile.self.cycles-pp.__inet_bind
0.12 ± 7% +0.1 0.19 ± 7% perf-profile.self.cycles-pp.__ip_finish_output
0.06 ± 11% +0.1 0.14 ± 5% perf-profile.self.cycles-pp.tcp_make_synack
0.06 ± 6% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
0.01 ±251% +0.1 0.09 ± 8% perf-profile.self.cycles-pp.inet_lhash2_lookup
0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.sock_prot_inuse_add
0.00 +0.1 0.09 ± 9% perf-profile.self.cycles-pp.tcp_push
0.07 ± 10% +0.1 0.16 ± 8% perf-profile.self.cycles-pp.tcp_select_initial_window
0.06 ± 10% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.tcp_sync_mss
0.00 +0.1 0.09 ± 11% perf-profile.self.cycles-pp.tcp_v4_send_synack
0.05 ± 22% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.tcp_create_openreq_child
0.04 ± 55% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.tcp_get_metrics
0.03 ± 91% +0.1 0.13 ± 7% perf-profile.self.cycles-pp.tcp_openreq_init_rwin
0.06 ± 11% +0.1 0.16 ± 8% perf-profile.self.cycles-pp.tcp_connect_init
0.02 ±120% +0.1 0.13 ± 10% perf-profile.self.cycles-pp.inet_lookup_ifaddr_rcu
0.08 ± 15% +0.1 0.19 ± 9% perf-profile.self.cycles-pp.tcp_update_metrics
0.06 ± 13% +0.1 0.19 ± 7% perf-profile.self.cycles-pp.skb_entail
0.14 ± 6% +0.1 0.26 ± 7% perf-profile.self.cycles-pp.tcp_parse_options
0.00 +0.1 0.12 ± 8% perf-profile.self.cycles-pp.tcp_stream_memory_free
0.04 ± 55% +0.1 0.17 ± 8% perf-profile.self.cycles-pp.tcp_init_metrics
0.46 ± 6% +0.1 0.59 ± 6% perf-profile.self.cycles-pp.tcp_ack
0.11 ± 7% +0.1 0.24 ± 8% perf-profile.self.cycles-pp.tcp_event_new_data_sent
0.12 ± 8% +0.1 0.25 ± 8% perf-profile.self.cycles-pp.tcp_update_pacing_rate
0.00 ±458% +0.1 0.13 ± 8% perf-profile.self.cycles-pp.secure_tcp_ts_off
0.44 ± 4% +0.1 0.58 ± 8% perf-profile.self.cycles-pp.tcp_v4_rcv
0.00 +0.1 0.15 ± 9% perf-profile.self.cycles-pp.__sk_free
0.06 ± 8% +0.2 0.21 ± 8% perf-profile.self.cycles-pp.tcp_set_state
0.08 ± 6% +0.2 0.23 ± 6% perf-profile.self.cycles-pp.tcp_assign_congestion_control
0.02 ±146% +0.2 0.21 ± 7% perf-profile.self.cycles-pp.tcp_init_sock
0.10 ± 7% +0.2 0.30 ± 7% perf-profile.self.cycles-pp.ip_rcv_finish_core
0.16 ± 7% +0.2 0.37 ± 7% perf-profile.self.cycles-pp.tcp_schedule_loss_probe
0.07 ± 9% +0.2 0.28 ± 7% perf-profile.self.cycles-pp.fib_lookup_good_nhc
0.03 ± 83% +0.2 0.25 ± 6% perf-profile.self.cycles-pp.tcp_newly_delivered
0.10 ± 10% +0.3 0.37 ± 8% perf-profile.self.cycles-pp.__tcp_select_window
0.28 ± 6% +0.3 0.58 ± 8% perf-profile.self.cycles-pp.tcp_write_xmit
0.11 ± 6% +0.3 0.46 ± 8% perf-profile.self.cycles-pp.tcp_ack_update_rtt
0.14 ± 9% +0.3 0.49 ± 7% perf-profile.self.cycles-pp.inet_ehashfn
0.16 ± 8% +0.4 0.53 ± 7% perf-profile.self.cycles-pp.ip_rcv_core
0.75 ± 5% +0.4 1.17 ± 7% perf-profile.self.cycles-pp.__tcp_transmit_skb
0.08 ± 8% +0.5 0.56 ± 5% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
0.24 ± 5% +0.5 0.77 ± 6% perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
0.23 ± 5% +0.6 0.84 ± 7% perf-profile.self.cycles-pp.ip_output
0.24 ± 6% +0.9 1.17 ± 7% perf-profile.self.cycles-pp.ipv4_dst_check

[3]

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30212.148125000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30112.125625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30178.713125000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30206.963125000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30208.091875000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30139.37625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30163.055625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30238.72875,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30188.946874999994,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30239.25625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30241.643125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30208.288125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30232.425,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30157.838125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30220.53375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30256.23625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30214.021250000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30203.740624999995,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30209.22375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30245.735624999998
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_total_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483394.37000000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 481794.01,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482859.41000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483311.41000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483329.47000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482230.02,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482608.89,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483819.66,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483023.1499999999,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483828.1,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483866.29,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483332.61,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483718.8,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482525.41,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483528.54,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 484099.78,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483424.3400000001,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483259.8499999999,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483347.58,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483931.76999999996
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],

[4]

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp2/TCP_CRR/netperf/0xd000280

commit:
652c1b17b8 ("bpf: Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
b89fbfbb85 ("bpf: Implement minimal BPF perf link")

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
522472 -5.3% 494763 netperf.Throughput_total_tps
32654 -5.3% 30922 netperf.Throughput_tps
305.01 -0.0% 305.01 netperf.time.elapsed_time
305.01 -0.0% 305.01 netperf.time.elapsed_time.max
514102 +0.3% 515832 netperf.time.involuntary_context_switches
0.25 ±214% -20.0% 0.20 ±200% netperf.time.major_page_faults
2497 -0.0% 2497 netperf.time.maximum_resident_set_size
7473 ± 5% -0.3% 7451 ± 5% netperf.time.minor_page_faults
4096 +0.0% 4096 netperf.time.page_size
940.65 -1.0% 931.30 netperf.time.percent_of_cpu_this_job_got
2762 -0.7% 2743 netperf.time.system_time
107.63 -8.0% 99.06 netperf.time.user_time
1.601e+08 -1.5% 1.577e+08 netperf.time.voluntary_context_switches
1.567e+08 -5.3% 1.484e+08 netperf.workload
305.01 -0.0% 305.01 time.elapsed_time
305.01 -0.0% 305.01 time.elapsed_time.max
514102 +0.3% 515832 time.involuntary_context_switches
0.25 ±214% -20.0% 0.20 ±200% time.major_page_faults
2497 -0.0% 2497 time.maximum_resident_set_size
7473 ± 5% -0.3% 7451 ± 5% time.minor_page_faults
4096 +0.0% 4096 time.page_size
940.65 -1.0% 931.30 time.percent_of_cpu_this_job_got
2762 -0.7% 2743 time.system_time
107.63 -8.0% 99.06 time.user_time
1.601e+08 -1.5% 1.577e+08 time.voluntary_context_switches
>
> >
> >
> > in testcase: netperf
> > on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242
> > CPU @ 2.30GHz with 192G memory with following parameters:
> >
> > ip: ipv4
> > runtime: 300s
> > nr_threads: 16
> > cluster: cs-localhost
> > test: TCP_CRR
> > cpufreq_governor: performance
> > ucode: 0x5003006
> >
> > test-description: Netperf is a benchmark that can be use to measure various
> aspect of networking performance.
> > test-url: http://www.netperf.org/netperf/
> >
> >
> >
> [...]
> _______________________________________________
> LKP mailing list -- [email protected]
> To unsubscribe send an email to [email protected]