LinuxLists.cc - [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

2020-06-06 15:51:46

Subject: [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

Greeting,

FYI, we noticed a

commit: a5949d3faedf492fa7

If you fix the issue, Reported-by: kernel

Details are as below:
--------------------------

To reproduce:

git clone bin/lkp install job.yaml bin/lkp run job.yaml

========================== compiler/cpufreq_governor/ gcc-9/performance/1BRD_4
commit:
590b16516e ("xfs: a5949d3fae ("xfs:
590b16516ef38e2e ---------------- fail:runs %reproduction | | :4 50% %stddev %change \ | 35272 -33.6% 102.13 102.13 1388038 43420 ± 2% +13.4% 3123 +44.2% 59.31 48595108 1.44 -28.8% 0.07 ± 6% +0.4 1.44 -0.4 8632 ± 50% +75.6% 6583 ±136% +106.0% 63325 ± 11% +14.3% 8647 ± 50% +75.3% 1656 ±136% +104.6% 15831 ± 11% +14.3% 93640 ± 3% 21641 +39.9% 129269 28000 -31.2% 1269307 149.75 ± 3% 718992 231397 6.774e+08 18203372 2.569e+08 ± 18% 2691402 ± 13% 990350 520061 77231 19868 381302 4388 -2.7% 69865 21615 +40.0% 7363 -3.2% 12595 ± 3% 19619 32316 +12.3% 77231 4388 -2.7% 69865 19165 9845 ± 27% +68.0% 8195135 8179414 15720 8281098 342925 ± 2% 8132789 1.327e+08 394.00 +151.7% 394.00 +152.1% 21028 ± 6% 21520 ± 5% 4955 +234.9% 77.25 +238.5% 4955 +238.4% 77.25 +238.5% 3368 ± 2% +22.6% 3446 ± 2% +21.9% 11815 ± 8% +29.0% 11824 ± 9% +29.0% 6849 ± 5% +9.0% 6906 ± 5% +8.9% 765.00 ± 4% 765.00 ± 4% 13866 +12.5% 13905 +12.2% 1954 +24.4% 1954 +24.4% 2596 +13.7% 2596 +13.7% 4368 ± 6% -11.7% 822.25 ± 11% 822.25 ± 11% 1612 +32.1% 540.50 1622 +31.6% 540.50 6905 +20.0% 6933 +20.0% 646.25 ± 7% +195.2% 646.25 ± 7% +195.2% 1.78 ± 2% -0.3 64079048 17.99 -4.2 66630506 3.899e+08 731029 3.79 -1.7% 113320 1498 +20.4% 50450547 ± 4% 0.20 ± 3% -0.0 4587096 ± 3% 2.297e+09 86.68 -2.4 19484791 3060023 0.29 -1.5% 0.21 ± 5% -20.5% 3120 -5.8% 42.61 +1.2 24887573 31814752 36.62 +0.8 22617213 35707079 3120 -5.8% 13.96 0.99 +0.0 17.09 -4.0 1555 +19.4% 0.69 ± 4% -0.1 0.20 ± 3% -0.0 86.43 -2.5 1433 +6.7% 43.89 +0.7 6.389e+09 63452800 65977234 3.861e+08 723870 112209 49955941 ± 4% 4542094 ± 3% 2.275e+09 19294019 3030049 3089 -5.5% 24643655 31502901 22395376 35356897 3089 -5.5% 2.85e+12 84193 ± 36% +135.9% 919053 256686 ± 16% +114.7% 19497 +106.7% 21240 +101.1% 19079 +107.3% 394.52 ± 2% 965.25 ± 15% +59.7% 2866 ± 21% +57.1% 659.77 ± 15% +68.9% 84193 ± 36% +135.9% 919053 256686 ± 16% +114.7% 901765 941794 872925 18358 ± 44% +146.6% 1.22 ± 8% +159.9% 6.75 ± 23% +65.4% 1.38 ± 11% +61.4% 18357 ± 44% +146.5% 773.25 ± 3% 285624 576507 ± 2% 51029 ± 68% -83.4% 135149 ± 7% 57492 +51.0% 57495 +51.0% 57487 +51.0% 1.98 ± 4% +74.8% 57492 +51.0% 57495 +51.0% 57487 +51.0% 1.98 ± 4% +74.8% 2831 +43.0% 417519 +133.3% 427854 409942 +131.8% 4770 ± 42% +238.7% 4.70 ± 23% +37.1% 776.62 ± 19% +86.1% -472.62 289.80 ± 5% +109.7% 415893 +133.8% 423960 +135.5% 409113 +132.0% 4194 ± 50% +274.3% 110094 +125.8% 112236 +125.6% 108281 +125.8% 772.04 ± 16% +175.8% 297267 +137.2% 320112 +135.9% 272487 ± 2% +142.2% 10662 ± 10% +110.9% 12295 ± 2% +834.9% 13726 ± 2% +769.7% 10620 ± 4% +942.4% 693.29 ± 14% +228.7% 57488 +51.0% 56998 +51.5% 57977 +50.6% 72.50 ± 37% 53.25 ± 2% +84.5% 73.50 ± 26% +116.3% 204541 290268 ± 2% 204580 290342 ± 2% 204537 291605 ± 2% 204140 291988 ± 2% 204367 288632 204555 289354 ± 2% 204536 4937 ± 25% +41.6% 4937 ± 25% +41.6% 287759 204527 286950 204556 5444 ± 32% -19.9% 5444 ± 32% -19.9% 287623 204528 287734 204465 5382 ± 31% -21.8% 5382 ± 31% -21.8% 288778 204302 289794 204552 291056 ± 2% 204468 290778 204546 289974 204694 290312 204374 290266 204518 288815 ± 2% 204414 288779 ± 2% 204536 287615 ± 2% 72.50 ± 37% 204514 287014 ± 2% 53.25 ± 2% +84.5% 204331 287278 ± 2% 73.50 ± 26% +116.3% 204427 287382 ± 2% 204538 5034 ± 27% -29.4% 5034 ± 27% -29.4% 288881 ± 2% 204750 287139 ± 2% 204429 288411 ± 2% 204395 289178 ± 2% 204347 289689 ± 2% 204280 6418 ± 24% -44.5% 6418 ± 24% -44.5% 290382 ± 2% 204331 289736 ± 2% 204578 7298 -39.0% 7298 -39.0% 287187 204538 288676 204504 6301 ± 24% -29.8% 6301 ± 24% -29.8% 287510 204566 286909 204445 288608 ± 2% 204538 286219 204551 5170 ± 29% -18.5% 5170 ± 29% -18.5% 286946 204492 287384 204545 289100 204489 6393 ± 24% -32.4% 6393 ± 24% -32.4% 288835 204473 289778 204493 289485 204511 286611 204441 289245 ± 2% 204246 290006 ± 2% 204445 290524 ± 2% 204178 291625 ± 2% 204502 291615 ± 2% 9814627 13869791 25728 +47.6% 34944 +41.7% 42375 +38.5% 25348 +53.8% 30986 +46.2% 41265 +44.3% 25149 +48.3% 29752 +47.1% 40609 +41.5% 25218 +48.3% 30294 +44.6% 41371 +39.2% 24427 +48.9% 30438 +44.0% 41045 ± 3% +43.0% 24247 +47.0% 30155 +44.5% 41148 ± 2% +40.3% 24506 +46.3% 29991 +44.1% 41190 ± 2% +39.7% 24412 +46.7% 30135 +44.1% 45081 ± 15% +28.0% 25327 +47.0% 30230 +43.8% 41157 +39.6% 25136 +47.8% 29894 +44.5% 41961 ± 5% +36.5% 25431 +46.1% 30062 +44.0% 40956 ± 2% +42.1% 25496 +44.8% 29989 +46.5% 41740 ± 2% +67.8% 25414 +48.3% 30905 +43.4% 41083 +40.6% 25124 +46.8% 30001 +44.7% 41060 ± 7% +39.2% 24992 +47.6% 30109 +44.4% 40055 ± 3% +42.5% 25072 +46.2% 30296 +43.3% 24882 +48.8% 29900 +43.7% 40388 ± 3% +44.8% 25007 +48.1% 29755 +45.9% 40786 +40.6% 25031 +47.3% 29841 ± 2% +45.3% 40971 +39.8% 25145 +47.6% 29462 ± 2% +48.5% 40553 +41.8% 25184 +47.6% 29867 +45.5% 41118 +39.3% 25452 +47.1% 29714 +46.2% 41209 ± 2% +37.8% 25334 +48.8% 29894 +46.3% 41413 ± 2% +43.3% 25514 +47.0% 30016 +46.0% 41371 ± 2% +39.3% 25206 +47.4% 30167 +44.0% 41166 ± 2% +58.1% 25211 +46.8% 30004 ± 2% +45.2% 41541 ± 2% +36.9% 24432 +46.9% 30206 +43.4% 41069 +40.7% 24303 +48.0% 30238 +43.8% 41161 +38.2% 24308 +47.6% 29803 +46.5% 40809 +39.2% 24291 +47.7% 30092 +44.2% 40607 +41.5% 25605 +49.5% 29900 +44.6% 45207 ± 15% +27.4% 25555 ± 2% +49.0% 29926 +45.2% 42619 ± 8% +36.7% 27683 ± 9% +50.2% 29700 +45.6% 41779 ± 4% +43.5% 26699 ± 7% +42.1% 30061 +43.4% 42312 ± 5% +36.1% 25219 +47.7% 30363 ± 3% +45.9% 41108 ± 2% +39.6% 25413 +50.6% 29910 +46.0% 39987 ± 2% +45.9% 25569 +49.5% 29867 +44.8% 25492 +49.8% 29871 +45.4% 40076 +44.8% 25281 +48.9% 30025 +43.9% 40222 ± 3% +44.7% 25183 +49.3% 30123 +43.6% 41090 ± 6% +42.5% 25314 +49.6% 29998 +43.2% 39705 ± 3% +43.6% 25500 +46.2% 29921 +44.7% 41088 ± 5% +39.9% 25564 +48.4% 30178 +42.4% 40738 ± 4% +42.2% 26467 ± 8% +40.8% 30334 ± 2% +44.1% 42268 ± 4% +35.8% 25343 +47.0% 30100 +45.0% 41270 +43.6% 25331 +47.7% 30210 +44.7% 41028 +41.9% 25483 +47.2% 30324 ± 2% +43.4% 41027 +40.4% 25546 +47.5% 30218 +44.8% 41193 +40.1% 1212606 1448198 1990064 8.56 ± 3% -4.3 8.50 ± 3% -4.3 8.47 ± 3% -4.3 78.55 -4.1 8.74 ± 3% -4.0 77.69 -3.9 77.64 -3.9 77.13 -3.8 77.08 -3.8 76.61 -3.7 76.43 -3.6 69.22 -2.6 4.70 ± 2% -2.2 4.63 ± 2% -2.2 4.60 ± 2% -2.2 4.32 ± 2% -2.1 3.72 ± 4% -2.1 3.71 ± 4% -2.1 2.63 ± 4% -2.0 7.72 -2.0 3.08 ± 5% -1.9 2.49 ± 5% -1.9 2.48 ± 5% -1.9 2.41 ± 6% -1.8 4.90 -1.5 4.78 -1.5 2.35 -0.6 2.36 -0.6 3.40 -0.5 1.83 -0.5 3.26 -0.5 2.85 -0.4 2.79 -0.4 2.83 -0.4 1.48 -0.4 0.61 ± 2% -0.4 1.89 -0.3 1.71 -0.3 1.65 -0.3 0.69 -0.3 1.59 -0.3 0.86 -0.3 1.52 -0.3 1.75 -0.2 1.74 -0.2 1.74 -0.2 2.15 -0.2 0.79 ± 3% -0.2 1.81 -0.2 1.87 -0.2 0.70 ± 4% -0.2 0.81 -0.1 0.65 ± 2% -0.1 0.62 ± 2% -0.1 1.18 -0.0 1.01 -0.0 1.00 -0.0 0.96 -0.0 1.01 -0.0 0.59 ± 3% +0.1 0.58 ± 3% +0.1 0.53 +0.1 0.91 ± 3% +0.1 0.54 +0.1 0.57 +0.1 1.83 +0.2 0.00 +0.6 0.00 +0.6 0.00 +0.6 0.00 +0.7 0.00 +0.8 0.00 +0.8 0.00 +0.8 0.00 +0.8 0.00 +0.8 0.00 +0.8 0.00 +0.9 0.00 +0.9 0.00 +1.0 35.68 +1.0 0.00 +1.0 0.00 +1.0 9.35 +1.2 9.61 +1.2 9.63 +1.2 12.24 +1.3 12.25 +1.3 12.25 +1.3 12.49 +1.4 14.38 +1.4 13.41 +1.4 13.37 +1.5 13.34 +1.5 58.91 +1.6 0.00 +2.2 5.63 +2.5 5.65 +2.5 5.69 +2.5 7.04 +2.8 7.54 +2.9 7.55 +2.9 7.55 +2.9 0.00 +3.3 0.00 +3.3 8.56 ± 3% -4.3 8.47 ± 3% -4.3 8.50 ± 3% -4.3 78.64 -4.1 78.55 -4.1 78.50 -4.1 8.75 ± 3% -4.1 77.14 -3.8 77.09 -3.8 76.62 -3.7 76.44 -3.6 69.23 -2.6 4.70 ± 2% -2.2 4.89 ± 2% -2.1 4.85 ± 2% -2.1 3.73 ± 4% -2.1 3.72 ± 4% -2.1 4.43 ± 2% -2.1 7.73 -2.0 3.08 ± 4% -1.9 6.00 -1.8 2.50 ± 4% -1.8 2.50 ± 5% -1.8 2.42 ± 6% -1.8 2.08 ± 7% -1.6 2.08 ± 6% -1.6 2.70 ± 4% -1.0 2.36 -0.6 2.35 -0.6 1.84 -0.5 2.41 ± 4% -0.5 3.44 ± 2% -0.5 2.87 -0.5 2.83 -0.4 2.28 ± 4% -0.4 1.56 -0.4 1.89 -0.4 1.71 -0.3 0.88 -0.2 0.69 -0.2 0.67 ± 2% -0.2 1.33 ± 5% -0.2 0.51 -0.2 0.47 -0.1 0.51 ± 3% -0.1 0.57 ± 2% -0.1 0.38 -0.1 0.34 -0.1 0.33 -0.1 0.65 ± 2% -0.1 0.37 -0.1 0.31 ± 4% -0.1 0.56 ± 3% -0.1 0.26 ± 3% -0.1 0.28 ± 2% -0.1 0.26 ± 6% -0.1 0.29 -0.1 0.34 ± 2% -0.1 0.30 ± 5% -0.1 0.32 ± 2% -0.1 0.18 ± 2% -0.1 0.15 ± 2% -0.1 0.15 ± 5% -0.1 0.21 ± 5% -0.1 0.13 ± 3% -0.0 0.20 ± 2% -0.0 0.20 ± 3% -0.0 0.20 ± 5% -0.0 0.19 ± 2% -0.0 0.40 -0.0 0.17 ± 2% -0.0 0.20 ± 3% -0.0 0.18 ± 4% -0.0 0.10 -0.0 0.10 -0.0 0.06 ± 6% -0.0 0.08 ± 6% -0.0 0.26 ± 2% -0.0 0.12 ± 6% -0.0 0.10 ± 7% -0.0 0.10 ± 7% -0.0 0.17 ± 2% -0.0 0.13 ± 5% -0.0 0.10 ± 5% -0.0 0.15 ± 4% -0.0 0.11 ± 9% -0.0 0.10 ± 5% -0.0 0.09 ± 4% -0.0 0.16 ± 8% -0.0 0.12 ± 8% -0.0 0.10 ± 4% -0.0 0.14 ± 3% -0.0 0.11 ± 3% -0.0 0.23 -0.0 0.18 ± 2% -0.0 0.12 ± 4% -0.0 0.09 ± 4% -0.0 0.09 ± 4% -0.0 0.15 ± 7% -0.0 0.08 ± 8% -0.0 0.09 ± 5% -0.0 0.24 ± 4% -0.0 0.13 ± 3% -0.0 0.10 ± 4% -0.0 0.08 ± 10% -0.0 0.08 ± 10% -0.0 0.08 ± 5% -0.0 0.08 ± 8% -0.0 0.07 ± 10% -0.0 0.18 ± 2% -0.0 0.12 ± 3% -0.0 0.14 ± 3% -0.0 0.07 ± 10% -0.0 0.08 ± 5% -0.0 0.08 ± 6% -0.0 0.08 ± 5% -0.0 0.16 ± 2% +0.0 0.05 +0.0 0.10 ± 4% +0.0 0.07 ± 7% +0.0 0.16 ± 2% +0.0 0.19 ± 4% +0.0 0.56 ± 3% +0.0 0.20 ± 6% +0.0 0.07 ± 7% +0.0 0.82 ± 2% +0.0 0.31 ± 3% +0.0 0.08 ± 8% +0.0 0.08 ± 10% +0.0 0.08 ± 10% +0.0 0.08 ± 15% +0.0 3.96 +0.0 0.28 +0.0 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.16 ± 4% +0.1 0.04 ± 57% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.56 +0.1 0.03 ±100% +0.1 0.08 ± 5% +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.00 +0.1 0.60 ± 3% +0.1 0.39 ± 2% +0.1 0.00 +0.1 0.93 ± 3% +0.1 2.58 +0.1 1.52 +0.1 0.00 +0.1 4.62 +0.1 0.00 +0.1 1.83 +0.2 5.94 +0.2 0.00 +0.2 5.93 +0.2 5.88 +0.2 0.00 +0.2 5.41 +0.2 3.14 +0.2 0.00 +0.2 0.00 +0.2 0.00 +0.2 0.11 ± 3% +0.3 0.00 +0.3 0.00 +0.3 0.08 ± 14% +0.4 0.08 ± 8% +0.4 0.00 +0.4 3.51 +0.4 0.20 ± 9% +0.5 0.06 ± 11% +0.5 0.45 ± 5% +0.6 0.65 ± 3% +0.6 0.48 ± 5% +0.6 0.04 ± 57% +0.6 4.83 +0.7 0.00 +0.7 0.00 +0.8 0.04 ± 57% +0.8 0.05 ± 8% +0.8 1.50 +0.9 0.00 +0.9 24.29 +0.9 0.07 ± 5% +1.0 35.68 +1.0 9.53 +1.2 9.81 +1.3 9.82 +1.3 12.25 +1.3 12.49 +1.4 12.49 +1.4 12.48 +1.4 14.78 +1.4 58.91 +1.7 0.00 +2.2 7.05 +2.8 7.54 +2.9 7.55 +2.9 7.55 +2.9 0.10 +3.2 0.10 ± 4% +3.2 19.57 +4.0 1.20 -0.3 0.55 ± 3% -0.2 0.31 -0.1 0.31 ± 3% -0.1 0.54 ± 3% -0.1 0.23 ± 3% -0.1 0.10 ± 4% -0.1 0.18 ± 4% -0.1 0.60 ± 4% -0.1 0.77 -0.0 0.14 -0.0 0.14 ± 3% -0.0 0.17 ± 4% -0.0 0.30 ± 2% -0.0 0.18 ± 5% -0.0 0.08 ± 5% -0.0 0.12 ± 3% -0.0 0.12 ± 7% -0.0 0.18 ± 4% -0.0 0.13 -0.0 0.11 ± 4% -0.0 0.11 ± 6% -0.0 0.12 ± 3% -0.0 0.08 ± 8% -0.0 0.06 ± 6% -0.0 0.09 ± 5% -0.0 0.06 -0.0 0.18 ± 2% -0.0 0.08 ± 6% -0.0 0.08 ± 6% -0.0 0.11 ± 4% -0.0 0.07 -0.0 0.23 ± 2% -0.0 0.18 ± 4% -0.0 0.20 ± 2% -0.0 0.10 ± 5% -0.0 0.07 ± 5% -0.0 0.07 ± 6% -0.0 0.11 -0.0 0.23 -0.0 0.08 ± 6% -0.0 0.10 ± 5% -0.0 0.08 ± 5% -0.0 0.10 -0.0 0.05 ± 8% +0.0 0.16 ± 4% +0.0 0.11 ± 4% +0.0 0.15 ± 3% +0.0 0.09 ± 4% +0.0 0.10 ± 4% +0.0 0.01 ±173% +0.0 0.01 ±173% +0.0 0.45 ± 2% +0.1 0.15 ± 6% +0.1 0.00 +0.1 0.08 ± 8% +0.1 9.53 +1.2

45000 +----------------- | + + 40000 |.+.+ : + .+. +. 35000 |-+ : : | : : 30000 |-+ : : 25000 |-O :O O:O OO | : : O 20000 |-+ : : 15000 |-+ : : | : : 10000 |-+ : : 5000 |-+ : : | : : 0 +----------------- 5000 +------------------ 4500 |-+ O OO O | O OO O O O O O O 4000 |-+ 3500 |-+ | 3000 |.+.+ .+ + + 2500 |-+ : +.+ 2000 |-+ : : | : : 1500 |-+ : : 1000 |-+ : : | : : 500 |-+ : : 0 +------------------ 160 +------------------- | O OO O O O O OO O O O O 140 |-O 120 |-+ | 100 |-+ |.+.+ +. .+ + + 80 |-+ : : + | : : 60 |-+ : : 40 |-+ : : | : : 20 |-+ : : | :: 0 +------------------- 160 +------------------- | O OO O O O O OO O O O O 140 |-O 120 |-+ | 100 |-+ |.+.+ +. .+ + + 80 |-+ : : + | : : 60 |-+ : : 40 |-+ : : | : : 20 |-+ : : | :: 0 +------------------- 8e+07 +----------------- | O 7e+07 |-+ 6e+07 |-+ | 5e+07 |-+ |.+.+ .+ 4e+07 |-+ : +.+ | : : 3e+07 |-+ : : 2e+07 |-+ : : | : : 1e+07 |-+ : : | : : 0 +----------------- [*] bisect-good sample
[O] bisect-bad sample

Disclaimer:
Results have been for informational design or configuration

Thanks,
Rong Chen

-33.6% regression of aim7.jobs-per-min due to commit:
863b914da408047ab46eb0 ("xfs: force writes to delalloc regions to unwritten")
el.org/cgit/linux/kernel/git/torvalds/linux.git">https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
//sourceforge.net/projects/aimbench/files/aim-suite7/">https://sourceforge.net/projects/aimbench/files/aim-suite7/
kindly add following tag
test robot <[email protected]>
------------------------------------------------------------------------>
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
# job file is attached in this email
===============================================================
disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
8G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
refactor xfs_iomap_prealloc_size")
force writes to delalloc regions to unwritten")
a5949d3faedf492fa7863b914da
---------------------------
fail:runs
|
2:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
%stddev
\
23430 aim7.jobs-per-min
+50.5% 153.75 aim7.time.elapsed_time
+50.5% 153.75 aim7.time.elapsed_time.max
+40.2% 1945838 aim7.time.involuntary_context_switches
49255 ± 2% aim7.time.minor_page_faults
4504 ± 2% aim7.time.system_time
+6.5% 63.18 aim7.time.user_time
+58.6% 77064959 aim7.time.voluntary_context_switches
1.02 iostat.cpu.user
0.44 ± 7% mpstat.cpu.all.iowait%
1.02 mpstat.cpu.all.usr%
15156 ± 34% numa-meminfo.node0.KernelStack
13562 ± 82% numa-meminfo.node0.PageTables
72352 ± 12% numa-meminfo.node0.SUnreclaim
15156 ± 34% numa-vmstat.node0.nr_kernel_stack
3389 ± 82% numa-vmstat.node0.nr_page_table_pages
18087 ± 12% numa-vmstat.node0.nr_slab_unreclaimable
+41.2% 132211 ± 2% meminfo.AnonHugePages
30271 ± 4% meminfo.KernelStack
+12.3% 145114 meminfo.SUnreclaim
19275 meminfo.max_used_kB
-26.9% 927657 vmstat.io.bo
-17.4% 123.75 ± 4% vmstat.procs.r
+13.3% 814567 vmstat.system.cs
-9.3% 209881 ± 2% vmstat.system.in
+70.0% 1.152e+09 cpuidle.C1.time
+60.4% 29198744 cpuidle.C1.usage
+81.8% 4.672e+08 ± 5% cpuidle.C1E.time
+98.7% 5346901 ± 3% cpuidle.C1E.usage
+95.0% 1931226 ± 2% cpuidle.POLL.time
+97.7% 1028004 ± 2% cpuidle.POLL.usage
+1.8% 78602 proc-vmstat.nr_active_anon
+3.8% 20615 proc-vmstat.nr_dirty
+1.0% 384969 proc-vmstat.nr_file_pages
4270 proc-vmstat.nr_inactive_anon
+4.7% 73155 proc-vmstat.nr_inactive_file
30251 ± 4% proc-vmstat.nr_kernel_stack
7127 proc-vmstat.nr_mapped
+5.2% 13255 ± 4% proc-vmstat.nr_shmem
+3.2% 20247 proc-vmstat.nr_slab_reclaimable
36280 proc-vmstat.nr_slab_unreclaimable
+1.8% 78602 proc-vmstat.nr_zone_active_anon
4270 proc-vmstat.nr_zone_inactive_anon
+4.7% 73155 proc-vmstat.nr_zone_inactive_file
+3.4% 19824 proc-vmstat.nr_zone_write_pending
16539 ± 37% proc-vmstat.numa_hint_faults_local
+2.5% 8402537 proc-vmstat.numa_hit
+2.5% 8386209 proc-vmstat.numa_local
+3.9% 16328 proc-vmstat.numa_other
+2.4% 8477157 proc-vmstat.pgalloc_normal
+39.8% 479279 proc-vmstat.pgfault
+2.0% 8298189 proc-vmstat.pgfree
+9.7% 1.455e+08 proc-vmstat.pgpgout
991.75 ± 5% slabinfo.biovec-max.active_objs
993.25 ± 5% slabinfo.biovec-max.num_objs
-7.8% 19389 ± 5% slabinfo.filp.active_objs
-7.6% 19886 ± 5% slabinfo.filp.num_objs
16594 ± 2% slabinfo.kmalloc-256.active_objs
261.50 ± 2% slabinfo.kmalloc-256.active_slabs
16768 ± 2% slabinfo.kmalloc-256.num_objs
261.50 ± 2% slabinfo.kmalloc-256.num_slabs
4128 ± 2% slabinfo.kmalloc-2k.active_objs
4200 ± 2% slabinfo.kmalloc-2k.num_objs
15239 ± 3% slabinfo.kmalloc-512.active_objs
15249 ± 3% slabinfo.kmalloc-512.num_objs
7463 ± 2% slabinfo.kmalloc-96.active_objs
7521 ± 2% slabinfo.kmalloc-96.num_objs
+25.0% 956.25 ± 5% slabinfo.mnt_cache.active_objs
+25.0% 956.25 ± 5% slabinfo.mnt_cache.num_objs
15594 slabinfo.proc_inode_cache.active_objs
15598 slabinfo.proc_inode_cache.num_objs
2431 ± 3% slabinfo.sighand_cache.active_objs
2431 ± 3% slabinfo.sighand_cache.num_objs
2951 ± 2% slabinfo.signal_cache.active_objs
2951 ± 2% slabinfo.signal_cache.num_objs
3856 ± 2% slabinfo.skbuff_head_cache.num_objs
+21.4% 998.50 ± 3% slabinfo.task_group.active_objs
+21.4% 998.50 ± 3% slabinfo.task_group.num_objs
2129 ± 3% slabinfo.task_struct.active_objs
+31.6% 711.25 ± 3% slabinfo.task_struct.active_slabs
2134 ± 3% slabinfo.task_struct.num_objs
+31.6% 711.25 ± 3% slabinfo.task_struct.num_slabs
8289 slabinfo.vmap_area.active_objs
8322 slabinfo.vmap_area.num_objs
1907 slabinfo.xfs_buf.active_objs
1907 slabinfo.xfs_buf.num_objs
1.51 ± 2% perf-stat.i.branch-miss-rate%
+3.9% 66551536 perf-stat.i.branch-misses
13.84 perf-stat.i.cache-miss-rate%
-16.5% 55647472 perf-stat.i.cache-misses
+9.4% 4.265e+08 perf-stat.i.cache-references
+13.1% 826756 perf-stat.i.context-switches
3.73 perf-stat.i.cpi
+8.3% 122693 perf-stat.i.cpu-migrations
1803 perf-stat.i.cycles-between-cache-misses
-8.0% 46402743 ± 5% perf-stat.i.dTLB-load-misses
0.16 ± 2% perf-stat.i.dTLB-store-miss-rate%
-22.9% 3538422 ± 2% perf-stat.i.dTLB-store-misses
-5.6% 2.168e+09 perf-stat.i.dTLB-stores
84.25 perf-stat.i.iTLB-load-miss-rate%
-6.1% 18297125 perf-stat.i.iTLB-load-misses
+14.4% 3501587 ± 4% perf-stat.i.iTLB-loads
0.28 perf-stat.i.ipc
0.17 ± 6% perf-stat.i.metric.K/sec
2938 perf-stat.i.minor-faults
43.83 perf-stat.i.node-load-miss-rate%
-10.1% 22384693 perf-stat.i.node-load-misses
-12.5% 27830232 perf-stat.i.node-loads
37.45 perf-stat.i.node-store-miss-rate%
-20.3% 18015607 perf-stat.i.node-store-misses
-19.9% 28604167 perf-stat.i.node-stores
2938 perf-stat.i.page-faults
+9.3% 15.25 perf-stat.overall.MPKI
1.03 perf-stat.overall.branch-miss-rate%
13.05 perf-stat.overall.cache-miss-rate%
1856 perf-stat.overall.cycles-between-cache-misses
0.64 ± 5% perf-stat.overall.dTLB-load-miss-rate%
0.16 ± 2% perf-stat.overall.dTLB-store-miss-rate%
83.94 perf-stat.overall.iTLB-load-miss-rate%
1528 perf-stat.overall.instructions-per-iTLB-miss
44.58 perf-stat.overall.node-load-miss-rate%
+0.8% 6.438e+09 perf-stat.ps.branch-instructions
+4.2% 66110454 perf-stat.ps.branch-misses
-16.2% 55277568 perf-stat.ps.cache-misses
+9.8% 4.237e+08 perf-stat.ps.cache-references
+13.5% 821265 perf-stat.ps.context-switches
+8.6% 121881 perf-stat.ps.cpu-migrations
-7.7% 46094243 ± 5% perf-stat.ps.dTLB-load-misses
-22.6% 3514899 ± 2% perf-stat.ps.dTLB-store-misses
-5.3% 2.154e+09 perf-stat.ps.dTLB-stores
-5.8% 18175580 perf-stat.ps.iTLB-load-misses
+14.8% 3478314 ± 4% perf-stat.ps.iTLB-loads
2918 perf-stat.ps.minor-faults
-9.8% 22235787 perf-stat.ps.node-load-misses
-12.2% 27645143 perf-stat.ps.node-loads
-20.1% 17895417 perf-stat.ps.node-store-misses
-19.6% 28413366 perf-stat.ps.node-stores
2918 perf-stat.ps.page-faults
+50.6% 4.293e+12 perf-stat.total.instructions
198575 ± 25% sched_debug.cfs_rq:/.MIN_vruntime.avg
+98.2% 1821460 ± 2% sched_debug.cfs_rq:/.MIN_vruntime.max
551162 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev
40307 sched_debug.cfs_rq:/.exec_clock.avg
42717 sched_debug.cfs_rq:/.exec_clock.max
39542 sched_debug.cfs_rq:/.exec_clock.min
+54.3% 608.79 ± 13% sched_debug.cfs_rq:/.exec_clock.stddev
1541 ± 4% sched_debug.cfs_rq:/.load_avg.avg
4503 ± 10% sched_debug.cfs_rq:/.load_avg.max
1114 ± 9% sched_debug.cfs_rq:/.load_avg.stddev
198575 ± 25% sched_debug.cfs_rq:/.max_vruntime.avg
+98.2% 1821462 ± 2% sched_debug.cfs_rq:/.max_vruntime.max
551163 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev
+97.1% 1777741 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg
+95.8% 1844445 ± 2% sched_debug.cfs_rq:/.min_vruntime.max
+96.5% 1715658 ± 2% sched_debug.cfs_rq:/.min_vruntime.min
45281 ± 34% sched_debug.cfs_rq:/.min_vruntime.stddev
3.18 ± 20% sched_debug.cfs_rq:/.nr_spread_over.avg
11.17 ± 17% sched_debug.cfs_rq:/.nr_spread_over.max
2.22 ± 12% sched_debug.cfs_rq:/.nr_spread_over.stddev
45250 ± 34% sched_debug.cfs_rq:/.spread0.stddev
+13.2% 875.02 sched_debug.cfs_rq:/.util_avg.avg
-30.3% 199111 ± 4% sched_debug.cpu.avg_idle.avg
-26.2% 425271 ± 7% sched_debug.cpu.avg_idle.max
8469 ± 29% sched_debug.cpu.avg_idle.min
-23.4% 103460 ± 10% sched_debug.cpu.avg_idle.stddev
86835 sched_debug.cpu.clock.avg
86841 sched_debug.cpu.clock.max
86828 sched_debug.cpu.clock.min
3.46 ± 10% sched_debug.cpu.clock.stddev
86835 sched_debug.cpu.clock_task.avg
86841 sched_debug.cpu.clock_task.max
86828 sched_debug.cpu.clock_task.min
3.46 ± 10% sched_debug.cpu.clock_task.stddev
4049 sched_debug.cpu.curr->pid.max
974078 sched_debug.cpu.nr_switches.avg
+134.0% 1001016 sched_debug.cpu.nr_switches.max
950218 sched_debug.cpu.nr_switches.min
16156 ± 22% sched_debug.cpu.nr_switches.stddev
6.44 ± 4% sched_debug.cpu.nr_uninterruptible.avg
1445 ± 12% sched_debug.cpu.nr_uninterruptible.max
+98.3% -937.33 sched_debug.cpu.nr_uninterruptible.min
607.68 ± 3% sched_debug.cpu.nr_uninterruptible.stddev
972526 sched_debug.cpu.sched_count.avg
998457 sched_debug.cpu.sched_count.max
949068 sched_debug.cpu.sched_count.min
15698 ± 22% sched_debug.cpu.sched_count.stddev
248571 sched_debug.cpu.sched_goidle.avg
253163 sched_debug.cpu.sched_goidle.max
244506 sched_debug.cpu.sched_goidle.min
2129 ± 18% sched_debug.cpu.sched_goidle.stddev
705173 sched_debug.cpu.ttwu_count.avg
755096 sched_debug.cpu.ttwu_count.max
660099 sched_debug.cpu.ttwu_count.min
22491 ± 18% sched_debug.cpu.ttwu_count.stddev
114954 ± 2% sched_debug.cpu.ttwu_local.avg
119376 ± 2% sched_debug.cpu.ttwu_local.max
110710 sched_debug.cpu.ttwu_local.min
2278 ± 18% sched_debug.cpu.ttwu_local.stddev
86829 sched_debug.cpu_clk
86337 sched_debug.ktime
87317 sched_debug.sched_clk
+55.5% 112.75 ± 26% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
98.25 ± 22% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
159.00 ± 60% interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
+49.0% 304702 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts
+29.1% 374641 ± 4% interrupts.CPU0.RES:Rescheduling_interrupts
+48.9% 304667 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts
+28.5% 373110 ± 4% interrupts.CPU1.RES:Rescheduling_interrupts
+48.9% 304619 ± 2% interrupts.CPU10.LOC:Local_timer_interrupts
+28.3% 374190 ± 4% interrupts.CPU10.RES:Rescheduling_interrupts
+49.2% 304587 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts
+28.1% 374024 ± 4% interrupts.CPU11.RES:Rescheduling_interrupts
+49.0% 304559 ± 2% interrupts.CPU12.LOC:Local_timer_interrupts
+28.4% 370669 ± 3% interrupts.CPU12.RES:Rescheduling_interrupts
+48.9% 304669 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts
+28.5% 371782 ± 3% interrupts.CPU13.RES:Rescheduling_interrupts
+49.0% 304726 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts
6991 ± 2% interrupts.CPU14.NMI:Non-maskable_interrupts
6991 ± 2% interrupts.CPU14.PMI:Performance_monitoring_interrupts
+28.2% 368986 ± 3% interrupts.CPU14.RES:Rescheduling_interrupts
+49.0% 304707 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts
+28.6% 369104 ± 3% interrupts.CPU15.RES:Rescheduling_interrupts
+49.0% 304718 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts
4360 ± 32% interrupts.CPU16.NMI:Non-maskable_interrupts
4360 ± 32% interrupts.CPU16.PMI:Performance_monitoring_interrupts
+28.1% 368583 ± 3% interrupts.CPU16.RES:Rescheduling_interrupts
+49.0% 304728 ± 2% interrupts.CPU17.LOC:Local_timer_interrupts
+28.2% 368920 ± 3% interrupts.CPU17.RES:Rescheduling_interrupts
+49.0% 304707 ± 2% interrupts.CPU18.LOC:Local_timer_interrupts
4211 ± 27% interrupts.CPU18.NMI:Non-maskable_interrupts
4211 ± 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts
+28.1% 369871 ± 3% interrupts.CPU18.RES:Rescheduling_interrupts
+49.2% 304727 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts
+27.8% 370247 ± 3% interrupts.CPU19.RES:Rescheduling_interrupts
+49.0% 304821 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts
+28.3% 373290 ± 4% interrupts.CPU2.RES:Rescheduling_interrupts
+48.9% 304484 ± 2% interrupts.CPU20.LOC:Local_timer_interrupts
+27.6% 371106 ± 3% interrupts.CPU20.RES:Rescheduling_interrupts
+48.9% 304619 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts
+28.0% 371074 ± 3% interrupts.CPU21.RES:Rescheduling_interrupts
+48.8% 304581 ± 2% interrupts.CPU22.LOC:Local_timer_interrupts
+28.0% 371734 ± 3% interrupts.CPU22.RES:Rescheduling_interrupts
+49.0% 304493 ± 2% interrupts.CPU23.LOC:Local_timer_interrupts
+28.5% 372865 ± 3% interrupts.CPU23.RES:Rescheduling_interrupts
+49.0% 304653 ± 2% interrupts.CPU24.LOC:Local_timer_interrupts
+28.7% 371621 ± 4% interrupts.CPU24.RES:Rescheduling_interrupts
+49.0% 304640 ± 2% interrupts.CPU25.LOC:Local_timer_interrupts
+28.2% 370202 ± 4% interrupts.CPU25.RES:Rescheduling_interrupts
+49.0% 304730 ± 2% interrupts.CPU26.LOC:Local_timer_interrupts
+28.6% 369843 ± 4% interrupts.CPU26.RES:Rescheduling_interrupts
+55.5% 112.75 ± 26% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
+49.0% 304668 ± 2% interrupts.CPU27.LOC:Local_timer_interrupts
+28.5% 368825 ± 4% interrupts.CPU27.RES:Rescheduling_interrupts
98.25 ± 22% interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
+49.1% 304759 ± 2% interrupts.CPU28.LOC:Local_timer_interrupts
+28.4% 368876 ± 4% interrupts.CPU28.RES:Rescheduling_interrupts
159.00 ± 60% interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
+49.0% 304651 ± 2% interrupts.CPU29.LOC:Local_timer_interrupts
+28.9% 370434 ± 4% interrupts.CPU29.RES:Rescheduling_interrupts
+49.0% 304765 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts
3552 interrupts.CPU3.NMI:Non-maskable_interrupts
3552 interrupts.CPU3.PMI:Performance_monitoring_interrupts
+28.3% 370740 ± 4% interrupts.CPU3.RES:Rescheduling_interrupts
+48.8% 304674 ± 2% interrupts.CPU30.LOC:Local_timer_interrupts
+28.8% 369825 ± 4% interrupts.CPU30.RES:Rescheduling_interrupts
+48.9% 304416 ± 2% interrupts.CPU31.LOC:Local_timer_interrupts
+28.4% 370199 ± 4% interrupts.CPU31.RES:Rescheduling_interrupts
+49.3% 305106 ± 2% interrupts.CPU32.LOC:Local_timer_interrupts
+28.6% 371830 ± 4% interrupts.CPU32.RES:Rescheduling_interrupts
+49.2% 304799 ± 2% interrupts.CPU33.LOC:Local_timer_interrupts
+28.2% 371280 ± 4% interrupts.CPU33.RES:Rescheduling_interrupts
+49.1% 304566 ± 2% interrupts.CPU34.LOC:Local_timer_interrupts
3564 interrupts.CPU34.NMI:Non-maskable_interrupts
3564 interrupts.CPU34.PMI:Performance_monitoring_interrupts
+28.3% 372606 ± 4% interrupts.CPU34.RES:Rescheduling_interrupts
+49.0% 304517 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts
+28.3% 371869 ± 4% interrupts.CPU35.RES:Rescheduling_interrupts
+48.8% 304423 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts
4450 ± 34% interrupts.CPU36.NMI:Non-maskable_interrupts
4450 ± 34% interrupts.CPU36.PMI:Performance_monitoring_interrupts
+28.3% 368516 ± 3% interrupts.CPU36.RES:Rescheduling_interrupts
+49.0% 304729 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts
+28.1% 369792 ± 3% interrupts.CPU37.RES:Rescheduling_interrupts
+49.0% 304687 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts
4424 ± 34% interrupts.CPU38.NMI:Non-maskable_interrupts
4424 ± 34% interrupts.CPU38.PMI:Performance_monitoring_interrupts
+28.1% 368349 ± 3% interrupts.CPU38.RES:Rescheduling_interrupts
+48.9% 304668 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts
+28.1% 367622 ± 3% interrupts.CPU39.RES:Rescheduling_interrupts
+49.0% 304686 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts
+28.7% 371493 ± 3% interrupts.CPU4.RES:Rescheduling_interrupts
+48.9% 304582 ± 2% interrupts.CPU40.LOC:Local_timer_interrupts
+27.9% 365983 ± 3% interrupts.CPU40.RES:Rescheduling_interrupts
+49.0% 304739 ± 2% interrupts.CPU41.LOC:Local_timer_interrupts
4215 ± 27% interrupts.CPU41.NMI:Non-maskable_interrupts
4215 ± 27% interrupts.CPU41.PMI:Performance_monitoring_interrupts
+27.8% 366717 ± 3% interrupts.CPU41.RES:Rescheduling_interrupts
+48.9% 304442 ± 2% interrupts.CPU42.LOC:Local_timer_interrupts
+28.0% 367805 ± 3% interrupts.CPU42.RES:Rescheduling_interrupts
+49.0% 304679 ± 2% interrupts.CPU43.LOC:Local_timer_interrupts
+27.3% 368059 ± 3% interrupts.CPU43.RES:Rescheduling_interrupts
+49.0% 304635 ± 2% interrupts.CPU44.LOC:Local_timer_interrupts
4319 ± 31% interrupts.CPU44.NMI:Non-maskable_interrupts
4319 ± 31% interrupts.CPU44.PMI:Performance_monitoring_interrupts
+27.8% 369182 ± 3% interrupts.CPU44.RES:Rescheduling_interrupts
+49.2% 304983 ± 2% interrupts.CPU45.LOC:Local_timer_interrupts
+27.4% 369059 ± 3% interrupts.CPU45.RES:Rescheduling_interrupts
+49.0% 304598 ± 2% interrupts.CPU46.LOC:Local_timer_interrupts
+27.9% 370172 ± 3% interrupts.CPU46.RES:Rescheduling_interrupts
+49.2% 305090 ± 2% interrupts.CPU47.LOC:Local_timer_interrupts
+28.8% 369021 ± 3% interrupts.CPU47.RES:Rescheduling_interrupts
+49.1% 304794 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts
+28.4% 371335 ± 4% interrupts.CPU5.RES:Rescheduling_interrupts
+49.0% 304414 ± 2% interrupts.CPU6.LOC:Local_timer_interrupts
+28.3% 372180 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts
+49.0% 304719 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts
+28.2% 372414 ± 4% interrupts.CPU7.RES:Rescheduling_interrupts
+49.2% 304663 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts
+27.8% 372747 ± 4% interrupts.CPU8.RES:Rescheduling_interrupts
+48.9% 304407 ± 3% interrupts.CPU9.LOC:Local_timer_interrupts
+27.9% 373024 ± 4% interrupts.CPU9.RES:Rescheduling_interrupts
+49.0% 14624016 ± 2% interrupts.LOC:Local_timer_interrupts
+28.2% 17785831 ± 3% interrupts.RES:Rescheduling_interrupts
37969 softirqs.CPU0.RCU
49503 ± 3% softirqs.CPU0.SCHED
58673 ± 2% softirqs.CPU0.TIMER
38976 ± 5% softirqs.CPU1.RCU
45315 ± 3% softirqs.CPU1.SCHED
59538 ± 3% softirqs.CPU1.TIMER
37288 softirqs.CPU10.RCU
43764 ± 2% softirqs.CPU10.SCHED
57449 softirqs.CPU10.TIMER
37399 softirqs.CPU11.RCU
43800 softirqs.CPU11.SCHED
57578 softirqs.CPU11.TIMER
36361 softirqs.CPU12.RCU
43835 softirqs.CPU12.SCHED
58687 softirqs.CPU12.TIMER
35645 softirqs.CPU13.RCU
43577 softirqs.CPU13.SCHED
57720 softirqs.CPU13.TIMER
35856 softirqs.CPU14.RCU
43229 softirqs.CPU14.SCHED
57552 softirqs.CPU14.TIMER
35800 softirqs.CPU15.RCU
43430 softirqs.CPU15.SCHED
57684 softirqs.CPU15.TIMER
37225 softirqs.CPU16.RCU
43461 softirqs.CPU16.SCHED
57448 softirqs.CPU16.TIMER
37142 softirqs.CPU17.RCU
43212 softirqs.CPU17.SCHED
57281 softirqs.CPU17.TIMER
37160 softirqs.CPU18.RCU
43281 softirqs.CPU18.SCHED
58217 softirqs.CPU18.TIMER
36923 ± 2% softirqs.CPU19.RCU
43946 softirqs.CPU19.SCHED
70057 ± 17% softirqs.CPU19.TIMER
37678 softirqs.CPU2.RCU
44309 ± 2% softirqs.CPU2.SCHED
57768 ± 2% softirqs.CPU2.TIMER
36892 softirqs.CPU20.RCU
43416 softirqs.CPU20.SCHED
57151 softirqs.CPU20.TIMER
36896 softirqs.CPU21.RCU
43467 softirqs.CPU21.SCHED
57081 ± 2% softirqs.CPU21.TIMER
36656 softirqs.CPU22.RCU
43425 softirqs.CPU22.SCHED
37031 softirqs.CPU23.RCU
42971 softirqs.CPU23.SCHED
58494 ± 3% softirqs.CPU23.TIMER
37036 softirqs.CPU24.RCU
43399 softirqs.CPU24.SCHED
57353 softirqs.CPU24.TIMER
36875 softirqs.CPU25.RCU
43355 softirqs.CPU25.SCHED
57266 softirqs.CPU25.TIMER
37114 softirqs.CPU26.RCU
43739 ± 2% softirqs.CPU26.SCHED
57521 softirqs.CPU26.TIMER
37169 softirqs.CPU27.RCU
43450 ± 3% softirqs.CPU27.SCHED
57261 softirqs.CPU27.TIMER
37440 softirqs.CPU28.RCU
43440 ± 2% softirqs.CPU28.SCHED
56783 ± 2% softirqs.CPU28.TIMER
37696 ± 2% softirqs.CPU29.RCU
43749 softirqs.CPU29.SCHED
59346 ± 5% softirqs.CPU29.TIMER
37495 softirqs.CPU3.RCU
43831 ± 2% softirqs.CPU3.SCHED
57611 softirqs.CPU3.TIMER
37149 softirqs.CPU30.RCU
43451 ± 2% softirqs.CPU30.SCHED
65092 ± 19% softirqs.CPU30.TIMER
37016 softirqs.CPU31.RCU
43569 softirqs.CPU31.SCHED
56874 softirqs.CPU31.TIMER
35885 softirqs.CPU32.RCU
43322 ± 2% softirqs.CPU32.SCHED
57781 ± 2% softirqs.CPU32.TIMER
35973 softirqs.CPU33.RCU
43489 softirqs.CPU33.SCHED
56872 softirqs.CPU33.TIMER
35876 softirqs.CPU34.RCU
43647 ± 2% softirqs.CPU34.SCHED
56817 softirqs.CPU34.TIMER
35877 softirqs.CPU35.RCU
43386 softirqs.CPU35.SCHED
57480 ± 2% softirqs.CPU35.TIMER
38286 softirqs.CPU36.RCU
43238 softirqs.CPU36.SCHED
57602 softirqs.CPU36.TIMER
38070 softirqs.CPU37.RCU
43460 softirqs.CPU37.SCHED
58242 softirqs.CPU37.TIMER
41577 ± 4% softirqs.CPU38.RCU
43230 softirqs.CPU38.SCHED
59948 softirqs.CPU38.TIMER
37951 softirqs.CPU39.RCU
43102 softirqs.CPU39.SCHED
57581 ± 2% softirqs.CPU39.TIMER
37247 softirqs.CPU4.RCU
44295 softirqs.CPU4.SCHED
57371 ± 2% softirqs.CPU4.TIMER
38268 softirqs.CPU40.RCU
43665 softirqs.CPU40.SCHED
58322 softirqs.CPU40.TIMER
38238 softirqs.CPU41.RCU
43247 softirqs.CPU41.SCHED
38185 softirqs.CPU42.RCU
43440 softirqs.CPU42.SCHED
58020 softirqs.CPU42.TIMER
37640 softirqs.CPU43.RCU
43220 softirqs.CPU43.SCHED
58195 softirqs.CPU43.TIMER
37611 softirqs.CPU44.RCU
43249 softirqs.CPU44.SCHED
58537 softirqs.CPU44.TIMER
37863 softirqs.CPU45.RCU
42957 softirqs.CPU45.SCHED
57033 softirqs.CPU45.TIMER
37274 softirqs.CPU46.RCU
43286 softirqs.CPU46.SCHED
57463 ± 3% softirqs.CPU46.TIMER
37931 softirqs.CPU47.RCU
42980 softirqs.CPU47.SCHED
57939 ± 3% softirqs.CPU47.TIMER
37278 softirqs.CPU5.RCU
43714 softirqs.CPU5.SCHED
57394 softirqs.CPU5.TIMER
37245 softirqs.CPU6.RCU
43643 ± 2% softirqs.CPU6.SCHED
59249 ± 5% softirqs.CPU6.TIMER
37418 softirqs.CPU7.RCU
43727 ± 2% softirqs.CPU7.SCHED
58206 ± 2% softirqs.CPU7.TIMER
37514 softirqs.CPU8.RCU
43483 ± 2% softirqs.CPU8.SCHED
57591 ± 2% softirqs.CPU8.TIMER
37687 softirqs.CPU9.RCU
43771 ± 2% softirqs.CPU9.SCHED
57716 ± 2% softirqs.CPU9.TIMER
+47.7% 1790808 softirqs.RCU
+44.7% 2095500 softirqs.SCHED
+40.7% 2800202 softirqs.TIMER
4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
74.44 perf-profile.calltrace.cycles-pp.write
4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
73.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
0.63 ± 3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
5.74 ± 2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
0.55 ± 3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
0.55 ± 3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
0.61 ± 2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
3.40 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
3.30 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
1.77 ± 2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
2.87 ± 6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
1.33 ± 2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
2.75 ± 6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
1.05 ± 2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
0.26 ±100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
0.38 ± 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
0.58 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
1.26 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
1.50 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
1.49 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
1.50 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
0.60 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
0.54 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
0.55 ± 2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
0.52 ± 3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.63 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
1.02 ± 2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
0.65 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
0.69 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
0.57 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.57 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.57 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
0.70 ± 2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
0.75 ± 4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
0.80 ± 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
0.89 ± 2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
0.92 ± 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
0.97 ± 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
13.60 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
13.89 perf-profile.calltrace.cycles-pp.secondary_startup_64
15.81 ± 2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
14.86 ± 2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
14.82 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
14.80 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
2.24 ± 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
9.80 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
10.45 ± 2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
10.46 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork
10.46 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
3.30 ± 9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
3.32 ± 8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
4.25 perf-profile.children.cycles-pp.__filemap_fdatawrite_range
4.18 perf-profile.children.cycles-pp.xfs_vm_writepages
4.20 perf-profile.children.cycles-pp.do_writepages
74.51 perf-profile.children.cycles-pp.write
74.44 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
74.41 perf-profile.children.cycles-pp.do_syscall_64
4.70 perf-profile.children.cycles-pp.file_write_and_wait_range
73.37 perf-profile.children.cycles-pp.ksys_write
73.33 perf-profile.children.cycles-pp.vfs_write
72.97 perf-profile.children.cycles-pp.new_sync_write
72.81 perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
66.62 perf-profile.children.cycles-pp.xfs_file_fsync
2.50 perf-profile.children.cycles-pp.iomap_submit_ioend
2.75 perf-profile.children.cycles-pp.submit_bio
2.71 perf-profile.children.cycles-pp.generic_make_request
1.63 perf-profile.children.cycles-pp.iomap_writepages
1.63 perf-profile.children.cycles-pp.write_cache_pages
2.38 perf-profile.children.cycles-pp.brd_make_request
5.75 ± 2% perf-profile.children.cycles-pp.__flush_work
1.14 perf-profile.children.cycles-pp.iomap_writepage_map
4.20 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.70 ± 3% perf-profile.children.cycles-pp.end_page_writeback
0.70 ± 4% perf-profile.children.cycles-pp.test_clear_page_writeback
0.61 ± 2% perf-profile.children.cycles-pp.__test_set_page_writeback
0.44 ± 4% perf-profile.children.cycles-pp.sb_mark_inode_writeback
0.44 ± 5% perf-profile.children.cycles-pp.sb_clear_inode_writeback
1.67 ± 2% perf-profile.children.cycles-pp.iomap_finish_ioend
1.79 perf-profile.children.cycles-pp.iomap_file_buffered_write
1.78 perf-profile.children.cycles-pp.iomap_apply
1.33 ± 2% perf-profile.children.cycles-pp.iomap_write_actor
1.94 ± 2% perf-profile.children.cycles-pp.reschedule_interrupt
2.98 perf-profile.children.cycles-pp.sched_ttwu_pending
2.42 perf-profile.children.cycles-pp.complete
2.38 perf-profile.children.cycles-pp.swake_up_locked
1.84 ± 2% perf-profile.children.cycles-pp.scheduler_ipi
1.17 ± 2% perf-profile.children.cycles-pp.brd_do_bvec
1.54 perf-profile.children.cycles-pp.wait_for_completion
1.38 perf-profile.children.cycles-pp.schedule_timeout
0.68 ± 2% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
0.51 ± 2% perf-profile.children.cycles-pp.iomap_write_begin
0.49 perf-profile.children.cycles-pp.lseek64
1.17 ± 3% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.36 perf-profile.children.cycles-pp.grab_cache_page_write_begin
0.33 perf-profile.children.cycles-pp.pagecache_get_page
0.38 ± 2% perf-profile.children.cycles-pp.iomap_write_end
0.46 perf-profile.children.cycles-pp.blkdev_issue_flush
0.26 ± 5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
0.23 ± 6% perf-profile.children.cycles-pp.copyin
0.22 ± 6% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.56 ± 2% perf-profile.children.cycles-pp.rwsem_optimistic_spin
0.28 ± 2% perf-profile.children.cycles-pp.bio_alloc_bioset
0.22 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.48 ± 3% perf-profile.children.cycles-pp.osq_lock
0.18 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64
0.21 ± 5% perf-profile.children.cycles-pp.mempool_alloc
0.18 ± 8% perf-profile.children.cycles-pp.xlog_state_get_iclog_space
0.22 ± 3% perf-profile.children.cycles-pp.find_get_entry
0.27 ± 4% perf-profile.children.cycles-pp.generic_make_request_checks
0.23 ± 3% perf-profile.children.cycles-pp.iomap_set_page_dirty
0.26 ± 5% perf-profile.children.cycles-pp.brd_lookup_page
0.12 ± 6% perf-profile.children.cycles-pp.iomap_set_range_uptodate
0.09 ± 4% perf-profile.children.cycles-pp.__mod_lruvec_state
0.10 ± 9% perf-profile.children.cycles-pp.wake_up_q
0.15 ± 5% perf-profile.children.cycles-pp.xfs_ilock
0.08 ± 10% perf-profile.children.cycles-pp.rwsem_wake
0.15 ± 5% perf-profile.children.cycles-pp.ksys_lseek
0.15 ± 3% perf-profile.children.cycles-pp.xas_load
0.16 ± 2% perf-profile.children.cycles-pp.__set_page_dirty
0.15 ± 2% perf-profile.children.cycles-pp.xfs_iunlock
0.36 ± 4% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
0.12 ± 4% perf-profile.children.cycles-pp.clear_page_dirty_for_io
0.16 ± 4% perf-profile.children.cycles-pp.submit_bio_wait
0.15 ± 7% perf-profile.children.cycles-pp.security_file_permission
0.06 ± 6% perf-profile.children.cycles-pp.evict
0.06 ± 6% perf-profile.children.cycles-pp.truncate_inode_pages_range
0.03 ±100% perf-profile.children.cycles-pp.vfs_unlink
0.04 ± 58% perf-profile.children.cycles-pp.add_to_page_cache_lru
0.23 ± 3% perf-profile.children.cycles-pp.reweight_entity
0.09 ± 4% perf-profile.children.cycles-pp.bio_endio
0.07 ± 7% perf-profile.children.cycles-pp.do_sys_open
0.07 ± 7% perf-profile.children.cycles-pp.do_sys_openat2
0.14 perf-profile.children.cycles-pp.__pagevec_release
0.10 ± 4% perf-profile.children.cycles-pp.down_write
0.06 ± 6% perf-profile.children.cycles-pp.do_filp_open
0.12 ± 5% perf-profile.children.cycles-pp.tag_pages_for_writeback
0.08 ± 5% perf-profile.children.cycles-pp.bio_associate_blkg
0.07 ± 7% perf-profile.children.cycles-pp.open64
0.06 ± 6% perf-profile.children.cycles-pp.path_openat
0.13 ± 3% perf-profile.children.cycles-pp.pagevec_lookup_range_tag
0.10 ± 5% perf-profile.children.cycles-pp.account_page_dirtied
0.08 ± 6% perf-profile.children.cycles-pp.__fdget_pos
0.11 ± 7% perf-profile.children.cycles-pp.memset_erms
0.09 ± 4% perf-profile.children.cycles-pp.switch_fpu_return
0.21 ± 3% perf-profile.children.cycles-pp.__radix_tree_lookup
0.16 ± 9% perf-profile.children.cycles-pp.insert_work
0.10 ± 7% perf-profile.children.cycles-pp.common_file_perm
0.07 ± 6% perf-profile.children.cycles-pp.__fget_light
0.07 ± 6% perf-profile.children.cycles-pp.bvec_alloc
0.12 ± 3% perf-profile.children.cycles-pp.find_get_pages_range_tag
0.06 ± 7% perf-profile.children.cycles-pp.iov_iter_fault_in_readable
0.06 ± 13% perf-profile.children.cycles-pp.blk_throtl_bio
0.22 ± 3% perf-profile.children.cycles-pp._cond_resched
0.11 ± 4% perf-profile.children.cycles-pp.release_pages
0.08 ± 6% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
0.06 perf-profile.children.cycles-pp.unlink
0.06 ± 11% perf-profile.children.cycles-pp.__sb_start_write
0.06 ± 7% perf-profile.children.cycles-pp.generic_write_checks
0.06 perf-profile.children.cycles-pp.do_unlinkat
0.05 perf-profile.children.cycles-pp.xfs_break_layouts
0.16 ± 2% perf-profile.children.cycles-pp.__list_add_valid
0.10 ± 8% perf-profile.children.cycles-pp.__might_sleep
0.12 ± 7% perf-profile.children.cycles-pp.xas_find_marked
0.05 ± 8% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
0.06 ± 11% perf-profile.children.cycles-pp.bio_free
0.06 perf-profile.children.cycles-pp.xfs_file_llseek
0.06 ± 6% perf-profile.children.cycles-pp.xfs_file_write_iter
0.17 ± 2% perf-profile.children.cycles-pp.__switch_to_asm
0.07 ± 7% perf-profile.children.cycles-pp.cpumask_next_wrap
0.12 ± 3% perf-profile.children.cycles-pp._find_next_bit
0.09 ± 5% perf-profile.children.cycles-pp.xlog_state_done_syncing
0.19 ± 2% perf-profile.children.cycles-pp.check_preempt_curr
0.22 perf-profile.children.cycles-pp.ttwu_do_wakeup
0.60 perf-profile.children.cycles-pp.dequeue_entity
0.23 ± 4% perf-profile.children.cycles-pp.xfs_trans_committed_bulk
0.10 ± 7% perf-profile.children.cycles-pp.check_preempt_wakeup
0.86 perf-profile.children.cycles-pp.update_load_avg
0.35 perf-profile.children.cycles-pp.__unwind_start
0.12 ± 3% perf-profile.children.cycles-pp.destroy_inode
0.12 ± 3% perf-profile.children.cycles-pp.xfs_fs_destroy_inode
0.12 ± 3% perf-profile.children.cycles-pp.xfs_inactive
0.12 ± 5% perf-profile.children.cycles-pp.xfs_bmap_validate_extent
4.01 perf-profile.children.cycles-pp.stack_trace_save_tsk
0.33 perf-profile.children.cycles-pp.set_next_entity
0.05 perf-profile.children.cycles-pp.xfs_trans_roll
0.05 ± 8% perf-profile.children.cycles-pp.xfs_btree_delrec
0.05 ± 8% perf-profile.children.cycles-pp.xfs_btree_insert
0.21 ± 5% perf-profile.children.cycles-pp.memcpy_erms
0.10 ± 4% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
0.06 ± 11% perf-profile.children.cycles-pp.xfs_buf_item_format_segment
0.06 ± 11% perf-profile.children.cycles-pp.xfs_btree_delete
0.06 perf-profile.children.cycles-pp.xfs_btree_update
0.07 ± 7% perf-profile.children.cycles-pp.xfs_cil_prepare_item
0.07 ± 6% perf-profile.children.cycles-pp.xfs_bmbt_update
0.07 ± 10% perf-profile.children.cycles-pp.xfs_buf_item_format
0.64 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair
0.10 ± 4% perf-profile.children.cycles-pp.xfs_inactive_truncate
0.16 ± 11% perf-profile.children.cycles-pp.xfs_iext_lookup_extent
0.08 ± 5% perf-profile.children.cycles-pp.xfs_defer_finish
0.08 ± 5% perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
0.08 ± 5% perf-profile.children.cycles-pp.xfs_verify_agbno
0.09 ± 9% perf-profile.children.cycles-pp.xfs_defer_finish_noroll
0.70 perf-profile.children.cycles-pp.schedule_idle
0.49 perf-profile.children.cycles-pp.xlog_state_release_iclog
0.10 ± 9% perf-profile.children.cycles-pp.xfs_buf_find
1.03 ± 2% perf-profile.children.cycles-pp.xlog_write
2.69 perf-profile.children.cycles-pp.schedule
1.63 perf-profile.children.cycles-pp.__wake_up_common_lock
0.11 ± 7% perf-profile.children.cycles-pp.xfs_buf_get_map
4.73 perf-profile.children.cycles-pp.__account_scheduler_latency
0.12 ± 10% perf-profile.children.cycles-pp.xfs_buf_read_map
1.99 perf-profile.children.cycles-pp.xlog_cil_push_work
6.12 perf-profile.children.cycles-pp.ttwu_do_activate
0.17 ± 6% perf-profile.children.cycles-pp.xfs_btree_read_buf_block
6.11 perf-profile.children.cycles-pp.activate_task
6.06 perf-profile.children.cycles-pp.enqueue_task_fair
0.18 ± 5% perf-profile.children.cycles-pp.xfs_trans_read_buf_map
5.59 perf-profile.children.cycles-pp.enqueue_entity
3.33 perf-profile.children.cycles-pp.__sched_text_start
0.21 ± 6% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
0.23 ± 57% perf-profile.children.cycles-pp.xlog_grant_head_wait
0.24 ± 6% perf-profile.children.cycles-pp.io_schedule
0.38 ± 4% perf-profile.children.cycles-pp.__filemap_fdatawait_range
0.27 ± 8% perf-profile.children.cycles-pp.wait_on_page_bit
0.34 ± 5% perf-profile.children.cycles-pp.xfs_btree_lookup
0.43 perf-profile.children.cycles-pp.__queue_work
0.44 perf-profile.children.cycles-pp.queue_work_on
0.41 perf-profile.children.cycles-pp.xfs_end_bio
3.94 ± 9% perf-profile.children.cycles-pp.__xfs_trans_commit
0.72 ± 54% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
0.58 ± 67% perf-profile.children.cycles-pp.xfs_log_space_wake
1.04 ± 36% perf-profile.children.cycles-pp.xfs_log_reserve
1.25 ± 29% perf-profile.children.cycles-pp.xfs_trans_alloc
1.09 ± 34% perf-profile.children.cycles-pp.xfs_trans_reserve
0.65 ± 60% perf-profile.children.cycles-pp.xlog_grant_head_check
5.52 perf-profile.children.cycles-pp.try_to_wake_up
0.70 ± 2% perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
0.75 ± 4% perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
0.80 perf-profile.children.cycles-pp.autoremove_wake_function
0.85 perf-profile.children.cycles-pp.wake_up_page_bit
2.36 perf-profile.children.cycles-pp.__wake_up_common
0.89 ± 2% perf-profile.children.cycles-pp.xfs_bmapi_write
25.18 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.05 perf-profile.children.cycles-pp.iomap_finish_ioends
36.69 perf-profile.children.cycles-pp.xlog_cil_force_lsn
10.76 perf-profile.children.cycles-pp.intel_idle
11.10 perf-profile.children.cycles-pp.cpuidle_enter_state
11.11 perf-profile.children.cycles-pp.cpuidle_enter
13.60 perf-profile.children.cycles-pp.start_secondary
13.89 perf-profile.children.cycles-pp.secondary_startup_64
13.89 perf-profile.children.cycles-pp.cpu_startup_entry
13.88 perf-profile.children.cycles-pp.do_idle
16.20 ± 2% perf-profile.children.cycles-pp.xlog_wait_on_iclog
60.57 perf-profile.children.cycles-pp.xfs_log_force_lsn
2.24 ± 13% perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
9.81 ± 2% perf-profile.children.cycles-pp.process_one_work
10.45 ± 2% perf-profile.children.cycles-pp.worker_thread
10.46 ± 2% perf-profile.children.cycles-pp.ret_from_fork
10.46 ± 2% perf-profile.children.cycles-pp.kthread
3.30 ± 9% perf-profile.children.cycles-pp.xfs_end_ioend
3.32 ± 8% perf-profile.children.cycles-pp.xfs_end_io
23.62 perf-profile.children.cycles-pp.remove_wait_queue
0.90 ± 2% perf-profile.self.cycles-pp.brd_do_bvec
0.39 ± 4% perf-profile.self.cycles-pp.do_syscall_64
0.22 ± 5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.22 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.47 ± 3% perf-profile.self.cycles-pp.osq_lock
0.16 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64
0.04 ± 57% perf-profile.self.cycles-pp.__mod_lruvec_state
0.12 ± 5% perf-profile.self.cycles-pp.iomap_set_range_uptodate
0.54 ± 2% perf-profile.self.cycles-pp._raw_spin_lock
0.72 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.10 ± 8% perf-profile.self.cycles-pp.find_get_entry
0.11 ± 4% perf-profile.self.cycles-pp.xas_load
0.14 ± 3% perf-profile.self.cycles-pp.write
0.27 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.15 ± 7% perf-profile.self.cycles-pp.__flush_work
0.05 perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
0.09 ± 4% perf-profile.self.cycles-pp.brd_lookup_page
0.09 ± 9% perf-profile.self.cycles-pp.vfs_write
0.15 ± 7% perf-profile.self.cycles-pp.insert_work
0.10 ± 8% perf-profile.self.cycles-pp.memset_erms
0.08 perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
0.08 ± 5% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
0.09 ± 4% perf-profile.self.cycles-pp.iomap_writepage_map
0.06 ± 15% perf-profile.self.cycles-pp.bio_alloc_bioset
0.04 ± 57% perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
0.06 ± 6% perf-profile.self.cycles-pp.__fget_light
0.04 ± 57% perf-profile.self.cycles-pp.xfs_map_blocks
0.15 ± 3% perf-profile.self.cycles-pp.__list_add_valid
0.06 ± 9% perf-profile.self.cycles-pp.xfs_ilock
0.06 ± 9% perf-profile.self.cycles-pp.iov_iter_fault_in_readable
0.09 ± 4% perf-profile.self.cycles-pp.switch_fpu_return
0.05 perf-profile.self.cycles-pp.iomap_write_actor
0.21 ± 5% perf-profile.self.cycles-pp.__radix_tree_lookup
0.16 ± 6% perf-profile.self.cycles-pp.xlog_cil_force_lsn
0.18 ± 4% perf-profile.self.cycles-pp.kmem_cache_alloc
0.08 ± 11% perf-profile.self.cycles-pp.xfs_file_fsync
0.05 ± 9% perf-profile.self.cycles-pp.clear_page_dirty_for_io
0.05 perf-profile.self.cycles-pp.__test_set_page_writeback
0.09 ± 8% perf-profile.self.cycles-pp.common_file_perm
0.21 ± 4% perf-profile.self.cycles-pp.update_rq_clock
0.06 perf-profile.self.cycles-pp.xfs_file_write_iter
0.08 ± 8% perf-profile.self.cycles-pp.up_read
0.07 perf-profile.self.cycles-pp.write_cache_pages
0.09 perf-profile.self.cycles-pp.reweight_entity
0.07 ± 7% perf-profile.self.cycles-pp.dequeue_entity
0.17 ± 2% perf-profile.self.cycles-pp.__switch_to_asm
0.12 ± 3% perf-profile.self.cycles-pp.set_next_entity
0.16 ± 2% perf-profile.self.cycles-pp.orc_find
0.11 ± 6% perf-profile.self.cycles-pp.enqueue_entity
0.12 ± 3% perf-profile.self.cycles-pp._find_next_bit
0.06 ± 7% perf-profile.self.cycles-pp.check_preempt_wakeup
0.06 ± 11% perf-profile.self.cycles-pp.xfs_trans_committed_bulk
0.51 ± 2% perf-profile.self.cycles-pp.__sched_text_start
0.21 ± 3% perf-profile.self.cycles-pp.memcpy_erms
0.06 ± 6% perf-profile.self.cycles-pp.xfs_verify_agbno
0.16 ± 9% perf-profile.self.cycles-pp.xfs_iext_lookup_extent
10.76 perf-profile.self.cycles-pp.intel_idle

aim7.jobs-per-min

--------------------------------------------------+
|
.+. |
+ +.+ +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+ ++.+.|
|
|
|
O O OO O O O O |
|
|
|
|
|
|
--------------------------------------------------+

aim7.time.system_time

--------------------------------------------------+
O O O |
|
|
|
.+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+ .+.+.+. .++.+.|
+ + |
|
|
|
|
|
|
|
--------------------------------------------------+

aim7.time.elapsed_time

--------------------------------------------------+
|
|
|
|
.+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
+ |
|
|
|
|
|
|
|
--------------------------------------------------+

aim7.time.elapsed_time.max

--------------------------------------------------+
|
|
|
|
.+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
+ |
|
|
|
|
|
|
|
--------------------------------------------------+

aim7.time.voluntary_context_switches

--------------------------------------------------+
O O O OO O O |
|
|
|
+.+.+.++.+.+.+.++.+.+. .+ .+.+. .+. .+. +.+.|
+ +.+ ++.+ ++.+ +.+ |
|
|
|
|
|
|
|
--------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.

Attachments:

(No filename) (88.29 kB)
config-5.7.0-rc4-00128-ga5949d3faedf4 (205.89 kB)
job-script (7.89 kB)
job.yaml (5.59 kB)
reproduce (762.00 B)
Download all attachments

2020-06-10 03:57:55

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

Hi Darrick,

Do you have time to take a look at this? Thanks.

On 6/6/2020 11:48 PM, kernel test robot wrote:
> Greeting,
>
> FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:
>
>
> commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes to delalloc regions to unwritten")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: aim7
> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
> with following parameters:
>
> disk: 1BRD_48G
> fs: xfs
> test: sync_disk_rw
> load: 600
> cpufreq_governor: performance
> ucode: 0x42e
>
> test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
>
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <[email protected]>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
> gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
>
> commit:
> 590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
> a5949d3fae ("xfs: force writes to delalloc regions to unwritten")
>
> 590b16516ef38e2e a5949d3faedf492fa7863b914da
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :4 50% 2:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
> %stddev %change %stddev
> \ | \
> 35272 -33.6% 23430 aim7.jobs-per-min
> 102.13 +50.5% 153.75 aim7.time.elapsed_time
> 102.13 +50.5% 153.75 aim7.time.elapsed_time.max
> 1388038 +40.2% 1945838 aim7.time.involuntary_context_switches
> 43420 ? 2% +13.4% 49255 ? 2% aim7.time.minor_page_faults
> 3123 +44.2% 4504 ? 2% aim7.time.system_time
> 59.31 +6.5% 63.18 aim7.time.user_time
> 48595108 +58.6% 77064959 aim7.time.voluntary_context_switches
> 1.44 -28.8% 1.02 iostat.cpu.user
> 0.07 ? 6% +0.4 0.44 ? 7% mpstat.cpu.all.iowait%
> 1.44 -0.4 1.02 mpstat.cpu.all.usr%
> 8632 ? 50% +75.6% 15156 ? 34% numa-meminfo.node0.KernelStack
> 6583 ?136% +106.0% 13562 ? 82% numa-meminfo.node0.PageTables
> 63325 ? 11% +14.3% 72352 ? 12% numa-meminfo.node0.SUnreclaim
> 8647 ? 50% +75.3% 15156 ? 34% numa-vmstat.node0.nr_kernel_stack
> 1656 ?136% +104.6% 3389 ? 82% numa-vmstat.node0.nr_page_table_pages
> 15831 ? 11% +14.3% 18087 ? 12% numa-vmstat.node0.nr_slab_unreclaimable
> 93640 ? 3% +41.2% 132211 ? 2% meminfo.AnonHugePages
> 21641 +39.9% 30271 ? 4% meminfo.KernelStack
> 129269 +12.3% 145114 meminfo.SUnreclaim
> 28000 -31.2% 19275 meminfo.max_used_kB
> 1269307 -26.9% 927657 vmstat.io.bo
> 149.75 ? 3% -17.4% 123.75 ? 4% vmstat.procs.r
> 718992 +13.3% 814567 vmstat.system.cs
> 231397 -9.3% 209881 ? 2% vmstat.system.in
> 6.774e+08 +70.0% 1.152e+09 cpuidle.C1.time
> 18203372 +60.4% 29198744 cpuidle.C1.usage
> 2.569e+08 ? 18% +81.8% 4.672e+08 ? 5% cpuidle.C1E.time
> 2691402 ? 13% +98.7% 5346901 ? 3% cpuidle.C1E.usage
> 990350 +95.0% 1931226 ? 2% cpuidle.POLL.time
> 520061 +97.7% 1028004 ? 2% cpuidle.POLL.usage
> 77231 +1.8% 78602 proc-vmstat.nr_active_anon
> 19868 +3.8% 20615 proc-vmstat.nr_dirty
> 381302 +1.0% 384969 proc-vmstat.nr_file_pages
> 4388 -2.7% 4270 proc-vmstat.nr_inactive_anon
> 69865 +4.7% 73155 proc-vmstat.nr_inactive_file
> 21615 +40.0% 30251 ? 4% proc-vmstat.nr_kernel_stack
> 7363 -3.2% 7127 proc-vmstat.nr_mapped
> 12595 ? 3% +5.2% 13255 ? 4% proc-vmstat.nr_shmem
> 19619 +3.2% 20247 proc-vmstat.nr_slab_reclaimable
> 32316 +12.3% 36280 proc-vmstat.nr_slab_unreclaimable
> 77231 +1.8% 78602 proc-vmstat.nr_zone_active_anon
> 4388 -2.7% 4270 proc-vmstat.nr_zone_inactive_anon
> 69865 +4.7% 73155 proc-vmstat.nr_zone_inactive_file
> 19165 +3.4% 19824 proc-vmstat.nr_zone_write_pending
> 9845 ? 27% +68.0% 16539 ? 37% proc-vmstat.numa_hint_faults_local
> 8195135 +2.5% 8402537 proc-vmstat.numa_hit
> 8179414 +2.5% 8386209 proc-vmstat.numa_local
> 15720 +3.9% 16328 proc-vmstat.numa_other
> 8281098 +2.4% 8477157 proc-vmstat.pgalloc_normal
> 342925 ? 2% +39.8% 479279 proc-vmstat.pgfault
> 8132789 +2.0% 8298189 proc-vmstat.pgfree
> 1.327e+08 +9.7% 1.455e+08 proc-vmstat.pgpgout
> 394.00 +151.7% 991.75 ? 5% slabinfo.biovec-max.active_objs
> 394.00 +152.1% 993.25 ? 5% slabinfo.biovec-max.num_objs
> 21028 ? 6% -7.8% 19389 ? 5% slabinfo.filp.active_objs
> 21520 ? 5% -7.6% 19886 ? 5% slabinfo.filp.num_objs
> 4955 +234.9% 16594 ? 2% slabinfo.kmalloc-256.active_objs
> 77.25 +238.5% 261.50 ? 2% slabinfo.kmalloc-256.active_slabs
> 4955 +238.4% 16768 ? 2% slabinfo.kmalloc-256.num_objs
> 77.25 +238.5% 261.50 ? 2% slabinfo.kmalloc-256.num_slabs
> 3368 ? 2% +22.6% 4128 ? 2% slabinfo.kmalloc-2k.active_objs
> 3446 ? 2% +21.9% 4200 ? 2% slabinfo.kmalloc-2k.num_objs
> 11815 ? 8% +29.0% 15239 ? 3% slabinfo.kmalloc-512.active_objs
> 11824 ? 9% +29.0% 15249 ? 3% slabinfo.kmalloc-512.num_objs
> 6849 ? 5% +9.0% 7463 ? 2% slabinfo.kmalloc-96.active_objs
> 6906 ? 5% +8.9% 7521 ? 2% slabinfo.kmalloc-96.num_objs
> 765.00 ? 4% +25.0% 956.25 ? 5% slabinfo.mnt_cache.active_objs
> 765.00 ? 4% +25.0% 956.25 ? 5% slabinfo.mnt_cache.num_objs
> 13866 +12.5% 15594 slabinfo.proc_inode_cache.active_objs
> 13905 +12.2% 15598 slabinfo.proc_inode_cache.num_objs
> 1954 +24.4% 2431 ? 3% slabinfo.sighand_cache.active_objs
> 1954 +24.4% 2431 ? 3% slabinfo.sighand_cache.num_objs
> 2596 +13.7% 2951 ? 2% slabinfo.signal_cache.active_objs
> 2596 +13.7% 2951 ? 2% slabinfo.signal_cache.num_objs
> 4368 ? 6% -11.7% 3856 ? 2% slabinfo.skbuff_head_cache.num_objs
> 822.25 ? 11% +21.4% 998.50 ? 3% slabinfo.task_group.active_objs
> 822.25 ? 11% +21.4% 998.50 ? 3% slabinfo.task_group.num_objs
> 1612 +32.1% 2129 ? 3% slabinfo.task_struct.active_objs
> 540.50 +31.6% 711.25 ? 3% slabinfo.task_struct.active_slabs
> 1622 +31.6% 2134 ? 3% slabinfo.task_struct.num_objs
> 540.50 +31.6% 711.25 ? 3% slabinfo.task_struct.num_slabs
> 6905 +20.0% 8289 slabinfo.vmap_area.active_objs
> 6933 +20.0% 8322 slabinfo.vmap_area.num_objs
> 646.25 ? 7% +195.2% 1907 slabinfo.xfs_buf.active_objs
> 646.25 ? 7% +195.2% 1907 slabinfo.xfs_buf.num_objs
> 1.78 ? 2% -0.3 1.51 ? 2% perf-stat.i.branch-miss-rate%
> 64079048 +3.9% 66551536 perf-stat.i.branch-misses
> 17.99 -4.2 13.84 perf-stat.i.cache-miss-rate%
> 66630506 -16.5% 55647472 perf-stat.i.cache-misses
> 3.899e+08 +9.4% 4.265e+08 perf-stat.i.cache-references
> 731029 +13.1% 826756 perf-stat.i.context-switches
> 3.79 -1.7% 3.73 perf-stat.i.cpi
> 113320 +8.3% 122693 perf-stat.i.cpu-migrations
> 1498 +20.4% 1803 perf-stat.i.cycles-between-cache-misses
> 50450547 ? 4% -8.0% 46402743 ? 5% perf-stat.i.dTLB-load-misses
> 0.20 ? 3% -0.0 0.16 ? 2% perf-stat.i.dTLB-store-miss-rate%
> 4587096 ? 3% -22.9% 3538422 ? 2% perf-stat.i.dTLB-store-misses
> 2.297e+09 -5.6% 2.168e+09 perf-stat.i.dTLB-stores
> 86.68 -2.4 84.25 perf-stat.i.iTLB-load-miss-rate%
> 19484791 -6.1% 18297125 perf-stat.i.iTLB-load-misses
> 3060023 +14.4% 3501587 ? 4% perf-stat.i.iTLB-loads
> 0.29 -1.5% 0.28 perf-stat.i.ipc
> 0.21 ? 5% -20.5% 0.17 ? 6% perf-stat.i.metric.K/sec
> 3120 -5.8% 2938 perf-stat.i.minor-faults
> 42.61 +1.2 43.83 perf-stat.i.node-load-miss-rate%
> 24887573 -10.1% 22384693 perf-stat.i.node-load-misses
> 31814752 -12.5% 27830232 perf-stat.i.node-loads
> 36.62 +0.8 37.45 perf-stat.i.node-store-miss-rate%
> 22617213 -20.3% 18015607 perf-stat.i.node-store-misses
> 35707079 -19.9% 28604167 perf-stat.i.node-stores
> 3120 -5.8% 2938 perf-stat.i.page-faults
> 13.96 +9.3% 15.25 perf-stat.overall.MPKI
> 0.99 +0.0 1.03 perf-stat.overall.branch-miss-rate%
> 17.09 -4.0 13.05 perf-stat.overall.cache-miss-rate%
> 1555 +19.4% 1856 perf-stat.overall.cycles-between-cache-misses
> 0.69 ? 4% -0.1 0.64 ? 5% perf-stat.overall.dTLB-load-miss-rate%
> 0.20 ? 3% -0.0 0.16 ? 2% perf-stat.overall.dTLB-store-miss-rate%
> 86.43 -2.5 83.94 perf-stat.overall.iTLB-load-miss-rate%
> 1433 +6.7% 1528 perf-stat.overall.instructions-per-iTLB-miss
> 43.89 +0.7 44.58 perf-stat.overall.node-load-miss-rate%
> 6.389e+09 +0.8% 6.438e+09 perf-stat.ps.branch-instructions
> 63452800 +4.2% 66110454 perf-stat.ps.branch-misses
> 65977234 -16.2% 55277568 perf-stat.ps.cache-misses
> 3.861e+08 +9.8% 4.237e+08 perf-stat.ps.cache-references
> 723870 +13.5% 821265 perf-stat.ps.context-switches
> 112209 +8.6% 121881 perf-stat.ps.cpu-migrations
> 49955941 ? 4% -7.7% 46094243 ? 5% perf-stat.ps.dTLB-load-misses
> 4542094 ? 3% -22.6% 3514899 ? 2% perf-stat.ps.dTLB-store-misses
> 2.275e+09 -5.3% 2.154e+09 perf-stat.ps.dTLB-stores
> 19294019 -5.8% 18175580 perf-stat.ps.iTLB-load-misses
> 3030049 +14.8% 3478314 ? 4% perf-stat.ps.iTLB-loads
> 3089 -5.5% 2918 perf-stat.ps.minor-faults
> 24643655 -9.8% 22235787 perf-stat.ps.node-load-misses
> 31502901 -12.2% 27645143 perf-stat.ps.node-loads
> 22395376 -20.1% 17895417 perf-stat.ps.node-store-misses
> 35356897 -19.6% 28413366 perf-stat.ps.node-stores
> 3089 -5.5% 2918 perf-stat.ps.page-faults
> 2.85e+12 +50.6% 4.293e+12 perf-stat.total.instructions
> 84193 ? 36% +135.9% 198575 ? 25% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 919053 +98.2% 1821460 ? 2% sched_debug.cfs_rq:/.MIN_vruntime.max
> 256686 ? 16% +114.7% 551162 ? 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 19497 +106.7% 40307 sched_debug.cfs_rq:/.exec_clock.avg
> 21240 +101.1% 42717 sched_debug.cfs_rq:/.exec_clock.max
> 19079 +107.3% 39542 sched_debug.cfs_rq:/.exec_clock.min
> 394.52 ? 2% +54.3% 608.79 ? 13% sched_debug.cfs_rq:/.exec_clock.stddev
> 965.25 ? 15% +59.7% 1541 ? 4% sched_debug.cfs_rq:/.load_avg.avg
> 2866 ? 21% +57.1% 4503 ? 10% sched_debug.cfs_rq:/.load_avg.max
> 659.77 ? 15% +68.9% 1114 ? 9% sched_debug.cfs_rq:/.load_avg.stddev
> 84193 ? 36% +135.9% 198575 ? 25% sched_debug.cfs_rq:/.max_vruntime.avg
> 919053 +98.2% 1821462 ? 2% sched_debug.cfs_rq:/.max_vruntime.max
> 256686 ? 16% +114.7% 551163 ? 13% sched_debug.cfs_rq:/.max_vruntime.stddev
> 901765 +97.1% 1777741 ? 2% sched_debug.cfs_rq:/.min_vruntime.avg
> 941794 +95.8% 1844445 ? 2% sched_debug.cfs_rq:/.min_vruntime.max
> 872925 +96.5% 1715658 ? 2% sched_debug.cfs_rq:/.min_vruntime.min
> 18358 ? 44% +146.6% 45281 ? 34% sched_debug.cfs_rq:/.min_vruntime.stddev
> 1.22 ? 8% +159.9% 3.18 ? 20% sched_debug.cfs_rq:/.nr_spread_over.avg
> 6.75 ? 23% +65.4% 11.17 ? 17% sched_debug.cfs_rq:/.nr_spread_over.max
> 1.38 ? 11% +61.4% 2.22 ? 12% sched_debug.cfs_rq:/.nr_spread_over.stddev
> 18357 ? 44% +146.5% 45250 ? 34% sched_debug.cfs_rq:/.spread0.stddev
> 773.25 ? 3% +13.2% 875.02 sched_debug.cfs_rq:/.util_avg.avg
> 285624 -30.3% 199111 ? 4% sched_debug.cpu.avg_idle.avg
> 576507 ? 2% -26.2% 425271 ? 7% sched_debug.cpu.avg_idle.max
> 51029 ? 68% -83.4% 8469 ? 29% sched_debug.cpu.avg_idle.min
> 135149 ? 7% -23.4% 103460 ? 10% sched_debug.cpu.avg_idle.stddev
> 57492 +51.0% 86835 sched_debug.cpu.clock.avg
> 57495 +51.0% 86841 sched_debug.cpu.clock.max
> 57487 +51.0% 86828 sched_debug.cpu.clock.min
> 1.98 ? 4% +74.8% 3.46 ? 10% sched_debug.cpu.clock.stddev
> 57492 +51.0% 86835 sched_debug.cpu.clock_task.avg
> 57495 +51.0% 86841 sched_debug.cpu.clock_task.max
> 57487 +51.0% 86828 sched_debug.cpu.clock_task.min
> 1.98 ? 4% +74.8% 3.46 ? 10% sched_debug.cpu.clock_task.stddev
> 2831 +43.0% 4049 sched_debug.cpu.curr->pid.max
> 417519 +133.3% 974078 sched_debug.cpu.nr_switches.avg
> 427854 +134.0% 1001016 sched_debug.cpu.nr_switches.max
> 409942 +131.8% 950218 sched_debug.cpu.nr_switches.min
> 4770 ? 42% +238.7% 16156 ? 22% sched_debug.cpu.nr_switches.stddev
> 4.70 ? 23% +37.1% 6.44 ? 4% sched_debug.cpu.nr_uninterruptible.avg
> 776.62 ? 19% +86.1% 1445 ? 12% sched_debug.cpu.nr_uninterruptible.max
> -472.62 +98.3% -937.33 sched_debug.cpu.nr_uninterruptible.min
> 289.80 ? 5% +109.7% 607.68 ? 3% sched_debug.cpu.nr_uninterruptible.stddev
> 415893 +133.8% 972526 sched_debug.cpu.sched_count.avg
> 423960 +135.5% 998457 sched_debug.cpu.sched_count.max
> 409113 +132.0% 949068 sched_debug.cpu.sched_count.min
> 4194 ? 50% +274.3% 15698 ? 22% sched_debug.cpu.sched_count.stddev
> 110094 +125.8% 248571 sched_debug.cpu.sched_goidle.avg
> 112236 +125.6% 253163 sched_debug.cpu.sched_goidle.max
> 108281 +125.8% 244506 sched_debug.cpu.sched_goidle.min
> 772.04 ? 16% +175.8% 2129 ? 18% sched_debug.cpu.sched_goidle.stddev
> 297267 +137.2% 705173 sched_debug.cpu.ttwu_count.avg
> 320112 +135.9% 755096 sched_debug.cpu.ttwu_count.max
> 272487 ? 2% +142.2% 660099 sched_debug.cpu.ttwu_count.min
> 10662 ? 10% +110.9% 22491 ? 18% sched_debug.cpu.ttwu_count.stddev
> 12295 ? 2% +834.9% 114954 ? 2% sched_debug.cpu.ttwu_local.avg
> 13726 ? 2% +769.7% 119376 ? 2% sched_debug.cpu.ttwu_local.max
> 10620 ? 4% +942.4% 110710 sched_debug.cpu.ttwu_local.min
> 693.29 ? 14% +228.7% 2278 ? 18% sched_debug.cpu.ttwu_local.stddev
> 57488 +51.0% 86829 sched_debug.cpu_clk
> 56998 +51.5% 86337 sched_debug.ktime
> 57977 +50.6% 87317 sched_debug.sched_clk
> 72.50 ? 37% +55.5% 112.75 ? 26% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 53.25 ? 2% +84.5% 98.25 ? 22% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
> 73.50 ? 26% +116.3% 159.00 ? 60% interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
> 204541 +49.0% 304702 ? 2% interrupts.CPU0.LOC:Local_timer_interrupts
> 290268 ? 2% +29.1% 374641 ? 4% interrupts.CPU0.RES:Rescheduling_interrupts
> 204580 +48.9% 304667 ? 2% interrupts.CPU1.LOC:Local_timer_interrupts
> 290342 ? 2% +28.5% 373110 ? 4% interrupts.CPU1.RES:Rescheduling_interrupts
> 204537 +48.9% 304619 ? 2% interrupts.CPU10.LOC:Local_timer_interrupts
> 291605 ? 2% +28.3% 374190 ? 4% interrupts.CPU10.RES:Rescheduling_interrupts
> 204140 +49.2% 304587 ? 2% interrupts.CPU11.LOC:Local_timer_interrupts
> 291988 ? 2% +28.1% 374024 ? 4% interrupts.CPU11.RES:Rescheduling_interrupts
> 204367 +49.0% 304559 ? 2% interrupts.CPU12.LOC:Local_timer_interrupts
> 288632 +28.4% 370669 ? 3% interrupts.CPU12.RES:Rescheduling_interrupts
> 204555 +48.9% 304669 ? 2% interrupts.CPU13.LOC:Local_timer_interrupts
> 289354 ? 2% +28.5% 371782 ? 3% interrupts.CPU13.RES:Rescheduling_interrupts
> 204536 +49.0% 304726 ? 2% interrupts.CPU14.LOC:Local_timer_interrupts
> 4937 ? 25% +41.6% 6991 ? 2% interrupts.CPU14.NMI:Non-maskable_interrupts
> 4937 ? 25% +41.6% 6991 ? 2% interrupts.CPU14.PMI:Performance_monitoring_interrupts
> 287759 +28.2% 368986 ? 3% interrupts.CPU14.RES:Rescheduling_interrupts
> 204527 +49.0% 304707 ? 2% interrupts.CPU15.LOC:Local_timer_interrupts
> 286950 +28.6% 369104 ? 3% interrupts.CPU15.RES:Rescheduling_interrupts
> 204556 +49.0% 304718 ? 2% interrupts.CPU16.LOC:Local_timer_interrupts
> 5444 ? 32% -19.9% 4360 ? 32% interrupts.CPU16.NMI:Non-maskable_interrupts
> 5444 ? 32% -19.9% 4360 ? 32% interrupts.CPU16.PMI:Performance_monitoring_interrupts
> 287623 +28.1% 368583 ? 3% interrupts.CPU16.RES:Rescheduling_interrupts
> 204528 +49.0% 304728 ? 2% interrupts.CPU17.LOC:Local_timer_interrupts
> 287734 +28.2% 368920 ? 3% interrupts.CPU17.RES:Rescheduling_interrupts
> 204465 +49.0% 304707 ? 2% interrupts.CPU18.LOC:Local_timer_interrupts
> 5382 ? 31% -21.8% 4211 ? 27% interrupts.CPU18.NMI:Non-maskable_interrupts
> 5382 ? 31% -21.8% 4211 ? 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts
> 288778 +28.1% 369871 ? 3% interrupts.CPU18.RES:Rescheduling_interrupts
> 204302 +49.2% 304727 ? 2% interrupts.CPU19.LOC:Local_timer_interrupts
> 289794 +27.8% 370247 ? 3% interrupts.CPU19.RES:Rescheduling_interrupts
> 204552 +49.0% 304821 ? 2% interrupts.CPU2.LOC:Local_timer_interrupts
> 291056 ? 2% +28.3% 373290 ? 4% interrupts.CPU2.RES:Rescheduling_interrupts
> 204468 +48.9% 304484 ? 2% interrupts.CPU20.LOC:Local_timer_interrupts
> 290778 +27.6% 371106 ? 3% interrupts.CPU20.RES:Rescheduling_interrupts
> 204546 +48.9% 304619 ? 2% interrupts.CPU21.LOC:Local_timer_interrupts
> 289974 +28.0% 371074 ? 3% interrupts.CPU21.RES:Rescheduling_interrupts
> 204694 +48.8% 304581 ? 2% interrupts.CPU22.LOC:Local_timer_interrupts
> 290312 +28.0% 371734 ? 3% interrupts.CPU22.RES:Rescheduling_interrupts
> 204374 +49.0% 304493 ? 2% interrupts.CPU23.LOC:Local_timer_interrupts
> 290266 +28.5% 372865 ? 3% interrupts.CPU23.RES:Rescheduling_interrupts
> 204518 +49.0% 304653 ? 2% interrupts.CPU24.LOC:Local_timer_interrupts
> 288815 ? 2% +28.7% 371621 ? 4% interrupts.CPU24.RES:Rescheduling_interrupts
> 204414 +49.0% 304640 ? 2% interrupts.CPU25.LOC:Local_timer_interrupts
> 288779 ? 2% +28.2% 370202 ? 4% interrupts.CPU25.RES:Rescheduling_interrupts
> 204536 +49.0% 304730 ? 2% interrupts.CPU26.LOC:Local_timer_interrupts
> 287615 ? 2% +28.6% 369843 ? 4% interrupts.CPU26.RES:Rescheduling_interrupts
> 72.50 ? 37% +55.5% 112.75 ? 26% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> 204514 +49.0% 304668 ? 2% interrupts.CPU27.LOC:Local_timer_interrupts
> 287014 ? 2% +28.5% 368825 ? 4% interrupts.CPU27.RES:Rescheduling_interrupts
> 53.25 ? 2% +84.5% 98.25 ? 22% interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
> 204331 +49.1% 304759 ? 2% interrupts.CPU28.LOC:Local_timer_interrupts
> 287278 ? 2% +28.4% 368876 ? 4% interrupts.CPU28.RES:Rescheduling_interrupts
> 73.50 ? 26% +116.3% 159.00 ? 60% interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
> 204427 +49.0% 304651 ? 2% interrupts.CPU29.LOC:Local_timer_interrupts
> 287382 ? 2% +28.9% 370434 ? 4% interrupts.CPU29.RES:Rescheduling_interrupts
> 204538 +49.0% 304765 ? 2% interrupts.CPU3.LOC:Local_timer_interrupts
> 5034 ? 27% -29.4% 3552 interrupts.CPU3.NMI:Non-maskable_interrupts
> 5034 ? 27% -29.4% 3552 interrupts.CPU3.PMI:Performance_monitoring_interrupts
> 288881 ? 2% +28.3% 370740 ? 4% interrupts.CPU3.RES:Rescheduling_interrupts
> 204750 +48.8% 304674 ? 2% interrupts.CPU30.LOC:Local_timer_interrupts
> 287139 ? 2% +28.8% 369825 ? 4% interrupts.CPU30.RES:Rescheduling_interrupts
> 204429 +48.9% 304416 ? 2% interrupts.CPU31.LOC:Local_timer_interrupts
> 288411 ? 2% +28.4% 370199 ? 4% interrupts.CPU31.RES:Rescheduling_interrupts
> 204395 +49.3% 305106 ? 2% interrupts.CPU32.LOC:Local_timer_interrupts
> 289178 ? 2% +28.6% 371830 ? 4% interrupts.CPU32.RES:Rescheduling_interrupts
> 204347 +49.2% 304799 ? 2% interrupts.CPU33.LOC:Local_timer_interrupts
> 289689 ? 2% +28.2% 371280 ? 4% interrupts.CPU33.RES:Rescheduling_interrupts
> 204280 +49.1% 304566 ? 2% interrupts.CPU34.LOC:Local_timer_interrupts
> 6418 ? 24% -44.5% 3564 interrupts.CPU34.NMI:Non-maskable_interrupts
> 6418 ? 24% -44.5% 3564 interrupts.CPU34.PMI:Performance_monitoring_interrupts
> 290382 ? 2% +28.3% 372606 ? 4% interrupts.CPU34.RES:Rescheduling_interrupts
> 204331 +49.0% 304517 ? 2% interrupts.CPU35.LOC:Local_timer_interrupts
> 289736 ? 2% +28.3% 371869 ? 4% interrupts.CPU35.RES:Rescheduling_interrupts
> 204578 +48.8% 304423 ? 2% interrupts.CPU36.LOC:Local_timer_interrupts
> 7298 -39.0% 4450 ? 34% interrupts.CPU36.NMI:Non-maskable_interrupts
> 7298 -39.0% 4450 ? 34% interrupts.CPU36.PMI:Performance_monitoring_interrupts
> 287187 +28.3% 368516 ? 3% interrupts.CPU36.RES:Rescheduling_interrupts
> 204538 +49.0% 304729 ? 2% interrupts.CPU37.LOC:Local_timer_interrupts
> 288676 +28.1% 369792 ? 3% interrupts.CPU37.RES:Rescheduling_interrupts
> 204504 +49.0% 304687 ? 2% interrupts.CPU38.LOC:Local_timer_interrupts
> 6301 ? 24% -29.8% 4424 ? 34% interrupts.CPU38.NMI:Non-maskable_interrupts
> 6301 ? 24% -29.8% 4424 ? 34% interrupts.CPU38.PMI:Performance_monitoring_interrupts
> 287510 +28.1% 368349 ? 3% interrupts.CPU38.RES:Rescheduling_interrupts
> 204566 +48.9% 304668 ? 2% interrupts.CPU39.LOC:Local_timer_interrupts
> 286909 +28.1% 367622 ? 3% interrupts.CPU39.RES:Rescheduling_interrupts
> 204445 +49.0% 304686 ? 2% interrupts.CPU4.LOC:Local_timer_interrupts
> 288608 ? 2% +28.7% 371493 ? 3% interrupts.CPU4.RES:Rescheduling_interrupts
> 204538 +48.9% 304582 ? 2% interrupts.CPU40.LOC:Local_timer_interrupts
> 286219 +27.9% 365983 ? 3% interrupts.CPU40.RES:Rescheduling_interrupts
> 204551 +49.0% 304739 ? 2% interrupts.CPU41.LOC:Local_timer_interrupts
> 5170 ? 29% -18.5% 4215 ? 27% interrupts.CPU41.NMI:Non-maskable_interrupts
> 5170 ? 29% -18.5% 4215 ? 27% interrupts.CPU41.PMI:Performance_monitoring_interrupts
> 286946 +27.8% 366717 ? 3% interrupts.CPU41.RES:Rescheduling_interrupts
> 204492 +48.9% 304442 ? 2% interrupts.CPU42.LOC:Local_timer_interrupts
> 287384 +28.0% 367805 ? 3% interrupts.CPU42.RES:Rescheduling_interrupts
> 204545 +49.0% 304679 ? 2% interrupts.CPU43.LOC:Local_timer_interrupts
> 289100 +27.3% 368059 ? 3% interrupts.CPU43.RES:Rescheduling_interrupts
> 204489 +49.0% 304635 ? 2% interrupts.CPU44.LOC:Local_timer_interrupts
> 6393 ? 24% -32.4% 4319 ? 31% interrupts.CPU44.NMI:Non-maskable_interrupts
> 6393 ? 24% -32.4% 4319 ? 31% interrupts.CPU44.PMI:Performance_monitoring_interrupts
> 288835 +27.8% 369182 ? 3% interrupts.CPU44.RES:Rescheduling_interrupts
> 204473 +49.2% 304983 ? 2% interrupts.CPU45.LOC:Local_timer_interrupts
> 289778 +27.4% 369059 ? 3% interrupts.CPU45.RES:Rescheduling_interrupts
> 204493 +49.0% 304598 ? 2% interrupts.CPU46.LOC:Local_timer_interrupts
> 289485 +27.9% 370172 ? 3% interrupts.CPU46.RES:Rescheduling_interrupts
> 204511 +49.2% 305090 ? 2% interrupts.CPU47.LOC:Local_timer_interrupts
> 286611 +28.8% 369021 ? 3% interrupts.CPU47.RES:Rescheduling_interrupts
> 204441 +49.1% 304794 ? 2% interrupts.CPU5.LOC:Local_timer_interrupts
> 289245 ? 2% +28.4% 371335 ? 4% interrupts.CPU5.RES:Rescheduling_interrupts
> 204246 +49.0% 304414 ? 2% interrupts.CPU6.LOC:Local_timer_interrupts
> 290006 ? 2% +28.3% 372180 ? 4% interrupts.CPU6.RES:Rescheduling_interrupts
> 204445 +49.0% 304719 ? 2% interrupts.CPU7.LOC:Local_timer_interrupts
> 290524 ? 2% +28.2% 372414 ? 4% interrupts.CPU7.RES:Rescheduling_interrupts
> 204178 +49.2% 304663 ? 2% interrupts.CPU8.LOC:Local_timer_interrupts
> 291625 ? 2% +27.8% 372747 ? 4% interrupts.CPU8.RES:Rescheduling_interrupts
> 204502 +48.9% 304407 ? 3% interrupts.CPU9.LOC:Local_timer_interrupts
> 291615 ? 2% +27.9% 373024 ? 4% interrupts.CPU9.RES:Rescheduling_interrupts
> 9814627 +49.0% 14624016 ? 2% interrupts.LOC:Local_timer_interrupts
> 13869791 +28.2% 17785831 ? 3% interrupts.RES:Rescheduling_interrupts
> 25728 +47.6% 37969 softirqs.CPU0.RCU
> 34944 +41.7% 49503 ? 3% softirqs.CPU0.SCHED
> 42375 +38.5% 58673 ? 2% softirqs.CPU0.TIMER
> 25348 +53.8% 38976 ? 5% softirqs.CPU1.RCU
> 30986 +46.2% 45315 ? 3% softirqs.CPU1.SCHED
> 41265 +44.3% 59538 ? 3% softirqs.CPU1.TIMER
> 25149 +48.3% 37288 softirqs.CPU10.RCU
> 29752 +47.1% 43764 ? 2% softirqs.CPU10.SCHED
> 40609 +41.5% 57449 softirqs.CPU10.TIMER
> 25218 +48.3% 37399 softirqs.CPU11.RCU
> 30294 +44.6% 43800 softirqs.CPU11.SCHED
> 41371 +39.2% 57578 softirqs.CPU11.TIMER
> 24427 +48.9% 36361 softirqs.CPU12.RCU
> 30438 +44.0% 43835 softirqs.CPU12.SCHED
> 41045 ? 3% +43.0% 58687 softirqs.CPU12.TIMER
> 24247 +47.0% 35645 softirqs.CPU13.RCU
> 30155 +44.5% 43577 softirqs.CPU13.SCHED
> 41148 ? 2% +40.3% 57720 softirqs.CPU13.TIMER
> 24506 +46.3% 35856 softirqs.CPU14.RCU
> 29991 +44.1% 43229 softirqs.CPU14.SCHED
> 41190 ? 2% +39.7% 57552 softirqs.CPU14.TIMER
> 24412 +46.7% 35800 softirqs.CPU15.RCU
> 30135 +44.1% 43430 softirqs.CPU15.SCHED
> 45081 ? 15% +28.0% 57684 softirqs.CPU15.TIMER
> 25327 +47.0% 37225 softirqs.CPU16.RCU
> 30230 +43.8% 43461 softirqs.CPU16.SCHED
> 41157 +39.6% 57448 softirqs.CPU16.TIMER
> 25136 +47.8% 37142 softirqs.CPU17.RCU
> 29894 +44.5% 43212 softirqs.CPU17.SCHED
> 41961 ? 5% +36.5% 57281 softirqs.CPU17.TIMER
> 25431 +46.1% 37160 softirqs.CPU18.RCU
> 30062 +44.0% 43281 softirqs.CPU18.SCHED
> 40956 ? 2% +42.1% 58217 softirqs.CPU18.TIMER
> 25496 +44.8% 36923 ? 2% softirqs.CPU19.RCU
> 29989 +46.5% 43946 softirqs.CPU19.SCHED
> 41740 ? 2% +67.8% 70057 ? 17% softirqs.CPU19.TIMER
> 25414 +48.3% 37678 softirqs.CPU2.RCU
> 30905 +43.4% 44309 ? 2% softirqs.CPU2.SCHED
> 41083 +40.6% 57768 ? 2% softirqs.CPU2.TIMER
> 25124 +46.8% 36892 softirqs.CPU20.RCU
> 30001 +44.7% 43416 softirqs.CPU20.SCHED
> 41060 ? 7% +39.2% 57151 softirqs.CPU20.TIMER
> 24992 +47.6% 36896 softirqs.CPU21.RCU
> 30109 +44.4% 43467 softirqs.CPU21.SCHED
> 40055 ? 3% +42.5% 57081 ? 2% softirqs.CPU21.TIMER
> 25072 +46.2% 36656 softirqs.CPU22.RCU
> 30296 +43.3% 43425 softirqs.CPU22.SCHED
> 24882 +48.8% 37031 softirqs.CPU23.RCU
> 29900 +43.7% 42971 softirqs.CPU23.SCHED
> 40388 ? 3% +44.8% 58494 ? 3% softirqs.CPU23.TIMER
> 25007 +48.1% 37036 softirqs.CPU24.RCU
> 29755 +45.9% 43399 softirqs.CPU24.SCHED
> 40786 +40.6% 57353 softirqs.CPU24.TIMER
> 25031 +47.3% 36875 softirqs.CPU25.RCU
> 29841 ? 2% +45.3% 43355 softirqs.CPU25.SCHED
> 40971 +39.8% 57266 softirqs.CPU25.TIMER
> 25145 +47.6% 37114 softirqs.CPU26.RCU
> 29462 ? 2% +48.5% 43739 ? 2% softirqs.CPU26.SCHED
> 40553 +41.8% 57521 softirqs.CPU26.TIMER
> 25184 +47.6% 37169 softirqs.CPU27.RCU
> 29867 +45.5% 43450 ? 3% softirqs.CPU27.SCHED
> 41118 +39.3% 57261 softirqs.CPU27.TIMER
> 25452 +47.1% 37440 softirqs.CPU28.RCU
> 29714 +46.2% 43440 ? 2% softirqs.CPU28.SCHED
> 41209 ? 2% +37.8% 56783 ? 2% softirqs.CPU28.TIMER
> 25334 +48.8% 37696 ? 2% softirqs.CPU29.RCU
> 29894 +46.3% 43749 softirqs.CPU29.SCHED
> 41413 ? 2% +43.3% 59346 ? 5% softirqs.CPU29.TIMER
> 25514 +47.0% 37495 softirqs.CPU3.RCU
> 30016 +46.0% 43831 ? 2% softirqs.CPU3.SCHED
> 41371 ? 2% +39.3% 57611 softirqs.CPU3.TIMER
> 25206 +47.4% 37149 softirqs.CPU30.RCU
> 30167 +44.0% 43451 ? 2% softirqs.CPU30.SCHED
> 41166 ? 2% +58.1% 65092 ? 19% softirqs.CPU30.TIMER
> 25211 +46.8% 37016 softirqs.CPU31.RCU
> 30004 ? 2% +45.2% 43569 softirqs.CPU31.SCHED
> 41541 ? 2% +36.9% 56874 softirqs.CPU31.TIMER
> 24432 +46.9% 35885 softirqs.CPU32.RCU
> 30206 +43.4% 43322 ? 2% softirqs.CPU32.SCHED
> 41069 +40.7% 57781 ? 2% softirqs.CPU32.TIMER
> 24303 +48.0% 35973 softirqs.CPU33.RCU
> 30238 +43.8% 43489 softirqs.CPU33.SCHED
> 41161 +38.2% 56872 softirqs.CPU33.TIMER
> 24308 +47.6% 35876 softirqs.CPU34.RCU
> 29803 +46.5% 43647 ? 2% softirqs.CPU34.SCHED
> 40809 +39.2% 56817 softirqs.CPU34.TIMER
> 24291 +47.7% 35877 softirqs.CPU35.RCU
> 30092 +44.2% 43386 softirqs.CPU35.SCHED
> 40607 +41.5% 57480 ? 2% softirqs.CPU35.TIMER
> 25605 +49.5% 38286 softirqs.CPU36.RCU
> 29900 +44.6% 43238 softirqs.CPU36.SCHED
> 45207 ? 15% +27.4% 57602 softirqs.CPU36.TIMER
> 25555 ? 2% +49.0% 38070 softirqs.CPU37.RCU
> 29926 +45.2% 43460 softirqs.CPU37.SCHED
> 42619 ? 8% +36.7% 58242 softirqs.CPU37.TIMER
> 27683 ? 9% +50.2% 41577 ? 4% softirqs.CPU38.RCU
> 29700 +45.6% 43230 softirqs.CPU38.SCHED
> 41779 ? 4% +43.5% 59948 softirqs.CPU38.TIMER
> 26699 ? 7% +42.1% 37951 softirqs.CPU39.RCU
> 30061 +43.4% 43102 softirqs.CPU39.SCHED
> 42312 ? 5% +36.1% 57581 ? 2% softirqs.CPU39.TIMER
> 25219 +47.7% 37247 softirqs.CPU4.RCU
> 30363 ? 3% +45.9% 44295 softirqs.CPU4.SCHED
> 41108 ? 2% +39.6% 57371 ? 2% softirqs.CPU4.TIMER
> 25413 +50.6% 38268 softirqs.CPU40.RCU
> 29910 +46.0% 43665 softirqs.CPU40.SCHED
> 39987 ? 2% +45.9% 58322 softirqs.CPU40.TIMER
> 25569 +49.5% 38238 softirqs.CPU41.RCU
> 29867 +44.8% 43247 softirqs.CPU41.SCHED
> 25492 +49.8% 38185 softirqs.CPU42.RCU
> 29871 +45.4% 43440 softirqs.CPU42.SCHED
> 40076 +44.8% 58020 softirqs.CPU42.TIMER
> 25281 +48.9% 37640 softirqs.CPU43.RCU
> 30025 +43.9% 43220 softirqs.CPU43.SCHED
> 40222 ? 3% +44.7% 58195 softirqs.CPU43.TIMER
> 25183 +49.3% 37611 softirqs.CPU44.RCU
> 30123 +43.6% 43249 softirqs.CPU44.SCHED
> 41090 ? 6% +42.5% 58537 softirqs.CPU44.TIMER
> 25314 +49.6% 37863 softirqs.CPU45.RCU
> 29998 +43.2% 42957 softirqs.CPU45.SCHED
> 39705 ? 3% +43.6% 57033 softirqs.CPU45.TIMER
> 25500 +46.2% 37274 softirqs.CPU46.RCU
> 29921 +44.7% 43286 softirqs.CPU46.SCHED
> 41088 ? 5% +39.9% 57463 ? 3% softirqs.CPU46.TIMER
> 25564 +48.4% 37931 softirqs.CPU47.RCU
> 30178 +42.4% 42980 softirqs.CPU47.SCHED
> 40738 ? 4% +42.2% 57939 ? 3% softirqs.CPU47.TIMER
> 26467 ? 8% +40.8% 37278 softirqs.CPU5.RCU
> 30334 ? 2% +44.1% 43714 softirqs.CPU5.SCHED
> 42268 ? 4% +35.8% 57394 softirqs.CPU5.TIMER
> 25343 +47.0% 37245 softirqs.CPU6.RCU
> 30100 +45.0% 43643 ? 2% softirqs.CPU6.SCHED
> 41270 +43.6% 59249 ? 5% softirqs.CPU6.TIMER
> 25331 +47.7% 37418 softirqs.CPU7.RCU
> 30210 +44.7% 43727 ? 2% softirqs.CPU7.SCHED
> 41028 +41.9% 58206 ? 2% softirqs.CPU7.TIMER
> 25483 +47.2% 37514 softirqs.CPU8.RCU
> 30324 ? 2% +43.4% 43483 ? 2% softirqs.CPU8.SCHED
> 41027 +40.4% 57591 ? 2% softirqs.CPU8.TIMER
> 25546 +47.5% 37687 softirqs.CPU9.RCU
> 30218 +44.8% 43771 ? 2% softirqs.CPU9.SCHED
> 41193 +40.1% 57716 ? 2% softirqs.CPU9.TIMER
> 1212606 +47.7% 1790808 softirqs.RCU
> 1448198 +44.7% 2095500 softirqs.SCHED
> 1990064 +40.7% 2800202 softirqs.TIMER
> 8.56 ? 3% -4.3 4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
> 8.50 ? 3% -4.3 4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
> 8.47 ? 3% -4.3 4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> 78.55 -4.1 74.44 perf-profile.calltrace.cycles-pp.write
> 8.74 ? 3% -4.0 4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> 77.69 -3.9 73.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> 77.64 -3.9 73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 77.13 -3.8 73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 77.08 -3.8 73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 76.61 -3.7 72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 76.43 -3.6 72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
> 69.22 -2.6 66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
> 4.70 ? 2% -2.2 2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 4.63 ? 2% -2.2 2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 4.60 ? 2% -2.2 2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
> 4.32 ? 2% -2.1 2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
> 3.72 ? 4% -2.1 1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> 3.71 ? 4% -2.1 1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> 2.63 ? 4% -2.0 0.63 ? 3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
> 7.72 -2.0 5.74 ? 2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> 3.08 ? 5% -1.9 1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
> 2.49 ? 5% -1.9 0.55 ? 3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
> 2.48 ? 5% -1.9 0.55 ? 3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
> 2.41 ? 6% -1.8 0.61 ? 2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
> 4.90 -1.5 3.40 ? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 4.78 -1.5 3.30 ? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
> 2.35 -0.6 1.77 ? 2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> 2.36 -0.6 1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
> 3.40 -0.5 2.87 ? 6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
> 1.83 -0.5 1.33 ? 2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
> 3.26 -0.5 2.75 ? 6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
> 2.85 -0.4 2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
> 2.79 -0.4 2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
> 2.83 -0.4 2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
> 1.48 -0.4 1.05 ? 2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
> 0.61 ? 2% -0.4 0.26 ?100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
> 1.89 -0.3 1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 1.71 -0.3 1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
> 1.65 -0.3 1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
> 0.69 -0.3 0.38 ? 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
> 1.59 -0.3 1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
> 0.86 -0.3 0.58 ? 2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
> 1.52 -0.3 1.26 ? 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
> 1.75 -0.2 1.50 ? 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
> 1.74 -0.2 1.49 ? 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
> 1.74 -0.2 1.50 ? 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
> 2.15 -0.2 1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
> 0.79 ? 3% -0.2 0.60 ? 4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
> 1.81 -0.2 1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
> 1.87 -0.2 1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
> 0.70 ? 4% -0.2 0.54 ? 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
> 0.81 -0.1 0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
> 0.65 ? 2% -0.1 0.55 ? 2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
> 0.62 ? 2% -0.1 0.52 ? 3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
> 1.18 -0.0 1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 1.01 -0.0 0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
> 1.00 -0.0 0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
> 0.96 -0.0 0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
> 1.01 -0.0 0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
> 0.59 ? 3% +0.1 0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.58 ? 3% +0.1 0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
> 0.53 +0.1 0.63 ? 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
> 0.91 ? 3% +0.1 1.02 ? 2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
> 0.54 +0.1 0.65 ? 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
> 0.57 +0.1 0.69 ? 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
> 1.83 +0.2 1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
> 0.00 +0.6 0.57 ? 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
> 0.00 +0.6 0.57 ? 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
> 0.00 +0.6 0.57 ? 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
> 0.00 +0.7 0.70 ? 2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
> 0.00 +0.8 0.75 ? 4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
> 0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
> 0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
> 0.00 +0.8 0.80 ? 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
> 0.00 +0.8 0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
> 0.00 +0.8 0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
> 0.00 +0.9 0.89 ? 2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
> 0.00 +0.9 0.92 ? 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
> 0.00 +1.0 0.97 ? 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
> 35.68 +1.0 36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
> 0.00 +1.0 1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
> 0.00 +1.0 1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
> 9.35 +1.2 10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 9.61 +1.2 10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> 9.63 +1.2 10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 12.24 +1.3 13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 12.25 +1.3 13.60 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 12.25 +1.3 13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 12.49 +1.4 13.89 perf-profile.calltrace.cycles-pp.secondary_startup_64
> 14.38 +1.4 15.81 ? 2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> 13.41 +1.4 14.86 ? 2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 13.37 +1.5 14.82 ? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
> 13.34 +1.5 14.80 ? 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
> 58.91 +1.6 60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> 0.00 +2.2 2.24 ? 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
> 5.63 +2.5 8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
> 5.65 +2.5 8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> 5.69 +2.5 8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> 7.04 +2.8 9.80 ? 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> 7.54 +2.9 10.45 ? 2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> 7.55 +2.9 10.46 ? 2% perf-profile.calltrace.cycles-pp.ret_from_fork
> 7.55 +2.9 10.46 ? 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> 0.00 +3.3 3.30 ? 9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
> 0.00 +3.3 3.32 ? 8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
> 8.56 ? 3% -4.3 4.25 perf-profile.children.cycles-pp.__filemap_fdatawrite_range
> 8.47 ? 3% -4.3 4.18 perf-profile.children.cycles-pp.xfs_vm_writepages
> 8.50 ? 3% -4.3 4.20 perf-profile.children.cycles-pp.do_writepages
> 78.64 -4.1 74.51 perf-profile.children.cycles-pp.write
> 78.55 -4.1 74.44 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 78.50 -4.1 74.41 perf-profile.children.cycles-pp.do_syscall_64
> 8.75 ? 3% -4.1 4.70 perf-profile.children.cycles-pp.file_write_and_wait_range
> 77.14 -3.8 73.37 perf-profile.children.cycles-pp.ksys_write
> 77.09 -3.8 73.33 perf-profile.children.cycles-pp.vfs_write
> 76.62 -3.7 72.97 perf-profile.children.cycles-pp.new_sync_write
> 76.44 -3.6 72.81 perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
> 69.23 -2.6 66.62 perf-profile.children.cycles-pp.xfs_file_fsync
> 4.70 ? 2% -2.2 2.50 perf-profile.children.cycles-pp.iomap_submit_ioend
> 4.89 ? 2% -2.1 2.75 perf-profile.children.cycles-pp.submit_bio
> 4.85 ? 2% -2.1 2.71 perf-profile.children.cycles-pp.generic_make_request
> 3.73 ? 4% -2.1 1.63 perf-profile.children.cycles-pp.iomap_writepages
> 3.72 ? 4% -2.1 1.63 perf-profile.children.cycles-pp.write_cache_pages
> 4.43 ? 2% -2.1 2.38 perf-profile.children.cycles-pp.brd_make_request
> 7.73 -2.0 5.75 ? 2% perf-profile.children.cycles-pp.__flush_work
> 3.08 ? 4% -1.9 1.14 perf-profile.children.cycles-pp.iomap_writepage_map
> 6.00 -1.8 4.20 ? 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 2.50 ? 4% -1.8 0.70 ? 3% perf-profile.children.cycles-pp.end_page_writeback
> 2.50 ? 5% -1.8 0.70 ? 4% perf-profile.children.cycles-pp.test_clear_page_writeback
> 2.42 ? 6% -1.8 0.61 ? 2% perf-profile.children.cycles-pp.__test_set_page_writeback
> 2.08 ? 7% -1.6 0.44 ? 4% perf-profile.children.cycles-pp.sb_mark_inode_writeback
> 2.08 ? 6% -1.6 0.44 ? 5% perf-profile.children.cycles-pp.sb_clear_inode_writeback
> 2.70 ? 4% -1.0 1.67 ? 2% perf-profile.children.cycles-pp.iomap_finish_ioend
> 2.36 -0.6 1.79 perf-profile.children.cycles-pp.iomap_file_buffered_write
> 2.35 -0.6 1.78 perf-profile.children.cycles-pp.iomap_apply
> 1.84 -0.5 1.33 ? 2% perf-profile.children.cycles-pp.iomap_write_actor
> 2.41 ? 4% -0.5 1.94 ? 2% perf-profile.children.cycles-pp.reschedule_interrupt
> 3.44 ? 2% -0.5 2.98 perf-profile.children.cycles-pp.sched_ttwu_pending
> 2.87 -0.5 2.42 perf-profile.children.cycles-pp.complete
> 2.83 -0.4 2.38 perf-profile.children.cycles-pp.swake_up_locked
> 2.28 ? 4% -0.4 1.84 ? 2% perf-profile.children.cycles-pp.scheduler_ipi
> 1.56 -0.4 1.17 ? 2% perf-profile.children.cycles-pp.brd_do_bvec
> 1.89 -0.4 1.54 perf-profile.children.cycles-pp.wait_for_completion
> 1.71 -0.3 1.38 perf-profile.children.cycles-pp.schedule_timeout
> 0.88 -0.2 0.68 ? 2% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
> 0.69 -0.2 0.51 ? 2% perf-profile.children.cycles-pp.iomap_write_begin
> 0.67 ? 2% -0.2 0.49 perf-profile.children.cycles-pp.lseek64
> 1.33 ? 5% -0.2 1.17 ? 3% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.51 -0.2 0.36 perf-profile.children.cycles-pp.grab_cache_page_write_begin
> 0.47 -0.1 0.33 perf-profile.children.cycles-pp.pagecache_get_page
> 0.51 ? 3% -0.1 0.38 ? 2% perf-profile.children.cycles-pp.iomap_write_end
> 0.57 ? 2% -0.1 0.46 perf-profile.children.cycles-pp.blkdev_issue_flush
> 0.38 -0.1 0.26 ? 5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
> 0.34 -0.1 0.23 ? 6% perf-profile.children.cycles-pp.copyin
> 0.33 -0.1 0.22 ? 6% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
> 0.65 ? 2% -0.1 0.56 ? 2% perf-profile.children.cycles-pp.rwsem_optimistic_spin
> 0.37 -0.1 0.28 ? 2% perf-profile.children.cycles-pp.bio_alloc_bioset
> 0.31 ? 4% -0.1 0.22 perf-profile.children.cycles-pp.syscall_return_via_sysret
> 0.56 ? 3% -0.1 0.48 ? 3% perf-profile.children.cycles-pp.osq_lock
> 0.26 ? 3% -0.1 0.18 ? 4% perf-profile.children.cycles-pp.entry_SYSCALL_64
> 0.28 ? 2% -0.1 0.21 ? 5% perf-profile.children.cycles-pp.mempool_alloc
> 0.26 ? 6% -0.1 0.18 ? 8% perf-profile.children.cycles-pp.xlog_state_get_iclog_space
> 0.29 -0.1 0.22 ? 3% perf-profile.children.cycles-pp.find_get_entry
> 0.34 ? 2% -0.1 0.27 ? 4% perf-profile.children.cycles-pp.generic_make_request_checks
> 0.30 ? 5% -0.1 0.23 ? 3% perf-profile.children.cycles-pp.iomap_set_page_dirty
> 0.32 ? 2% -0.1 0.26 ? 5% perf-profile.children.cycles-pp.brd_lookup_page
> 0.18 ? 2% -0.1 0.12 ? 6% perf-profile.children.cycles-pp.iomap_set_range_uptodate
> 0.15 ? 2% -0.1 0.09 ? 4% perf-profile.children.cycles-pp.__mod_lruvec_state
> 0.15 ? 5% -0.1 0.10 ? 9% perf-profile.children.cycles-pp.wake_up_q
> 0.21 ? 5% -0.1 0.15 ? 5% perf-profile.children.cycles-pp.xfs_ilock
> 0.13 ? 3% -0.0 0.08 ? 10% perf-profile.children.cycles-pp.rwsem_wake
> 0.20 ? 2% -0.0 0.15 ? 5% perf-profile.children.cycles-pp.ksys_lseek
> 0.20 ? 3% -0.0 0.15 ? 3% perf-profile.children.cycles-pp.xas_load
> 0.20 ? 5% -0.0 0.16 ? 2% perf-profile.children.cycles-pp.__set_page_dirty
> 0.19 ? 2% -0.0 0.15 ? 2% perf-profile.children.cycles-pp.xfs_iunlock
> 0.40 -0.0 0.36 ? 4% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
> 0.17 ? 2% -0.0 0.12 ? 4% perf-profile.children.cycles-pp.clear_page_dirty_for_io
> 0.20 ? 3% -0.0 0.16 ? 4% perf-profile.children.cycles-pp.submit_bio_wait
> 0.18 ? 4% -0.0 0.15 ? 7% perf-profile.children.cycles-pp.security_file_permission
> 0.10 -0.0 0.06 ? 6% perf-profile.children.cycles-pp.evict
> 0.10 -0.0 0.06 ? 6% perf-profile.children.cycles-pp.truncate_inode_pages_range
> 0.06 ? 6% -0.0 0.03 ?100% perf-profile.children.cycles-pp.vfs_unlink
> 0.08 ? 6% -0.0 0.04 ? 58% perf-profile.children.cycles-pp.add_to_page_cache_lru
> 0.26 ? 2% -0.0 0.23 ? 3% perf-profile.children.cycles-pp.reweight_entity
> 0.12 ? 6% -0.0 0.09 ? 4% perf-profile.children.cycles-pp.bio_endio
> 0.10 ? 7% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.do_sys_open
> 0.10 ? 7% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.do_sys_openat2
> 0.17 ? 2% -0.0 0.14 perf-profile.children.cycles-pp.__pagevec_release
> 0.13 ? 5% -0.0 0.10 ? 4% perf-profile.children.cycles-pp.down_write
> 0.10 ? 5% -0.0 0.06 ? 6% perf-profile.children.cycles-pp.do_filp_open
> 0.15 ? 4% -0.0 0.12 ? 5% perf-profile.children.cycles-pp.tag_pages_for_writeback
> 0.11 ? 9% -0.0 0.08 ? 5% perf-profile.children.cycles-pp.bio_associate_blkg
> 0.10 ? 5% -0.0 0.07 ? 7% perf-profile.children.cycles-pp.open64
> 0.09 ? 4% -0.0 0.06 ? 6% perf-profile.children.cycles-pp.path_openat
> 0.16 ? 8% -0.0 0.13 ? 3% perf-profile.children.cycles-pp.pagevec_lookup_range_tag
> 0.12 ? 8% -0.0 0.10 ? 5% perf-profile.children.cycles-pp.account_page_dirtied
> 0.10 ? 4% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.__fdget_pos
> 0.14 ? 3% -0.0 0.11 ? 7% perf-profile.children.cycles-pp.memset_erms
> 0.11 ? 3% -0.0 0.09 ? 4% perf-profile.children.cycles-pp.switch_fpu_return
> 0.23 -0.0 0.21 ? 3% perf-profile.children.cycles-pp.__radix_tree_lookup
> 0.18 ? 2% -0.0 0.16 ? 9% perf-profile.children.cycles-pp.insert_work
> 0.12 ? 4% -0.0 0.10 ? 7% perf-profile.children.cycles-pp.common_file_perm
> 0.09 ? 4% -0.0 0.07 ? 6% perf-profile.children.cycles-pp.__fget_light
> 0.09 ? 4% -0.0 0.07 ? 6% perf-profile.children.cycles-pp.bvec_alloc
> 0.15 ? 7% -0.0 0.12 ? 3% perf-profile.children.cycles-pp.find_get_pages_range_tag
> 0.08 ? 8% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.iov_iter_fault_in_readable
> 0.09 ? 5% -0.0 0.06 ? 13% perf-profile.children.cycles-pp.blk_throtl_bio
> 0.24 ? 4% -0.0 0.22 ? 3% perf-profile.children.cycles-pp._cond_resched
> 0.13 ? 3% -0.0 0.11 ? 4% perf-profile.children.cycles-pp.release_pages
> 0.10 ? 4% -0.0 0.08 ? 6% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
> 0.08 ? 10% -0.0 0.06 perf-profile.children.cycles-pp.unlink
> 0.08 ? 10% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.__sb_start_write
> 0.08 ? 5% -0.0 0.06 ? 7% perf-profile.children.cycles-pp.generic_write_checks
> 0.08 ? 8% -0.0 0.06 perf-profile.children.cycles-pp.do_unlinkat
> 0.07 ? 10% -0.0 0.05 perf-profile.children.cycles-pp.xfs_break_layouts
> 0.18 ? 2% -0.0 0.16 ? 2% perf-profile.children.cycles-pp.__list_add_valid
> 0.12 ? 3% -0.0 0.10 ? 8% perf-profile.children.cycles-pp.__might_sleep
> 0.14 ? 3% -0.0 0.12 ? 7% perf-profile.children.cycles-pp.xas_find_marked
> 0.07 ? 10% -0.0 0.05 ? 8% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
> 0.08 ? 5% -0.0 0.06 ? 11% perf-profile.children.cycles-pp.bio_free
> 0.08 ? 6% -0.0 0.06 perf-profile.children.cycles-pp.xfs_file_llseek
> 0.08 ? 5% -0.0 0.06 ? 6% perf-profile.children.cycles-pp.xfs_file_write_iter
> 0.16 ? 2% +0.0 0.17 ? 2% perf-profile.children.cycles-pp.__switch_to_asm
> 0.05 +0.0 0.07 ? 7% perf-profile.children.cycles-pp.cpumask_next_wrap
> 0.10 ? 4% +0.0 0.12 ? 3% perf-profile.children.cycles-pp._find_next_bit
> 0.07 ? 7% +0.0 0.09 ? 5% perf-profile.children.cycles-pp.xlog_state_done_syncing
> 0.16 ? 2% +0.0 0.19 ? 2% perf-profile.children.cycles-pp.check_preempt_curr
> 0.19 ? 4% +0.0 0.22 perf-profile.children.cycles-pp.ttwu_do_wakeup
> 0.56 ? 3% +0.0 0.60 perf-profile.children.cycles-pp.dequeue_entity
> 0.20 ? 6% +0.0 0.23 ? 4% perf-profile.children.cycles-pp.xfs_trans_committed_bulk
> 0.07 ? 7% +0.0 0.10 ? 7% perf-profile.children.cycles-pp.check_preempt_wakeup
> 0.82 ? 2% +0.0 0.86 perf-profile.children.cycles-pp.update_load_avg
> 0.31 ? 3% +0.0 0.35 perf-profile.children.cycles-pp.__unwind_start
> 0.08 ? 8% +0.0 0.12 ? 3% perf-profile.children.cycles-pp.destroy_inode
> 0.08 ? 10% +0.0 0.12 ? 3% perf-profile.children.cycles-pp.xfs_fs_destroy_inode
> 0.08 ? 10% +0.0 0.12 ? 3% perf-profile.children.cycles-pp.xfs_inactive
> 0.08 ? 15% +0.0 0.12 ? 5% perf-profile.children.cycles-pp.xfs_bmap_validate_extent
> 3.96 +0.0 4.01 perf-profile.children.cycles-pp.stack_trace_save_tsk
> 0.28 +0.0 0.33 perf-profile.children.cycles-pp.set_next_entity
> 0.00 +0.1 0.05 perf-profile.children.cycles-pp.xfs_trans_roll
> 0.00 +0.1 0.05 ? 8% perf-profile.children.cycles-pp.xfs_btree_delrec
> 0.00 +0.1 0.05 ? 8% perf-profile.children.cycles-pp.xfs_btree_insert
> 0.16 ? 4% +0.1 0.21 ? 5% perf-profile.children.cycles-pp.memcpy_erms
> 0.04 ? 57% +0.1 0.10 ? 4% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
> 0.00 +0.1 0.06 ? 11% perf-profile.children.cycles-pp.xfs_buf_item_format_segment
> 0.00 +0.1 0.06 ? 11% perf-profile.children.cycles-pp.xfs_btree_delete
> 0.00 +0.1 0.06 perf-profile.children.cycles-pp.xfs_btree_update
> 0.00 +0.1 0.07 ? 7% perf-profile.children.cycles-pp.xfs_cil_prepare_item
> 0.00 +0.1 0.07 ? 6% perf-profile.children.cycles-pp.xfs_bmbt_update
> 0.00 +0.1 0.07 ? 10% perf-profile.children.cycles-pp.xfs_buf_item_format
> 0.56 +0.1 0.64 ? 2% perf-profile.children.cycles-pp.pick_next_task_fair
> 0.03 ?100% +0.1 0.10 ? 4% perf-profile.children.cycles-pp.xfs_inactive_truncate
> 0.08 ? 5% +0.1 0.16 ? 11% perf-profile.children.cycles-pp.xfs_iext_lookup_extent
> 0.00 +0.1 0.08 ? 5% perf-profile.children.cycles-pp.xfs_defer_finish
> 0.00 +0.1 0.08 ? 5% perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
> 0.00 +0.1 0.08 ? 5% perf-profile.children.cycles-pp.xfs_verify_agbno
> 0.00 +0.1 0.09 ? 9% perf-profile.children.cycles-pp.xfs_defer_finish_noroll
> 0.60 ? 3% +0.1 0.70 perf-profile.children.cycles-pp.schedule_idle
> 0.39 ? 2% +0.1 0.49 perf-profile.children.cycles-pp.xlog_state_release_iclog
> 0.00 +0.1 0.10 ? 9% perf-profile.children.cycles-pp.xfs_buf_find
> 0.93 ? 3% +0.1 1.03 ? 2% perf-profile.children.cycles-pp.xlog_write
> 2.58 +0.1 2.69 perf-profile.children.cycles-pp.schedule
> 1.52 +0.1 1.63 perf-profile.children.cycles-pp.__wake_up_common_lock
> 0.00 +0.1 0.11 ? 7% perf-profile.children.cycles-pp.xfs_buf_get_map
> 4.62 +0.1 4.73 perf-profile.children.cycles-pp.__account_scheduler_latency
> 0.00 +0.1 0.12 ? 10% perf-profile.children.cycles-pp.xfs_buf_read_map
> 1.83 +0.2 1.99 perf-profile.children.cycles-pp.xlog_cil_push_work
> 5.94 +0.2 6.12 perf-profile.children.cycles-pp.ttwu_do_activate
> 0.00 +0.2 0.17 ? 6% perf-profile.children.cycles-pp.xfs_btree_read_buf_block
> 5.93 +0.2 6.11 perf-profile.children.cycles-pp.activate_task
> 5.88 +0.2 6.06 perf-profile.children.cycles-pp.enqueue_task_fair
> 0.00 +0.2 0.18 ? 5% perf-profile.children.cycles-pp.xfs_trans_read_buf_map
> 5.41 +0.2 5.59 perf-profile.children.cycles-pp.enqueue_entity
> 3.14 +0.2 3.33 perf-profile.children.cycles-pp.__sched_text_start
> 0.00 +0.2 0.21 ? 6% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
> 0.00 +0.2 0.23 ? 57% perf-profile.children.cycles-pp.xlog_grant_head_wait
> 0.00 +0.2 0.24 ? 6% perf-profile.children.cycles-pp.io_schedule
> 0.11 ? 3% +0.3 0.38 ? 4% perf-profile.children.cycles-pp.__filemap_fdatawait_range
> 0.00 +0.3 0.27 ? 8% perf-profile.children.cycles-pp.wait_on_page_bit
> 0.00 +0.3 0.34 ? 5% perf-profile.children.cycles-pp.xfs_btree_lookup
> 0.08 ? 14% +0.4 0.43 perf-profile.children.cycles-pp.__queue_work
> 0.08 ? 8% +0.4 0.44 perf-profile.children.cycles-pp.queue_work_on
> 0.00 +0.4 0.41 perf-profile.children.cycles-pp.xfs_end_bio
> 3.51 +0.4 3.94 ? 9% perf-profile.children.cycles-pp.__xfs_trans_commit
> 0.20 ? 9% +0.5 0.72 ? 54% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
> 0.06 ? 11% +0.5 0.58 ? 67% perf-profile.children.cycles-pp.xfs_log_space_wake
> 0.45 ? 5% +0.6 1.04 ? 36% perf-profile.children.cycles-pp.xfs_log_reserve
> 0.65 ? 3% +0.6 1.25 ? 29% perf-profile.children.cycles-pp.xfs_trans_alloc
> 0.48 ? 5% +0.6 1.09 ? 34% perf-profile.children.cycles-pp.xfs_trans_reserve
> 0.04 ? 57% +0.6 0.65 ? 60% perf-profile.children.cycles-pp.xlog_grant_head_check
> 4.83 +0.7 5.52 perf-profile.children.cycles-pp.try_to_wake_up
> 0.00 +0.7 0.70 ? 2% perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
> 0.00 +0.8 0.75 ? 4% perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
> 0.04 ? 57% +0.8 0.80 perf-profile.children.cycles-pp.autoremove_wake_function
> 0.05 ? 8% +0.8 0.85 perf-profile.children.cycles-pp.wake_up_page_bit
> 1.50 +0.9 2.36 perf-profile.children.cycles-pp.__wake_up_common
> 0.00 +0.9 0.89 ? 2% perf-profile.children.cycles-pp.xfs_bmapi_write
> 24.29 +0.9 25.18 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.07 ? 5% +1.0 1.05 perf-profile.children.cycles-pp.iomap_finish_ioends
> 35.68 +1.0 36.69 perf-profile.children.cycles-pp.xlog_cil_force_lsn
> 9.53 +1.2 10.76 perf-profile.children.cycles-pp.intel_idle
> 9.81 +1.3 11.10 perf-profile.children.cycles-pp.cpuidle_enter_state
> 9.82 +1.3 11.11 perf-profile.children.cycles-pp.cpuidle_enter
> 12.25 +1.3 13.60 perf-profile.children.cycles-pp.start_secondary
> 12.49 +1.4 13.89 perf-profile.children.cycles-pp.secondary_startup_64
> 12.49 +1.4 13.89 perf-profile.children.cycles-pp.cpu_startup_entry
> 12.48 +1.4 13.88 perf-profile.children.cycles-pp.do_idle
> 14.78 +1.4 16.20 ? 2% perf-profile.children.cycles-pp.xlog_wait_on_iclog
> 58.91 +1.7 60.57 perf-profile.children.cycles-pp.xfs_log_force_lsn
> 0.00 +2.2 2.24 ? 13% perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
> 7.05 +2.8 9.81 ? 2% perf-profile.children.cycles-pp.process_one_work
> 7.54 +2.9 10.45 ? 2% perf-profile.children.cycles-pp.worker_thread
> 7.55 +2.9 10.46 ? 2% perf-profile.children.cycles-pp.ret_from_fork
> 7.55 +2.9 10.46 ? 2% perf-profile.children.cycles-pp.kthread
> 0.10 +3.2 3.30 ? 9% perf-profile.children.cycles-pp.xfs_end_ioend
> 0.10 ? 4% +3.2 3.32 ? 8% perf-profile.children.cycles-pp.xfs_end_io
> 19.57 +4.0 23.62 perf-profile.children.cycles-pp.remove_wait_queue
> 1.20 -0.3 0.90 ? 2% perf-profile.self.cycles-pp.brd_do_bvec
> 0.55 ? 3% -0.2 0.39 ? 4% perf-profile.self.cycles-pp.do_syscall_64
> 0.31 -0.1 0.22 ? 5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
> 0.31 ? 3% -0.1 0.22 perf-profile.self.cycles-pp.syscall_return_via_sysret
> 0.54 ? 3% -0.1 0.47 ? 3% perf-profile.self.cycles-pp.osq_lock
> 0.23 ? 3% -0.1 0.16 ? 5% perf-profile.self.cycles-pp.entry_SYSCALL_64
> 0.10 ? 4% -0.1 0.04 ? 57% perf-profile.self.cycles-pp.__mod_lruvec_state
> 0.18 ? 4% -0.1 0.12 ? 5% perf-profile.self.cycles-pp.iomap_set_range_uptodate
> 0.60 ? 4% -0.1 0.54 ? 2% perf-profile.self.cycles-pp._raw_spin_lock
> 0.77 -0.0 0.72 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.14 -0.0 0.10 ? 8% perf-profile.self.cycles-pp.find_get_entry
> 0.14 ? 3% -0.0 0.11 ? 4% perf-profile.self.cycles-pp.xas_load
> 0.17 ? 4% -0.0 0.14 ? 3% perf-profile.self.cycles-pp.write
> 0.30 ? 2% -0.0 0.27 ? 6% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 0.18 ? 5% -0.0 0.15 ? 7% perf-profile.self.cycles-pp.__flush_work
> 0.08 ? 5% -0.0 0.05 perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
> 0.12 ? 3% -0.0 0.09 ? 4% perf-profile.self.cycles-pp.brd_lookup_page
> 0.12 ? 7% -0.0 0.09 ? 9% perf-profile.self.cycles-pp.vfs_write
> 0.18 ? 4% -0.0 0.15 ? 7% perf-profile.self.cycles-pp.insert_work
> 0.13 -0.0 0.10 ? 8% perf-profile.self.cycles-pp.memset_erms
> 0.11 ? 4% -0.0 0.08 perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
> 0.11 ? 6% -0.0 0.08 ? 5% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
> 0.12 ? 3% -0.0 0.09 ? 4% perf-profile.self.cycles-pp.iomap_writepage_map
> 0.08 ? 8% -0.0 0.06 ? 15% perf-profile.self.cycles-pp.bio_alloc_bioset
> 0.06 ? 6% -0.0 0.04 ? 57% perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
> 0.09 ? 5% -0.0 0.06 ? 6% perf-profile.self.cycles-pp.__fget_light
> 0.06 -0.0 0.04 ? 57% perf-profile.self.cycles-pp.xfs_map_blocks
> 0.18 ? 2% -0.0 0.15 ? 3% perf-profile.self.cycles-pp.__list_add_valid
> 0.08 ? 6% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.xfs_ilock
> 0.08 ? 6% -0.0 0.06 ? 9% perf-profile.self.cycles-pp.iov_iter_fault_in_readable
> 0.11 ? 4% -0.0 0.09 ? 4% perf-profile.self.cycles-pp.switch_fpu_return
> 0.07 -0.0 0.05 perf-profile.self.cycles-pp.iomap_write_actor
> 0.23 ? 2% -0.0 0.21 ? 5% perf-profile.self.cycles-pp.__radix_tree_lookup
> 0.18 ? 4% -0.0 0.16 ? 6% perf-profile.self.cycles-pp.xlog_cil_force_lsn
> 0.20 ? 2% -0.0 0.18 ? 4% perf-profile.self.cycles-pp.kmem_cache_alloc
> 0.10 ? 5% -0.0 0.08 ? 11% perf-profile.self.cycles-pp.xfs_file_fsync
> 0.07 ? 5% -0.0 0.05 ? 9% perf-profile.self.cycles-pp.clear_page_dirty_for_io
> 0.07 ? 6% -0.0 0.05 perf-profile.self.cycles-pp.__test_set_page_writeback
> 0.11 -0.0 0.09 ? 8% perf-profile.self.cycles-pp.common_file_perm
> 0.23 -0.0 0.21 ? 4% perf-profile.self.cycles-pp.update_rq_clock
> 0.08 ? 6% -0.0 0.06 perf-profile.self.cycles-pp.xfs_file_write_iter
> 0.10 ? 5% -0.0 0.08 ? 8% perf-profile.self.cycles-pp.up_read
> 0.08 ? 5% -0.0 0.07 perf-profile.self.cycles-pp.write_cache_pages
> 0.10 -0.0 0.09 perf-profile.self.cycles-pp.reweight_entity
> 0.05 ? 8% +0.0 0.07 ? 7% perf-profile.self.cycles-pp.dequeue_entity
> 0.16 ? 4% +0.0 0.17 ? 2% perf-profile.self.cycles-pp.__switch_to_asm
> 0.11 ? 4% +0.0 0.12 ? 3% perf-profile.self.cycles-pp.set_next_entity
> 0.15 ? 3% +0.0 0.16 ? 2% perf-profile.self.cycles-pp.orc_find
> 0.09 ? 4% +0.0 0.11 ? 6% perf-profile.self.cycles-pp.enqueue_entity
> 0.10 ? 4% +0.0 0.12 ? 3% perf-profile.self.cycles-pp._find_next_bit
> 0.01 ?173% +0.0 0.06 ? 7% perf-profile.self.cycles-pp.check_preempt_wakeup
> 0.01 ?173% +0.0 0.06 ? 11% perf-profile.self.cycles-pp.xfs_trans_committed_bulk
> 0.45 ? 2% +0.1 0.51 ? 2% perf-profile.self.cycles-pp.__sched_text_start
> 0.15 ? 6% +0.1 0.21 ? 3% perf-profile.self.cycles-pp.memcpy_erms
> 0.00 +0.1 0.06 ? 6% perf-profile.self.cycles-pp.xfs_verify_agbno
> 0.08 ? 8% +0.1 0.16 ? 9% perf-profile.self.cycles-pp.xfs_iext_lookup_extent
> 9.53 +1.2 10.76 perf-profile.self.cycles-pp.intel_idle
>
>
>
> aim7.jobs-per-min
>
> 45000 +-------------------------------------------------------------------+
> | + + |
> 40000 |.+.+ : + .+. +. .+. |
> 35000 |-+ : : + +.+ +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+ ++.+.|
> | : : |
> 30000 |-+ : : |
> 25000 |-O :O O:O OO |
> | : : O O O OO O O O O |
> 20000 |-+ : : |
> 15000 |-+ : : |
> | : : |
> 10000 |-+ : : |
> 5000 |-+ : : |
> | : : |
> 0 +-------------------------------------------------------------------+
>
>
> aim7.time.system_time
>
> 5000 +--------------------------------------------------------------------+
> 4500 |-+ O OO O O O O |
> | O OO O O O O O O |
> 4000 |-+ |
> 3500 |-+ |
> | .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+ .+.+.+. .++.+.|
> 3000 |.+.+ .+ + + + + |
> 2500 |-+ : +.+ |
> 2000 |-+ : : |
> | : : |
> 1500 |-+ : : |
> 1000 |-+ : : |
> | : : |
> 500 |-+ : : |
> 0 +--------------------------------------------------------------------+
>
>
> aim7.time.elapsed_time
>
> 160 +---------------------------------------------------------------------+
> | O OO O O O O OO O O O O |
> 140 |-O |
> 120 |-+ |
> | |
> 100 |-+ .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
> |.+.+ +. .+ + + + |
> 80 |-+ : : + |
> | : : |
> 60 |-+ : : |
> 40 |-+ : : |
> | : : |
> 20 |-+ : : |
> | :: |
> 0 +---------------------------------------------------------------------+
>
>
> aim7.time.elapsed_time.max
>
> 160 +---------------------------------------------------------------------+
> | O OO O O O O OO O O O O |
> 140 |-O |
> 120 |-+ |
> | |
> 100 |-+ .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
> |.+.+ +. .+ + + + |
> 80 |-+ : : + |
> | : : |
> 60 |-+ : : |
> 40 |-+ : : |
> | : : |
> 20 |-+ : : |
> | :: |
> 0 +---------------------------------------------------------------------+
>
>
> aim7.time.voluntary_context_switches
>
> 8e+07 +-------------------------------------------------------------------+
> | O O O O OO O O |
> 7e+07 |-+ |
> 6e+07 |-+ |
> | |
> 5e+07 |-+ +.+.+.++.+.+.+.++.+.+. .+ .+.+. .+. .+. +.+.|
> |.+.+ .+ + +.+ ++.+ ++.+ +.+ |
> 4e+07 |-+ : +.+ |
> | : : |
> 3e+07 |-+ : : |
> 2e+07 |-+ : : |
> | : : |
> 1e+07 |-+ : : |
> | : : |
> 0 +-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>
>
> _______________________________________________
> LKP mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>

--
Zhengjun Xing

2020-07-01 08:36:18

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

On 6/10/2020 11:07 AM, Xing Zhengjun wrote:
> Hi Darrick,
>
> ?? Do you have time to take a look at this? Thanks.
>

Ping...

> On 6/6/2020 11:48 PM, kernel test robot wrote:
>> Greeting,
>>
>> FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:
>>
>>
>> commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes
>> to delalloc regions to unwritten")
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>
>> in testcase: aim7
>> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
>> with 64G memory
>> with following parameters:
>>
>> ????disk: 1BRD_48G
>> ????fs: xfs
>> ????test: sync_disk_rw
>> ????load: 600
>> ????cpufreq_governor: performance
>> ????ucode: 0x42e
>>
>> test-description: AIM7 is a traditional UNIX system level benchmark
>> suite which is used to test and measure the performance of multiuser
>> system.
>> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
>>
>>
>>
>> If you fix the issue, kindly add following tag
>> Reported-by: kernel test robot <[email protected]>
>>
>>
>> Details are as below:
>> -------------------------------------------------------------------------------------------------->
>>
>>
>>
>> To reproduce:
>>
>> ???????? git clone https://github.com/intel/lkp-tests.git
>> ???????? cd lkp-tests
>> ???????? bin/lkp install job.yaml? # job file is attached in this email
>> ???????? bin/lkp run???? job.yaml
>>
>> =========================================================================================
>>
>> compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
>>
>>
>> gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
>>
>>
>> commit:
>> ?? 590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
>> ?? a5949d3fae ("xfs: force writes to delalloc regions to unwritten")
>>
>> 590b16516ef38e2e a5949d3faedf492fa7863b914da
>> ---------------- ---------------------------
>> ??????? fail:runs? %reproduction??? fail:runs
>> ??????????? |???????????? |???????????? |
>> ??????????? :4?????????? 50%?????????? 2:4
>> dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
>> ????????? %stddev???? %change???????? %stddev
>> ????????????? \????????? |??????????????? \
>> ????? 35272?????????? -33.6%????? 23430??????? aim7.jobs-per-min
>> ???? 102.13?????????? +50.5%???? 153.75??????? aim7.time.elapsed_time
>> ???? 102.13?????????? +50.5%???? 153.75??????? aim7.time.elapsed_time.max
>> ??? 1388038?????????? +40.2%??? 1945838
>> aim7.time.involuntary_context_switches
>> ????? 43420 ?? 2%???? +13.4%????? 49255 ?? 2%
>> aim7.time.minor_page_faults
>> ?????? 3123?????????? +44.2%?????? 4504 ?? 2%? aim7.time.system_time
>> ????? 59.31??????????? +6.5%????? 63.18??????? aim7.time.user_time
>> ?? 48595108?????????? +58.6%?? 77064959
>> aim7.time.voluntary_context_switches
>> ?????? 1.44?????????? -28.8%?????? 1.02??????? iostat.cpu.user
>> ?????? 0.07 ?? 6%????? +0.4??????? 0.44 ?? 7%? mpstat.cpu.all.iowait%
>> ?????? 1.44??????????? -0.4??????? 1.02??????? mpstat.cpu.all.usr%
>> ?????? 8632 ? 50%???? +75.6%????? 15156 ? 34%
>> numa-meminfo.node0.KernelStack
>> ?????? 6583 ?136%??? +106.0%????? 13562 ? 82%
>> numa-meminfo.node0.PageTables
>> ????? 63325 ? 11%???? +14.3%????? 72352 ? 12%
>> numa-meminfo.node0.SUnreclaim
>> ?????? 8647 ? 50%???? +75.3%????? 15156 ? 34%
>> numa-vmstat.node0.nr_kernel_stack
>> ?????? 1656 ?136%??? +104.6%?????? 3389 ? 82%
>> numa-vmstat.node0.nr_page_table_pages
>> ????? 15831 ? 11%???? +14.3%????? 18087 ? 12%
>> numa-vmstat.node0.nr_slab_unreclaimable
>> ????? 93640 ?? 3%???? +41.2%???? 132211 ?? 2%? meminfo.AnonHugePages
>> ????? 21641?????????? +39.9%????? 30271 ?? 4%? meminfo.KernelStack
>> ???? 129269?????????? +12.3%???? 145114??????? meminfo.SUnreclaim
>> ????? 28000?????????? -31.2%????? 19275??????? meminfo.max_used_kB
>> ??? 1269307?????????? -26.9%???? 927657??????? vmstat.io.bo
>> ???? 149.75 ?? 3%???? -17.4%???? 123.75 ?? 4%? vmstat.procs.r
>> ???? 718992?????????? +13.3%???? 814567??????? vmstat.system.cs
>> ???? 231397??????????? -9.3%???? 209881 ?? 2%? vmstat.system.in
>> ? 6.774e+08?????????? +70.0%? 1.152e+09??????? cpuidle.C1.time
>> ?? 18203372?????????? +60.4%?? 29198744??????? cpuidle.C1.usage
>> ? 2.569e+08 ? 18%???? +81.8%? 4.672e+08 ?? 5%? cpuidle.C1E.time
>> ??? 2691402 ? 13%???? +98.7%??? 5346901 ?? 3%? cpuidle.C1E.usage
>> ???? 990350?????????? +95.0%??? 1931226 ?? 2%? cpuidle.POLL.time
>> ???? 520061?????????? +97.7%??? 1028004 ?? 2%? cpuidle.POLL.usage
>> ????? 77231??????????? +1.8%????? 78602??????? proc-vmstat.nr_active_anon
>> ????? 19868??????????? +3.8%????? 20615??????? proc-vmstat.nr_dirty
>> ???? 381302??????????? +1.0%???? 384969??????? proc-vmstat.nr_file_pages
>> ?????? 4388??????????? -2.7%?????? 4270
>> proc-vmstat.nr_inactive_anon
>> ????? 69865??????????? +4.7%????? 73155
>> proc-vmstat.nr_inactive_file
>> ????? 21615?????????? +40.0%????? 30251 ?? 4%
>> proc-vmstat.nr_kernel_stack
>> ?????? 7363??????????? -3.2%?????? 7127??????? proc-vmstat.nr_mapped
>> ????? 12595 ?? 3%????? +5.2%????? 13255 ?? 4%? proc-vmstat.nr_shmem
>> ????? 19619??????????? +3.2%????? 20247
>> proc-vmstat.nr_slab_reclaimable
>> ????? 32316?????????? +12.3%????? 36280
>> proc-vmstat.nr_slab_unreclaimable
>> ????? 77231??????????? +1.8%????? 78602
>> proc-vmstat.nr_zone_active_anon
>> ?????? 4388??????????? -2.7%?????? 4270
>> proc-vmstat.nr_zone_inactive_anon
>> ????? 69865??????????? +4.7%????? 73155
>> proc-vmstat.nr_zone_inactive_file
>> ????? 19165??????????? +3.4%????? 19824
>> proc-vmstat.nr_zone_write_pending
>> ?????? 9845 ? 27%???? +68.0%????? 16539 ? 37%
>> proc-vmstat.numa_hint_faults_local
>> ??? 8195135??????????? +2.5%??? 8402537??????? proc-vmstat.numa_hit
>> ??? 8179414??????????? +2.5%??? 8386209??????? proc-vmstat.numa_local
>> ????? 15720??????????? +3.9%????? 16328??????? proc-vmstat.numa_other
>> ??? 8281098??????????? +2.4%??? 8477157??????? proc-vmstat.pgalloc_normal
>> ???? 342925 ?? 2%???? +39.8%???? 479279??????? proc-vmstat.pgfault
>> ??? 8132789??????????? +2.0%??? 8298189??????? proc-vmstat.pgfree
>> ? 1.327e+08??????????? +9.7%? 1.455e+08??????? proc-vmstat.pgpgout
>> ???? 394.00????????? +151.7%???? 991.75 ?? 5%
>> slabinfo.biovec-max.active_objs
>> ???? 394.00????????? +152.1%???? 993.25 ?? 5%
>> slabinfo.biovec-max.num_objs
>> ????? 21028 ?? 6%????? -7.8%????? 19389 ?? 5%? slabinfo.filp.active_objs
>> ????? 21520 ?? 5%????? -7.6%????? 19886 ?? 5%? slabinfo.filp.num_objs
>> ?????? 4955????????? +234.9%????? 16594 ?? 2%
>> slabinfo.kmalloc-256.active_objs
>> ????? 77.25????????? +238.5%???? 261.50 ?? 2%
>> slabinfo.kmalloc-256.active_slabs
>> ?????? 4955????????? +238.4%????? 16768 ?? 2%
>> slabinfo.kmalloc-256.num_objs
>> ????? 77.25????????? +238.5%???? 261.50 ?? 2%
>> slabinfo.kmalloc-256.num_slabs
>> ?????? 3368 ?? 2%???? +22.6%?????? 4128 ?? 2%
>> slabinfo.kmalloc-2k.active_objs
>> ?????? 3446 ?? 2%???? +21.9%?????? 4200 ?? 2%
>> slabinfo.kmalloc-2k.num_objs
>> ????? 11815 ?? 8%???? +29.0%????? 15239 ?? 3%
>> slabinfo.kmalloc-512.active_objs
>> ????? 11824 ?? 9%???? +29.0%????? 15249 ?? 3%
>> slabinfo.kmalloc-512.num_objs
>> ?????? 6849 ?? 5%????? +9.0%?????? 7463 ?? 2%
>> slabinfo.kmalloc-96.active_objs
>> ?????? 6906 ?? 5%????? +8.9%?????? 7521 ?? 2%
>> slabinfo.kmalloc-96.num_objs
>> ???? 765.00 ?? 4%???? +25.0%???? 956.25 ?? 5%
>> slabinfo.mnt_cache.active_objs
>> ???? 765.00 ?? 4%???? +25.0%???? 956.25 ?? 5%
>> slabinfo.mnt_cache.num_objs
>> ????? 13866?????????? +12.5%????? 15594
>> slabinfo.proc_inode_cache.active_objs
>> ????? 13905?????????? +12.2%????? 15598
>> slabinfo.proc_inode_cache.num_objs
>> ?????? 1954?????????? +24.4%?????? 2431 ?? 3%
>> slabinfo.sighand_cache.active_objs
>> ?????? 1954?????????? +24.4%?????? 2431 ?? 3%
>> slabinfo.sighand_cache.num_objs
>> ?????? 2596?????????? +13.7%?????? 2951 ?? 2%
>> slabinfo.signal_cache.active_objs
>> ?????? 2596?????????? +13.7%?????? 2951 ?? 2%
>> slabinfo.signal_cache.num_objs
>> ?????? 4368 ?? 6%???? -11.7%?????? 3856 ?? 2%
>> slabinfo.skbuff_head_cache.num_objs
>> ???? 822.25 ? 11%???? +21.4%???? 998.50 ?? 3%
>> slabinfo.task_group.active_objs
>> ???? 822.25 ? 11%???? +21.4%???? 998.50 ?? 3%
>> slabinfo.task_group.num_objs
>> ?????? 1612?????????? +32.1%?????? 2129 ?? 3%
>> slabinfo.task_struct.active_objs
>> ???? 540.50?????????? +31.6%???? 711.25 ?? 3%
>> slabinfo.task_struct.active_slabs
>> ?????? 1622?????????? +31.6%?????? 2134 ?? 3%
>> slabinfo.task_struct.num_objs
>> ???? 540.50?????????? +31.6%???? 711.25 ?? 3%
>> slabinfo.task_struct.num_slabs
>> ?????? 6905?????????? +20.0%?????? 8289
>> slabinfo.vmap_area.active_objs
>> ?????? 6933?????????? +20.0%?????? 8322
>> slabinfo.vmap_area.num_objs
>> ???? 646.25 ?? 7%??? +195.2%?????? 1907
>> slabinfo.xfs_buf.active_objs
>> ???? 646.25 ?? 7%??? +195.2%?????? 1907??????? slabinfo.xfs_buf.num_objs
>> ?????? 1.78 ?? 2%????? -0.3??????? 1.51 ?? 2%
>> perf-stat.i.branch-miss-rate%
>> ?? 64079048??????????? +3.9%?? 66551536??????? perf-stat.i.branch-misses
>> ????? 17.99??????????? -4.2?????? 13.84
>> perf-stat.i.cache-miss-rate%
>> ?? 66630506?????????? -16.5%?? 55647472??????? perf-stat.i.cache-misses
>> ? 3.899e+08??????????? +9.4%? 4.265e+08
>> perf-stat.i.cache-references
>> ???? 731029?????????? +13.1%???? 826756
>> perf-stat.i.context-switches
>> ?????? 3.79??????????? -1.7%?????? 3.73??????? perf-stat.i.cpi
>> ???? 113320??????????? +8.3%???? 122693??????? perf-stat.i.cpu-migrations
>> ?????? 1498?????????? +20.4%?????? 1803
>> perf-stat.i.cycles-between-cache-misses
>> ?? 50450547 ?? 4%????? -8.0%?? 46402743 ?? 5%
>> perf-stat.i.dTLB-load-misses
>> ?????? 0.20 ?? 3%????? -0.0??????? 0.16 ?? 2%
>> perf-stat.i.dTLB-store-miss-rate%
>> ??? 4587096 ?? 3%???? -22.9%??? 3538422 ?? 2%
>> perf-stat.i.dTLB-store-misses
>> ? 2.297e+09??????????? -5.6%? 2.168e+09??????? perf-stat.i.dTLB-stores
>> ????? 86.68??????????? -2.4?????? 84.25
>> perf-stat.i.iTLB-load-miss-rate%
>> ?? 19484791??????????? -6.1%?? 18297125
>> perf-stat.i.iTLB-load-misses
>> ??? 3060023?????????? +14.4%??? 3501587 ?? 4%? perf-stat.i.iTLB-loads
>> ?????? 0.29??????????? -1.5%?????? 0.28??????? perf-stat.i.ipc
>> ?????? 0.21 ?? 5%???? -20.5%?????? 0.17 ?? 6%? perf-stat.i.metric.K/sec
>> ?????? 3120??????????? -5.8%?????? 2938??????? perf-stat.i.minor-faults
>> ????? 42.61??????????? +1.2?????? 43.83
>> perf-stat.i.node-load-miss-rate%
>> ?? 24887573?????????? -10.1%?? 22384693
>> perf-stat.i.node-load-misses
>> ?? 31814752?????????? -12.5%?? 27830232??????? perf-stat.i.node-loads
>> ????? 36.62??????????? +0.8?????? 37.45
>> perf-stat.i.node-store-miss-rate%
>> ?? 22617213?????????? -20.3%?? 18015607
>> perf-stat.i.node-store-misses
>> ?? 35707079?????????? -19.9%?? 28604167??????? perf-stat.i.node-stores
>> ?????? 3120??????????? -5.8%?????? 2938??????? perf-stat.i.page-faults
>> ????? 13.96??????????? +9.3%????? 15.25??????? perf-stat.overall.MPKI
>> ?????? 0.99??????????? +0.0??????? 1.03
>> perf-stat.overall.branch-miss-rate%
>> ????? 17.09??????????? -4.0?????? 13.05
>> perf-stat.overall.cache-miss-rate%
>> ?????? 1555?????????? +19.4%?????? 1856
>> perf-stat.overall.cycles-between-cache-misses
>> ?????? 0.69 ?? 4%????? -0.1??????? 0.64 ?? 5%
>> perf-stat.overall.dTLB-load-miss-rate%
>> ?????? 0.20 ?? 3%????? -0.0??????? 0.16 ?? 2%
>> perf-stat.overall.dTLB-store-miss-rate%
>> ????? 86.43??????????? -2.5?????? 83.94
>> perf-stat.overall.iTLB-load-miss-rate%
>> ?????? 1433??????????? +6.7%?????? 1528
>> perf-stat.overall.instructions-per-iTLB-miss
>> ????? 43.89??????????? +0.7?????? 44.58
>> perf-stat.overall.node-load-miss-rate%
>> ? 6.389e+09??????????? +0.8%? 6.438e+09
>> perf-stat.ps.branch-instructions
>> ?? 63452800??????????? +4.2%?? 66110454??????? perf-stat.ps.branch-misses
>> ?? 65977234?????????? -16.2%?? 55277568??????? perf-stat.ps.cache-misses
>> ? 3.861e+08??????????? +9.8%? 4.237e+08
>> perf-stat.ps.cache-references
>> ???? 723870?????????? +13.5%???? 821265
>> perf-stat.ps.context-switches
>> ???? 112209??????????? +8.6%???? 121881
>> perf-stat.ps.cpu-migrations
>> ?? 49955941 ?? 4%????? -7.7%?? 46094243 ?? 5%
>> perf-stat.ps.dTLB-load-misses
>> ??? 4542094 ?? 3%???? -22.6%??? 3514899 ?? 2%
>> perf-stat.ps.dTLB-store-misses
>> ? 2.275e+09??????????? -5.3%? 2.154e+09??????? perf-stat.ps.dTLB-stores
>> ?? 19294019??????????? -5.8%?? 18175580
>> perf-stat.ps.iTLB-load-misses
>> ??? 3030049?????????? +14.8%??? 3478314 ?? 4%? perf-stat.ps.iTLB-loads
>> ?????? 3089??????????? -5.5%?????? 2918??????? perf-stat.ps.minor-faults
>> ?? 24643655??????????? -9.8%?? 22235787
>> perf-stat.ps.node-load-misses
>> ?? 31502901?????????? -12.2%?? 27645143??????? perf-stat.ps.node-loads
>> ?? 22395376?????????? -20.1%?? 17895417
>> perf-stat.ps.node-store-misses
>> ?? 35356897?????????? -19.6%?? 28413366??????? perf-stat.ps.node-stores
>> ?????? 3089??????????? -5.5%?????? 2918??????? perf-stat.ps.page-faults
>> ?? 2.85e+12?????????? +50.6%? 4.293e+12
>> perf-stat.total.instructions
>> ????? 84193 ? 36%??? +135.9%???? 198575 ? 25%
>> sched_debug.cfs_rq:/.MIN_vruntime.avg
>> ???? 919053?????????? +98.2%??? 1821460 ?? 2%
>> sched_debug.cfs_rq:/.MIN_vruntime.max
>> ???? 256686 ? 16%??? +114.7%???? 551162 ? 13%
>> sched_debug.cfs_rq:/.MIN_vruntime.stddev
>> ????? 19497????????? +106.7%????? 40307
>> sched_debug.cfs_rq:/.exec_clock.avg
>> ????? 21240????????? +101.1%????? 42717
>> sched_debug.cfs_rq:/.exec_clock.max
>> ????? 19079????????? +107.3%????? 39542
>> sched_debug.cfs_rq:/.exec_clock.min
>> ???? 394.52 ?? 2%???? +54.3%???? 608.79 ? 13%
>> sched_debug.cfs_rq:/.exec_clock.stddev
>> ???? 965.25 ? 15%???? +59.7%?????? 1541 ?? 4%
>> sched_debug.cfs_rq:/.load_avg.avg
>> ?????? 2866 ? 21%???? +57.1%?????? 4503 ? 10%
>> sched_debug.cfs_rq:/.load_avg.max
>> ???? 659.77 ? 15%???? +68.9%?????? 1114 ?? 9%
>> sched_debug.cfs_rq:/.load_avg.stddev
>> ????? 84193 ? 36%??? +135.9%???? 198575 ? 25%
>> sched_debug.cfs_rq:/.max_vruntime.avg
>> ???? 919053?????????? +98.2%??? 1821462 ?? 2%
>> sched_debug.cfs_rq:/.max_vruntime.max
>> ???? 256686 ? 16%??? +114.7%???? 551163 ? 13%
>> sched_debug.cfs_rq:/.max_vruntime.stddev
>> ???? 901765?????????? +97.1%??? 1777741 ?? 2%
>> sched_debug.cfs_rq:/.min_vruntime.avg
>> ???? 941794?????????? +95.8%??? 1844445 ?? 2%
>> sched_debug.cfs_rq:/.min_vruntime.max
>> ???? 872925?????????? +96.5%??? 1715658 ?? 2%
>> sched_debug.cfs_rq:/.min_vruntime.min
>> ????? 18358 ? 44%??? +146.6%????? 45281 ? 34%
>> sched_debug.cfs_rq:/.min_vruntime.stddev
>> ?????? 1.22 ?? 8%??? +159.9%?????? 3.18 ? 20%
>> sched_debug.cfs_rq:/.nr_spread_over.avg
>> ?????? 6.75 ? 23%???? +65.4%????? 11.17 ? 17%
>> sched_debug.cfs_rq:/.nr_spread_over.max
>> ?????? 1.38 ? 11%???? +61.4%?????? 2.22 ? 12%
>> sched_debug.cfs_rq:/.nr_spread_over.stddev
>> ????? 18357 ? 44%??? +146.5%????? 45250 ? 34%
>> sched_debug.cfs_rq:/.spread0.stddev
>> ???? 773.25 ?? 3%???? +13.2%???? 875.02
>> sched_debug.cfs_rq:/.util_avg.avg
>> ???? 285624?????????? -30.3%???? 199111 ?? 4%
>> sched_debug.cpu.avg_idle.avg
>> ???? 576507 ?? 2%???? -26.2%???? 425271 ?? 7%
>> sched_debug.cpu.avg_idle.max
>> ????? 51029 ? 68%???? -83.4%?????? 8469 ? 29%
>> sched_debug.cpu.avg_idle.min
>> ???? 135149 ?? 7%???? -23.4%???? 103460 ? 10%
>> sched_debug.cpu.avg_idle.stddev
>> ????? 57492?????????? +51.0%????? 86835??????? sched_debug.cpu.clock.avg
>> ????? 57495?????????? +51.0%????? 86841??????? sched_debug.cpu.clock.max
>> ????? 57487?????????? +51.0%????? 86828??????? sched_debug.cpu.clock.min
>> ?????? 1.98 ?? 4%???? +74.8%?????? 3.46 ? 10%
>> sched_debug.cpu.clock.stddev
>> ????? 57492?????????? +51.0%????? 86835
>> sched_debug.cpu.clock_task.avg
>> ????? 57495?????????? +51.0%????? 86841
>> sched_debug.cpu.clock_task.max
>> ????? 57487?????????? +51.0%????? 86828
>> sched_debug.cpu.clock_task.min
>> ?????? 1.98 ?? 4%???? +74.8%?????? 3.46 ? 10%
>> sched_debug.cpu.clock_task.stddev
>> ?????? 2831?????????? +43.0%?????? 4049
>> sched_debug.cpu.curr->pid.max
>> ???? 417519????????? +133.3%???? 974078
>> sched_debug.cpu.nr_switches.avg
>> ???? 427854????????? +134.0%??? 1001016
>> sched_debug.cpu.nr_switches.max
>> ???? 409942????????? +131.8%???? 950218
>> sched_debug.cpu.nr_switches.min
>> ?????? 4770 ? 42%??? +238.7%????? 16156 ? 22%
>> sched_debug.cpu.nr_switches.stddev
>> ?????? 4.70 ? 23%???? +37.1%?????? 6.44 ?? 4%
>> sched_debug.cpu.nr_uninterruptible.avg
>> ???? 776.62 ? 19%???? +86.1%?????? 1445 ? 12%
>> sched_debug.cpu.nr_uninterruptible.max
>> ??? -472.62?????????? +98.3%??? -937.33
>> sched_debug.cpu.nr_uninterruptible.min
>> ???? 289.80 ?? 5%??? +109.7%???? 607.68 ?? 3%
>> sched_debug.cpu.nr_uninterruptible.stddev
>> ???? 415893????????? +133.8%???? 972526
>> sched_debug.cpu.sched_count.avg
>> ???? 423960????????? +135.5%???? 998457
>> sched_debug.cpu.sched_count.max
>> ???? 409113????????? +132.0%???? 949068
>> sched_debug.cpu.sched_count.min
>> ?????? 4194 ? 50%??? +274.3%????? 15698 ? 22%
>> sched_debug.cpu.sched_count.stddev
>> ???? 110094????????? +125.8%???? 248571
>> sched_debug.cpu.sched_goidle.avg
>> ???? 112236????????? +125.6%???? 253163
>> sched_debug.cpu.sched_goidle.max
>> ???? 108281????????? +125.8%???? 244506
>> sched_debug.cpu.sched_goidle.min
>> ???? 772.04 ? 16%??? +175.8%?????? 2129 ? 18%
>> sched_debug.cpu.sched_goidle.stddev
>> ???? 297267????????? +137.2%???? 705173
>> sched_debug.cpu.ttwu_count.avg
>> ???? 320112????????? +135.9%???? 755096
>> sched_debug.cpu.ttwu_count.max
>> ???? 272487 ?? 2%??? +142.2%???? 660099
>> sched_debug.cpu.ttwu_count.min
>> ????? 10662 ? 10%??? +110.9%????? 22491 ? 18%
>> sched_debug.cpu.ttwu_count.stddev
>> ????? 12295 ?? 2%??? +834.9%???? 114954 ?? 2%
>> sched_debug.cpu.ttwu_local.avg
>> ????? 13726 ?? 2%??? +769.7%???? 119376 ?? 2%
>> sched_debug.cpu.ttwu_local.max
>> ????? 10620 ?? 4%??? +942.4%???? 110710
>> sched_debug.cpu.ttwu_local.min
>> ???? 693.29 ? 14%??? +228.7%?????? 2278 ? 18%
>> sched_debug.cpu.ttwu_local.stddev
>> ????? 57488?????????? +51.0%????? 86829??????? sched_debug.cpu_clk
>> ????? 56998?????????? +51.5%????? 86337??????? sched_debug.ktime
>> ????? 57977?????????? +50.6%????? 87317??????? sched_debug.sched_clk
>> ????? 72.50 ? 37%???? +55.5%???? 112.75 ? 26%
>> interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
>> ????? 53.25 ?? 2%???? +84.5%????? 98.25 ? 22%
>> interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
>> ????? 73.50 ? 26%??? +116.3%???? 159.00 ? 60%
>> interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
>> ???? 204541?????????? +49.0%???? 304702 ?? 2%
>> interrupts.CPU0.LOC:Local_timer_interrupts
>> ???? 290268 ?? 2%???? +29.1%???? 374641 ?? 4%
>> interrupts.CPU0.RES:Rescheduling_interrupts
>> ???? 204580?????????? +48.9%???? 304667 ?? 2%
>> interrupts.CPU1.LOC:Local_timer_interrupts
>> ???? 290342 ?? 2%???? +28.5%???? 373110 ?? 4%
>> interrupts.CPU1.RES:Rescheduling_interrupts
>> ???? 204537?????????? +48.9%???? 304619 ?? 2%
>> interrupts.CPU10.LOC:Local_timer_interrupts
>> ???? 291605 ?? 2%???? +28.3%???? 374190 ?? 4%
>> interrupts.CPU10.RES:Rescheduling_interrupts
>> ???? 204140?????????? +49.2%???? 304587 ?? 2%
>> interrupts.CPU11.LOC:Local_timer_interrupts
>> ???? 291988 ?? 2%???? +28.1%???? 374024 ?? 4%
>> interrupts.CPU11.RES:Rescheduling_interrupts
>> ???? 204367?????????? +49.0%???? 304559 ?? 2%
>> interrupts.CPU12.LOC:Local_timer_interrupts
>> ???? 288632?????????? +28.4%???? 370669 ?? 3%
>> interrupts.CPU12.RES:Rescheduling_interrupts
>> ???? 204555?????????? +48.9%???? 304669 ?? 2%
>> interrupts.CPU13.LOC:Local_timer_interrupts
>> ???? 289354 ?? 2%???? +28.5%???? 371782 ?? 3%
>> interrupts.CPU13.RES:Rescheduling_interrupts
>> ???? 204536?????????? +49.0%???? 304726 ?? 2%
>> interrupts.CPU14.LOC:Local_timer_interrupts
>> ?????? 4937 ? 25%???? +41.6%?????? 6991 ?? 2%
>> interrupts.CPU14.NMI:Non-maskable_interrupts
>> ?????? 4937 ? 25%???? +41.6%?????? 6991 ?? 2%
>> interrupts.CPU14.PMI:Performance_monitoring_interrupts
>> ???? 287759?????????? +28.2%???? 368986 ?? 3%
>> interrupts.CPU14.RES:Rescheduling_interrupts
>> ???? 204527?????????? +49.0%???? 304707 ?? 2%
>> interrupts.CPU15.LOC:Local_timer_interrupts
>> ???? 286950?????????? +28.6%???? 369104 ?? 3%
>> interrupts.CPU15.RES:Rescheduling_interrupts
>> ???? 204556?????????? +49.0%???? 304718 ?? 2%
>> interrupts.CPU16.LOC:Local_timer_interrupts
>> ?????? 5444 ? 32%???? -19.9%?????? 4360 ? 32%
>> interrupts.CPU16.NMI:Non-maskable_interrupts
>> ?????? 5444 ? 32%???? -19.9%?????? 4360 ? 32%
>> interrupts.CPU16.PMI:Performance_monitoring_interrupts
>> ???? 287623?????????? +28.1%???? 368583 ?? 3%
>> interrupts.CPU16.RES:Rescheduling_interrupts
>> ???? 204528?????????? +49.0%???? 304728 ?? 2%
>> interrupts.CPU17.LOC:Local_timer_interrupts
>> ???? 287734?????????? +28.2%???? 368920 ?? 3%
>> interrupts.CPU17.RES:Rescheduling_interrupts
>> ???? 204465?????????? +49.0%???? 304707 ?? 2%
>> interrupts.CPU18.LOC:Local_timer_interrupts
>> ?????? 5382 ? 31%???? -21.8%?????? 4211 ? 27%
>> interrupts.CPU18.NMI:Non-maskable_interrupts
>> ?????? 5382 ? 31%???? -21.8%?????? 4211 ? 27%
>> interrupts.CPU18.PMI:Performance_monitoring_interrupts
>> ???? 288778?????????? +28.1%???? 369871 ?? 3%
>> interrupts.CPU18.RES:Rescheduling_interrupts
>> ???? 204302?????????? +49.2%???? 304727 ?? 2%
>> interrupts.CPU19.LOC:Local_timer_interrupts
>> ???? 289794?????????? +27.8%???? 370247 ?? 3%
>> interrupts.CPU19.RES:Rescheduling_interrupts
>> ???? 204552?????????? +49.0%???? 304821 ?? 2%
>> interrupts.CPU2.LOC:Local_timer_interrupts
>> ???? 291056 ?? 2%???? +28.3%???? 373290 ?? 4%
>> interrupts.CPU2.RES:Rescheduling_interrupts
>> ???? 204468?????????? +48.9%???? 304484 ?? 2%
>> interrupts.CPU20.LOC:Local_timer_interrupts
>> ???? 290778?????????? +27.6%???? 371106 ?? 3%
>> interrupts.CPU20.RES:Rescheduling_interrupts
>> ???? 204546?????????? +48.9%???? 304619 ?? 2%
>> interrupts.CPU21.LOC:Local_timer_interrupts
>> ???? 289974?????????? +28.0%???? 371074 ?? 3%
>> interrupts.CPU21.RES:Rescheduling_interrupts
>> ???? 204694?????????? +48.8%???? 304581 ?? 2%
>> interrupts.CPU22.LOC:Local_timer_interrupts
>> ???? 290312?????????? +28.0%???? 371734 ?? 3%
>> interrupts.CPU22.RES:Rescheduling_interrupts
>> ???? 204374?????????? +49.0%???? 304493 ?? 2%
>> interrupts.CPU23.LOC:Local_timer_interrupts
>> ???? 290266?????????? +28.5%???? 372865 ?? 3%
>> interrupts.CPU23.RES:Rescheduling_interrupts
>> ???? 204518?????????? +49.0%???? 304653 ?? 2%
>> interrupts.CPU24.LOC:Local_timer_interrupts
>> ???? 288815 ?? 2%???? +28.7%???? 371621 ?? 4%
>> interrupts.CPU24.RES:Rescheduling_interrupts
>> ???? 204414?????????? +49.0%???? 304640 ?? 2%
>> interrupts.CPU25.LOC:Local_timer_interrupts
>> ???? 288779 ?? 2%???? +28.2%???? 370202 ?? 4%
>> interrupts.CPU25.RES:Rescheduling_interrupts
>> ???? 204536?????????? +49.0%???? 304730 ?? 2%
>> interrupts.CPU26.LOC:Local_timer_interrupts
>> ???? 287615 ?? 2%???? +28.6%???? 369843 ?? 4%
>> interrupts.CPU26.RES:Rescheduling_interrupts
>> ????? 72.50 ? 37%???? +55.5%???? 112.75 ? 26%
>> interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
>> ???? 204514?????????? +49.0%???? 304668 ?? 2%
>> interrupts.CPU27.LOC:Local_timer_interrupts
>> ???? 287014 ?? 2%???? +28.5%???? 368825 ?? 4%
>> interrupts.CPU27.RES:Rescheduling_interrupts
>> ????? 53.25 ?? 2%???? +84.5%????? 98.25 ? 22%
>> interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
>> ???? 204331?????????? +49.1%???? 304759 ?? 2%
>> interrupts.CPU28.LOC:Local_timer_interrupts
>> ???? 287278 ?? 2%???? +28.4%???? 368876 ?? 4%
>> interrupts.CPU28.RES:Rescheduling_interrupts
>> ????? 73.50 ? 26%??? +116.3%???? 159.00 ? 60%
>> interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
>> ???? 204427?????????? +49.0%???? 304651 ?? 2%
>> interrupts.CPU29.LOC:Local_timer_interrupts
>> ???? 287382 ?? 2%???? +28.9%???? 370434 ?? 4%
>> interrupts.CPU29.RES:Rescheduling_interrupts
>> ???? 204538?????????? +49.0%???? 304765 ?? 2%
>> interrupts.CPU3.LOC:Local_timer_interrupts
>> ?????? 5034 ? 27%???? -29.4%?????? 3552
>> interrupts.CPU3.NMI:Non-maskable_interrupts
>> ?????? 5034 ? 27%???? -29.4%?????? 3552
>> interrupts.CPU3.PMI:Performance_monitoring_interrupts
>> ???? 288881 ?? 2%???? +28.3%???? 370740 ?? 4%
>> interrupts.CPU3.RES:Rescheduling_interrupts
>> ???? 204750?????????? +48.8%???? 304674 ?? 2%
>> interrupts.CPU30.LOC:Local_timer_interrupts
>> ???? 287139 ?? 2%???? +28.8%???? 369825 ?? 4%
>> interrupts.CPU30.RES:Rescheduling_interrupts
>> ???? 204429?????????? +48.9%???? 304416 ?? 2%
>> interrupts.CPU31.LOC:Local_timer_interrupts
>> ???? 288411 ?? 2%???? +28.4%???? 370199 ?? 4%
>> interrupts.CPU31.RES:Rescheduling_interrupts
>> ???? 204395?????????? +49.3%???? 305106 ?? 2%
>> interrupts.CPU32.LOC:Local_timer_interrupts
>> ???? 289178 ?? 2%???? +28.6%???? 371830 ?? 4%
>> interrupts.CPU32.RES:Rescheduling_interrupts
>> ???? 204347?????????? +49.2%???? 304799 ?? 2%
>> interrupts.CPU33.LOC:Local_timer_interrupts
>> ???? 289689 ?? 2%???? +28.2%???? 371280 ?? 4%
>> interrupts.CPU33.RES:Rescheduling_interrupts
>> ???? 204280?????????? +49.1%???? 304566 ?? 2%
>> interrupts.CPU34.LOC:Local_timer_interrupts
>> ?????? 6418 ? 24%???? -44.5%?????? 3564
>> interrupts.CPU34.NMI:Non-maskable_interrupts
>> ?????? 6418 ? 24%???? -44.5%?????? 3564
>> interrupts.CPU34.PMI:Performance_monitoring_interrupts
>> ???? 290382 ?? 2%???? +28.3%???? 372606 ?? 4%
>> interrupts.CPU34.RES:Rescheduling_interrupts
>> ???? 204331?????????? +49.0%???? 304517 ?? 2%
>> interrupts.CPU35.LOC:Local_timer_interrupts
>> ???? 289736 ?? 2%???? +28.3%???? 371869 ?? 4%
>> interrupts.CPU35.RES:Rescheduling_interrupts
>> ???? 204578?????????? +48.8%???? 304423 ?? 2%
>> interrupts.CPU36.LOC:Local_timer_interrupts
>> ?????? 7298?????????? -39.0%?????? 4450 ? 34%
>> interrupts.CPU36.NMI:Non-maskable_interrupts
>> ?????? 7298?????????? -39.0%?????? 4450 ? 34%
>> interrupts.CPU36.PMI:Performance_monitoring_interrupts
>> ???? 287187?????????? +28.3%???? 368516 ?? 3%
>> interrupts.CPU36.RES:Rescheduling_interrupts
>> ???? 204538?????????? +49.0%???? 304729 ?? 2%
>> interrupts.CPU37.LOC:Local_timer_interrupts
>> ???? 288676?????????? +28.1%???? 369792 ?? 3%
>> interrupts.CPU37.RES:Rescheduling_interrupts
>> ???? 204504?????????? +49.0%???? 304687 ?? 2%
>> interrupts.CPU38.LOC:Local_timer_interrupts
>> ?????? 6301 ? 24%???? -29.8%?????? 4424 ? 34%
>> interrupts.CPU38.NMI:Non-maskable_interrupts
>> ?????? 6301 ? 24%???? -29.8%?????? 4424 ? 34%
>> interrupts.CPU38.PMI:Performance_monitoring_interrupts
>> ???? 287510?????????? +28.1%???? 368349 ?? 3%
>> interrupts.CPU38.RES:Rescheduling_interrupts
>> ???? 204566?????????? +48.9%???? 304668 ?? 2%
>> interrupts.CPU39.LOC:Local_timer_interrupts
>> ???? 286909?????????? +28.1%???? 367622 ?? 3%
>> interrupts.CPU39.RES:Rescheduling_interrupts
>> ???? 204445?????????? +49.0%???? 304686 ?? 2%
>> interrupts.CPU4.LOC:Local_timer_interrupts
>> ???? 288608 ?? 2%???? +28.7%???? 371493 ?? 3%
>> interrupts.CPU4.RES:Rescheduling_interrupts
>> ???? 204538?????????? +48.9%???? 304582 ?? 2%
>> interrupts.CPU40.LOC:Local_timer_interrupts
>> ???? 286219?????????? +27.9%???? 365983 ?? 3%
>> interrupts.CPU40.RES:Rescheduling_interrupts
>> ???? 204551?????????? +49.0%???? 304739 ?? 2%
>> interrupts.CPU41.LOC:Local_timer_interrupts
>> ?????? 5170 ? 29%???? -18.5%?????? 4215 ? 27%
>> interrupts.CPU41.NMI:Non-maskable_interrupts
>> ?????? 5170 ? 29%???? -18.5%?????? 4215 ? 27%
>> interrupts.CPU41.PMI:Performance_monitoring_interrupts
>> ???? 286946?????????? +27.8%???? 366717 ?? 3%
>> interrupts.CPU41.RES:Rescheduling_interrupts
>> ???? 204492?????????? +48.9%???? 304442 ?? 2%
>> interrupts.CPU42.LOC:Local_timer_interrupts
>> ???? 287384?????????? +28.0%???? 367805 ?? 3%
>> interrupts.CPU42.RES:Rescheduling_interrupts
>> ???? 204545?????????? +49.0%???? 304679 ?? 2%
>> interrupts.CPU43.LOC:Local_timer_interrupts
>> ???? 289100?????????? +27.3%???? 368059 ?? 3%
>> interrupts.CPU43.RES:Rescheduling_interrupts
>> ???? 204489?????????? +49.0%???? 304635 ?? 2%
>> interrupts.CPU44.LOC:Local_timer_interrupts
>> ?????? 6393 ? 24%???? -32.4%?????? 4319 ? 31%
>> interrupts.CPU44.NMI:Non-maskable_interrupts
>> ?????? 6393 ? 24%???? -32.4%?????? 4319 ? 31%
>> interrupts.CPU44.PMI:Performance_monitoring_interrupts
>> ???? 288835?????????? +27.8%???? 369182 ?? 3%
>> interrupts.CPU44.RES:Rescheduling_interrupts
>> ???? 204473?????????? +49.2%???? 304983 ?? 2%
>> interrupts.CPU45.LOC:Local_timer_interrupts
>> ???? 289778?????????? +27.4%???? 369059 ?? 3%
>> interrupts.CPU45.RES:Rescheduling_interrupts
>> ???? 204493?????????? +49.0%???? 304598 ?? 2%
>> interrupts.CPU46.LOC:Local_timer_interrupts
>> ???? 289485?????????? +27.9%???? 370172 ?? 3%
>> interrupts.CPU46.RES:Rescheduling_interrupts
>> ???? 204511?????????? +49.2%???? 305090 ?? 2%
>> interrupts.CPU47.LOC:Local_timer_interrupts
>> ???? 286611?????????? +28.8%???? 369021 ?? 3%
>> interrupts.CPU47.RES:Rescheduling_interrupts
>> ???? 204441?????????? +49.1%???? 304794 ?? 2%
>> interrupts.CPU5.LOC:Local_timer_interrupts
>> ???? 289245 ?? 2%???? +28.4%???? 371335 ?? 4%
>> interrupts.CPU5.RES:Rescheduling_interrupts
>> ???? 204246?????????? +49.0%???? 304414 ?? 2%
>> interrupts.CPU6.LOC:Local_timer_interrupts
>> ???? 290006 ?? 2%???? +28.3%???? 372180 ?? 4%
>> interrupts.CPU6.RES:Rescheduling_interrupts
>> ???? 204445?????????? +49.0%???? 304719 ?? 2%
>> interrupts.CPU7.LOC:Local_timer_interrupts
>> ???? 290524 ?? 2%???? +28.2%???? 372414 ?? 4%
>> interrupts.CPU7.RES:Rescheduling_interrupts
>> ???? 204178?????????? +49.2%???? 304663 ?? 2%
>> interrupts.CPU8.LOC:Local_timer_interrupts
>> ???? 291625 ?? 2%???? +27.8%???? 372747 ?? 4%
>> interrupts.CPU8.RES:Rescheduling_interrupts
>> ???? 204502?????????? +48.9%???? 304407 ?? 3%
>> interrupts.CPU9.LOC:Local_timer_interrupts
>> ???? 291615 ?? 2%???? +27.9%???? 373024 ?? 4%
>> interrupts.CPU9.RES:Rescheduling_interrupts
>> ??? 9814627?????????? +49.0%?? 14624016 ?? 2%
>> interrupts.LOC:Local_timer_interrupts
>> ?? 13869791?????????? +28.2%?? 17785831 ?? 3%
>> interrupts.RES:Rescheduling_interrupts
>> ????? 25728?????????? +47.6%????? 37969??????? softirqs.CPU0.RCU
>> ????? 34944?????????? +41.7%????? 49503 ?? 3%? softirqs.CPU0.SCHED
>> ????? 42375?????????? +38.5%????? 58673 ?? 2%? softirqs.CPU0.TIMER
>> ????? 25348?????????? +53.8%????? 38976 ?? 5%? softirqs.CPU1.RCU
>> ????? 30986?????????? +46.2%????? 45315 ?? 3%? softirqs.CPU1.SCHED
>> ????? 41265?????????? +44.3%????? 59538 ?? 3%? softirqs.CPU1.TIMER
>> ????? 25149?????????? +48.3%????? 37288??????? softirqs.CPU10.RCU
>> ????? 29752?????????? +47.1%????? 43764 ?? 2%? softirqs.CPU10.SCHED
>> ????? 40609?????????? +41.5%????? 57449??????? softirqs.CPU10.TIMER
>> ????? 25218?????????? +48.3%????? 37399??????? softirqs.CPU11.RCU
>> ????? 30294?????????? +44.6%????? 43800??????? softirqs.CPU11.SCHED
>> ????? 41371?????????? +39.2%????? 57578??????? softirqs.CPU11.TIMER
>> ????? 24427?????????? +48.9%????? 36361??????? softirqs.CPU12.RCU
>> ????? 30438?????????? +44.0%????? 43835??????? softirqs.CPU12.SCHED
>> ????? 41045 ?? 3%???? +43.0%????? 58687??????? softirqs.CPU12.TIMER
>> ????? 24247?????????? +47.0%????? 35645??????? softirqs.CPU13.RCU
>> ????? 30155?????????? +44.5%????? 43577??????? softirqs.CPU13.SCHED
>> ????? 41148 ?? 2%???? +40.3%????? 57720??????? softirqs.CPU13.TIMER
>> ????? 24506?????????? +46.3%????? 35856??????? softirqs.CPU14.RCU
>> ????? 29991?????????? +44.1%????? 43229??????? softirqs.CPU14.SCHED
>> ????? 41190 ?? 2%???? +39.7%????? 57552??????? softirqs.CPU14.TIMER
>> ????? 24412?????????? +46.7%????? 35800??????? softirqs.CPU15.RCU
>> ????? 30135?????????? +44.1%????? 43430??????? softirqs.CPU15.SCHED
>> ????? 45081 ? 15%???? +28.0%????? 57684??????? softirqs.CPU15.TIMER
>> ????? 25327?????????? +47.0%????? 37225??????? softirqs.CPU16.RCU
>> ????? 30230?????????? +43.8%????? 43461??????? softirqs.CPU16.SCHED
>> ????? 41157?????????? +39.6%????? 57448??????? softirqs.CPU16.TIMER
>> ????? 25136?????????? +47.8%????? 37142??????? softirqs.CPU17.RCU
>> ????? 29894?????????? +44.5%????? 43212??????? softirqs.CPU17.SCHED
>> ????? 41961 ?? 5%???? +36.5%????? 57281??????? softirqs.CPU17.TIMER
>> ????? 25431?????????? +46.1%????? 37160??????? softirqs.CPU18.RCU
>> ????? 30062?????????? +44.0%????? 43281??????? softirqs.CPU18.SCHED
>> ????? 40956 ?? 2%???? +42.1%????? 58217??????? softirqs.CPU18.TIMER
>> ????? 25496?????????? +44.8%????? 36923 ?? 2%? softirqs.CPU19.RCU
>> ????? 29989?????????? +46.5%????? 43946??????? softirqs.CPU19.SCHED
>> ????? 41740 ?? 2%???? +67.8%????? 70057 ? 17%? softirqs.CPU19.TIMER
>> ????? 25414?????????? +48.3%????? 37678??????? softirqs.CPU2.RCU
>> ????? 30905?????????? +43.4%????? 44309 ?? 2%? softirqs.CPU2.SCHED
>> ????? 41083?????????? +40.6%????? 57768 ?? 2%? softirqs.CPU2.TIMER
>> ????? 25124?????????? +46.8%????? 36892??????? softirqs.CPU20.RCU
>> ????? 30001?????????? +44.7%????? 43416??????? softirqs.CPU20.SCHED
>> ????? 41060 ?? 7%???? +39.2%????? 57151??????? softirqs.CPU20.TIMER
>> ????? 24992?????????? +47.6%????? 36896??????? softirqs.CPU21.RCU
>> ????? 30109?????????? +44.4%????? 43467??????? softirqs.CPU21.SCHED
>> ????? 40055 ?? 3%???? +42.5%????? 57081 ?? 2%? softirqs.CPU21.TIMER
>> ????? 25072?????????? +46.2%????? 36656??????? softirqs.CPU22.RCU
>> ????? 30296?????????? +43.3%????? 43425??????? softirqs.CPU22.SCHED
>> ????? 24882?????????? +48.8%????? 37031??????? softirqs.CPU23.RCU
>> ????? 29900?????????? +43.7%????? 42971??????? softirqs.CPU23.SCHED
>> ????? 40388 ?? 3%???? +44.8%????? 58494 ?? 3%? softirqs.CPU23.TIMER
>> ????? 25007?????????? +48.1%????? 37036??????? softirqs.CPU24.RCU
>> ????? 29755?????????? +45.9%????? 43399??????? softirqs.CPU24.SCHED
>> ????? 40786?????????? +40.6%????? 57353??????? softirqs.CPU24.TIMER
>> ????? 25031?????????? +47.3%????? 36875??????? softirqs.CPU25.RCU
>> ????? 29841 ?? 2%???? +45.3%????? 43355??????? softirqs.CPU25.SCHED
>> ????? 40971?????????? +39.8%????? 57266??????? softirqs.CPU25.TIMER
>> ????? 25145?????????? +47.6%????? 37114??????? softirqs.CPU26.RCU
>> ????? 29462 ?? 2%???? +48.5%????? 43739 ?? 2%? softirqs.CPU26.SCHED
>> ????? 40553?????????? +41.8%????? 57521??????? softirqs.CPU26.TIMER
>> ????? 25184?????????? +47.6%????? 37169??????? softirqs.CPU27.RCU
>> ????? 29867?????????? +45.5%????? 43450 ?? 3%? softirqs.CPU27.SCHED
>> ????? 41118?????????? +39.3%????? 57261??????? softirqs.CPU27.TIMER
>> ????? 25452?????????? +47.1%????? 37440??????? softirqs.CPU28.RCU
>> ????? 29714?????????? +46.2%????? 43440 ?? 2%? softirqs.CPU28.SCHED
>> ????? 41209 ?? 2%???? +37.8%????? 56783 ?? 2%? softirqs.CPU28.TIMER
>> ????? 25334?????????? +48.8%????? 37696 ?? 2%? softirqs.CPU29.RCU
>> ????? 29894?????????? +46.3%????? 43749??????? softirqs.CPU29.SCHED
>> ????? 41413 ?? 2%???? +43.3%????? 59346 ?? 5%? softirqs.CPU29.TIMER
>> ????? 25514?????????? +47.0%????? 37495??????? softirqs.CPU3.RCU
>> ????? 30016?????????? +46.0%????? 43831 ?? 2%? softirqs.CPU3.SCHED
>> ????? 41371 ?? 2%???? +39.3%????? 57611??????? softirqs.CPU3.TIMER
>> ????? 25206?????????? +47.4%????? 37149??????? softirqs.CPU30.RCU
>> ????? 30167?????????? +44.0%????? 43451 ?? 2%? softirqs.CPU30.SCHED
>> ????? 41166 ?? 2%???? +58.1%????? 65092 ? 19%? softirqs.CPU30.TIMER
>> ????? 25211?????????? +46.8%????? 37016??????? softirqs.CPU31.RCU
>> ????? 30004 ?? 2%???? +45.2%????? 43569??????? softirqs.CPU31.SCHED
>> ????? 41541 ?? 2%???? +36.9%????? 56874??????? softirqs.CPU31.TIMER
>> ????? 24432?????????? +46.9%????? 35885??????? softirqs.CPU32.RCU
>> ????? 30206?????????? +43.4%????? 43322 ?? 2%? softirqs.CPU32.SCHED
>> ????? 41069?????????? +40.7%????? 57781 ?? 2%? softirqs.CPU32.TIMER
>> ????? 24303?????????? +48.0%????? 35973??????? softirqs.CPU33.RCU
>> ????? 30238?????????? +43.8%????? 43489??????? softirqs.CPU33.SCHED
>> ????? 41161?????????? +38.2%????? 56872??????? softirqs.CPU33.TIMER
>> ????? 24308?????????? +47.6%????? 35876??????? softirqs.CPU34.RCU
>> ????? 29803?????????? +46.5%????? 43647 ?? 2%? softirqs.CPU34.SCHED
>> ????? 40809?????????? +39.2%????? 56817??????? softirqs.CPU34.TIMER
>> ????? 24291?????????? +47.7%????? 35877??????? softirqs.CPU35.RCU
>> ????? 30092?????????? +44.2%????? 43386??????? softirqs.CPU35.SCHED
>> ????? 40607?????????? +41.5%????? 57480 ?? 2%? softirqs.CPU35.TIMER
>> ????? 25605?????????? +49.5%????? 38286??????? softirqs.CPU36.RCU
>> ????? 29900?????????? +44.6%????? 43238??????? softirqs.CPU36.SCHED
>> ????? 45207 ? 15%???? +27.4%????? 57602??????? softirqs.CPU36.TIMER
>> ????? 25555 ?? 2%???? +49.0%????? 38070??????? softirqs.CPU37.RCU
>> ????? 29926?????????? +45.2%????? 43460??????? softirqs.CPU37.SCHED
>> ????? 42619 ?? 8%???? +36.7%????? 58242??????? softirqs.CPU37.TIMER
>> ????? 27683 ?? 9%???? +50.2%????? 41577 ?? 4%? softirqs.CPU38.RCU
>> ????? 29700?????????? +45.6%????? 43230??????? softirqs.CPU38.SCHED
>> ????? 41779 ?? 4%???? +43.5%????? 59948??????? softirqs.CPU38.TIMER
>> ????? 26699 ?? 7%???? +42.1%????? 37951??????? softirqs.CPU39.RCU
>> ????? 30061?????????? +43.4%????? 43102??????? softirqs.CPU39.SCHED
>> ????? 42312 ?? 5%???? +36.1%????? 57581 ?? 2%? softirqs.CPU39.TIMER
>> ????? 25219?????????? +47.7%????? 37247??????? softirqs.CPU4.RCU
>> ????? 30363 ?? 3%???? +45.9%????? 44295??????? softirqs.CPU4.SCHED
>> ????? 41108 ?? 2%???? +39.6%????? 57371 ?? 2%? softirqs.CPU4.TIMER
>> ????? 25413?????????? +50.6%????? 38268??????? softirqs.CPU40.RCU
>> ????? 29910?????????? +46.0%????? 43665??????? softirqs.CPU40.SCHED
>> ????? 39987 ?? 2%???? +45.9%????? 58322??????? softirqs.CPU40.TIMER
>> ????? 25569?????????? +49.5%????? 38238??????? softirqs.CPU41.RCU
>> ????? 29867?????????? +44.8%????? 43247??????? softirqs.CPU41.SCHED
>> ????? 25492?????????? +49.8%????? 38185??????? softirqs.CPU42.RCU
>> ????? 29871?????????? +45.4%????? 43440??????? softirqs.CPU42.SCHED
>> ????? 40076?????????? +44.8%????? 58020??????? softirqs.CPU42.TIMER
>> ????? 25281?????????? +48.9%????? 37640??????? softirqs.CPU43.RCU
>> ????? 30025?????????? +43.9%????? 43220??????? softirqs.CPU43.SCHED
>> ????? 40222 ?? 3%???? +44.7%????? 58195??????? softirqs.CPU43.TIMER
>> ????? 25183?????????? +49.3%????? 37611??????? softirqs.CPU44.RCU
>> ????? 30123?????????? +43.6%????? 43249??????? softirqs.CPU44.SCHED
>> ????? 41090 ?? 6%???? +42.5%????? 58537??????? softirqs.CPU44.TIMER
>> ????? 25314?????????? +49.6%????? 37863??????? softirqs.CPU45.RCU
>> ????? 29998?????????? +43.2%????? 42957??????? softirqs.CPU45.SCHED
>> ????? 39705 ?? 3%???? +43.6%????? 57033??????? softirqs.CPU45.TIMER
>> ????? 25500?????????? +46.2%????? 37274??????? softirqs.CPU46.RCU
>> ????? 29921?????????? +44.7%????? 43286??????? softirqs.CPU46.SCHED
>> ????? 41088 ?? 5%???? +39.9%????? 57463 ?? 3%? softirqs.CPU46.TIMER
>> ????? 25564?????????? +48.4%????? 37931??????? softirqs.CPU47.RCU
>> ????? 30178?????????? +42.4%????? 42980??????? softirqs.CPU47.SCHED
>> ????? 40738 ?? 4%???? +42.2%????? 57939 ?? 3%? softirqs.CPU47.TIMER
>> ????? 26467 ?? 8%???? +40.8%????? 37278??????? softirqs.CPU5.RCU
>> ????? 30334 ?? 2%???? +44.1%????? 43714??????? softirqs.CPU5.SCHED
>> ????? 42268 ?? 4%???? +35.8%????? 57394??????? softirqs.CPU5.TIMER
>> ????? 25343?????????? +47.0%????? 37245??????? softirqs.CPU6.RCU
>> ????? 30100?????????? +45.0%????? 43643 ?? 2%? softirqs.CPU6.SCHED
>> ????? 41270?????????? +43.6%????? 59249 ?? 5%? softirqs.CPU6.TIMER
>> ????? 25331?????????? +47.7%????? 37418??????? softirqs.CPU7.RCU
>> ????? 30210?????????? +44.7%????? 43727 ?? 2%? softirqs.CPU7.SCHED
>> ????? 41028?????????? +41.9%????? 58206 ?? 2%? softirqs.CPU7.TIMER
>> ????? 25483?????????? +47.2%????? 37514??????? softirqs.CPU8.RCU
>> ????? 30324 ?? 2%???? +43.4%????? 43483 ?? 2%? softirqs.CPU8.SCHED
>> ????? 41027?????????? +40.4%????? 57591 ?? 2%? softirqs.CPU8.TIMER
>> ????? 25546?????????? +47.5%????? 37687??????? softirqs.CPU9.RCU
>> ????? 30218?????????? +44.8%????? 43771 ?? 2%? softirqs.CPU9.SCHED
>> ????? 41193?????????? +40.1%????? 57716 ?? 2%? softirqs.CPU9.TIMER
>> ??? 1212606?????????? +47.7%??? 1790808??????? softirqs.RCU
>> ??? 1448198?????????? +44.7%??? 2095500??????? softirqs.SCHED
>> ??? 1990064?????????? +40.7%??? 2800202??????? softirqs.TIMER
>> ?????? 8.56 ?? 3%????? -4.3??????? 4.25
>> perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
>>
>> ?????? 8.50 ?? 3%????? -4.3??????? 4.20
>> perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
>>
>> ?????? 8.47 ?? 3%????? -4.3??????? 4.17
>> perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
>>
>> ????? 78.55??????????? -4.1?????? 74.44
>> perf-profile.calltrace.cycles-pp.write
>> ?????? 8.74 ?? 3%????? -4.0??????? 4.70
>> perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
>>
>> ????? 77.69??????????? -3.9?????? 73.75
>> perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
>> ????? 77.64??????????? -3.9?????? 73.71
>> perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
>>
>> ????? 77.13??????????? -3.8?????? 73.36
>> perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
>>
>> ????? 77.08??????????? -3.8?????? 73.31
>> perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
>>
>> ????? 76.61??????????? -3.7?????? 72.96
>> perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
>>
>> ????? 76.43??????????? -3.6?????? 72.80
>> perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
>>
>> ????? 69.22??????????? -2.6?????? 66.62
>> perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
>>
>> ?????? 4.70 ?? 2%????? -2.2??????? 2.50
>> perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
>>
>> ?????? 4.63 ?? 2%????? -2.2??????? 2.45
>> perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
>>
>> ?????? 4.60 ?? 2%????? -2.2??????? 2.43
>> perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
>>
>> ?????? 4.32 ?? 2%????? -2.1??????? 2.22
>> perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
>>
>> ?????? 3.72 ?? 4%????? -2.1??????? 1.63
>> perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
>>
>> ?????? 3.71 ?? 4%????? -2.1??????? 1.62
>> perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
>>
>> ?????? 2.63 ?? 4%????? -2.0??????? 0.63 ?? 3%
>> perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
>>
>> ?????? 7.72??????????? -2.0??????? 5.74 ?? 2%
>> perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
>>
>> ?????? 3.08 ?? 5%????? -1.9??????? 1.13
>> perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
>>
>> ?????? 2.49 ?? 5%????? -1.9??????? 0.55 ?? 3%
>> perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
>>
>> ?????? 2.48 ?? 5%????? -1.9??????? 0.55 ?? 3%
>> perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
>>
>> ?????? 2.41 ?? 6%????? -1.8??????? 0.61 ?? 2%
>> perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
>>
>> ?????? 4.90??????????? -1.5??????? 3.40 ?? 3%
>> perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>
>> ?????? 4.78??????????? -1.5??????? 3.30 ?? 3%
>> perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
>>
>> ?????? 2.35??????????? -0.6??????? 1.77 ?? 2%
>> perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
>>
>> ?????? 2.36??????????? -0.6??????? 1.79
>> perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
>>
>> ?????? 3.40??????????? -0.5??????? 2.87 ?? 6%
>> perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
>>
>> ?????? 1.83??????????? -0.5??????? 1.33 ?? 2%
>> perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
>>
>> ?????? 3.26??????????? -0.5??????? 2.75 ?? 6%
>> perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
>>
>> ?????? 2.85??????????? -0.4??????? 2.40
>> perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
>>
>> ?????? 2.79??????????? -0.4??????? 2.34
>> perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
>>
>> ?????? 2.83??????????? -0.4??????? 2.38
>> perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
>>
>> ?????? 1.48??????????? -0.4??????? 1.05 ?? 2%
>> perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
>>
>> ?????? 0.61 ?? 2%????? -0.4??????? 0.26 ?100%
>> perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
>>
>> ?????? 1.89??????????? -0.3??????? 1.54
>> perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>
>> ?????? 1.71??????????? -0.3??????? 1.38
>> perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
>>
>> ?????? 1.65??????????? -0.3??????? 1.34
>> perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
>>
>> ?????? 0.69??????????? -0.3??????? 0.38 ? 57%
>> perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
>>
>> ?????? 1.59??????????? -0.3??????? 1.29
>> perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
>>
>> ?????? 0.86??????????? -0.3??????? 0.58 ?? 2%
>> perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
>>
>> ?????? 1.52??????????? -0.3??????? 1.26 ?? 3%
>> perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
>>
>> ?????? 1.75??????????? -0.2??????? 1.50 ?? 2%
>> perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
>>
>> ?????? 1.74??????????? -0.2??????? 1.49 ?? 2%
>> perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
>>
>> ?????? 1.74??????????? -0.2??????? 1.50 ?? 2%
>> perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
>>
>> ?????? 2.15??????????? -0.2??????? 1.94
>> perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
>>
>> ?????? 0.79 ?? 3%????? -0.2??????? 0.60 ?? 4%
>> perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
>>
>> ?????? 1.81??????????? -0.2??????? 1.62
>> perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
>>
>> ?????? 1.87??????????? -0.2??????? 1.68
>> perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
>>
>> ?????? 0.70 ?? 4%????? -0.2??????? 0.54 ?? 4%
>> perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
>>
>> ?????? 0.81??????????? -0.1??????? 0.68
>> perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
>>
>> ?????? 0.65 ?? 2%????? -0.1??????? 0.55 ?? 2%
>> perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
>>
>> ?????? 0.62 ?? 2%????? -0.1??????? 0.52 ?? 3%
>> perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
>>
>> ?????? 1.18??????????? -0.0??????? 1.16
>> perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>
>> ?????? 1.01??????????? -0.0??????? 0.99
>> perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
>>
>> ?????? 1.00??????????? -0.0??????? 0.98
>> perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
>>
>> ?????? 0.96??????????? -0.0??????? 0.93
>> perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
>>
>> ?????? 1.01??????????? -0.0??????? 0.99
>> perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
>>
>> ?????? 0.59 ?? 3%????? +0.1??????? 0.68
>> perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>
>> ?????? 0.58 ?? 3%????? +0.1??????? 0.67
>> perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
>>
>> ?????? 0.53??????????? +0.1??????? 0.63 ?? 3%
>> perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
>>
>> ?????? 0.91 ?? 3%????? +0.1??????? 1.02 ?? 2%
>> perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
>>
>> ?????? 0.54??????????? +0.1??????? 0.65 ?? 3%
>> perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
>>
>> ?????? 0.57??????????? +0.1??????? 0.69 ?? 3%
>> perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
>>
>> ?????? 1.83??????????? +0.2??????? 1.99
>> perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
>>
>> ?????? 0.00??????????? +0.6??????? 0.57 ?? 2%
>> perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
>>
>> ?????? 0.00??????????? +0.6??????? 0.57 ?? 2%
>> perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
>>
>> ?????? 0.00??????????? +0.6??????? 0.57 ?? 2%
>> perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
>>
>> ?????? 0.00??????????? +0.7??????? 0.70 ?? 2%
>> perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
>>
>> ?????? 0.00??????????? +0.8??????? 0.75 ?? 4%
>> perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
>>
>> ?????? 0.00??????????? +0.8??????? 0.80
>> perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
>>
>> ?????? 0.00??????????? +0.8??????? 0.80
>> perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
>>
>> ?????? 0.00??????????? +0.8??????? 0.80 ? 30%
>> perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
>>
>> ?????? 0.00??????????? +0.8??????? 0.82
>> perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
>>
>> ?????? 0.00??????????? +0.8??????? 0.85
>> perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
>>
>> ?????? 0.00??????????? +0.9??????? 0.89 ?? 2%
>> perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
>>
>> ?????? 0.00??????????? +0.9??????? 0.92 ? 21%
>> perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
>>
>> ?????? 0.00??????????? +1.0??????? 0.97 ? 20%
>> perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
>>
>> ????? 35.68??????????? +1.0?????? 36.68
>> perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
>>
>> ?????? 0.00??????????? +1.0??????? 1.04
>> perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
>>
>> ?????? 0.00??????????? +1.0??????? 1.05
>> perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
>>
>> ?????? 9.35??????????? +1.2?????? 10.54
>> perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
>>
>> ?????? 9.61??????????? +1.2?????? 10.85
>> perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
>>
>> ?????? 9.63??????????? +1.2?????? 10.87
>> perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>
>> ????? 12.24??????????? +1.3?????? 13.58
>> perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>
>> ????? 12.25??????????? +1.3?????? 13.60
>> perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
>> ????? 12.25??????????? +1.3?????? 13.60
>> perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
>>
>> ????? 12.49??????????? +1.4?????? 13.89
>> perf-profile.calltrace.cycles-pp.secondary_startup_64
>> ????? 14.38??????????? +1.4?????? 15.81 ?? 2%
>> perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
>>
>> ????? 13.41??????????? +1.4?????? 14.86 ?? 2%
>> perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>
>> ????? 13.37??????????? +1.5?????? 14.82 ?? 2%
>> perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
>>
>> ????? 13.34??????????? +1.5?????? 14.80 ?? 2%
>> perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
>>
>> ????? 58.91??????????? +1.6?????? 60.56
>> perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
>>
>> ?????? 0.00??????????? +2.2??????? 2.24 ? 13%
>> perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
>>
>> ?????? 5.63??????????? +2.5??????? 8.15
>> perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
>>
>> ?????? 5.65??????????? +2.5??????? 8.19
>> perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>
>> ?????? 5.69??????????? +2.5??????? 8.23
>> perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
>>
>> ?????? 7.04??????????? +2.8??????? 9.80 ?? 2%
>> perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
>>
>> ?????? 7.54??????????? +2.9?????? 10.45 ?? 2%
>> perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
>> ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
>> perf-profile.calltrace.cycles-pp.ret_from_fork
>> ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
>> perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
>> ?????? 0.00??????????? +3.3??????? 3.30 ?? 9%
>> perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
>>
>> ?????? 0.00??????????? +3.3??????? 3.32 ?? 8%
>> perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
>>
>> ?????? 8.56 ?? 3%????? -4.3??????? 4.25
>> perf-profile.children.cycles-pp.__filemap_fdatawrite_range
>> ?????? 8.47 ?? 3%????? -4.3??????? 4.18
>> perf-profile.children.cycles-pp.xfs_vm_writepages
>> ?????? 8.50 ?? 3%????? -4.3??????? 4.20
>> perf-profile.children.cycles-pp.do_writepages
>> ????? 78.64??????????? -4.1?????? 74.51
>> perf-profile.children.cycles-pp.write
>> ????? 78.55??????????? -4.1?????? 74.44
>> perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
>> ????? 78.50??????????? -4.1?????? 74.41
>> perf-profile.children.cycles-pp.do_syscall_64
>> ?????? 8.75 ?? 3%????? -4.1??????? 4.70
>> perf-profile.children.cycles-pp.file_write_and_wait_range
>> ????? 77.14??????????? -3.8?????? 73.37
>> perf-profile.children.cycles-pp.ksys_write
>> ????? 77.09??????????? -3.8?????? 73.33
>> perf-profile.children.cycles-pp.vfs_write
>> ????? 76.62??????????? -3.7?????? 72.97
>> perf-profile.children.cycles-pp.new_sync_write
>> ????? 76.44??????????? -3.6?????? 72.81
>> perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
>> ????? 69.23??????????? -2.6?????? 66.62
>> perf-profile.children.cycles-pp.xfs_file_fsync
>> ?????? 4.70 ?? 2%????? -2.2??????? 2.50
>> perf-profile.children.cycles-pp.iomap_submit_ioend
>> ?????? 4.89 ?? 2%????? -2.1??????? 2.75
>> perf-profile.children.cycles-pp.submit_bio
>> ?????? 4.85 ?? 2%????? -2.1??????? 2.71
>> perf-profile.children.cycles-pp.generic_make_request
>> ?????? 3.73 ?? 4%????? -2.1??????? 1.63
>> perf-profile.children.cycles-pp.iomap_writepages
>> ?????? 3.72 ?? 4%????? -2.1??????? 1.63
>> perf-profile.children.cycles-pp.write_cache_pages
>> ?????? 4.43 ?? 2%????? -2.1??????? 2.38
>> perf-profile.children.cycles-pp.brd_make_request
>> ?????? 7.73??????????? -2.0??????? 5.75 ?? 2%
>> perf-profile.children.cycles-pp.__flush_work
>> ?????? 3.08 ?? 4%????? -1.9??????? 1.14
>> perf-profile.children.cycles-pp.iomap_writepage_map
>> ?????? 6.00??????????? -1.8??????? 4.20 ?? 2%
>> perf-profile.children.cycles-pp._raw_spin_lock_irq
>> ?????? 2.50 ?? 4%????? -1.8??????? 0.70 ?? 3%
>> perf-profile.children.cycles-pp.end_page_writeback
>> ?????? 2.50 ?? 5%????? -1.8??????? 0.70 ?? 4%
>> perf-profile.children.cycles-pp.test_clear_page_writeback
>> ?????? 2.42 ?? 6%????? -1.8??????? 0.61 ?? 2%
>> perf-profile.children.cycles-pp.__test_set_page_writeback
>> ?????? 2.08 ?? 7%????? -1.6??????? 0.44 ?? 4%
>> perf-profile.children.cycles-pp.sb_mark_inode_writeback
>> ?????? 2.08 ?? 6%????? -1.6??????? 0.44 ?? 5%
>> perf-profile.children.cycles-pp.sb_clear_inode_writeback
>> ?????? 2.70 ?? 4%????? -1.0??????? 1.67 ?? 2%
>> perf-profile.children.cycles-pp.iomap_finish_ioend
>> ?????? 2.36??????????? -0.6??????? 1.79
>> perf-profile.children.cycles-pp.iomap_file_buffered_write
>> ?????? 2.35??????????? -0.6??????? 1.78
>> perf-profile.children.cycles-pp.iomap_apply
>> ?????? 1.84??????????? -0.5??????? 1.33 ?? 2%
>> perf-profile.children.cycles-pp.iomap_write_actor
>> ?????? 2.41 ?? 4%????? -0.5??????? 1.94 ?? 2%
>> perf-profile.children.cycles-pp.reschedule_interrupt
>> ?????? 3.44 ?? 2%????? -0.5??????? 2.98
>> perf-profile.children.cycles-pp.sched_ttwu_pending
>> ?????? 2.87??????????? -0.5??????? 2.42
>> perf-profile.children.cycles-pp.complete
>> ?????? 2.83??????????? -0.4??????? 2.38
>> perf-profile.children.cycles-pp.swake_up_locked
>> ?????? 2.28 ?? 4%????? -0.4??????? 1.84 ?? 2%
>> perf-profile.children.cycles-pp.scheduler_ipi
>> ?????? 1.56??????????? -0.4??????? 1.17 ?? 2%
>> perf-profile.children.cycles-pp.brd_do_bvec
>> ?????? 1.89??????????? -0.4??????? 1.54
>> perf-profile.children.cycles-pp.wait_for_completion
>> ?????? 1.71??????????? -0.3??????? 1.38
>> perf-profile.children.cycles-pp.schedule_timeout
>> ?????? 0.88??????????? -0.2??????? 0.68 ?? 2%
>> perf-profile.children.cycles-pp.rwsem_down_read_slowpath
>> ?????? 0.69??????????? -0.2??????? 0.51 ?? 2%
>> perf-profile.children.cycles-pp.iomap_write_begin
>> ?????? 0.67 ?? 2%????? -0.2??????? 0.49
>> perf-profile.children.cycles-pp.lseek64
>> ?????? 1.33 ?? 5%????? -0.2??????? 1.17 ?? 3%
>> perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
>> ?????? 0.51??????????? -0.2??????? 0.36
>> perf-profile.children.cycles-pp.grab_cache_page_write_begin
>> ?????? 0.47??????????? -0.1??????? 0.33
>> perf-profile.children.cycles-pp.pagecache_get_page
>> ?????? 0.51 ?? 3%????? -0.1??????? 0.38 ?? 2%
>> perf-profile.children.cycles-pp.iomap_write_end
>> ?????? 0.57 ?? 2%????? -0.1??????? 0.46
>> perf-profile.children.cycles-pp.blkdev_issue_flush
>> ?????? 0.38??????????? -0.1??????? 0.26 ?? 5%
>> perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
>> ?????? 0.34??????????? -0.1??????? 0.23 ?? 6%
>> perf-profile.children.cycles-pp.copyin
>> ?????? 0.33??????????? -0.1??????? 0.22 ?? 6%
>> perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
>> ?????? 0.65 ?? 2%????? -0.1??????? 0.56 ?? 2%
>> perf-profile.children.cycles-pp.rwsem_optimistic_spin
>> ?????? 0.37??????????? -0.1??????? 0.28 ?? 2%
>> perf-profile.children.cycles-pp.bio_alloc_bioset
>> ?????? 0.31 ?? 4%????? -0.1??????? 0.22
>> perf-profile.children.cycles-pp.syscall_return_via_sysret
>> ?????? 0.56 ?? 3%????? -0.1??????? 0.48 ?? 3%
>> perf-profile.children.cycles-pp.osq_lock
>> ?????? 0.26 ?? 3%????? -0.1??????? 0.18 ?? 4%
>> perf-profile.children.cycles-pp.entry_SYSCALL_64
>> ?????? 0.28 ?? 2%????? -0.1??????? 0.21 ?? 5%
>> perf-profile.children.cycles-pp.mempool_alloc
>> ?????? 0.26 ?? 6%????? -0.1??????? 0.18 ?? 8%
>> perf-profile.children.cycles-pp.xlog_state_get_iclog_space
>> ?????? 0.29??????????? -0.1??????? 0.22 ?? 3%
>> perf-profile.children.cycles-pp.find_get_entry
>> ?????? 0.34 ?? 2%????? -0.1??????? 0.27 ?? 4%
>> perf-profile.children.cycles-pp.generic_make_request_checks
>> ?????? 0.30 ?? 5%????? -0.1??????? 0.23 ?? 3%
>> perf-profile.children.cycles-pp.iomap_set_page_dirty
>> ?????? 0.32 ?? 2%????? -0.1??????? 0.26 ?? 5%
>> perf-profile.children.cycles-pp.brd_lookup_page
>> ?????? 0.18 ?? 2%????? -0.1??????? 0.12 ?? 6%
>> perf-profile.children.cycles-pp.iomap_set_range_uptodate
>> ?????? 0.15 ?? 2%????? -0.1??????? 0.09 ?? 4%
>> perf-profile.children.cycles-pp.__mod_lruvec_state
>> ?????? 0.15 ?? 5%????? -0.1??????? 0.10 ?? 9%
>> perf-profile.children.cycles-pp.wake_up_q
>> ?????? 0.21 ?? 5%????? -0.1??????? 0.15 ?? 5%
>> perf-profile.children.cycles-pp.xfs_ilock
>> ?????? 0.13 ?? 3%????? -0.0??????? 0.08 ? 10%
>> perf-profile.children.cycles-pp.rwsem_wake
>> ?????? 0.20 ?? 2%????? -0.0??????? 0.15 ?? 5%
>> perf-profile.children.cycles-pp.ksys_lseek
>> ?????? 0.20 ?? 3%????? -0.0??????? 0.15 ?? 3%
>> perf-profile.children.cycles-pp.xas_load
>> ?????? 0.20 ?? 5%????? -0.0??????? 0.16 ?? 2%
>> perf-profile.children.cycles-pp.__set_page_dirty
>> ?????? 0.19 ?? 2%????? -0.0??????? 0.15 ?? 2%
>> perf-profile.children.cycles-pp.xfs_iunlock
>> ?????? 0.40??????????? -0.0??????? 0.36 ?? 4%
>> perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
>> ?????? 0.17 ?? 2%????? -0.0??????? 0.12 ?? 4%
>> perf-profile.children.cycles-pp.clear_page_dirty_for_io
>> ?????? 0.20 ?? 3%????? -0.0??????? 0.16 ?? 4%
>> perf-profile.children.cycles-pp.submit_bio_wait
>> ?????? 0.18 ?? 4%????? -0.0??????? 0.15 ?? 7%
>> perf-profile.children.cycles-pp.security_file_permission
>> ?????? 0.10??????????? -0.0??????? 0.06 ?? 6%
>> perf-profile.children.cycles-pp.evict
>> ?????? 0.10??????????? -0.0??????? 0.06 ?? 6%
>> perf-profile.children.cycles-pp.truncate_inode_pages_range
>> ?????? 0.06 ?? 6%????? -0.0??????? 0.03 ?100%
>> perf-profile.children.cycles-pp.vfs_unlink
>> ?????? 0.08 ?? 6%????? -0.0??????? 0.04 ? 58%
>> perf-profile.children.cycles-pp.add_to_page_cache_lru
>> ?????? 0.26 ?? 2%????? -0.0??????? 0.23 ?? 3%
>> perf-profile.children.cycles-pp.reweight_entity
>> ?????? 0.12 ?? 6%????? -0.0??????? 0.09 ?? 4%
>> perf-profile.children.cycles-pp.bio_endio
>> ?????? 0.10 ?? 7%????? -0.0??????? 0.07 ?? 7%
>> perf-profile.children.cycles-pp.do_sys_open
>> ?????? 0.10 ?? 7%????? -0.0??????? 0.07 ?? 7%
>> perf-profile.children.cycles-pp.do_sys_openat2
>> ?????? 0.17 ?? 2%????? -0.0??????? 0.14
>> perf-profile.children.cycles-pp.__pagevec_release
>> ?????? 0.13 ?? 5%????? -0.0??????? 0.10 ?? 4%
>> perf-profile.children.cycles-pp.down_write
>> ?????? 0.10 ?? 5%????? -0.0??????? 0.06 ?? 6%
>> perf-profile.children.cycles-pp.do_filp_open
>> ?????? 0.15 ?? 4%????? -0.0??????? 0.12 ?? 5%
>> perf-profile.children.cycles-pp.tag_pages_for_writeback
>> ?????? 0.11 ?? 9%????? -0.0??????? 0.08 ?? 5%
>> perf-profile.children.cycles-pp.bio_associate_blkg
>> ?????? 0.10 ?? 5%????? -0.0??????? 0.07 ?? 7%
>> perf-profile.children.cycles-pp.open64
>> ?????? 0.09 ?? 4%????? -0.0??????? 0.06 ?? 6%
>> perf-profile.children.cycles-pp.path_openat
>> ?????? 0.16 ?? 8%????? -0.0??????? 0.13 ?? 3%
>> perf-profile.children.cycles-pp.pagevec_lookup_range_tag
>> ?????? 0.12 ?? 8%????? -0.0??????? 0.10 ?? 5%
>> perf-profile.children.cycles-pp.account_page_dirtied
>> ?????? 0.10 ?? 4%????? -0.0??????? 0.08 ?? 6%
>> perf-profile.children.cycles-pp.__fdget_pos
>> ?????? 0.14 ?? 3%????? -0.0??????? 0.11 ?? 7%
>> perf-profile.children.cycles-pp.memset_erms
>> ?????? 0.11 ?? 3%????? -0.0??????? 0.09 ?? 4%
>> perf-profile.children.cycles-pp.switch_fpu_return
>> ?????? 0.23??????????? -0.0??????? 0.21 ?? 3%
>> perf-profile.children.cycles-pp.__radix_tree_lookup
>> ?????? 0.18 ?? 2%????? -0.0??????? 0.16 ?? 9%
>> perf-profile.children.cycles-pp.insert_work
>> ?????? 0.12 ?? 4%????? -0.0??????? 0.10 ?? 7%
>> perf-profile.children.cycles-pp.common_file_perm
>> ?????? 0.09 ?? 4%????? -0.0??????? 0.07 ?? 6%
>> perf-profile.children.cycles-pp.__fget_light
>> ?????? 0.09 ?? 4%????? -0.0??????? 0.07 ?? 6%
>> perf-profile.children.cycles-pp.bvec_alloc
>> ?????? 0.15 ?? 7%????? -0.0??????? 0.12 ?? 3%
>> perf-profile.children.cycles-pp.find_get_pages_range_tag
>> ?????? 0.08 ?? 8%????? -0.0??????? 0.06 ?? 7%
>> perf-profile.children.cycles-pp.iov_iter_fault_in_readable
>> ?????? 0.09 ?? 5%????? -0.0??????? 0.06 ? 13%
>> perf-profile.children.cycles-pp.blk_throtl_bio
>> ?????? 0.24 ?? 4%????? -0.0??????? 0.22 ?? 3%
>> perf-profile.children.cycles-pp._cond_resched
>> ?????? 0.13 ?? 3%????? -0.0??????? 0.11 ?? 4%
>> perf-profile.children.cycles-pp.release_pages
>> ?????? 0.10 ?? 4%????? -0.0??????? 0.08 ?? 6%
>> perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
>> ?????? 0.08 ? 10%????? -0.0??????? 0.06
>> perf-profile.children.cycles-pp.unlink
>> ?????? 0.08 ? 10%????? -0.0??????? 0.06 ? 11%
>> perf-profile.children.cycles-pp.__sb_start_write
>> ?????? 0.08 ?? 5%????? -0.0??????? 0.06 ?? 7%
>> perf-profile.children.cycles-pp.generic_write_checks
>> ?????? 0.08 ?? 8%????? -0.0??????? 0.06
>> perf-profile.children.cycles-pp.do_unlinkat
>> ?????? 0.07 ? 10%????? -0.0??????? 0.05
>> perf-profile.children.cycles-pp.xfs_break_layouts
>> ?????? 0.18 ?? 2%????? -0.0??????? 0.16 ?? 2%
>> perf-profile.children.cycles-pp.__list_add_valid
>> ?????? 0.12 ?? 3%????? -0.0??????? 0.10 ?? 8%
>> perf-profile.children.cycles-pp.__might_sleep
>> ?????? 0.14 ?? 3%????? -0.0??????? 0.12 ?? 7%
>> perf-profile.children.cycles-pp.xas_find_marked
>> ?????? 0.07 ? 10%????? -0.0??????? 0.05 ?? 8%
>> perf-profile.children.cycles-pp.bio_associate_blkg_from_css
>> ?????? 0.08 ?? 5%????? -0.0??????? 0.06 ? 11%
>> perf-profile.children.cycles-pp.bio_free
>> ?????? 0.08 ?? 6%????? -0.0??????? 0.06
>> perf-profile.children.cycles-pp.xfs_file_llseek
>> ?????? 0.08 ?? 5%????? -0.0??????? 0.06 ?? 6%
>> perf-profile.children.cycles-pp.xfs_file_write_iter
>> ?????? 0.16 ?? 2%????? +0.0??????? 0.17 ?? 2%
>> perf-profile.children.cycles-pp.__switch_to_asm
>> ?????? 0.05??????????? +0.0??????? 0.07 ?? 7%
>> perf-profile.children.cycles-pp.cpumask_next_wrap
>> ?????? 0.10 ?? 4%????? +0.0??????? 0.12 ?? 3%
>> perf-profile.children.cycles-pp._find_next_bit
>> ?????? 0.07 ?? 7%????? +0.0??????? 0.09 ?? 5%
>> perf-profile.children.cycles-pp.xlog_state_done_syncing
>> ?????? 0.16 ?? 2%????? +0.0??????? 0.19 ?? 2%
>> perf-profile.children.cycles-pp.check_preempt_curr
>> ?????? 0.19 ?? 4%????? +0.0??????? 0.22
>> perf-profile.children.cycles-pp.ttwu_do_wakeup
>> ?????? 0.56 ?? 3%????? +0.0??????? 0.60
>> perf-profile.children.cycles-pp.dequeue_entity
>> ?????? 0.20 ?? 6%????? +0.0??????? 0.23 ?? 4%
>> perf-profile.children.cycles-pp.xfs_trans_committed_bulk
>> ?????? 0.07 ?? 7%????? +0.0??????? 0.10 ?? 7%
>> perf-profile.children.cycles-pp.check_preempt_wakeup
>> ?????? 0.82 ?? 2%????? +0.0??????? 0.86
>> perf-profile.children.cycles-pp.update_load_avg
>> ?????? 0.31 ?? 3%????? +0.0??????? 0.35
>> perf-profile.children.cycles-pp.__unwind_start
>> ?????? 0.08 ?? 8%????? +0.0??????? 0.12 ?? 3%
>> perf-profile.children.cycles-pp.destroy_inode
>> ?????? 0.08 ? 10%????? +0.0??????? 0.12 ?? 3%
>> perf-profile.children.cycles-pp.xfs_fs_destroy_inode
>> ?????? 0.08 ? 10%????? +0.0??????? 0.12 ?? 3%
>> perf-profile.children.cycles-pp.xfs_inactive
>> ?????? 0.08 ? 15%????? +0.0??????? 0.12 ?? 5%
>> perf-profile.children.cycles-pp.xfs_bmap_validate_extent
>> ?????? 3.96??????????? +0.0??????? 4.01
>> perf-profile.children.cycles-pp.stack_trace_save_tsk
>> ?????? 0.28??????????? +0.0??????? 0.33
>> perf-profile.children.cycles-pp.set_next_entity
>> ?????? 0.00??????????? +0.1??????? 0.05
>> perf-profile.children.cycles-pp.xfs_trans_roll
>> ?????? 0.00??????????? +0.1??????? 0.05 ?? 8%
>> perf-profile.children.cycles-pp.xfs_btree_delrec
>> ?????? 0.00??????????? +0.1??????? 0.05 ?? 8%
>> perf-profile.children.cycles-pp.xfs_btree_insert
>> ?????? 0.16 ?? 4%????? +0.1??????? 0.21 ?? 5%
>> perf-profile.children.cycles-pp.memcpy_erms
>> ?????? 0.04 ? 57%????? +0.1??????? 0.10 ?? 4%
>> perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
>> ?????? 0.00??????????? +0.1??????? 0.06 ? 11%
>> perf-profile.children.cycles-pp.xfs_buf_item_format_segment
>> ?????? 0.00??????????? +0.1??????? 0.06 ? 11%
>> perf-profile.children.cycles-pp.xfs_btree_delete
>> ?????? 0.00??????????? +0.1??????? 0.06
>> perf-profile.children.cycles-pp.xfs_btree_update
>> ?????? 0.00??????????? +0.1??????? 0.07 ?? 7%
>> perf-profile.children.cycles-pp.xfs_cil_prepare_item
>> ?????? 0.00??????????? +0.1??????? 0.07 ?? 6%
>> perf-profile.children.cycles-pp.xfs_bmbt_update
>> ?????? 0.00??????????? +0.1??????? 0.07 ? 10%
>> perf-profile.children.cycles-pp.xfs_buf_item_format
>> ?????? 0.56??????????? +0.1??????? 0.64 ?? 2%
>> perf-profile.children.cycles-pp.pick_next_task_fair
>> ?????? 0.03 ?100%????? +0.1??????? 0.10 ?? 4%
>> perf-profile.children.cycles-pp.xfs_inactive_truncate
>> ?????? 0.08 ?? 5%????? +0.1??????? 0.16 ? 11%
>> perf-profile.children.cycles-pp.xfs_iext_lookup_extent
>> ?????? 0.00??????????? +0.1??????? 0.08 ?? 5%
>> perf-profile.children.cycles-pp.xfs_defer_finish
>> ?????? 0.00??????????? +0.1??????? 0.08 ?? 5%
>> perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
>> ?????? 0.00??????????? +0.1??????? 0.08 ?? 5%
>> perf-profile.children.cycles-pp.xfs_verify_agbno
>> ?????? 0.00??????????? +0.1??????? 0.09 ?? 9%
>> perf-profile.children.cycles-pp.xfs_defer_finish_noroll
>> ?????? 0.60 ?? 3%????? +0.1??????? 0.70
>> perf-profile.children.cycles-pp.schedule_idle
>> ?????? 0.39 ?? 2%????? +0.1??????? 0.49
>> perf-profile.children.cycles-pp.xlog_state_release_iclog
>> ?????? 0.00??????????? +0.1??????? 0.10 ?? 9%
>> perf-profile.children.cycles-pp.xfs_buf_find
>> ?????? 0.93 ?? 3%????? +0.1??????? 1.03 ?? 2%
>> perf-profile.children.cycles-pp.xlog_write
>> ?????? 2.58??????????? +0.1??????? 2.69
>> perf-profile.children.cycles-pp.schedule
>> ?????? 1.52??????????? +0.1??????? 1.63
>> perf-profile.children.cycles-pp.__wake_up_common_lock
>> ?????? 0.00??????????? +0.1??????? 0.11 ?? 7%
>> perf-profile.children.cycles-pp.xfs_buf_get_map
>> ?????? 4.62??????????? +0.1??????? 4.73
>> perf-profile.children.cycles-pp.__account_scheduler_latency
>> ?????? 0.00??????????? +0.1??????? 0.12 ? 10%
>> perf-profile.children.cycles-pp.xfs_buf_read_map
>> ?????? 1.83??????????? +0.2??????? 1.99
>> perf-profile.children.cycles-pp.xlog_cil_push_work
>> ?????? 5.94??????????? +0.2??????? 6.12
>> perf-profile.children.cycles-pp.ttwu_do_activate
>> ?????? 0.00??????????? +0.2??????? 0.17 ?? 6%
>> perf-profile.children.cycles-pp.xfs_btree_read_buf_block
>> ?????? 5.93??????????? +0.2??????? 6.11
>> perf-profile.children.cycles-pp.activate_task
>> ?????? 5.88??????????? +0.2??????? 6.06
>> perf-profile.children.cycles-pp.enqueue_task_fair
>> ?????? 0.00??????????? +0.2??????? 0.18 ?? 5%
>> perf-profile.children.cycles-pp.xfs_trans_read_buf_map
>> ?????? 5.41??????????? +0.2??????? 5.59
>> perf-profile.children.cycles-pp.enqueue_entity
>> ?????? 3.14??????????? +0.2??????? 3.33
>> perf-profile.children.cycles-pp.__sched_text_start
>> ?????? 0.00??????????? +0.2??????? 0.21 ?? 6%
>> perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
>> ?????? 0.00??????????? +0.2??????? 0.23 ? 57%
>> perf-profile.children.cycles-pp.xlog_grant_head_wait
>> ?????? 0.00??????????? +0.2??????? 0.24 ?? 6%
>> perf-profile.children.cycles-pp.io_schedule
>> ?????? 0.11 ?? 3%????? +0.3??????? 0.38 ?? 4%
>> perf-profile.children.cycles-pp.__filemap_fdatawait_range
>> ?????? 0.00??????????? +0.3??????? 0.27 ?? 8%
>> perf-profile.children.cycles-pp.wait_on_page_bit
>> ?????? 0.00??????????? +0.3??????? 0.34 ?? 5%
>> perf-profile.children.cycles-pp.xfs_btree_lookup
>> ?????? 0.08 ? 14%????? +0.4??????? 0.43
>> perf-profile.children.cycles-pp.__queue_work
>> ?????? 0.08 ?? 8%????? +0.4??????? 0.44
>> perf-profile.children.cycles-pp.queue_work_on
>> ?????? 0.00??????????? +0.4??????? 0.41
>> perf-profile.children.cycles-pp.xfs_end_bio
>> ?????? 3.51??????????? +0.4??????? 3.94 ?? 9%
>> perf-profile.children.cycles-pp.__xfs_trans_commit
>> ?????? 0.20 ?? 9%????? +0.5??????? 0.72 ? 54%
>> perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
>> ?????? 0.06 ? 11%????? +0.5??????? 0.58 ? 67%
>> perf-profile.children.cycles-pp.xfs_log_space_wake
>> ?????? 0.45 ?? 5%????? +0.6??????? 1.04 ? 36%
>> perf-profile.children.cycles-pp.xfs_log_reserve
>> ?????? 0.65 ?? 3%????? +0.6??????? 1.25 ? 29%
>> perf-profile.children.cycles-pp.xfs_trans_alloc
>> ?????? 0.48 ?? 5%????? +0.6??????? 1.09 ? 34%
>> perf-profile.children.cycles-pp.xfs_trans_reserve
>> ?????? 0.04 ? 57%????? +0.6??????? 0.65 ? 60%
>> perf-profile.children.cycles-pp.xlog_grant_head_check
>> ?????? 4.83??????????? +0.7??????? 5.52
>> perf-profile.children.cycles-pp.try_to_wake_up
>> ?????? 0.00??????????? +0.7??????? 0.70 ?? 2%
>> perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
>> ?????? 0.00??????????? +0.8??????? 0.75 ?? 4%
>> perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
>> ?????? 0.04 ? 57%????? +0.8??????? 0.80
>> perf-profile.children.cycles-pp.autoremove_wake_function
>> ?????? 0.05 ?? 8%????? +0.8??????? 0.85
>> perf-profile.children.cycles-pp.wake_up_page_bit
>> ?????? 1.50??????????? +0.9??????? 2.36
>> perf-profile.children.cycles-pp.__wake_up_common
>> ?????? 0.00??????????? +0.9??????? 0.89 ?? 2%
>> perf-profile.children.cycles-pp.xfs_bmapi_write
>> ????? 24.29??????????? +0.9?????? 25.18
>> perf-profile.children.cycles-pp._raw_spin_lock_irqsave
>> ?????? 0.07 ?? 5%????? +1.0??????? 1.05
>> perf-profile.children.cycles-pp.iomap_finish_ioends
>> ????? 35.68??????????? +1.0?????? 36.69
>> perf-profile.children.cycles-pp.xlog_cil_force_lsn
>> ?????? 9.53??????????? +1.2?????? 10.76
>> perf-profile.children.cycles-pp.intel_idle
>> ?????? 9.81??????????? +1.3?????? 11.10
>> perf-profile.children.cycles-pp.cpuidle_enter_state
>> ?????? 9.82??????????? +1.3?????? 11.11
>> perf-profile.children.cycles-pp.cpuidle_enter
>> ????? 12.25??????????? +1.3?????? 13.60
>> perf-profile.children.cycles-pp.start_secondary
>> ????? 12.49??????????? +1.4?????? 13.89
>> perf-profile.children.cycles-pp.secondary_startup_64
>> ????? 12.49??????????? +1.4?????? 13.89
>> perf-profile.children.cycles-pp.cpu_startup_entry
>> ????? 12.48??????????? +1.4?????? 13.88
>> perf-profile.children.cycles-pp.do_idle
>> ????? 14.78??????????? +1.4?????? 16.20 ?? 2%
>> perf-profile.children.cycles-pp.xlog_wait_on_iclog
>> ????? 58.91??????????? +1.7?????? 60.57
>> perf-profile.children.cycles-pp.xfs_log_force_lsn
>> ?????? 0.00??????????? +2.2??????? 2.24 ? 13%
>> perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
>> ?????? 7.05??????????? +2.8??????? 9.81 ?? 2%
>> perf-profile.children.cycles-pp.process_one_work
>> ?????? 7.54??????????? +2.9?????? 10.45 ?? 2%
>> perf-profile.children.cycles-pp.worker_thread
>> ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
>> perf-profile.children.cycles-pp.ret_from_fork
>> ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
>> perf-profile.children.cycles-pp.kthread
>> ?????? 0.10??????????? +3.2??????? 3.30 ?? 9%
>> perf-profile.children.cycles-pp.xfs_end_ioend
>> ?????? 0.10 ?? 4%????? +3.2??????? 3.32 ?? 8%
>> perf-profile.children.cycles-pp.xfs_end_io
>> ????? 19.57??????????? +4.0?????? 23.62
>> perf-profile.children.cycles-pp.remove_wait_queue
>> ?????? 1.20??????????? -0.3??????? 0.90 ?? 2%
>> perf-profile.self.cycles-pp.brd_do_bvec
>> ?????? 0.55 ?? 3%????? -0.2??????? 0.39 ?? 4%
>> perf-profile.self.cycles-pp.do_syscall_64
>> ?????? 0.31??????????? -0.1??????? 0.22 ?? 5%
>> perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
>> ?????? 0.31 ?? 3%????? -0.1??????? 0.22
>> perf-profile.self.cycles-pp.syscall_return_via_sysret
>> ?????? 0.54 ?? 3%????? -0.1??????? 0.47 ?? 3%
>> perf-profile.self.cycles-pp.osq_lock
>> ?????? 0.23 ?? 3%????? -0.1??????? 0.16 ?? 5%
>> perf-profile.self.cycles-pp.entry_SYSCALL_64
>> ?????? 0.10 ?? 4%????? -0.1??????? 0.04 ? 57%
>> perf-profile.self.cycles-pp.__mod_lruvec_state
>> ?????? 0.18 ?? 4%????? -0.1??????? 0.12 ?? 5%
>> perf-profile.self.cycles-pp.iomap_set_range_uptodate
>> ?????? 0.60 ?? 4%????? -0.1??????? 0.54 ?? 2%
>> perf-profile.self.cycles-pp._raw_spin_lock
>> ?????? 0.77??????????? -0.0??????? 0.72
>> perf-profile.self.cycles-pp._raw_spin_lock_irqsave
>> ?????? 0.14??????????? -0.0??????? 0.10 ?? 8%
>> perf-profile.self.cycles-pp.find_get_entry
>> ?????? 0.14 ?? 3%????? -0.0??????? 0.11 ?? 4%
>> perf-profile.self.cycles-pp.xas_load
>> ?????? 0.17 ?? 4%????? -0.0??????? 0.14 ?? 3%
>> perf-profile.self.cycles-pp.write
>> ?????? 0.30 ?? 2%????? -0.0??????? 0.27 ?? 6%
>> perf-profile.self.cycles-pp._raw_spin_lock_irq
>> ?????? 0.18 ?? 5%????? -0.0??????? 0.15 ?? 7%
>> perf-profile.self.cycles-pp.__flush_work
>> ?????? 0.08 ?? 5%????? -0.0??????? 0.05
>> perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
>> ?????? 0.12 ?? 3%????? -0.0??????? 0.09 ?? 4%
>> perf-profile.self.cycles-pp.brd_lookup_page
>> ?????? 0.12 ?? 7%????? -0.0??????? 0.09 ?? 9%
>> perf-profile.self.cycles-pp.vfs_write
>> ?????? 0.18 ?? 4%????? -0.0??????? 0.15 ?? 7%
>> perf-profile.self.cycles-pp.insert_work
>> ?????? 0.13??????????? -0.0??????? 0.10 ?? 8%
>> perf-profile.self.cycles-pp.memset_erms
>> ?????? 0.11 ?? 4%????? -0.0??????? 0.08
>> perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
>> ?????? 0.11 ?? 6%????? -0.0??????? 0.08 ?? 5%
>> perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
>> ?????? 0.12 ?? 3%????? -0.0??????? 0.09 ?? 4%
>> perf-profile.self.cycles-pp.iomap_writepage_map
>> ?????? 0.08 ?? 8%????? -0.0??????? 0.06 ? 15%
>> perf-profile.self.cycles-pp.bio_alloc_bioset
>> ?????? 0.06 ?? 6%????? -0.0??????? 0.04 ? 57%
>> perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
>> ?????? 0.09 ?? 5%????? -0.0??????? 0.06 ?? 6%
>> perf-profile.self.cycles-pp.__fget_light
>> ?????? 0.06??????????? -0.0??????? 0.04 ? 57%
>> perf-profile.self.cycles-pp.xfs_map_blocks
>> ?????? 0.18 ?? 2%????? -0.0??????? 0.15 ?? 3%
>> perf-profile.self.cycles-pp.__list_add_valid
>> ?????? 0.08 ?? 6%????? -0.0??????? 0.06 ?? 9%
>> perf-profile.self.cycles-pp.xfs_ilock
>> ?????? 0.08 ?? 6%????? -0.0??????? 0.06 ?? 9%
>> perf-profile.self.cycles-pp.iov_iter_fault_in_readable
>> ?????? 0.11 ?? 4%????? -0.0??????? 0.09 ?? 4%
>> perf-profile.self.cycles-pp.switch_fpu_return
>> ?????? 0.07??????????? -0.0??????? 0.05
>> perf-profile.self.cycles-pp.iomap_write_actor
>> ?????? 0.23 ?? 2%????? -0.0??????? 0.21 ?? 5%
>> perf-profile.self.cycles-pp.__radix_tree_lookup
>> ?????? 0.18 ?? 4%????? -0.0??????? 0.16 ?? 6%
>> perf-profile.self.cycles-pp.xlog_cil_force_lsn
>> ?????? 0.20 ?? 2%????? -0.0??????? 0.18 ?? 4%
>> perf-profile.self.cycles-pp.kmem_cache_alloc
>> ?????? 0.10 ?? 5%????? -0.0??????? 0.08 ? 11%
>> perf-profile.self.cycles-pp.xfs_file_fsync
>> ?????? 0.07 ?? 5%????? -0.0??????? 0.05 ?? 9%
>> perf-profile.self.cycles-pp.clear_page_dirty_for_io
>> ?????? 0.07 ?? 6%????? -0.0??????? 0.05
>> perf-profile.self.cycles-pp.__test_set_page_writeback
>> ?????? 0.11??????????? -0.0??????? 0.09 ?? 8%
>> perf-profile.self.cycles-pp.common_file_perm
>> ?????? 0.23??????????? -0.0??????? 0.21 ?? 4%
>> perf-profile.self.cycles-pp.update_rq_clock
>> ?????? 0.08 ?? 6%????? -0.0??????? 0.06
>> perf-profile.self.cycles-pp.xfs_file_write_iter
>> ?????? 0.10 ?? 5%????? -0.0??????? 0.08 ?? 8%
>> perf-profile.self.cycles-pp.up_read
>> ?????? 0.08 ?? 5%????? -0.0??????? 0.07
>> perf-profile.self.cycles-pp.write_cache_pages
>> ?????? 0.10??????????? -0.0??????? 0.09
>> perf-profile.self.cycles-pp.reweight_entity
>> ?????? 0.05 ?? 8%????? +0.0??????? 0.07 ?? 7%
>> perf-profile.self.cycles-pp.dequeue_entity
>> ?????? 0.16 ?? 4%????? +0.0??????? 0.17 ?? 2%
>> perf-profile.self.cycles-pp.__switch_to_asm
>> ?????? 0.11 ?? 4%????? +0.0??????? 0.12 ?? 3%
>> perf-profile.self.cycles-pp.set_next_entity
>> ?????? 0.15 ?? 3%????? +0.0??????? 0.16 ?? 2%
>> perf-profile.self.cycles-pp.orc_find
>> ?????? 0.09 ?? 4%????? +0.0??????? 0.11 ?? 6%
>> perf-profile.self.cycles-pp.enqueue_entity
>> ?????? 0.10 ?? 4%????? +0.0??????? 0.12 ?? 3%
>> perf-profile.self.cycles-pp._find_next_bit
>> ?????? 0.01 ?173%????? +0.0??????? 0.06 ?? 7%
>> perf-profile.self.cycles-pp.check_preempt_wakeup
>> ?????? 0.01 ?173%????? +0.0??????? 0.06 ? 11%
>> perf-profile.self.cycles-pp.xfs_trans_committed_bulk
>> ?????? 0.45 ?? 2%????? +0.1??????? 0.51 ?? 2%
>> perf-profile.self.cycles-pp.__sched_text_start
>> ?????? 0.15 ?? 6%????? +0.1??????? 0.21 ?? 3%
>> perf-profile.self.cycles-pp.memcpy_erms
>> ?????? 0.00??????????? +0.1??????? 0.06 ?? 6%
>> perf-profile.self.cycles-pp.xfs_verify_agbno
>> ?????? 0.08 ?? 8%????? +0.1??????? 0.16 ?? 9%
>> perf-profile.self.cycles-pp.xfs_iext_lookup_extent
>> ?????? 9.53??????????? +1.2?????? 10.76
>> perf-profile.self.cycles-pp.intel_idle
>>
>>
>> ?????????????????????????????????? aim7.jobs-per-min
>> ?? 45000
>> +-------------------------------------------------------------------+
>> ???????? |??????? +
>> +?????????????????????????????????????????????????????? |
>> ?? 40000 |.+.+??? :?? + .+.?? +.
>> .+.???? |
>> ?? 35000 |-+ :??? :??? +?? +.+
>> +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+?? ++.+.|
>> ???????? |?? :
>> :????????????????????????????????????????????????????????? |
>> ?? 30000 |-+ :
>> :?????????????????????????????????????????????????????????? |
>> ?? 25000 |-O :O O:O
>> OO???????????????????????????????????????????????????? |
>> ???????? |?? :?? :? O??? O O OO O O O
>> O????????????????????????????????????? |
>> ?? 20000 |-+? :
>> :?????????????????????????????????????????????????????????? |
>> ?? 15000 |-+? :
>> :?????????????????????????????????????????????????????????? |
>> ???????? |??? :
>> :?????????????????????????????????????????????????????????? |
>> ?? 10000 |-+? :
>> :??????????????????????????????????????????????????????????? |
>> ??? 5000 |-+? :
>> :??????????????????????????????????????????????????????????? |
>> ???????? |??? :
>> :??????????????????????????????????????????????????????????? |
>> ?????? 0
>> +-------------------------------------------------------------------+
>> ??????????????????????????????? aim7.time.system_time
>> ?? 5000
>> +--------------------------------------------------------------------+
>> ?? 4500 |-+??????? O?? OO?? O??? O O
>> O?????????????????????????????????????? |
>> ??????? | O OO O O?? O??? O?? O
>> O??????????????????????????????????????????? |
>> ?? 4000
>> |-+????????????????????????????????????????????????????????????????? |
>> ?? 3500
>> |-+????????????????????????????????????????????????????????????????? |
>> ??????? |???????????? .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+ .+.+.+.
>> .++.+.|
>> ?? 3000 |.+.+?????? .+? +???? +?????????????????????????????? +
>> +????? |
>> ?? 2500 |-+ :
>> +.+???????????????????????????????????????????????????????? |
>> ?? 2000 |-+ :
>> :?????????????????????????????????????????????????????????? |
>> ??????? |?? :
>> :??????????????????????????????????????????????????????????? |
>> ?? 1500 |-+? :
>> :??????????????????????????????????????????????????????????? |
>> ?? 1000 |-+? :
>> :??????????????????????????????????????????????????????????? |
>> ??????? |??? :
>> :??????????????????????????????????????????????????????????? |
>> ??? 500 |-+? :
>> :???????????????????????????????????????????????????????????? |
>> ????? 0
>> +--------------------------------------------------------------------+
>> ?????????????????????????????? aim7.time.elapsed_time
>> ?? 160
>> +---------------------------------------------------------------------+
>> ?????? |?? O OO O O O O OO O O O
>> O???????????????????????????????????????? |
>> ?? 140
>> |-O?????????????????????????????????????????????????????????????????? |
>> ?? 120
>> |-+?????????????????????????????????????????????????????????????????? |
>>
>> |???????????????????????????????????????????????????????????????????? |
>> ?? 100 |-+?????????? .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+.
>> +.+.+.|
>> ?????? |.+.+??? +. .+?? +??? +
>> +????? |
>> ??? 80 |-+ :??? :
>> +????????????????????????????????????????????????????????? |
>> ?????? |??? :
>> :??????????????????????????????????????????????????????????? |
>> ??? 60 |-+? :
>> :???????????????????????????????????????????????????????????? |
>> ??? 40 |-+? :
>> :???????????????????????????????????????????????????????????? |
>> ?????? |??? :
>> :???????????????????????????????????????????????????????????? |
>> ??? 20 |-+?? :
>> :???????????????????????????????????????????????????????????? |
>> ?????? |
>> ::????????????????????????????????????????????????????????????? |
>> ???? 0
>> +---------------------------------------------------------------------+
>> ???????????????????????????? aim7.time.elapsed_time.max
>> ?? 160
>> +---------------------------------------------------------------------+
>> ?????? |?? O OO O O O O OO O O O
>> O???????????????????????????????????????? |
>> ?? 140
>> |-O?????????????????????????????????????????????????????????????????? |
>> ?? 120
>> |-+?????????????????????????????????????????????????????????????????? |
>>
>> |???????????????????????????????????????????????????????????????????? |
>> ?? 100 |-+?????????? .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+.
>> +.+.+.|
>> ?????? |.+.+??? +. .+?? +??? +
>> +????? |
>> ??? 80 |-+ :??? :
>> +????????????????????????????????????????????????????????? |
>> ?????? |??? :
>> :??????????????????????????????????????????????????????????? |
>> ??? 60 |-+? :
>> :???????????????????????????????????????????????????????????? |
>> ??? 40 |-+? :
>> :???????????????????????????????????????????????????????????? |
>> ?????? |??? :
>> :???????????????????????????????????????????????????????????? |
>> ??? 20 |-+?? :
>> :???????????????????????????????????????????????????????????? |
>> ?????? |
>> ::????????????????????????????????????????????????????????????? |
>> ???? 0
>> +---------------------------------------------------------------------+
>> ???????????????????????? aim7.time.voluntary_context_switches
>> ?? 8e+07
>> +-------------------------------------------------------------------+
>> ???????? | O????????? O? O O OO O
>> O??????????????????????????????????????? |
>> ?? 7e+07
>> |-+???????????????????????????????????????????????????????????????? |
>> ?? 6e+07
>> |-+???????????????????????????????????????????????????????????????? |
>>
>> |?????????????????????????????????????????????????????????????????? |
>> ?? 5e+07 |-+?????????? +.+.+.++.+.+.+.++.+.+. .+?? .+.+.??? .+.
>> .+.?? +.+.|
>> ???????? |.+.+?????? .+????????????????????? +? +.+???? ++.+?? ++.+
>> +.+??? |
>> ?? 4e+07 |-+ :
>> +.+??????????????????????????????????????????????????????? |
>> ???????? |?? :
>> :????????????????????????????????????????????????????????? |
>> ?? 3e+07 |-+ :
>> :?????????????????????????????????????????????????????????? |
>> ?? 2e+07 |-+? :
>> :?????????????????????????????????????????????????????????? |
>> ???????? |??? :
>> :?????????????????????????????????????????????????????????? |
>> ?? 1e+07 |-+? :
>> :?????????????????????????????????????????????????????????? |
>> ???????? |??? :
>> :??????????????????????????????????????????????????????????? |
>> ?????? 0
>> +-------------------------------------------------------------------+
>> [*] bisect-good sample
>> [O] bisect-bad? sample
>>
>>
>>
>> Disclaimer:
>> Results have been estimated based on internal Intel analysis and are
>> provided
>> for informational purposes only. Any difference in system hardware or
>> software
>> design or configuration may affect actual performance.
>>
>>
>> Thanks,
>> Rong Chen
>>
>>
>> _______________________________________________
>> LKP mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>>
>

--
Zhengjun Xing

2020-07-06 18:34:05

by Darrick J. Wong

[permalink] [raw]

Subject: Re: [LKP] [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

On Wed, Jul 01, 2020 at 03:49:52PM +0800, Xing Zhengjun wrote:
>
>
> On 6/10/2020 11:07 AM, Xing Zhengjun wrote:
> > Hi Darrick,
> >
> > ?? Do you have time to take a look at this? Thanks.
> >
>
>
> Ping...

Yes, that decrease is the expected end result of making the write path
take a longer route to avoid a file corruption vector.

--D

>
>
> > On 6/6/2020 11:48 PM, kernel test robot wrote:
> > > Greeting,
> > >
> > > FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:
> > >
> > >
> > > commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes
> > > to delalloc regions to unwritten")
> > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > >
> > > in testcase: aim7
> > > on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @
> > > 2.70GHz with 64G memory
> > > with following parameters:
> > >
> > > ????disk: 1BRD_48G
> > > ????fs: xfs
> > > ????test: sync_disk_rw
> > > ????load: 600
> > > ????cpufreq_governor: performance
> > > ????ucode: 0x42e
> > >
> > > test-description: AIM7 is a traditional UNIX system level benchmark
> > > suite which is used to test and measure the performance of multiuser
> > > system.
> > > test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
> > >
> > >
> > >
> > > If you fix the issue, kindly add following tag
> > > Reported-by: kernel test robot <[email protected]>
> > >
> > >
> > > Details are as below:
> > > -------------------------------------------------------------------------------------------------->
> > >
> > >
> > >
> > > To reproduce:
> > >
> > > ???????? git clone https://github.com/intel/lkp-tests.git
> > > ???????? cd lkp-tests
> > > ???????? bin/lkp install job.yaml? # job file is attached in this email
> > > ???????? bin/lkp run???? job.yaml
> > >
> > > =========================================================================================
> > >
> > > compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
> > >
> > > gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
> > >
> > >
> > > commit:
> > > ?? 590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
> > > ?? a5949d3fae ("xfs: force writes to delalloc regions to unwritten")
> > >
> > > 590b16516ef38e2e a5949d3faedf492fa7863b914da
> > > ---------------- ---------------------------
> > > ??????? fail:runs? %reproduction??? fail:runs
> > > ??????????? |???????????? |???????????? |
> > > ??????????? :4?????????? 50%?????????? 2:4
> > > dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
> > > ????????? %stddev???? %change???????? %stddev
> > > ????????????? \????????? |??????????????? \
> > > ????? 35272?????????? -33.6%????? 23430??????? aim7.jobs-per-min
> > > ???? 102.13?????????? +50.5%???? 153.75??????? aim7.time.elapsed_time
> > > ???? 102.13?????????? +50.5%???? 153.75??????? aim7.time.elapsed_time.max
> > > ??? 1388038?????????? +40.2%??? 1945838
> > > aim7.time.involuntary_context_switches
> > > ????? 43420 ?? 2%???? +13.4%????? 49255 ?? 2%
> > > aim7.time.minor_page_faults
> > > ?????? 3123?????????? +44.2%?????? 4504 ?? 2%? aim7.time.system_time
> > > ????? 59.31??????????? +6.5%????? 63.18??????? aim7.time.user_time
> > > ?? 48595108?????????? +58.6%?? 77064959
> > > aim7.time.voluntary_context_switches
> > > ?????? 1.44?????????? -28.8%?????? 1.02??????? iostat.cpu.user
> > > ?????? 0.07 ?? 6%????? +0.4??????? 0.44 ?? 7%? mpstat.cpu.all.iowait%
> > > ?????? 1.44??????????? -0.4??????? 1.02??????? mpstat.cpu.all.usr%
> > > ?????? 8632 ? 50%???? +75.6%????? 15156 ? 34%
> > > numa-meminfo.node0.KernelStack
> > > ?????? 6583 ?136%??? +106.0%????? 13562 ? 82%
> > > numa-meminfo.node0.PageTables
> > > ????? 63325 ? 11%???? +14.3%????? 72352 ? 12%
> > > numa-meminfo.node0.SUnreclaim
> > > ?????? 8647 ? 50%???? +75.3%????? 15156 ? 34%
> > > numa-vmstat.node0.nr_kernel_stack
> > > ?????? 1656 ?136%??? +104.6%?????? 3389 ? 82%
> > > numa-vmstat.node0.nr_page_table_pages
> > > ????? 15831 ? 11%???? +14.3%????? 18087 ? 12%
> > > numa-vmstat.node0.nr_slab_unreclaimable
> > > ????? 93640 ?? 3%???? +41.2%???? 132211 ?? 2%? meminfo.AnonHugePages
> > > ????? 21641?????????? +39.9%????? 30271 ?? 4%? meminfo.KernelStack
> > > ???? 129269?????????? +12.3%???? 145114??????? meminfo.SUnreclaim
> > > ????? 28000?????????? -31.2%????? 19275??????? meminfo.max_used_kB
> > > ??? 1269307?????????? -26.9%???? 927657??????? vmstat.io.bo
> > > ???? 149.75 ?? 3%???? -17.4%???? 123.75 ?? 4%? vmstat.procs.r
> > > ???? 718992?????????? +13.3%???? 814567??????? vmstat.system.cs
> > > ???? 231397??????????? -9.3%???? 209881 ?? 2%? vmstat.system.in
> > > ? 6.774e+08?????????? +70.0%? 1.152e+09??????? cpuidle.C1.time
> > > ?? 18203372?????????? +60.4%?? 29198744??????? cpuidle.C1.usage
> > > ? 2.569e+08 ? 18%???? +81.8%? 4.672e+08 ?? 5%? cpuidle.C1E.time
> > > ??? 2691402 ? 13%???? +98.7%??? 5346901 ?? 3%? cpuidle.C1E.usage
> > > ???? 990350?????????? +95.0%??? 1931226 ?? 2%? cpuidle.POLL.time
> > > ???? 520061?????????? +97.7%??? 1028004 ?? 2%? cpuidle.POLL.usage
> > > ????? 77231??????????? +1.8%????? 78602??????? proc-vmstat.nr_active_anon
> > > ????? 19868??????????? +3.8%????? 20615??????? proc-vmstat.nr_dirty
> > > ???? 381302??????????? +1.0%???? 384969??????? proc-vmstat.nr_file_pages
> > > ?????? 4388??????????? -2.7%?????? 4270
> > > proc-vmstat.nr_inactive_anon
> > > ????? 69865??????????? +4.7%????? 73155
> > > proc-vmstat.nr_inactive_file
> > > ????? 21615?????????? +40.0%????? 30251 ?? 4%
> > > proc-vmstat.nr_kernel_stack
> > > ?????? 7363??????????? -3.2%?????? 7127??????? proc-vmstat.nr_mapped
> > > ????? 12595 ?? 3%????? +5.2%????? 13255 ?? 4%? proc-vmstat.nr_shmem
> > > ????? 19619??????????? +3.2%????? 20247
> > > proc-vmstat.nr_slab_reclaimable
> > > ????? 32316?????????? +12.3%????? 36280
> > > proc-vmstat.nr_slab_unreclaimable
> > > ????? 77231??????????? +1.8%????? 78602
> > > proc-vmstat.nr_zone_active_anon
> > > ?????? 4388??????????? -2.7%?????? 4270
> > > proc-vmstat.nr_zone_inactive_anon
> > > ????? 69865??????????? +4.7%????? 73155
> > > proc-vmstat.nr_zone_inactive_file
> > > ????? 19165??????????? +3.4%????? 19824
> > > proc-vmstat.nr_zone_write_pending
> > > ?????? 9845 ? 27%???? +68.0%????? 16539 ? 37%
> > > proc-vmstat.numa_hint_faults_local
> > > ??? 8195135??????????? +2.5%??? 8402537??????? proc-vmstat.numa_hit
> > > ??? 8179414??????????? +2.5%??? 8386209??????? proc-vmstat.numa_local
> > > ????? 15720??????????? +3.9%????? 16328??????? proc-vmstat.numa_other
> > > ??? 8281098??????????? +2.4%??? 8477157??????? proc-vmstat.pgalloc_normal
> > > ???? 342925 ?? 2%???? +39.8%???? 479279??????? proc-vmstat.pgfault
> > > ??? 8132789??????????? +2.0%??? 8298189??????? proc-vmstat.pgfree
> > > ? 1.327e+08??????????? +9.7%? 1.455e+08??????? proc-vmstat.pgpgout
> > > ???? 394.00????????? +151.7%???? 991.75 ?? 5%
> > > slabinfo.biovec-max.active_objs
> > > ???? 394.00????????? +152.1%???? 993.25 ?? 5%
> > > slabinfo.biovec-max.num_objs
> > > ????? 21028 ?? 6%????? -7.8%????? 19389 ?? 5%? slabinfo.filp.active_objs
> > > ????? 21520 ?? 5%????? -7.6%????? 19886 ?? 5%? slabinfo.filp.num_objs
> > > ?????? 4955????????? +234.9%????? 16594 ?? 2%
> > > slabinfo.kmalloc-256.active_objs
> > > ????? 77.25????????? +238.5%???? 261.50 ?? 2%
> > > slabinfo.kmalloc-256.active_slabs
> > > ?????? 4955????????? +238.4%????? 16768 ?? 2%
> > > slabinfo.kmalloc-256.num_objs
> > > ????? 77.25????????? +238.5%???? 261.50 ?? 2%
> > > slabinfo.kmalloc-256.num_slabs
> > > ?????? 3368 ?? 2%???? +22.6%?????? 4128 ?? 2%
> > > slabinfo.kmalloc-2k.active_objs
> > > ?????? 3446 ?? 2%???? +21.9%?????? 4200 ?? 2%
> > > slabinfo.kmalloc-2k.num_objs
> > > ????? 11815 ?? 8%???? +29.0%????? 15239 ?? 3%
> > > slabinfo.kmalloc-512.active_objs
> > > ????? 11824 ?? 9%???? +29.0%????? 15249 ?? 3%
> > > slabinfo.kmalloc-512.num_objs
> > > ?????? 6849 ?? 5%????? +9.0%?????? 7463 ?? 2%
> > > slabinfo.kmalloc-96.active_objs
> > > ?????? 6906 ?? 5%????? +8.9%?????? 7521 ?? 2%
> > > slabinfo.kmalloc-96.num_objs
> > > ???? 765.00 ?? 4%???? +25.0%???? 956.25 ?? 5%
> > > slabinfo.mnt_cache.active_objs
> > > ???? 765.00 ?? 4%???? +25.0%???? 956.25 ?? 5%
> > > slabinfo.mnt_cache.num_objs
> > > ????? 13866?????????? +12.5%????? 15594
> > > slabinfo.proc_inode_cache.active_objs
> > > ????? 13905?????????? +12.2%????? 15598
> > > slabinfo.proc_inode_cache.num_objs
> > > ?????? 1954?????????? +24.4%?????? 2431 ?? 3%
> > > slabinfo.sighand_cache.active_objs
> > > ?????? 1954?????????? +24.4%?????? 2431 ?? 3%
> > > slabinfo.sighand_cache.num_objs
> > > ?????? 2596?????????? +13.7%?????? 2951 ?? 2%
> > > slabinfo.signal_cache.active_objs
> > > ?????? 2596?????????? +13.7%?????? 2951 ?? 2%
> > > slabinfo.signal_cache.num_objs
> > > ?????? 4368 ?? 6%???? -11.7%?????? 3856 ?? 2%
> > > slabinfo.skbuff_head_cache.num_objs
> > > ???? 822.25 ? 11%???? +21.4%???? 998.50 ?? 3%
> > > slabinfo.task_group.active_objs
> > > ???? 822.25 ? 11%???? +21.4%???? 998.50 ?? 3%
> > > slabinfo.task_group.num_objs
> > > ?????? 1612?????????? +32.1%?????? 2129 ?? 3%
> > > slabinfo.task_struct.active_objs
> > > ???? 540.50?????????? +31.6%???? 711.25 ?? 3%
> > > slabinfo.task_struct.active_slabs
> > > ?????? 1622?????????? +31.6%?????? 2134 ?? 3%
> > > slabinfo.task_struct.num_objs
> > > ???? 540.50?????????? +31.6%???? 711.25 ?? 3%
> > > slabinfo.task_struct.num_slabs
> > > ?????? 6905?????????? +20.0%?????? 8289
> > > slabinfo.vmap_area.active_objs
> > > ?????? 6933?????????? +20.0%?????? 8322
> > > slabinfo.vmap_area.num_objs
> > > ???? 646.25 ?? 7%??? +195.2%?????? 1907
> > > slabinfo.xfs_buf.active_objs
> > > ???? 646.25 ?? 7%??? +195.2%?????? 1907??????? slabinfo.xfs_buf.num_objs
> > > ?????? 1.78 ?? 2%????? -0.3??????? 1.51 ?? 2%
> > > perf-stat.i.branch-miss-rate%
> > > ?? 64079048??????????? +3.9%?? 66551536??????? perf-stat.i.branch-misses
> > > ????? 17.99??????????? -4.2?????? 13.84
> > > perf-stat.i.cache-miss-rate%
> > > ?? 66630506?????????? -16.5%?? 55647472??????? perf-stat.i.cache-misses
> > > ? 3.899e+08??????????? +9.4%? 4.265e+08
> > > perf-stat.i.cache-references
> > > ???? 731029?????????? +13.1%???? 826756
> > > perf-stat.i.context-switches
> > > ?????? 3.79??????????? -1.7%?????? 3.73??????? perf-stat.i.cpi
> > > ???? 113320??????????? +8.3%???? 122693??????? perf-stat.i.cpu-migrations
> > > ?????? 1498?????????? +20.4%?????? 1803
> > > perf-stat.i.cycles-between-cache-misses
> > > ?? 50450547 ?? 4%????? -8.0%?? 46402743 ?? 5%
> > > perf-stat.i.dTLB-load-misses
> > > ?????? 0.20 ?? 3%????? -0.0??????? 0.16 ?? 2%
> > > perf-stat.i.dTLB-store-miss-rate%
> > > ??? 4587096 ?? 3%???? -22.9%??? 3538422 ?? 2%
> > > perf-stat.i.dTLB-store-misses
> > > ? 2.297e+09??????????? -5.6%? 2.168e+09??????? perf-stat.i.dTLB-stores
> > > ????? 86.68??????????? -2.4?????? 84.25
> > > perf-stat.i.iTLB-load-miss-rate%
> > > ?? 19484791??????????? -6.1%?? 18297125
> > > perf-stat.i.iTLB-load-misses
> > > ??? 3060023?????????? +14.4%??? 3501587 ?? 4%? perf-stat.i.iTLB-loads
> > > ?????? 0.29??????????? -1.5%?????? 0.28??????? perf-stat.i.ipc
> > > ?????? 0.21 ?? 5%???? -20.5%?????? 0.17 ?? 6%? perf-stat.i.metric.K/sec
> > > ?????? 3120??????????? -5.8%?????? 2938??????? perf-stat.i.minor-faults
> > > ????? 42.61??????????? +1.2?????? 43.83
> > > perf-stat.i.node-load-miss-rate%
> > > ?? 24887573?????????? -10.1%?? 22384693
> > > perf-stat.i.node-load-misses
> > > ?? 31814752?????????? -12.5%?? 27830232??????? perf-stat.i.node-loads
> > > ????? 36.62??????????? +0.8?????? 37.45
> > > perf-stat.i.node-store-miss-rate%
> > > ?? 22617213?????????? -20.3%?? 18015607
> > > perf-stat.i.node-store-misses
> > > ?? 35707079?????????? -19.9%?? 28604167??????? perf-stat.i.node-stores
> > > ?????? 3120??????????? -5.8%?????? 2938??????? perf-stat.i.page-faults
> > > ????? 13.96??????????? +9.3%????? 15.25??????? perf-stat.overall.MPKI
> > > ?????? 0.99??????????? +0.0??????? 1.03
> > > perf-stat.overall.branch-miss-rate%
> > > ????? 17.09??????????? -4.0?????? 13.05
> > > perf-stat.overall.cache-miss-rate%
> > > ?????? 1555?????????? +19.4%?????? 1856
> > > perf-stat.overall.cycles-between-cache-misses
> > > ?????? 0.69 ?? 4%????? -0.1??????? 0.64 ?? 5%
> > > perf-stat.overall.dTLB-load-miss-rate%
> > > ?????? 0.20 ?? 3%????? -0.0??????? 0.16 ?? 2%
> > > perf-stat.overall.dTLB-store-miss-rate%
> > > ????? 86.43??????????? -2.5?????? 83.94
> > > perf-stat.overall.iTLB-load-miss-rate%
> > > ?????? 1433??????????? +6.7%?????? 1528
> > > perf-stat.overall.instructions-per-iTLB-miss
> > > ????? 43.89??????????? +0.7?????? 44.58
> > > perf-stat.overall.node-load-miss-rate%
> > > ? 6.389e+09??????????? +0.8%? 6.438e+09
> > > perf-stat.ps.branch-instructions
> > > ?? 63452800??????????? +4.2%?? 66110454??????? perf-stat.ps.branch-misses
> > > ?? 65977234?????????? -16.2%?? 55277568??????? perf-stat.ps.cache-misses
> > > ? 3.861e+08??????????? +9.8%? 4.237e+08
> > > perf-stat.ps.cache-references
> > > ???? 723870?????????? +13.5%???? 821265
> > > perf-stat.ps.context-switches
> > > ???? 112209??????????? +8.6%???? 121881
> > > perf-stat.ps.cpu-migrations
> > > ?? 49955941 ?? 4%????? -7.7%?? 46094243 ?? 5%
> > > perf-stat.ps.dTLB-load-misses
> > > ??? 4542094 ?? 3%???? -22.6%??? 3514899 ?? 2%
> > > perf-stat.ps.dTLB-store-misses
> > > ? 2.275e+09??????????? -5.3%? 2.154e+09??????? perf-stat.ps.dTLB-stores
> > > ?? 19294019??????????? -5.8%?? 18175580
> > > perf-stat.ps.iTLB-load-misses
> > > ??? 3030049?????????? +14.8%??? 3478314 ?? 4%? perf-stat.ps.iTLB-loads
> > > ?????? 3089??????????? -5.5%?????? 2918??????? perf-stat.ps.minor-faults
> > > ?? 24643655??????????? -9.8%?? 22235787
> > > perf-stat.ps.node-load-misses
> > > ?? 31502901?????????? -12.2%?? 27645143??????? perf-stat.ps.node-loads
> > > ?? 22395376?????????? -20.1%?? 17895417
> > > perf-stat.ps.node-store-misses
> > > ?? 35356897?????????? -19.6%?? 28413366??????? perf-stat.ps.node-stores
> > > ?????? 3089??????????? -5.5%?????? 2918??????? perf-stat.ps.page-faults
> > > ?? 2.85e+12?????????? +50.6%? 4.293e+12
> > > perf-stat.total.instructions
> > > ????? 84193 ? 36%??? +135.9%???? 198575 ? 25%
> > > sched_debug.cfs_rq:/.MIN_vruntime.avg
> > > ???? 919053?????????? +98.2%??? 1821460 ?? 2%
> > > sched_debug.cfs_rq:/.MIN_vruntime.max
> > > ???? 256686 ? 16%??? +114.7%???? 551162 ? 13%
> > > sched_debug.cfs_rq:/.MIN_vruntime.stddev
> > > ????? 19497????????? +106.7%????? 40307
> > > sched_debug.cfs_rq:/.exec_clock.avg
> > > ????? 21240????????? +101.1%????? 42717
> > > sched_debug.cfs_rq:/.exec_clock.max
> > > ????? 19079????????? +107.3%????? 39542
> > > sched_debug.cfs_rq:/.exec_clock.min
> > > ???? 394.52 ?? 2%???? +54.3%???? 608.79 ? 13%
> > > sched_debug.cfs_rq:/.exec_clock.stddev
> > > ???? 965.25 ? 15%???? +59.7%?????? 1541 ?? 4%
> > > sched_debug.cfs_rq:/.load_avg.avg
> > > ?????? 2866 ? 21%???? +57.1%?????? 4503 ? 10%
> > > sched_debug.cfs_rq:/.load_avg.max
> > > ???? 659.77 ? 15%???? +68.9%?????? 1114 ?? 9%
> > > sched_debug.cfs_rq:/.load_avg.stddev
> > > ????? 84193 ? 36%??? +135.9%???? 198575 ? 25%
> > > sched_debug.cfs_rq:/.max_vruntime.avg
> > > ???? 919053?????????? +98.2%??? 1821462 ?? 2%
> > > sched_debug.cfs_rq:/.max_vruntime.max
> > > ???? 256686 ? 16%??? +114.7%???? 551163 ? 13%
> > > sched_debug.cfs_rq:/.max_vruntime.stddev
> > > ???? 901765?????????? +97.1%??? 1777741 ?? 2%
> > > sched_debug.cfs_rq:/.min_vruntime.avg
> > > ???? 941794?????????? +95.8%??? 1844445 ?? 2%
> > > sched_debug.cfs_rq:/.min_vruntime.max
> > > ???? 872925?????????? +96.5%??? 1715658 ?? 2%
> > > sched_debug.cfs_rq:/.min_vruntime.min
> > > ????? 18358 ? 44%??? +146.6%????? 45281 ? 34%
> > > sched_debug.cfs_rq:/.min_vruntime.stddev
> > > ?????? 1.22 ?? 8%??? +159.9%?????? 3.18 ? 20%
> > > sched_debug.cfs_rq:/.nr_spread_over.avg
> > > ?????? 6.75 ? 23%???? +65.4%????? 11.17 ? 17%
> > > sched_debug.cfs_rq:/.nr_spread_over.max
> > > ?????? 1.38 ? 11%???? +61.4%?????? 2.22 ? 12%
> > > sched_debug.cfs_rq:/.nr_spread_over.stddev
> > > ????? 18357 ? 44%??? +146.5%????? 45250 ? 34%
> > > sched_debug.cfs_rq:/.spread0.stddev
> > > ???? 773.25 ?? 3%???? +13.2%???? 875.02
> > > sched_debug.cfs_rq:/.util_avg.avg
> > > ???? 285624?????????? -30.3%???? 199111 ?? 4%
> > > sched_debug.cpu.avg_idle.avg
> > > ???? 576507 ?? 2%???? -26.2%???? 425271 ?? 7%
> > > sched_debug.cpu.avg_idle.max
> > > ????? 51029 ? 68%???? -83.4%?????? 8469 ? 29%
> > > sched_debug.cpu.avg_idle.min
> > > ???? 135149 ?? 7%???? -23.4%???? 103460 ? 10%
> > > sched_debug.cpu.avg_idle.stddev
> > > ????? 57492?????????? +51.0%????? 86835??????? sched_debug.cpu.clock.avg
> > > ????? 57495?????????? +51.0%????? 86841??????? sched_debug.cpu.clock.max
> > > ????? 57487?????????? +51.0%????? 86828??????? sched_debug.cpu.clock.min
> > > ?????? 1.98 ?? 4%???? +74.8%?????? 3.46 ? 10%
> > > sched_debug.cpu.clock.stddev
> > > ????? 57492?????????? +51.0%????? 86835
> > > sched_debug.cpu.clock_task.avg
> > > ????? 57495?????????? +51.0%????? 86841
> > > sched_debug.cpu.clock_task.max
> > > ????? 57487?????????? +51.0%????? 86828
> > > sched_debug.cpu.clock_task.min
> > > ?????? 1.98 ?? 4%???? +74.8%?????? 3.46 ? 10%
> > > sched_debug.cpu.clock_task.stddev
> > > ?????? 2831?????????? +43.0%?????? 4049
> > > sched_debug.cpu.curr->pid.max
> > > ???? 417519????????? +133.3%???? 974078
> > > sched_debug.cpu.nr_switches.avg
> > > ???? 427854????????? +134.0%??? 1001016
> > > sched_debug.cpu.nr_switches.max
> > > ???? 409942????????? +131.8%???? 950218
> > > sched_debug.cpu.nr_switches.min
> > > ?????? 4770 ? 42%??? +238.7%????? 16156 ? 22%
> > > sched_debug.cpu.nr_switches.stddev
> > > ?????? 4.70 ? 23%???? +37.1%?????? 6.44 ?? 4%
> > > sched_debug.cpu.nr_uninterruptible.avg
> > > ???? 776.62 ? 19%???? +86.1%?????? 1445 ? 12%
> > > sched_debug.cpu.nr_uninterruptible.max
> > > ??? -472.62?????????? +98.3%??? -937.33
> > > sched_debug.cpu.nr_uninterruptible.min
> > > ???? 289.80 ?? 5%??? +109.7%???? 607.68 ?? 3%
> > > sched_debug.cpu.nr_uninterruptible.stddev
> > > ???? 415893????????? +133.8%???? 972526
> > > sched_debug.cpu.sched_count.avg
> > > ???? 423960????????? +135.5%???? 998457
> > > sched_debug.cpu.sched_count.max
> > > ???? 409113????????? +132.0%???? 949068
> > > sched_debug.cpu.sched_count.min
> > > ?????? 4194 ? 50%??? +274.3%????? 15698 ? 22%
> > > sched_debug.cpu.sched_count.stddev
> > > ???? 110094????????? +125.8%???? 248571
> > > sched_debug.cpu.sched_goidle.avg
> > > ???? 112236????????? +125.6%???? 253163
> > > sched_debug.cpu.sched_goidle.max
> > > ???? 108281????????? +125.8%???? 244506
> > > sched_debug.cpu.sched_goidle.min
> > > ???? 772.04 ? 16%??? +175.8%?????? 2129 ? 18%
> > > sched_debug.cpu.sched_goidle.stddev
> > > ???? 297267????????? +137.2%???? 705173
> > > sched_debug.cpu.ttwu_count.avg
> > > ???? 320112????????? +135.9%???? 755096
> > > sched_debug.cpu.ttwu_count.max
> > > ???? 272487 ?? 2%??? +142.2%???? 660099
> > > sched_debug.cpu.ttwu_count.min
> > > ????? 10662 ? 10%??? +110.9%????? 22491 ? 18%
> > > sched_debug.cpu.ttwu_count.stddev
> > > ????? 12295 ?? 2%??? +834.9%???? 114954 ?? 2%
> > > sched_debug.cpu.ttwu_local.avg
> > > ????? 13726 ?? 2%??? +769.7%???? 119376 ?? 2%
> > > sched_debug.cpu.ttwu_local.max
> > > ????? 10620 ?? 4%??? +942.4%???? 110710
> > > sched_debug.cpu.ttwu_local.min
> > > ???? 693.29 ? 14%??? +228.7%?????? 2278 ? 18%
> > > sched_debug.cpu.ttwu_local.stddev
> > > ????? 57488?????????? +51.0%????? 86829??????? sched_debug.cpu_clk
> > > ????? 56998?????????? +51.5%????? 86337??????? sched_debug.ktime
> > > ????? 57977?????????? +50.6%????? 87317??????? sched_debug.sched_clk
> > > ????? 72.50 ? 37%???? +55.5%???? 112.75 ? 26%
> > > interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> > > ????? 53.25 ?? 2%???? +84.5%????? 98.25 ? 22%
> > > interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
> > > ????? 73.50 ? 26%??? +116.3%???? 159.00 ? 60%
> > > interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
> > > ???? 204541?????????? +49.0%???? 304702 ?? 2%
> > > interrupts.CPU0.LOC:Local_timer_interrupts
> > > ???? 290268 ?? 2%???? +29.1%???? 374641 ?? 4%
> > > interrupts.CPU0.RES:Rescheduling_interrupts
> > > ???? 204580?????????? +48.9%???? 304667 ?? 2%
> > > interrupts.CPU1.LOC:Local_timer_interrupts
> > > ???? 290342 ?? 2%???? +28.5%???? 373110 ?? 4%
> > > interrupts.CPU1.RES:Rescheduling_interrupts
> > > ???? 204537?????????? +48.9%???? 304619 ?? 2%
> > > interrupts.CPU10.LOC:Local_timer_interrupts
> > > ???? 291605 ?? 2%???? +28.3%???? 374190 ?? 4%
> > > interrupts.CPU10.RES:Rescheduling_interrupts
> > > ???? 204140?????????? +49.2%???? 304587 ?? 2%
> > > interrupts.CPU11.LOC:Local_timer_interrupts
> > > ???? 291988 ?? 2%???? +28.1%???? 374024 ?? 4%
> > > interrupts.CPU11.RES:Rescheduling_interrupts
> > > ???? 204367?????????? +49.0%???? 304559 ?? 2%
> > > interrupts.CPU12.LOC:Local_timer_interrupts
> > > ???? 288632?????????? +28.4%???? 370669 ?? 3%
> > > interrupts.CPU12.RES:Rescheduling_interrupts
> > > ???? 204555?????????? +48.9%???? 304669 ?? 2%
> > > interrupts.CPU13.LOC:Local_timer_interrupts
> > > ???? 289354 ?? 2%???? +28.5%???? 371782 ?? 3%
> > > interrupts.CPU13.RES:Rescheduling_interrupts
> > > ???? 204536?????????? +49.0%???? 304726 ?? 2%
> > > interrupts.CPU14.LOC:Local_timer_interrupts
> > > ?????? 4937 ? 25%???? +41.6%?????? 6991 ?? 2%
> > > interrupts.CPU14.NMI:Non-maskable_interrupts
> > > ?????? 4937 ? 25%???? +41.6%?????? 6991 ?? 2%
> > > interrupts.CPU14.PMI:Performance_monitoring_interrupts
> > > ???? 287759?????????? +28.2%???? 368986 ?? 3%
> > > interrupts.CPU14.RES:Rescheduling_interrupts
> > > ???? 204527?????????? +49.0%???? 304707 ?? 2%
> > > interrupts.CPU15.LOC:Local_timer_interrupts
> > > ???? 286950?????????? +28.6%???? 369104 ?? 3%
> > > interrupts.CPU15.RES:Rescheduling_interrupts
> > > ???? 204556?????????? +49.0%???? 304718 ?? 2%
> > > interrupts.CPU16.LOC:Local_timer_interrupts
> > > ?????? 5444 ? 32%???? -19.9%?????? 4360 ? 32%
> > > interrupts.CPU16.NMI:Non-maskable_interrupts
> > > ?????? 5444 ? 32%???? -19.9%?????? 4360 ? 32%
> > > interrupts.CPU16.PMI:Performance_monitoring_interrupts
> > > ???? 287623?????????? +28.1%???? 368583 ?? 3%
> > > interrupts.CPU16.RES:Rescheduling_interrupts
> > > ???? 204528?????????? +49.0%???? 304728 ?? 2%
> > > interrupts.CPU17.LOC:Local_timer_interrupts
> > > ???? 287734?????????? +28.2%???? 368920 ?? 3%
> > > interrupts.CPU17.RES:Rescheduling_interrupts
> > > ???? 204465?????????? +49.0%???? 304707 ?? 2%
> > > interrupts.CPU18.LOC:Local_timer_interrupts
> > > ?????? 5382 ? 31%???? -21.8%?????? 4211 ? 27%
> > > interrupts.CPU18.NMI:Non-maskable_interrupts
> > > ?????? 5382 ? 31%???? -21.8%?????? 4211 ? 27%
> > > interrupts.CPU18.PMI:Performance_monitoring_interrupts
> > > ???? 288778?????????? +28.1%???? 369871 ?? 3%
> > > interrupts.CPU18.RES:Rescheduling_interrupts
> > > ???? 204302?????????? +49.2%???? 304727 ?? 2%
> > > interrupts.CPU19.LOC:Local_timer_interrupts
> > > ???? 289794?????????? +27.8%???? 370247 ?? 3%
> > > interrupts.CPU19.RES:Rescheduling_interrupts
> > > ???? 204552?????????? +49.0%???? 304821 ?? 2%
> > > interrupts.CPU2.LOC:Local_timer_interrupts
> > > ???? 291056 ?? 2%???? +28.3%???? 373290 ?? 4%
> > > interrupts.CPU2.RES:Rescheduling_interrupts
> > > ???? 204468?????????? +48.9%???? 304484 ?? 2%
> > > interrupts.CPU20.LOC:Local_timer_interrupts
> > > ???? 290778?????????? +27.6%???? 371106 ?? 3%
> > > interrupts.CPU20.RES:Rescheduling_interrupts
> > > ???? 204546?????????? +48.9%???? 304619 ?? 2%
> > > interrupts.CPU21.LOC:Local_timer_interrupts
> > > ???? 289974?????????? +28.0%???? 371074 ?? 3%
> > > interrupts.CPU21.RES:Rescheduling_interrupts
> > > ???? 204694?????????? +48.8%???? 304581 ?? 2%
> > > interrupts.CPU22.LOC:Local_timer_interrupts
> > > ???? 290312?????????? +28.0%???? 371734 ?? 3%
> > > interrupts.CPU22.RES:Rescheduling_interrupts
> > > ???? 204374?????????? +49.0%???? 304493 ?? 2%
> > > interrupts.CPU23.LOC:Local_timer_interrupts
> > > ???? 290266?????????? +28.5%???? 372865 ?? 3%
> > > interrupts.CPU23.RES:Rescheduling_interrupts
> > > ???? 204518?????????? +49.0%???? 304653 ?? 2%
> > > interrupts.CPU24.LOC:Local_timer_interrupts
> > > ???? 288815 ?? 2%???? +28.7%???? 371621 ?? 4%
> > > interrupts.CPU24.RES:Rescheduling_interrupts
> > > ???? 204414?????????? +49.0%???? 304640 ?? 2%
> > > interrupts.CPU25.LOC:Local_timer_interrupts
> > > ???? 288779 ?? 2%???? +28.2%???? 370202 ?? 4%
> > > interrupts.CPU25.RES:Rescheduling_interrupts
> > > ???? 204536?????????? +49.0%???? 304730 ?? 2%
> > > interrupts.CPU26.LOC:Local_timer_interrupts
> > > ???? 287615 ?? 2%???? +28.6%???? 369843 ?? 4%
> > > interrupts.CPU26.RES:Rescheduling_interrupts
> > > ????? 72.50 ? 37%???? +55.5%???? 112.75 ? 26%
> > > interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
> > > ???? 204514?????????? +49.0%???? 304668 ?? 2%
> > > interrupts.CPU27.LOC:Local_timer_interrupts
> > > ???? 287014 ?? 2%???? +28.5%???? 368825 ?? 4%
> > > interrupts.CPU27.RES:Rescheduling_interrupts
> > > ????? 53.25 ?? 2%???? +84.5%????? 98.25 ? 22%
> > > interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
> > > ???? 204331?????????? +49.1%???? 304759 ?? 2%
> > > interrupts.CPU28.LOC:Local_timer_interrupts
> > > ???? 287278 ?? 2%???? +28.4%???? 368876 ?? 4%
> > > interrupts.CPU28.RES:Rescheduling_interrupts
> > > ????? 73.50 ? 26%??? +116.3%???? 159.00 ? 60%
> > > interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
> > > ???? 204427?????????? +49.0%???? 304651 ?? 2%
> > > interrupts.CPU29.LOC:Local_timer_interrupts
> > > ???? 287382 ?? 2%???? +28.9%???? 370434 ?? 4%
> > > interrupts.CPU29.RES:Rescheduling_interrupts
> > > ???? 204538?????????? +49.0%???? 304765 ?? 2%
> > > interrupts.CPU3.LOC:Local_timer_interrupts
> > > ?????? 5034 ? 27%???? -29.4%?????? 3552
> > > interrupts.CPU3.NMI:Non-maskable_interrupts
> > > ?????? 5034 ? 27%???? -29.4%?????? 3552
> > > interrupts.CPU3.PMI:Performance_monitoring_interrupts
> > > ???? 288881 ?? 2%???? +28.3%???? 370740 ?? 4%
> > > interrupts.CPU3.RES:Rescheduling_interrupts
> > > ???? 204750?????????? +48.8%???? 304674 ?? 2%
> > > interrupts.CPU30.LOC:Local_timer_interrupts
> > > ???? 287139 ?? 2%???? +28.8%???? 369825 ?? 4%
> > > interrupts.CPU30.RES:Rescheduling_interrupts
> > > ???? 204429?????????? +48.9%???? 304416 ?? 2%
> > > interrupts.CPU31.LOC:Local_timer_interrupts
> > > ???? 288411 ?? 2%???? +28.4%???? 370199 ?? 4%
> > > interrupts.CPU31.RES:Rescheduling_interrupts
> > > ???? 204395?????????? +49.3%???? 305106 ?? 2%
> > > interrupts.CPU32.LOC:Local_timer_interrupts
> > > ???? 289178 ?? 2%???? +28.6%???? 371830 ?? 4%
> > > interrupts.CPU32.RES:Rescheduling_interrupts
> > > ???? 204347?????????? +49.2%???? 304799 ?? 2%
> > > interrupts.CPU33.LOC:Local_timer_interrupts
> > > ???? 289689 ?? 2%???? +28.2%???? 371280 ?? 4%
> > > interrupts.CPU33.RES:Rescheduling_interrupts
> > > ???? 204280?????????? +49.1%???? 304566 ?? 2%
> > > interrupts.CPU34.LOC:Local_timer_interrupts
> > > ?????? 6418 ? 24%???? -44.5%?????? 3564
> > > interrupts.CPU34.NMI:Non-maskable_interrupts
> > > ?????? 6418 ? 24%???? -44.5%?????? 3564
> > > interrupts.CPU34.PMI:Performance_monitoring_interrupts
> > > ???? 290382 ?? 2%???? +28.3%???? 372606 ?? 4%
> > > interrupts.CPU34.RES:Rescheduling_interrupts
> > > ???? 204331?????????? +49.0%???? 304517 ?? 2%
> > > interrupts.CPU35.LOC:Local_timer_interrupts
> > > ???? 289736 ?? 2%???? +28.3%???? 371869 ?? 4%
> > > interrupts.CPU35.RES:Rescheduling_interrupts
> > > ???? 204578?????????? +48.8%???? 304423 ?? 2%
> > > interrupts.CPU36.LOC:Local_timer_interrupts
> > > ?????? 7298?????????? -39.0%?????? 4450 ? 34%
> > > interrupts.CPU36.NMI:Non-maskable_interrupts
> > > ?????? 7298?????????? -39.0%?????? 4450 ? 34%
> > > interrupts.CPU36.PMI:Performance_monitoring_interrupts
> > > ???? 287187?????????? +28.3%???? 368516 ?? 3%
> > > interrupts.CPU36.RES:Rescheduling_interrupts
> > > ???? 204538?????????? +49.0%???? 304729 ?? 2%
> > > interrupts.CPU37.LOC:Local_timer_interrupts
> > > ???? 288676?????????? +28.1%???? 369792 ?? 3%
> > > interrupts.CPU37.RES:Rescheduling_interrupts
> > > ???? 204504?????????? +49.0%???? 304687 ?? 2%
> > > interrupts.CPU38.LOC:Local_timer_interrupts
> > > ?????? 6301 ? 24%???? -29.8%?????? 4424 ? 34%
> > > interrupts.CPU38.NMI:Non-maskable_interrupts
> > > ?????? 6301 ? 24%???? -29.8%?????? 4424 ? 34%
> > > interrupts.CPU38.PMI:Performance_monitoring_interrupts
> > > ???? 287510?????????? +28.1%???? 368349 ?? 3%
> > > interrupts.CPU38.RES:Rescheduling_interrupts
> > > ???? 204566?????????? +48.9%???? 304668 ?? 2%
> > > interrupts.CPU39.LOC:Local_timer_interrupts
> > > ???? 286909?????????? +28.1%???? 367622 ?? 3%
> > > interrupts.CPU39.RES:Rescheduling_interrupts
> > > ???? 204445?????????? +49.0%???? 304686 ?? 2%
> > > interrupts.CPU4.LOC:Local_timer_interrupts
> > > ???? 288608 ?? 2%???? +28.7%???? 371493 ?? 3%
> > > interrupts.CPU4.RES:Rescheduling_interrupts
> > > ???? 204538?????????? +48.9%???? 304582 ?? 2%
> > > interrupts.CPU40.LOC:Local_timer_interrupts
> > > ???? 286219?????????? +27.9%???? 365983 ?? 3%
> > > interrupts.CPU40.RES:Rescheduling_interrupts
> > > ???? 204551?????????? +49.0%???? 304739 ?? 2%
> > > interrupts.CPU41.LOC:Local_timer_interrupts
> > > ?????? 5170 ? 29%???? -18.5%?????? 4215 ? 27%
> > > interrupts.CPU41.NMI:Non-maskable_interrupts
> > > ?????? 5170 ? 29%???? -18.5%?????? 4215 ? 27%
> > > interrupts.CPU41.PMI:Performance_monitoring_interrupts
> > > ???? 286946?????????? +27.8%???? 366717 ?? 3%
> > > interrupts.CPU41.RES:Rescheduling_interrupts
> > > ???? 204492?????????? +48.9%???? 304442 ?? 2%
> > > interrupts.CPU42.LOC:Local_timer_interrupts
> > > ???? 287384?????????? +28.0%???? 367805 ?? 3%
> > > interrupts.CPU42.RES:Rescheduling_interrupts
> > > ???? 204545?????????? +49.0%???? 304679 ?? 2%
> > > interrupts.CPU43.LOC:Local_timer_interrupts
> > > ???? 289100?????????? +27.3%???? 368059 ?? 3%
> > > interrupts.CPU43.RES:Rescheduling_interrupts
> > > ???? 204489?????????? +49.0%???? 304635 ?? 2%
> > > interrupts.CPU44.LOC:Local_timer_interrupts
> > > ?????? 6393 ? 24%???? -32.4%?????? 4319 ? 31%
> > > interrupts.CPU44.NMI:Non-maskable_interrupts
> > > ?????? 6393 ? 24%???? -32.4%?????? 4319 ? 31%
> > > interrupts.CPU44.PMI:Performance_monitoring_interrupts
> > > ???? 288835?????????? +27.8%???? 369182 ?? 3%
> > > interrupts.CPU44.RES:Rescheduling_interrupts
> > > ???? 204473?????????? +49.2%???? 304983 ?? 2%
> > > interrupts.CPU45.LOC:Local_timer_interrupts
> > > ???? 289778?????????? +27.4%???? 369059 ?? 3%
> > > interrupts.CPU45.RES:Rescheduling_interrupts
> > > ???? 204493?????????? +49.0%???? 304598 ?? 2%
> > > interrupts.CPU46.LOC:Local_timer_interrupts
> > > ???? 289485?????????? +27.9%???? 370172 ?? 3%
> > > interrupts.CPU46.RES:Rescheduling_interrupts
> > > ???? 204511?????????? +49.2%???? 305090 ?? 2%
> > > interrupts.CPU47.LOC:Local_timer_interrupts
> > > ???? 286611?????????? +28.8%???? 369021 ?? 3%
> > > interrupts.CPU47.RES:Rescheduling_interrupts
> > > ???? 204441?????????? +49.1%???? 304794 ?? 2%
> > > interrupts.CPU5.LOC:Local_timer_interrupts
> > > ???? 289245 ?? 2%???? +28.4%???? 371335 ?? 4%
> > > interrupts.CPU5.RES:Rescheduling_interrupts
> > > ???? 204246?????????? +49.0%???? 304414 ?? 2%
> > > interrupts.CPU6.LOC:Local_timer_interrupts
> > > ???? 290006 ?? 2%???? +28.3%???? 372180 ?? 4%
> > > interrupts.CPU6.RES:Rescheduling_interrupts
> > > ???? 204445?????????? +49.0%???? 304719 ?? 2%
> > > interrupts.CPU7.LOC:Local_timer_interrupts
> > > ???? 290524 ?? 2%???? +28.2%???? 372414 ?? 4%
> > > interrupts.CPU7.RES:Rescheduling_interrupts
> > > ???? 204178?????????? +49.2%???? 304663 ?? 2%
> > > interrupts.CPU8.LOC:Local_timer_interrupts
> > > ???? 291625 ?? 2%???? +27.8%???? 372747 ?? 4%
> > > interrupts.CPU8.RES:Rescheduling_interrupts
> > > ???? 204502?????????? +48.9%???? 304407 ?? 3%
> > > interrupts.CPU9.LOC:Local_timer_interrupts
> > > ???? 291615 ?? 2%???? +27.9%???? 373024 ?? 4%
> > > interrupts.CPU9.RES:Rescheduling_interrupts
> > > ??? 9814627?????????? +49.0%?? 14624016 ?? 2%
> > > interrupts.LOC:Local_timer_interrupts
> > > ?? 13869791?????????? +28.2%?? 17785831 ?? 3%
> > > interrupts.RES:Rescheduling_interrupts
> > > ????? 25728?????????? +47.6%????? 37969??????? softirqs.CPU0.RCU
> > > ????? 34944?????????? +41.7%????? 49503 ?? 3%? softirqs.CPU0.SCHED
> > > ????? 42375?????????? +38.5%????? 58673 ?? 2%? softirqs.CPU0.TIMER
> > > ????? 25348?????????? +53.8%????? 38976 ?? 5%? softirqs.CPU1.RCU
> > > ????? 30986?????????? +46.2%????? 45315 ?? 3%? softirqs.CPU1.SCHED
> > > ????? 41265?????????? +44.3%????? 59538 ?? 3%? softirqs.CPU1.TIMER
> > > ????? 25149?????????? +48.3%????? 37288??????? softirqs.CPU10.RCU
> > > ????? 29752?????????? +47.1%????? 43764 ?? 2%? softirqs.CPU10.SCHED
> > > ????? 40609?????????? +41.5%????? 57449??????? softirqs.CPU10.TIMER
> > > ????? 25218?????????? +48.3%????? 37399??????? softirqs.CPU11.RCU
> > > ????? 30294?????????? +44.6%????? 43800??????? softirqs.CPU11.SCHED
> > > ????? 41371?????????? +39.2%????? 57578??????? softirqs.CPU11.TIMER
> > > ????? 24427?????????? +48.9%????? 36361??????? softirqs.CPU12.RCU
> > > ????? 30438?????????? +44.0%????? 43835??????? softirqs.CPU12.SCHED
> > > ????? 41045 ?? 3%???? +43.0%????? 58687??????? softirqs.CPU12.TIMER
> > > ????? 24247?????????? +47.0%????? 35645??????? softirqs.CPU13.RCU
> > > ????? 30155?????????? +44.5%????? 43577??????? softirqs.CPU13.SCHED
> > > ????? 41148 ?? 2%???? +40.3%????? 57720??????? softirqs.CPU13.TIMER
> > > ????? 24506?????????? +46.3%????? 35856??????? softirqs.CPU14.RCU
> > > ????? 29991?????????? +44.1%????? 43229??????? softirqs.CPU14.SCHED
> > > ????? 41190 ?? 2%???? +39.7%????? 57552??????? softirqs.CPU14.TIMER
> > > ????? 24412?????????? +46.7%????? 35800??????? softirqs.CPU15.RCU
> > > ????? 30135?????????? +44.1%????? 43430??????? softirqs.CPU15.SCHED
> > > ????? 45081 ? 15%???? +28.0%????? 57684??????? softirqs.CPU15.TIMER
> > > ????? 25327?????????? +47.0%????? 37225??????? softirqs.CPU16.RCU
> > > ????? 30230?????????? +43.8%????? 43461??????? softirqs.CPU16.SCHED
> > > ????? 41157?????????? +39.6%????? 57448??????? softirqs.CPU16.TIMER
> > > ????? 25136?????????? +47.8%????? 37142??????? softirqs.CPU17.RCU
> > > ????? 29894?????????? +44.5%????? 43212??????? softirqs.CPU17.SCHED
> > > ????? 41961 ?? 5%???? +36.5%????? 57281??????? softirqs.CPU17.TIMER
> > > ????? 25431?????????? +46.1%????? 37160??????? softirqs.CPU18.RCU
> > > ????? 30062?????????? +44.0%????? 43281??????? softirqs.CPU18.SCHED
> > > ????? 40956 ?? 2%???? +42.1%????? 58217??????? softirqs.CPU18.TIMER
> > > ????? 25496?????????? +44.8%????? 36923 ?? 2%? softirqs.CPU19.RCU
> > > ????? 29989?????????? +46.5%????? 43946??????? softirqs.CPU19.SCHED
> > > ????? 41740 ?? 2%???? +67.8%????? 70057 ? 17%? softirqs.CPU19.TIMER
> > > ????? 25414?????????? +48.3%????? 37678??????? softirqs.CPU2.RCU
> > > ????? 30905?????????? +43.4%????? 44309 ?? 2%? softirqs.CPU2.SCHED
> > > ????? 41083?????????? +40.6%????? 57768 ?? 2%? softirqs.CPU2.TIMER
> > > ????? 25124?????????? +46.8%????? 36892??????? softirqs.CPU20.RCU
> > > ????? 30001?????????? +44.7%????? 43416??????? softirqs.CPU20.SCHED
> > > ????? 41060 ?? 7%???? +39.2%????? 57151??????? softirqs.CPU20.TIMER
> > > ????? 24992?????????? +47.6%????? 36896??????? softirqs.CPU21.RCU
> > > ????? 30109?????????? +44.4%????? 43467??????? softirqs.CPU21.SCHED
> > > ????? 40055 ?? 3%???? +42.5%????? 57081 ?? 2%? softirqs.CPU21.TIMER
> > > ????? 25072?????????? +46.2%????? 36656??????? softirqs.CPU22.RCU
> > > ????? 30296?????????? +43.3%????? 43425??????? softirqs.CPU22.SCHED
> > > ????? 24882?????????? +48.8%????? 37031??????? softirqs.CPU23.RCU
> > > ????? 29900?????????? +43.7%????? 42971??????? softirqs.CPU23.SCHED
> > > ????? 40388 ?? 3%???? +44.8%????? 58494 ?? 3%? softirqs.CPU23.TIMER
> > > ????? 25007?????????? +48.1%????? 37036??????? softirqs.CPU24.RCU
> > > ????? 29755?????????? +45.9%????? 43399??????? softirqs.CPU24.SCHED
> > > ????? 40786?????????? +40.6%????? 57353??????? softirqs.CPU24.TIMER
> > > ????? 25031?????????? +47.3%????? 36875??????? softirqs.CPU25.RCU
> > > ????? 29841 ?? 2%???? +45.3%????? 43355??????? softirqs.CPU25.SCHED
> > > ????? 40971?????????? +39.8%????? 57266??????? softirqs.CPU25.TIMER
> > > ????? 25145?????????? +47.6%????? 37114??????? softirqs.CPU26.RCU
> > > ????? 29462 ?? 2%???? +48.5%????? 43739 ?? 2%? softirqs.CPU26.SCHED
> > > ????? 40553?????????? +41.8%????? 57521??????? softirqs.CPU26.TIMER
> > > ????? 25184?????????? +47.6%????? 37169??????? softirqs.CPU27.RCU
> > > ????? 29867?????????? +45.5%????? 43450 ?? 3%? softirqs.CPU27.SCHED
> > > ????? 41118?????????? +39.3%????? 57261??????? softirqs.CPU27.TIMER
> > > ????? 25452?????????? +47.1%????? 37440??????? softirqs.CPU28.RCU
> > > ????? 29714?????????? +46.2%????? 43440 ?? 2%? softirqs.CPU28.SCHED
> > > ????? 41209 ?? 2%???? +37.8%????? 56783 ?? 2%? softirqs.CPU28.TIMER
> > > ????? 25334?????????? +48.8%????? 37696 ?? 2%? softirqs.CPU29.RCU
> > > ????? 29894?????????? +46.3%????? 43749??????? softirqs.CPU29.SCHED
> > > ????? 41413 ?? 2%???? +43.3%????? 59346 ?? 5%? softirqs.CPU29.TIMER
> > > ????? 25514?????????? +47.0%????? 37495??????? softirqs.CPU3.RCU
> > > ????? 30016?????????? +46.0%????? 43831 ?? 2%? softirqs.CPU3.SCHED
> > > ????? 41371 ?? 2%???? +39.3%????? 57611??????? softirqs.CPU3.TIMER
> > > ????? 25206?????????? +47.4%????? 37149??????? softirqs.CPU30.RCU
> > > ????? 30167?????????? +44.0%????? 43451 ?? 2%? softirqs.CPU30.SCHED
> > > ????? 41166 ?? 2%???? +58.1%????? 65092 ? 19%? softirqs.CPU30.TIMER
> > > ????? 25211?????????? +46.8%????? 37016??????? softirqs.CPU31.RCU
> > > ????? 30004 ?? 2%???? +45.2%????? 43569??????? softirqs.CPU31.SCHED
> > > ????? 41541 ?? 2%???? +36.9%????? 56874??????? softirqs.CPU31.TIMER
> > > ????? 24432?????????? +46.9%????? 35885??????? softirqs.CPU32.RCU
> > > ????? 30206?????????? +43.4%????? 43322 ?? 2%? softirqs.CPU32.SCHED
> > > ????? 41069?????????? +40.7%????? 57781 ?? 2%? softirqs.CPU32.TIMER
> > > ????? 24303?????????? +48.0%????? 35973??????? softirqs.CPU33.RCU
> > > ????? 30238?????????? +43.8%????? 43489??????? softirqs.CPU33.SCHED
> > > ????? 41161?????????? +38.2%????? 56872??????? softirqs.CPU33.TIMER
> > > ????? 24308?????????? +47.6%????? 35876??????? softirqs.CPU34.RCU
> > > ????? 29803?????????? +46.5%????? 43647 ?? 2%? softirqs.CPU34.SCHED
> > > ????? 40809?????????? +39.2%????? 56817??????? softirqs.CPU34.TIMER
> > > ????? 24291?????????? +47.7%????? 35877??????? softirqs.CPU35.RCU
> > > ????? 30092?????????? +44.2%????? 43386??????? softirqs.CPU35.SCHED
> > > ????? 40607?????????? +41.5%????? 57480 ?? 2%? softirqs.CPU35.TIMER
> > > ????? 25605?????????? +49.5%????? 38286??????? softirqs.CPU36.RCU
> > > ????? 29900?????????? +44.6%????? 43238??????? softirqs.CPU36.SCHED
> > > ????? 45207 ? 15%???? +27.4%????? 57602??????? softirqs.CPU36.TIMER
> > > ????? 25555 ?? 2%???? +49.0%????? 38070??????? softirqs.CPU37.RCU
> > > ????? 29926?????????? +45.2%????? 43460??????? softirqs.CPU37.SCHED
> > > ????? 42619 ?? 8%???? +36.7%????? 58242??????? softirqs.CPU37.TIMER
> > > ????? 27683 ?? 9%???? +50.2%????? 41577 ?? 4%? softirqs.CPU38.RCU
> > > ????? 29700?????????? +45.6%????? 43230??????? softirqs.CPU38.SCHED
> > > ????? 41779 ?? 4%???? +43.5%????? 59948??????? softirqs.CPU38.TIMER
> > > ????? 26699 ?? 7%???? +42.1%????? 37951??????? softirqs.CPU39.RCU
> > > ????? 30061?????????? +43.4%????? 43102??????? softirqs.CPU39.SCHED
> > > ????? 42312 ?? 5%???? +36.1%????? 57581 ?? 2%? softirqs.CPU39.TIMER
> > > ????? 25219?????????? +47.7%????? 37247??????? softirqs.CPU4.RCU
> > > ????? 30363 ?? 3%???? +45.9%????? 44295??????? softirqs.CPU4.SCHED
> > > ????? 41108 ?? 2%???? +39.6%????? 57371 ?? 2%? softirqs.CPU4.TIMER
> > > ????? 25413?????????? +50.6%????? 38268??????? softirqs.CPU40.RCU
> > > ????? 29910?????????? +46.0%????? 43665??????? softirqs.CPU40.SCHED
> > > ????? 39987 ?? 2%???? +45.9%????? 58322??????? softirqs.CPU40.TIMER
> > > ????? 25569?????????? +49.5%????? 38238??????? softirqs.CPU41.RCU
> > > ????? 29867?????????? +44.8%????? 43247??????? softirqs.CPU41.SCHED
> > > ????? 25492?????????? +49.8%????? 38185??????? softirqs.CPU42.RCU
> > > ????? 29871?????????? +45.4%????? 43440??????? softirqs.CPU42.SCHED
> > > ????? 40076?????????? +44.8%????? 58020??????? softirqs.CPU42.TIMER
> > > ????? 25281?????????? +48.9%????? 37640??????? softirqs.CPU43.RCU
> > > ????? 30025?????????? +43.9%????? 43220??????? softirqs.CPU43.SCHED
> > > ????? 40222 ?? 3%???? +44.7%????? 58195??????? softirqs.CPU43.TIMER
> > > ????? 25183?????????? +49.3%????? 37611??????? softirqs.CPU44.RCU
> > > ????? 30123?????????? +43.6%????? 43249??????? softirqs.CPU44.SCHED
> > > ????? 41090 ?? 6%???? +42.5%????? 58537??????? softirqs.CPU44.TIMER
> > > ????? 25314?????????? +49.6%????? 37863??????? softirqs.CPU45.RCU
> > > ????? 29998?????????? +43.2%????? 42957??????? softirqs.CPU45.SCHED
> > > ????? 39705 ?? 3%???? +43.6%????? 57033??????? softirqs.CPU45.TIMER
> > > ????? 25500?????????? +46.2%????? 37274??????? softirqs.CPU46.RCU
> > > ????? 29921?????????? +44.7%????? 43286??????? softirqs.CPU46.SCHED
> > > ????? 41088 ?? 5%???? +39.9%????? 57463 ?? 3%? softirqs.CPU46.TIMER
> > > ????? 25564?????????? +48.4%????? 37931??????? softirqs.CPU47.RCU
> > > ????? 30178?????????? +42.4%????? 42980??????? softirqs.CPU47.SCHED
> > > ????? 40738 ?? 4%???? +42.2%????? 57939 ?? 3%? softirqs.CPU47.TIMER
> > > ????? 26467 ?? 8%???? +40.8%????? 37278??????? softirqs.CPU5.RCU
> > > ????? 30334 ?? 2%???? +44.1%????? 43714??????? softirqs.CPU5.SCHED
> > > ????? 42268 ?? 4%???? +35.8%????? 57394??????? softirqs.CPU5.TIMER
> > > ????? 25343?????????? +47.0%????? 37245??????? softirqs.CPU6.RCU
> > > ????? 30100?????????? +45.0%????? 43643 ?? 2%? softirqs.CPU6.SCHED
> > > ????? 41270?????????? +43.6%????? 59249 ?? 5%? softirqs.CPU6.TIMER
> > > ????? 25331?????????? +47.7%????? 37418??????? softirqs.CPU7.RCU
> > > ????? 30210?????????? +44.7%????? 43727 ?? 2%? softirqs.CPU7.SCHED
> > > ????? 41028?????????? +41.9%????? 58206 ?? 2%? softirqs.CPU7.TIMER
> > > ????? 25483?????????? +47.2%????? 37514??????? softirqs.CPU8.RCU
> > > ????? 30324 ?? 2%???? +43.4%????? 43483 ?? 2%? softirqs.CPU8.SCHED
> > > ????? 41027?????????? +40.4%????? 57591 ?? 2%? softirqs.CPU8.TIMER
> > > ????? 25546?????????? +47.5%????? 37687??????? softirqs.CPU9.RCU
> > > ????? 30218?????????? +44.8%????? 43771 ?? 2%? softirqs.CPU9.SCHED
> > > ????? 41193?????????? +40.1%????? 57716 ?? 2%? softirqs.CPU9.TIMER
> > > ??? 1212606?????????? +47.7%??? 1790808??????? softirqs.RCU
> > > ??? 1448198?????????? +44.7%??? 2095500??????? softirqs.SCHED
> > > ??? 1990064?????????? +40.7%??? 2800202??????? softirqs.TIMER
> > > ?????? 8.56 ?? 3%????? -4.3??????? 4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
> > >
> > > ?????? 8.50 ?? 3%????? -4.3??????? 4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
> > >
> > > ?????? 8.47 ?? 3%????? -4.3??????? 4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> > >
> > > ????? 78.55??????????? -4.1?????? 74.44
> > > perf-profile.calltrace.cycles-pp.write
> > > ?????? 8.74 ?? 3%????? -4.0??????? 4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> > >
> > > ????? 77.69??????????? -3.9?????? 73.75
> > > perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> > > ????? 77.64??????????? -3.9?????? 73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> > >
> > > ????? 77.13??????????? -3.8?????? 73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> > >
> > > ????? 77.08??????????? -3.8?????? 73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> > >
> > > ????? 76.61??????????? -3.7?????? 72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > >
> > > ????? 76.43??????????? -3.6?????? 72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
> > >
> > > ????? 69.22??????????? -2.6?????? 66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
> > >
> > > ?????? 4.70 ?? 2%????? -2.2??????? 2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> > >
> > > ?????? 4.63 ?? 2%????? -2.2??????? 2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> > >
> > > ?????? 4.60 ?? 2%????? -2.2??????? 2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
> > >
> > > ?????? 4.32 ?? 2%????? -2.1??????? 2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
> > >
> > > ?????? 3.72 ?? 4%????? -2.1??????? 1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
> > >
> > > ?????? 3.71 ?? 4%????? -2.1??????? 1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
> > >
> > > ?????? 2.63 ?? 4%????? -2.0??????? 0.63 ?? 3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
> > >
> > > ?????? 7.72??????????? -2.0??????? 5.74 ?? 2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> > >
> > > ?????? 3.08 ?? 5%????? -1.9??????? 1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
> > >
> > > ?????? 2.49 ?? 5%????? -1.9??????? 0.55 ?? 3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
> > >
> > > ?????? 2.48 ?? 5%????? -1.9??????? 0.55 ?? 3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
> > >
> > > ?????? 2.41 ?? 6%????? -1.8??????? 0.61 ?? 2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
> > >
> > > ?????? 4.90??????????? -1.5??????? 3.40 ?? 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> > >
> > > ?????? 4.78??????????? -1.5??????? 3.30 ?? 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
> > >
> > > ?????? 2.35??????????? -0.6??????? 1.77 ?? 2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> > >
> > > ?????? 2.36??????????? -0.6??????? 1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
> > >
> > > ?????? 3.40??????????? -0.5??????? 2.87 ?? 6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
> > >
> > > ?????? 1.83??????????? -0.5??????? 1.33 ?? 2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
> > >
> > > ?????? 3.26??????????? -0.5??????? 2.75 ?? 6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
> > >
> > > ?????? 2.85??????????? -0.4??????? 2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
> > >
> > > ?????? 2.79??????????? -0.4??????? 2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
> > >
> > > ?????? 2.83??????????? -0.4??????? 2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
> > >
> > > ?????? 1.48??????????? -0.4??????? 1.05 ?? 2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
> > >
> > > ?????? 0.61 ?? 2%????? -0.4??????? 0.26 ?100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
> > >
> > > ?????? 1.89??????????? -0.3??????? 1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> > >
> > > ?????? 1.71??????????? -0.3??????? 1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
> > >
> > > ?????? 1.65??????????? -0.3??????? 1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
> > >
> > > ?????? 0.69??????????? -0.3??????? 0.38 ? 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
> > >
> > > ?????? 1.59??????????? -0.3??????? 1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
> > >
> > > ?????? 0.86??????????? -0.3??????? 0.58 ?? 2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
> > >
> > > ?????? 1.52??????????? -0.3??????? 1.26 ?? 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
> > >
> > > ?????? 1.75??????????? -0.2??????? 1.50 ?? 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
> > >
> > > ?????? 1.74??????????? -0.2??????? 1.49 ?? 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
> > >
> > > ?????? 1.74??????????? -0.2??????? 1.50 ?? 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
> > >
> > > ?????? 2.15??????????? -0.2??????? 1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
> > >
> > > ?????? 0.79 ?? 3%????? -0.2??????? 0.60 ?? 4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
> > >
> > > ?????? 1.81??????????? -0.2??????? 1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
> > >
> > > ?????? 1.87??????????? -0.2??????? 1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
> > >
> > > ?????? 0.70 ?? 4%????? -0.2??????? 0.54 ?? 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
> > >
> > > ?????? 0.81??????????? -0.1??????? 0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
> > >
> > > ?????? 0.65 ?? 2%????? -0.1??????? 0.55 ?? 2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
> > >
> > > ?????? 0.62 ?? 2%????? -0.1??????? 0.52 ?? 3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
> > >
> > > ?????? 1.18??????????? -0.0??????? 1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > >
> > > ?????? 1.01??????????? -0.0??????? 0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
> > >
> > > ?????? 1.00??????????? -0.0??????? 0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
> > >
> > > ?????? 0.96??????????? -0.0??????? 0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
> > >
> > > ?????? 1.01??????????? -0.0??????? 0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
> > >
> > > ?????? 0.59 ?? 3%????? +0.1??????? 0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > >
> > > ?????? 0.58 ?? 3%????? +0.1??????? 0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
> > >
> > > ?????? 0.53??????????? +0.1??????? 0.63 ?? 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
> > >
> > > ?????? 0.91 ?? 3%????? +0.1??????? 1.02 ?? 2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
> > >
> > > ?????? 0.54??????????? +0.1??????? 0.65 ?? 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
> > >
> > > ?????? 0.57??????????? +0.1??????? 0.69 ?? 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
> > >
> > > ?????? 1.83??????????? +0.2??????? 1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
> > >
> > > ?????? 0.00??????????? +0.6??????? 0.57 ?? 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
> > >
> > > ?????? 0.00??????????? +0.6??????? 0.57 ?? 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
> > >
> > > ?????? 0.00??????????? +0.6??????? 0.57 ?? 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
> > >
> > > ?????? 0.00??????????? +0.7??????? 0.70 ?? 2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
> > >
> > > ?????? 0.00??????????? +0.8??????? 0.75 ?? 4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
> > >
> > > ?????? 0.00??????????? +0.8??????? 0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
> > >
> > > ?????? 0.00??????????? +0.8??????? 0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
> > >
> > > ?????? 0.00??????????? +0.8??????? 0.80 ? 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
> > >
> > > ?????? 0.00??????????? +0.8??????? 0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
> > >
> > > ?????? 0.00??????????? +0.8??????? 0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
> > >
> > > ?????? 0.00??????????? +0.9??????? 0.89 ?? 2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
> > >
> > > ?????? 0.00??????????? +0.9??????? 0.92 ? 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
> > >
> > > ?????? 0.00??????????? +1.0??????? 0.97 ? 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
> > >
> > > ????? 35.68??????????? +1.0?????? 36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
> > >
> > > ?????? 0.00??????????? +1.0??????? 1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
> > >
> > > ?????? 0.00??????????? +1.0??????? 1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
> > >
> > > ?????? 9.35??????????? +1.2?????? 10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> > >
> > > ?????? 9.61??????????? +1.2?????? 10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> > >
> > > ?????? 9.63??????????? +1.2?????? 10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > >
> > > ????? 12.24??????????? +1.3?????? 13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> > >
> > > ????? 12.25??????????? +1.3?????? 13.60
> > > perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> > > ????? 12.25??????????? +1.3?????? 13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> > >
> > > ????? 12.49??????????? +1.4?????? 13.89
> > > perf-profile.calltrace.cycles-pp.secondary_startup_64
> > > ????? 14.38??????????? +1.4?????? 15.81 ?? 2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> > >
> > > ????? 13.41??????????? +1.4?????? 14.86 ?? 2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> > >
> > > ????? 13.37??????????? +1.5?????? 14.82 ?? 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
> > >
> > > ????? 13.34??????????? +1.5?????? 14.80 ?? 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
> > >
> > > ????? 58.91??????????? +1.6?????? 60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
> > >
> > > ?????? 0.00??????????? +2.2??????? 2.24 ? 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
> > >
> > > ?????? 5.63??????????? +2.5??????? 8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
> > >
> > > ?????? 5.65??????????? +2.5??????? 8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
> > >
> > > ?????? 5.69??????????? +2.5??????? 8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
> > >
> > > ?????? 7.04??????????? +2.8??????? 9.80 ?? 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
> > >
> > > ?????? 7.54??????????? +2.9?????? 10.45 ?? 2%
> > > perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
> > > ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
> > > perf-profile.calltrace.cycles-pp.ret_from_fork
> > > ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
> > > perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> > > ?????? 0.00??????????? +3.3??????? 3.30 ?? 9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
> > >
> > > ?????? 0.00??????????? +3.3??????? 3.32 ?? 8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
> > >
> > > ?????? 8.56 ?? 3%????? -4.3??????? 4.25
> > > perf-profile.children.cycles-pp.__filemap_fdatawrite_range
> > > ?????? 8.47 ?? 3%????? -4.3??????? 4.18
> > > perf-profile.children.cycles-pp.xfs_vm_writepages
> > > ?????? 8.50 ?? 3%????? -4.3??????? 4.20
> > > perf-profile.children.cycles-pp.do_writepages
> > > ????? 78.64??????????? -4.1?????? 74.51
> > > perf-profile.children.cycles-pp.write
> > > ????? 78.55??????????? -4.1?????? 74.44
> > > perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> > > ????? 78.50??????????? -4.1?????? 74.41
> > > perf-profile.children.cycles-pp.do_syscall_64
> > > ?????? 8.75 ?? 3%????? -4.1??????? 4.70
> > > perf-profile.children.cycles-pp.file_write_and_wait_range
> > > ????? 77.14??????????? -3.8?????? 73.37
> > > perf-profile.children.cycles-pp.ksys_write
> > > ????? 77.09??????????? -3.8?????? 73.33
> > > perf-profile.children.cycles-pp.vfs_write
> > > ????? 76.62??????????? -3.7?????? 72.97
> > > perf-profile.children.cycles-pp.new_sync_write
> > > ????? 76.44??????????? -3.6?????? 72.81
> > > perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
> > > ????? 69.23??????????? -2.6?????? 66.62
> > > perf-profile.children.cycles-pp.xfs_file_fsync
> > > ?????? 4.70 ?? 2%????? -2.2??????? 2.50
> > > perf-profile.children.cycles-pp.iomap_submit_ioend
> > > ?????? 4.89 ?? 2%????? -2.1??????? 2.75
> > > perf-profile.children.cycles-pp.submit_bio
> > > ?????? 4.85 ?? 2%????? -2.1??????? 2.71
> > > perf-profile.children.cycles-pp.generic_make_request
> > > ?????? 3.73 ?? 4%????? -2.1??????? 1.63
> > > perf-profile.children.cycles-pp.iomap_writepages
> > > ?????? 3.72 ?? 4%????? -2.1??????? 1.63
> > > perf-profile.children.cycles-pp.write_cache_pages
> > > ?????? 4.43 ?? 2%????? -2.1??????? 2.38
> > > perf-profile.children.cycles-pp.brd_make_request
> > > ?????? 7.73??????????? -2.0??????? 5.75 ?? 2%
> > > perf-profile.children.cycles-pp.__flush_work
> > > ?????? 3.08 ?? 4%????? -1.9??????? 1.14
> > > perf-profile.children.cycles-pp.iomap_writepage_map
> > > ?????? 6.00??????????? -1.8??????? 4.20 ?? 2%
> > > perf-profile.children.cycles-pp._raw_spin_lock_irq
> > > ?????? 2.50 ?? 4%????? -1.8??????? 0.70 ?? 3%
> > > perf-profile.children.cycles-pp.end_page_writeback
> > > ?????? 2.50 ?? 5%????? -1.8??????? 0.70 ?? 4%
> > > perf-profile.children.cycles-pp.test_clear_page_writeback
> > > ?????? 2.42 ?? 6%????? -1.8??????? 0.61 ?? 2%
> > > perf-profile.children.cycles-pp.__test_set_page_writeback
> > > ?????? 2.08 ?? 7%????? -1.6??????? 0.44 ?? 4%
> > > perf-profile.children.cycles-pp.sb_mark_inode_writeback
> > > ?????? 2.08 ?? 6%????? -1.6??????? 0.44 ?? 5%
> > > perf-profile.children.cycles-pp.sb_clear_inode_writeback
> > > ?????? 2.70 ?? 4%????? -1.0??????? 1.67 ?? 2%
> > > perf-profile.children.cycles-pp.iomap_finish_ioend
> > > ?????? 2.36??????????? -0.6??????? 1.79
> > > perf-profile.children.cycles-pp.iomap_file_buffered_write
> > > ?????? 2.35??????????? -0.6??????? 1.78
> > > perf-profile.children.cycles-pp.iomap_apply
> > > ?????? 1.84??????????? -0.5??????? 1.33 ?? 2%
> > > perf-profile.children.cycles-pp.iomap_write_actor
> > > ?????? 2.41 ?? 4%????? -0.5??????? 1.94 ?? 2%
> > > perf-profile.children.cycles-pp.reschedule_interrupt
> > > ?????? 3.44 ?? 2%????? -0.5??????? 2.98
> > > perf-profile.children.cycles-pp.sched_ttwu_pending
> > > ?????? 2.87??????????? -0.5??????? 2.42
> > > perf-profile.children.cycles-pp.complete
> > > ?????? 2.83??????????? -0.4??????? 2.38
> > > perf-profile.children.cycles-pp.swake_up_locked
> > > ?????? 2.28 ?? 4%????? -0.4??????? 1.84 ?? 2%
> > > perf-profile.children.cycles-pp.scheduler_ipi
> > > ?????? 1.56??????????? -0.4??????? 1.17 ?? 2%
> > > perf-profile.children.cycles-pp.brd_do_bvec
> > > ?????? 1.89??????????? -0.4??????? 1.54
> > > perf-profile.children.cycles-pp.wait_for_completion
> > > ?????? 1.71??????????? -0.3??????? 1.38
> > > perf-profile.children.cycles-pp.schedule_timeout
> > > ?????? 0.88??????????? -0.2??????? 0.68 ?? 2%
> > > perf-profile.children.cycles-pp.rwsem_down_read_slowpath
> > > ?????? 0.69??????????? -0.2??????? 0.51 ?? 2%
> > > perf-profile.children.cycles-pp.iomap_write_begin
> > > ?????? 0.67 ?? 2%????? -0.2??????? 0.49
> > > perf-profile.children.cycles-pp.lseek64
> > > ?????? 1.33 ?? 5%????? -0.2??????? 1.17 ?? 3%
> > > perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> > > ?????? 0.51??????????? -0.2??????? 0.36
> > > perf-profile.children.cycles-pp.grab_cache_page_write_begin
> > > ?????? 0.47??????????? -0.1??????? 0.33
> > > perf-profile.children.cycles-pp.pagecache_get_page
> > > ?????? 0.51 ?? 3%????? -0.1??????? 0.38 ?? 2%
> > > perf-profile.children.cycles-pp.iomap_write_end
> > > ?????? 0.57 ?? 2%????? -0.1??????? 0.46
> > > perf-profile.children.cycles-pp.blkdev_issue_flush
> > > ?????? 0.38??????????? -0.1??????? 0.26 ?? 5%
> > > perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
> > > ?????? 0.34??????????? -0.1??????? 0.23 ?? 6%
> > > perf-profile.children.cycles-pp.copyin
> > > ?????? 0.33??????????? -0.1??????? 0.22 ?? 6%
> > > perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
> > > ?????? 0.65 ?? 2%????? -0.1??????? 0.56 ?? 2%
> > > perf-profile.children.cycles-pp.rwsem_optimistic_spin
> > > ?????? 0.37??????????? -0.1??????? 0.28 ?? 2%
> > > perf-profile.children.cycles-pp.bio_alloc_bioset
> > > ?????? 0.31 ?? 4%????? -0.1??????? 0.22
> > > perf-profile.children.cycles-pp.syscall_return_via_sysret
> > > ?????? 0.56 ?? 3%????? -0.1??????? 0.48 ?? 3%
> > > perf-profile.children.cycles-pp.osq_lock
> > > ?????? 0.26 ?? 3%????? -0.1??????? 0.18 ?? 4%
> > > perf-profile.children.cycles-pp.entry_SYSCALL_64
> > > ?????? 0.28 ?? 2%????? -0.1??????? 0.21 ?? 5%
> > > perf-profile.children.cycles-pp.mempool_alloc
> > > ?????? 0.26 ?? 6%????? -0.1??????? 0.18 ?? 8%
> > > perf-profile.children.cycles-pp.xlog_state_get_iclog_space
> > > ?????? 0.29??????????? -0.1??????? 0.22 ?? 3%
> > > perf-profile.children.cycles-pp.find_get_entry
> > > ?????? 0.34 ?? 2%????? -0.1??????? 0.27 ?? 4%
> > > perf-profile.children.cycles-pp.generic_make_request_checks
> > > ?????? 0.30 ?? 5%????? -0.1??????? 0.23 ?? 3%
> > > perf-profile.children.cycles-pp.iomap_set_page_dirty
> > > ?????? 0.32 ?? 2%????? -0.1??????? 0.26 ?? 5%
> > > perf-profile.children.cycles-pp.brd_lookup_page
> > > ?????? 0.18 ?? 2%????? -0.1??????? 0.12 ?? 6%
> > > perf-profile.children.cycles-pp.iomap_set_range_uptodate
> > > ?????? 0.15 ?? 2%????? -0.1??????? 0.09 ?? 4%
> > > perf-profile.children.cycles-pp.__mod_lruvec_state
> > > ?????? 0.15 ?? 5%????? -0.1??????? 0.10 ?? 9%
> > > perf-profile.children.cycles-pp.wake_up_q
> > > ?????? 0.21 ?? 5%????? -0.1??????? 0.15 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_ilock
> > > ?????? 0.13 ?? 3%????? -0.0??????? 0.08 ? 10%
> > > perf-profile.children.cycles-pp.rwsem_wake
> > > ?????? 0.20 ?? 2%????? -0.0??????? 0.15 ?? 5%
> > > perf-profile.children.cycles-pp.ksys_lseek
> > > ?????? 0.20 ?? 3%????? -0.0??????? 0.15 ?? 3%
> > > perf-profile.children.cycles-pp.xas_load
> > > ?????? 0.20 ?? 5%????? -0.0??????? 0.16 ?? 2%
> > > perf-profile.children.cycles-pp.__set_page_dirty
> > > ?????? 0.19 ?? 2%????? -0.0??????? 0.15 ?? 2%
> > > perf-profile.children.cycles-pp.xfs_iunlock
> > > ?????? 0.40??????????? -0.0??????? 0.36 ?? 4%
> > > perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
> > > ?????? 0.17 ?? 2%????? -0.0??????? 0.12 ?? 4%
> > > perf-profile.children.cycles-pp.clear_page_dirty_for_io
> > > ?????? 0.20 ?? 3%????? -0.0??????? 0.16 ?? 4%
> > > perf-profile.children.cycles-pp.submit_bio_wait
> > > ?????? 0.18 ?? 4%????? -0.0??????? 0.15 ?? 7%
> > > perf-profile.children.cycles-pp.security_file_permission
> > > ?????? 0.10??????????? -0.0??????? 0.06 ?? 6%
> > > perf-profile.children.cycles-pp.evict
> > > ?????? 0.10??????????? -0.0??????? 0.06 ?? 6%
> > > perf-profile.children.cycles-pp.truncate_inode_pages_range
> > > ?????? 0.06 ?? 6%????? -0.0??????? 0.03 ?100%
> > > perf-profile.children.cycles-pp.vfs_unlink
> > > ?????? 0.08 ?? 6%????? -0.0??????? 0.04 ? 58%
> > > perf-profile.children.cycles-pp.add_to_page_cache_lru
> > > ?????? 0.26 ?? 2%????? -0.0??????? 0.23 ?? 3%
> > > perf-profile.children.cycles-pp.reweight_entity
> > > ?????? 0.12 ?? 6%????? -0.0??????? 0.09 ?? 4%
> > > perf-profile.children.cycles-pp.bio_endio
> > > ?????? 0.10 ?? 7%????? -0.0??????? 0.07 ?? 7%
> > > perf-profile.children.cycles-pp.do_sys_open
> > > ?????? 0.10 ?? 7%????? -0.0??????? 0.07 ?? 7%
> > > perf-profile.children.cycles-pp.do_sys_openat2
> > > ?????? 0.17 ?? 2%????? -0.0??????? 0.14
> > > perf-profile.children.cycles-pp.__pagevec_release
> > > ?????? 0.13 ?? 5%????? -0.0??????? 0.10 ?? 4%
> > > perf-profile.children.cycles-pp.down_write
> > > ?????? 0.10 ?? 5%????? -0.0??????? 0.06 ?? 6%
> > > perf-profile.children.cycles-pp.do_filp_open
> > > ?????? 0.15 ?? 4%????? -0.0??????? 0.12 ?? 5%
> > > perf-profile.children.cycles-pp.tag_pages_for_writeback
> > > ?????? 0.11 ?? 9%????? -0.0??????? 0.08 ?? 5%
> > > perf-profile.children.cycles-pp.bio_associate_blkg
> > > ?????? 0.10 ?? 5%????? -0.0??????? 0.07 ?? 7%
> > > perf-profile.children.cycles-pp.open64
> > > ?????? 0.09 ?? 4%????? -0.0??????? 0.06 ?? 6%
> > > perf-profile.children.cycles-pp.path_openat
> > > ?????? 0.16 ?? 8%????? -0.0??????? 0.13 ?? 3%
> > > perf-profile.children.cycles-pp.pagevec_lookup_range_tag
> > > ?????? 0.12 ?? 8%????? -0.0??????? 0.10 ?? 5%
> > > perf-profile.children.cycles-pp.account_page_dirtied
> > > ?????? 0.10 ?? 4%????? -0.0??????? 0.08 ?? 6%
> > > perf-profile.children.cycles-pp.__fdget_pos
> > > ?????? 0.14 ?? 3%????? -0.0??????? 0.11 ?? 7%
> > > perf-profile.children.cycles-pp.memset_erms
> > > ?????? 0.11 ?? 3%????? -0.0??????? 0.09 ?? 4%
> > > perf-profile.children.cycles-pp.switch_fpu_return
> > > ?????? 0.23??????????? -0.0??????? 0.21 ?? 3%
> > > perf-profile.children.cycles-pp.__radix_tree_lookup
> > > ?????? 0.18 ?? 2%????? -0.0??????? 0.16 ?? 9%
> > > perf-profile.children.cycles-pp.insert_work
> > > ?????? 0.12 ?? 4%????? -0.0??????? 0.10 ?? 7%
> > > perf-profile.children.cycles-pp.common_file_perm
> > > ?????? 0.09 ?? 4%????? -0.0??????? 0.07 ?? 6%
> > > perf-profile.children.cycles-pp.__fget_light
> > > ?????? 0.09 ?? 4%????? -0.0??????? 0.07 ?? 6%
> > > perf-profile.children.cycles-pp.bvec_alloc
> > > ?????? 0.15 ?? 7%????? -0.0??????? 0.12 ?? 3%
> > > perf-profile.children.cycles-pp.find_get_pages_range_tag
> > > ?????? 0.08 ?? 8%????? -0.0??????? 0.06 ?? 7%
> > > perf-profile.children.cycles-pp.iov_iter_fault_in_readable
> > > ?????? 0.09 ?? 5%????? -0.0??????? 0.06 ? 13%
> > > perf-profile.children.cycles-pp.blk_throtl_bio
> > > ?????? 0.24 ?? 4%????? -0.0??????? 0.22 ?? 3%
> > > perf-profile.children.cycles-pp._cond_resched
> > > ?????? 0.13 ?? 3%????? -0.0??????? 0.11 ?? 4%
> > > perf-profile.children.cycles-pp.release_pages
> > > ?????? 0.10 ?? 4%????? -0.0??????? 0.08 ?? 6%
> > > perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
> > > ?????? 0.08 ? 10%????? -0.0??????? 0.06
> > > perf-profile.children.cycles-pp.unlink
> > > ?????? 0.08 ? 10%????? -0.0??????? 0.06 ? 11%
> > > perf-profile.children.cycles-pp.__sb_start_write
> > > ?????? 0.08 ?? 5%????? -0.0??????? 0.06 ?? 7%
> > > perf-profile.children.cycles-pp.generic_write_checks
> > > ?????? 0.08 ?? 8%????? -0.0??????? 0.06
> > > perf-profile.children.cycles-pp.do_unlinkat
> > > ?????? 0.07 ? 10%????? -0.0??????? 0.05
> > > perf-profile.children.cycles-pp.xfs_break_layouts
> > > ?????? 0.18 ?? 2%????? -0.0??????? 0.16 ?? 2%
> > > perf-profile.children.cycles-pp.__list_add_valid
> > > ?????? 0.12 ?? 3%????? -0.0??????? 0.10 ?? 8%
> > > perf-profile.children.cycles-pp.__might_sleep
> > > ?????? 0.14 ?? 3%????? -0.0??????? 0.12 ?? 7%
> > > perf-profile.children.cycles-pp.xas_find_marked
> > > ?????? 0.07 ? 10%????? -0.0??????? 0.05 ?? 8%
> > > perf-profile.children.cycles-pp.bio_associate_blkg_from_css
> > > ?????? 0.08 ?? 5%????? -0.0??????? 0.06 ? 11%
> > > perf-profile.children.cycles-pp.bio_free
> > > ?????? 0.08 ?? 6%????? -0.0??????? 0.06
> > > perf-profile.children.cycles-pp.xfs_file_llseek
> > > ?????? 0.08 ?? 5%????? -0.0??????? 0.06 ?? 6%
> > > perf-profile.children.cycles-pp.xfs_file_write_iter
> > > ?????? 0.16 ?? 2%????? +0.0??????? 0.17 ?? 2%
> > > perf-profile.children.cycles-pp.__switch_to_asm
> > > ?????? 0.05??????????? +0.0??????? 0.07 ?? 7%
> > > perf-profile.children.cycles-pp.cpumask_next_wrap
> > > ?????? 0.10 ?? 4%????? +0.0??????? 0.12 ?? 3%
> > > perf-profile.children.cycles-pp._find_next_bit
> > > ?????? 0.07 ?? 7%????? +0.0??????? 0.09 ?? 5%
> > > perf-profile.children.cycles-pp.xlog_state_done_syncing
> > > ?????? 0.16 ?? 2%????? +0.0??????? 0.19 ?? 2%
> > > perf-profile.children.cycles-pp.check_preempt_curr
> > > ?????? 0.19 ?? 4%????? +0.0??????? 0.22
> > > perf-profile.children.cycles-pp.ttwu_do_wakeup
> > > ?????? 0.56 ?? 3%????? +0.0??????? 0.60
> > > perf-profile.children.cycles-pp.dequeue_entity
> > > ?????? 0.20 ?? 6%????? +0.0??????? 0.23 ?? 4%
> > > perf-profile.children.cycles-pp.xfs_trans_committed_bulk
> > > ?????? 0.07 ?? 7%????? +0.0??????? 0.10 ?? 7%
> > > perf-profile.children.cycles-pp.check_preempt_wakeup
> > > ?????? 0.82 ?? 2%????? +0.0??????? 0.86
> > > perf-profile.children.cycles-pp.update_load_avg
> > > ?????? 0.31 ?? 3%????? +0.0??????? 0.35
> > > perf-profile.children.cycles-pp.__unwind_start
> > > ?????? 0.08 ?? 8%????? +0.0??????? 0.12 ?? 3%
> > > perf-profile.children.cycles-pp.destroy_inode
> > > ?????? 0.08 ? 10%????? +0.0??????? 0.12 ?? 3%
> > > perf-profile.children.cycles-pp.xfs_fs_destroy_inode
> > > ?????? 0.08 ? 10%????? +0.0??????? 0.12 ?? 3%
> > > perf-profile.children.cycles-pp.xfs_inactive
> > > ?????? 0.08 ? 15%????? +0.0??????? 0.12 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_bmap_validate_extent
> > > ?????? 3.96??????????? +0.0??????? 4.01
> > > perf-profile.children.cycles-pp.stack_trace_save_tsk
> > > ?????? 0.28??????????? +0.0??????? 0.33
> > > perf-profile.children.cycles-pp.set_next_entity
> > > ?????? 0.00??????????? +0.1??????? 0.05
> > > perf-profile.children.cycles-pp.xfs_trans_roll
> > > ?????? 0.00??????????? +0.1??????? 0.05 ?? 8%
> > > perf-profile.children.cycles-pp.xfs_btree_delrec
> > > ?????? 0.00??????????? +0.1??????? 0.05 ?? 8%
> > > perf-profile.children.cycles-pp.xfs_btree_insert
> > > ?????? 0.16 ?? 4%????? +0.1??????? 0.21 ?? 5%
> > > perf-profile.children.cycles-pp.memcpy_erms
> > > ?????? 0.04 ? 57%????? +0.1??????? 0.10 ?? 4%
> > > perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
> > > ?????? 0.00??????????? +0.1??????? 0.06 ? 11%
> > > perf-profile.children.cycles-pp.xfs_buf_item_format_segment
> > > ?????? 0.00??????????? +0.1??????? 0.06 ? 11%
> > > perf-profile.children.cycles-pp.xfs_btree_delete
> > > ?????? 0.00??????????? +0.1??????? 0.06
> > > perf-profile.children.cycles-pp.xfs_btree_update
> > > ?????? 0.00??????????? +0.1??????? 0.07 ?? 7%
> > > perf-profile.children.cycles-pp.xfs_cil_prepare_item
> > > ?????? 0.00??????????? +0.1??????? 0.07 ?? 6%
> > > perf-profile.children.cycles-pp.xfs_bmbt_update
> > > ?????? 0.00??????????? +0.1??????? 0.07 ? 10%
> > > perf-profile.children.cycles-pp.xfs_buf_item_format
> > > ?????? 0.56??????????? +0.1??????? 0.64 ?? 2%
> > > perf-profile.children.cycles-pp.pick_next_task_fair
> > > ?????? 0.03 ?100%????? +0.1??????? 0.10 ?? 4%
> > > perf-profile.children.cycles-pp.xfs_inactive_truncate
> > > ?????? 0.08 ?? 5%????? +0.1??????? 0.16 ? 11%
> > > perf-profile.children.cycles-pp.xfs_iext_lookup_extent
> > > ?????? 0.00??????????? +0.1??????? 0.08 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_defer_finish
> > > ?????? 0.00??????????? +0.1??????? 0.08 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
> > > ?????? 0.00??????????? +0.1??????? 0.08 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_verify_agbno
> > > ?????? 0.00??????????? +0.1??????? 0.09 ?? 9%
> > > perf-profile.children.cycles-pp.xfs_defer_finish_noroll
> > > ?????? 0.60 ?? 3%????? +0.1??????? 0.70
> > > perf-profile.children.cycles-pp.schedule_idle
> > > ?????? 0.39 ?? 2%????? +0.1??????? 0.49
> > > perf-profile.children.cycles-pp.xlog_state_release_iclog
> > > ?????? 0.00??????????? +0.1??????? 0.10 ?? 9%
> > > perf-profile.children.cycles-pp.xfs_buf_find
> > > ?????? 0.93 ?? 3%????? +0.1??????? 1.03 ?? 2%
> > > perf-profile.children.cycles-pp.xlog_write
> > > ?????? 2.58??????????? +0.1??????? 2.69
> > > perf-profile.children.cycles-pp.schedule
> > > ?????? 1.52??????????? +0.1??????? 1.63
> > > perf-profile.children.cycles-pp.__wake_up_common_lock
> > > ?????? 0.00??????????? +0.1??????? 0.11 ?? 7%
> > > perf-profile.children.cycles-pp.xfs_buf_get_map
> > > ?????? 4.62??????????? +0.1??????? 4.73
> > > perf-profile.children.cycles-pp.__account_scheduler_latency
> > > ?????? 0.00??????????? +0.1??????? 0.12 ? 10%
> > > perf-profile.children.cycles-pp.xfs_buf_read_map
> > > ?????? 1.83??????????? +0.2??????? 1.99
> > > perf-profile.children.cycles-pp.xlog_cil_push_work
> > > ?????? 5.94??????????? +0.2??????? 6.12
> > > perf-profile.children.cycles-pp.ttwu_do_activate
> > > ?????? 0.00??????????? +0.2??????? 0.17 ?? 6%
> > > perf-profile.children.cycles-pp.xfs_btree_read_buf_block
> > > ?????? 5.93??????????? +0.2??????? 6.11
> > > perf-profile.children.cycles-pp.activate_task
> > > ?????? 5.88??????????? +0.2??????? 6.06
> > > perf-profile.children.cycles-pp.enqueue_task_fair
> > > ?????? 0.00??????????? +0.2??????? 0.18 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_trans_read_buf_map
> > > ?????? 5.41??????????? +0.2??????? 5.59
> > > perf-profile.children.cycles-pp.enqueue_entity
> > > ?????? 3.14??????????? +0.2??????? 3.33
> > > perf-profile.children.cycles-pp.__sched_text_start
> > > ?????? 0.00??????????? +0.2??????? 0.21 ?? 6%
> > > perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
> > > ?????? 0.00??????????? +0.2??????? 0.23 ? 57%
> > > perf-profile.children.cycles-pp.xlog_grant_head_wait
> > > ?????? 0.00??????????? +0.2??????? 0.24 ?? 6%
> > > perf-profile.children.cycles-pp.io_schedule
> > > ?????? 0.11 ?? 3%????? +0.3??????? 0.38 ?? 4%
> > > perf-profile.children.cycles-pp.__filemap_fdatawait_range
> > > ?????? 0.00??????????? +0.3??????? 0.27 ?? 8%
> > > perf-profile.children.cycles-pp.wait_on_page_bit
> > > ?????? 0.00??????????? +0.3??????? 0.34 ?? 5%
> > > perf-profile.children.cycles-pp.xfs_btree_lookup
> > > ?????? 0.08 ? 14%????? +0.4??????? 0.43
> > > perf-profile.children.cycles-pp.__queue_work
> > > ?????? 0.08 ?? 8%????? +0.4??????? 0.44
> > > perf-profile.children.cycles-pp.queue_work_on
> > > ?????? 0.00??????????? +0.4??????? 0.41
> > > perf-profile.children.cycles-pp.xfs_end_bio
> > > ?????? 3.51??????????? +0.4??????? 3.94 ?? 9%
> > > perf-profile.children.cycles-pp.__xfs_trans_commit
> > > ?????? 0.20 ?? 9%????? +0.5??????? 0.72 ? 54%
> > > perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
> > > ?????? 0.06 ? 11%????? +0.5??????? 0.58 ? 67%
> > > perf-profile.children.cycles-pp.xfs_log_space_wake
> > > ?????? 0.45 ?? 5%????? +0.6??????? 1.04 ? 36%
> > > perf-profile.children.cycles-pp.xfs_log_reserve
> > > ?????? 0.65 ?? 3%????? +0.6??????? 1.25 ? 29%
> > > perf-profile.children.cycles-pp.xfs_trans_alloc
> > > ?????? 0.48 ?? 5%????? +0.6??????? 1.09 ? 34%
> > > perf-profile.children.cycles-pp.xfs_trans_reserve
> > > ?????? 0.04 ? 57%????? +0.6??????? 0.65 ? 60%
> > > perf-profile.children.cycles-pp.xlog_grant_head_check
> > > ?????? 4.83??????????? +0.7??????? 5.52
> > > perf-profile.children.cycles-pp.try_to_wake_up
> > > ?????? 0.00??????????? +0.7??????? 0.70 ?? 2%
> > > perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
> > > ?????? 0.00??????????? +0.8??????? 0.75 ?? 4%
> > > perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
> > > ?????? 0.04 ? 57%????? +0.8??????? 0.80
> > > perf-profile.children.cycles-pp.autoremove_wake_function
> > > ?????? 0.05 ?? 8%????? +0.8??????? 0.85
> > > perf-profile.children.cycles-pp.wake_up_page_bit
> > > ?????? 1.50??????????? +0.9??????? 2.36
> > > perf-profile.children.cycles-pp.__wake_up_common
> > > ?????? 0.00??????????? +0.9??????? 0.89 ?? 2%
> > > perf-profile.children.cycles-pp.xfs_bmapi_write
> > > ????? 24.29??????????? +0.9?????? 25.18
> > > perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> > > ?????? 0.07 ?? 5%????? +1.0??????? 1.05
> > > perf-profile.children.cycles-pp.iomap_finish_ioends
> > > ????? 35.68??????????? +1.0?????? 36.69
> > > perf-profile.children.cycles-pp.xlog_cil_force_lsn
> > > ?????? 9.53??????????? +1.2?????? 10.76
> > > perf-profile.children.cycles-pp.intel_idle
> > > ?????? 9.81??????????? +1.3?????? 11.10
> > > perf-profile.children.cycles-pp.cpuidle_enter_state
> > > ?????? 9.82??????????? +1.3?????? 11.11
> > > perf-profile.children.cycles-pp.cpuidle_enter
> > > ????? 12.25??????????? +1.3?????? 13.60
> > > perf-profile.children.cycles-pp.start_secondary
> > > ????? 12.49??????????? +1.4?????? 13.89
> > > perf-profile.children.cycles-pp.secondary_startup_64
> > > ????? 12.49??????????? +1.4?????? 13.89
> > > perf-profile.children.cycles-pp.cpu_startup_entry
> > > ????? 12.48??????????? +1.4?????? 13.88
> > > perf-profile.children.cycles-pp.do_idle
> > > ????? 14.78??????????? +1.4?????? 16.20 ?? 2%
> > > perf-profile.children.cycles-pp.xlog_wait_on_iclog
> > > ????? 58.91??????????? +1.7?????? 60.57
> > > perf-profile.children.cycles-pp.xfs_log_force_lsn
> > > ?????? 0.00??????????? +2.2??????? 2.24 ? 13%
> > > perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
> > > ?????? 7.05??????????? +2.8??????? 9.81 ?? 2%
> > > perf-profile.children.cycles-pp.process_one_work
> > > ?????? 7.54??????????? +2.9?????? 10.45 ?? 2%
> > > perf-profile.children.cycles-pp.worker_thread
> > > ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
> > > perf-profile.children.cycles-pp.ret_from_fork
> > > ?????? 7.55??????????? +2.9?????? 10.46 ?? 2%
> > > perf-profile.children.cycles-pp.kthread
> > > ?????? 0.10??????????? +3.2??????? 3.30 ?? 9%
> > > perf-profile.children.cycles-pp.xfs_end_ioend
> > > ?????? 0.10 ?? 4%????? +3.2??????? 3.32 ?? 8%
> > > perf-profile.children.cycles-pp.xfs_end_io
> > > ????? 19.57??????????? +4.0?????? 23.62
> > > perf-profile.children.cycles-pp.remove_wait_queue
> > > ?????? 1.20??????????? -0.3??????? 0.90 ?? 2%
> > > perf-profile.self.cycles-pp.brd_do_bvec
> > > ?????? 0.55 ?? 3%????? -0.2??????? 0.39 ?? 4%
> > > perf-profile.self.cycles-pp.do_syscall_64
> > > ?????? 0.31??????????? -0.1??????? 0.22 ?? 5%
> > > perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
> > > ?????? 0.31 ?? 3%????? -0.1??????? 0.22
> > > perf-profile.self.cycles-pp.syscall_return_via_sysret
> > > ?????? 0.54 ?? 3%????? -0.1??????? 0.47 ?? 3%
> > > perf-profile.self.cycles-pp.osq_lock
> > > ?????? 0.23 ?? 3%????? -0.1??????? 0.16 ?? 5%
> > > perf-profile.self.cycles-pp.entry_SYSCALL_64
> > > ?????? 0.10 ?? 4%????? -0.1??????? 0.04 ? 57%
> > > perf-profile.self.cycles-pp.__mod_lruvec_state
> > > ?????? 0.18 ?? 4%????? -0.1??????? 0.12 ?? 5%
> > > perf-profile.self.cycles-pp.iomap_set_range_uptodate
> > > ?????? 0.60 ?? 4%????? -0.1??????? 0.54 ?? 2%
> > > perf-profile.self.cycles-pp._raw_spin_lock
> > > ?????? 0.77??????????? -0.0??????? 0.72
> > > perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> > > ?????? 0.14??????????? -0.0??????? 0.10 ?? 8%
> > > perf-profile.self.cycles-pp.find_get_entry
> > > ?????? 0.14 ?? 3%????? -0.0??????? 0.11 ?? 4%
> > > perf-profile.self.cycles-pp.xas_load
> > > ?????? 0.17 ?? 4%????? -0.0??????? 0.14 ?? 3%
> > > perf-profile.self.cycles-pp.write
> > > ?????? 0.30 ?? 2%????? -0.0??????? 0.27 ?? 6%
> > > perf-profile.self.cycles-pp._raw_spin_lock_irq
> > > ?????? 0.18 ?? 5%????? -0.0??????? 0.15 ?? 7%
> > > perf-profile.self.cycles-pp.__flush_work
> > > ?????? 0.08 ?? 5%????? -0.0??????? 0.05
> > > perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
> > > ?????? 0.12 ?? 3%????? -0.0??????? 0.09 ?? 4%
> > > perf-profile.self.cycles-pp.brd_lookup_page
> > > ?????? 0.12 ?? 7%????? -0.0??????? 0.09 ?? 9%
> > > perf-profile.self.cycles-pp.vfs_write
> > > ?????? 0.18 ?? 4%????? -0.0??????? 0.15 ?? 7%
> > > perf-profile.self.cycles-pp.insert_work
> > > ?????? 0.13??????????? -0.0??????? 0.10 ?? 8%
> > > perf-profile.self.cycles-pp.memset_erms
> > > ?????? 0.11 ?? 4%????? -0.0??????? 0.08
> > > perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
> > > ?????? 0.11 ?? 6%????? -0.0??????? 0.08 ?? 5%
> > > perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
> > > ?????? 0.12 ?? 3%????? -0.0??????? 0.09 ?? 4%
> > > perf-profile.self.cycles-pp.iomap_writepage_map
> > > ?????? 0.08 ?? 8%????? -0.0??????? 0.06 ? 15%
> > > perf-profile.self.cycles-pp.bio_alloc_bioset
> > > ?????? 0.06 ?? 6%????? -0.0??????? 0.04 ? 57%
> > > perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
> > > ?????? 0.09 ?? 5%????? -0.0??????? 0.06 ?? 6%
> > > perf-profile.self.cycles-pp.__fget_light
> > > ?????? 0.06??????????? -0.0??????? 0.04 ? 57%
> > > perf-profile.self.cycles-pp.xfs_map_blocks
> > > ?????? 0.18 ?? 2%????? -0.0??????? 0.15 ?? 3%
> > > perf-profile.self.cycles-pp.__list_add_valid
> > > ?????? 0.08 ?? 6%????? -0.0??????? 0.06 ?? 9%
> > > perf-profile.self.cycles-pp.xfs_ilock
> > > ?????? 0.08 ?? 6%????? -0.0??????? 0.06 ?? 9%
> > > perf-profile.self.cycles-pp.iov_iter_fault_in_readable
> > > ?????? 0.11 ?? 4%????? -0.0??????? 0.09 ?? 4%
> > > perf-profile.self.cycles-pp.switch_fpu_return
> > > ?????? 0.07??????????? -0.0??????? 0.05
> > > perf-profile.self.cycles-pp.iomap_write_actor
> > > ?????? 0.23 ?? 2%????? -0.0??????? 0.21 ?? 5%
> > > perf-profile.self.cycles-pp.__radix_tree_lookup
> > > ?????? 0.18 ?? 4%????? -0.0??????? 0.16 ?? 6%
> > > perf-profile.self.cycles-pp.xlog_cil_force_lsn
> > > ?????? 0.20 ?? 2%????? -0.0??????? 0.18 ?? 4%
> > > perf-profile.self.cycles-pp.kmem_cache_alloc
> > > ?????? 0.10 ?? 5%????? -0.0??????? 0.08 ? 11%
> > > perf-profile.self.cycles-pp.xfs_file_fsync
> > > ?????? 0.07 ?? 5%????? -0.0??????? 0.05 ?? 9%
> > > perf-profile.self.cycles-pp.clear_page_dirty_for_io
> > > ?????? 0.07 ?? 6%????? -0.0??????? 0.05
> > > perf-profile.self.cycles-pp.__test_set_page_writeback
> > > ?????? 0.11??????????? -0.0??????? 0.09 ?? 8%
> > > perf-profile.self.cycles-pp.common_file_perm
> > > ?????? 0.23??????????? -0.0??????? 0.21 ?? 4%
> > > perf-profile.self.cycles-pp.update_rq_clock
> > > ?????? 0.08 ?? 6%????? -0.0??????? 0.06
> > > perf-profile.self.cycles-pp.xfs_file_write_iter
> > > ?????? 0.10 ?? 5%????? -0.0??????? 0.08 ?? 8%
> > > perf-profile.self.cycles-pp.up_read
> > > ?????? 0.08 ?? 5%????? -0.0??????? 0.07
> > > perf-profile.self.cycles-pp.write_cache_pages
> > > ?????? 0.10??????????? -0.0??????? 0.09
> > > perf-profile.self.cycles-pp.reweight_entity
> > > ?????? 0.05 ?? 8%????? +0.0??????? 0.07 ?? 7%
> > > perf-profile.self.cycles-pp.dequeue_entity
> > > ?????? 0.16 ?? 4%????? +0.0??????? 0.17 ?? 2%
> > > perf-profile.self.cycles-pp.__switch_to_asm
> > > ?????? 0.11 ?? 4%????? +0.0??????? 0.12 ?? 3%
> > > perf-profile.self.cycles-pp.set_next_entity
> > > ?????? 0.15 ?? 3%????? +0.0??????? 0.16 ?? 2%
> > > perf-profile.self.cycles-pp.orc_find
> > > ?????? 0.09 ?? 4%????? +0.0??????? 0.11 ?? 6%
> > > perf-profile.self.cycles-pp.enqueue_entity
> > > ?????? 0.10 ?? 4%????? +0.0??????? 0.12 ?? 3%
> > > perf-profile.self.cycles-pp._find_next_bit
> > > ?????? 0.01 ?173%????? +0.0??????? 0.06 ?? 7%
> > > perf-profile.self.cycles-pp.check_preempt_wakeup
> > > ?????? 0.01 ?173%????? +0.0??????? 0.06 ? 11%
> > > perf-profile.self.cycles-pp.xfs_trans_committed_bulk
> > > ?????? 0.45 ?? 2%????? +0.1??????? 0.51 ?? 2%
> > > perf-profile.self.cycles-pp.__sched_text_start
> > > ?????? 0.15 ?? 6%????? +0.1??????? 0.21 ?? 3%
> > > perf-profile.self.cycles-pp.memcpy_erms
> > > ?????? 0.00??????????? +0.1??????? 0.06 ?? 6%
> > > perf-profile.self.cycles-pp.xfs_verify_agbno
> > > ?????? 0.08 ?? 8%????? +0.1??????? 0.16 ?? 9%
> > > perf-profile.self.cycles-pp.xfs_iext_lookup_extent
> > > ?????? 9.53??????????? +1.2?????? 10.76
> > > perf-profile.self.cycles-pp.intel_idle
> > >
> > >
> > > ?????????????????????????????????? aim7.jobs-per-min
> > > ?? 45000
> > > +-------------------------------------------------------------------+
> > > ???????? |??????? +
> > > +?????????????????????????????????????????????????????? |
> > > ?? 40000 |.+.+??? :?? + .+.?? +.
> > > .+.???? |
> > > ?? 35000 |-+ :??? :??? +?? +.+
> > > +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+?? ++.+.|
> > > ???????? |?? :
> > > :????????????????????????????????????????????????????????? |
> > > ?? 30000 |-+ :
> > > :?????????????????????????????????????????????????????????? |
> > > ?? 25000 |-O :O O:O
> > > OO???????????????????????????????????????????????????? |
> > > ???????? |?? :?? :? O??? O O OO O O O
> > > O????????????????????????????????????? |
> > > ?? 20000 |-+? :
> > > :?????????????????????????????????????????????????????????? |
> > > ?? 15000 |-+? :
> > > :?????????????????????????????????????????????????????????? |
> > > ???????? |??? :
> > > :?????????????????????????????????????????????????????????? |
> > > ?? 10000 |-+? :
> > > :??????????????????????????????????????????????????????????? |
> > > ??? 5000 |-+? :
> > > :??????????????????????????????????????????????????????????? |
> > > ???????? |??? :
> > > :??????????????????????????????????????????????????????????? |
> > > ?????? 0
> > > +-------------------------------------------------------------------+
> > > ??????????????????????????????? aim7.time.system_time
> > > ?? 5000
> > > +--------------------------------------------------------------------+
> > > ?? 4500 |-+??????? O?? OO?? O??? O O
> > > O?????????????????????????????????????? |
> > > ??????? | O OO O O?? O??? O?? O
> > > O??????????????????????????????????????????? |
> > > ?? 4000
> > > |-+?????????????????????????????????????????????????????????????????
> > > |
> > > ?? 3500
> > > |-+?????????????????????????????????????????????????????????????????
> > > |
> > > ??????? |???????????? .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+
> > > .+.+.+. .++.+.|
> > > ?? 3000 |.+.+?????? .+? +???? +?????????????????????????????? +
> > > +????? |
> > > ?? 2500 |-+ :
> > > +.+???????????????????????????????????????????????????????? |
> > > ?? 2000 |-+ :
> > > :?????????????????????????????????????????????????????????? |
> > > ??????? |?? :
> > > :??????????????????????????????????????????????????????????? |
> > > ?? 1500 |-+? :
> > > :??????????????????????????????????????????????????????????? |
> > > ?? 1000 |-+? :
> > > :??????????????????????????????????????????????????????????? |
> > > ??????? |??? :
> > > :??????????????????????????????????????????????????????????? |
> > > ??? 500 |-+? :
> > > :???????????????????????????????????????????????????????????? |
> > > ????? 0
> > > +--------------------------------------------------------------------+
> > > ?????????????????????????????? aim7.time.elapsed_time
> > > ?? 160
> > > +---------------------------------------------------------------------+
> > > ?????? |?? O OO O O O O OO O O O
> > > O???????????????????????????????????????? |
> > > ?? 140
> > > |-O??????????????????????????????????????????????????????????????????
> > > |
> > > ?? 120
> > > |-+??????????????????????????????????????????????????????????????????
> > > |
> > > |???????????????????????????????????????????????????????????????????? |
> > > ?? 100 |-+?????????? .+. +.+.
> > > .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
> > > ?????? |.+.+??? +. .+?? +??? +
> > > +????? |
> > > ??? 80 |-+ :??? :
> > > +????????????????????????????????????????????????????????? |
> > > ?????? |??? :
> > > :??????????????????????????????????????????????????????????? |
> > > ??? 60 |-+? :
> > > :???????????????????????????????????????????????????????????? |
> > > ??? 40 |-+? :
> > > :???????????????????????????????????????????????????????????? |
> > > ?????? |??? :
> > > :???????????????????????????????????????????????????????????? |
> > > ??? 20 |-+?? :
> > > :???????????????????????????????????????????????????????????? |
> > > ?????? |
> > > ::????????????????????????????????????????????????????????????? |
> > > ???? 0
> > > +---------------------------------------------------------------------+
> > > ???????????????????????????? aim7.time.elapsed_time.max
> > > ?? 160
> > > +---------------------------------------------------------------------+
> > > ?????? |?? O OO O O O O OO O O O
> > > O???????????????????????????????????????? |
> > > ?? 140
> > > |-O??????????????????????????????????????????????????????????????????
> > > |
> > > ?? 120
> > > |-+??????????????????????????????????????????????????????????????????
> > > |
> > > |???????????????????????????????????????????????????????????????????? |
> > > ?? 100 |-+?????????? .+. +.+.
> > > .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
> > > ?????? |.+.+??? +. .+?? +??? +
> > > +????? |
> > > ??? 80 |-+ :??? :
> > > +????????????????????????????????????????????????????????? |
> > > ?????? |??? :
> > > :??????????????????????????????????????????????????????????? |
> > > ??? 60 |-+? :
> > > :???????????????????????????????????????????????????????????? |
> > > ??? 40 |-+? :
> > > :???????????????????????????????????????????????????????????? |
> > > ?????? |??? :
> > > :???????????????????????????????????????????????????????????? |
> > > ??? 20 |-+?? :
> > > :???????????????????????????????????????????????????????????? |
> > > ?????? |
> > > ::????????????????????????????????????????????????????????????? |
> > > ???? 0
> > > +---------------------------------------------------------------------+
> > > ???????????????????????? aim7.time.voluntary_context_switches
> > > ?? 8e+07
> > > +-------------------------------------------------------------------+
> > > ???????? | O????????? O? O O OO O
> > > O??????????????????????????????????????? |
> > > ?? 7e+07
> > > |-+????????????????????????????????????????????????????????????????
> > > |
> > > ?? 6e+07
> > > |-+????????????????????????????????????????????????????????????????
> > > |
> > > |?????????????????????????????????????????????????????????????????? |
> > > ?? 5e+07 |-+?????????? +.+.+.++.+.+.+.++.+.+. .+?? .+.+.??? .+.
> > > .+.?? +.+.|
> > > ???????? |.+.+?????? .+????????????????????? +? +.+???? ++.+?? ++.+
> > > +.+??? |
> > > ?? 4e+07 |-+ :
> > > +.+??????????????????????????????????????????????????????? |
> > > ???????? |?? :
> > > :????????????????????????????????????????????????????????? |
> > > ?? 3e+07 |-+ :
> > > :?????????????????????????????????????????????????????????? |
> > > ?? 2e+07 |-+? :
> > > :?????????????????????????????????????????????????????????? |
> > > ???????? |??? :
> > > :?????????????????????????????????????????????????????????? |
> > > ?? 1e+07 |-+? :
> > > :?????????????????????????????????????????????????????????? |
> > > ???????? |??? :
> > > :??????????????????????????????????????????????????????????? |
> > > ?????? 0
> > > +-------------------------------------------------------------------+
> > > [*] bisect-good sample
> > > [O] bisect-bad? sample
> > >
> > >
> > >
> > > Disclaimer:
> > > Results have been estimated based on internal Intel analysis and are
> > > provided
> > > for informational purposes only. Any difference in system hardware
> > > or software
> > > design or configuration may affect actual performance.
> > >
> > >
> > > Thanks,
> > > Rong Chen
> > >
> > >
> > > _______________________________________________
> > > LKP mailing list -- [email protected]
> > > To unsubscribe send an email to [email protected]
> > >
> >
>
> --
> Zhengjun Xing

2020-07-22 07:22:35

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

On 7/7/2020 2:30 AM, Darrick J. Wong wrote:
> On Wed, Jul 01, 2020 at 03:49:52PM +0800, Xing Zhengjun wrote:
>>
>>
>> On 6/10/2020 11:07 AM, Xing Zhengjun wrote:
>>> Hi Darrick,
>>>
>>>    Do you have time to take a look at this? Thanks.
>>>
>>
>>
>> Ping...
>
> Yes, that decrease is the expected end result of making the write path
> take a longer route to avoid a file corruption vector.
>
> --D
>

Thanks for the explanation, We will stop tracking it.

>>
>>
>>> On 6/6/2020 11:48 PM, kernel test robot wrote:
>>>> Greeting,
>>>>
>>>> FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:
>>>>
>>>>
>>>> commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes
>>>> to delalloc regions to unwritten")
>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>>>
>>>> in testcase: aim7
>>>> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @
>>>> 2.70GHz with 64G memory
>>>> with following parameters:
>>>>
>>>>     disk: 1BRD_48G
>>>>     fs: xfs
>>>>     test: sync_disk_rw
>>>>     load: 600
>>>>     cpufreq_governor: performance
>>>>     ucode: 0x42e
>>>>
>>>> test-description: AIM7 is a traditional UNIX system level benchmark
>>>> suite which is used to test and measure the performance of multiuser
>>>> system.
>>>> test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
>>>>
>>>>
>>>>
>>>> If you fix the issue, kindly add following tag
>>>> Reported-by: kernel test robot <[email protected]>
>>>>
>>>>
>>>> Details are as below:
>>>> -------------------------------------------------------------------------------------------------->
>>>>
>>>>
>>>>
>>>> To reproduce:
>>>>
>>>>          git clone https://github.com/intel/lkp-tests.git
>>>>          cd lkp-tests
>>>>          bin/lkp install job.yaml # job file is attached in this email
>>>>          bin/lkp run     job.yaml
>>>>
>>>> =========================================================================================
>>>>
>>>> compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
>>>>
>>>> gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e
>>>>
>>>>
>>>> commit:
>>>>    590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
>>>>    a5949d3fae ("xfs: force writes to delalloc regions to unwritten")
>>>>
>>>> 590b16516ef38e2e a5949d3faedf492fa7863b914da
>>>> ---------------- ---------------------------
>>>>         fail:runs %reproduction    fail:runs
>>>>             |             |             |
>>>>             :4           50%           2:4
>>>> dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
>>>>           %stddev     %change         %stddev
>>>>               \          |                \
>>>>       35272           -33.6%      23430        aim7.jobs-per-min
>>>>      102.13           +50.5%     153.75        aim7.time.elapsed_time
>>>>      102.13           +50.5%     153.75        aim7.time.elapsed_time.max
>>>>     1388038           +40.2%    1945838
>>>> aim7.time.involuntary_context_switches
>>>>       43420 ± 2%     +13.4%      49255 ± 2%
>>>> aim7.time.minor_page_faults
>>>>        3123           +44.2%       4504 ± 2% aim7.time.system_time
>>>>       59.31            +6.5%      63.18        aim7.time.user_time
>>>>    48595108           +58.6%   77064959
>>>> aim7.time.voluntary_context_switches
>>>>        1.44           -28.8%       1.02        iostat.cpu.user
>>>>        0.07 ± 6%      +0.4        0.44 ± 7% mpstat.cpu.all.iowait%
>>>>        1.44            -0.4        1.02        mpstat.cpu.all.usr%
>>>>        8632 ± 50%     +75.6%      15156 ± 34%
>>>> numa-meminfo.node0.KernelStack
>>>>        6583 ±136%    +106.0%      13562 ± 82%
>>>> numa-meminfo.node0.PageTables
>>>>       63325 ± 11%     +14.3%      72352 ± 12%
>>>> numa-meminfo.node0.SUnreclaim
>>>>        8647 ± 50%     +75.3%      15156 ± 34%
>>>> numa-vmstat.node0.nr_kernel_stack
>>>>        1656 ±136%    +104.6%       3389 ± 82%
>>>> numa-vmstat.node0.nr_page_table_pages
>>>>       15831 ± 11%     +14.3%      18087 ± 12%
>>>> numa-vmstat.node0.nr_slab_unreclaimable
>>>>       93640 ± 3%     +41.2%     132211 ± 2% meminfo.AnonHugePages
>>>>       21641           +39.9%      30271 ± 4% meminfo.KernelStack
>>>>      129269           +12.3%     145114        meminfo.SUnreclaim
>>>>       28000           -31.2%      19275        meminfo.max_used_kB
>>>>     1269307           -26.9%     927657        vmstat.io.bo
>>>>      149.75 ± 3%     -17.4%     123.75 ± 4% vmstat.procs.r
>>>>      718992           +13.3%     814567        vmstat.system.cs
>>>>      231397            -9.3%     209881 ± 2% vmstat.system.in
>>>> 6.774e+08           +70.0% 1.152e+09        cpuidle.C1.time
>>>>    18203372           +60.4%   29198744        cpuidle.C1.usage
>>>> 2.569e+08 ± 18%     +81.8% 4.672e+08 ± 5% cpuidle.C1E.time
>>>>     2691402 ± 13%     +98.7%    5346901 ± 3% cpuidle.C1E.usage
>>>>      990350           +95.0%    1931226 ± 2% cpuidle.POLL.time
>>>>      520061           +97.7%    1028004 ± 2% cpuidle.POLL.usage
>>>>       77231            +1.8%      78602        proc-vmstat.nr_active_anon
>>>>       19868            +3.8%      20615        proc-vmstat.nr_dirty
>>>>      381302            +1.0%     384969        proc-vmstat.nr_file_pages
>>>>        4388            -2.7%       4270
>>>> proc-vmstat.nr_inactive_anon
>>>>       69865            +4.7%      73155
>>>> proc-vmstat.nr_inactive_file
>>>>       21615           +40.0%      30251 ± 4%
>>>> proc-vmstat.nr_kernel_stack
>>>>        7363            -3.2%       7127        proc-vmstat.nr_mapped
>>>>       12595 ± 3%      +5.2%      13255 ± 4% proc-vmstat.nr_shmem
>>>>       19619            +3.2%      20247
>>>> proc-vmstat.nr_slab_reclaimable
>>>>       32316           +12.3%      36280
>>>> proc-vmstat.nr_slab_unreclaimable
>>>>       77231            +1.8%      78602
>>>> proc-vmstat.nr_zone_active_anon
>>>>        4388            -2.7%       4270
>>>> proc-vmstat.nr_zone_inactive_anon
>>>>       69865            +4.7%      73155
>>>> proc-vmstat.nr_zone_inactive_file
>>>>       19165            +3.4%      19824
>>>> proc-vmstat.nr_zone_write_pending
>>>>        9845 ± 27%     +68.0%      16539 ± 37%
>>>> proc-vmstat.numa_hint_faults_local
>>>>     8195135            +2.5%    8402537        proc-vmstat.numa_hit
>>>>     8179414            +2.5%    8386209        proc-vmstat.numa_local
>>>>       15720            +3.9%      16328        proc-vmstat.numa_other
>>>>     8281098            +2.4%    8477157        proc-vmstat.pgalloc_normal
>>>>      342925 ± 2%     +39.8%     479279        proc-vmstat.pgfault
>>>>     8132789            +2.0%    8298189        proc-vmstat.pgfree
>>>> 1.327e+08            +9.7% 1.455e+08        proc-vmstat.pgpgout
>>>>      394.00          +151.7%     991.75 ± 5%
>>>> slabinfo.biovec-max.active_objs
>>>>      394.00          +152.1%     993.25 ± 5%
>>>> slabinfo.biovec-max.num_objs
>>>>       21028 ± 6%      -7.8%      19389 ± 5% slabinfo.filp.active_objs
>>>>       21520 ± 5%      -7.6%      19886 ± 5% slabinfo.filp.num_objs
>>>>        4955          +234.9%      16594 ± 2%
>>>> slabinfo.kmalloc-256.active_objs
>>>>       77.25          +238.5%     261.50 ± 2%
>>>> slabinfo.kmalloc-256.active_slabs
>>>>        4955          +238.4%      16768 ± 2%
>>>> slabinfo.kmalloc-256.num_objs
>>>>       77.25          +238.5%     261.50 ± 2%
>>>> slabinfo.kmalloc-256.num_slabs
>>>>        3368 ± 2%     +22.6%       4128 ± 2%
>>>> slabinfo.kmalloc-2k.active_objs
>>>>        3446 ± 2%     +21.9%       4200 ± 2%
>>>> slabinfo.kmalloc-2k.num_objs
>>>>       11815 ± 8%     +29.0%      15239 ± 3%
>>>> slabinfo.kmalloc-512.active_objs
>>>>       11824 ± 9%     +29.0%      15249 ± 3%
>>>> slabinfo.kmalloc-512.num_objs
>>>>        6849 ± 5%      +9.0%       7463 ± 2%
>>>> slabinfo.kmalloc-96.active_objs
>>>>        6906 ± 5%      +8.9%       7521 ± 2%
>>>> slabinfo.kmalloc-96.num_objs
>>>>      765.00 ± 4%     +25.0%     956.25 ± 5%
>>>> slabinfo.mnt_cache.active_objs
>>>>      765.00 ± 4%     +25.0%     956.25 ± 5%
>>>> slabinfo.mnt_cache.num_objs
>>>>       13866           +12.5%      15594
>>>> slabinfo.proc_inode_cache.active_objs
>>>>       13905           +12.2%      15598
>>>> slabinfo.proc_inode_cache.num_objs
>>>>        1954           +24.4%       2431 ± 3%
>>>> slabinfo.sighand_cache.active_objs
>>>>        1954           +24.4%       2431 ± 3%
>>>> slabinfo.sighand_cache.num_objs
>>>>        2596           +13.7%       2951 ± 2%
>>>> slabinfo.signal_cache.active_objs
>>>>        2596           +13.7%       2951 ± 2%
>>>> slabinfo.signal_cache.num_objs
>>>>        4368 ± 6%     -11.7%       3856 ± 2%
>>>> slabinfo.skbuff_head_cache.num_objs
>>>>      822.25 ± 11%     +21.4%     998.50 ± 3%
>>>> slabinfo.task_group.active_objs
>>>>      822.25 ± 11%     +21.4%     998.50 ± 3%
>>>> slabinfo.task_group.num_objs
>>>>        1612           +32.1%       2129 ± 3%
>>>> slabinfo.task_struct.active_objs
>>>>      540.50           +31.6%     711.25 ± 3%
>>>> slabinfo.task_struct.active_slabs
>>>>        1622           +31.6%       2134 ± 3%
>>>> slabinfo.task_struct.num_objs
>>>>      540.50           +31.6%     711.25 ± 3%
>>>> slabinfo.task_struct.num_slabs
>>>>        6905           +20.0%       8289
>>>> slabinfo.vmap_area.active_objs
>>>>        6933           +20.0%       8322
>>>> slabinfo.vmap_area.num_objs
>>>>      646.25 ± 7%    +195.2%       1907
>>>> slabinfo.xfs_buf.active_objs
>>>>      646.25 ± 7%    +195.2%       1907        slabinfo.xfs_buf.num_objs
>>>>        1.78 ± 2%      -0.3        1.51 ± 2%
>>>> perf-stat.i.branch-miss-rate%
>>>>    64079048            +3.9%   66551536        perf-stat.i.branch-misses
>>>>       17.99            -4.2       13.84
>>>> perf-stat.i.cache-miss-rate%
>>>>    66630506           -16.5%   55647472        perf-stat.i.cache-misses
>>>> 3.899e+08            +9.4% 4.265e+08
>>>> perf-stat.i.cache-references
>>>>      731029           +13.1%     826756
>>>> perf-stat.i.context-switches
>>>>        3.79            -1.7%       3.73        perf-stat.i.cpi
>>>>      113320            +8.3%     122693        perf-stat.i.cpu-migrations
>>>>        1498           +20.4%       1803
>>>> perf-stat.i.cycles-between-cache-misses
>>>>    50450547 ± 4%      -8.0%   46402743 ± 5%
>>>> perf-stat.i.dTLB-load-misses
>>>>        0.20 ± 3%      -0.0        0.16 ± 2%
>>>> perf-stat.i.dTLB-store-miss-rate%
>>>>     4587096 ± 3%     -22.9%    3538422 ± 2%
>>>> perf-stat.i.dTLB-store-misses
>>>> 2.297e+09            -5.6% 2.168e+09        perf-stat.i.dTLB-stores
>>>>       86.68            -2.4       84.25
>>>> perf-stat.i.iTLB-load-miss-rate%
>>>>    19484791            -6.1%   18297125
>>>> perf-stat.i.iTLB-load-misses
>>>>     3060023           +14.4%    3501587 ± 4% perf-stat.i.iTLB-loads
>>>>        0.29            -1.5%       0.28        perf-stat.i.ipc
>>>>        0.21 ± 5%     -20.5%       0.17 ± 6% perf-stat.i.metric.K/sec
>>>>        3120            -5.8%       2938        perf-stat.i.minor-faults
>>>>       42.61            +1.2       43.83
>>>> perf-stat.i.node-load-miss-rate%
>>>>    24887573           -10.1%   22384693
>>>> perf-stat.i.node-load-misses
>>>>    31814752           -12.5%   27830232        perf-stat.i.node-loads
>>>>       36.62            +0.8       37.45
>>>> perf-stat.i.node-store-miss-rate%
>>>>    22617213           -20.3%   18015607
>>>> perf-stat.i.node-store-misses
>>>>    35707079           -19.9%   28604167        perf-stat.i.node-stores
>>>>        3120            -5.8%       2938        perf-stat.i.page-faults
>>>>       13.96            +9.3%      15.25        perf-stat.overall.MPKI
>>>>        0.99            +0.0        1.03
>>>> perf-stat.overall.branch-miss-rate%
>>>>       17.09            -4.0       13.05
>>>> perf-stat.overall.cache-miss-rate%
>>>>        1555           +19.4%       1856
>>>> perf-stat.overall.cycles-between-cache-misses
>>>>        0.69 ± 4%      -0.1        0.64 ± 5%
>>>> perf-stat.overall.dTLB-load-miss-rate%
>>>>        0.20 ± 3%      -0.0        0.16 ± 2%
>>>> perf-stat.overall.dTLB-store-miss-rate%
>>>>       86.43            -2.5       83.94
>>>> perf-stat.overall.iTLB-load-miss-rate%
>>>>        1433            +6.7%       1528
>>>> perf-stat.overall.instructions-per-iTLB-miss
>>>>       43.89            +0.7       44.58
>>>> perf-stat.overall.node-load-miss-rate%
>>>> 6.389e+09            +0.8% 6.438e+09
>>>> perf-stat.ps.branch-instructions
>>>>    63452800            +4.2%   66110454        perf-stat.ps.branch-misses
>>>>    65977234           -16.2%   55277568        perf-stat.ps.cache-misses
>>>> 3.861e+08            +9.8% 4.237e+08
>>>> perf-stat.ps.cache-references
>>>>      723870           +13.5%     821265
>>>> perf-stat.ps.context-switches
>>>>      112209            +8.6%     121881
>>>> perf-stat.ps.cpu-migrations
>>>>    49955941 ± 4%      -7.7%   46094243 ± 5%
>>>> perf-stat.ps.dTLB-load-misses
>>>>     4542094 ± 3%     -22.6%    3514899 ± 2%
>>>> perf-stat.ps.dTLB-store-misses
>>>> 2.275e+09            -5.3% 2.154e+09        perf-stat.ps.dTLB-stores
>>>>    19294019            -5.8%   18175580
>>>> perf-stat.ps.iTLB-load-misses
>>>>     3030049           +14.8%    3478314 ± 4% perf-stat.ps.iTLB-loads
>>>>        3089            -5.5%       2918        perf-stat.ps.minor-faults
>>>>    24643655            -9.8%   22235787
>>>> perf-stat.ps.node-load-misses
>>>>    31502901           -12.2%   27645143        perf-stat.ps.node-loads
>>>>    22395376           -20.1%   17895417
>>>> perf-stat.ps.node-store-misses
>>>>    35356897           -19.6%   28413366        perf-stat.ps.node-stores
>>>>        3089            -5.5%       2918        perf-stat.ps.page-faults
>>>>    2.85e+12           +50.6% 4.293e+12
>>>> perf-stat.total.instructions
>>>>       84193 ± 36%    +135.9%     198575 ± 25%
>>>> sched_debug.cfs_rq:/.MIN_vruntime.avg
>>>>      919053           +98.2%    1821460 ± 2%
>>>> sched_debug.cfs_rq:/.MIN_vruntime.max
>>>>      256686 ± 16%    +114.7%     551162 ± 13%
>>>> sched_debug.cfs_rq:/.MIN_vruntime.stddev
>>>>       19497          +106.7%      40307
>>>> sched_debug.cfs_rq:/.exec_clock.avg
>>>>       21240          +101.1%      42717
>>>> sched_debug.cfs_rq:/.exec_clock.max
>>>>       19079          +107.3%      39542
>>>> sched_debug.cfs_rq:/.exec_clock.min
>>>>      394.52 ± 2%     +54.3%     608.79 ± 13%
>>>> sched_debug.cfs_rq:/.exec_clock.stddev
>>>>      965.25 ± 15%     +59.7%       1541 ± 4%
>>>> sched_debug.cfs_rq:/.load_avg.avg
>>>>        2866 ± 21%     +57.1%       4503 ± 10%
>>>> sched_debug.cfs_rq:/.load_avg.max
>>>>      659.77 ± 15%     +68.9%       1114 ± 9%
>>>> sched_debug.cfs_rq:/.load_avg.stddev
>>>>       84193 ± 36%    +135.9%     198575 ± 25%
>>>> sched_debug.cfs_rq:/.max_vruntime.avg
>>>>      919053           +98.2%    1821462 ± 2%
>>>> sched_debug.cfs_rq:/.max_vruntime.max
>>>>      256686 ± 16%    +114.7%     551163 ± 13%
>>>> sched_debug.cfs_rq:/.max_vruntime.stddev
>>>>      901765           +97.1%    1777741 ± 2%
>>>> sched_debug.cfs_rq:/.min_vruntime.avg
>>>>      941794           +95.8%    1844445 ± 2%
>>>> sched_debug.cfs_rq:/.min_vruntime.max
>>>>      872925           +96.5%    1715658 ± 2%
>>>> sched_debug.cfs_rq:/.min_vruntime.min
>>>>       18358 ± 44%    +146.6%      45281 ± 34%
>>>> sched_debug.cfs_rq:/.min_vruntime.stddev
>>>>        1.22 ± 8%    +159.9%       3.18 ± 20%
>>>> sched_debug.cfs_rq:/.nr_spread_over.avg
>>>>        6.75 ± 23%     +65.4%      11.17 ± 17%
>>>> sched_debug.cfs_rq:/.nr_spread_over.max
>>>>        1.38 ± 11%     +61.4%       2.22 ± 12%
>>>> sched_debug.cfs_rq:/.nr_spread_over.stddev
>>>>       18357 ± 44%    +146.5%      45250 ± 34%
>>>> sched_debug.cfs_rq:/.spread0.stddev
>>>>      773.25 ± 3%     +13.2%     875.02
>>>> sched_debug.cfs_rq:/.util_avg.avg
>>>>      285624           -30.3%     199111 ± 4%
>>>> sched_debug.cpu.avg_idle.avg
>>>>      576507 ± 2%     -26.2%     425271 ± 7%
>>>> sched_debug.cpu.avg_idle.max
>>>>       51029 ± 68%     -83.4%       8469 ± 29%
>>>> sched_debug.cpu.avg_idle.min
>>>>      135149 ± 7%     -23.4%     103460 ± 10%
>>>> sched_debug.cpu.avg_idle.stddev
>>>>       57492           +51.0%      86835        sched_debug.cpu.clock.avg
>>>>       57495           +51.0%      86841        sched_debug.cpu.clock.max
>>>>       57487           +51.0%      86828        sched_debug.cpu.clock.min
>>>>        1.98 ± 4%     +74.8%       3.46 ± 10%
>>>> sched_debug.cpu.clock.stddev
>>>>       57492           +51.0%      86835
>>>> sched_debug.cpu.clock_task.avg
>>>>       57495           +51.0%      86841
>>>> sched_debug.cpu.clock_task.max
>>>>       57487           +51.0%      86828
>>>> sched_debug.cpu.clock_task.min
>>>>        1.98 ± 4%     +74.8%       3.46 ± 10%
>>>> sched_debug.cpu.clock_task.stddev
>>>>        2831           +43.0%       4049
>>>> sched_debug.cpu.curr->pid.max
>>>>      417519          +133.3%     974078
>>>> sched_debug.cpu.nr_switches.avg
>>>>      427854          +134.0%    1001016
>>>> sched_debug.cpu.nr_switches.max
>>>>      409942          +131.8%     950218
>>>> sched_debug.cpu.nr_switches.min
>>>>        4770 ± 42%    +238.7%      16156 ± 22%
>>>> sched_debug.cpu.nr_switches.stddev
>>>>        4.70 ± 23%     +37.1%       6.44 ± 4%
>>>> sched_debug.cpu.nr_uninterruptible.avg
>>>>      776.62 ± 19%     +86.1%       1445 ± 12%
>>>> sched_debug.cpu.nr_uninterruptible.max
>>>>     -472.62           +98.3%    -937.33
>>>> sched_debug.cpu.nr_uninterruptible.min
>>>>      289.80 ± 5%    +109.7%     607.68 ± 3%
>>>> sched_debug.cpu.nr_uninterruptible.stddev
>>>>      415893          +133.8%     972526
>>>> sched_debug.cpu.sched_count.avg
>>>>      423960          +135.5%     998457
>>>> sched_debug.cpu.sched_count.max
>>>>      409113          +132.0%     949068
>>>> sched_debug.cpu.sched_count.min
>>>>        4194 ± 50%    +274.3%      15698 ± 22%
>>>> sched_debug.cpu.sched_count.stddev
>>>>      110094          +125.8%     248571
>>>> sched_debug.cpu.sched_goidle.avg
>>>>      112236          +125.6%     253163
>>>> sched_debug.cpu.sched_goidle.max
>>>>      108281          +125.8%     244506
>>>> sched_debug.cpu.sched_goidle.min
>>>>      772.04 ± 16%    +175.8%       2129 ± 18%
>>>> sched_debug.cpu.sched_goidle.stddev
>>>>      297267          +137.2%     705173
>>>> sched_debug.cpu.ttwu_count.avg
>>>>      320112          +135.9%     755096
>>>> sched_debug.cpu.ttwu_count.max
>>>>      272487 ± 2%    +142.2%     660099
>>>> sched_debug.cpu.ttwu_count.min
>>>>       10662 ± 10%    +110.9%      22491 ± 18%
>>>> sched_debug.cpu.ttwu_count.stddev
>>>>       12295 ± 2%    +834.9%     114954 ± 2%
>>>> sched_debug.cpu.ttwu_local.avg
>>>>       13726 ± 2%    +769.7%     119376 ± 2%
>>>> sched_debug.cpu.ttwu_local.max
>>>>       10620 ± 4%    +942.4%     110710
>>>> sched_debug.cpu.ttwu_local.min
>>>>      693.29 ± 14%    +228.7%       2278 ± 18%
>>>> sched_debug.cpu.ttwu_local.stddev
>>>>       57488           +51.0%      86829        sched_debug.cpu_clk
>>>>       56998           +51.5%      86337        sched_debug.ktime
>>>>       57977           +50.6%      87317        sched_debug.sched_clk
>>>>       72.50 ± 37%     +55.5%     112.75 ± 26%
>>>> interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
>>>>       53.25 ± 2%     +84.5%      98.25 ± 22%
>>>> interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
>>>>       73.50 ± 26%    +116.3%     159.00 ± 60%
>>>> interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
>>>>      204541           +49.0%     304702 ± 2%
>>>> interrupts.CPU0.LOC:Local_timer_interrupts
>>>>      290268 ± 2%     +29.1%     374641 ± 4%
>>>> interrupts.CPU0.RES:Rescheduling_interrupts
>>>>      204580           +48.9%     304667 ± 2%
>>>> interrupts.CPU1.LOC:Local_timer_interrupts
>>>>      290342 ± 2%     +28.5%     373110 ± 4%
>>>> interrupts.CPU1.RES:Rescheduling_interrupts
>>>>      204537           +48.9%     304619 ± 2%
>>>> interrupts.CPU10.LOC:Local_timer_interrupts
>>>>      291605 ± 2%     +28.3%     374190 ± 4%
>>>> interrupts.CPU10.RES:Rescheduling_interrupts
>>>>      204140           +49.2%     304587 ± 2%
>>>> interrupts.CPU11.LOC:Local_timer_interrupts
>>>>      291988 ± 2%     +28.1%     374024 ± 4%
>>>> interrupts.CPU11.RES:Rescheduling_interrupts
>>>>      204367           +49.0%     304559 ± 2%
>>>> interrupts.CPU12.LOC:Local_timer_interrupts
>>>>      288632           +28.4%     370669 ± 3%
>>>> interrupts.CPU12.RES:Rescheduling_interrupts
>>>>      204555           +48.9%     304669 ± 2%
>>>> interrupts.CPU13.LOC:Local_timer_interrupts
>>>>      289354 ± 2%     +28.5%     371782 ± 3%
>>>> interrupts.CPU13.RES:Rescheduling_interrupts
>>>>      204536           +49.0%     304726 ± 2%
>>>> interrupts.CPU14.LOC:Local_timer_interrupts
>>>>        4937 ± 25%     +41.6%       6991 ± 2%
>>>> interrupts.CPU14.NMI:Non-maskable_interrupts
>>>>        4937 ± 25%     +41.6%       6991 ± 2%
>>>> interrupts.CPU14.PMI:Performance_monitoring_interrupts
>>>>      287759           +28.2%     368986 ± 3%
>>>> interrupts.CPU14.RES:Rescheduling_interrupts
>>>>      204527           +49.0%     304707 ± 2%
>>>> interrupts.CPU15.LOC:Local_timer_interrupts
>>>>      286950           +28.6%     369104 ± 3%
>>>> interrupts.CPU15.RES:Rescheduling_interrupts
>>>>      204556           +49.0%     304718 ± 2%
>>>> interrupts.CPU16.LOC:Local_timer_interrupts
>>>>        5444 ± 32%     -19.9%       4360 ± 32%
>>>> interrupts.CPU16.NMI:Non-maskable_interrupts
>>>>        5444 ± 32%     -19.9%       4360 ± 32%
>>>> interrupts.CPU16.PMI:Performance_monitoring_interrupts
>>>>      287623           +28.1%     368583 ± 3%
>>>> interrupts.CPU16.RES:Rescheduling_interrupts
>>>>      204528           +49.0%     304728 ± 2%
>>>> interrupts.CPU17.LOC:Local_timer_interrupts
>>>>      287734           +28.2%     368920 ± 3%
>>>> interrupts.CPU17.RES:Rescheduling_interrupts
>>>>      204465           +49.0%     304707 ± 2%
>>>> interrupts.CPU18.LOC:Local_timer_interrupts
>>>>        5382 ± 31%     -21.8%       4211 ± 27%
>>>> interrupts.CPU18.NMI:Non-maskable_interrupts
>>>>        5382 ± 31%     -21.8%       4211 ± 27%
>>>> interrupts.CPU18.PMI:Performance_monitoring_interrupts
>>>>      288778           +28.1%     369871 ± 3%
>>>> interrupts.CPU18.RES:Rescheduling_interrupts
>>>>      204302           +49.2%     304727 ± 2%
>>>> interrupts.CPU19.LOC:Local_timer_interrupts
>>>>      289794           +27.8%     370247 ± 3%
>>>> interrupts.CPU19.RES:Rescheduling_interrupts
>>>>      204552           +49.0%     304821 ± 2%
>>>> interrupts.CPU2.LOC:Local_timer_interrupts
>>>>      291056 ± 2%     +28.3%     373290 ± 4%
>>>> interrupts.CPU2.RES:Rescheduling_interrupts
>>>>      204468           +48.9%     304484 ± 2%
>>>> interrupts.CPU20.LOC:Local_timer_interrupts
>>>>      290778           +27.6%     371106 ± 3%
>>>> interrupts.CPU20.RES:Rescheduling_interrupts
>>>>      204546           +48.9%     304619 ± 2%
>>>> interrupts.CPU21.LOC:Local_timer_interrupts
>>>>      289974           +28.0%     371074 ± 3%
>>>> interrupts.CPU21.RES:Rescheduling_interrupts
>>>>      204694           +48.8%     304581 ± 2%
>>>> interrupts.CPU22.LOC:Local_timer_interrupts
>>>>      290312           +28.0%     371734 ± 3%
>>>> interrupts.CPU22.RES:Rescheduling_interrupts
>>>>      204374           +49.0%     304493 ± 2%
>>>> interrupts.CPU23.LOC:Local_timer_interrupts
>>>>      290266           +28.5%     372865 ± 3%
>>>> interrupts.CPU23.RES:Rescheduling_interrupts
>>>>      204518           +49.0%     304653 ± 2%
>>>> interrupts.CPU24.LOC:Local_timer_interrupts
>>>>      288815 ± 2%     +28.7%     371621 ± 4%
>>>> interrupts.CPU24.RES:Rescheduling_interrupts
>>>>      204414           +49.0%     304640 ± 2%
>>>> interrupts.CPU25.LOC:Local_timer_interrupts
>>>>      288779 ± 2%     +28.2%     370202 ± 4%
>>>> interrupts.CPU25.RES:Rescheduling_interrupts
>>>>      204536           +49.0%     304730 ± 2%
>>>> interrupts.CPU26.LOC:Local_timer_interrupts
>>>>      287615 ± 2%     +28.6%     369843 ± 4%
>>>> interrupts.CPU26.RES:Rescheduling_interrupts
>>>>       72.50 ± 37%     +55.5%     112.75 ± 26%
>>>> interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
>>>>      204514           +49.0%     304668 ± 2%
>>>> interrupts.CPU27.LOC:Local_timer_interrupts
>>>>      287014 ± 2%     +28.5%     368825 ± 4%
>>>> interrupts.CPU27.RES:Rescheduling_interrupts
>>>>       53.25 ± 2%     +84.5%      98.25 ± 22%
>>>> interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
>>>>      204331           +49.1%     304759 ± 2%
>>>> interrupts.CPU28.LOC:Local_timer_interrupts
>>>>      287278 ± 2%     +28.4%     368876 ± 4%
>>>> interrupts.CPU28.RES:Rescheduling_interrupts
>>>>       73.50 ± 26%    +116.3%     159.00 ± 60%
>>>> interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
>>>>      204427           +49.0%     304651 ± 2%
>>>> interrupts.CPU29.LOC:Local_timer_interrupts
>>>>      287382 ± 2%     +28.9%     370434 ± 4%
>>>> interrupts.CPU29.RES:Rescheduling_interrupts
>>>>      204538           +49.0%     304765 ± 2%
>>>> interrupts.CPU3.LOC:Local_timer_interrupts
>>>>        5034 ± 27%     -29.4%       3552
>>>> interrupts.CPU3.NMI:Non-maskable_interrupts
>>>>        5034 ± 27%     -29.4%       3552
>>>> interrupts.CPU3.PMI:Performance_monitoring_interrupts
>>>>      288881 ± 2%     +28.3%     370740 ± 4%
>>>> interrupts.CPU3.RES:Rescheduling_interrupts
>>>>      204750           +48.8%     304674 ± 2%
>>>> interrupts.CPU30.LOC:Local_timer_interrupts
>>>>      287139 ± 2%     +28.8%     369825 ± 4%
>>>> interrupts.CPU30.RES:Rescheduling_interrupts
>>>>      204429           +48.9%     304416 ± 2%
>>>> interrupts.CPU31.LOC:Local_timer_interrupts
>>>>      288411 ± 2%     +28.4%     370199 ± 4%
>>>> interrupts.CPU31.RES:Rescheduling_interrupts
>>>>      204395           +49.3%     305106 ± 2%
>>>> interrupts.CPU32.LOC:Local_timer_interrupts
>>>>      289178 ± 2%     +28.6%     371830 ± 4%
>>>> interrupts.CPU32.RES:Rescheduling_interrupts
>>>>      204347           +49.2%     304799 ± 2%
>>>> interrupts.CPU33.LOC:Local_timer_interrupts
>>>>      289689 ± 2%     +28.2%     371280 ± 4%
>>>> interrupts.CPU33.RES:Rescheduling_interrupts
>>>>      204280           +49.1%     304566 ± 2%
>>>> interrupts.CPU34.LOC:Local_timer_interrupts
>>>>        6418 ± 24%     -44.5%       3564
>>>> interrupts.CPU34.NMI:Non-maskable_interrupts
>>>>        6418 ± 24%     -44.5%       3564
>>>> interrupts.CPU34.PMI:Performance_monitoring_interrupts
>>>>      290382 ± 2%     +28.3%     372606 ± 4%
>>>> interrupts.CPU34.RES:Rescheduling_interrupts
>>>>      204331           +49.0%     304517 ± 2%
>>>> interrupts.CPU35.LOC:Local_timer_interrupts
>>>>      289736 ± 2%     +28.3%     371869 ± 4%
>>>> interrupts.CPU35.RES:Rescheduling_interrupts
>>>>      204578           +48.8%     304423 ± 2%
>>>> interrupts.CPU36.LOC:Local_timer_interrupts
>>>>        7298           -39.0%       4450 ± 34%
>>>> interrupts.CPU36.NMI:Non-maskable_interrupts
>>>>        7298           -39.0%       4450 ± 34%
>>>> interrupts.CPU36.PMI:Performance_monitoring_interrupts
>>>>      287187           +28.3%     368516 ± 3%
>>>> interrupts.CPU36.RES:Rescheduling_interrupts
>>>>      204538           +49.0%     304729 ± 2%
>>>> interrupts.CPU37.LOC:Local_timer_interrupts
>>>>      288676           +28.1%     369792 ± 3%
>>>> interrupts.CPU37.RES:Rescheduling_interrupts
>>>>      204504           +49.0%     304687 ± 2%
>>>> interrupts.CPU38.LOC:Local_timer_interrupts
>>>>        6301 ± 24%     -29.8%       4424 ± 34%
>>>> interrupts.CPU38.NMI:Non-maskable_interrupts
>>>>        6301 ± 24%     -29.8%       4424 ± 34%
>>>> interrupts.CPU38.PMI:Performance_monitoring_interrupts
>>>>      287510           +28.1%     368349 ± 3%
>>>> interrupts.CPU38.RES:Rescheduling_interrupts
>>>>      204566           +48.9%     304668 ± 2%
>>>> interrupts.CPU39.LOC:Local_timer_interrupts
>>>>      286909           +28.1%     367622 ± 3%
>>>> interrupts.CPU39.RES:Rescheduling_interrupts
>>>>      204445           +49.0%     304686 ± 2%
>>>> interrupts.CPU4.LOC:Local_timer_interrupts
>>>>      288608 ± 2%     +28.7%     371493 ± 3%
>>>> interrupts.CPU4.RES:Rescheduling_interrupts
>>>>      204538           +48.9%     304582 ± 2%
>>>> interrupts.CPU40.LOC:Local_timer_interrupts
>>>>      286219           +27.9%     365983 ± 3%
>>>> interrupts.CPU40.RES:Rescheduling_interrupts
>>>>      204551           +49.0%     304739 ± 2%
>>>> interrupts.CPU41.LOC:Local_timer_interrupts
>>>>        5170 ± 29%     -18.5%       4215 ± 27%
>>>> interrupts.CPU41.NMI:Non-maskable_interrupts
>>>>        5170 ± 29%     -18.5%       4215 ± 27%
>>>> interrupts.CPU41.PMI:Performance_monitoring_interrupts
>>>>      286946           +27.8%     366717 ± 3%
>>>> interrupts.CPU41.RES:Rescheduling_interrupts
>>>>      204492           +48.9%     304442 ± 2%
>>>> interrupts.CPU42.LOC:Local_timer_interrupts
>>>>      287384           +28.0%     367805 ± 3%
>>>> interrupts.CPU42.RES:Rescheduling_interrupts
>>>>      204545           +49.0%     304679 ± 2%
>>>> interrupts.CPU43.LOC:Local_timer_interrupts
>>>>      289100           +27.3%     368059 ± 3%
>>>> interrupts.CPU43.RES:Rescheduling_interrupts
>>>>      204489           +49.0%     304635 ± 2%
>>>> interrupts.CPU44.LOC:Local_timer_interrupts
>>>>        6393 ± 24%     -32.4%       4319 ± 31%
>>>> interrupts.CPU44.NMI:Non-maskable_interrupts
>>>>        6393 ± 24%     -32.4%       4319 ± 31%
>>>> interrupts.CPU44.PMI:Performance_monitoring_interrupts
>>>>      288835           +27.8%     369182 ± 3%
>>>> interrupts.CPU44.RES:Rescheduling_interrupts
>>>>      204473           +49.2%     304983 ± 2%
>>>> interrupts.CPU45.LOC:Local_timer_interrupts
>>>>      289778           +27.4%     369059 ± 3%
>>>> interrupts.CPU45.RES:Rescheduling_interrupts
>>>>      204493           +49.0%     304598 ± 2%
>>>> interrupts.CPU46.LOC:Local_timer_interrupts
>>>>      289485           +27.9%     370172 ± 3%
>>>> interrupts.CPU46.RES:Rescheduling_interrupts
>>>>      204511           +49.2%     305090 ± 2%
>>>> interrupts.CPU47.LOC:Local_timer_interrupts
>>>>      286611           +28.8%     369021 ± 3%
>>>> interrupts.CPU47.RES:Rescheduling_interrupts
>>>>      204441           +49.1%     304794 ± 2%
>>>> interrupts.CPU5.LOC:Local_timer_interrupts
>>>>      289245 ± 2%     +28.4%     371335 ± 4%
>>>> interrupts.CPU5.RES:Rescheduling_interrupts
>>>>      204246           +49.0%     304414 ± 2%
>>>> interrupts.CPU6.LOC:Local_timer_interrupts
>>>>      290006 ± 2%     +28.3%     372180 ± 4%
>>>> interrupts.CPU6.RES:Rescheduling_interrupts
>>>>      204445           +49.0%     304719 ± 2%
>>>> interrupts.CPU7.LOC:Local_timer_interrupts
>>>>      290524 ± 2%     +28.2%     372414 ± 4%
>>>> interrupts.CPU7.RES:Rescheduling_interrupts
>>>>      204178           +49.2%     304663 ± 2%
>>>> interrupts.CPU8.LOC:Local_timer_interrupts
>>>>      291625 ± 2%     +27.8%     372747 ± 4%
>>>> interrupts.CPU8.RES:Rescheduling_interrupts
>>>>      204502           +48.9%     304407 ± 3%
>>>> interrupts.CPU9.LOC:Local_timer_interrupts
>>>>      291615 ± 2%     +27.9%     373024 ± 4%
>>>> interrupts.CPU9.RES:Rescheduling_interrupts
>>>>     9814627           +49.0%   14624016 ± 2%
>>>> interrupts.LOC:Local_timer_interrupts
>>>>    13869791           +28.2%   17785831 ± 3%
>>>> interrupts.RES:Rescheduling_interrupts
>>>>       25728           +47.6%      37969        softirqs.CPU0.RCU
>>>>       34944           +41.7%      49503 ± 3% softirqs.CPU0.SCHED
>>>>       42375           +38.5%      58673 ± 2% softirqs.CPU0.TIMER
>>>>       25348           +53.8%      38976 ± 5% softirqs.CPU1.RCU
>>>>       30986           +46.2%      45315 ± 3% softirqs.CPU1.SCHED
>>>>       41265           +44.3%      59538 ± 3% softirqs.CPU1.TIMER
>>>>       25149           +48.3%      37288        softirqs.CPU10.RCU
>>>>       29752           +47.1%      43764 ± 2% softirqs.CPU10.SCHED
>>>>       40609           +41.5%      57449        softirqs.CPU10.TIMER
>>>>       25218           +48.3%      37399        softirqs.CPU11.RCU
>>>>       30294           +44.6%      43800        softirqs.CPU11.SCHED
>>>>       41371           +39.2%      57578        softirqs.CPU11.TIMER
>>>>       24427           +48.9%      36361        softirqs.CPU12.RCU
>>>>       30438           +44.0%      43835        softirqs.CPU12.SCHED
>>>>       41045 ± 3%     +43.0%      58687        softirqs.CPU12.TIMER
>>>>       24247           +47.0%      35645        softirqs.CPU13.RCU
>>>>       30155           +44.5%      43577        softirqs.CPU13.SCHED
>>>>       41148 ± 2%     +40.3%      57720        softirqs.CPU13.TIMER
>>>>       24506           +46.3%      35856        softirqs.CPU14.RCU
>>>>       29991           +44.1%      43229        softirqs.CPU14.SCHED
>>>>       41190 ± 2%     +39.7%      57552        softirqs.CPU14.TIMER
>>>>       24412           +46.7%      35800        softirqs.CPU15.RCU
>>>>       30135           +44.1%      43430        softirqs.CPU15.SCHED
>>>>       45081 ± 15%     +28.0%      57684        softirqs.CPU15.TIMER
>>>>       25327           +47.0%      37225        softirqs.CPU16.RCU
>>>>       30230           +43.8%      43461        softirqs.CPU16.SCHED
>>>>       41157           +39.6%      57448        softirqs.CPU16.TIMER
>>>>       25136           +47.8%      37142        softirqs.CPU17.RCU
>>>>       29894           +44.5%      43212        softirqs.CPU17.SCHED
>>>>       41961 ± 5%     +36.5%      57281        softirqs.CPU17.TIMER
>>>>       25431           +46.1%      37160        softirqs.CPU18.RCU
>>>>       30062           +44.0%      43281        softirqs.CPU18.SCHED
>>>>       40956 ± 2%     +42.1%      58217        softirqs.CPU18.TIMER
>>>>       25496           +44.8%      36923 ± 2% softirqs.CPU19.RCU
>>>>       29989           +46.5%      43946        softirqs.CPU19.SCHED
>>>>       41740 ± 2%     +67.8%      70057 ± 17% softirqs.CPU19.TIMER
>>>>       25414           +48.3%      37678        softirqs.CPU2.RCU
>>>>       30905           +43.4%      44309 ± 2% softirqs.CPU2.SCHED
>>>>       41083           +40.6%      57768 ± 2% softirqs.CPU2.TIMER
>>>>       25124           +46.8%      36892        softirqs.CPU20.RCU
>>>>       30001           +44.7%      43416        softirqs.CPU20.SCHED
>>>>       41060 ± 7%     +39.2%      57151        softirqs.CPU20.TIMER
>>>>       24992           +47.6%      36896        softirqs.CPU21.RCU
>>>>       30109           +44.4%      43467        softirqs.CPU21.SCHED
>>>>       40055 ± 3%     +42.5%      57081 ± 2% softirqs.CPU21.TIMER
>>>>       25072           +46.2%      36656        softirqs.CPU22.RCU
>>>>       30296           +43.3%      43425        softirqs.CPU22.SCHED
>>>>       24882           +48.8%      37031        softirqs.CPU23.RCU
>>>>       29900           +43.7%      42971        softirqs.CPU23.SCHED
>>>>       40388 ± 3%     +44.8%      58494 ± 3% softirqs.CPU23.TIMER
>>>>       25007           +48.1%      37036        softirqs.CPU24.RCU
>>>>       29755           +45.9%      43399        softirqs.CPU24.SCHED
>>>>       40786           +40.6%      57353        softirqs.CPU24.TIMER
>>>>       25031           +47.3%      36875        softirqs.CPU25.RCU
>>>>       29841 ± 2%     +45.3%      43355        softirqs.CPU25.SCHED
>>>>       40971           +39.8%      57266        softirqs.CPU25.TIMER
>>>>       25145           +47.6%      37114        softirqs.CPU26.RCU
>>>>       29462 ± 2%     +48.5%      43739 ± 2% softirqs.CPU26.SCHED
>>>>       40553           +41.8%      57521        softirqs.CPU26.TIMER
>>>>       25184           +47.6%      37169        softirqs.CPU27.RCU
>>>>       29867           +45.5%      43450 ± 3% softirqs.CPU27.SCHED
>>>>       41118           +39.3%      57261        softirqs.CPU27.TIMER
>>>>       25452           +47.1%      37440        softirqs.CPU28.RCU
>>>>       29714           +46.2%      43440 ± 2% softirqs.CPU28.SCHED
>>>>       41209 ± 2%     +37.8%      56783 ± 2% softirqs.CPU28.TIMER
>>>>       25334           +48.8%      37696 ± 2% softirqs.CPU29.RCU
>>>>       29894           +46.3%      43749        softirqs.CPU29.SCHED
>>>>       41413 ± 2%     +43.3%      59346 ± 5% softirqs.CPU29.TIMER
>>>>       25514           +47.0%      37495        softirqs.CPU3.RCU
>>>>       30016           +46.0%      43831 ± 2% softirqs.CPU3.SCHED
>>>>       41371 ± 2%     +39.3%      57611        softirqs.CPU3.TIMER
>>>>       25206           +47.4%      37149        softirqs.CPU30.RCU
>>>>       30167           +44.0%      43451 ± 2% softirqs.CPU30.SCHED
>>>>       41166 ± 2%     +58.1%      65092 ± 19% softirqs.CPU30.TIMER
>>>>       25211           +46.8%      37016        softirqs.CPU31.RCU
>>>>       30004 ± 2%     +45.2%      43569        softirqs.CPU31.SCHED
>>>>       41541 ± 2%     +36.9%      56874        softirqs.CPU31.TIMER
>>>>       24432           +46.9%      35885        softirqs.CPU32.RCU
>>>>       30206           +43.4%      43322 ± 2% softirqs.CPU32.SCHED
>>>>       41069           +40.7%      57781 ± 2% softirqs.CPU32.TIMER
>>>>       24303           +48.0%      35973        softirqs.CPU33.RCU
>>>>       30238           +43.8%      43489        softirqs.CPU33.SCHED
>>>>       41161           +38.2%      56872        softirqs.CPU33.TIMER
>>>>       24308           +47.6%      35876        softirqs.CPU34.RCU
>>>>       29803           +46.5%      43647 ± 2% softirqs.CPU34.SCHED
>>>>       40809           +39.2%      56817        softirqs.CPU34.TIMER
>>>>       24291           +47.7%      35877        softirqs.CPU35.RCU
>>>>       30092           +44.2%      43386        softirqs.CPU35.SCHED
>>>>       40607           +41.5%      57480 ± 2% softirqs.CPU35.TIMER
>>>>       25605           +49.5%      38286        softirqs.CPU36.RCU
>>>>       29900           +44.6%      43238        softirqs.CPU36.SCHED
>>>>       45207 ± 15%     +27.4%      57602        softirqs.CPU36.TIMER
>>>>       25555 ± 2%     +49.0%      38070        softirqs.CPU37.RCU
>>>>       29926           +45.2%      43460        softirqs.CPU37.SCHED
>>>>       42619 ± 8%     +36.7%      58242        softirqs.CPU37.TIMER
>>>>       27683 ± 9%     +50.2%      41577 ± 4% softirqs.CPU38.RCU
>>>>       29700           +45.6%      43230        softirqs.CPU38.SCHED
>>>>       41779 ± 4%     +43.5%      59948        softirqs.CPU38.TIMER
>>>>       26699 ± 7%     +42.1%      37951        softirqs.CPU39.RCU
>>>>       30061           +43.4%      43102        softirqs.CPU39.SCHED
>>>>       42312 ± 5%     +36.1%      57581 ± 2% softirqs.CPU39.TIMER
>>>>       25219           +47.7%      37247        softirqs.CPU4.RCU
>>>>       30363 ± 3%     +45.9%      44295        softirqs.CPU4.SCHED
>>>>       41108 ± 2%     +39.6%      57371 ± 2% softirqs.CPU4.TIMER
>>>>       25413           +50.6%      38268        softirqs.CPU40.RCU
>>>>       29910           +46.0%      43665        softirqs.CPU40.SCHED
>>>>       39987 ± 2%     +45.9%      58322        softirqs.CPU40.TIMER
>>>>       25569           +49.5%      38238        softirqs.CPU41.RCU
>>>>       29867           +44.8%      43247        softirqs.CPU41.SCHED
>>>>       25492           +49.8%      38185        softirqs.CPU42.RCU
>>>>       29871           +45.4%      43440        softirqs.CPU42.SCHED
>>>>       40076           +44.8%      58020        softirqs.CPU42.TIMER
>>>>       25281           +48.9%      37640        softirqs.CPU43.RCU
>>>>       30025           +43.9%      43220        softirqs.CPU43.SCHED
>>>>       40222 ± 3%     +44.7%      58195        softirqs.CPU43.TIMER
>>>>       25183           +49.3%      37611        softirqs.CPU44.RCU
>>>>       30123           +43.6%      43249        softirqs.CPU44.SCHED
>>>>       41090 ± 6%     +42.5%      58537        softirqs.CPU44.TIMER
>>>>       25314           +49.6%      37863        softirqs.CPU45.RCU
>>>>       29998           +43.2%      42957        softirqs.CPU45.SCHED
>>>>       39705 ± 3%     +43.6%      57033        softirqs.CPU45.TIMER
>>>>       25500           +46.2%      37274        softirqs.CPU46.RCU
>>>>       29921           +44.7%      43286        softirqs.CPU46.SCHED
>>>>       41088 ± 5%     +39.9%      57463 ± 3% softirqs.CPU46.TIMER
>>>>       25564           +48.4%      37931        softirqs.CPU47.RCU
>>>>       30178           +42.4%      42980        softirqs.CPU47.SCHED
>>>>       40738 ± 4%     +42.2%      57939 ± 3% softirqs.CPU47.TIMER
>>>>       26467 ± 8%     +40.8%      37278        softirqs.CPU5.RCU
>>>>       30334 ± 2%     +44.1%      43714        softirqs.CPU5.SCHED
>>>>       42268 ± 4%     +35.8%      57394        softirqs.CPU5.TIMER
>>>>       25343           +47.0%      37245        softirqs.CPU6.RCU
>>>>       30100           +45.0%      43643 ± 2% softirqs.CPU6.SCHED
>>>>       41270           +43.6%      59249 ± 5% softirqs.CPU6.TIMER
>>>>       25331           +47.7%      37418        softirqs.CPU7.RCU
>>>>       30210           +44.7%      43727 ± 2% softirqs.CPU7.SCHED
>>>>       41028           +41.9%      58206 ± 2% softirqs.CPU7.TIMER
>>>>       25483           +47.2%      37514        softirqs.CPU8.RCU
>>>>       30324 ± 2%     +43.4%      43483 ± 2% softirqs.CPU8.SCHED
>>>>       41027           +40.4%      57591 ± 2% softirqs.CPU8.TIMER
>>>>       25546           +47.5%      37687        softirqs.CPU9.RCU
>>>>       30218           +44.8%      43771 ± 2% softirqs.CPU9.SCHED
>>>>       41193           +40.1%      57716 ± 2% softirqs.CPU9.TIMER
>>>>     1212606           +47.7%    1790808        softirqs.RCU
>>>>     1448198           +44.7%    2095500        softirqs.SCHED
>>>>     1990064           +40.7%    2800202        softirqs.TIMER
>>>>        8.56 ± 3%      -4.3        4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
>>>>
>>>>        8.50 ± 3%      -4.3        4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
>>>>
>>>>        8.47 ± 3%      -4.3        4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
>>>>
>>>>       78.55            -4.1       74.44
>>>> perf-profile.calltrace.cycles-pp.write
>>>>        8.74 ± 3%      -4.0        4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
>>>>
>>>>       77.69            -3.9       73.75
>>>> perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
>>>>       77.64            -3.9       73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
>>>>
>>>>       77.13            -3.8       73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
>>>>
>>>>       77.08            -3.8       73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
>>>>
>>>>       76.61            -3.7       72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
>>>>
>>>>       76.43            -3.6       72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
>>>>
>>>>       69.22            -2.6       66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
>>>>
>>>>        4.70 ± 2%      -2.2        2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
>>>>
>>>>        4.63 ± 2%      -2.2        2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
>>>>
>>>>        4.60 ± 2%      -2.2        2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
>>>>
>>>>        4.32 ± 2%      -2.1        2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
>>>>
>>>>        3.72 ± 4%      -2.1        1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
>>>>
>>>>        3.71 ± 4%      -2.1        1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
>>>>
>>>>        2.63 ± 4%      -2.0        0.63 ± 3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
>>>>
>>>>        7.72            -2.0        5.74 ± 2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
>>>>
>>>>        3.08 ± 5%      -1.9        1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
>>>>
>>>>        2.49 ± 5%      -1.9        0.55 ± 3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
>>>>
>>>>        2.48 ± 5%      -1.9        0.55 ± 3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
>>>>
>>>>        2.41 ± 6%      -1.8        0.61 ± 2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
>>>>
>>>>        4.90            -1.5        3.40 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>>>
>>>>        4.78            -1.5        3.30 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
>>>>
>>>>        2.35            -0.6        1.77 ± 2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
>>>>
>>>>        2.36            -0.6        1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
>>>>
>>>>        3.40            -0.5        2.87 ± 6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
>>>>
>>>>        1.83            -0.5        1.33 ± 2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
>>>>
>>>>        3.26            -0.5        2.75 ± 6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
>>>>
>>>>        2.85            -0.4        2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
>>>>
>>>>        2.79            -0.4        2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
>>>>
>>>>        2.83            -0.4        2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
>>>>
>>>>        1.48            -0.4        1.05 ± 2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
>>>>
>>>>        0.61 ± 2%      -0.4        0.26 ±100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
>>>>
>>>>        1.89            -0.3        1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>>>
>>>>        1.71            -0.3        1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
>>>>
>>>>        1.65            -0.3        1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
>>>>
>>>>        0.69            -0.3        0.38 ± 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
>>>>
>>>>        1.59            -0.3        1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
>>>>
>>>>        0.86            -0.3        0.58 ± 2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
>>>>
>>>>        1.52            -0.3        1.26 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
>>>>
>>>>        1.75            -0.2        1.50 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
>>>>
>>>>        1.74            -0.2        1.49 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
>>>>
>>>>        1.74            -0.2        1.50 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
>>>>
>>>>        2.15            -0.2        1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
>>>>
>>>>        0.79 ± 3%      -0.2        0.60 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
>>>>
>>>>        1.81            -0.2        1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
>>>>
>>>>        1.87            -0.2        1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
>>>>
>>>>        0.70 ± 4%      -0.2        0.54 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
>>>>
>>>>        0.81            -0.1        0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
>>>>
>>>>        0.65 ± 2%      -0.1        0.55 ± 2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
>>>>
>>>>        0.62 ± 2%      -0.1        0.52 ± 3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
>>>>
>>>>        1.18            -0.0        1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>>>
>>>>        1.01            -0.0        0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
>>>>
>>>>        1.00            -0.0        0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
>>>>
>>>>        0.96            -0.0        0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
>>>>
>>>>        1.01            -0.0        0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
>>>>
>>>>        0.59 ± 3%      +0.1        0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>>>
>>>>        0.58 ± 3%      +0.1        0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
>>>>
>>>>        0.53            +0.1        0.63 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
>>>>
>>>>        0.91 ± 3%      +0.1        1.02 ± 2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
>>>>
>>>>        0.54            +0.1        0.65 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
>>>>
>>>>        0.57            +0.1        0.69 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
>>>>
>>>>        1.83            +0.2        1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
>>>>
>>>>        0.00            +0.6        0.57 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
>>>>
>>>>        0.00            +0.6        0.57 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
>>>>
>>>>        0.00            +0.6        0.57 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
>>>>
>>>>        0.00            +0.7        0.70 ± 2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
>>>>
>>>>        0.00            +0.8        0.75 ± 4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
>>>>
>>>>        0.00            +0.8        0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
>>>>
>>>>        0.00            +0.8        0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
>>>>
>>>>        0.00            +0.8        0.80 ± 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
>>>>
>>>>        0.00            +0.8        0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
>>>>
>>>>        0.00            +0.8        0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
>>>>
>>>>        0.00            +0.9        0.89 ± 2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
>>>>
>>>>        0.00            +0.9        0.92 ± 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
>>>>
>>>>        0.00            +1.0        0.97 ± 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
>>>>
>>>>       35.68            +1.0       36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
>>>>
>>>>        0.00            +1.0        1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
>>>>
>>>>        0.00            +1.0        1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
>>>>
>>>>        9.35            +1.2       10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
>>>>
>>>>        9.61            +1.2       10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
>>>>
>>>>        9.63            +1.2       10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>>>
>>>>       12.24            +1.3       13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
>>>>
>>>>       12.25            +1.3       13.60
>>>> perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
>>>>       12.25            +1.3       13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
>>>>
>>>>       12.49            +1.4       13.89
>>>> perf-profile.calltrace.cycles-pp.secondary_startup_64
>>>>       14.38            +1.4       15.81 ± 2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
>>>>
>>>>       13.41            +1.4       14.86 ± 2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>>>
>>>>       13.37            +1.5       14.82 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
>>>>
>>>>       13.34            +1.5       14.80 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
>>>>
>>>>       58.91            +1.6       60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
>>>>
>>>>        0.00            +2.2        2.24 ± 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
>>>>
>>>>        5.63            +2.5        8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
>>>>
>>>>        5.65            +2.5        8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
>>>>
>>>>        5.69            +2.5        8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
>>>>
>>>>        7.04            +2.8        9.80 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
>>>>
>>>>        7.54            +2.9       10.45 ± 2%
>>>> perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
>>>>        7.55            +2.9       10.46 ± 2%
>>>> perf-profile.calltrace.cycles-pp.ret_from_fork
>>>>        7.55            +2.9       10.46 ± 2%
>>>> perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
>>>>        0.00            +3.3        3.30 ± 9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
>>>>
>>>>        0.00            +3.3        3.32 ± 8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
>>>>
>>>>        8.56 ± 3%      -4.3        4.25
>>>> perf-profile.children.cycles-pp.__filemap_fdatawrite_range
>>>>        8.47 ± 3%      -4.3        4.18
>>>> perf-profile.children.cycles-pp.xfs_vm_writepages
>>>>        8.50 ± 3%      -4.3        4.20
>>>> perf-profile.children.cycles-pp.do_writepages
>>>>       78.64            -4.1       74.51
>>>> perf-profile.children.cycles-pp.write
>>>>       78.55            -4.1       74.44
>>>> perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
>>>>       78.50            -4.1       74.41
>>>> perf-profile.children.cycles-pp.do_syscall_64
>>>>        8.75 ± 3%      -4.1        4.70
>>>> perf-profile.children.cycles-pp.file_write_and_wait_range
>>>>       77.14            -3.8       73.37
>>>> perf-profile.children.cycles-pp.ksys_write
>>>>       77.09            -3.8       73.33
>>>> perf-profile.children.cycles-pp.vfs_write
>>>>       76.62            -3.7       72.97
>>>> perf-profile.children.cycles-pp.new_sync_write
>>>>       76.44            -3.6       72.81
>>>> perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
>>>>       69.23            -2.6       66.62
>>>> perf-profile.children.cycles-pp.xfs_file_fsync
>>>>        4.70 ± 2%      -2.2        2.50
>>>> perf-profile.children.cycles-pp.iomap_submit_ioend
>>>>        4.89 ± 2%      -2.1        2.75
>>>> perf-profile.children.cycles-pp.submit_bio
>>>>        4.85 ± 2%      -2.1        2.71
>>>> perf-profile.children.cycles-pp.generic_make_request
>>>>        3.73 ± 4%      -2.1        1.63
>>>> perf-profile.children.cycles-pp.iomap_writepages
>>>>        3.72 ± 4%      -2.1        1.63
>>>> perf-profile.children.cycles-pp.write_cache_pages
>>>>        4.43 ± 2%      -2.1        2.38
>>>> perf-profile.children.cycles-pp.brd_make_request
>>>>        7.73            -2.0        5.75 ± 2%
>>>> perf-profile.children.cycles-pp.__flush_work
>>>>        3.08 ± 4%      -1.9        1.14
>>>> perf-profile.children.cycles-pp.iomap_writepage_map
>>>>        6.00            -1.8        4.20 ± 2%
>>>> perf-profile.children.cycles-pp._raw_spin_lock_irq
>>>>        2.50 ± 4%      -1.8        0.70 ± 3%
>>>> perf-profile.children.cycles-pp.end_page_writeback
>>>>        2.50 ± 5%      -1.8        0.70 ± 4%
>>>> perf-profile.children.cycles-pp.test_clear_page_writeback
>>>>        2.42 ± 6%      -1.8        0.61 ± 2%
>>>> perf-profile.children.cycles-pp.__test_set_page_writeback
>>>>        2.08 ± 7%      -1.6        0.44 ± 4%
>>>> perf-profile.children.cycles-pp.sb_mark_inode_writeback
>>>>        2.08 ± 6%      -1.6        0.44 ± 5%
>>>> perf-profile.children.cycles-pp.sb_clear_inode_writeback
>>>>        2.70 ± 4%      -1.0        1.67 ± 2%
>>>> perf-profile.children.cycles-pp.iomap_finish_ioend
>>>>        2.36            -0.6        1.79
>>>> perf-profile.children.cycles-pp.iomap_file_buffered_write
>>>>        2.35            -0.6        1.78
>>>> perf-profile.children.cycles-pp.iomap_apply
>>>>        1.84            -0.5        1.33 ± 2%
>>>> perf-profile.children.cycles-pp.iomap_write_actor
>>>>        2.41 ± 4%      -0.5        1.94 ± 2%
>>>> perf-profile.children.cycles-pp.reschedule_interrupt
>>>>        3.44 ± 2%      -0.5        2.98
>>>> perf-profile.children.cycles-pp.sched_ttwu_pending
>>>>        2.87            -0.5        2.42
>>>> perf-profile.children.cycles-pp.complete
>>>>        2.83            -0.4        2.38
>>>> perf-profile.children.cycles-pp.swake_up_locked
>>>>        2.28 ± 4%      -0.4        1.84 ± 2%
>>>> perf-profile.children.cycles-pp.scheduler_ipi
>>>>        1.56            -0.4        1.17 ± 2%
>>>> perf-profile.children.cycles-pp.brd_do_bvec
>>>>        1.89            -0.4        1.54
>>>> perf-profile.children.cycles-pp.wait_for_completion
>>>>        1.71            -0.3        1.38
>>>> perf-profile.children.cycles-pp.schedule_timeout
>>>>        0.88            -0.2        0.68 ± 2%
>>>> perf-profile.children.cycles-pp.rwsem_down_read_slowpath
>>>>        0.69            -0.2        0.51 ± 2%
>>>> perf-profile.children.cycles-pp.iomap_write_begin
>>>>        0.67 ± 2%      -0.2        0.49
>>>> perf-profile.children.cycles-pp.lseek64
>>>>        1.33 ± 5%      -0.2        1.17 ± 3%
>>>> perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
>>>>        0.51            -0.2        0.36
>>>> perf-profile.children.cycles-pp.grab_cache_page_write_begin
>>>>        0.47            -0.1        0.33
>>>> perf-profile.children.cycles-pp.pagecache_get_page
>>>>        0.51 ± 3%      -0.1        0.38 ± 2%
>>>> perf-profile.children.cycles-pp.iomap_write_end
>>>>        0.57 ± 2%      -0.1        0.46
>>>> perf-profile.children.cycles-pp.blkdev_issue_flush
>>>>        0.38            -0.1        0.26 ± 5%
>>>> perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
>>>>        0.34            -0.1        0.23 ± 6%
>>>> perf-profile.children.cycles-pp.copyin
>>>>        0.33            -0.1        0.22 ± 6%
>>>> perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
>>>>        0.65 ± 2%      -0.1        0.56 ± 2%
>>>> perf-profile.children.cycles-pp.rwsem_optimistic_spin
>>>>        0.37            -0.1        0.28 ± 2%
>>>> perf-profile.children.cycles-pp.bio_alloc_bioset
>>>>        0.31 ± 4%      -0.1        0.22
>>>> perf-profile.children.cycles-pp.syscall_return_via_sysret
>>>>        0.56 ± 3%      -0.1        0.48 ± 3%
>>>> perf-profile.children.cycles-pp.osq_lock
>>>>        0.26 ± 3%      -0.1        0.18 ± 4%
>>>> perf-profile.children.cycles-pp.entry_SYSCALL_64
>>>>        0.28 ± 2%      -0.1        0.21 ± 5%
>>>> perf-profile.children.cycles-pp.mempool_alloc
>>>>        0.26 ± 6%      -0.1        0.18 ± 8%
>>>> perf-profile.children.cycles-pp.xlog_state_get_iclog_space
>>>>        0.29            -0.1        0.22 ± 3%
>>>> perf-profile.children.cycles-pp.find_get_entry
>>>>        0.34 ± 2%      -0.1        0.27 ± 4%
>>>> perf-profile.children.cycles-pp.generic_make_request_checks
>>>>        0.30 ± 5%      -0.1        0.23 ± 3%
>>>> perf-profile.children.cycles-pp.iomap_set_page_dirty
>>>>        0.32 ± 2%      -0.1        0.26 ± 5%
>>>> perf-profile.children.cycles-pp.brd_lookup_page
>>>>        0.18 ± 2%      -0.1        0.12 ± 6%
>>>> perf-profile.children.cycles-pp.iomap_set_range_uptodate
>>>>        0.15 ± 2%      -0.1        0.09 ± 4%
>>>> perf-profile.children.cycles-pp.__mod_lruvec_state
>>>>        0.15 ± 5%      -0.1        0.10 ± 9%
>>>> perf-profile.children.cycles-pp.wake_up_q
>>>>        0.21 ± 5%      -0.1        0.15 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_ilock
>>>>        0.13 ± 3%      -0.0        0.08 ± 10%
>>>> perf-profile.children.cycles-pp.rwsem_wake
>>>>        0.20 ± 2%      -0.0        0.15 ± 5%
>>>> perf-profile.children.cycles-pp.ksys_lseek
>>>>        0.20 ± 3%      -0.0        0.15 ± 3%
>>>> perf-profile.children.cycles-pp.xas_load
>>>>        0.20 ± 5%      -0.0        0.16 ± 2%
>>>> perf-profile.children.cycles-pp.__set_page_dirty
>>>>        0.19 ± 2%      -0.0        0.15 ± 2%
>>>> perf-profile.children.cycles-pp.xfs_iunlock
>>>>        0.40            -0.0        0.36 ± 4%
>>>> perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
>>>>        0.17 ± 2%      -0.0        0.12 ± 4%
>>>> perf-profile.children.cycles-pp.clear_page_dirty_for_io
>>>>        0.20 ± 3%      -0.0        0.16 ± 4%
>>>> perf-profile.children.cycles-pp.submit_bio_wait
>>>>        0.18 ± 4%      -0.0        0.15 ± 7%
>>>> perf-profile.children.cycles-pp.security_file_permission
>>>>        0.10            -0.0        0.06 ± 6%
>>>> perf-profile.children.cycles-pp.evict
>>>>        0.10            -0.0        0.06 ± 6%
>>>> perf-profile.children.cycles-pp.truncate_inode_pages_range
>>>>        0.06 ± 6%      -0.0        0.03 ±100%
>>>> perf-profile.children.cycles-pp.vfs_unlink
>>>>        0.08 ± 6%      -0.0        0.04 ± 58%
>>>> perf-profile.children.cycles-pp.add_to_page_cache_lru
>>>>        0.26 ± 2%      -0.0        0.23 ± 3%
>>>> perf-profile.children.cycles-pp.reweight_entity
>>>>        0.12 ± 6%      -0.0        0.09 ± 4%
>>>> perf-profile.children.cycles-pp.bio_endio
>>>>        0.10 ± 7%      -0.0        0.07 ± 7%
>>>> perf-profile.children.cycles-pp.do_sys_open
>>>>        0.10 ± 7%      -0.0        0.07 ± 7%
>>>> perf-profile.children.cycles-pp.do_sys_openat2
>>>>        0.17 ± 2%      -0.0        0.14
>>>> perf-profile.children.cycles-pp.__pagevec_release
>>>>        0.13 ± 5%      -0.0        0.10 ± 4%
>>>> perf-profile.children.cycles-pp.down_write
>>>>        0.10 ± 5%      -0.0        0.06 ± 6%
>>>> perf-profile.children.cycles-pp.do_filp_open
>>>>        0.15 ± 4%      -0.0        0.12 ± 5%
>>>> perf-profile.children.cycles-pp.tag_pages_for_writeback
>>>>        0.11 ± 9%      -0.0        0.08 ± 5%
>>>> perf-profile.children.cycles-pp.bio_associate_blkg
>>>>        0.10 ± 5%      -0.0        0.07 ± 7%
>>>> perf-profile.children.cycles-pp.open64
>>>>        0.09 ± 4%      -0.0        0.06 ± 6%
>>>> perf-profile.children.cycles-pp.path_openat
>>>>        0.16 ± 8%      -0.0        0.13 ± 3%
>>>> perf-profile.children.cycles-pp.pagevec_lookup_range_tag
>>>>        0.12 ± 8%      -0.0        0.10 ± 5%
>>>> perf-profile.children.cycles-pp.account_page_dirtied
>>>>        0.10 ± 4%      -0.0        0.08 ± 6%
>>>> perf-profile.children.cycles-pp.__fdget_pos
>>>>        0.14 ± 3%      -0.0        0.11 ± 7%
>>>> perf-profile.children.cycles-pp.memset_erms
>>>>        0.11 ± 3%      -0.0        0.09 ± 4%
>>>> perf-profile.children.cycles-pp.switch_fpu_return
>>>>        0.23            -0.0        0.21 ± 3%
>>>> perf-profile.children.cycles-pp.__radix_tree_lookup
>>>>        0.18 ± 2%      -0.0        0.16 ± 9%
>>>> perf-profile.children.cycles-pp.insert_work
>>>>        0.12 ± 4%      -0.0        0.10 ± 7%
>>>> perf-profile.children.cycles-pp.common_file_perm
>>>>        0.09 ± 4%      -0.0        0.07 ± 6%
>>>> perf-profile.children.cycles-pp.__fget_light
>>>>        0.09 ± 4%      -0.0        0.07 ± 6%
>>>> perf-profile.children.cycles-pp.bvec_alloc
>>>>        0.15 ± 7%      -0.0        0.12 ± 3%
>>>> perf-profile.children.cycles-pp.find_get_pages_range_tag
>>>>        0.08 ± 8%      -0.0        0.06 ± 7%
>>>> perf-profile.children.cycles-pp.iov_iter_fault_in_readable
>>>>        0.09 ± 5%      -0.0        0.06 ± 13%
>>>> perf-profile.children.cycles-pp.blk_throtl_bio
>>>>        0.24 ± 4%      -0.0        0.22 ± 3%
>>>> perf-profile.children.cycles-pp._cond_resched
>>>>        0.13 ± 3%      -0.0        0.11 ± 4%
>>>> perf-profile.children.cycles-pp.release_pages
>>>>        0.10 ± 4%      -0.0        0.08 ± 6%
>>>> perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
>>>>        0.08 ± 10%      -0.0        0.06
>>>> perf-profile.children.cycles-pp.unlink
>>>>        0.08 ± 10%      -0.0        0.06 ± 11%
>>>> perf-profile.children.cycles-pp.__sb_start_write
>>>>        0.08 ± 5%      -0.0        0.06 ± 7%
>>>> perf-profile.children.cycles-pp.generic_write_checks
>>>>        0.08 ± 8%      -0.0        0.06
>>>> perf-profile.children.cycles-pp.do_unlinkat
>>>>        0.07 ± 10%      -0.0        0.05
>>>> perf-profile.children.cycles-pp.xfs_break_layouts
>>>>        0.18 ± 2%      -0.0        0.16 ± 2%
>>>> perf-profile.children.cycles-pp.__list_add_valid
>>>>        0.12 ± 3%      -0.0        0.10 ± 8%
>>>> perf-profile.children.cycles-pp.__might_sleep
>>>>        0.14 ± 3%      -0.0        0.12 ± 7%
>>>> perf-profile.children.cycles-pp.xas_find_marked
>>>>        0.07 ± 10%      -0.0        0.05 ± 8%
>>>> perf-profile.children.cycles-pp.bio_associate_blkg_from_css
>>>>        0.08 ± 5%      -0.0        0.06 ± 11%
>>>> perf-profile.children.cycles-pp.bio_free
>>>>        0.08 ± 6%      -0.0        0.06
>>>> perf-profile.children.cycles-pp.xfs_file_llseek
>>>>        0.08 ± 5%      -0.0        0.06 ± 6%
>>>> perf-profile.children.cycles-pp.xfs_file_write_iter
>>>>        0.16 ± 2%      +0.0        0.17 ± 2%
>>>> perf-profile.children.cycles-pp.__switch_to_asm
>>>>        0.05            +0.0        0.07 ± 7%
>>>> perf-profile.children.cycles-pp.cpumask_next_wrap
>>>>        0.10 ± 4%      +0.0        0.12 ± 3%
>>>> perf-profile.children.cycles-pp._find_next_bit
>>>>        0.07 ± 7%      +0.0        0.09 ± 5%
>>>> perf-profile.children.cycles-pp.xlog_state_done_syncing
>>>>        0.16 ± 2%      +0.0        0.19 ± 2%
>>>> perf-profile.children.cycles-pp.check_preempt_curr
>>>>        0.19 ± 4%      +0.0        0.22
>>>> perf-profile.children.cycles-pp.ttwu_do_wakeup
>>>>        0.56 ± 3%      +0.0        0.60
>>>> perf-profile.children.cycles-pp.dequeue_entity
>>>>        0.20 ± 6%      +0.0        0.23 ± 4%
>>>> perf-profile.children.cycles-pp.xfs_trans_committed_bulk
>>>>        0.07 ± 7%      +0.0        0.10 ± 7%
>>>> perf-profile.children.cycles-pp.check_preempt_wakeup
>>>>        0.82 ± 2%      +0.0        0.86
>>>> perf-profile.children.cycles-pp.update_load_avg
>>>>        0.31 ± 3%      +0.0        0.35
>>>> perf-profile.children.cycles-pp.__unwind_start
>>>>        0.08 ± 8%      +0.0        0.12 ± 3%
>>>> perf-profile.children.cycles-pp.destroy_inode
>>>>        0.08 ± 10%      +0.0        0.12 ± 3%
>>>> perf-profile.children.cycles-pp.xfs_fs_destroy_inode
>>>>        0.08 ± 10%      +0.0        0.12 ± 3%
>>>> perf-profile.children.cycles-pp.xfs_inactive
>>>>        0.08 ± 15%      +0.0        0.12 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_bmap_validate_extent
>>>>        3.96            +0.0        4.01
>>>> perf-profile.children.cycles-pp.stack_trace_save_tsk
>>>>        0.28            +0.0        0.33
>>>> perf-profile.children.cycles-pp.set_next_entity
>>>>        0.00            +0.1        0.05
>>>> perf-profile.children.cycles-pp.xfs_trans_roll
>>>>        0.00            +0.1        0.05 ± 8%
>>>> perf-profile.children.cycles-pp.xfs_btree_delrec
>>>>        0.00            +0.1        0.05 ± 8%
>>>> perf-profile.children.cycles-pp.xfs_btree_insert
>>>>        0.16 ± 4%      +0.1        0.21 ± 5%
>>>> perf-profile.children.cycles-pp.memcpy_erms
>>>>        0.04 ± 57%      +0.1        0.10 ± 4%
>>>> perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
>>>>        0.00            +0.1        0.06 ± 11%
>>>> perf-profile.children.cycles-pp.xfs_buf_item_format_segment
>>>>        0.00            +0.1        0.06 ± 11%
>>>> perf-profile.children.cycles-pp.xfs_btree_delete
>>>>        0.00            +0.1        0.06
>>>> perf-profile.children.cycles-pp.xfs_btree_update
>>>>        0.00            +0.1        0.07 ± 7%
>>>> perf-profile.children.cycles-pp.xfs_cil_prepare_item
>>>>        0.00            +0.1        0.07 ± 6%
>>>> perf-profile.children.cycles-pp.xfs_bmbt_update
>>>>        0.00            +0.1        0.07 ± 10%
>>>> perf-profile.children.cycles-pp.xfs_buf_item_format
>>>>        0.56            +0.1        0.64 ± 2%
>>>> perf-profile.children.cycles-pp.pick_next_task_fair
>>>>        0.03 ±100%      +0.1        0.10 ± 4%
>>>> perf-profile.children.cycles-pp.xfs_inactive_truncate
>>>>        0.08 ± 5%      +0.1        0.16 ± 11%
>>>> perf-profile.children.cycles-pp.xfs_iext_lookup_extent
>>>>        0.00            +0.1        0.08 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_defer_finish
>>>>        0.00            +0.1        0.08 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
>>>>        0.00            +0.1        0.08 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_verify_agbno
>>>>        0.00            +0.1        0.09 ± 9%
>>>> perf-profile.children.cycles-pp.xfs_defer_finish_noroll
>>>>        0.60 ± 3%      +0.1        0.70
>>>> perf-profile.children.cycles-pp.schedule_idle
>>>>        0.39 ± 2%      +0.1        0.49
>>>> perf-profile.children.cycles-pp.xlog_state_release_iclog
>>>>        0.00            +0.1        0.10 ± 9%
>>>> perf-profile.children.cycles-pp.xfs_buf_find
>>>>        0.93 ± 3%      +0.1        1.03 ± 2%
>>>> perf-profile.children.cycles-pp.xlog_write
>>>>        2.58            +0.1        2.69
>>>> perf-profile.children.cycles-pp.schedule
>>>>        1.52            +0.1        1.63
>>>> perf-profile.children.cycles-pp.__wake_up_common_lock
>>>>        0.00            +0.1        0.11 ± 7%
>>>> perf-profile.children.cycles-pp.xfs_buf_get_map
>>>>        4.62            +0.1        4.73
>>>> perf-profile.children.cycles-pp.__account_scheduler_latency
>>>>        0.00            +0.1        0.12 ± 10%
>>>> perf-profile.children.cycles-pp.xfs_buf_read_map
>>>>        1.83            +0.2        1.99
>>>> perf-profile.children.cycles-pp.xlog_cil_push_work
>>>>        5.94            +0.2        6.12
>>>> perf-profile.children.cycles-pp.ttwu_do_activate
>>>>        0.00            +0.2        0.17 ± 6%
>>>> perf-profile.children.cycles-pp.xfs_btree_read_buf_block
>>>>        5.93            +0.2        6.11
>>>> perf-profile.children.cycles-pp.activate_task
>>>>        5.88            +0.2        6.06
>>>> perf-profile.children.cycles-pp.enqueue_task_fair
>>>>        0.00            +0.2        0.18 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_trans_read_buf_map
>>>>        5.41            +0.2        5.59
>>>> perf-profile.children.cycles-pp.enqueue_entity
>>>>        3.14            +0.2        3.33
>>>> perf-profile.children.cycles-pp.__sched_text_start
>>>>        0.00            +0.2        0.21 ± 6%
>>>> perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
>>>>        0.00            +0.2        0.23 ± 57%
>>>> perf-profile.children.cycles-pp.xlog_grant_head_wait
>>>>        0.00            +0.2        0.24 ± 6%
>>>> perf-profile.children.cycles-pp.io_schedule
>>>>        0.11 ± 3%      +0.3        0.38 ± 4%
>>>> perf-profile.children.cycles-pp.__filemap_fdatawait_range
>>>>        0.00            +0.3        0.27 ± 8%
>>>> perf-profile.children.cycles-pp.wait_on_page_bit
>>>>        0.00            +0.3        0.34 ± 5%
>>>> perf-profile.children.cycles-pp.xfs_btree_lookup
>>>>        0.08 ± 14%      +0.4        0.43
>>>> perf-profile.children.cycles-pp.__queue_work
>>>>        0.08 ± 8%      +0.4        0.44
>>>> perf-profile.children.cycles-pp.queue_work_on
>>>>        0.00            +0.4        0.41
>>>> perf-profile.children.cycles-pp.xfs_end_bio
>>>>        3.51            +0.4        3.94 ± 9%
>>>> perf-profile.children.cycles-pp.__xfs_trans_commit
>>>>        0.20 ± 9%      +0.5        0.72 ± 54%
>>>> perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
>>>>        0.06 ± 11%      +0.5        0.58 ± 67%
>>>> perf-profile.children.cycles-pp.xfs_log_space_wake
>>>>        0.45 ± 5%      +0.6        1.04 ± 36%
>>>> perf-profile.children.cycles-pp.xfs_log_reserve
>>>>        0.65 ± 3%      +0.6        1.25 ± 29%
>>>> perf-profile.children.cycles-pp.xfs_trans_alloc
>>>>        0.48 ± 5%      +0.6        1.09 ± 34%
>>>> perf-profile.children.cycles-pp.xfs_trans_reserve
>>>>        0.04 ± 57%      +0.6        0.65 ± 60%
>>>> perf-profile.children.cycles-pp.xlog_grant_head_check
>>>>        4.83            +0.7        5.52
>>>> perf-profile.children.cycles-pp.try_to_wake_up
>>>>        0.00            +0.7        0.70 ± 2%
>>>> perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
>>>>        0.00            +0.8        0.75 ± 4%
>>>> perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
>>>>        0.04 ± 57%      +0.8        0.80
>>>> perf-profile.children.cycles-pp.autoremove_wake_function
>>>>        0.05 ± 8%      +0.8        0.85
>>>> perf-profile.children.cycles-pp.wake_up_page_bit
>>>>        1.50            +0.9        2.36
>>>> perf-profile.children.cycles-pp.__wake_up_common
>>>>        0.00            +0.9        0.89 ± 2%
>>>> perf-profile.children.cycles-pp.xfs_bmapi_write
>>>>       24.29            +0.9       25.18
>>>> perf-profile.children.cycles-pp._raw_spin_lock_irqsave
>>>>        0.07 ± 5%      +1.0        1.05
>>>> perf-profile.children.cycles-pp.iomap_finish_ioends
>>>>       35.68            +1.0       36.69
>>>> perf-profile.children.cycles-pp.xlog_cil_force_lsn
>>>>        9.53            +1.2       10.76
>>>> perf-profile.children.cycles-pp.intel_idle
>>>>        9.81            +1.3       11.10
>>>> perf-profile.children.cycles-pp.cpuidle_enter_state
>>>>        9.82            +1.3       11.11
>>>> perf-profile.children.cycles-pp.cpuidle_enter
>>>>       12.25            +1.3       13.60
>>>> perf-profile.children.cycles-pp.start_secondary
>>>>       12.49            +1.4       13.89
>>>> perf-profile.children.cycles-pp.secondary_startup_64
>>>>       12.49            +1.4       13.89
>>>> perf-profile.children.cycles-pp.cpu_startup_entry
>>>>       12.48            +1.4       13.88
>>>> perf-profile.children.cycles-pp.do_idle
>>>>       14.78            +1.4       16.20 ± 2%
>>>> perf-profile.children.cycles-pp.xlog_wait_on_iclog
>>>>       58.91            +1.7       60.57
>>>> perf-profile.children.cycles-pp.xfs_log_force_lsn
>>>>        0.00            +2.2        2.24 ± 13%
>>>> perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
>>>>        7.05            +2.8        9.81 ± 2%
>>>> perf-profile.children.cycles-pp.process_one_work
>>>>        7.54            +2.9       10.45 ± 2%
>>>> perf-profile.children.cycles-pp.worker_thread
>>>>        7.55            +2.9       10.46 ± 2%
>>>> perf-profile.children.cycles-pp.ret_from_fork
>>>>        7.55            +2.9       10.46 ± 2%
>>>> perf-profile.children.cycles-pp.kthread
>>>>        0.10            +3.2        3.30 ± 9%
>>>> perf-profile.children.cycles-pp.xfs_end_ioend
>>>>        0.10 ± 4%      +3.2        3.32 ± 8%
>>>> perf-profile.children.cycles-pp.xfs_end_io
>>>>       19.57            +4.0       23.62
>>>> perf-profile.children.cycles-pp.remove_wait_queue
>>>>        1.20            -0.3        0.90 ± 2%
>>>> perf-profile.self.cycles-pp.brd_do_bvec
>>>>        0.55 ± 3%      -0.2        0.39 ± 4%
>>>> perf-profile.self.cycles-pp.do_syscall_64
>>>>        0.31            -0.1        0.22 ± 5%
>>>> perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
>>>>        0.31 ± 3%      -0.1        0.22
>>>> perf-profile.self.cycles-pp.syscall_return_via_sysret
>>>>        0.54 ± 3%      -0.1        0.47 ± 3%
>>>> perf-profile.self.cycles-pp.osq_lock
>>>>        0.23 ± 3%      -0.1        0.16 ± 5%
>>>> perf-profile.self.cycles-pp.entry_SYSCALL_64
>>>>        0.10 ± 4%      -0.1        0.04 ± 57%
>>>> perf-profile.self.cycles-pp.__mod_lruvec_state
>>>>        0.18 ± 4%      -0.1        0.12 ± 5%
>>>> perf-profile.self.cycles-pp.iomap_set_range_uptodate
>>>>        0.60 ± 4%      -0.1        0.54 ± 2%
>>>> perf-profile.self.cycles-pp._raw_spin_lock
>>>>        0.77            -0.0        0.72
>>>> perf-profile.self.cycles-pp._raw_spin_lock_irqsave
>>>>        0.14            -0.0        0.10 ± 8%
>>>> perf-profile.self.cycles-pp.find_get_entry
>>>>        0.14 ± 3%      -0.0        0.11 ± 4%
>>>> perf-profile.self.cycles-pp.xas_load
>>>>        0.17 ± 4%      -0.0        0.14 ± 3%
>>>> perf-profile.self.cycles-pp.write
>>>>        0.30 ± 2%      -0.0        0.27 ± 6%
>>>> perf-profile.self.cycles-pp._raw_spin_lock_irq
>>>>        0.18 ± 5%      -0.0        0.15 ± 7%
>>>> perf-profile.self.cycles-pp.__flush_work
>>>>        0.08 ± 5%      -0.0        0.05
>>>> perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
>>>>        0.12 ± 3%      -0.0        0.09 ± 4%
>>>> perf-profile.self.cycles-pp.brd_lookup_page
>>>>        0.12 ± 7%      -0.0        0.09 ± 9%
>>>> perf-profile.self.cycles-pp.vfs_write
>>>>        0.18 ± 4%      -0.0        0.15 ± 7%
>>>> perf-profile.self.cycles-pp.insert_work
>>>>        0.13            -0.0        0.10 ± 8%
>>>> perf-profile.self.cycles-pp.memset_erms
>>>>        0.11 ± 4%      -0.0        0.08
>>>> perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
>>>>        0.11 ± 6%      -0.0        0.08 ± 5%
>>>> perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
>>>>        0.12 ± 3%      -0.0        0.09 ± 4%
>>>> perf-profile.self.cycles-pp.iomap_writepage_map
>>>>        0.08 ± 8%      -0.0        0.06 ± 15%
>>>> perf-profile.self.cycles-pp.bio_alloc_bioset
>>>>        0.06 ± 6%      -0.0        0.04 ± 57%
>>>> perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
>>>>        0.09 ± 5%      -0.0        0.06 ± 6%
>>>> perf-profile.self.cycles-pp.__fget_light
>>>>        0.06            -0.0        0.04 ± 57%
>>>> perf-profile.self.cycles-pp.xfs_map_blocks
>>>>        0.18 ± 2%      -0.0        0.15 ± 3%
>>>> perf-profile.self.cycles-pp.__list_add_valid
>>>>        0.08 ± 6%      -0.0        0.06 ± 9%
>>>> perf-profile.self.cycles-pp.xfs_ilock
>>>>        0.08 ± 6%      -0.0        0.06 ± 9%
>>>> perf-profile.self.cycles-pp.iov_iter_fault_in_readable
>>>>        0.11 ± 4%      -0.0        0.09 ± 4%
>>>> perf-profile.self.cycles-pp.switch_fpu_return
>>>>        0.07            -0.0        0.05
>>>> perf-profile.self.cycles-pp.iomap_write_actor
>>>>        0.23 ± 2%      -0.0        0.21 ± 5%
>>>> perf-profile.self.cycles-pp.__radix_tree_lookup
>>>>        0.18 ± 4%      -0.0        0.16 ± 6%
>>>> perf-profile.self.cycles-pp.xlog_cil_force_lsn
>>>>        0.20 ± 2%      -0.0        0.18 ± 4%
>>>> perf-profile.self.cycles-pp.kmem_cache_alloc
>>>>        0.10 ± 5%      -0.0        0.08 ± 11%
>>>> perf-profile.self.cycles-pp.xfs_file_fsync
>>>>        0.07 �� 5%      -0.0        0.05 ± 9%
>>>> perf-profile.self.cycles-pp.clear_page_dirty_for_io
>>>>        0.07 ± 6%      -0.0        0.05
>>>> perf-profile.self.cycles-pp.__test_set_page_writeback
>>>>        0.11            -0.0        0.09 ± 8%
>>>> perf-profile.self.cycles-pp.common_file_perm
>>>>        0.23            -0.0        0.21 ± 4%
>>>> perf-profile.self.cycles-pp.update_rq_clock
>>>>        0.08 ± 6%      -0.0        0.06
>>>> perf-profile.self.cycles-pp.xfs_file_write_iter
>>>>        0.10 ± 5%      -0.0        0.08 ± 8%
>>>> perf-profile.self.cycles-pp.up_read
>>>>        0.08 ± 5%      -0.0        0.07
>>>> perf-profile.self.cycles-pp.write_cache_pages
>>>>        0.10            -0.0        0.09
>>>> perf-profile.self.cycles-pp.reweight_entity
>>>>        0.05 ± 8%      +0.0        0.07 ± 7%
>>>> perf-profile.self.cycles-pp.dequeue_entity
>>>>        0.16 ± 4%      +0.0        0.17 ± 2%
>>>> perf-profile.self.cycles-pp.__switch_to_asm
>>>>        0.11 ± 4%      +0.0        0.12 ± 3%
>>>> perf-profile.self.cycles-pp.set_next_entity
>>>>        0.15 ± 3%      +0.0        0.16 ± 2%
>>>> perf-profile.self.cycles-pp.orc_find
>>>>        0.09 ± 4%      +0.0        0.11 ± 6%
>>>> perf-profile.self.cycles-pp.enqueue_entity
>>>>        0.10 ± 4%      +0.0        0.12 ± 3%
>>>> perf-profile.self.cycles-pp._find_next_bit
>>>>        0.01 ±173%      +0.0        0.06 ± 7%
>>>> perf-profile.self.cycles-pp.check_preempt_wakeup
>>>>        0.01 ±173%      +0.0        0.06 ± 11%
>>>> perf-profile.self.cycles-pp.xfs_trans_committed_bulk
>>>>        0.45 ± 2%      +0.1        0.51 ± 2%
>>>> perf-profile.self.cycles-pp.__sched_text_start
>>>>        0.15 ± 6%      +0.1        0.21 ± 3%
>>>> perf-profile.self.cycles-pp.memcpy_erms
>>>>        0.00            +0.1        0.06 ± 6%
>>>> perf-profile.self.cycles-pp.xfs_verify_agbno
>>>>        0.08 ± 8%      +0.1        0.16 ± 9%
>>>> perf-profile.self.cycles-pp.xfs_iext_lookup_extent
>>>>        9.53            +1.2       10.76
>>>> perf-profile.self.cycles-pp.intel_idle
>>>>
>>>>
>>>>                                    aim7.jobs-per-min
>>>>    45000
>>>> +-------------------------------------------------------------------+
>>>>          |        +
>>>> +                                                       |
>>>>    40000 |.+.+    :   + .+.   +.
>>>> .+.     |
>>>>    35000 |-+ :    :    +   +.+
>>>> +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+   ++.+.|
>>>>          |   :
>>>> :                                                          |
>>>>    30000 |-+ :
>>>> :                                                           |
>>>>    25000 |-O :O O:O
>>>> OO                                                     |
>>>>          |   :   : O    O O OO O O O
>>>> O                                      |
>>>>    20000 |-+ :
>>>> :                                                           |
>>>>    15000 |-+ :
>>>> :                                                           |
>>>>          |    :
>>>> :                                                           |
>>>>    10000 |-+ :
>>>> :                                                            |
>>>>     5000 |-+ :
>>>> :                                                            |
>>>>          |    :
>>>> :                                                            |
>>>>        0
>>>> +-------------------------------------------------------------------+
>>>>                                 aim7.time.system_time
>>>>    5000
>>>> +--------------------------------------------------------------------+
>>>>    4500 |-+        O   OO   O    O O
>>>> O                                       |
>>>>         | O OO O O   O    O   O
>>>> O                                            |
>>>>    4000
>>>> |-+
>>>> |
>>>>    3500
>>>> |-+
>>>> |
>>>>         |             .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+
>>>> .+.+.+. .++.+.|
>>>>    3000 |.+.+       .+ +     +                               +
>>>> +      |
>>>>    2500 |-+ :
>>>> +.+                                                         |
>>>>    2000 |-+ :
>>>> :                                                           |
>>>>         |   :
>>>> :                                                            |
>>>>    1500 |-+ :
>>>> :                                                            |
>>>>    1000 |-+ :
>>>> :                                                            |
>>>>         |    :
>>>> :                                                            |
>>>>     500 |-+ :
>>>> :                                                             |
>>>>       0
>>>> +--------------------------------------------------------------------+
>>>>                                aim7.time.elapsed_time
>>>>    160
>>>> +---------------------------------------------------------------------+
>>>>        |   O OO O O O O OO O O O
>>>> O                                         |
>>>>    140
>>>> |-O
>>>> |
>>>>    120
>>>> |-+
>>>> |
>>>> |                                                                     |
>>>>    100 |-+           .+. +.+.
>>>> .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
>>>>        |.+.+    +. .+   +    +
>>>> +      |
>>>>     80 |-+ :    :
>>>> +                                                          |
>>>>        |    :
>>>> :                                                            |
>>>>     60 |-+ :
>>>> :                                                             |
>>>>     40 |-+ :
>>>> :                                                             |
>>>>        |    :
>>>> :                                                             |
>>>>     20 |-+   :
>>>> :                                                             |
>>>>        |
>>>> ::                                                              |
>>>>      0
>>>> +---------------------------------------------------------------------+
>>>>                              aim7.time.elapsed_time.max
>>>>    160
>>>> +---------------------------------------------------------------------+
>>>>        |   O OO O O O O OO O O O
>>>> O                                         |
>>>>    140
>>>> |-O
>>>> |
>>>>    120
>>>> |-+
>>>> |
>>>> |                                                                     |
>>>>    100 |-+           .+. +.+.
>>>> .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
>>>>        |.+.+    +. .+   +    +
>>>> +      |
>>>>     80 |-+ :    :
>>>> +                                                          |
>>>>        |    :
>>>> :                                                            |
>>>>     60 |-+ :
>>>> :                                                             |
>>>>     40 |-+ :
>>>> :                                                             |
>>>>        |    :
>>>> :                                                             |
>>>>     20 |-+   :
>>>> :                                                             |
>>>>        |
>>>> ::                                                              |
>>>>      0
>>>> +---------------------------------------------------------------------+
>>>>                          aim7.time.voluntary_context_switches
>>>>    8e+07
>>>> +-------------------------------------------------------------------+
>>>>          | O          O O O OO O
>>>> O                                        |
>>>>    7e+07
>>>> |-+
>>>> |
>>>>    6e+07
>>>> |-+
>>>> |
>>>> |                                                                   |
>>>>    5e+07 |-+           +.+.+.++.+.+.+.++.+.+. .+   .+.+.    .+.
>>>> .+.   +.+.|
>>>>          |.+.+       .+                      + +.+     ++.+   ++.+
>>>> +.+    |
>>>>    4e+07 |-+ :
>>>> +.+                                                        |
>>>>          |   :
>>>> :                                                          |
>>>>    3e+07 |-+ :
>>>> :                                                           |
>>>>    2e+07 |-+ :
>>>> :                                                           |
>>>>          |    :
>>>> :                                                           |
>>>>    1e+07 |-+ :
>>>> :                                                           |
>>>>          |    :
>>>> :                                                            |
>>>>        0
>>>> +-------------------------------------------------------------------+
>>>> [*] bisect-good sample
>>>> [O] bisect-bad sample
>>>>
>>>>
>>>>
>>>> Disclaimer:
>>>> Results have been estimated based on internal Intel analysis and are
>>>> provided
>>>> for informational purposes only. Any difference in system hardware
>>>> or software
>>>> design or configuration may affect actual performance.
>>>>
>>>>
>>>> Thanks,
>>>> Rong Chen
>>>>
>>>>
>>>> _______________________________________________
>>>> LKP mailing list -- [email protected]
>>>> To unsubscribe send an email to [email protected]
>>>>
>>>
>>
>> --
>> Zhengjun Xing

--
Zhengjun Xing