LinuxLists.cc - [memcg, kmem] 58056f7750: hackbench.throughput 10.3% improvement

2021-11-24 08:34:49

Subject: [memcg, kmem] 58056f7750: hackbench.throughput 10.3% improvement

Greeting,

FYI, we noticed a

commit: 58056f77502f3567b76

Details are as below:
---------------------------

To reproduce:

git clone sudo bin/lkp install job.yaml bin/lkp split-job sudo bin/lkp
# if come # please remove
=========================== compiler/cpufreq_governor/i gcc-9/performance/socket/
commit:
16f6bf266c ("mm/list_lru.c: 58056f7750 ("memcg,
16f6bf266c94017c 58056f7750 ---------------- ---------- %stddev %change \ | 124164 +10.3% 553.20 553.20 3.519e+08 ? 4% 76944 -10.0% 1046 -15.9% 9.209e+08 ? 2% 516011 ? 17% -33.5% 260286 ? 8% -16.7% 607.74 164520 137719 0.42 ? 6% +0.1 2.966e+08 2746 -12.3% 2336697 ? 3% 534308 5546 ? 5% -12.6% 5546 ? 5% -12.6% 157208 3128 +11.3% 200236 +11.3% 3128 +11.3% 157390 3134 +11.2% 200626 +11.2% 3134 +11.2% 1.669e+10 0.80 -0.2 1.312e+08 27.80 +1.1 8.619e+08 2349252 ? 3% 143032 ? 2% -14.7% 2001 -3.5% 0.07 ? 5% -0.0 16549027 ? 4% 2.392e+10 0.03 ? 11% +0.0 4025883 ? 11% 56000903 23047129 ? 3% 8.356e+10 1548 +47.8% 0.18 -4.4% 869.08 384.93 7130 +3.7% 66.31 -6.3 39678105 20549470 43.76 -6.0 15613166 ? 2% 20301635 7172 +3.8% 0.79 -0.2 28.13 +0.9 5.65 +2.5% 1942 -2.3% 0.07 ? 5% -0.0 0.03 ? 11% +0.0 1500 +49.7% 0.18 -2.4% 64.79 -5.9 40.92 -5.0 1.641e+10 1.288e+08 2337457 ? 3% 130209 ? 2% -13.6% 16228057 ? 4% 3947999 ? 10% 54846930 22682918 ? 2% 8.23e+10 20.85 ? 3% +15.9% 5283 +10.5% 39315654 21363970 14602656 21089334 5304 +10.5% 4.56e+13 0.14 ?149% +6903.8% 146.03 ? 20% -59.7% 20.00 ? 16% -39.8% 45.20 ? 21% -45.1% 168.56 ? 19% -77.4% 87.74 ? 39% -69.7% 142.77 ? 20% -39.2% 22.06 ? 12% -51.5% 6.08 ?146% +12273.2% 22236 ? 14% -62.5% 26048 ? 9% -52.9% 24687 ? 8% -61.5% 24838 ? 8% -70.9% 21611 ? 13% -65.7% 26017 ? 10% -53.3% 26199 ? 9% -57.5% 26.44 ? 13% -36.5% 26690 ? 10% -52.4% 92.56 ? 13% -34.8% 53236 ? 9% -51.9% 66.12 ? 13% -34.1% 26766 ? 8% -47.8% 549.67 ? 18% -64.8% 75.13 ? 14% -41.6% 167.49 ? 17% -45.5% 594.39 ? 14% -76.9% 317.87 ? 21% -65.9% 1953 ?115% -90.1% 154.92 ? 33% -82.5% 7364 ? 51% -83.3% 532.84 ? 22% -42.2% 73.20 ? 12% -48.1% 131.17 ? 12% +54.1% 1020 ? 16% +30.8% 12199 ? 15% +39.9% 1423 ? 12% +44.5% 6.00 ? 19% +88.9% 48952 ? 19% +143.5% 240.00 ? 17% +52.6% 119.67 ? 28% +60.3% 45862 ? 11% -63.5% 52359 ? 9% -52.3% 49522 ? 8% -61.6% 49837 ? 8% -70.5% 43808 ? 12% -66.0% 36717 ? 23% -82.8% 31347 ? 40% -75.5% 52076 ? 10% -52.5% 52504 ? 9% -57.3% 1.82 ?163% +2252.9% 403.64 ? 24% -66.7% 55.12 ? 14% -42.3% 0.02 ?108% +534.1% 122.29 ? 16% -45.7% 425.82 ? 16% -76.7% 465.09 ?223% -100.0% 9.43 ?125% +219.2% 230.14 ? 17% -64.5% 70.21 ? 31% -56.3% 1362 ?123% -88.1% 0.17 ?118% +1194.7% 154.92 ? 33% -64.9% 5277 ? 37% -81.9% 390.06 ? 23% -43.2% 51.14 ? 12% -46.7% 42.41 ?143% +7008.4% 17958 ? 24% -60.2% 24588 ? 7% -65.6% 26488 ? 8% -51.3% 20113 ? 16% -45.9% 25666 ? 9% -57.7% 19730 ? 11% -59.6% 25151 ? 8% -64.3% 1860 ?223% -100.0% 4.30 ?194% -97.7% 320.22 ?175% +1509.4% 24087 ? 12% -61.8% 17911 ? 16% -62.7% 20724 ? 17% -79.9% 1.20 ?151% +2217.5% 19696 ? 23% -77.1% 26233 ? 10% -51.2% 26305 ? 9% -56.7% 24961 ? 11% -45.5% 25452 ? 10% -47.3% 84420 ? 6% -11.5% 86481 ? 5% -11.8% 87185 ? 5% -13.1% 92504 ? 6% -14.8% 92105 ? 6% -15.2% 92008 ? 6% -14.9% 92262 ? 6% -15.3% 92609 ? 6% -14.8% 92568 ? 6% -15.0% 93113 ? 6% -15.2% 94050 ? 6% -15.6% 86336 ? 6% -12.2% 85713 ? 6% -12.9% 85964 ? 5% -12.0% 85393 ? 7% -12.6% 85002 ? 6% -12.3% 95689 ? 6% -14.9% 95426 ? 6% -13.5% 93843 ? 6% -13.2% 94782 ? 6% -13.7% 95329 ? 6% -13.2% 95098 ? 6% -13.3% 94061 ? 6% -13.4% 94978 ? 6% -13.7% 85245 ? 6% -10.5% 94735 ? 6% -13.3% 95239 ? 6% -13.3% 94890 ? 6% -13.5% 94265 ? 6% -13.5% 95082 ? 6% -14.6% 95351 ? 6% -13.6% 81884 ? 5% -11.7% 81461 ? 6% -12.1% 88358 ? 5% -13.9% 88086 ? 5% -13.4% 14400 +9.9% 88026 ? 5% -13.4% 88351 ? 5% -13.7% 87490 ? 5% -13.3% 87529 ? 5% -13.6% 87798 ? 5% -12.5% 88136 ? 5% -13.5% 88294 ? 5% -13.6% 87208 ? 5% -13.5% 87695 ? 5% -13.3% 88320 ? 5% -13.6% 86555 ? 6% -12.3% 14298 +7.8% 88184 ? 5% -13.7% 87603 ? 5% -13.1% 87625 ? 5% -13.3% 88256 ? 5% -13.3% 87050 ? 5% -12.2% 81130 ? 6% -12.1% 82146 ? 6% -12.5% 14082 +10.2% 93814 ? 5% -15.8% 92946 ? 5% -15.5% 86506 ? 5% -9.5% 92407 ? 5% -15.2% 92212 ? 6% -15.7% 92309 ? 6% -15.1% 92929 ? 5% -15.4% 93052 ? 6% -15.7% 92080 ? 6% -15.3% 91828 ? 5% -15.3% 92903 ? 5% -15.6% 92628 ? 6% -15.3% 92249 ? 6% -16.2% 86877 ? 5% -12.0% 91350 ? 6% -14.7% 91443 ? 6% -16.0% 86014 ? 6% -14.2% 85380 ? 6% -13.7% 86155 ? 5% -14.3% 86462 ? 5% -14.1% 92387 ? 6% -13.1% 92117 ? 6% -12.9% 90761 ? 8% -11.9% 90705 ? 5% -11.5% 86170 ? 5% -11.2% 13627 ? 6% +14.0% 92368 ? 6% -13.7% 92055 ? 6% -13.0% 91447 ? 6% -13.0% 91880 ? 6% -13.2% 92254 ? 6% -12.7% 91783 ? 6% -12.8% 91421 ? 6% -12.9% 91390 ? 6% -12.8% 88932 ? 6% -12.8% 88850 ? 6% -12.4% 86897 ? 4% -12.3% 88709 ? 6% -12.5% 88614 ? 6% -12.5% 88876 ? 6% -13.3% 88779 ? 6% -12.4% 89712 ? 5% -13.5% 88569 ? 5% -13.2% 88777 ? 5% -13.1% 89211 ? 5% -13.8% 88989 ? 5% -13.0% 88940 ? 5% -13.1% 86373 ? 5% -12.3% 88320 ? 5% -13.0% 88405 ? 5% -13.2% 88722 ? 5% -12.8% 88541 ? 5% -12.9% 88636 ? 5% -13.6% 88387 ? 5% -13.4% 88139 ? 5% -13.3% 88420 ? 5% -13.3% 88484 ? 5% -13.2% 88366 ? 5% -12.7% 86334 ? 5% -12.1% 88438 ? 5% -13.4% 88125 ? 5% -13.3% 87488 ? 6% -12.1% 13664 ? 6% +12.1% 87669 ? 6% -11.6% 87675 ? 5% -11.5% 87948 ? 5% -12.0% 88247 ? 5% -12.2% 87612 ? 5% -11.7% 87923 ? 6% -12.4% 86391 ? 5% -11.4% 88597 ? 6% -12.0% 89101 ? 6% -12.5% 14183 ? 2% +8.3% 88545 ? 6% -12.4% 87757 ? 7% -11.5% 88307 ? 6% -11.5% 14211 +9.8% 89048 ? 6% -13.0% 89209 ? 5% -12.3% 88691 ? 5% -12.3% 88012 ? 5% -11.8% 88781 ? 5% -11.9% 86346 ? 6% -10.6% 90192 ? 5% -14.8% 89520 ? 6% -14.7% 89174 ? 5% -14.4% 88839 ? 6% -14.4% 89065 ? 5% -14.4% 89354 ? 6% -14.3% 92662 ? 6% -14.8% 91610 ? 6% -14.8% 92230 ? 6% -15.0% 92985 ? 6% -14.9% 12886523 ? 6% 112751 ? 3% -10.3% 6866613 ? 7% 1109865 -13.2% 746403 ? 6% -26.6% 1109844 -11.5% 784948 ? 6% -26.8% 1109908 -11.5% 768936 ? 6% -25.0% 1109673 -11.3% 933365 ? 9% -33.5% 1109648 -11.2% 905179 ? 4% -32.6% 55524 ? 32% -37.9% 1109644 -11.3% 919161 ? 4% -33.6% 1109645 -11.3% 913475 ? 5% -33.8% 1109644 -11.4% 891411 ? 4% -30.6% 1109685 -11.3% 916161 ? 6% -33.4% 1109651 -11.4% 940033 ? 4% -36.6% 45963 ? 35% -31.8% 1109713 -11.3% 934019 ? 10% -35.0% 1109681 -11.2% 907098 ? 10% -26.3% 1109685 -11.2% 926079 ? 10% -27.0% 1109842 -11.5% 777408 ? 7% -24.9% 1109698 -11.2% 927794 ? 6% -26.1% 1109662 -11.1% 910204 ? 4% -26.3% 1109686 -11.3% 956972 ? 6% -29.5% 1109660 -11.1% 911937 ? 6% -25.3% 1109640 -11.1% 945548 ? 10% -27.3% 1109707 -11.1% 965953 ? 5% -28.4% 1109635 -11.1% 961063 ? 9% -30.4% 1109648 -11.1% 953449 ? 5% -29.3% 1109688 -11.1% 909710 ? 9% -22.2% 1109682 -11.1% 951262 ? 7% -27.6% 1109883 -11.5% 787531 ? 3% -27.2% 1109698 -11.1% 943666 ? 5% -28.5% 1109696 -11.1% 912017 ? 7% -24.5% 1109688 -11.1% 962703 ? 8% -29.7% 1109653 -11.1% 949945 ? 6% -25.6% 1109693 -11.1% 926407 ? 8% -25.0% 1109690 -11.1% 897337 ? 8% -24.5% 1109701 -11.2% 870200 ? 5% -27.7% 1109638 -11.2% 903216 ? 5% -26.2% 1109678 -11.2% 898010 ? 3% -27.7% 1109640 -11.2% 914045 ? 5% -27.6% 1109756 -11.5% 770920 ? 8% -26.9% 1109642 -11.2% 902719 ? 3% -27.4% 1109678 -11.2% 927667 ? 2% -29.9% 1109655 -11.2% 895668 ? 4% -26.7% 1109644 -11.2% 932719 ? 8% -28.6% 1109654 -11.2% 919080 ? 4% -28.9% 1109612 -11.2% 919061 ? 6% -29.3% 1109671 -11.2% 942893 ? 4% -30.7% 1109638 -11.2% 909483 ? 4% -26.2% 1109667 -11.2% 905979 -24.9% 1109666 -11.2% 914012 ? 2% -30.0% 1109769 -11.5% 786901 ? 6% -26.1% 1109661 -11.2% 891457 ? 4% -25.1% 1109567 -11.2% 920671 ? 4% -29.1% 1109583 -11.2% 901948 ? 3% -25.7% 1109693 -11.2% 897848 ? 3% -29.8% 1109764 -11.5% 769032 ? 4% -24.6% 1109749 -11.5% 753284 ? 7% -24.5% 1109740 -11.5% 762912 ? 3% -24.3% 1109715 -11.3% 862584 ? 9% -31.5% 1109709 -11.4% 914512 ? 5% -33.8% 1109849 -11.5% 763288 ? 2% -24.2% 1109697 -11.3% 935470 ? 7% -33.8% 1109689 -11.3% 953999 ? 5% -35.9% 1109713 -11.3% 949063 ? 4% -34.9% 1109677 -11.3% 916942 ? 2% -31.5% 1109676 -11.3% 946944 ? 7% -34.4% 1109671 -11.3% 927026 ? 3% -32.5% 1109686 -11.3% 925930 ? 4% -31.8% 1109675 -11.3% 930402 ? 7% -32.6% 1109729 -11.4% 928439 ? 4% -32.0% 1109731 -11.9% 917894 ? 3% -31.6% 1109813 -11.5% 763728 ? 4% -25.8% 1109702 -11.4% 959814 ? 6% -34.4% 1109691 -11.3% 953721 ? 7% -34.2% 1109716 -11.4% 929251 ? 5% -32.4% 1109720 -11.4% 955082 ? 8% -34.0% 1109706 -11.4% 933432 ? 6% -33.6% 1109724 -11.4% 904660 ? 3% -31.4% 1109716 -11.1% 908916 ? 7% -25.5% 1109635 -11.1% 926313 ? 8% -24.1% 1109685 -11.1% 951056 ? 6% -27.3% 1109743 -11.1% 927813 ? 7% -24.0% 1109737 -11.5% 766534 ? 8% -24.7% 1109783 -11.1% 929300 ? 3% -24.5% 1109714 -11.2% 952762 ? 9% -26.6% 1109708 -11.1% 961704 ? 6% -26.4% 1109757 -11.1% 952246 ? 6% -26.2% 1109751 -11.1% 980627 ? 2% -29.8% 1109689 -11.1% 976440 ? 3% -29.2% 1109645 -11.1% 952352 ? 2% -26.7% 1109682 -11.1% 957266 ? 4% -28.5% 49803 ? 13% -18.0% 1109702 -11.1% 977427 ? 6% -27.8% 1109658 -11.1% 937091 ? 7% -26.2% 1109735 -11.5% 741049 ? 6% -23.6% 1109692 -11.1% 942242 ? 7% -26.2% 1109688 -11.1% 978237 ? 5% -27.8% 1109680 -11.1% 942523 ? 6% -25.9% 1109709 -11.1% 968770 ? 8% -28.5% 1109706 -11.2% 892744 ? 3% -27.4% 1109711 -11.2% 905314 ? 4% -26.4% 1109760 -11.2% 923287 ? 5% -27.0% 1109750 -11.2% 941749 ? 6% -27.9% 1109711 -11.2% 911298 ? 3% -27.4% 1109694 -11.2% 934580 ? 3% -29.9% 1109735 -11.5% 740135 ? 6% -24.3% 1109690 -11.2% 927684 ? 3% -28.1% 1109694 -11.2% 958054 ? 4% -30.3% 1109679 -11.2% 946262 ? 4% -30.0% 1109678 -11.2% 931012 ? 4% -26.6% 1109691 -11.2% 929091 ? 6% -29.7% 1109744 -11.2% 933881 ? 3% -27.6% 1109710 -11.2% 956361 ? 5% -29.2% 1109687 -11.2% 909257 ? 3% -26.6% 1109693 -11.2% 925288 ? 2% -28.0% 1109675 -11.2% 958528 ? 4% -29.0% 31192 ? 5% -17.8% 1109763 -11.4% 766751 ? 7% -25.8% 1109572 -11.2% 934309 ? 2% -28.6% 1109770 -11.2% 923580 ? 5% -26.7% 48586 ? 18% -31.1% 1109690 -11.4% 732823 ? 6% -25.1% 1109780 -11.5% 748198 ? 6% -25.7% 1109731 -11.5% 732000 ? 5% -24.4% 1109689 -11.5% 732437 ? 4% -23.9% 1109783 -11.5% 727115 -22.8% 1109749 -11.5% 722320 ? 5% -23.9% 1109776 -11.5% 774003 ? 8% -27.3% 1109771 -11.5% 725034 ? 4% -21.5% 1109745 -11.5% 766391 ? 6% -26.4% 1109765 -11.5% 747959 ? 5% -24.2% 1109814 -11.5% 747605 ? 7% -24.3% 1109714 -11.5% 740632 ? 4% -25.4% 1109762 -11.5% 752433 ? 6% -23.9% 1109801 -11.5% 750543 ? 6% -25.7% 1109757 -11.5% 710008 ? 5% -20.3% 1109819 -11.5% 756402 ? 4% -26.1% 1109763 -11.5% 763567 ? 5% -27.5% 1109795 -11.4% 736222 ? 4% -24.9% 1109788 -11.4% 729712 ? 5% -24.6% 1109951 -11.5% 749632 ? 7% -25.7% 1109721 -11.4% 889032 ? 8% -34.4% 1109668 -11.4% 914855 ? 5% -35.1% 1109647 -11.3% 948214 ? 8% -35.0% 1109694 -11.4% 892302 ? 7% -32.9% 1109699 -11.4% 913830 ? 4% -33.8% 1109687 -11.4% 942816 ? 8% -35.3% 1109662 -11.3% 921056 ? 7% -33.8% 1109663 -11.3% 939132 ? 5% -35.0% 1109659 -11.3% 905810 ? 6% -32.7% 1109697 -11.3% 936646 ? 7% -34.8% 1.598e+08 1.274e+08 ? 2% 21.07 ? 13% -14.9 20.01 ? 13% -14.3 19.91 ? 13% -14.2 11.18 ? 11% -11.2 17.91 ? 12% -11.0 17.29 ? 12% -10.9 17.15 ? 12% -10.8 17.06 ? 12% -10.8 18.74 ? 11% -10.4 12.96 ? 13% -9.5 15.35 ? 11% -9.5 15.28 ? 11% -9.4 15.16 ? 11% -9.4 10.21 ? 12% -7.0 10.16 ? 12% -7.0 5.82 ? 26% -5.8 5.63 ? 11% -5.6 6.42 ? 13% -5.6 6.28 ? 14% -4.8 6.27 ? 14% -4.8 5.06 ? 17% -4.8 6.24 ? 14% -4.8 6.92 ? 13% -4.5 4.80 ? 13% -4.5 6.87 ? 13% -4.5 52.74 -4.5 52.06 -4.3 6.76 ? 11% -4.3 51.93 -4.3 6.17 ? 13% -4.1 6.08 ? 11% -4.0 5.56 ? 13% -3.8 5.78 ? 11% -3.8 5.51 ? 11% -3.7 5.25 ? 13% -3.7 5.45 ? 12% -3.6 5.36 ? 11% -3.6 5.08 ? 13% -3.6 5.31 ? 12% -3.6 5.25 ? 11% -3.5 4.98 ? 13% -3.5 5.10 ? 12% -3.4 5.09 ? 12% -3.4 5.07 ? 12% -3.4 4.63 ? 13% -3.2 4.45 ? 13% -3.1 4.07 ? 12% -2.9 4.01 ? 12% -2.9 3.78 ? 12% -2.8 3.74 ? 12% -2.8 2.40 ? 12% -1.5 2.09 ? 13% -1.4 47.70 -1.3 47.45 -1.3 46.94 -1.3 46.52 -1.2 46.20 -1.1 45.87 -1.1 1.33 ? 3% -0.6 0.62 ? 2% +0.2 0.65 ? 2% +0.2 0.75 ? 2% +0.2 0.62 ? 3% +0.3 0.65 ? 2% +0.3 0.81 ? 2% +0.3 0.58 ? 7% +0.4 0.18 ?141% +0.4 1.06 ? 6% +0.4 0.73 ? 7% +0.4 0.75 ? 7% +0.4 0.60 ? 12% +0.5 0.36 ? 71% +0.5 1.52 ? 2% +0.5 1.50 ? 2% +0.5 0.37 ? 71% +0.5 1.53 +0.5 0.09 ?223% +0.5 0.65 ? 5% +0.5 0.08 ?223% +0.5 0.41 ? 71% +0.6 0.41 ? 71% +0.6 0.00 +0.7 1.12 ? 7% +0.7 0.88 ? 10% +0.7 0.96 ? 13% +0.7 1.16 ? 9% +0.8 1.20 ? 8% +0.8 1.22 ? 8% +0.8 1.23 ? 8% +0.8 1.97 ? 12% +1.4 0.00 +1.7 3.75 ? 12% +1.8 6.27 ? 10% +2.2 6.32 ? 10% +2.2 7.85 ? 8% +2.4 4.76 ? 12% +2.6 5.38 ? 11% +3.0 7.80 ? 11% +3.3 0.00 +3.4 46.87 +3.5 45.87 +3.7 45.74 +3.7 44.83 +3.9 45.02 +3.9 43.59 +3.9 43.68 +3.9 44.11 +3.9 44.32 +3.9 7.54 ? 9% +4.1 12.45 ? 10% +4.3 12.50 ? 10% +4.3 14.17 ? 9% +4.5 14.23 ? 9% +4.5 9.88 ? 11% +4.8 11.16 ? 11% +5.6 12.06 ? 9% +5.9 13.11 ? 9% +6.5 22.76 ? 8% +7.1 22.70 ? 8% +7.1 0.00 +7.4 14.56 ? 9% +7.5 23.80 ? 8% +7.9 0.00 +14.6 17.18 ? 11% -16.6 23.30 ? 12% -15.5 22.67 ? 12% -15.1 22.63 ? 12% -15.1 22.53 ? 12% -15.0 22.12 ? 12% -14.8 21.00 ? 12% -14.1 20.89 ? 12% -14.0 19.54 ? 10% -12.0 19.55 ? 10% -11.9 18.11 ? 12% -10.8 17.47 ? 12% -10.7 17.34 ? 12% -10.6 17.24 ? 12% -10.6 18.74 ? 11% -10.4 13.76 ? 13% -9.3 15.56 ? 10% -9.2 12.23 ? 12% -7.8 11.26 ? 12% -7.3 10.42 ? 13% -7.0 6.96 ? 13% -4.8 6.72 ? 13% -4.5 7.06 ? 13% -4.5 52.87 -4.4 7.00 ? 13% -4.4 6.37 ? 12% -4.3 6.86 ? 11% -4.2 6.30 ? 13% -4.1 6.17 ? 11% -3.9 5.50 ? 12% -3.7 5.52 ? 11% -3.6 4.70 ? 11% -3.1 4.61 ? 11% -3.0 4.40 ? 13% -2.5 2.99 ? 13% -2.0 3.90 ? 10% -1.5 47.70 -1.3 47.46 -1.3 46.95 -1.3 46.55 -1.2 1.67 ? 13% -1.1 46.21 -1.1 45.90 -1.1 1.66 ? 12% -1.1 5.94 ? 5% -1.1 2.00 ? 3% -0.9 1.30 ? 12% -0.8 1.20 ? 10% -0.8 1.09 ? 13% -0.8 1.05 ? 14% -0.7 1.01 ? 13% -0.7 1.00 ? 12% -0.7 0.91 ? 13% -0.6 0.88 ? 13% -0.6 0.84 ? 14% -0.6 1.44 ? 14% -0.5 97.98 -0.5 0.95 ? 14% -0.5 97.74 -0.5 0.82 ? 11% -0.4 0.70 ? 14% -0.4 0.65 ? 9% -0.4 0.87 ? 12% -0.4 0.51 ? 14% -0.4 0.46 ? 11% -0.3 0.44 ? 9% -0.3 0.41 ? 15% -0.2 0.46 ? 8% -0.2 0.37 ? 8% -0.2 0.45 ? 3% -0.2 0.31 ? 11% -0.2 0.31 ? 12% -0.2 0.31 ? 9% -0.2 0.30 ? 8% -0.2 0.32 ? 16% -0.2 0.37 ? 5% -0.2 0.37 ? 16% -0.2 0.27 ? 10% -0.2 0.27 ? 17% -0.2 0.30 ? 9% -0.2 0.35 ? 5% -0.2 0.23 ? 13% -0.2 0.29 ? 10% -0.1 0.32 ? 9% -0.1 0.66 ? 5% -0.1 0.30 ? 4% -0.1 0.20 ? 11% -0.1 0.26 ? 11% -0.1 0.21 ? 16% -0.1 0.30 ? 15% -0.1 0.19 ? 14% -0.1 0.24 ? 11% -0.1 0.24 ? 12% -0.1 0.21 ? 16% -0.1 0.22 ? 13% -0.1 0.13 ? 15% -0.1 0.16 ? 11% -0.1 0.16 ? 16% -0.1 0.14 ? 14% -0.1 0.13 ? 11% -0.1 0.15 ? 11% -0.1 0.14 ? 15% -0.1 0.13 ? 11% -0.1 0.16 ? 4% -0.1 0.11 ? 11% -0.1 0.11 ? 14% -0.1 0.10 ? 14% -0.1 0.13 ? 15% -0.1 0.14 ? 14% -0.1 0.14 ? 11% -0.0 0.10 ? 9% -0.0 0.08 ? 6% -0.0 0.10 ? 7% -0.0 0.17 ? 4% -0.0 0.12 ? 6% -0.0 0.14 ? 3% -0.0 0.06 +0.0 0.06 +0.0 0.07 +0.0 0.15 ? 3% +0.0 0.04 ? 45% +0.0 0.10 ? 4% +0.0 0.13 ? 5% +0.0 0.19 ? 3% +0.0 0.18 +0.0 0.19 ? 2% +0.0 0.22 ? 2% +0.0 0.26 ? 3% +0.0 0.22 ? 2% +0.1 0.00 +0.1 0.37 +0.1 0.33 ? 2% +0.1 0.04 ? 73% +0.1 0.04 ? 73% +0.1 0.04 ? 73% +0.1 0.00 +0.1 0.42 ? 2% +0.1 0.38 +0.1 0.00 +0.1 0.26 ? 4% +0.1 0.01 ?223% +0.1 0.49 ? 3% +0.1 0.90 ? 3% +0.1 0.01 ?223% +0.1 0.65 +0.2 0.75 ? 2% +0.2 0.02 ?223% +0.2 0.76 ? 7% +0.2 0.76 ? 6% +0.2 0.92 +0.2 0.29 ? 15% +0.2 0.49 ? 12% +0.2 0.03 ?155% +0.2 0.34 ? 13% +0.3 0.40 ? 9% +0.3 0.34 ? 15% +0.3 0.03 ?161% +0.3 0.65 ? 2% +0.3 0.82 ? 2% +0.3 0.17 ? 38% +0.3 0.88 ? 2% +0.3 0.06 ?106% +0.4 1.28 ? 9% +0.4 0.06 ?106% +0.4 0.06 ?106% +0.4 0.06 ?108% +0.4 0.08 ? 81% +0.4 0.08 ? 84% +0.4 1.79 ? 8% +0.4 0.00 +0.4 0.58 ? 12% +0.4 0.76 ? 7% +0.4 0.01 ?223% +0.5 0.01 ?223% +0.5 0.60 ? 12% +0.5 1.52 +0.5 1.50 ? 2% +0.5 1.53 +0.5 0.65 ? 5% +0.5 0.02 ?144% +0.6 0.02 ?144% +0.6 0.02 ?144% +0.6 0.02 ?144% +0.6 1.16 ? 9% +0.8 1.21 ? 8% +0.8 1.22 ? 8% +0.8 1.23 ? 8% +0.8 1.58 ? 7% +1.0 1.47 ? 9% +1.1 2.99 ? 5% +1.1 7.88 ? 8% +2.4 47.00 +3.5 7.43 ? 12% +3.7 44.84 +3.9 45.04 +3.9 43.69 +3.9 43.61 +3.9 44.12 +3.9 44.34 +3.9 7.55 ? 9% +4.1 14.24 ? 9% +4.5 14.20 ? 9% +4.5 11.62 ? 11% +5.2 12.06 ? 9% +5.9 18.74 ? 10% +6.4 18.83 ? 10% +6.5 13.12 ? 9% +6.5 22.71 ? 8% +7.1 22.76 ? 8% +7.1 14.56 ? 9% +7.5 14.92 ? 11% +7.6 23.80 ? 8% +8.0 16.54 ? 11% +8.6 0.00 +22.0 12.79 ? 10% -12.6 4.56 ? 13% -3.1 3.43 ? 14% -2.3 2.98 ? 13% -2.0 3.89 ? 10% -1.5 1.61 ? 13% -1.1 1.22 ? 12% -0.8 1.20 ? 10% -0.8 0.92 ? 13% -0.6 1.54 ? 5% -0.6 0.82 ? 9% -0.4 0.50 ? 14% -0.3 0.50 ? 12% -0.3 0.48 ? 13% -0.3 0.49 ? 13% -0.3 0.36 ? 12% -0.3 0.44 ? 10% -0.3 0.34 ? 13% -0.2 0.30 ? 17% -0.2 0.45 ? 3% -0.2 0.27 ? 9% -0.2 0.33 ? 10% -0.2 0.28 ? 12% -0.2 0.28 ? 9% -0.2 0.26 ? 12% -0.2 0.26 ? 12% -0.2 0.26 ? 17% -0.2 0.27 ? 10% -0.2 0.26 ? 9% -0.2 0.22 ? 11% -0.2 0.30 ? 6% -0.1 0.28 ? 10% -0.1 0.22 ? 12% -0.1 0.40 ? 12% -0.1 0.28 ? 11% -0.1 0.21 ? 16% -0.1 0.24 ? 11% -0.1 0.24 ? 12% -0.1 0.17 ? 13% -0.1 0.22 ? 14% -0.1 0.20 ? 16% -0.1 0.21 ? 14% -0.1 0.22 ? 6% -0.1 0.13 ? 14% -0.1 0.28 ? 6% -0.1 0.17 ? 10% -0.1 0.13 ? 8% -0.1 0.13 ? 14% -0.1 0.14 ? 15% -0.1 0.19 ? 11% -0.1 0.16 ? 14% -0.1 0.16 ? 11% -0.1 0.15 ? 13% -0.1 0.34 ? 5% -0.1 0.15 ? 16% -0.1 0.16 ? 4% -0.1 0.15 ? 10% -0.1 0.14 ? 14% -0.1 0.13 ? 8% -0.1 0.10 ? 14% -0.1 0.14 ? 10% -0.1 0.12 ? 14% -0.1 0.13 ? 7% -0.1 0.29 ? 6% -0.0 0.13 ? 12% -0.0 0.10 ? 9% -0.0 0.11 ? 14% -0.0 0.10 ? 7% -0.0 0.08 ? 10% -0.0 0.10 ? 7% -0.0 0.09 ? 7% -0.0 0.24 ? 3% -0.0 0.08 ? 10% -0.0 0.08 ? 8% -0.0 0.08 ? 11% -0.0 0.07 ? 5% -0.0 0.11 ? 6% -0.0 0.07 ? 10% -0.0 0.21 ? 2% -0.0 0.08 ? 4% -0.0 0.23 ? 3% -0.0 0.13 ? 6% -0.0 0.06 ? 6% +0.0 0.05 ? 7% +0.0 0.09 ? 7% +0.0 0.05 ? 8% +0.0 0.05 ? 7% +0.0 0.10 ? 4% +0.0 0.18 ? 2% +0.0 0.14 ? 3% +0.0 0.16 +0.0 0.19 ? 4% +0.0 0.18 ? 4% +0.0 0.17 ? 2% +0.0 0.27 ? 2% +0.0 0.34 +0.0 0.04 ? 45% +0.0 0.21 ? 2% +0.0 0.50 ? 2% +0.1 0.46 ? 4% +0.1 0.36 +0.1 0.32 ? 2% +0.1 0.22 ? 5% +0.1 0.00 +0.1 0.14 ? 16% +0.1 0.19 ? 6% +0.1 0.40 ? 2% +0.1 0.37 ? 4% +0.1 0.45 ? 3% +0.1 0.21 ? 8% +0.1 0.02 ?223% +0.1 0.20 ? 11% +0.2 1.02 ? 3% +0.2 0.58 ? 2% +0.2 0.86 ? 2% +0.3 1.25 ? 9% +0.4 0.00 +0.4 0.57 ? 12% +0.4 0.76 ? 8% +0.4 0.64 ? 4% +0.5 0.73 ? 8% +0.5 0.74 ? 9% +0.6 0.81 ? 11% +0.6 2.22 ? 4% +0.7 2.79 ? 4% +0.9 1.56 ? 7% +0.9 1.45 ? 9% +1.1 7.36 ? 12% +3.6 11.50 ? 11% +5.2 0.00 +16.7

145000 +----------------- | 140000 |-+ | 135000 |-O O O | O O O O O 130000 |-+ O | OO 125000 |.++.+.+ | + ++.+ : 120000 |-+ :O : : : | : : :: 115000 |-+ :: : | + + 110000 +----------------- 1.4e+09 +---------------- | + 1.3e+09 |-+ + :: 1.2e+09 |-+ :+ : : | : + +.+. 1.1e+09 |.++. .+ 1e+09 |-+ + | 9e+08 |-+ 8e+08 |-+ | 7e+08 |-+ 6e+08 |-+ | OO O OO 5e+08 +---------------- 6.5e+08 +---------------- 6e+08 |-+ + | + : 5.5e+08 |-+ :: : : 5e+08 |-+ :: : +.+. |. : + 4.5e+08 |-++. .+ 4e+08 |-+ + 3.5e+08 |-+ | 3e+08 |-+ 2.5e+08 |-+ | 2e+08 |-+ O O O 1.5e+08 +---------------- [*] bisect-good sample
[O] bisect-bad sample

Disclaimer:
Results have been for informational design or configuration

---
0DAY/LKP+ Test Infrastructure https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory
Hackbench is both a benchmark and a stress test for the Linux kernel scheduler.
/github.com/linux-test-project/ltp/blob/master/testcases/kernel/sched/cfs-scheduler/hackbench.c">https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sched/cfs-scheduler/hackbench.c
----------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
# job file is attached in this email
--compatible job.yaml # generate the yaml file for lkp run
run generated-yaml-file
across any failure that blocks the test,
~/.lkp and /lkp dir to run from a clean state.
==============================================================
pc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
4/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-cpl-4sp1/hackbench/0x700001e
prefer struct_size over open coded arithmetic")
kmem: further deprecate kmem.limit_in_bytes")
2f3567b760c9a8fc8
-----------------
%stddev
\
137012 ? 2% hackbench.throughput
-9.9% 498.51 ? 2% hackbench.time.elapsed_time
-9.9% 498.51 ? 2% hackbench.time.elapsed_time.max
-56.3% 1.539e+08 ? 2% hackbench.time.involuntary_context_switches
69266 ? 2% hackbench.time.system_time
880.42 ? 2% hackbench.time.user_time
-41.2% 5.414e+08 hackbench.time.voluntary_context_switches
343117 ? 32% numa-numastat.node1.numa_hit
216841 ? 15% numa-vmstat.node3.nr_shmem
-8.8% 554.41 uptime.boot
+2.7% 168980 proc-vmstat.nr_slab_unreclaimable
-4.5% 131582 proc-vmstat.pgreuse
0.53 ? 6% turbostat.C1%
-18.8% 2.408e+08 ? 2% turbostat.IRQ
2408 vmstat.procs.r
-38.4% 1439816 vmstat.system.cs
-9.9% 481527 vmstat.system.in
4846 ? 6% slabinfo.kmalloc-cg-192.active_objs
4846 ? 6% slabinfo.kmalloc-cg-192.num_objs
+9.5% 172170 slabinfo.kmalloc-cg-512.active_objs
3480 slabinfo.kmalloc-cg-512.active_slabs
222802 slabinfo.kmalloc-cg-512.num_objs
3480 slabinfo.kmalloc-cg-512.num_slabs
+9.5% 172289 slabinfo.skbuff_head_cache.active_objs
3485 slabinfo.skbuff_head_cache.active_slabs
223129 slabinfo.skbuff_head_cache.num_objs
3485 slabinfo.skbuff_head_cache.num_slabs
-5.2% 1.582e+10 perf-stat.i.branch-instructions
0.64 ? 2% perf-stat.i.branch-miss-rate%
-24.3% 99283197 ? 2% perf-stat.i.branch-misses
28.86 perf-stat.i.cache-miss-rate%
-1.5% 8.491e+08 perf-stat.i.cache-references
-38.2% 1452325 perf-stat.i.context-switches
121977 ? 2% perf-stat.i.cpu-migrations
1931 perf-stat.i.cycles-between-cache-misses
0.06 ? 5% perf-stat.i.dTLB-load-miss-rate%
-12.9% 14414299 ? 3% perf-stat.i.dTLB-load-misses
-1.9% 2.348e+10 perf-stat.i.dTLB-loads
0.04 ? 6% perf-stat.i.dTLB-store-miss-rate%
+30.5% 5252098 ? 5% perf-stat.i.dTLB-store-misses
-35.7% 35989438 ? 8% perf-stat.i.iTLB-load-misses
-35.8% 14787249 perf-stat.i.iTLB-loads
-3.9% 8.027e+10 perf-stat.i.instructions
2288 ? 7% perf-stat.i.instructions-per-iTLB-miss
0.17 perf-stat.i.ipc
+4.5% 908.26 perf-stat.i.metric.K/sec
-2.7% 374.53 perf-stat.i.metric.M/sec
7398 ? 3% perf-stat.i.minor-faults
60.05 ? 2% perf-stat.i.node-load-miss-rate%
-3.8% 38161927 perf-stat.i.node-load-misses
+25.2% 25719712 ? 5% perf-stat.i.node-loads
37.72 ? 5% perf-stat.i.node-store-miss-rate%
+10.0% 17180670 ? 4% perf-stat.i.node-store-misses
+40.5% 28518438 ? 4% perf-stat.i.node-stores
7443 ? 3% perf-stat.i.page-faults
0.62 ? 2% perf-stat.overall.branch-miss-rate%
29.08 perf-stat.overall.cache-miss-rate%
5.79 perf-stat.overall.cpi
1899 perf-stat.overall.cycles-between-cache-misses
0.06 ? 5% perf-stat.overall.dTLB-load-miss-rate%
0.04 ? 6% perf-stat.overall.dTLB-store-miss-rate%
2246 ? 7% perf-stat.overall.instructions-per-iTLB-miss
0.17 perf-stat.overall.ipc
58.87 perf-stat.overall.node-load-miss-rate%
35.89 ? 3% perf-stat.overall.node-store-miss-rate%
-3.5% 1.584e+10 perf-stat.ps.branch-instructions
-23.6% 98487415 ? 2% perf-stat.ps.branch-misses
-38.5% 1437257 perf-stat.ps.context-switches
112490 perf-stat.ps.cpu-migrations
-11.2% 14403294 ? 3% perf-stat.ps.dTLB-load-misses
+31.5% 5191308 ? 5% perf-stat.ps.dTLB-store-misses
-34.4% 35981347 ? 8% perf-stat.ps.iTLB-load-misses
-35.8% 14572556 perf-stat.ps.iTLB-loads
-2.4% 8.035e+10 perf-stat.ps.instructions
24.16 ? 4% perf-stat.ps.major-faults
5839 perf-stat.ps.minor-faults
-4.2% 37669399 perf-stat.ps.node-load-misses
+23.3% 26334557 ? 4% perf-stat.ps.node-loads
+11.5% 16280558 ? 2% perf-stat.ps.node-store-misses
+38.0% 29102191 ? 3% perf-stat.ps.node-stores
5863 perf-stat.ps.page-faults
-12.0% 4.011e+13 perf-stat.total.instructions
10.10 ? 90% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
58.91 ? 32% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
12.05 ? 15% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
24.80 ? 17% perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
38.03 ? 34% perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
26.57 ? 32% perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
86.82 ? 21% perf-sched.sch_delay.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
10.70 ? 19% perf-sched.sch_delay.avg.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
752.85 ? 96% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
8341 ? 29% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
12272 ? 28% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
9499 ? 27% perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
7234 ? 34% perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
7403 ? 44% perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
12149 ? 32% perf-sched.sch_delay.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
11127 ? 28% perf-sched.sch_delay.max.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
16.79 ? 16% perf-sched.total_sch_delay.average.ms
12702 ? 29% perf-sched.total_sch_delay.max.ms
60.38 ? 16% perf-sched.total_wait_and_delay.average.ms
25601 ? 30% perf-sched.total_wait_and_delay.max.ms
43.58 ? 17% perf-sched.total_wait_time.average.ms
13965 ? 29% perf-sched.total_wait_time.max.ms
193.37 ? 28% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
43.84 ? 14% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
91.23 ? 16% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
137.13 ? 29% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
108.34 ? 28% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
193.28 ?107% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
27.16 ?152% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
1228 ?110% perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
308.22 ? 23% perf-sched.wait_and_delay.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
37.98 ? 19% perf-sched.wait_and_delay.avg.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
202.17 ? 14% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
1335 ? 15% perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
17063 ? 19% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
2056 ? 17% perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
11.33 ? 15% perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
119182 ? 12% perf-sched.wait_and_delay.count.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
366.17 ? 18% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
191.83 ? 17% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
16731 ? 29% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
24957 ? 30% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
19021 ? 27% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
14699 ? 33% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
14910 ? 44% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
6317 ?122% perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
7689 ?112% perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
24745 ? 34% perf-sched.wait_and_delay.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
22437 ? 28% perf-sched.wait_and_delay.max.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
42.73 ?129% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
134.46 ? 28% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
31.79 ? 14% perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.13 ? 85% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.do_anonymous_page
66.43 ? 16% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
99.10 ? 28% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
0.04 ?117% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.change_p4d_range.change_protection.change_prot_numa
30.11 ?104% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
81.77 ? 28% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
30.65 ? 43% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg
161.89 ? 95% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
2.26 ? 80% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
54.45 ? 45% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
953.75 ?103% perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
221.40 ? 23% perf-sched.wait_time.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
27.27 ? 19% perf-sched.wait_time.avg.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
3014 ?126% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
7144 ? 33% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
8461 ? 28% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
12906 ? 31% perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
10885 ? 38% perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
10860 ? 33% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
7972 ? 23% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg
8977 ? 31% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
0.06 ?105% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.change_p4d_range.change_protection.change_prot_numa
0.10 ?116% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map
5153 ? 71% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
9191 ? 42% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
6687 ? 29% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg
4168 ? 82% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
27.84 ? 75% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
4500 ? 99% perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
12801 ? 36% perf-sched.wait_time.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
11393 ? 28% perf-sched.wait_time.max.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
13607 ? 28% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
13401 ? 30% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
74716 ? 8% softirqs.CPU0.RCU
76240 ? 9% softirqs.CPU1.RCU
75796 ? 9% softirqs.CPU10.RCU
78845 ? 7% softirqs.CPU100.RCU
78097 ? 7% softirqs.CPU101.RCU
78278 ? 7% softirqs.CPU102.RCU
78190 ? 6% softirqs.CPU103.RCU
78921 ? 7% softirqs.CPU104.RCU
78674 ? 7% softirqs.CPU105.RCU
78921 ? 7% softirqs.CPU106.RCU
79354 ? 8% softirqs.CPU107.RCU
75835 ? 7% softirqs.CPU108.RCU
74646 ? 9% softirqs.CPU109.RCU
75619 ? 9% softirqs.CPU11.RCU
74646 ? 9% softirqs.CPU110.RCU
74549 ? 8% softirqs.CPU111.RCU
81465 ? 10% softirqs.CPU112.RCU
82531 ? 8% softirqs.CPU113.RCU
81424 ? 9% softirqs.CPU114.RCU
81765 ? 8% softirqs.CPU115.RCU
82770 ? 9% softirqs.CPU116.RCU
82488 ? 9% softirqs.CPU117.RCU
81501 ? 8% softirqs.CPU118.RCU
81931 ? 9% softirqs.CPU119.RCU
76276 ? 9% softirqs.CPU12.RCU
82148 ? 9% softirqs.CPU120.RCU
82558 ? 9% softirqs.CPU121.RCU
82113 ? 9% softirqs.CPU122.RCU
81563 ? 8% softirqs.CPU123.RCU
81160 ? 10% softirqs.CPU124.RCU
82390 ? 9% softirqs.CPU125.RCU
72288 ? 8% softirqs.CPU126.RCU
71593 ? 8% softirqs.CPU127.RCU
76053 ? 9% softirqs.CPU128.RCU
76318 ? 8% softirqs.CPU129.RCU
15821 ? 3% softirqs.CPU13.SCHED
76206 ? 8% softirqs.CPU130.RCU
76205 ? 9% softirqs.CPU131.RCU
75876 ? 9% softirqs.CPU132.RCU
75668 ? 9% softirqs.CPU133.RCU
76813 ? 9% softirqs.CPU134.RCU
76276 ? 9% softirqs.CPU135.RCU
76251 ? 9% softirqs.CPU136.RCU
75465 ? 9% softirqs.CPU137.RCU
75994 ? 9% softirqs.CPU138.RCU
76309 ? 9% softirqs.CPU139.RCU
75887 ? 8% softirqs.CPU14.RCU
15412 ? 4% softirqs.CPU14.SCHED
76139 ? 9% softirqs.CPU140.RCU
76136 ? 9% softirqs.CPU141.RCU
75972 ? 9% softirqs.CPU142.RCU
76557 ? 9% softirqs.CPU143.RCU
76430 ? 8% softirqs.CPU15.RCU
71332 ? 8% softirqs.CPU16.RCU
71850 ? 9% softirqs.CPU17.RCU
15524 ? 3% softirqs.CPU17.SCHED
79008 ? 7% softirqs.CPU18.RCU
78562 ? 7% softirqs.CPU19.RCU
78315 ? 6% softirqs.CPU2.RCU
78322 ? 7% softirqs.CPU20.RCU
77774 ? 7% softirqs.CPU21.RCU
78328 ? 7% softirqs.CPU22.RCU
78577 ? 7% softirqs.CPU23.RCU
78470 ? 7% softirqs.CPU24.RCU
77996 ? 7% softirqs.CPU25.RCU
77740 ? 7% softirqs.CPU26.RCU
78432 ? 7% softirqs.CPU27.RCU
78450 ? 7% softirqs.CPU28.RCU
77285 ? 8% softirqs.CPU29.RCU
76462 ? 8% softirqs.CPU3.RCU
77958 ? 8% softirqs.CPU30.RCU
76829 ? 5% softirqs.CPU31.RCU
73818 ? 8% softirqs.CPU32.RCU
73689 ? 8% softirqs.CPU33.RCU
73854 ? 8% softirqs.CPU34.RCU
74234 ? 8% softirqs.CPU35.RCU
80287 ? 9% softirqs.CPU36.RCU
80218 ? 8% softirqs.CPU37.RCU
79957 ? 9% softirqs.CPU38.RCU
80235 ? 9% softirqs.CPU39.RCU
76515 ? 7% softirqs.CPU4.RCU
15539 ? 3% softirqs.CPU4.SCHED
79721 ? 9% softirqs.CPU40.RCU
80123 ? 8% softirqs.CPU41.RCU
79565 ? 9% softirqs.CPU42.RCU
79751 ? 8% softirqs.CPU43.RCU
80501 ? 8% softirqs.CPU44.RCU
80015 ? 9% softirqs.CPU45.RCU
79645 ? 9% softirqs.CPU46.RCU
79722 ? 8% softirqs.CPU47.RCU
77543 ? 8% softirqs.CPU48.RCU
77848 ? 9% softirqs.CPU49.RCU
76176 ? 8% softirqs.CPU5.RCU
77631 ? 8% softirqs.CPU50.RCU
77564 ? 8% softirqs.CPU51.RCU
77084 ? 9% softirqs.CPU52.RCU
77752 ? 8% softirqs.CPU53.RCU
77589 ? 8% softirqs.CPU54.RCU
76854 ? 8% softirqs.CPU55.RCU
77130 ? 8% softirqs.CPU56.RCU
76929 ? 8% softirqs.CPU57.RCU
77400 ? 8% softirqs.CPU58.RCU
77250 ? 8% softirqs.CPU59.RCU
75749 ? 8% softirqs.CPU6.RCU
76839 ? 9% softirqs.CPU60.RCU
76706 ? 8% softirqs.CPU61.RCU
77340 ? 8% softirqs.CPU62.RCU
77089 ? 9% softirqs.CPU63.RCU
76600 ? 8% softirqs.CPU64.RCU
76579 ? 8% softirqs.CPU65.RCU
76411 ? 8% softirqs.CPU66.RCU
76702 ? 8% softirqs.CPU67.RCU
76794 ? 8% softirqs.CPU68.RCU
77154 ? 8% softirqs.CPU69.RCU
75872 ? 7% softirqs.CPU7.RCU
76599 ? 8% softirqs.CPU70.RCU
76369 ? 8% softirqs.CPU71.RCU
76899 ? 8% softirqs.CPU72.RCU
15314 ? 3% softirqs.CPU73.SCHED
77534 ? 7% softirqs.CPU74.RCU
77573 ? 8% softirqs.CPU75.RCU
77402 ? 8% softirqs.CPU76.RCU
77474 ? 7% softirqs.CPU77.RCU
77372 ? 8% softirqs.CPU78.RCU
77023 ? 8% softirqs.CPU79.RCU
76543 ? 8% softirqs.CPU8.RCU
78003 ? 8% softirqs.CPU80.RCU
77968 ? 8% softirqs.CPU81.RCU
15363 ? 3% softirqs.CPU81.SCHED
77527 ? 7% softirqs.CPU82.RCU
77621 ? 8% softirqs.CPU83.RCU
78128 ? 8% softirqs.CPU84.RCU
15605 ? 3% softirqs.CPU84.SCHED
77482 ? 8% softirqs.CPU85.RCU
78258 ? 7% softirqs.CPU86.RCU
77825 ? 8% softirqs.CPU87.RCU
77588 ? 8% softirqs.CPU88.RCU
78190 ? 8% softirqs.CPU89.RCU
77225 ? 8% softirqs.CPU9.RCU
76846 ? 7% softirqs.CPU90.RCU
76378 ? 7% softirqs.CPU91.RCU
76374 ? 7% softirqs.CPU92.RCU
76030 ? 7% softirqs.CPU93.RCU
76238 ? 7% softirqs.CPU94.RCU
76619 ? 7% softirqs.CPU95.RCU
78959 ? 7% softirqs.CPU96.RCU
78044 ? 7% softirqs.CPU97.RCU
78392 ? 7% softirqs.CPU98.RCU
79138 ? 7% softirqs.CPU99.RCU
-13.3% 11171782 ? 8% softirqs.RCU
101130 ? 7% softirqs.TIMER
-20.9% 5431401 ? 9% interrupts.CAL:Function_call_interrupts
963752 ? 7% interrupts.CPU0.LOC:Local_timer_interrupts
547987 ? 6% interrupts.CPU0.RES:Rescheduling_interrupts
982545 ? 4% interrupts.CPU1.LOC:Local_timer_interrupts
574218 ? 6% interrupts.CPU1.RES:Rescheduling_interrupts
982586 ? 4% interrupts.CPU10.LOC:Local_timer_interrupts
576330 ? 7% interrupts.CPU10.RES:Rescheduling_interrupts
983929 ? 3% interrupts.CPU100.LOC:Local_timer_interrupts
620758 ? 11% interrupts.CPU100.RES:Rescheduling_interrupts
985868 ? 3% interrupts.CPU101.LOC:Local_timer_interrupts
610073 ? 8% interrupts.CPU101.RES:Rescheduling_interrupts
34488 ? 22% interrupts.CPU102.CAL:Function_call_interrupts
983935 ? 3% interrupts.CPU102.LOC:Local_timer_interrupts
610117 ? 10% interrupts.CPU102.RES:Rescheduling_interrupts
983934 ? 3% interrupts.CPU103.LOC:Local_timer_interrupts
604948 ? 10% interrupts.CPU103.RES:Rescheduling_interrupts
983504 ? 3% interrupts.CPU104.LOC:Local_timer_interrupts
618904 ? 11% interrupts.CPU104.RES:Rescheduling_interrupts
983840 ? 3% interrupts.CPU105.LOC:Local_timer_interrupts
609904 ? 12% interrupts.CPU105.RES:Rescheduling_interrupts
983590 ? 3% interrupts.CPU106.LOC:Local_timer_interrupts
596222 ? 8% interrupts.CPU106.RES:Rescheduling_interrupts
31357 ? 20% interrupts.CPU107.CAL:Function_call_interrupts
983779 ? 3% interrupts.CPU107.LOC:Local_timer_interrupts
606683 ? 9% interrupts.CPU107.RES:Rescheduling_interrupts
985738 ? 3% interrupts.CPU108.LOC:Local_timer_interrupts
668850 ? 4% interrupts.CPU108.RES:Rescheduling_interrupts
985797 ? 3% interrupts.CPU109.LOC:Local_timer_interrupts
675677 ? 2% interrupts.CPU109.RES:Rescheduling_interrupts
982553 ? 3% interrupts.CPU11.LOC:Local_timer_interrupts
584102 ? 6% interrupts.CPU11.RES:Rescheduling_interrupts
985930 ? 3% interrupts.CPU110.LOC:Local_timer_interrupts
685774 ? 3% interrupts.CPU110.RES:Rescheduling_interrupts
986214 ? 3% interrupts.CPU111.LOC:Local_timer_interrupts
670493 ? 3% interrupts.CPU111.RES:Rescheduling_interrupts
984562 ? 3% interrupts.CPU112.LOC:Local_timer_interrupts
674228 ? 3% interrupts.CPU112.RES:Rescheduling_interrupts
986024 ? 3% interrupts.CPU113.LOC:Local_timer_interrupts
680865 ? 3% interrupts.CPU113.RES:Rescheduling_interrupts
986264 ? 3% interrupts.CPU114.LOC:Local_timer_interrupts
687639 ? 2% interrupts.CPU114.RES:Rescheduling_interrupts
986350 ? 3% interrupts.CPU115.LOC:Local_timer_interrupts
692092 ? 4% interrupts.CPU115.RES:Rescheduling_interrupts
985969 ? 3% interrupts.CPU116.LOC:Local_timer_interrupts
669281 ? 3% interrupts.CPU116.RES:Rescheduling_interrupts
986070 ? 3% interrupts.CPU117.LOC:Local_timer_interrupts
674189 ? 4% interrupts.CPU117.RES:Rescheduling_interrupts
986329 ? 3% interrupts.CPU118.LOC:Local_timer_interrupts
707837 ? 2% interrupts.CPU118.RES:Rescheduling_interrupts
986282 ? 3% interrupts.CPU119.LOC:Local_timer_interrupts
688722 ? 4% interrupts.CPU119.RES:Rescheduling_interrupts
982267 ? 4% interrupts.CPU12.LOC:Local_timer_interrupts
573213 ? 7% interrupts.CPU12.RES:Rescheduling_interrupts
986222 ? 3% interrupts.CPU120.LOC:Local_timer_interrupts
674336 ? 4% interrupts.CPU120.RES:Rescheduling_interrupts
986248 ? 3% interrupts.CPU121.LOC:Local_timer_interrupts
688310 interrupts.CPU121.RES:Rescheduling_interrupts
986359 ? 3% interrupts.CPU122.LOC:Local_timer_interrupts
676523 ? 2% interrupts.CPU122.RES:Rescheduling_interrupts
986405 ? 3% interrupts.CPU123.LOC:Local_timer_interrupts
706314 ? 4% interrupts.CPU123.RES:Rescheduling_interrupts
986142 ? 3% interrupts.CPU124.LOC:Local_timer_interrupts
695173 ? 3% interrupts.CPU124.RES:Rescheduling_interrupts
986479 ? 3% interrupts.CPU125.LOC:Local_timer_interrupts
677912 ? 4% interrupts.CPU125.RES:Rescheduling_interrupts
985053 ? 3% interrupts.CPU126.LOC:Local_timer_interrupts
629561 ? 12% interrupts.CPU126.RES:Rescheduling_interrupts
985218 ? 3% interrupts.CPU127.LOC:Local_timer_interrupts
667011 ? 11% interrupts.CPU127.RES:Rescheduling_interrupts
985554 ? 3% interrupts.CPU128.LOC:Local_timer_interrupts
649239 ? 10% interrupts.CPU128.RES:Rescheduling_interrupts
985393 ? 3% interrupts.CPU129.LOC:Local_timer_interrupts
661981 ? 13% interrupts.CPU129.RES:Rescheduling_interrupts
982011 ? 4% interrupts.CPU13.LOC:Local_timer_interrupts
563555 ? 7% interrupts.CPU13.RES:Rescheduling_interrupts
985483 ? 3% interrupts.CPU130.LOC:Local_timer_interrupts
655719 ? 12% interrupts.CPU130.RES:Rescheduling_interrupts
985411 ? 3% interrupts.CPU131.LOC:Local_timer_interrupts
650000 ? 12% interrupts.CPU131.RES:Rescheduling_interrupts
985585 ? 3% interrupts.CPU132.LOC:Local_timer_interrupts
656131 ? 9% interrupts.CPU132.RES:Rescheduling_interrupts
985607 ? 3% interrupts.CPU133.LOC:Local_timer_interrupts
665823 ? 11% interrupts.CPU133.RES:Rescheduling_interrupts
985756 ? 3% interrupts.CPU134.LOC:Local_timer_interrupts
653480 ? 10% interrupts.CPU134.RES:Rescheduling_interrupts
985559 ? 3% interrupts.CPU135.LOC:Local_timer_interrupts
649968 ? 10% interrupts.CPU135.RES:Rescheduling_interrupts
985352 ? 3% interrupts.CPU136.LOC:Local_timer_interrupts
653873 ? 10% interrupts.CPU136.RES:Rescheduling_interrupts
985586 ? 3% interrupts.CPU137.LOC:Local_timer_interrupts
671290 ? 12% interrupts.CPU137.RES:Rescheduling_interrupts
985336 ? 3% interrupts.CPU138.LOC:Local_timer_interrupts
680152 ? 12% interrupts.CPU138.RES:Rescheduling_interrupts
985457 ? 3% interrupts.CPU139.LOC:Local_timer_interrupts
639753 ? 11% interrupts.CPU139.RES:Rescheduling_interrupts
982477 ? 4% interrupts.CPU14.LOC:Local_timer_interrupts
581705 ? 5% interrupts.CPU14.RES:Rescheduling_interrupts
985582 ? 3% interrupts.CPU140.LOC:Local_timer_interrupts
667637 ? 9% interrupts.CPU140.RES:Rescheduling_interrupts
985643 ? 3% interrupts.CPU141.LOC:Local_timer_interrupts
652448 ? 10% interrupts.CPU141.RES:Rescheduling_interrupts
985732 ? 3% interrupts.CPU142.LOC:Local_timer_interrupts
670496 ? 11% interrupts.CPU142.RES:Rescheduling_interrupts
985489 ? 3% interrupts.CPU143.LOC:Local_timer_interrupts
630227 ? 11% interrupts.CPU143.RES:Rescheduling_interrupts
982288 ? 4% interrupts.CPU15.LOC:Local_timer_interrupts
579800 ? 5% interrupts.CPU15.RES:Rescheduling_interrupts
981954 ? 4% interrupts.CPU16.LOC:Local_timer_interrupts
568931 ? 7% interrupts.CPU16.RES:Rescheduling_interrupts
982067 ? 4% interrupts.CPU17.LOC:Local_timer_interrupts
577864 ? 6% interrupts.CPU17.RES:Rescheduling_interrupts
983766 ? 3% interrupts.CPU18.LOC:Local_timer_interrupts
591294 ? 8% interrupts.CPU18.RES:Rescheduling_interrupts
983571 ? 3% interrupts.CPU19.LOC:Local_timer_interrupts
605096 ? 9% interrupts.CPU19.RES:Rescheduling_interrupts
982602 ? 4% interrupts.CPU2.LOC:Local_timer_interrupts
578849 ? 8% interrupts.CPU2.RES:Rescheduling_interrupts
983869 ? 3% interrupts.CPU20.LOC:Local_timer_interrupts
619417 ? 8% interrupts.CPU20.RES:Rescheduling_interrupts
983811 ? 3% interrupts.CPU21.LOC:Local_timer_interrupts
611854 ? 9% interrupts.CPU21.RES:Rescheduling_interrupts
983859 ? 3% interrupts.CPU22.LOC:Local_timer_interrupts
617867 ? 10% interrupts.CPU22.RES:Rescheduling_interrupts
983782 ? 3% interrupts.CPU23.LOC:Local_timer_interrupts
627663 ? 9% interrupts.CPU23.RES:Rescheduling_interrupts
983778 ? 3% interrupts.CPU24.LOC:Local_timer_interrupts
621109 ? 9% interrupts.CPU24.RES:Rescheduling_interrupts
983788 ? 3% interrupts.CPU25.LOC:Local_timer_interrupts
625454 ? 10% interrupts.CPU25.RES:Rescheduling_interrupts
983799 ? 3% interrupts.CPU26.LOC:Local_timer_interrupts
631366 ? 9% interrupts.CPU26.RES:Rescheduling_interrupts
983850 ? 3% interrupts.CPU27.LOC:Local_timer_interrupts
627189 ? 8% interrupts.CPU27.RES:Rescheduling_interrupts
983703 ? 3% interrupts.CPU28.LOC:Local_timer_interrupts
630948 ? 7% interrupts.CPU28.RES:Rescheduling_interrupts
977654 ? 4% interrupts.CPU29.LOC:Local_timer_interrupts
627431 ? 7% interrupts.CPU29.RES:Rescheduling_interrupts
982425 ? 4% interrupts.CPU3.LOC:Local_timer_interrupts
566752 ? 7% interrupts.CPU3.RES:Rescheduling_interrupts
983710 ? 3% interrupts.CPU30.LOC:Local_timer_interrupts
630064 ? 11% interrupts.CPU30.RES:Rescheduling_interrupts
983822 ? 3% interrupts.CPU31.LOC:Local_timer_interrupts
627837 ? 12% interrupts.CPU31.RES:Rescheduling_interrupts
983559 ? 3% interrupts.CPU32.LOC:Local_timer_interrupts
628141 ? 10% interrupts.CPU32.RES:Rescheduling_interrupts
983527 ? 3% interrupts.CPU33.LOC:Local_timer_interrupts
629974 ? 9% interrupts.CPU33.RES:Rescheduling_interrupts
983432 ? 3% interrupts.CPU34.LOC:Local_timer_interrupts
619604 ? 9% interrupts.CPU34.RES:Rescheduling_interrupts
983426 ? 3% interrupts.CPU35.LOC:Local_timer_interrupts
620967 ? 9% interrupts.CPU35.RES:Rescheduling_interrupts
986109 ? 3% interrupts.CPU36.LOC:Local_timer_interrupts
677428 ? 2% interrupts.CPU36.RES:Rescheduling_interrupts
986076 ? 3% interrupts.CPU37.LOC:Local_timer_interrupts
702863 ? 3% interrupts.CPU37.RES:Rescheduling_interrupts
986009 ? 3% interrupts.CPU38.LOC:Local_timer_interrupts
691529 ? 4% interrupts.CPU38.RES:Rescheduling_interrupts
986389 ? 3% interrupts.CPU39.LOC:Local_timer_interrupts
705228 ? 4% interrupts.CPU39.RES:Rescheduling_interrupts
982501 ? 3% interrupts.CPU4.LOC:Local_timer_interrupts
577507 ? 6% interrupts.CPU4.RES:Rescheduling_interrupts
986790 ? 3% interrupts.CPU40.LOC:Local_timer_interrupts
702028 ? 3% interrupts.CPU40.RES:Rescheduling_interrupts
985893 ? 3% interrupts.CPU41.LOC:Local_timer_interrupts
698904 ? 3% interrupts.CPU41.RES:Rescheduling_interrupts
986368 ? 3% interrupts.CPU42.LOC:Local_timer_interrupts
707713 ? 3% interrupts.CPU42.RES:Rescheduling_interrupts
986228 ? 3% interrupts.CPU43.LOC:Local_timer_interrupts
702405 ? 2% interrupts.CPU43.RES:Rescheduling_interrupts
986140 ? 3% interrupts.CPU44.LOC:Local_timer_interrupts
688585 ? 4% interrupts.CPU44.RES:Rescheduling_interrupts
985981 ? 3% interrupts.CPU45.LOC:Local_timer_interrupts
691802 ? 3% interrupts.CPU45.RES:Rescheduling_interrupts
986013 ? 3% interrupts.CPU46.LOC:Local_timer_interrupts
697811 ? 4% interrupts.CPU46.RES:Rescheduling_interrupts
986172 ? 3% interrupts.CPU47.LOC:Local_timer_interrupts
684262 interrupts.CPU47.RES:Rescheduling_interrupts
40844 ? 12% interrupts.CPU48.CAL:Function_call_interrupts
986471 ? 3% interrupts.CPU48.LOC:Local_timer_interrupts
705625 ? 3% interrupts.CPU48.RES:Rescheduling_interrupts
986172 ? 3% interrupts.CPU49.LOC:Local_timer_interrupts
691409 ? 3% interrupts.CPU49.RES:Rescheduling_interrupts
982317 ? 4% interrupts.CPU5.LOC:Local_timer_interrupts
566103 ? 6% interrupts.CPU5.RES:Rescheduling_interrupts
986149 ? 3% interrupts.CPU50.LOC:Local_timer_interrupts
695845 ? 3% interrupts.CPU50.RES:Rescheduling_interrupts
986153 ? 3% interrupts.CPU51.LOC:Local_timer_interrupts
706561 ? 2% interrupts.CPU51.RES:Rescheduling_interrupts
986101 ? 3% interrupts.CPU52.LOC:Local_timer_interrupts
698844 ? 2% interrupts.CPU52.RES:Rescheduling_interrupts
986142 ? 3% interrupts.CPU53.LOC:Local_timer_interrupts
692330 ? 4% interrupts.CPU53.RES:Rescheduling_interrupts
985372 ? 3% interrupts.CPU54.LOC:Local_timer_interrupts
648201 ? 10% interrupts.CPU54.RES:Rescheduling_interrupts
985410 ? 3% interrupts.CPU55.LOC:Local_timer_interrupts
666564 ? 10% interrupts.CPU55.RES:Rescheduling_interrupts
985583 ? 3% interrupts.CPU56.LOC:Local_timer_interrupts
673797 ? 10% interrupts.CPU56.RES:Rescheduling_interrupts
985363 ? 3% interrupts.CPU57.LOC:Local_timer_interrupts
678794 ? 9% interrupts.CPU57.RES:Rescheduling_interrupts
985560 ? 3% interrupts.CPU58.LOC:Local_timer_interrupts
661172 ? 10% interrupts.CPU58.RES:Rescheduling_interrupts
985480 ? 3% interrupts.CPU59.LOC:Local_timer_interrupts
654916 ? 9% interrupts.CPU59.RES:Rescheduling_interrupts
982399 ? 4% interrupts.CPU6.LOC:Local_timer_interrupts
560193 ? 5% interrupts.CPU6.RES:Rescheduling_interrupts
985677 ? 3% interrupts.CPU60.LOC:Local_timer_interrupts
667287 ? 11% interrupts.CPU60.RES:Rescheduling_interrupts
985524 ? 3% interrupts.CPU61.LOC:Local_timer_interrupts
667940 ? 11% interrupts.CPU61.RES:Rescheduling_interrupts
985364 ? 3% interrupts.CPU62.LOC:Local_timer_interrupts
662460 ? 11% interrupts.CPU62.RES:Rescheduling_interrupts
985301 ? 3% interrupts.CPU63.LOC:Local_timer_interrupts
683774 ? 11% interrupts.CPU63.RES:Rescheduling_interrupts
985194 ? 3% interrupts.CPU64.LOC:Local_timer_interrupts
653309 ? 11% interrupts.CPU64.RES:Rescheduling_interrupts
985320 ? 3% interrupts.CPU65.LOC:Local_timer_interrupts
676297 ? 9% interrupts.CPU65.RES:Rescheduling_interrupts
985673 ? 3% interrupts.CPU66.LOC:Local_timer_interrupts
676634 ? 9% interrupts.CPU66.RES:Rescheduling_interrupts
985315 ? 3% interrupts.CPU67.LOC:Local_timer_interrupts
667439 ? 12% interrupts.CPU67.RES:Rescheduling_interrupts
985287 ? 3% interrupts.CPU68.LOC:Local_timer_interrupts
665809 ? 9% interrupts.CPU68.RES:Rescheduling_interrupts
985486 ? 3% interrupts.CPU69.LOC:Local_timer_interrupts
680393 ? 9% interrupts.CPU69.RES:Rescheduling_interrupts
25645 ? 15% interrupts.CPU7.CAL:Function_call_interrupts
982851 ? 3% interrupts.CPU7.LOC:Local_timer_interrupts
569219 ? 4% interrupts.CPU7.RES:Rescheduling_interrupts
985353 ? 3% interrupts.CPU70.LOC:Local_timer_interrupts
666929 ? 11% interrupts.CPU70.RES:Rescheduling_interrupts
985376 ? 3% interrupts.CPU71.LOC:Local_timer_interrupts
676802 ? 9% interrupts.CPU71.RES:Rescheduling_interrupts
33495 ? 20% interrupts.CPU72.CAL:Function_call_interrupts
983437 ? 3% interrupts.CPU72.LOC:Local_timer_interrupts
548780 ? 6% interrupts.CPU72.RES:Rescheduling_interrupts
982255 ? 4% interrupts.CPU73.LOC:Local_timer_interrupts
555808 ? 2% interrupts.CPU73.RES:Rescheduling_interrupts
982361 ? 4% interrupts.CPU74.LOC:Local_timer_interrupts
553248 ? 8% interrupts.CPU74.RES:Rescheduling_interrupts
982406 ? 4% interrupts.CPU75.LOC:Local_timer_interrupts
557346 ? 5% interrupts.CPU75.RES:Rescheduling_interrupts
982253 ? 4% interrupts.CPU76.LOC:Local_timer_interrupts
561035 ? 6% interrupts.CPU76.RES:Rescheduling_interrupts
981909 ? 4% interrupts.CPU77.LOC:Local_timer_interrupts
549545 ? 7% interrupts.CPU77.RES:Rescheduling_interrupts
982201 ? 4% interrupts.CPU78.LOC:Local_timer_interrupts
562990 ? 5% interrupts.CPU78.RES:Rescheduling_interrupts
982534 ? 3% interrupts.CPU79.LOC:Local_timer_interrupts
569192 ? 5% interrupts.CPU79.RES:Rescheduling_interrupts
982390 ? 4% interrupts.CPU8.LOC:Local_timer_interrupts
564224 ? 5% interrupts.CPU8.RES:Rescheduling_interrupts
982241 ? 4% interrupts.CPU80.LOC:Local_timer_interrupts
566670 ? 8% interrupts.CPU80.RES:Rescheduling_interrupts
981873 ? 4% interrupts.CPU81.LOC:Local_timer_interrupts
565728 ? 7% interrupts.CPU81.RES:Rescheduling_interrupts
982405 ? 4% interrupts.CPU82.LOC:Local_timer_interrupts
552433 ? 7% interrupts.CPU82.RES:Rescheduling_interrupts
982189 ? 4% interrupts.CPU83.LOC:Local_timer_interrupts
572430 ? 6% interrupts.CPU83.RES:Rescheduling_interrupts
982438 ? 4% interrupts.CPU84.LOC:Local_timer_interrupts
557447 ? 7% interrupts.CPU84.RES:Rescheduling_interrupts
982181 ? 4% interrupts.CPU85.LOC:Local_timer_interrupts
565565 ? 7% interrupts.CPU85.RES:Rescheduling_interrupts
982651 ? 3% interrupts.CPU86.LOC:Local_timer_interrupts
559174 ? 6% interrupts.CPU86.RES:Rescheduling_interrupts
982634 ? 3% interrupts.CPU87.LOC:Local_timer_interrupts
553678 ? 5% interrupts.CPU87.RES:Rescheduling_interrupts
982983 ? 3% interrupts.CPU88.LOC:Local_timer_interrupts
552717 ? 6% interrupts.CPU88.RES:Rescheduling_interrupts
982800 ? 3% interrupts.CPU89.LOC:Local_timer_interrupts
550308 ? 6% interrupts.CPU89.RES:Rescheduling_interrupts
982025 ? 4% interrupts.CPU9.LOC:Local_timer_interrupts
556858 ? 7% interrupts.CPU9.RES:Rescheduling_interrupts
983595 ? 3% interrupts.CPU90.LOC:Local_timer_interrupts
582776 ? 9% interrupts.CPU90.RES:Rescheduling_interrupts
983263 ? 3% interrupts.CPU91.LOC:Local_timer_interrupts
593653 ? 8% interrupts.CPU91.RES:Rescheduling_interrupts
983819 ? 3% interrupts.CPU92.LOC:Local_timer_interrupts
616661 ? 10% interrupts.CPU92.RES:Rescheduling_interrupts
983708 ? 3% interrupts.CPU93.LOC:Local_timer_interrupts
598886 ? 9% interrupts.CPU93.RES:Rescheduling_interrupts
983475 ? 3% interrupts.CPU94.LOC:Local_timer_interrupts
605237 ? 10% interrupts.CPU94.RES:Rescheduling_interrupts
983654 ? 3% interrupts.CPU95.LOC:Local_timer_interrupts
610115 ? 9% interrupts.CPU95.RES:Rescheduling_interrupts
983930 ? 3% interrupts.CPU96.LOC:Local_timer_interrupts
609373 ? 10% interrupts.CPU96.RES:Rescheduling_interrupts
983971 ? 3% interrupts.CPU97.LOC:Local_timer_interrupts
610299 ? 10% interrupts.CPU97.RES:Rescheduling_interrupts
983854 ? 3% interrupts.CPU98.LOC:Local_timer_interrupts
609848 ? 9% interrupts.CPU98.RES:Rescheduling_interrupts
983762 ? 3% interrupts.CPU99.LOC:Local_timer_interrupts
610801 ? 10% interrupts.CPU99.RES:Rescheduling_interrupts
-11.3% 1.417e+08 ? 3% interrupts.LOC:Local_timer_interrupts
-28.5% 91108791 interrupts.RES:Rescheduling_interrupts
6.16 ? 35% perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event
5.71 ? 38% perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow
5.68 ? 38% perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
0.00 perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve
6.86 ? 22% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
6.42 ? 23% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg
6.31 ? 23% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg
6.26 ? 23% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
8.30 ? 17% perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
3.45 ? 49% perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
5.88 ? 22% perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
5.84 ? 22% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
5.77 ? 23% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg
3.21 ? 32% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
3.19 ? 32% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime
0.00 perf-profile.calltrace.cycles-pp.asm_exc_page_fault.__get_user_nocheck_8.perf_callchain_user.get_perf_callchain.perf_callchain
0.00 perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb
0.81 ?114% perf-profile.calltrace.cycles-pp.perf_callchain_user.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
1.48 ? 46% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
1.47 ? 46% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
0.28 ?223% perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
1.46 ? 46% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch
2.40 ? 24% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.31 ?223% perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
2.38 ? 24% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
48.28 perf-profile.calltrace.cycles-pp.__libc_write
47.73 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
2.44 ? 26% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.unix_stream_read_generic
47.62 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
2.04 ? 27% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
2.11 ? 28% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
1.73 ? 30% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
1.95 ? 30% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
1.83 ? 31% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
1.59 ? 32% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate
1.83 ? 30% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
1.77 ? 32% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
1.52 ? 33% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair
1.76 ? 31% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common
1.73 ? 31% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
1.49 ? 33% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity
1.68 ? 31% perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function
1.67 ? 31% perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
1.66 ? 32% perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template
1.45 ? 33% perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.schedule_timeout.unix_stream_read_generic
1.38 ? 33% perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.schedule_timeout
1.15 ? 27% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
1.11 ? 28% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
0.99 ? 30% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.98 ? 30% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
0.90 ? 18% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.66 ? 46% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
46.38 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
46.14 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
45.63 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
45.29 perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
45.08 perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write.ksys_write
44.76 perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write
0.78 ? 6% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.unix_stream_read_generic
0.78 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
0.81 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
0.92 ? 4% perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
0.90 ? 3% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
0.94 ? 3% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
1.12 ? 3% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
0.96 ? 4% perf-profile.calltrace.cycles-pp.__slab_free.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
0.57 ? 6% perf-profile.calltrace.cycles-pp.mod_objcg_state.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
1.47 ? 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
1.18 ? 7% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
1.20 ? 7% perf-profile.calltrace.cycles-pp.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
1.06 ? 8% perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
0.83 ? 9% perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_objcg.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve
1.99 ? 2% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
1.97 ? 2% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
0.84 ? 8% perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_objcg.obj_cgroup_uncharge_pages.kfree.skb_release_data.consume_skb
2.01 ? 2% perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
0.58 ? 6% perf-profile.calltrace.cycles-pp.mod_objcg_state.kfree.skb_release_data.consume_skb.unix_stream_read_generic
1.15 ? 6% perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
0.59 ? 5% perf-profile.calltrace.cycles-pp.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
1.00 ? 7% perf-profile.calltrace.cycles-pp.mutex_unlock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
1.02 ? 8% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
0.67 ? 4% perf-profile.calltrace.cycles-pp.unix_write_space.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all
1.81 ? 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
1.57 ? 5% perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
1.69 ? 7% perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic
1.99 ? 4% perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb
2.03 ? 4% perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic
2.06 ? 4% perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
2.07 ? 4% perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
3.41 ? 7% perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_uncharge.obj_cgroup_uncharge_pages.kfree.skb_release_data
1.72 ? 7% perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node
5.55 ? 6% perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic
8.45 ? 5% perf-profile.calltrace.cycles-pp.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
8.50 ? 5% perf-profile.calltrace.cycles-pp.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
10.30 ? 4% perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
7.32 ? 5% perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg
8.34 ? 5% perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
11.14 ? 5% perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kfree.skb_release_data
3.40 ? 6% perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller
50.41 perf-profile.calltrace.cycles-pp.__libc_read
49.56 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
49.46 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
48.69 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
48.89 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
47.47 perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read
47.56 perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read
48.00 perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
48.22 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.64 ? 4% perf-profile.calltrace.cycles-pp.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
16.71 ? 5% perf-profile.calltrace.cycles-pp.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
16.77 ? 5% perf-profile.calltrace.cycles-pp.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
18.70 ? 4% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
18.75 ? 4% perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
14.72 ? 5% perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kfree.skb_release_data.consume_skb
16.78 ? 4% perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kfree.skb_release_data.consume_skb.unix_stream_read_generic
17.92 ? 4% perf-profile.calltrace.cycles-pp.kfree.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
19.61 ? 4% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
29.89 ? 4% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
29.82 ? 4% perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
7.38 ? 6% perf-profile.calltrace.cycles-pp.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb
22.04 ? 4% perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
31.75 ? 4% perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
14.60 ? 5% perf-profile.calltrace.cycles-pp.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve
0.59 ? 7% perf-profile.children.cycles-pp.page_counter_try_charge
7.83 ? 31% perf-profile.children.cycles-pp.perf_tp_event
7.58 ? 31% perf-profile.children.cycles-pp.perf_swevent_overflow
7.56 ? 31% perf-profile.children.cycles-pp.__perf_event_overflow
7.52 ? 31% perf-profile.children.cycles-pp.perf_event_output_forward
7.34 ? 31% perf-profile.children.cycles-pp.perf_prepare_sample
6.88 ? 32% perf-profile.children.cycles-pp.perf_callchain
6.85 ? 32% perf-profile.children.cycles-pp.get_perf_callchain
7.51 ? 22% perf-profile.children.cycles-pp.schedule
7.61 ? 21% perf-profile.children.cycles-pp.__schedule
7.26 ? 21% perf-profile.children.cycles-pp.__wake_up_common_lock
6.81 ? 21% perf-profile.children.cycles-pp.__wake_up_common
6.70 ? 22% perf-profile.children.cycles-pp.autoremove_wake_function
6.64 ? 22% perf-profile.children.cycles-pp.try_to_wake_up
8.30 ? 17% perf-profile.children.cycles-pp.sock_def_readable
4.50 ? 32% perf-profile.children.cycles-pp.perf_callchain_kernel
6.32 ? 21% perf-profile.children.cycles-pp.schedule_timeout
4.42 ? 28% perf-profile.children.cycles-pp.update_curr
3.96 ? 30% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
3.38 ? 32% perf-profile.children.cycles-pp.unwind_next_frame
2.12 ? 34% perf-profile.children.cycles-pp.perf_trace_sched_switch
2.22 ? 31% perf-profile.children.cycles-pp.perf_callchain_user
2.60 ? 24% perf-profile.children.cycles-pp.ttwu_do_activate
48.42 perf-profile.children.cycles-pp.__libc_write
2.58 ? 24% perf-profile.children.cycles-pp.enqueue_task_fair
2.11 ? 31% perf-profile.children.cycles-pp.__get_user_nocheck_8
2.63 ? 24% perf-profile.children.cycles-pp.dequeue_task_fair
2.21 ? 26% perf-profile.children.cycles-pp.enqueue_entity
2.28 ? 27% perf-profile.children.cycles-pp.dequeue_entity
1.76 ? 32% perf-profile.children.cycles-pp.__unwind_start
1.94 ? 29% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
1.64 ? 19% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
1.56 ? 20% perf-profile.children.cycles-pp.exit_to_user_mode_prepare
1.87 ? 28% perf-profile.children.cycles-pp.asm_exc_page_fault
1.00 ? 31% perf-profile.children.cycles-pp.__orc_find
2.38 ? 21% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
46.38 perf-profile.children.cycles-pp.ksys_write
46.14 perf-profile.children.cycles-pp.vfs_write
45.64 perf-profile.children.cycles-pp.new_sync_write
45.36 perf-profile.children.cycles-pp.sock_write_iter
0.55 ? 32% perf-profile.children.cycles-pp.orc_find
45.09 perf-profile.children.cycles-pp.sock_sendmsg
44.79 perf-profile.children.cycles-pp.unix_stream_sendmsg
0.57 ? 31% perf-profile.children.cycles-pp.unwind_get_return_address
4.86 ? 3% perf-profile.children.cycles-pp._raw_spin_lock
1.12 ? 6% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.46 ? 30% perf-profile.children.cycles-pp.__kernel_text_address
0.44 ? 28% perf-profile.children.cycles-pp.native_irq_return_iret
0.34 ? 31% perf-profile.children.cycles-pp.kernelmode_fixup_or_oops
0.32 ? 34% perf-profile.children.cycles-pp.stack_access_ok
0.31 ? 31% perf-profile.children.cycles-pp.fixup_exception
0.34 ? 30% perf-profile.children.cycles-pp.kernel_text_address
0.28 ? 32% perf-profile.children.cycles-pp.search_exception_tables
0.27 ? 32% perf-profile.children.cycles-pp.search_extable
0.26 ? 31% perf-profile.children.cycles-pp.bsearch
0.90 ? 29% perf-profile.children.cycles-pp.exc_page_fault
97.46 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
0.46 ? 13% perf-profile.children.cycles-pp.select_task_rq_fair
97.26 perf-profile.children.cycles-pp.do_syscall_64
0.42 ? 8% perf-profile.children.cycles-pp.pick_next_task_fair
0.31 ? 15% perf-profile.children.cycles-pp.select_idle_sibling
0.27 ? 28% perf-profile.children.cycles-pp.__perf_event_header__init_id
0.51 ? 9% perf-profile.children.cycles-pp.update_load_avg
0.16 ? 31% perf-profile.children.cycles-pp.cmp_ex_search
0.21 ? 13% perf-profile.children.cycles-pp.reweight_entity
0.19 ? 10% perf-profile.children.cycles-pp.load_new_mm_cr3
0.18 ? 16% perf-profile.children.cycles-pp.select_idle_cpu
0.24 ? 7% perf-profile.children.cycles-pp.switch_fpu_return
0.15 ? 26% perf-profile.children.cycles-pp.perf_event_pid_type
0.25 ? 6% perf-profile.children.cycles-pp.__switch_to
0.11 ? 32% perf-profile.children.cycles-pp.ftrace_graph_ret_addr
0.12 ? 30% perf-profile.children.cycles-pp.perf_output_begin_forward
0.13 ? 17% perf-profile.children.cycles-pp.sched_clock_cpu
0.12 ? 28% perf-profile.children.cycles-pp.__task_pid_nr_ns
0.15 ? 31% perf-profile.children.cycles-pp.perf_trace_sched_migrate_task
0.21 ? 7% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.21 ? 24% perf-profile.children.cycles-pp.set_task_cpu
0.11 ? 18% perf-profile.children.cycles-pp.native_sched_clock
0.11 ? 22% perf-profile.children.cycles-pp.available_idle_cpu
0.15 ? 10% perf-profile.children.cycles-pp.set_next_entity
0.20 ? 7% perf-profile.children.cycles-pp.check_preempt_curr
0.08 ? 48% perf-profile.children.cycles-pp.bad_get_user
0.14 ? 9% perf-profile.children.cycles-pp.__switch_to_asm
0.18 ? 14% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.52 ? 6% perf-profile.children.cycles-pp.prepare_to_wait
0.16 ? 9% perf-profile.children.cycles-pp.check_preempt_wakeup
0.06 ? 47% perf-profile.children.cycles-pp.error_entry
0.12 ? 10% perf-profile.children.cycles-pp.update_rq_clock
0.08 ? 8% perf-profile.children.cycles-pp.__enqueue_entity
0.17 ? 22% perf-profile.children.cycles-pp.finish_task_switch
0.07 ? 48% perf-profile.children.cycles-pp.is_bpf_text_address
0.11 ? 13% perf-profile.children.cycles-pp.__restore_fpregs_from_fpstate
0.12 ? 11% perf-profile.children.cycles-pp.__update_load_avg_se
0.09 ? 20% perf-profile.children.cycles-pp.cpuacct_charge
0.10 ? 11% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.03 ?102% perf-profile.children.cycles-pp.perf_tp_event_match
0.05 ? 45% perf-profile.children.cycles-pp.perf_trace_buf_alloc
0.06 ? 11% perf-profile.children.cycles-pp.put_prev_entity
0.04 ? 75% perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.04 ? 73% perf-profile.children.cycles-pp.get_callchain_entry
0.06 ? 46% perf-profile.children.cycles-pp.perf_trace_buf_update
0.05 ? 51% perf-profile.children.cycles-pp.bpf_ksym_find
0.05 ? 47% perf-profile.children.cycles-pp.get_stack_info
0.08 ? 8% perf-profile.children.cycles-pp.resched_curr
0.04 ? 71% perf-profile.children.cycles-pp.perf_instruction_pointer
0.04 ? 47% perf-profile.children.cycles-pp.___perf_sw_event
0.04 ? 71% perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.07 ? 10% perf-profile.children.cycles-pp.update_min_vruntime
0.08 ? 11% perf-profile.children.cycles-pp.__list_del_entry_valid
0.09 ? 12% perf-profile.children.cycles-pp.preempt_schedule_common
0.05 ? 8% perf-profile.children.cycles-pp.__list_add_valid
0.03 ? 70% perf-profile.children.cycles-pp.iov_iter_init
0.06 ? 7% perf-profile.children.cycles-pp.__calc_delta
0.14 ? 5% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
0.09 ? 6% perf-profile.children.cycles-pp.finish_wait
0.10 ? 4% perf-profile.children.cycles-pp.rcu_read_unlock_strict
0.07 perf-profile.children.cycles-pp.refill_obj_stock
0.07 ? 5% perf-profile.children.cycles-pp.check_stack_object
0.09 ? 7% perf-profile.children.cycles-pp.rcu_all_qs
0.17 ? 4% perf-profile.children.cycles-pp.__ksize
0.06 ? 7% perf-profile.children.cycles-pp.irq_exit_rcu
0.12 ? 3% perf-profile.children.cycles-pp.wait_for_unix_gc
0.16 ? 4% perf-profile.children.cycles-pp.__might_fault
0.22 ? 4% perf-profile.children.cycles-pp.__virt_addr_valid
0.21 ? 4% perf-profile.children.cycles-pp.__might_sleep
0.22 ? 2% perf-profile.children.cycles-pp.security_socket_recvmsg
0.25 perf-profile.children.cycles-pp.sock_recvmsg
0.30 ? 2% perf-profile.children.cycles-pp.copyin
0.27 ? 3% perf-profile.children.cycles-pp.__check_heap_object
0.05 ? 7% perf-profile.children.cycles-pp.load_balance
0.42 ? 2% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.38 ? 2% perf-profile.children.cycles-pp.aa_sk_perm
0.09 ? 13% perf-profile.children.cycles-pp.generic_file_write_iter
0.09 ? 13% perf-profile.children.cycles-pp.__generic_file_write_iter
0.09 ? 13% perf-profile.children.cycles-pp.generic_perform_write
0.06 ? 6% perf-profile.children.cycles-pp.__softirqentry_text_start
0.49 ? 4% perf-profile.children.cycles-pp._copy_from_iter
0.45 ? 3% perf-profile.children.cycles-pp.__entry_text_start
0.08 ? 17% perf-profile.children.cycles-pp.schedule_idle
0.35 ? 3% perf-profile.children.cycles-pp.__build_skb_around
0.11 ? 41% perf-profile.children.cycles-pp.try_to_migrate_one
0.60 ? 5% perf-profile.children.cycles-pp.mutex_lock
1.01 ? 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.14 ? 40% perf-profile.children.cycles-pp.try_to_migrate
0.81 perf-profile.children.cycles-pp.simple_copy_to_iter
0.92 ? 4% perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
0.22 ? 48% perf-profile.children.cycles-pp.remove_migration_pte
0.99 ? 12% perf-profile.children.cycles-pp.__slab_alloc
0.98 ? 11% perf-profile.children.cycles-pp.___slab_alloc
1.15 ? 2% perf-profile.children.cycles-pp.__check_object_size
0.53 ? 7% perf-profile.children.cycles-pp.drain_stock
0.74 ? 19% perf-profile.children.cycles-pp.__unfreeze_partials
0.27 ? 47% perf-profile.children.cycles-pp.remove_migration_ptes
0.60 ? 6% perf-profile.children.cycles-pp.refill_stock
0.67 ? 4% perf-profile.children.cycles-pp.unix_write_space
0.62 ? 7% perf-profile.children.cycles-pp.try_charge_memcg
0.32 ? 45% perf-profile.children.cycles-pp.page_vma_mapped_walk
0.94 ? 3% perf-profile.children.cycles-pp.copyout
1.12 ? 3% perf-profile.children.cycles-pp._copy_to_iter
0.49 ? 39% perf-profile.children.cycles-pp.do_user_addr_fault
1.20 ? 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.41 ? 44% perf-profile.children.cycles-pp.rmap_walk_anon
1.63 ? 7% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.42 ? 44% perf-profile.children.cycles-pp.migrate_misplaced_page
0.42 ? 44% perf-profile.children.cycles-pp.migrate_pages
0.42 ? 45% perf-profile.children.cycles-pp.do_numa_page
0.44 ? 43% perf-profile.children.cycles-pp.__handle_mm_fault
0.45 ? 42% perf-profile.children.cycles-pp.handle_mm_fault
2.22 ? 6% perf-profile.children.cycles-pp.mod_objcg_state
0.43 ? 22% perf-profile.children.cycles-pp.intel_idle
1.01 ? 7% perf-profile.children.cycles-pp.mutex_unlock
1.21 ? 7% perf-profile.children.cycles-pp.skb_queue_tail
0.46 ? 22% perf-profile.children.cycles-pp.cpuidle_enter_state
0.46 ? 22% perf-profile.children.cycles-pp.cpuidle_enter
1.06 ? 7% perf-profile.children.cycles-pp.skb_unlink
1.99 ? 2% perf-profile.children.cycles-pp.skb_copy_datagram_iter
1.98 ? 2% perf-profile.children.cycles-pp.__skb_datagram_iter
2.01 ? 2% perf-profile.children.cycles-pp.unix_stream_read_actor
1.15 ? 6% perf-profile.children.cycles-pp.skb_set_owner_w
0.58 ? 21% perf-profile.children.cycles-pp.start_secondary
0.59 ? 21% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
0.59 ? 21% perf-profile.children.cycles-pp.cpu_startup_entry
0.59 ? 21% perf-profile.children.cycles-pp.do_idle
1.99 ? 4% perf-profile.children.cycles-pp.sock_wfree
2.04 ? 4% perf-profile.children.cycles-pp.unix_destruct_scm
2.06 ? 4% perf-profile.children.cycles-pp.skb_release_head_state
2.07 ? 4% perf-profile.children.cycles-pp.skb_release_all
2.53 ? 8% perf-profile.children.cycles-pp.get_mem_cgroup_from_objcg
2.54 ? 5% perf-profile.children.cycles-pp.__slab_free
4.13 ? 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
10.33 ? 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node
50.55 perf-profile.children.cycles-pp.__libc_read
11.08 ? 7% perf-profile.children.cycles-pp.propagate_protected_usage
48.70 perf-profile.children.cycles-pp.vfs_read
48.91 perf-profile.children.cycles-pp.ksys_read
47.56 perf-profile.children.cycles-pp.unix_stream_recvmsg
47.49 perf-profile.children.cycles-pp.unix_stream_read_generic
48.00 perf-profile.children.cycles-pp.sock_read_iter
48.24 perf-profile.children.cycles-pp.new_sync_read
11.66 ? 4% perf-profile.children.cycles-pp.kmem_cache_free
18.76 ? 4% perf-profile.children.cycles-pp.kmalloc_reserve
18.72 ? 4% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
16.83 ? 6% perf-profile.children.cycles-pp.page_counter_cancel
17.92 ? 4% perf-profile.children.cycles-pp.kfree
25.17 ? 5% perf-profile.children.cycles-pp.obj_cgroup_charge_pages
25.28 ? 5% perf-profile.children.cycles-pp.obj_cgroup_charge
19.62 ? 4% perf-profile.children.cycles-pp.skb_release_data
29.84 ? 4% perf-profile.children.cycles-pp.__alloc_skb
29.89 ? 4% perf-profile.children.cycles-pp.alloc_skb_with_frags
22.04 ? 4% perf-profile.children.cycles-pp.consume_skb
22.56 ? 5% perf-profile.children.cycles-pp.page_counter_uncharge
31.76 ? 4% perf-profile.children.cycles-pp.sock_alloc_send_pskb
25.14 ? 5% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
22.04 ? 5% perf-profile.children.cycles-pp.page_counter_charge
0.17 ? 9% perf-profile.self.cycles-pp.page_counter_try_charge
1.44 ? 33% perf-profile.self.cycles-pp.unwind_next_frame
1.14 ? 32% perf-profile.self.cycles-pp.__get_user_nocheck_8
0.99 ? 32% perf-profile.self.cycles-pp.__orc_find
2.38 ? 21% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.53 ? 32% perf-profile.self.cycles-pp.orc_find
0.39 ? 34% perf-profile.self.cycles-pp.perf_callchain_kernel
0.44 ? 28% perf-profile.self.cycles-pp.native_irq_return_iret
0.29 ? 33% perf-profile.self.cycles-pp.stack_access_ok
0.92 ? 6% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.45 ? 9% perf-profile.self.cycles-pp.__schedule
0.16 ? 30% perf-profile.self.cycles-pp.cmp_ex_search
0.16 ? 30% perf-profile.self.cycles-pp.__unwind_start
0.16 ? 31% perf-profile.self.cycles-pp.kernel_text_address
0.22 ? 10% perf-profile.self.cycles-pp.update_curr
0.11 ? 31% perf-profile.self.cycles-pp.unwind_get_return_address
0.19 ? 10% perf-profile.self.cycles-pp.load_new_mm_cr3
0.10 ? 49% perf-profile.self.cycles-pp.bsearch
0.08 ? 47% perf-profile.self.cycles-pp.perf_callchain_user
0.25 ? 7% perf-profile.self.cycles-pp.__switch_to
0.08 ? 47% perf-profile.self.cycles-pp.get_perf_callchain
0.14 ? 21% perf-profile.self.cycles-pp.perf_tp_event
0.11 ? 30% perf-profile.self.cycles-pp.perf_output_begin_forward
0.11 ? 27% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.09 ? 48% perf-profile.self.cycles-pp.ftrace_graph_ret_addr
0.10 ? 46% perf-profile.self.cycles-pp.__kernel_text_address
0.10 ? 21% perf-profile.self.cycles-pp.available_idle_cpu
0.11 ? 27% perf-profile.self.cycles-pp.perf_prepare_sample
0.10 ? 19% perf-profile.self.cycles-pp.native_sched_clock
0.07 ? 48% perf-profile.self.cycles-pp.asm_exc_page_fault
0.16 ? 3% perf-profile.self.cycles-pp.new_sync_write
0.14 ? 9% perf-profile.self.cycles-pp.__switch_to_asm
0.08 ? 49% perf-profile.self.cycles-pp.perf_trace_sched_switch
0.27 ? 9% perf-profile.self.cycles-pp.update_load_avg
0.14 ? 14% perf-profile.self.cycles-pp.try_to_wake_up
0.08 ? 8% perf-profile.self.cycles-pp.__enqueue_entity
0.11 ? 13% perf-profile.self.cycles-pp.__restore_fpregs_from_fpstate
0.12 ? 10% perf-profile.self.cycles-pp.__update_load_avg_se
0.05 ? 47% perf-profile.self.cycles-pp.error_entry
0.11 ? 16% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
0.09 ? 16% perf-profile.self.cycles-pp.cpuacct_charge
0.10 ? 14% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.12 ? 8% perf-profile.self.cycles-pp.switch_fpu_return
0.03 ?102% perf-profile.self.cycles-pp.perf_tp_event_match
0.18 ? 5% perf-profile.self.cycles-pp.prepare_to_wait
0.07 ? 12% perf-profile.self.cycles-pp.reweight_entity
0.04 ? 71% perf-profile.self.cycles-pp.get_callchain_entry
0.04 ? 75% perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.05 ? 49% perf-profile.self.cycles-pp.bpf_ksym_find
0.10 ? 5% perf-profile.self.cycles-pp.pick_next_task_fair
0.08 ? 13% perf-profile.self.cycles-pp.update_rq_clock
0.08 ? 5% perf-profile.self.cycles-pp.enqueue_entity
0.07 ? 12% perf-profile.self.cycles-pp.select_idle_sibling
0.26 ? 3% perf-profile.self.cycles-pp.sock_write_iter
0.07 ? 18% perf-profile.self.cycles-pp.select_idle_cpu
0.08 ? 8% perf-profile.self.cycles-pp.resched_curr
0.08 ? 4% perf-profile.self.cycles-pp.select_task_rq_fair
0.08 ? 16% perf-profile.self.cycles-pp.enqueue_task_fair
0.06 ? 11% perf-profile.self.cycles-pp.schedule
0.04 ? 71% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.07 ? 6% perf-profile.self.cycles-pp.dequeue_task_fair
0.06 ? 7% perf-profile.self.cycles-pp.update_min_vruntime
0.08 ? 7% perf-profile.self.cycles-pp.do_syscall_64
0.24 ? 2% perf-profile.self.cycles-pp.__libc_read
0.08 ? 8% perf-profile.self.cycles-pp.__list_del_entry_valid
0.05 ? 8% perf-profile.self.cycles-pp.__list_add_valid
0.06 ? 14% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
0.06 ? 6% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.03 ? 70% perf-profile.self.cycles-pp.iov_iter_init
0.06 ? 7% perf-profile.self.cycles-pp.__calc_delta
0.05 ? 8% perf-profile.self.cycles-pp.finish_task_switch
0.20 ? 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.04 ? 45% perf-profile.self.cycles-pp.dequeue_entity
0.04 ? 45% perf-profile.self.cycles-pp.check_preempt_wakeup
0.05 ? 46% perf-profile.self.cycles-pp.__perf_event_header__init_id
0.04 ? 44% perf-profile.self.cycles-pp.sock_sendmsg
0.09 ? 5% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.05 ? 7% perf-profile.self.cycles-pp.schedule_timeout
0.18 ? 4% perf-profile.self.cycles-pp.sock_read_iter
0.06 ? 7% perf-profile.self.cycles-pp.rcu_read_unlock_strict
0.21 ? 3% perf-profile.self.cycles-pp.__libc_write
0.11 ? 6% perf-profile.self.cycles-pp.security_file_permission
0.07 perf-profile.self.cycles-pp.refill_obj_stock
0.06 ? 7% perf-profile.self.cycles-pp.check_stack_object
0.11 ? 6% perf-profile.self.cycles-pp.obj_cgroup_charge
0.07 perf-profile.self.cycles-pp.__cond_resched
0.07 ? 8% perf-profile.self.cycles-pp.rcu_all_qs
0.12 ? 6% perf-profile.self.cycles-pp._copy_from_iter
0.20 ? 4% perf-profile.self.cycles-pp.new_sync_read
0.16 ? 6% perf-profile.self.cycles-pp.__ksize
0.19 ? 3% perf-profile.self.cycles-pp.__might_sleep
0.22 ? 3% perf-profile.self.cycles-pp.aa_sk_perm
0.21 ? 4% perf-profile.self.cycles-pp.__virt_addr_valid
0.21 ? 3% perf-profile.self.cycles-pp.__entry_text_start
0.31 ? 3% perf-profile.self.cycles-pp.__alloc_skb
0.38 ? 4% perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.09 ? 9% perf-profile.self.cycles-pp.skb_copy_datagram_from_iter
0.26 ? 4% perf-profile.self.cycles-pp.__check_heap_object
0.54 ? 2% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.52 ? 5% perf-profile.self.cycles-pp.unix_stream_sendmsg
0.42 ? 2% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.38 ? 5% perf-profile.self.cycles-pp.___slab_alloc
0.28 ? 6% perf-profile.self.cycles-pp.unix_write_space
0.07 ? 7% perf-profile.self.cycles-pp.refill_stock
0.21 ? 3% perf-profile.self.cycles-pp.page_counter_uncharge
0.27 ? 5% perf-profile.self.cycles-pp.__build_skb_around
0.50 ? 4% perf-profile.self.cycles-pp.kfree
0.49 ? 6% perf-profile.self.cycles-pp.mutex_lock
0.59 ? 2% perf-profile.self.cycles-pp.__check_object_size
0.35 ? 9% perf-profile.self.cycles-pp.consume_skb
0.16 ? 45% perf-profile.self.cycles-pp.page_vma_mapped_walk
0.36 ? 7% perf-profile.self.cycles-pp.skb_release_data
1.22 ? 4% perf-profile.self.cycles-pp.unix_stream_read_generic
0.78 ? 3% perf-profile.self.cycles-pp.kmem_cache_free
1.19 ? 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
1.60 ? 6% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.43 ? 22% perf-profile.self.cycles-pp.intel_idle
1.00 ? 7% perf-profile.self.cycles-pp.mutex_unlock
1.20 ? 8% perf-profile.self.cycles-pp.obj_cgroup_uncharge_pages
1.14 ? 6% perf-profile.self.cycles-pp.skb_set_owner_w
1.25 ? 9% perf-profile.self.cycles-pp.obj_cgroup_charge_pages
1.30 ? 7% perf-profile.self.cycles-pp.sock_wfree
1.42 ? 7% perf-profile.self.cycles-pp.sock_def_readable
2.88 ? 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
3.68 ? 5% perf-profile.self.cycles-pp._raw_spin_lock
2.50 ? 8% perf-profile.self.cycles-pp.get_mem_cgroup_from_objcg
2.51 ? 5% perf-profile.self.cycles-pp.__slab_free
10.97 ? 7% perf-profile.self.cycles-pp.propagate_protected_usage
16.66 ? 6% perf-profile.self.cycles-pp.page_counter_cancel
16.74 ? 6% perf-profile.self.cycles-pp.page_counter_charge

hackbench.throughput

-------------------------------------------------+
O |
O O O |
OO OO O O O O O O |
OO O O O O |
O O |
O O O |
O ++ .+.+ |
+.+.++.+.++.+.+.++ : + + +.+.+ .+ |
O : : +.+ +.+ |
: : |
: |
+ |
|
-------------------------------------------------+

hackbench.time.voluntary_context_switches

-------------------------------------------------+
|
|
|
|
++.++. .++.+.+ .+ + |
+ + + + : |
+ + : .++. .+. |
+.+. : ++.+ ++ + |
+ |
|
|
|
O OO O OO OO O OO O OO O OO O OO O OO O O OO O OO O OO |
-------------------------------------------------+

hackbench.time.involuntary_context_switches

-------------------------------------------------+
|
|
|
|
++. +. |
++. .+ +.+ .+ .+ |
+ + + + : |
+ : : .++. .+. |
+.+.: ++.+ ++ + |
+ |
|
|
O OO |
-------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.
Open Source Technology Center
org/hyperkitty/list/lkp@lists.01.org">https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation

Attachments:

(No filename) (113.86 kB)
config-5.15.0-00059-g58056f77502f (168.59 kB)
job-script (7.95 kB)
job.yaml (5.45 kB)
reproduce (529.00 B)
Download all attachments

2021-11-24 17:01:18

by Michal Hocko

[permalink] [raw]

Subject: Re: [memcg, kmem] 58056f7750: hackbench.throughput 10.3% improvement

On Wed 24-11-21 16:34:35, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed a 10.3% improvement of hackbench.throughput due to commit:
>
>
> commit: 58056f77502f3567b760c9a8fc8d2e9081515b2d ("memcg, kmem: further deprecate kmem.limit_in_bytes")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

I am really surprised to see an improvement from this patch. I do not
expect your benchmarking would be using kmem limit. The above patch
hasn't really removed the page counter out of the picture so there
shouldn't be any real reason for performance improvement. I strongly
suspect this is just some benchmark artifact or unreliable evaluation.
--
Michal Hocko
SUSE Labs

2021-11-26 08:15:02

by Michal Hocko

[permalink] [raw]

Subject: Re: [memcg, kmem] 58056f7750: hackbench.throughput 10.3% improvement

On Fri 26-11-21 11:17:48, Oliver Sang wrote:
> Hi Michal Hocko,
>
> On Wed, Nov 24, 2021 at 06:01:12PM +0100, Michal Hocko wrote:
> > On Wed 24-11-21 16:34:35, kernel test robot wrote:
> > >
> > >
> > > Greeting,
> > >
> > > FYI, we noticed a 10.3% improvement of hackbench.throughput due to commit:
> > >
> > >
> > > commit: 58056f77502f3567b760c9a8fc8d2e9081515b2d ("memcg, kmem: further deprecate kmem.limit_in_bytes")
> > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >
> > I am really surprised to see an improvement from this patch. I do not
> > expect your benchmarking would be using kmem limit. The above patch
> > hasn't really removed the page counter out of the picture so there
> > shouldn't be any real reason for performance improvement. I strongly
> > suspect this is just some benchmark artifact or unreliable evaluation.
>
> Fengwei Yin helped further analyze this improvement.
>
> The patch changed the behavior of function obj_cgroup_charge_pages. It's shown
> in the perf-callstack as following line:
>
> 5.63 ? 11% -5.6 0.00 perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb
>
> So Fengwei prepared a patch which reverting the changes in
> obj_cgroup_charge_pages in 58056f7750 (as attached mod.patch)
>
> by this patch, the performance is similar to 16f6bf266c, the improvement
> disappear.

I am still quite surprised and do not understand it. The only practical
difference the said commit has done is
s@page_counter_try_charge@page_counter_charge@

Withtout a limit in place the try_charge always succeeds. There
should be only a single if (new > c->max) branch executed and always
false.
The code is also slightly larger but all that sounds like to little to
make such a larger change. Maybe this is some microarchitecture specific
result. Or can you reproduce on other HW as well.

--
Michal Hocko
SUSE Labs

2021-11-26 03:20:15

by kernel test robot

[permalink] [raw]

Subject: Re: [memcg, kmem] 58056f7750: hackbench.throughput 10.3% improvement

Hi Michal Hocko,

On Wed, Nov 24, 2021 at 06:01:12PM +0100, Michal Hocko wrote:
> On Wed 24-11-21 16:34:35, kernel test robot wrote:
> >
> >
> > Greeting,
> >
> > FYI, we noticed a 10.3% improvement of hackbench.throughput due to commit:
> >
> >
> > commit: 58056f77502f3567b760c9a8fc8d2e9081515b2d ("memcg, kmem: further deprecate kmem.limit_in_bytes")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> I am really surprised to see an improvement from this patch. I do not
> expect your benchmarking would be using kmem limit. The above patch
> hasn't really removed the page counter out of the picture so there
> shouldn't be any real reason for performance improvement. I strongly
> suspect this is just some benchmark artifact or unreliable evaluation.

Fengwei Yin helped further analyze this improvement.

The patch changed the behavior of function obj_cgroup_charge_pages. It's shown
in the perf-callstack as following line:

5.63 ? 11% -5.6 0.00 perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb

So Fengwei prepared a patch which reverting the changes in
obj_cgroup_charge_pages in 58056f7750 (as attached mod.patch)

by this patch, the performance is similar to 16f6bf266c, the improvement
disappear.

=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-9/performance/socket/4/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-cpl-4sp1/hackbench/0x700001e

commit:
16f6bf266c ("mm/list_lru.c: prefer struct_size over open coded arithmetic")
58056f7750 ("memcg, kmem: further deprecate kmem.limit_in_bytes")
ae12af515d ('58056f7750' minus 'changes in obj_cgroup_charge_pages', attached mod.patch)

16f6bf266c94017c 58056f77502f3567b760c9a8fc8 ae12af515da0d557c25f86e89b0
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
124966 +8.8% 136017 ? 2% -0.1% 124791 ? 2% hackbench.throughput
...
5.41 ? 12% -5.4 0.00 +0.3 5.73 ? 13% perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb

detail comparison data as attached 16f6b-58056-ae12a

in brief, the result prove what we suspect. The original patch removed code
- !page_counter_try_charge(&memcg->kmem, nr_pages, &counter)) {
which improved the hackbench throughput. Thanks.

> --
> Michal Hocko
> SUSE Labs

Attachments:

(No filename) (2.73 kB)
mod.patch (4.84 kB)
16f6b-58056-ae12a (698.40 kB)
Download all attachments