LinuxLists.cc - [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput

2015-06-15 07:02:17

Subject: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps

FYI, we
git://git.kernel. commit 52f518a3a7

testcase/path_par
3cb96f0c97330834 ---------------- %stddev \ 1444 ± 1% 899.43 ± 3% 10202 ± 0% 3565 ± 4% 98265 ± 3% 25540 ± 11% 69135723 ± 1% 37347 ± 4% 44 ± 5% 407528 ± 3% 446620 ± 4% 178635 ± 1% 82997 ± 2% 36056 ± 3% 496435 ± 3% 480933 ± 3% 3997 ± 1% 4183 ± 1% 313 ± 15% 313 ± 15% 1016370 ± 1% 1556902 ± 0% 3471142 ± 1% 4270170 ± 1% 2.347e+09 ± 6% 22319046 ± 3% 1.591e+10 ± 9% 7 ± 21% 25540 ± 11% 98265 ± 3% 3565 ± 4% 10202 ± 0% 899.43 ± 3% 69135723 ± 1% 268764 ± 14% 268758 ± 14% 438574 ± 26% 438574 ± 26% 333989 ± 23% 333989 ± 23% 381233 ± 26% 381238 ± 26% 482 ± 4% 251510 ± 9% 289771 ± 8% 454081 ± 12% 521 ± 1% 395201 ± 14% 529 ± 3% 525 ± 6% 51.25 ± 3% 1635 ± 3% 45.20 ± 4% 357.80 ± 3% 53 ± 2% 53 ± 1% 424 ± 2% 62.78 ± 1% 20746 ± 2% 2024 ± 1% 9013 ± 3% 49576 ± 5% 44013 ± 5% 1420966 ± 2% 1420955 ± 2% 53703 ± 5% 8470 ± 1% 162499 ± 5% 5918865 ± 2% 6085763 ± 2% 2579 ± 5% 858 ± 5% 2337 ± 4% 875 ± 7% 455 ± 1% 541 ± 14% 655 ± 20% 69119790 ± 1% 443 ± 3% 190 ± 6% 2157 ± 23% 4912 ± 0% 3.151e+10 ± 0% 16780629 ± 2% 25402986 ± 2% 12072 ± 31% 290547 ± 19% 24407 ± 9% 245753 ± 23% 17.85 ± 4% 0.97 ± 4% 2.19 ± 8% 3.88 ± 9% 6.29 ± 7% 0.94 ± 5% 0.94 ± 5% 4.19 ± 2% 15.10 ± 3% 15.19 ± 3% 15.35 ± 3% 15.44 ± 3% 14.72 ± 3% 1.57 ± 3% 15.03 ± 4% 23.41 ± 3% 1.19 ± 12% 4.24 ± 2% 15.05 ± 4% 2.81 ± 9% 9.12 ± 4% 0.46 ± 2% 2.52 ± 2% 1.58 ± 8% 2.49 ± 1% 15.06 ± 4% 1.35 ± 3% 23.23 ± 3% 5.90 ± 2% 1.72 ± 8% 1.55 ± 6% 1.35 ± 3% 15.75 ± 3% 1.54 ± 5% 3.70 ± 1% 72.13 ± 0% 8.28 ± 5% 3.44 ± 3% 2.75 ± 9% 8.36 ± 5% 0.33 ± 29% 15.13 ± 4% 72.31 ± 0% 14.20 ± 3% 1.30 ± 3% 1.34 ± 5% 8.93 ± 3% 15.18 ± 1% 4.09 ± 0% 8.08 ± 5% 3.87 ± 2% 5.80 ± 1% 0.36 ± 4% 4.72 ± 3% 10.92 ± 4% 2.95 ± 3% 3.83 ± 2% 8.92 ± 3% 7.38 ± 1% 11.19 ± 4% 15.76 ± 4% 7.73 ± 2% 13.61 ± 1% 2.97 ± 2% 7.79 ± 2% 4.63 ± 1% 2.87 ± 2% 6.59 ± 4% 3.04 ± 3% 11.17 ± 4% 23.39 ± 2% 7.03 ± 1% 4.16 ± 0% 4.15 ± 0% 3.05 ± 3% 6.33 ± 4% 6.78 ± 4% 6.75 ± 4% 4.73 ± 3% 10.95 ± 5% 4.62 ± 3% 4.21 ± 1% 1.29 ± 4% 9.15 ± 1% 10.94 ± 5% 0.17 ± 35% 4.32 ± 9% 4.54 ± 3% 16.82 ± 4% 16.81 ± 4% 4.28 ± 2% 24.19 ± 3% 23.93 ± 3% 4.03 ± 1% 9.82 ± 4% 13.12 ± 1% 6.71 ± 1% 8.57 ± 1% 16.71 ± 4% 0.14 ± 15% 0.10 ± 23% 0.08 ± 6% 4.48 ± 3% 4.57 ± 2% 8.57 ± 1% 17.72 ± 4% 4.57 ± 2% 0.96 ± 4% 18.03 ± 4% 4.99 ± 1% 18.00 ± 4% 3.16 ± 1% 4.87 ± 2% 2.36 ± 1% 0.23 ± 6% 0.47 ± 4% 1.37 ± 4% 1.34 ± 4% 0.43 ± 7% 0.54 ± 7% 0.54 ± 7% 0.55 ± 6% 1.28 ± 3% 4.21 ± 0% 5.58 ± 1% 4.77 ± 1% 1.33 ± 4% 1.92 ± 2% 1.93 ± 2% 2.20 ± 3% 16.33 ± 4% 0.99 ± 4% 0.97 ± 4% 1.13 ± 3% 1.15 ± 2% 1.02 ± 2% 1.15 ± 2% 2.34 ± 1% 1.62 ± 4% 1.61 ± 3% 1.56 ± 4% 1.48 ± 3% 0.21 ± 12% 1.72 ± 2% 1.02 ± 2% 1.02 ± 2% 1.02 ± 2% 4.21 ± 1% 528 ± 6% 24162 ± 6% 4882658 ± 9% 37303 ± 1% 77353 ± 9% 38349 ± 2% 38365 ± 2% 38369 ± 2% 38382 ± 2% 38383 ± 2% 38391 ± 2% 38388 ± 2% 38402 ± 2% 38401 ± 2% 38413 ± 2% 548 ± 3% 37450 ± 1% 5772463 ± 8% 76594 ± 8% 25036 ± 3% 38410 ± 2% 38417 ± 2% 38435 ± 2% 38435 ± 2% 38428 ± 2% 38439 ± 1% 38445 ± 2% 38449 ± 2% 38461 ± 2% 38460 ± 1% 76608 ± 7% 546 ± 2% 37465 ± 1% 24985 ± 2% 5775629 ± 7% 76166 ± 8% 5756491 ± 8% 5 ± 31% 37492 ± 1% 75892 ± 8% 37506 ± 1% 5727275 ± 7% 37528 ± 1% 25080 ± 5% 75791 ± 8% 550 ± 4% 6 ± 34% 5717919 ± 7% 37554 ± 1% 71299 ± 11% 5471648 ± 11% 71273 ± 12% 5469742 ± 11% 37563 ± 1% 2807093.70 ± 36% 6 ± 31% 37568 ± 1% 71654 ± 12% 5474236 ± 11% 2807093.70 ± 36% 37574 ± 1% 2611368.18 ± 47% 2611367.85 ± 47% 5469964 ± 11% 4 ± 28% 5460827 ± 11% 37603 ± 1% 5931126 ± 8% 37333 ± 1% 1048433 ± 16% 5501605 ± 11% 71374 ± 11% 37630 ± 1% 4 ± 49% 5435674 ± 11% 37645 ± 1% 7 ± 40% 122 ± 48% 70205 ± 12% 7 ± 44% 37661 ± 1% 5491828 ± 11% 6 ± 32% 4 ± 35% 90 ± 40% 37652 ± 1% 5499468 ± 11% 37651 ± 1% 7 ± 47% 2002338.86 ± 45% 5458480 ± 11% 2002338.86 ± 45% 499 ± 11% 22791 ± 11% 5501548 ± 11% 37675 ± 1% 37701 ± 1% 5460570 ± 11% 72021 ± 8% 5443461 ± 11% 37720 ± 1% 70407 ± 11% 37737 ± 1% 71813 ± 12% 5482457 ± 11% 5478035 ± 11% 37748 ± 1% 5924938 ± 8% 37344 ± 1% 1042225 ± 20% 37765 ± 1% 2514014.52 ± 44% 5791091 ± 11% 2514014.52 ± 44% 76335 ± 12% 76769 ± 11% 2370182.85 ± 47% 5806067 ± 11% 2370183.19 ± 47% 37783 ± 1% 77107 ± 12% 5865705 ± 12% 37808 ± 1% 24950 ± 9% 37805 ± 1% 5827248 ± 12% 76627 ± 13% 546 ± 8% 37804 ± 1% 76396 ± 12% 2052042.52 ± 30% 5789564 ± 11% 2052042.52 ± 30% 37810 ± 1% 5793549 ± 11% 76584 ± 12% 553 ± 8% 37800 ± 1% 25249 ± 8% 76722 ± 13% 5843915 ± 12% 75923 ± 12% 5768738 ± 11% 535 ± 9% 37811 ± 1% 24479 ± 9% 7 ± 32% 37833 ± 1% 5 ± 30% 5826004 ± 12% 544 ± 10% 6 ± 21% 76688 ± 13% 37837 ± 1% 24864 ± 9% 5835422 ± 12% 26232 ± 5% 574 ± 6% 37364 ± 1% 9 ± 49% 5891140 ± 8% 1008386 ± 18% 76125 ± 12% 37838 ± 1% 5784431 ± 11% 76346 ± 13% 2279884.43 ± 47% 5802087 ± 12% 5 ± 29% 2279884.76 ± 47% 37855 ± 1% 134 ± 42% 37858 ± 1% 5839210 ± 12% 76766 ± 13% 5 ± 20% 76865 ± 13% 5 ± 34% 7 ± 44% 3274498.62 ± 42% 5845980 ± 12% 3274498.62 ± 42% 37884 ± 1% 76461 ± 13% 5801262 ± 12% 37904 ± 1% 37919 ± 1% 75128 ± 8% 5733040 ± 8% 74823 ± 8% 2181820.38 ± 44% 5706201 ± 7% 37931 ± 1% 2181820.38 ± 44% 37937 ± 1% 75941 ± 7% 5795769 ± 7% 74784 ± 7% 5703468 ± 7% 37947 ± 1% 5 ± 33% 75045 ± 7% 2349144.49 ± 34% 5717654 ± 7% 2349144.49 ± 34% 37976 ± 1% 26117 ± 3% 37364 ± 1% 78451 ± 8% 571 ± 4% 5828333 ± 8% 945560 ± 28% 92 ± 24% 75109 ± 7% 6 ± 49% 5726100 ± 6% 543 ± 11% 24764 ± 11% 37997 ± 1% 4 ± 34% 5701479 ± 6% 38008 ± 1% 74668 ± 7% 25473 ± 12% 75465 ± 8% 3567331.44 ± 29% 5767898 ± 7% 8 ± 41% 3567331.77 ± 29% 38019 ± 1% 38036 ± 2% 75493 ± 8% 2564844.76 ± 34% 5768160 ± 7% 2564844.76 ± 34% 75523 ± 9% 38055 ± 2% 5757595 ± 8% 5731615 ± 7% 75160 ± 8% 38085 ± 2% 552 ± 13% 5759032 ± 7% 38101 ± 2% 25129 ± 13% 75483 ± 8% 5 ± 29% 74873 ± 8% 5707036 ± 7% 38113 ± 2% 2056049.76 ± 47% 2056049.76 ± 47% 5638172 ± 8% 38110 ± 1% 74655 ± 8% 9 ± 48% 38125 ± 1% 5687789 ± 7% 569 ± 5% 78491 ± 9% 37386 ± 1% 26026 ± 6% 5868941 ± 8% 50960 ± 9% 38133 ± 1% 51720 ± 14% 4337125 ± 13% 38160 ± 1% 3 ± 39% 52202 ± 13% 38148 ± 1% 14412 ± 18% 2 ± 40% 38154 ± 2% 1268924 ± 47% 64 ± 42% 38173 ± 2% 2993 ± 41% 38152 ± 2% 7 ± 47% 38162 ± 2% 10 ± 30% 15875 ± 49% 38175 ± 2% 38206 ± 2% 38221 ± 2% 15369 ± 47% 25441 ± 4% 557 ± 4% 6 ± 45% 5815571 ± 7% 77709 ± 8% 5 ± 33% 37407 ± 1% 38225 ± 2% 38235 ± 2% 38238 ± 2% 38253 ± 2% 9 ± 27% 7 ± 45% 38240 ± 2% 38235 ± 2% 38233 ± 2% 38239 ± 2% 38260 ± 2% 38284 ± 2% 25894 ± 3% 78030 ± 8% 5863768 ± 7% 566 ± 3% 37430 ± 1% 38292 ± 2% 38300 ± 2% 38303 ± 2% 38286 ± 2% 38294 ± 2% 38304 ± 2% 38321 ± 2% 38326 ± 2% 38322 ± 2% 38313 ± 2% 37445 ± 1% 553 ± 4% 25270 ± 4% 77680 ± 8% 5833584 ± 8% 38307 ± 2% 38316 ± 2% 4788 ± 48% 103 ± 49% 38320 ± 2% 38332 ± 2% 38341 ± 2% 38329 ± 2% 38315 ± 2% 38318 ± 2% 38321 ± 2% 38326 ± 2% 2599395.82 ± 28% 37435 ± 1% 7 ± 45% 25086 ± 4% 76760 ± 8% 2599396.15 ± 28% 550 ± 4% 5775897 ± 7% 7 ± 39% +11559.1% 8 ± 35% 10 ± 25% 6 ± 39% +14133.3% 866410 ± 20% 401816 ± 6% 312394 ± 14% 177603 ± 24% 303227 ± 7% 695685 ± 12% 699186 ± 12% 461875 ± 8% 262569 ± 5% 145636 ± 7% 4 ± 36% 3 ± 42% 3 ± 34% 534226 ± 10% 182313 ± 10% 912692 ± 29% 890003 ± 25% 895162 ± 26% 1279055 ± 16% 1210646 ± 23% 873261 ± 25% 864952 ± 24% 887413 ± 31% 854103 ± 27% 864081 ± 28% 898657 ± 28% 3 ± 42% 543726 ± 4% 145709 ± 7% 270996 ± 1% 2 ± 37% 185681 ± 10% 873654 ± 27% 858843 ± 30% 872547 ± 28% 850260 ± 28% 960366 ± 23% 869532 ± 27% 884039 ± 27% 855531 ± 30% 844778 ± 29% 867345 ± 27% 145505 ± 8% 5 ± 44% 186685 ± 11% 4 ± 49% 280891 ± 3% 3 ± 39% 187388 ± 11% 145475 ± 8% 546320 ± 8% 3 ± 27% 282818 ± 5% 145427 ± 7% 4 ± 41% 283614 ± 6% 548763 ± 10% 187787 ± 10% 4 ± 46% 910931 ± 10% 401751 ± 10% 532305 ± 9% 227026 ± 15% 2 ± 46% 907583 ± 10% 147307 ± 8% 271896 ± 28% 228547 ± 14% 146688 ± 7% 538118 ± 10% 401589 ± 9% 912282 ± 10% 274129 ± 10% 909937 ± 10% 399818 ± 10% 251250 ± 33% 146193 ± 8% 907652 ± 10% 905857 ± 10% 535902 ± 9% 535263 ± 9% 904084 ± 11% 274939 ± 9% 226681 ± 14% 905555 ± 11% 146165 ± 7% 399028 ± 10% 146308 ± 8% 529715 ± 9% 296225 ± 10% 3 ± 38% 683586 ± 9% 3 ± 41% 420236 ± 7% 154717 ± 17% 680385 ± 9% 300978 ± 12% 300267 ± 10% 226744 ± 15% 2 ± 41% 539626 ± 11% 3 ± 48% 264020 ± 16% 145629 ± 7% 525878 ± 11% 145597 ± 8% 6 ± 40% 226311 ± 17% 2 ± 47% 282162 ± 10% 2 ± 40% 4 ± 40% 3 ± 34% 145443 ± 7% 2 ± 47% 547461 ± 10% 293095 ± 9% 538395 ± 10% 145290 ± 8% 312631 ± 11% 5 ± 42% 274102 ± 9% 145062 ± 8% 305640 ± 8% 554385 ± 4% 146404 ± 6% 562613 ± 11% 2 ± 47% 291842 ± 8% 3 ± 37% 4 ± 44% 296953 ± 8% 227387 ± 14% 541565 ± 9% 287000 ± 9% 292577 ± 5% 2 ± 49% 2 ± 31% 3 ± 27% 538059 ± 11% 2 ± 40% 858348 ± 11% 583644 ± 14% 188558 ± 11% 861080 ± 11% 3 ± 28% 283363 ± 7% 3 ± 27% 365993 ± 11% 272296 ± 5% 886491 ± 7% 193322 ± 6% 890539 ± 7% 145061 ± 7% 385340 ± 7% 558855 ± 11% 607204 ± 10% 948582 ± 13% 147499 ± 6% 231115 ± 34% 414152 ± 16% 945517 ± 13% 269881 ± 11% 146258 ± 6% 212916 ± 19% 616801 ± 13% 886655 ± 10% 884019 ± 10% 287282 ± 4% 386416 ± 11% 381805 ± 6% 882037 ± 6% 264381 ± 4% 580711 ± 13% 2 ± 37% 198746 ± 7% 879687 ± 6% 145237 ± 7% 558432 ± 12% 271737 ± 6% 144986 ± 7% 192853 ± 6% 255816 ± 24% 144906 ± 7% 554138 ± 12% 379077 ± 6% 879048 ± 7% 876729 ± 7% 192396 ± 6% 144811 ± 7% 198260 ± 3% 3 ± 38% 608120 ± 19% 272324 ± 10% 4 ± 47% 3 ± 48% 144880 ± 8% 551377 ± 10% 193906 ± 7% 277677 ± 3% 4 ± 49% 271911 ± 6% 4 ± 37% 588979 ± 20% 195832 ± 6% 3 ± 42% 859499 ± 7% 197080 ± 5% 144875 ± 8% 861583 ± 7% 372809 ± 7% 585682 ± 15% 254598 ± 28% 384496 ± 7% 903009 ± 6% 900237 ± 6% 556589 ± 9% 193650 ± 19% 279229 ± 6% 564514 ± 13% 2 ± 37% 2 ± 47% 196449 ± 8% 195020 ± 4% 569391 ± 15% 250174 ± 24% 7 ± 46% 271444 ± 4% 3 ± 35% 4 ± 43% 3 ± 27% 594265 ± 15% 198649 ± 7% 270822 ± 9% 3 ± 14% 595801 ± 15% 2 ± 24% 199411 ± 4% 4 ± 25% 5 ± 23% 144742 ± 8% 276311 ± 3% 582806 ± 15% 144761 ± 8% 144710 ± 7% 851364 ± 13% 847235 ± 13% 372059 ± 14% 203310 ± 19% 281126 ± 7% 573414 ± 12% 861853 ± 12% 273192 ± 7% 864903 ± 12% 376696 ± 13% 552783 ± 9% 201314 ± 20% 144703 ± 7% 589 ± 34% 584555 ± 7% 203884 ± 19% 144799 ± 7% 274538 ± 6% 3 ± 35% 144673 ± 8% 543724 ± 4% 199514 ± 22% 2 ± 37% 3 ± 44% 280359 ± 8% 564757 ± 6% 203456 ± 21% 144637 ± 7% 302435 ± 19% 7 ± 35% 576004 ± 11% 184101 ± 10% 256732 ± 5% 2 ± 49% 200344 ± 20% 2 ± 31% 563146 ± 5% 2 ± 41% 244904 ± 22% 144490 ± 7% 273633 ± 7% 555030 ± 5% 202800 ± 21% 204402 ± 20% 6 ± 37% 266105 ± 8% 8 ± 40% 4 ± 33% 590313 ± 10% 246340 ± 20% 580316 ± 10% 202868 ± 20% 275898 ± 8% 580835 ± 12% 203860 ± 21% 561511 ± 12% 199859 ± 21% 269166 ± 3% 592219 ± 10% 203190 ± 20% 285148 ± 8% 289395 ± 8% 201174 ± 22% 561875 ± 9% 532410 ± 13% 199150 ± 23% 281144 ± 9% 206191 ± 20% 290618 ± 4% 568974 ± 8% 145823 ± 7% 179817 ± 10% 546345 ± 8% 278607 ± 3% 2 ± 47% 851921 ± 8% 854405 ± 8% 1 ± 35% 137215 ± 8% 916934 ± 7% 496648 ± 21% 523174 ± 17% 207882 ± 14% 925690 ± 4% 303435 ± 20% 203648 ± 13% 904843 ± 5% 305661 ± 24% 529 ± 32% 1080713 ± 10% 1699969 ± 10% 1096675 ± 9% 2374 ± 48% 1073352 ± 9% 1599502 ± 9% 39322 ± 42% 1409179 ± 19% 1076703 ± 7% 1086785 ± 9% 1411451 ± 16% 38684 ± 43% 1108217 ± 7% 1077891 ± 7% 566207 ± 7% 262075 ± 9% 838568 ± 9% 357789 ± 9% 836222 ± 9% 145736 ± 7% 182859 ± 10% 1073432 ± 10% 1530831 ± 15% 1054705 ± 6% 1069098 ± 9% 38445 ± 45% 1356913 ± 13% 1332336 ± 10% 1058495 ± 8% 2 ± 47% 1067890 ± 8% 682497 ± 27% 693411 ± 28% 734302 ± 27% 681179 ± 28% 540200 ± 8% 178504 ± 11% 145742 ± 7% 361282 ± 8% 276970 ± 8% 846491 ± 7% 843911 ± 7% 1359 ± 48% 682840 ± 27% 680252 ± 27% 688877 ± 33% 670276 ± 22% 680305 ± 22% 1390 ± 44% 711200 ± 26% 649693 ± 26% 531933 ± 7% 180813 ± 10% 855358 ± 8% 857752 ± 8% 366428 ± 8% 255284 ± 21% 145661 ± 7% 3 ± 33% 2 ± 38% 3 ± 40% 880077 ± 25% 1270327 ± 11% 886286 ± 22% 861842 ± 25% 877600 ± 26% 872812 ± 25% 880573 ± 26% 898390 ± 24% 1233750 ± 15% 1279764 ± 17% 887849 ± 23% 881541 ± 22% 883063 ± 25%
brickland3: Memory: 512G

1 ++--------- | | 0.8 ++ | | 0.6 ++ | 0.4 ++ | | 0.2 ++ | | 0 *+-*--*--*-

1 ++--------- | | 0.8 ++ | | 0.6 ++ | 0.4 ++ | | 0.2 ++ | | 0 *+--*--*---

6e+06 | 5e+06 ++ | .*. |.* *.*.*. .* 4e+06 *+ | 3e+06 ++ | 2e+06 ++ | | 1e+06 O+ O | O O 0

7e+07 | 6e+07 *+*.*.*.*.* * | 5e+07 ++ | 4e+07 ++ | 3e+07 ++ | 2e+07 ++ | 1e+07 ++ | 0

4.5e+06 | 4e+06 ++ .*. * 3.5e+06 ++* *.* : * * 3e+06 ++ 2.5e+06 ++ | 2e+06 ++ 1.5e+06 ++ | 1e+06 ++ 500000 ++

1.8e+06 | .** 1.6e+06 *+ .*.* + .* 1.4e+06 ++* | 1.2e+06 ++ 1e+06 ++ | 800000 ++ 600000 ++ | 400000 ++ 200000 O+ O | O

4e+07 3.5e+07 ++ 3e+07 ++ | 2.5e+07 ++ *.*.*.*.* * 2e+07 ++ | 1.5e+07 ++ 1e+07 ++ | 5e+06 ++ |

1.2e+11 | 1e+11 ++ | | 8e+10 ++ | 6e+10 ++ | 4e+10 ++ | 2e+10 *+*.*.*.** | O

1800 ++-------- |.*. .*.*.* * 1600 *+ * 1400 ++ | 1200 ++ 1000 ++ | 800 ++ 600 ++ | 400 ++ 200 ++ O 0 ++O---O---

60 ++---------- | .*.*.* 50 *+*.* | | 40 ++ | 30 ++ | 20 ++ | | 10 ++ O O 0 ++O---O-----

50 *+---------- 45 ++* *.*.* * | + + : : 40 ++ * 35 ++ | 30 ++ 25 ++ 20 ++ | 15 O+ O 10 ++ | 5 ++ 0 ++O---O-----

100 ++O---O---- 90 ++ O 80 ++ 70 ++ | 60 ++ 50 ++ 40 ++ | 30 ++ 20 ++ | 10 ++ .*. : : 0 *+*---*-*-*

60 ++---------- *.*.*.*.*.* * 50 ++ O O | 40 ++ | O O 30 ++ | 20 ++ | | 10 ++ | 0 ++----------

60 ++---------- *.*.*.*.*.* * 50 ++ | O O 40 ++ | O O 30 ++ | 20 ++ | | 10 ++ | 0 ++----------

70 ++---------- | 60 ++ O O 50 ++ O O | 40 ++ | O O 30 ++ | 20 ++ | 10 ++ | .*. : : 0 *+*---*-*-*-

450 *+*---*-*-* | * 400 ++ 350 ++ | 300 ++ 250 ++ | 200 ++ 150 O+ O | 100 ++ 50 ++O O | 0 ++---------

400 ++--------- *.*. .*.*.* * 350 ++ * 300 ++ | 250 ++ | 200 ++ | 150 ++ 100 O+ O | 50 ++ | O O 0 ++---------

1600 ++-------- *.*.*.*.*.* * 1400 ++ 1200 ++ | 1000 O+ O | 800 ++ | 600 ++ 400 ++ | 200 ++ | 0 ++O---O---

1000 ++*-*----- 900 ++ : * : .*.*.* 800 ++ :.* 700 ++ * | 600 ++ 500 ++ 400 ++ | 300 ++ 200 ++ | 100 O+ O 0 ++O---O---

12000 | .*. 10000 ++*.* *.* * * | 8000 ++ | 6000 ++ | 4000 ++ | | 2000 ++ O 0

4000 ++-------- |.*. .*.*.* * 3500 *+ * 3000 ++ | 2500 ++ | 2000 ++ | 1500 ++ 1000 ++ | 500 ++ O 0 ++O---O---

7e+07 | 6e+07 ++ | 5e+07 O+ O | 4e+07 ++ | 3e+07 ++ | 2e+07 ++ | 1e+07 ++ | 0

35000 | 30000 ++ * | + : 25000 ++ * : .* * | : 20000 ++ : *.* 15000 ++ | 10000 ++ | 5000 ++ O 0

1000 ++*-*----- 900 ++ : * : .*.*.* 800 ++ :.* 700 ++ * | 600 ++ 500 ++ 400 ++ | 300 ++ 200 ++ | 100 O+ O 0 ++O---O---

12000 | .*. 10000 ++*.* *.* * * | 8000 ++ | 6000 ++ | 4000 ++ | | 2000 ++ O 0

4000 ++-------- |.*. .*.*.* * 3500 *+ * 3000 ++ | 2500 ++ | 2000 ++ | 1500 ++ 1000 ++ | 500 ++ O 0 ++O---O---

7e+07 | 6e+07 ++ | 5e+07 O+ O | 4e+07 ++ | 3e+07 ++ | 2e+07 ++ | 1e+07 ++ | 0

35000 | 30000 ++ * | + : 25000 ++ * : .* * | : 20000 ++ : *.* 15000 ++ | 10000 ++ | 5000 ++ O 0

50 ++----*---*- | + * : * *.*.* 40 ++ | | 30 ++ | 20 ++ | | 10 ++ | O O 0 ++O---O-----

500000 450000 *+*. .*.*.* * | * 400000 ++ 350000 ++ | 300000 ++ 250000 ++ 200000 ++ | 150000 ++ 100000 ++ | 50000 O+ O 0

1.6e+06 | O 1.4e+06 O+ O | 1.2e+06 ++ 1e+06 ++ | 800000 ++ | 600000 ++ 400000 ++ | .*. : :.* 200000 *+*.* |

1.6e+06 | O 1.4e+06 O+ O | 1.2e+06 ++ 1e+06 ++ | 800000 ++ | 600000 ++ 400000 ++ | .*. : :.* 200000 *+*.* |

1e+06 900000 ++ | O 800000 O+ O 700000 ++ | 600000 ++ 500000 ++ * 400000 ++ | .*. : : 300000 *+*.* 200000 ++ O | O 100000 ++ 0

900000 | O 800000 O+ O 700000 ++ O | 600000 ++ 500000 ++ | 400000 ++ 300000 ++ : : |.*.*.*.*.* *.* 200000 *+ 100000 ++O O | 0

3e+06 | 2.5e+06 ++ | 2e+06 ++ | 1.5e+06 ++ .*.*.** *. *.* 1e+06 ++ | | 500000 ++ | O

3e+06 | 2.5e+06 ++ | 2e+06 ++ | 1.5e+06 ++ .*.*.** *. *.* 1e+06 ++ | | 500000 ++ | O

350000 O O O 300000 ++ O | 250000 ++ | 200000 ++ * * *. + + + *.* 150000 ++*.* * : | 100000 ++ | 50000 ++ | 0

1.2e+07 O O O 1e+07 ++ O | | 8e+06 ++ | 6e+06 *+*.*.*.** * | 4e+06 ++ | | 2e+06 ++ |

1.2e+07 O O 1e+07 ++ | | 8e+06 ++ | .*. .*. 6e+06 *+*.* ** * | 4e+06 ++ | | 2e+06 ++ |

70000 | 60000 ++ | 50000 ++ .* *. |.* + + *.*. 40000 *+ * | 30000 ++ | 20000 ++ O O O O 10000 ++ | 0

60000 | 50000 ++ | .* | * + .*. 40000 ++ * 30000 ++ | 20000 ++ | O O O O 10000 ++ | 0

[*] bisect-good sample
[O] bisect-bad sample

To reproduce:

apt-get install ruby
git clone cd lkp-tests
bin/setup-local bin/run-local job.yaml

Disclaimer:
Results for informational design or

Thanks,
Ying Huang

noticed the below changes on
org/pub/scm/linux/kernel/git/next/linux-next.git master
c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
ams/tbox_group: netperf/performance-300s-200%-TCP_RR/brickland3
52f518a3a7c2f80551a38d38be
--------------------------
%change %stddev
| \
-30.5% 1004 ± 1% netperf.Throughput_tps
-91.6% 75.48 ± 3% netperf.time.user_time
-92.4% 778 ± 1% netperf.time.system_time
-92.5% 268 ± 3% netperf.time.percent_of_cpu_this_job_got
-31.1% 67662 ± 4% netperf.time.minor_page_faults
-88.3% 2991 ± 16% netperf.time.involuntary_context_switches
-30.0% 48364754 ± 1% netperf.time.voluntary_context_switches
+13.1% 42256 ± 3% uptime.idle
-93.2% 3 ± 0% vmstat.procs.r
-26.3% 300511 ± 4% vmstat.system.cs
-92.6% 33083 ± 6% vmstat.system.in
-14.6% 152519 ± 0% meminfo.Active
-29.5% 58518 ± 1% meminfo.Active(anon)
-63.2% 13281 ± 0% meminfo.Shmem
+19.4% 592640 ± 2% numa-meminfo.node0.MemUsed
-7.3% 445927 ± 2% numa-meminfo.node2.MemUsed
+12.1% 4482 ± 3% slabinfo.kmalloc-2048.active_objs
+12.2% 4694 ± 3% slabinfo.kmalloc-2048.num_objs
-78.0% 69 ± 30% slabinfo.tw_sock_TCP.active_objs
-78.0% 69 ± 30% slabinfo.tw_sock_TCP.num_objs
-90.2% 99283 ± 1% softirqs.HRTIMER
-84.0% 248959 ± 2% softirqs.RCU
-92.3% 267496 ± 3% softirqs.SCHED
-82.2% 758706 ± 2% softirqs.TIMER
-63.5% 8.569e+08 ± 39% cpuidle.C1-IVT-4S.time
+62.6% 36299214 ± 3% cpuidle.C1-IVT-4S.usage
+128.3% 3.632e+10 ± 3% cpuidle.C6-IVT-4S.time
+514.0% 44 ± 31% cpuidle.POLL.usage
-88.3% 2991 ± 16% time.involuntary_context_switches
-31.1% 67662 ± 4% time.minor_page_faults
-92.5% 268 ± 3% time.percent_of_cpu_this_job_got
-92.4% 778 ± 1% time.system_time
-91.6% 75.48 ± 3% time.user_time
-30.0% 48364754 ± 1% time.voluntary_context_switches
+408.8% 1367464 ± 3% numa-numastat.node0.numa_hit
+408.8% 1367424 ± 3% numa-numastat.node0.local_node
-45.0% 241074 ± 2% numa-numastat.node1.local_node
-45.0% 241076 ± 2% numa-numastat.node1.numa_hit
-27.9% 240841 ± 3% numa-numastat.node2.local_node
-27.9% 240844 ± 3% numa-numastat.node2.numa_hit
-35.6% 245556 ± 3% numa-numastat.node3.local_node
-35.6% 245561 ± 3% numa-numastat.node3.numa_hit
+11.9% 540 ± 1% numa-vmstat.node0.nr_alloc_batch
+212.0% 784815 ± 8% numa-vmstat.node0.numa_local
+184.0% 823025 ± 7% numa-vmstat.node0.numa_hit
-25.4% 338719 ± 4% numa-vmstat.node1.numa_hit
+13.5% 592 ± 3% numa-vmstat.node1.nr_alloc_batch
-28.9% 281018 ± 4% numa-vmstat.node1.numa_local
+10.1% 583 ± 1% numa-vmstat.node2.nr_alloc_batch
+12.8% 593 ± 1% numa-vmstat.node3.nr_alloc_batch
-92.9% 3.65 ± 3% turbostat.%Busy
-93.0% 114 ± 3% turbostat.Avg_MHz
-70.1% 13.52 ± 21% turbostat.CPU%c1
-71.5% 101.83 ± 6% turbostat.CorWatt
-21.2% 42 ± 6% turbostat.CoreTmp
-20.7% 42 ± 6% turbostat.PkgTmp
-61.0% 165 ± 3% turbostat.PkgWatt
-8.9% 57.17 ± 3% turbostat.RAMWatt
-29.5% 14628 ± 1% proc-vmstat.nr_active_anon
+13.4% 2295 ± 0% proc-vmstat.nr_alloc_batch
-63.2% 3319 ± 0% proc-vmstat.nr_shmem
-60.3% 19667 ± 13% proc-vmstat.numa_hint_faults
-64.6% 15598 ± 16% proc-vmstat.numa_hint_faults_local
+47.3% 2092933 ± 1% proc-vmstat.numa_hit
+47.3% 2092883 ± 1% proc-vmstat.numa_local
-57.9% 22601 ± 13% proc-vmstat.numa_pte_updates
-81.8% 1543 ± 3% proc-vmstat.pgactivate
+90.3% 309245 ± 5% proc-vmstat.pgalloc_dma32
+85.7% 10992818 ± 3% proc-vmstat.pgalloc_normal
+86.0% 11317294 ± 3% proc-vmstat.pgfree
-79.4% 532 ± 2% latency_stats.avg.do_wait.SyS_wait4.system_call_fastpath
+60.1% 1375 ± 4% latency_stats.avg.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
-79.5% 479 ± 2% latency_stats.avg.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
+74.6% 1528 ± 10% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
+108.8% 951 ± 1% latency_stats.avg.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
+137.6% 1287 ± 4% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open
+96.9% 1291 ± 9% latency_stats.avg.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
-30.0% 48351473 ± 1% latency_stats.hits.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
-10.0% 399 ± 2% latency_stats.hits.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
-95.8% 8 ± 38% latency_stats.hits.stop_one_cpu.sched_exec.do_execveat_common.SyS_execve.return_from_execve
+66.0% 3581 ± 22% latency_stats.max.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
+1.6% 4992 ± 0% latency_stats.max.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
+46.1% 4.603e+10 ± 0% latency_stats.sum.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
-78.9% 3539898 ± 4% latency_stats.sum.do_wait.SyS_wait4.system_call_fastpath
-78.5% 5473439 ± 4% latency_stats.sum.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
+112.0% 25588 ± 10% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open
+77.5% 515581 ± 9% latency_stats.sum.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
+74.1% 42503 ± 9% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
+44.5% 355113 ± 9% latency_stats.sum.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
-16.4% 14.92 ± 2% perf-profile.cpu-cycles.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive
-85.4% 0.14 ± 32% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle
-93.6% 0.14 ± 10% perf-profile.cpu-cycles._raw_spin_lock.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
-94.1% 0.23 ± 21% perf-profile.cpu-cycles.load_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
-96.7% 0.21 ± 40% perf-profile.cpu-cycles._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task
-90.7% 0.09 ± 13% perf-profile.cpu-cycles.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule.schedule
-90.6% 0.09 ± 13% perf-profile.cpu-cycles.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule
+61.7% 6.77 ± 3% perf-profile.cpu-cycles.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
-27.5% 10.94 ± 1% perf-profile.cpu-cycles.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
-27.2% 11.07 ± 1% perf-profile.cpu-cycles.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
-27.0% 11.20 ± 1% perf-profile.cpu-cycles.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
-27.0% 11.27 ± 1% perf-profile.cpu-cycles.sys_recvfrom.system_call_fastpath
-26.5% 10.82 ± 1% perf-profile.cpu-cycles.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom
+33.4% 2.10 ± 4% perf-profile.cpu-cycles.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
-25.0% 11.27 ± 2% perf-profile.cpu-cycles.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpu_startup_entry.start_secondary
-80.5% 0.23 ± 19% perf-profile.cpu-cycles.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer
+58.3% 6.72 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry
-24.9% 11.30 ± 2% perf-profile.cpu-cycles.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue
-90.6% 0.26 ± 20% perf-profile.cpu-cycles.find_busiest_group.load_balance.pick_next_task_fair.__schedule.schedule
-52.1% 4.36 ± 4% perf-profile.cpu-cycles.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
+127.4% 1.04 ± 6% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry
-62.9% 0.94 ± 31% perf-profile.cpu-cycles.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
-81.7% 0.29 ± 21% perf-profile.cpu-cycles.tick_sched_handle.isra.18.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt
-64.0% 0.90 ± 33% perf-profile.cpu-cycles.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt
-24.9% 11.32 ± 2% perf-profile.cpu-cycles.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv
+32.3% 1.79 ± 8% perf-profile.cpu-cycles.deactivate_task.__schedule.schedule.schedule_timeout.sk_wait_data
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
-23.8% 4.50 ± 6% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle
-83.2% 0.29 ± 21% perf-profile.cpu-cycles.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer.hrtimer_interrupt
+48.4% 2.31 ± 5% perf-profile.cpu-cycles.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit
+30.7% 1.76 ± 7% perf-profile.cpu-cycles.dequeue_task.deactivate_task.__schedule.schedule.schedule_timeout
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
+48.3% 2.28 ± 5% perf-profile.cpu-cycles.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk
+49.0% 5.51 ± 1% perf-profile.cpu-cycles.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg
-24.2% 54.66 ± 0% perf-profile.cpu-cycles.cpu_startup_entry.start_secondary
-61.0% 3.23 ± 5% perf-profile.cpu-cycles.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg
+51.2% 5.19 ± 1% perf-profile.cpu-cycles.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg
-91.1% 0.24 ± 20% perf-profile.cpu-cycles.update_sd_lb_stats.find_busiest_group.load_balance.pick_next_task_fair.__schedule
-61.0% 3.26 ± 5% perf-profile.cpu-cycles.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg
+218.5% 1.06 ± 10% perf-profile.cpu-cycles.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
-24.2% 11.46 ± 2% perf-profile.cpu-cycles.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish
-24.2% 54.83 ± 0% perf-profile.cpu-cycles.start_secondary
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry
+24.4% 1.62 ± 8% perf-profile.cpu-cycles.dequeue_task_fair.dequeue_task.deactivate_task.__schedule.schedule
+30.0% 1.74 ± 4% perf-profile.cpu-cycles.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary
+26.1% 19.14 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
+9.8% 4.49 ± 4% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns
-62.1% 3.06 ± 6% perf-profile.cpu-cycles.__schedule.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg
+12.2% 4.35 ± 7% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry
-12.5% 5.07 ± 5% perf-profile.cpu-cycles.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll
+188.3% 1.02 ± 5% perf-profile.cpu-cycles.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output
+186.8% 13.52 ± 6% perf-profile.cpu-cycles.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
-11.1% 9.71 ± 4% perf-profile.cpu-cycles.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate
-18.6% 2.40 ± 3% perf-profile.cpu-cycles.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
+14.0% 4.36 ± 7% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary
+13.6% 8.38 ± 2% perf-profile.cpu-cycles.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath
-11.1% 9.95 ± 2% perf-profile.cpu-cycles.ttwu_do_activate.constprop.88.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
-20.9% 12.46 ± 3% perf-profile.cpu-cycles.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
+14.4% 8.85 ± 2% perf-profile.cpu-cycles.SYSC_sendto.sys_sendto.system_call_fastpath
+12.6% 15.31 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
+14.2% 3.40 ± 4% perf-profile.cpu-cycles.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
+14.7% 8.93 ± 2% perf-profile.cpu-cycles.sys_sendto.system_call_fastpath
+24.6% 5.77 ± 1% perf-profile.cpu-cycles.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
+13.8% 3.26 ± 3% perf-profile.cpu-cycles.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry
-17.9% 2.49 ± 3% perf-profile.cpu-cycles.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
-18.6% 9.09 ± 2% perf-profile.cpu-cycles.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up
-12.3% 20.51 ± 1% perf-profile.cpu-cycles.system_call_fastpath
+16.0% 8.15 ± 2% perf-profile.cpu-cycles.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath
+14.6% 4.76 ± 4% perf-profile.cpu-cycles.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick
+14.7% 4.76 ± 4% perf-profile.cpu-cycles.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start
-16.3% 2.55 ± 3% perf-profile.cpu-cycles.schedule_preempt_disabled.cpu_startup_entry.start_secondary
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter
-13.5% 9.47 ± 2% perf-profile.cpu-cycles.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter
+21.2% 5.10 ± 4% perf-profile.cpu-cycles.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
+21.1% 1.57 ± 3% perf-profile.cpu-cycles.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
+16.1% 10.63 ± 2% perf-profile.cpu-cycles.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
-13.5% 9.46 ± 2% perf-profile.cpu-cycles.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
+479.8% 0.96 ± 10% perf-profile.cpu-cycles.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg
-88.6% 0.49 ± 13% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_timeout.sk_wait_data
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr
-15.0% 14.30 ± 3% perf-profile.cpu-cycles.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb
-15.0% 14.28 ± 3% perf-profile.cpu-cycles.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core
-14.4% 3.66 ± 6% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.hrtimer_try_to_cancel
-15.8% 20.36 ± 3% perf-profile.cpu-cycles.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq
-15.8% 20.15 ± 3% perf-profile.cpu-cycles.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action
+15.2% 4.64 ± 3% perf-profile.cpu-cycles.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg
-14.5% 8.40 ± 3% perf-profile.cpu-cycles.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task
+15.9% 15.21 ± 3% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
+18.5% 7.95 ± 2% perf-profile.cpu-cycles.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto
+17.2% 10.05 ± 3% perf-profile.cpu-cycles.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry
-16.1% 14.02 ± 3% perf-profile.cpu-cycles.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv
+573.8% 0.94 ± 9% perf-profile.cpu-cycles.tcp_send_delayed_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
+830.0% 0.93 ± 5% perf-profile.cpu-cycles.ixgbe_xmit_frame.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk
+1213.0% 1.01 ± 4% perf-profile.cpu-cycles.is_module_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr
+16.9% 10.02 ± 3% perf-profile.cpu-cycles.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter
-16.3% 14.84 ± 2% perf-profile.cpu-cycles.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_irq_enter.irq_enter.do_IRQ
-85.4% 0.14 ± 35% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times
-16.6% 15.03 ± 2% perf-profile.cpu-cycles.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll
+17.0% 5.84 ± 2% perf-profile.cpu-cycles.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
-16.7% 15.00 ± 2% perf-profile.cpu-cycles.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq
+20.8% 3.82 ± 3% perf-profile.cpu-cycles.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push
+18.7% 5.78 ± 2% perf-profile.cpu-cycles.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry
+443.4% 1.23 ± 3% perf-profile.cpu-cycles.is_module_text_address.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk
+684.2% 3.65 ± 3% perf-profile.cpu-cycles.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency
+239.8% 4.65 ± 7% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
+243.2% 4.61 ± 7% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry
+1444.2% 6.64 ± 15% perf-profile.cpu-cycles.intel_idle.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
+3863.4% 21.27 ± 3% perf-profile.cpu-cycles.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd
+4049.4% 22.41 ± 2% perf-profile.cpu-cycles.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn
+3975.2% 22.48 ± 2% perf-profile.cpu-cycles.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread
+246.7% 4.43 ± 7% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit
+18.0% 4.97 ± 3% perf-profile.cpu-cycles.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter
-15.9% 4.69 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq
+19.6% 5.70 ± 2% perf-profile.cpu-cycles.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg
+248.1% 4.62 ± 7% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
+283.1% 7.36 ± 2% perf-profile.cpu-cycles.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
+285.6% 7.46 ± 2% perf-profile.cpu-cycles.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task
+19.0% 2.61 ± 4% perf-profile.cpu-cycles.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb
-17.9% 13.40 ± 3% perf-profile.cpu-cycles.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
+2179.8% 22.61 ± 2% perf-profile.cpu-cycles.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
+2217.9% 22.60 ± 2% perf-profile.cpu-cycles.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
+1906.2% 22.67 ± 2% perf-profile.cpu-cycles.smpboot_thread_fn.kthread.ret_from_fork
+1878.6% 22.82 ± 2% perf-profile.cpu-cycles.kthread.ret_from_fork
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_reservations.x86_64_start_kernel
+1879.0% 22.82 ± 2% perf-profile.cpu-cycles.ret_from_fork
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ
+351.6% 0.96 ± 8% perf-profile.cpu-cycles.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
+301.4% 6.90 ± 2% perf-profile.cpu-cycles.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_kernel
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
-100.0% 0.00 ± 0% perf-profile.cpu-cycles.start_kernel.x86_64_start_reservations.x86_64_start_kernel
+20.6% 5.08 ± 4% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
+61.7% 854 ± 1% sched_debug.cfs_rq[0]:/.tg_runnable_contrib
+62.3% 39214 ± 1% sched_debug.cfs_rq[0]:/.avg->runnable_avg_sum
-96.6% 167048 ± 8% sched_debug.cfs_rq[0]:/.min_vruntime
-89.7% 3826 ± 4% sched_debug.cfs_rq[0]:/.tg->runnable_avg
+81.9% 140694 ± 7% sched_debug.cfs_rq[0]:/.exec_clock
-89.4% 4070 ± 3% sched_debug.cfs_rq[100]:/.tg->runnable_avg
-89.4% 4071 ± 3% sched_debug.cfs_rq[101]:/.tg->runnable_avg
-89.4% 4074 ± 3% sched_debug.cfs_rq[102]:/.tg->runnable_avg
-89.4% 4075 ± 3% sched_debug.cfs_rq[103]:/.tg->runnable_avg
-89.4% 4077 ± 3% sched_debug.cfs_rq[104]:/.tg->runnable_avg
-89.4% 4079 ± 3% sched_debug.cfs_rq[105]:/.tg->runnable_avg
-89.4% 4083 ± 3% sched_debug.cfs_rq[106]:/.tg->runnable_avg
-89.4% 4084 ± 3% sched_debug.cfs_rq[107]:/.tg->runnable_avg
-89.4% 4087 ± 3% sched_debug.cfs_rq[108]:/.tg->runnable_avg
-89.4% 4088 ± 3% sched_debug.cfs_rq[109]:/.tg->runnable_avg
-93.4% 36 ± 22% sched_debug.cfs_rq[10]:/.tg_runnable_contrib
-89.7% 3851 ± 4% sched_debug.cfs_rq[10]:/.tg->runnable_avg
-98.4% 93465 ± 22% sched_debug.cfs_rq[10]:/.min_vruntime
-91.7% 6366 ± 15% sched_debug.cfs_rq[10]:/.exec_clock
-93.3% 1688 ± 20% sched_debug.cfs_rq[10]:/.avg->runnable_avg_sum
-89.4% 4089 ± 3% sched_debug.cfs_rq[110]:/.tg->runnable_avg
-89.4% 4088 ± 3% sched_debug.cfs_rq[111]:/.tg->runnable_avg
-89.4% 4091 ± 3% sched_debug.cfs_rq[112]:/.tg->runnable_avg
-89.4% 4092 ± 3% sched_debug.cfs_rq[113]:/.tg->runnable_avg
-89.3% 4095 ± 3% sched_debug.cfs_rq[114]:/.tg->runnable_avg
-89.3% 4096 ± 3% sched_debug.cfs_rq[115]:/.tg->runnable_avg
-89.3% 4095 ± 3% sched_debug.cfs_rq[116]:/.tg->runnable_avg
-89.3% 4097 ± 3% sched_debug.cfs_rq[117]:/.tg->runnable_avg
-89.3% 4100 ± 3% sched_debug.cfs_rq[118]:/.tg->runnable_avg
-89.3% 4102 ± 3% sched_debug.cfs_rq[119]:/.tg->runnable_avg
-91.7% 6356 ± 16% sched_debug.cfs_rq[11]:/.exec_clock
-93.5% 35 ± 25% sched_debug.cfs_rq[11]:/.tg_runnable_contrib
-89.7% 3852 ± 4% sched_debug.cfs_rq[11]:/.tg->runnable_avg
-93.3% 1664 ± 24% sched_debug.cfs_rq[11]:/.avg->runnable_avg_sum
-98.4% 91917 ± 12% sched_debug.cfs_rq[11]:/.min_vruntime
-92.2% 5923 ± 22% sched_debug.cfs_rq[12]:/.exec_clock
-98.5% 86026 ± 20% sched_debug.cfs_rq[12]:/.min_vruntime
-100.0% 0 ± 0% sched_debug.cfs_rq[12]:/.runnable_load_avg
-89.7% 3853 ± 4% sched_debug.cfs_rq[12]:/.tg->runnable_avg
-91.8% 6211 ± 19% sched_debug.cfs_rq[13]:/.exec_clock
-89.7% 3856 ± 4% sched_debug.cfs_rq[13]:/.tg->runnable_avg
-98.4% 93734 ± 18% sched_debug.cfs_rq[13]:/.min_vruntime
-89.7% 3857 ± 4% sched_debug.cfs_rq[14]:/.tg->runnable_avg
-94.4% 1394 ± 43% sched_debug.cfs_rq[14]:/.avg->runnable_avg_sum
-92.2% 5907 ± 24% sched_debug.cfs_rq[14]:/.exec_clock
-94.6% 29 ± 43% sched_debug.cfs_rq[14]:/.tg_runnable_contrib
-100.0% 0 ± 0% sched_debug.cfs_rq[14]:/.runnable_load_avg
-98.4% 90360 ± 22% sched_debug.cfs_rq[14]:/.min_vruntime
-89.7% 3860 ± 4% sched_debug.cfs_rq[15]:/.tg->runnable_avg
-96.4% 2537 ± 18% sched_debug.cfs_rq[15]:/.exec_clock
-99.1% 50259 ± 22% sched_debug.cfs_rq[15]:/.min_vruntime
-96.7% 2353 ± 10% sched_debug.cfs_rq[16]:/.exec_clock
-99.0% 53306 ± 24% sched_debug.cfs_rq[16]:/.min_vruntime
-89.7% 3864 ± 4% sched_debug.cfs_rq[16]:/.tg->runnable_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.MIN_vruntime
-100.0% 0 ± 0% sched_debug.cfs_rq[17]:/.runnable_load_avg
-89.7% 3865 ± 4% sched_debug.cfs_rq[17]:/.tg->runnable_avg
-96.9% 2229 ± 14% sched_debug.cfs_rq[17]:/.exec_clock
-99.1% 50364 ± 25% sched_debug.cfs_rq[17]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.max_vruntime
-89.7% 3866 ± 4% sched_debug.cfs_rq[18]:/.tg->runnable_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.max_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.MIN_vruntime
-99.2% 42263 ± 36% sched_debug.cfs_rq[18]:/.min_vruntime
-100.0% 0 ± 0% sched_debug.cfs_rq[19]:/.runnable_load_avg
-99.2% 46250 ± 33% sched_debug.cfs_rq[19]:/.min_vruntime
-89.7% 3871 ± 4% sched_debug.cfs_rq[19]:/.tg->runnable_avg
-86.0% 829332 ± 28% sched_debug.cfs_rq[1]:/.min_vruntime
-89.7% 3834 ± 4% sched_debug.cfs_rq[1]:/.tg->runnable_avg
-36.8% 662283 ± 34% sched_debug.cfs_rq[1]:/.spread0
-99.1% 46820 ± 29% sched_debug.cfs_rq[20]:/.min_vruntime
-97.4% 1825 ± 22% sched_debug.cfs_rq[20]:/.exec_clock
-89.7% 3876 ± 4% sched_debug.cfs_rq[20]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[20]:/.runnable_load_avg
-99.1% 48697 ± 31% sched_debug.cfs_rq[21]:/.min_vruntime
-89.7% 3875 ± 4% sched_debug.cfs_rq[21]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.load
-100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.utilization_load_avg
-97.3% 1867 ± 36% sched_debug.cfs_rq[21]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[22]:/.load
-89.7% 3876 ± 4% sched_debug.cfs_rq[22]:/.tg->runnable_avg
-99.2% 45143 ± 26% sched_debug.cfs_rq[22]:/.min_vruntime
-100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.load
-100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.runnable_load_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.utilization_load_avg
-89.7% 3879 ± 4% sched_debug.cfs_rq[23]:/.tg->runnable_avg
-99.2% 44829 ± 30% sched_debug.cfs_rq[23]:/.min_vruntime
-89.7% 3882 ± 4% sched_debug.cfs_rq[24]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[24]:/.load
-100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.MIN_vruntime
-99.2% 45851 ± 22% sched_debug.cfs_rq[24]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.max_vruntime
-98.4% 7 ± 49% sched_debug.cfs_rq[24]:/.tg_runnable_contrib
-98.3% 393 ± 47% sched_debug.cfs_rq[24]:/.avg->runnable_avg_sum
-99.2% 42017 ± 39% sched_debug.cfs_rq[25]:/.min_vruntime
-89.7% 3883 ± 4% sched_debug.cfs_rq[25]:/.tg->runnable_avg
-89.7% 3887 ± 4% sched_debug.cfs_rq[26]:/.tg->runnable_avg
-99.2% 42120 ± 20% sched_debug.cfs_rq[26]:/.min_vruntime
-97.9% 1500 ± 39% sched_debug.cfs_rq[26]:/.exec_clock
-99.3% 39281 ± 30% sched_debug.cfs_rq[27]:/.min_vruntime
-89.7% 3890 ± 4% sched_debug.cfs_rq[27]:/.tg->runnable_avg
-97.6% 1657 ± 41% sched_debug.cfs_rq[27]:/.exec_clock
-89.7% 3890 ± 4% sched_debug.cfs_rq[28]:/.tg->runnable_avg
-98.0% 1424 ± 44% sched_debug.cfs_rq[28]:/.exec_clock
-99.2% 42096 ± 24% sched_debug.cfs_rq[28]:/.min_vruntime
-99.2% 41368 ± 31% sched_debug.cfs_rq[29]:/.min_vruntime
-89.7% 3893 ± 4% sched_debug.cfs_rq[29]:/.tg->runnable_avg
-86.5% 799517 ± 29% sched_debug.cfs_rq[2]:/.min_vruntime
-89.7% 3839 ± 4% sched_debug.cfs_rq[2]:/.tg->runnable_avg
-39.3% 632468 ± 35% sched_debug.cfs_rq[2]:/.spread0
-89.7% 3895 ± 4% sched_debug.cfs_rq[30]:/.tg->runnable_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.max_vruntime
-99.1% 49961 ± 26% sched_debug.cfs_rq[30]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.MIN_vruntime
-97.0% 2285 ± 16% sched_debug.cfs_rq[30]:/.exec_clock
-97.4% 2026 ± 16% sched_debug.cfs_rq[31]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.MIN_vruntime
-99.2% 46410 ± 22% sched_debug.cfs_rq[31]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.max_vruntime
-89.7% 3903 ± 4% sched_debug.cfs_rq[31]:/.tg->runnable_avg
-97.4% 1980 ± 30% sched_debug.cfs_rq[32]:/.exec_clock
-99.2% 45792 ± 32% sched_debug.cfs_rq[32]:/.min_vruntime
-89.7% 3909 ± 4% sched_debug.cfs_rq[32]:/.tg->runnable_avg
-96.2% 943 ± 23% sched_debug.cfs_rq[33]:/.avg->runnable_avg_sum
-89.6% 3914 ± 4% sched_debug.cfs_rq[33]:/.tg->runnable_avg
-99.2% 44413 ± 27% sched_debug.cfs_rq[33]:/.min_vruntime
-97.6% 1812 ± 27% sched_debug.cfs_rq[33]:/.exec_clock
-96.4% 19 ± 22% sched_debug.cfs_rq[33]:/.tg_runnable_contrib
-89.6% 3915 ± 4% sched_debug.cfs_rq[34]:/.tg->runnable_avg
-97.7% 1738 ± 29% sched_debug.cfs_rq[34]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.MIN_vruntime
-99.2% 43627 ± 23% sched_debug.cfs_rq[34]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.max_vruntime
-89.6% 3915 ± 4% sched_debug.cfs_rq[35]:/.tg->runnable_avg
-99.3% 40722 ± 15% sched_debug.cfs_rq[35]:/.min_vruntime
-97.8% 1682 ± 27% sched_debug.cfs_rq[35]:/.exec_clock
-98.9% 6 ± 36% sched_debug.cfs_rq[36]:/.tg_runnable_contrib
-89.6% 3917 ± 4% sched_debug.cfs_rq[36]:/.tg->runnable_avg
-98.7% 330 ± 28% sched_debug.cfs_rq[36]:/.avg->runnable_avg_sum
-97.9% 1576 ± 41% sched_debug.cfs_rq[36]:/.exec_clock
-99.3% 40252 ± 43% sched_debug.cfs_rq[36]:/.min_vruntime
-97.7% 1718 ± 38% sched_debug.cfs_rq[37]:/.exec_clock
-99.3% 42192 ± 39% sched_debug.cfs_rq[37]:/.min_vruntime
-98.8% 6 ± 28% sched_debug.cfs_rq[37]:/.tg_runnable_contrib
-89.6% 3924 ± 4% sched_debug.cfs_rq[37]:/.tg->runnable_avg
-98.7% 316 ± 26% sched_debug.cfs_rq[37]:/.avg->runnable_avg_sum
-100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.load
-89.6% 3925 ± 4% sched_debug.cfs_rq[38]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.runnable_load_avg
-99.3% 43456 ± 44% sched_debug.cfs_rq[38]:/.min_vruntime
-98.4% 8 ± 27% sched_debug.cfs_rq[39]:/.tg_runnable_contrib
-100.0% 0 ± 0% sched_debug.cfs_rq[39]:/.runnable_load_avg
-98.2% 1415 ± 35% sched_debug.cfs_rq[39]:/.exec_clock
-89.6% 3925 ± 4% sched_debug.cfs_rq[39]:/.tg->runnable_avg
-98.2% 454 ± 23% sched_debug.cfs_rq[39]:/.avg->runnable_avg_sum
-99.4% 36160 ± 31% sched_debug.cfs_rq[39]:/.min_vruntime
-22.4% 20344 ± 8% sched_debug.cfs_rq[3]:/.avg->runnable_avg_sum
-22.5% 445 ± 8% sched_debug.cfs_rq[3]:/.tg_runnable_contrib
-89.7% 3840 ± 4% sched_debug.cfs_rq[3]:/.tg->runnable_avg
+335.2% 39 ± 44% sched_debug.cfs_rq[3]:/.load
-87.4% 740621 ± 29% sched_debug.cfs_rq[3]:/.min_vruntime
-43.1% 573573 ± 35% sched_debug.cfs_rq[3]:/.spread0
-98.0% 1494 ± 47% sched_debug.cfs_rq[40]:/.exec_clock
-89.6% 3926 ± 4% sched_debug.cfs_rq[40]:/.tg->runnable_avg
-99.3% 38831 ± 36% sched_debug.cfs_rq[40]:/.min_vruntime
-98.1% 1455 ± 41% sched_debug.cfs_rq[41]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.MIN_vruntime
-99.3% 40195 ± 34% sched_debug.cfs_rq[41]:/.min_vruntime
-100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.runnable_load_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.max_vruntime
-89.6% 3929 ± 4% sched_debug.cfs_rq[41]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.utilization_load_avg
-89.6% 3932 ± 4% sched_debug.cfs_rq[42]:/.tg->runnable_avg
-99.3% 39152 ± 47% sched_debug.cfs_rq[42]:/.min_vruntime
-98.2% 1387 ± 46% sched_debug.cfs_rq[42]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[42]:/.runnable_load_avg
-98.1% 1491 ± 35% sched_debug.cfs_rq[43]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.runnable_load_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.load
-100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.max_vruntime
-99.3% 42925 ± 28% sched_debug.cfs_rq[43]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.MIN_vruntime
-89.6% 3934 ± 4% sched_debug.cfs_rq[43]:/.tg->runnable_avg
-97.9% 1582 ± 36% sched_debug.cfs_rq[44]:/.exec_clock
-99.3% 39028 ± 29% sched_debug.cfs_rq[44]:/.min_vruntime
-89.6% 3939 ± 4% sched_debug.cfs_rq[44]:/.tg->runnable_avg
-89.6% 3945 ± 4% sched_debug.cfs_rq[45]:/.tg->runnable_avg
-96.8% 2372 ± 25% sched_debug.cfs_rq[45]:/.exec_clock
-99.2% 47682 ± 38% sched_debug.cfs_rq[45]:/.min_vruntime
-97.4% 1957 ± 15% sched_debug.cfs_rq[46]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.MIN_vruntime
-99.2% 45069 ± 29% sched_debug.cfs_rq[46]:/.min_vruntime
-89.6% 3949 ± 4% sched_debug.cfs_rq[46]:/.tg->runnable_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.max_vruntime
-89.6% 3952 ± 4% sched_debug.cfs_rq[47]:/.tg->runnable_avg
-97.7% 1740 ± 26% sched_debug.cfs_rq[47]:/.exec_clock
-99.3% 43188 ± 29% sched_debug.cfs_rq[47]:/.min_vruntime
-97.9% 1582 ± 27% sched_debug.cfs_rq[48]:/.exec_clock
-99.3% 39942 ± 31% sched_debug.cfs_rq[48]:/.min_vruntime
-89.6% 3954 ± 4% sched_debug.cfs_rq[48]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[49]:/.runnable_load_avg
-97.9% 1592 ± 23% sched_debug.cfs_rq[49]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.MIN_vruntime
-99.3% 41062 ± 37% sched_debug.cfs_rq[49]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.max_vruntime
-89.6% 3956 ± 4% sched_debug.cfs_rq[49]:/.tg->runnable_avg
-50.7% 12870 ± 7% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sum
-89.7% 3843 ± 4% sched_debug.cfs_rq[4]:/.tg->runnable_avg
-47.9% 40905 ± 20% sched_debug.cfs_rq[4]:/.exec_clock
-50.5% 282 ± 7% sched_debug.cfs_rq[4]:/.tg_runnable_contrib
-91.7% 486365 ± 28% sched_debug.cfs_rq[4]:/.min_vruntime
-66.2% 319316 ± 41% sched_debug.cfs_rq[4]:/.spread0
-100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.utilization_load_avg
-97.7% 1699 ± 37% sched_debug.cfs_rq[50]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.load
-99.2% 45365 ± 30% sched_debug.cfs_rq[50]:/.min_vruntime
-98.9% 6 ± 49% sched_debug.cfs_rq[50]:/.tg_runnable_contrib
-98.7% 328 ± 39% sched_debug.cfs_rq[50]:/.avg->runnable_avg_sum
-89.6% 3959 ± 4% sched_debug.cfs_rq[50]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.runnable_load_avg
-99.3% 37520 ± 36% sched_debug.cfs_rq[51]:/.min_vruntime
-89.6% 3959 ± 4% sched_debug.cfs_rq[51]:/.tg->runnable_avg
-98.1% 1411 ± 41% sched_debug.cfs_rq[51]:/.exec_clock
-98.4% 408 ± 48% sched_debug.cfs_rq[52]:/.avg->runnable_avg_sum
-98.1% 1450 ± 23% sched_debug.cfs_rq[52]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.MIN_vruntime
-99.3% 38845 ± 30% sched_debug.cfs_rq[52]:/.min_vruntime
-100.0% 0 ± 0% sched_debug.cfs_rq[52]:/.runnable_load_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.max_vruntime
-89.6% 3962 ± 4% sched_debug.cfs_rq[52]:/.tg->runnable_avg
-89.6% 3966 ± 4% sched_debug.cfs_rq[53]:/.tg->runnable_avg
-98.0% 1520 ± 30% sched_debug.cfs_rq[53]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.max_vruntime
-99.2% 44245 ± 35% sched_debug.cfs_rq[53]:/.min_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.MIN_vruntime
-98.1% 1443 ± 40% sched_debug.cfs_rq[54]:/.exec_clock
-89.6% 3969 ± 4% sched_debug.cfs_rq[54]:/.tg->runnable_avg
-99.3% 41312 ± 28% sched_debug.cfs_rq[54]:/.min_vruntime
-99.3% 38668 ± 27% sched_debug.cfs_rq[55]:/.min_vruntime
-98.0% 1473 ± 39% sched_debug.cfs_rq[55]:/.exec_clock
-89.6% 3971 ± 4% sched_debug.cfs_rq[55]:/.tg->runnable_avg
-99.0% 5 ± 49% sched_debug.cfs_rq[56]:/.tg_runnable_contrib
-99.3% 37467 ± 32% sched_debug.cfs_rq[56]:/.min_vruntime
-89.6% 3980 ± 4% sched_debug.cfs_rq[56]:/.tg->runnable_avg
-98.8% 291 ± 44% sched_debug.cfs_rq[56]:/.avg->runnable_avg_sum
-98.1% 1421 ± 48% sched_debug.cfs_rq[56]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[57]:/.runnable_load_avg
-97.9% 1582 ± 49% sched_debug.cfs_rq[57]:/.exec_clock
-99.2% 43462 ± 45% sched_debug.cfs_rq[57]:/.min_vruntime
-89.6% 3981 ± 4% sched_debug.cfs_rq[57]:/.tg->runnable_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.MIN_vruntime
-100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.max_vruntime
-99.3% 40579 ± 41% sched_debug.cfs_rq[58]:/.min_vruntime
-89.6% 3979 ± 4% sched_debug.cfs_rq[58]:/.tg->runnable_avg
-98.3% 1291 ± 34% sched_debug.cfs_rq[59]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[59]:/.load
-89.6% 3980 ± 4% sched_debug.cfs_rq[59]:/.tg->runnable_avg
-99.3% 37622 ± 41% sched_debug.cfs_rq[59]:/.min_vruntime
-89.4% 60 ± 18% sched_debug.cfs_rq[5]:/.tg_runnable_contrib
-87.5% 9836 ± 6% sched_debug.cfs_rq[5]:/.exec_clock
-89.7% 3845 ± 4% sched_debug.cfs_rq[5]:/.tg->runnable_avg
-89.3% 2782 ± 17% sched_debug.cfs_rq[5]:/.avg->runnable_avg_sum
-97.8% 132022 ± 13% sched_debug.cfs_rq[5]:/.min_vruntime
-99.1% 438 ± 44% sched_debug.cfs_rq[60]:/.exec_clock
-89.6% 3981 ± 4% sched_debug.cfs_rq[60]:/.tg->runnable_avg
-97.9% 1072 ± 40% sched_debug.cfs_rq[61]:/.exec_clock
-99.2% 35916 ± 38% sched_debug.cfs_rq[61]:/.min_vruntime
-89.6% 3984 ± 4% sched_debug.cfs_rq[61]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.load
-98.8% 626 ± 34% sched_debug.cfs_rq[62]:/.exec_clock
-89.5% 3986 ± 4% sched_debug.cfs_rq[62]:/.tg->runnable_avg
-98.6% 194 ± 48% sched_debug.cfs_rq[62]:/.avg->runnable_avg_sum
-100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.runnable_load_avg
-89.6% 3986 ± 4% sched_debug.cfs_rq[63]:/.tg->runnable_avg
-97.9% 26358 ± 34% sched_debug.cfs_rq[64]:/.min_vruntime
-95.8% 2 ± 41% sched_debug.cfs_rq[64]:/.tg_runnable_contrib
-89.6% 3988 ± 4% sched_debug.cfs_rq[64]:/.tg->runnable_avg
-95.1% 148 ± 36% sched_debug.cfs_rq[64]:/.avg->runnable_avg_sum
-89.6% 3985 ± 4% sched_debug.cfs_rq[65]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[65]:/.nr_spread_over
-89.5% 3989 ± 3% sched_debug.cfs_rq[66]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[66]:/.nr_spread_over
-95.7% 680 ± 31% sched_debug.cfs_rq[66]:/.exec_clock
-89.5% 3990 ± 3% sched_debug.cfs_rq[67]:/.tg->runnable_avg
-89.6% 3991 ± 3% sched_debug.cfs_rq[68]:/.tg->runnable_avg
-89.6% 3992 ± 3% sched_debug.cfs_rq[69]:/.tg->runnable_avg
-94.7% 812 ± 43% sched_debug.cfs_rq[69]:/.exec_clock
-91.8% 2076 ± 13% sched_debug.cfs_rq[6]:/.avg->runnable_avg_sum
-92.0% 44 ± 13% sched_debug.cfs_rq[6]:/.tg_runnable_contrib
-100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.load
-98.1% 110120 ± 10% sched_debug.cfs_rq[6]:/.min_vruntime
-89.8% 7891 ± 9% sched_debug.cfs_rq[6]:/.exec_clock
-100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.runnable_load_avg
-89.7% 3849 ± 4% sched_debug.cfs_rq[6]:/.tg->runnable_avg
-89.5% 4001 ± 4% sched_debug.cfs_rq[70]:/.tg->runnable_avg
-89.5% 4005 ± 4% sched_debug.cfs_rq[71]:/.tg->runnable_avg
-89.5% 4007 ± 4% sched_debug.cfs_rq[72]:/.tg->runnable_avg
-89.5% 4007 ± 4% sched_debug.cfs_rq[73]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[73]:/.nr_spread_over
-100.0% 0 ± 0% sched_debug.cfs_rq[74]:/.nr_spread_over
-89.5% 4012 ± 4% sched_debug.cfs_rq[74]:/.tg->runnable_avg
-89.5% 4010 ± 4% sched_debug.cfs_rq[75]:/.tg->runnable_avg
-89.5% 4010 ± 4% sched_debug.cfs_rq[76]:/.tg->runnable_avg
-89.5% 4012 ± 4% sched_debug.cfs_rq[77]:/.tg->runnable_avg
-89.5% 4015 ± 4% sched_debug.cfs_rq[78]:/.tg->runnable_avg
-89.5% 4019 ± 3% sched_debug.cfs_rq[79]:/.tg->runnable_avg
-92.4% 1956 ± 12% sched_debug.cfs_rq[7]:/.avg->runnable_avg_sum
-90.8% 7193 ± 8% sched_debug.cfs_rq[7]:/.exec_clock
-98.2% 105821 ± 12% sched_debug.cfs_rq[7]:/.min_vruntime
-92.6% 42 ± 12% sched_debug.cfs_rq[7]:/.tg_runnable_contrib
-89.7% 3850 ± 4% sched_debug.cfs_rq[7]:/.tg->runnable_avg
-89.5% 4021 ± 3% sched_debug.cfs_rq[80]:/.tg->runnable_avg
-89.5% 4022 ± 3% sched_debug.cfs_rq[81]:/.tg->runnable_avg
-89.5% 4025 ± 3% sched_debug.cfs_rq[82]:/.tg->runnable_avg
-89.5% 4028 ± 3% sched_debug.cfs_rq[83]:/.tg->runnable_avg
-89.5% 4032 ± 3% sched_debug.cfs_rq[84]:/.tg->runnable_avg
-89.5% 4034 ± 3% sched_debug.cfs_rq[85]:/.tg->runnable_avg
-89.5% 4035 ± 3% sched_debug.cfs_rq[86]:/.tg->runnable_avg
-89.5% 4039 ± 3% sched_debug.cfs_rq[87]:/.tg->runnable_avg
-89.5% 4040 ± 3% sched_debug.cfs_rq[88]:/.tg->runnable_avg
-89.4% 4044 ± 3% sched_debug.cfs_rq[89]:/.tg->runnable_avg
-89.7% 3853 ± 4% sched_debug.cfs_rq[8]:/.tg->runnable_avg
-92.0% 44 ± 5% sched_debug.cfs_rq[8]:/.tg_runnable_contrib
-91.8% 2076 ± 5% sched_debug.cfs_rq[8]:/.avg->runnable_avg_sum
-91.2% 6799 ± 6% sched_debug.cfs_rq[8]:/.exec_clock
-98.3% 99293 ± 11% sched_debug.cfs_rq[8]:/.min_vruntime
-89.4% 4044 ± 3% sched_debug.cfs_rq[90]:/.tg->runnable_avg
-89.4% 4047 ± 3% sched_debug.cfs_rq[91]:/.tg->runnable_avg
-95.1% 235 ± 35% sched_debug.cfs_rq[91]:/.avg->runnable_avg_sum
-95.8% 4 ± 39% sched_debug.cfs_rq[91]:/.tg_runnable_contrib
-89.4% 4051 ± 3% sched_debug.cfs_rq[92]:/.tg->runnable_avg
-89.4% 4052 ± 3% sched_debug.cfs_rq[93]:/.tg->runnable_avg
-89.4% 4053 ± 3% sched_debug.cfs_rq[94]:/.tg->runnable_avg
-89.4% 4056 ± 3% sched_debug.cfs_rq[95]:/.tg->runnable_avg
-89.4% 4060 ± 3% sched_debug.cfs_rq[96]:/.tg->runnable_avg
-89.4% 4061 ± 3% sched_debug.cfs_rq[97]:/.tg->runnable_avg
-89.4% 4064 ± 3% sched_debug.cfs_rq[98]:/.tg->runnable_avg
-89.4% 4066 ± 3% sched_debug.cfs_rq[99]:/.tg->runnable_avg
-100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.MIN_vruntime
-89.7% 3849 ± 4% sched_debug.cfs_rq[9]:/.tg->runnable_avg
-100.0% 0 ± 0% sched_debug.cfs_rq[9]:/.runnable_load_avg
-93.1% 1741 ± 14% sched_debug.cfs_rq[9]:/.avg->runnable_avg_sum
-91.3% 6708 ± 10% sched_debug.cfs_rq[9]:/.exec_clock
-100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.max_vruntime
-93.2% 37 ± 15% sched_debug.cfs_rq[9]:/.tg_runnable_contrib
-98.3% 99243 ± 18% sched_debug.cfs_rq[9]:/.min_vruntime
855 ± 0% sched_debug.cpu#0.cpu_load[3]
+9767.3% 855 ± 0% sched_debug.cpu#0.cpu_load[2]
+8443.3% 854 ± 0% sched_debug.cpu#0.cpu_load[1]
854 ± 0% sched_debug.cpu#0.cpu_load[4]
-42.2% 500354 ± 0% sched_debug.cpu#0.max_idle_balance_cost
+5302.7% 21709084 ± 9% sched_debug.cpu#0.ttwu_count
+1631.2% 5408220 ± 18% sched_debug.cpu#1.sched_goidle
-87.5% 22288 ± 17% sched_debug.cpu#1.ttwu_local
-46.2% 163110 ± 12% sched_debug.cpu#1.avg_idle
+1457.9% 10837841 ± 18% sched_debug.cpu#1.nr_switches
+1450.1% 10837966 ± 18% sched_debug.cpu#1.sched_count
-94.9% 23602 ± 15% sched_debug.cpu#1.ttwu_count
+240.9% 895031 ± 14% sched_debug.cpu#10.avg_idle
-30.0% 101980 ± 10% sched_debug.cpu#10.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[1]
-100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[3]
-100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[2]
-99.7% 1612 ± 29% sched_debug.cpu#10.ttwu_count
-99.3% 1351 ± 29% sched_debug.cpu#10.ttwu_local
-45.2% 500000 ± 0% sched_debug.cpu#100.max_idle_balance_cost
-43.5% 502419 ± 0% sched_debug.cpu#101.max_idle_balance_cost
-42.0% 518876 ± 8% sched_debug.cpu#102.max_idle_balance_cost
-18.8% 1038141 ± 8% sched_debug.cpu#102.avg_idle
-19.2% 978478 ± 1% sched_debug.cpu#103.avg_idle
-42.7% 500000 ± 0% sched_debug.cpu#103.max_idle_balance_cost
-42.2% 500000 ± 0% sched_debug.cpu#104.max_idle_balance_cost
-43.7% 500000 ± 0% sched_debug.cpu#105.max_idle_balance_cost
-40.9% 504735 ± 2% sched_debug.cpu#106.max_idle_balance_cost
-42.0% 501124 ± 0% sched_debug.cpu#107.max_idle_balance_cost
-42.5% 516711 ± 6% sched_debug.cpu#108.max_idle_balance_cost
-100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[2]
-99.7% 1760 ± 32% sched_debug.cpu#11.ttwu_count
-30.3% 101564 ± 9% sched_debug.cpu#11.nr_load_updates
+229.9% 894099 ± 14% sched_debug.cpu#11.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[3]
-99.2% 1455 ± 30% sched_debug.cpu#11.ttwu_local
-42.5% 502080 ± 0% sched_debug.cpu#110.max_idle_balance_cost
-41.1% 505597 ± 1% sched_debug.cpu#111.max_idle_balance_cost
-42.7% 500133 ± 0% sched_debug.cpu#112.max_idle_balance_cost
-40.6% 505082 ± 2% sched_debug.cpu#113.max_idle_balance_cost
-47.8% 500984 ± 0% sched_debug.cpu#114.max_idle_balance_cost
-42.2% 502156 ± 0% sched_debug.cpu#115.max_idle_balance_cost
-42.9% 505044 ± 1% sched_debug.cpu#116.max_idle_balance_cost
-41.2% 502901 ± 1% sched_debug.cpu#117.max_idle_balance_cost
-40.2% 505544 ± 1% sched_debug.cpu#118.max_idle_balance_cost
-41.1% 510969 ± 4% sched_debug.cpu#119.max_idle_balance_cost
-34.3% 95650 ± 15% sched_debug.cpu#12.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[1]
-99.3% 1223 ± 33% sched_debug.cpu#12.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[2]
+222.5% 905991 ± 14% sched_debug.cpu#12.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[3]
-99.3% 1223 ± 30% sched_debug.cpu#13.ttwu_local
-31.9% 99013 ± 11% sched_debug.cpu#13.nr_load_updates
-99.7% 1508 ± 21% sched_debug.cpu#13.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#13.cpu_load[1]
+212.5% 883684 ± 13% sched_debug.cpu#13.avg_idle
-34.5% 95288 ± 17% sched_debug.cpu#14.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[3]
+204.3% 862952 ± 17% sched_debug.cpu#14.avg_idle
-99.7% 1508 ± 26% sched_debug.cpu#14.ttwu_count
-99.3% 1258 ± 24% sched_debug.cpu#14.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[2]
-90.9% 83254 ± 28% sched_debug.cpu#15.sched_count
-89.7% 41393 ± 28% sched_debug.cpu#15.sched_goidle
-99.3% 3824 ± 15% sched_debug.cpu#15.ttwu_count
-98.7% 2840 ± 12% sched_debug.cpu#15.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#15.cpu_load[2]
-90.8% 83244 ± 28% sched_debug.cpu#15.nr_switches
-64.1% 52925 ± 18% sched_debug.cpu#15.nr_load_updates
+268.2% 1001194 ± 0% sched_debug.cpu#15.avg_idle
-99.6% 1006 ± 46% sched_debug.cpu#16.ttwu_local
-64.8% 51642 ± 18% sched_debug.cpu#16.nr_load_updates
-99.7% 1836 ± 36% sched_debug.cpu#16.ttwu_count
-89.9% 40532 ± 27% sched_debug.cpu#16.sched_goidle
-91.1% 81551 ± 27% sched_debug.cpu#16.sched_count
+258.6% 983056 ± 2% sched_debug.cpu#16.avg_idle
-91.0% 81542 ± 27% sched_debug.cpu#16.nr_switches
-90.8% 36949 ± 41% sched_debug.cpu#17.sched_goidle
+290.1% 980211 ± 2% sched_debug.cpu#17.avg_idle
-67.5% 47493 ± 30% sched_debug.cpu#17.nr_load_updates
-91.8% 74306 ± 41% sched_debug.cpu#17.sched_count
-91.8% 74297 ± 41% sched_debug.cpu#17.nr_switches
-99.7% 1645 ± 28% sched_debug.cpu#17.ttwu_count
-99.6% 2125 ± 33% sched_debug.cpu#18.ttwu_count
-93.0% 62907 ± 44% sched_debug.cpu#18.nr_switches
+266.2% 1006769 ± 2% sched_debug.cpu#18.avg_idle
-99.5% 1123 ± 40% sched_debug.cpu#18.ttwu_local
-93.1% 62916 ± 43% sched_debug.cpu#18.sched_count
-71.0% 42321 ± 30% sched_debug.cpu#18.nr_load_updates
-92.2% 31022 ± 44% sched_debug.cpu#18.sched_goidle
-71.5% 41686 ± 35% sched_debug.cpu#19.nr_load_updates
-99.6% 2212 ± 43% sched_debug.cpu#19.ttwu_count
+232.9% 986018 ± 1% sched_debug.cpu#19.avg_idle
+605.3% 22 ± 29% sched_debug.cpu#2.cpu_load[2]
+1403.7% 10278920 ± 19% sched_debug.cpu#2.sched_count
+460.9% 21 ± 46% sched_debug.cpu#2.cpu_load[1]
-95.2% 20376 ± 17% sched_debug.cpu#2.ttwu_count
-87.2% 19810 ± 18% sched_debug.cpu#2.ttwu_local
+1410.7% 10278585 ± 19% sched_debug.cpu#2.nr_switches
-50.5% 149090 ± 40% sched_debug.cpu#2.avg_idle
+1608.3% 5129477 ± 19% sched_debug.cpu#2.sched_goidle
-99.7% 791 ± 12% sched_debug.cpu#20.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[3]
-99.7% 1714 ± 13% sched_debug.cpu#20.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[1]
+277.1% 995578 ± 0% sched_debug.cpu#20.avg_idle
-72.8% 39540 ± 38% sched_debug.cpu#20.nr_load_updates
-99.6% 1905 ± 16% sched_debug.cpu#21.ttwu_count
-72.4% 40198 ± 40% sched_debug.cpu#21.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#21.load
-99.6% 954 ± 17% sched_debug.cpu#21.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[3]
+268.8% 1040716 ± 4% sched_debug.cpu#21.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[2]
-100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[1]
-100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[2]
-73.8% 38078 ± 39% sched_debug.cpu#22.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[3]
-99.6% 2080 ± 41% sched_debug.cpu#22.ttwu_count
+242.6% 1004209 ± 2% sched_debug.cpu#22.avg_idle
-99.7% 1761 ± 30% sched_debug.cpu#23.ttwu_count
-77.3% 33007 ± 45% sched_debug.cpu#23.nr_load_updates
+217.5% 992648 ± 1% sched_debug.cpu#23.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#23.load
+261.8% 991658 ± 1% sched_debug.cpu#24.avg_idle
-75.6% 35376 ± 39% sched_debug.cpu#24.nr_load_updates
+215.8% 965318 ± 7% sched_debug.cpu#25.avg_idle
-99.6% 2368 ± 47% sched_debug.cpu#25.ttwu_count
-77.2% 33321 ± 47% sched_debug.cpu#25.nr_load_updates
-99.6% 2233 ± 32% sched_debug.cpu#26.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#26.cpu_load[2]
+239.5% 990850 ± 4% sched_debug.cpu#26.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[2]
-100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[1]
+234.6% 993660 ± 3% sched_debug.cpu#27.avg_idle
-99.6% 875 ± 46% sched_debug.cpu#28.ttwu_local
-99.7% 1729 ± 35% sched_debug.cpu#28.ttwu_count
+247.4% 996956 ± 1% sched_debug.cpu#28.avg_idle
+240.4% 995812 ± 9% sched_debug.cpu#29.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[0]
-100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[1]
-100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[2]
-99.6% 1942 ± 46% sched_debug.cpu#29.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[3]
+988.5% 9343383 ± 19% sched_debug.cpu#3.nr_switches
-96.9% 18374 ± 19% sched_debug.cpu#3.ttwu_count
-90.6% 17658 ± 20% sched_debug.cpu#3.ttwu_local
+985.1% 9343434 ± 19% sched_debug.cpu#3.sched_count
+325.0% 14 ± 46% sched_debug.cpu#3.cpu_load[2]
-36.9% 178917 ± 5% sched_debug.cpu#3.avg_idle
+366.7% 14 ± 35% sched_debug.cpu#3.cpu_load[3]
+1174.1% 4662957 ± 19% sched_debug.cpu#3.sched_goidle
+263.0% 988390 ± 3% sched_debug.cpu#30.avg_idle
-90.7% 82339 ± 23% sched_debug.cpu#30.nr_switches
-99.6% 752 ± 14% sched_debug.cpu#30.ttwu_local
-90.7% 82392 ± 23% sched_debug.cpu#30.sched_count
-66.4% 48735 ± 20% sched_debug.cpu#30.nr_load_updates
-89.4% 40937 ± 23% sched_debug.cpu#30.sched_goidle
-99.7% 1486 ± 14% sched_debug.cpu#30.ttwu_count
-99.7% 1669 ± 28% sched_debug.cpu#31.ttwu_count
-92.4% 72412 ± 23% sched_debug.cpu#31.sched_count
-70.2% 44027 ± 20% sched_debug.cpu#31.nr_load_updates
-99.7% 707 ± 17% sched_debug.cpu#31.ttwu_local
-91.3% 35947 ± 23% sched_debug.cpu#31.sched_goidle
-92.3% 72360 ± 23% sched_debug.cpu#31.nr_switches
+260.8% 973666 ± 4% sched_debug.cpu#31.avg_idle
-72.3% 40536 ± 31% sched_debug.cpu#32.nr_load_updates
-99.7% 682 ± 15% sched_debug.cpu#32.ttwu_local
-99.8% 1439 ± 11% sched_debug.cpu#32.ttwu_count
-92.6% 65901 ± 36% sched_debug.cpu#32.sched_count
-92.6% 65851 ± 36% sched_debug.cpu#32.nr_switches
+221.9% 924747 ± 3% sched_debug.cpu#32.avg_idle
-91.5% 32707 ± 36% sched_debug.cpu#32.sched_goidle
-92.2% 29619 ± 35% sched_debug.cpu#33.sched_goidle
-93.2% 59641 ± 35% sched_debug.cpu#33.sched_count
+251.9% 930329 ± 7% sched_debug.cpu#33.avg_idle
-99.8% 1425 ± 8% sched_debug.cpu#33.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#33.cpu_load[2]
-99.7% 677 ± 13% sched_debug.cpu#33.ttwu_local
-93.2% 59612 ± 35% sched_debug.cpu#33.nr_switches
-73.9% 37912 ± 30% sched_debug.cpu#33.nr_load_updates
-99.7% 1937 ± 40% sched_debug.cpu#34.ttwu_count
+267.3% 998140 ± 8% sched_debug.cpu#34.avg_idle
-74.4% 37118 ± 42% sched_debug.cpu#34.nr_load_updates
-99.6% 781 ± 15% sched_debug.cpu#34.ttwu_local
+287.7% 991833 ± 1% sched_debug.cpu#35.avg_idle
-74.4% 37168 ± 35% sched_debug.cpu#35.nr_load_updates
-99.7% 1813 ± 9% sched_debug.cpu#35.ttwu_count
-92.2% 29594 ± 42% sched_debug.cpu#35.sched_goidle
-93.2% 59608 ± 42% sched_debug.cpu#35.sched_count
-93.2% 59559 ± 42% sched_debug.cpu#35.nr_switches
-99.6% 796 ± 19% sched_debug.cpu#35.ttwu_local
-76.8% 33645 ± 44% sched_debug.cpu#36.nr_load_updates
-99.7% 681 ± 15% sched_debug.cpu#36.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[2]
-99.7% 1607 ± 10% sched_debug.cpu#36.ttwu_count
+271.7% 1012309 ± 4% sched_debug.cpu#36.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[0]
-100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[1]
-75.5% 35491 ± 42% sched_debug.cpu#37.nr_load_updates
-99.7% 1487 ± 10% sched_debug.cpu#37.ttwu_count
-99.6% 680 ± 18% sched_debug.cpu#37.ttwu_local
+256.2% 989065 ± 1% sched_debug.cpu#37.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[1]
+259.7% 978143 ± 1% sched_debug.cpu#38.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[2]
-99.7% 1876 ± 49% sched_debug.cpu#38.ttwu_count
-99.7% 662 ± 13% sched_debug.cpu#38.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[3]
-94.2% 49501 ± 48% sched_debug.cpu#39.nr_switches
-99.7% 616 ± 14% sched_debug.cpu#39.ttwu_local
-78.5% 31180 ± 38% sched_debug.cpu#39.nr_load_updates
-94.2% 50175 ± 46% sched_debug.cpu#39.sched_count
-93.4% 24588 ± 48% sched_debug.cpu#39.sched_goidle
-99.8% 1422 ± 11% sched_debug.cpu#39.ttwu_count
+291.0% 995522 ± 0% sched_debug.cpu#39.avg_idle
+624.1% 2784133 ± 20% sched_debug.cpu#4.sched_goidle
+517.8% 5579241 ± 20% sched_debug.cpu#4.sched_count
+519.7% 5579190 ± 20% sched_debug.cpu#4.nr_switches
-97.9% 11720 ± 21% sched_debug.cpu#4.ttwu_count
-94.2% 11155 ± 20% sched_debug.cpu#4.ttwu_local
+261.3% 1008863 ± 1% sched_debug.cpu#40.avg_idle
-99.8% 1408 ± 18% sched_debug.cpu#40.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[0]
-100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[4]
-99.7% 599 ± 25% sched_debug.cpu#40.ttwu_local
-99.7% 665 ± 32% sched_debug.cpu#41.ttwu_local
-99.7% 1443 ± 22% sched_debug.cpu#41.ttwu_count
+299.0% 998144 ± 0% sched_debug.cpu#41.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#41.load
+267.2% 996765 ± 3% sched_debug.cpu#42.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[2]
-100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[1]
-100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[3]
-99.8% 1417 ± 26% sched_debug.cpu#42.ttwu_count
-99.7% 578 ± 29% sched_debug.cpu#42.ttwu_local
+262.7% 982391 ± 2% sched_debug.cpu#43.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[2]
-99.8% 1474 ± 35% sched_debug.cpu#43.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[3]
-99.7% 613 ± 22% sched_debug.cpu#43.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[1]
-100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[0]
-79.2% 30152 ± 47% sched_debug.cpu#43.nr_load_updates
+258.0% 989070 ± 2% sched_debug.cpu#44.avg_idle
-99.8% 1362 ± 30% sched_debug.cpu#44.ttwu_count
-80.1% 28756 ± 45% sched_debug.cpu#44.nr_load_updates
-67.6% 46849 ± 27% sched_debug.cpu#45.nr_load_updates
-90.0% 85464 ± 31% sched_debug.cpu#45.sched_count
-89.9% 85423 ± 31% sched_debug.cpu#45.nr_switches
-88.6% 42469 ± 31% sched_debug.cpu#45.sched_goidle
-99.6% 843 ± 17% sched_debug.cpu#45.ttwu_local
+253.4% 993587 ± 5% sched_debug.cpu#45.avg_idle
-99.7% 1515 ± 21% sched_debug.cpu#45.ttwu_count
-92.4% 65635 ± 43% sched_debug.cpu#46.nr_switches
+268.5% 1006815 ± 2% sched_debug.cpu#46.avg_idle
-92.4% 65678 ± 43% sched_debug.cpu#46.sched_count
-91.3% 32592 ± 43% sched_debug.cpu#46.sched_goidle
-99.7% 1450 ± 14% sched_debug.cpu#46.ttwu_count
-99.6% 706 ± 19% sched_debug.cpu#46.ttwu_local
-74.6% 36762 ± 34% sched_debug.cpu#46.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#46.curr->pid
-99.7% 1599 ± 16% sched_debug.cpu#47.ttwu_count
-99.6% 723 ± 23% sched_debug.cpu#47.ttwu_local
-76.4% 34218 ± 46% sched_debug.cpu#47.nr_load_updates
+260.1% 988726 ± 1% sched_debug.cpu#47.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[1]
-78.2% 31538 ± 48% sched_debug.cpu#48.nr_load_updates
-99.7% 1672 ± 18% sched_debug.cpu#48.ttwu_count
-99.7% 684 ± 34% sched_debug.cpu#48.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[2]
-100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[0]
+263.5% 1019216 ± 4% sched_debug.cpu#48.avg_idle
-99.7% 1635 ± 18% sched_debug.cpu#49.ttwu_count
-99.7% 710 ± 29% sched_debug.cpu#49.ttwu_local
-78.9% 30527 ± 47% sched_debug.cpu#49.nr_load_updates
+233.8% 1009483 ± 3% sched_debug.cpu#49.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#5.load
-99.5% 2984 ± 18% sched_debug.cpu#5.ttwu_count
-98.6% 2497 ± 17% sched_debug.cpu#5.ttwu_local
+131.1% 593404 ± 27% sched_debug.cpu#5.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[3]
-99.7% 688 ± 28% sched_debug.cpu#50.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[2]
-99.7% 1908 ± 44% sched_debug.cpu#50.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[1]
+302.4% 985599 ± 1% sched_debug.cpu#50.avg_idle
-79.9% 29035 ± 47% sched_debug.cpu#51.nr_load_updates
+265.2% 999259 ± 2% sched_debug.cpu#51.avg_idle
-99.7% 1398 ± 23% sched_debug.cpu#51.ttwu_count
-99.7% 617 ± 28% sched_debug.cpu#51.ttwu_local
-99.7% 672 ± 35% sched_debug.cpu#52.ttwu_local
-100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[2]
+268.6% 980753 ± 3% sched_debug.cpu#52.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[1]
-100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[3]
-99.8% 1394 ± 15% sched_debug.cpu#52.ttwu_count
+293.7% 969729 ± 5% sched_debug.cpu#53.avg_idle
-99.7% 1479 ± 17% sched_debug.cpu#53.ttwu_count
-99.7% 559 ± 24% sched_debug.cpu#53.ttwu_local
+251.8% 970687 ± 6% sched_debug.cpu#54.avg_idle
-99.8% 1420 ± 27% sched_debug.cpu#54.ttwu_count
-99.7% 669 ± 25% sched_debug.cpu#54.ttwu_local
-99.8% 1320 ± 32% sched_debug.cpu#55.ttwu_count
-99.7% 615 ± 44% sched_debug.cpu#55.ttwu_local
+267.9% 990163 ± 1% sched_debug.cpu#55.avg_idle
-99.8% 1394 ± 20% sched_debug.cpu#56.ttwu_count
-99.7% 622 ± 21% sched_debug.cpu#56.ttwu_local
+244.9% 983467 ± 2% sched_debug.cpu#56.avg_idle
+243.8% 994802 ± 1% sched_debug.cpu#57.avg_idle
-99.7% 588 ± 32% sched_debug.cpu#57.ttwu_local
-99.8% 1333 ± 32% sched_debug.cpu#57.ttwu_count
-99.8% 1321 ± 14% sched_debug.cpu#58.ttwu_count
-99.7% 623 ± 30% sched_debug.cpu#58.ttwu_local
+248.6% 979942 ± 4% sched_debug.cpu#58.avg_idle
-99.7% 576 ± 34% sched_debug.cpu#59.ttwu_local
+237.8% 981621 ± 2% sched_debug.cpu#59.avg_idle
-99.8% 1378 ± 21% sched_debug.cpu#59.ttwu_count
-17.5% 120332 ± 3% sched_debug.cpu#6.nr_load_updates
-99.1% 1693 ± 19% sched_debug.cpu#6.ttwu_local
-99.6% 2115 ± 21% sched_debug.cpu#6.ttwu_count
+199.1% 833241 ± 18% sched_debug.cpu#6.avg_idle
-100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[3]
-40.4% 508151 ± 46% sched_debug.cpu#6.nr_switches
-40.5% 508206 ± 46% sched_debug.cpu#6.sched_count
-100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[4]
-92.7% 9982 ± 37% sched_debug.cpu#60.nr_load_updates
-39.4% 555767 ± 14% sched_debug.cpu#60.max_idle_balance_cost
+124.0% 1112535 ± 14% sched_debug.cpu#60.avg_idle
+90.8% 997981 ± 1% sched_debug.cpu#61.avg_idle
-99.9% 283 ± 41% sched_debug.cpu#61.ttwu_local
-45.3% 506045 ± 2% sched_debug.cpu#61.max_idle_balance_cost
-99.8% 571 ± 21% sched_debug.cpu#61.ttwu_count
-99.9% 158 ± 29% sched_debug.cpu#62.ttwu_local
-44.4% 502645 ± 1% sched_debug.cpu#62.max_idle_balance_cost
-99.9% 261 ± 30% sched_debug.cpu#62.ttwu_count
-100.0% 0 ± 0% sched_debug.cpu#62.curr->pid
-53.7% 500459 ± 0% sched_debug.cpu#63.max_idle_balance_cost
-41.0% 1003536 ± 2% sched_debug.cpu#64.avg_idle
-54.0% 504808 ± 2% sched_debug.cpu#64.max_idle_balance_cost
-91.1% 211 ± 27% sched_debug.cpu#64.ttwu_count
-51.1% 524554 ± 7% sched_debug.cpu#65.max_idle_balance_cost
-34.4% 1049932 ± 7% sched_debug.cpu#65.avg_idle
-72.4% 10848 ± 40% sched_debug.cpu#66.nr_load_updates
-27.9% 1015680 ± 3% sched_debug.cpu#66.avg_idle
-52.6% 510769 ± 4% sched_debug.cpu#66.max_idle_balance_cost
-53.0% 510290 ± 4% sched_debug.cpu#67.max_idle_balance_cost
-29.9% 989837 ± 2% sched_debug.cpu#68.avg_idle
-66.9% 12790 ± 34% sched_debug.cpu#68.nr_load_updates
-54.9% 500229 ± 0% sched_debug.cpu#68.max_idle_balance_cost
-47.7% 563733 ± 22% sched_debug.cpu#69.max_idle_balance_cost
-99.7% 1748 ± 21% sched_debug.cpu#7.ttwu_count
+236.5% 881956 ± 13% sched_debug.cpu#7.avg_idle
-53.9% 386340 ± 44% sched_debug.cpu#7.sched_count
-46.2% 192547 ± 44% sched_debug.cpu#7.sched_goidle
-53.8% 386289 ± 44% sched_debug.cpu#7.nr_switches
-22.7% 112707 ± 3% sched_debug.cpu#7.nr_load_updates
-99.2% 1443 ± 23% sched_debug.cpu#7.ttwu_local
-51.3% 522486 ± 8% sched_debug.cpu#70.max_idle_balance_cost
-35.2% 992013 ± 1% sched_debug.cpu#71.avg_idle
-52.6% 500000 ± 0% sched_debug.cpu#71.max_idle_balance_cost
-50.7% 527311 ± 9% sched_debug.cpu#72.max_idle_balance_cost
-61.5% 14811 ± 40% sched_debug.cpu#72.nr_load_updates
-22.4% 1053465 ± 9% sched_debug.cpu#72.avg_idle
-26.8% 975029 ± 6% sched_debug.cpu#73.avg_idle
-51.8% 510063 ± 3% sched_debug.cpu#73.max_idle_balance_cost
-100.0% 0 ± 0% sched_debug.cpu#73.cpu_load[1]
-52.7% 504615 ± 1% sched_debug.cpu#74.max_idle_balance_cost
-26.7% 500478 ± 0% sched_debug.cpu#76.max_idle_balance_cost
-27.4% 503241 ± 1% sched_debug.cpu#77.max_idle_balance_cost
-30.9% 507513 ± 2% sched_debug.cpu#78.max_idle_balance_cost
-21.9% 531938 ± 10% sched_debug.cpu#79.max_idle_balance_cost
-99.7% 1700 ± 13% sched_debug.cpu#8.ttwu_count
-99.2% 1373 ± 19% sched_debug.cpu#8.ttwu_local
-25.5% 108570 ± 3% sched_debug.cpu#8.nr_load_updates
-51.3% 176052 ± 43% sched_debug.cpu#8.sched_goidle
+230.5% 915429 ± 10% sched_debug.cpu#8.avg_idle
-58.3% 353331 ± 43% sched_debug.cpu#8.sched_count
-58.1% 353246 ± 43% sched_debug.cpu#8.nr_switches
-59.4% 552 ± 9% sched_debug.cpu#81.ttwu_count
-26.2% 503819 ± 1% sched_debug.cpu#81.max_idle_balance_cost
-24.4% 513984 ± 4% sched_debug.cpu#82.max_idle_balance_cost
-26.0% 509483 ± 3% sched_debug.cpu#83.max_idle_balance_cost
-24.9% 503665 ± 1% sched_debug.cpu#85.max_idle_balance_cost
-25.8% 504652 ± 1% sched_debug.cpu#86.max_idle_balance_cost
-65.2% 483 ± 6% sched_debug.cpu#87.ttwu_count
-29.2% 503212 ± 1% sched_debug.cpu#88.max_idle_balance_cost
-22.7% 502075 ± 0% sched_debug.cpu#89.max_idle_balance_cost
-99.7% 1700 ± 30% sched_debug.cpu#9.ttwu_count
-99.2% 1424 ± 27% sched_debug.cpu#9.ttwu_local
-58.0% 359635 ± 48% sched_debug.cpu#9.nr_switches
-58.1% 359679 ± 48% sched_debug.cpu#9.sched_count
-51.1% 179269 ± 48% sched_debug.cpu#9.sched_goidle
+256.0% 908885 ± 12% sched_debug.cpu#9.avg_idle
-28.0% 104899 ± 5% sched_debug.cpu#9.nr_load_updates
-100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[2]
-100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[3]
-100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[1]
-43.2% 500000 ± 0% sched_debug.cpu#90.max_idle_balance_cost
-21.3% 1000000 ± 0% sched_debug.cpu#91.avg_idle
-43.6% 500000 ± 0% sched_debug.cpu#91.max_idle_balance_cost
-42.0% 500000 ± 0% sched_debug.cpu#92.max_idle_balance_cost
-37.3% 550542 ± 20% sched_debug.cpu#93.max_idle_balance_cost
-42.7% 500149 ± 0% sched_debug.cpu#94.max_idle_balance_cost
-43.0% 501897 ± 0% sched_debug.cpu#95.max_idle_balance_cost
-43.6% 506730 ± 2% sched_debug.cpu#96.max_idle_balance_cost
-17.9% 1013461 ± 2% sched_debug.cpu#96.avg_idle
-22.4% 993620 ± 1% sched_debug.cpu#97.avg_idle
-43.7% 500000 ± 0% sched_debug.cpu#97.max_idle_balance_cost
-43.0% 502208 ± 0% sched_debug.cpu#98.max_idle_balance_cost
-43.1% 502857 ± 1% sched_debug.cpu#99.max_idle_balance_cost
Brickland Ivy Bridge-EX
stderr.netperf:send_omni:recv_data_failed:No_route_to_host
-------*-----*--------*------*-----------------------------*
: : : : |
: : :: : |
: : : : : : : : :|
: : : : : : : : :|
: : : : : : : : :|
: : : : : : : : :|
: : : : : : : : :|
: : : : : : : : : |
: : : : : : : : : |
: : : : : : : : : |
: : : : : : : : : |
: : : : :: : : |
: : : : : : : |
-*--*-----*-----*--*------*-----*--*--*--*--*--*--*--*--*--+
last_state.booting
---------------------------------------------*-------------*
: |
:: |
: : :|
: : :|
: : :|
: : :|
: : :|
: : : |
: : : |
: : : |
: : : |
: : : |
: : : |
*--*---*--*---*--*---*--*---*--*---*--*---*------*--*---*--+
softirqs.TIMER
++------------------------------------------------------------------+
|
* |
+: |
* * *.* *.* *.* : .*.*.*.*.*.*.*
* : : : : : : : * : : * |
: : :: : : : : : : : : |
: : : : : : : : : :: : : : |
: : : : : : : : : : : : :: |
: : : : : : : : : : : : :: |
: : : : : : : : : : :: |
*: : * : : : : * :: |
O O O O O O O O : O O : O O O OO O O O O |
O* O * : : O : : O * |
++----------O------------O---O-*-O-----*-O--------------------------+
softirqs.NET_RX
++------------------------------------------------------------------+
|
* * *.* *.* * *.*.* *.*.*.*.*.*.*.*
: :: : : : : : : : : : : |
: :: : : : : : : : : : : |
* : :: :: : : : : :: : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : :: |
: : : : : : : : : : : :: |
: : : : : : : : : : :: |
: : :: * : : : : : :: |
*: : : : : : * : |
: : : : : : : |
O+O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-OO-O-O-O-O-------+
softirqs.SCHED
++-----------------------------------------------*----------------+
:: |
* :: |
: * *.* *.* * *.* : *.**.*.*.*.*
: +: : : : : : : : : : + |
* : :: :: : : : : : : : * |
: : : : : : : : : :: : : : |
: : : : : : : : : :: : : : |
: : : : :: : : : :: : : : |
: : : : : : : : : : : : : |
: : : : : : : : : : : : |
: : :: * : : : : : :: |
* : : : : : : * : |
O O OO O +O O O O O : O O :O O O O O O O OO |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
softirqs.RCU
++-----------------------------------------------*----------------+
* .* .* +: .*. |
: * * : * : * *.* : * **.*.*.*.*
* : : : : : : : : : : : |
: :: :: : : : : : : : : |
: : : : : : : : : :: : : : |
: : : : : : : : : :: : : : |
: : : : :: : : : :: : : : |
: : : : : : : : :: : : : |
: : : : : : : : : : : : |
: : :: * : : : : : :: |
* : : : : : : * : |
OO O +O O O O O : O O :O O O O O O OO |
O * * : : O : : O O * |
0 ++----O----O-------O-----O--O-*-O-----*-O-------------------------+
cpuidle.C1-IVT-4S.usage
++----------------------------------------------------------------+
O O OO O O O O O O O O O O O O O O |
O O O |
|
|
|
* * *.* *.* * *.*. *. .**.*. .*.*
*. +: : : : : : : : : * : * * |
* : :: :: : : : : :: : : : |
: : : : : : : : : :: : : : |
: : : : :: : : : :: : : : |
: : : : : : : : : : : : : |
: : :: * : : : : : : : |
*.: : : : : : * : |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
cpuidle.C6-IVT-4S.time
++----------------------------------------------------------------+
|
* * |
: : |
: : * |
: : :: : |
: : :: :: |
: : * : : : : |
: : : : : : : |
: : : : : : : : |
O O OO O: :O O O O O O O O: O O O O O O OO |
: : : : : : : |
* :.*.*.* *.* *.* * *.*.* *.*.**.*.*.*.*
* O + + + : O + + |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
turbostat.Avg_MHz
-----------------------------------------*-----*-----------+
* * .* .* * *. +: + *.*. .*.*.|
: : : : * : * : : : * : * * *
: : : : : : : : : : : : |
: : : : :: : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : |
: : : : : :: : : : : :: :: |
* : : : : : : : : : : |
: : : : : : : : : : |
O O O O *.O O O * O O :O O : O * O O O O O O O O |
--O---O-*-O-*---O---O-*-O--O-*-O---O-----------------------+
turbostat.%Busy
-----------------------------------------------------------+
* * .*. |
: * : * .* .* * *. +: * *.*. .*.*.|
: : : : * : * : : : * : : * *
: : : : : : : : : : : : |
: : : : : :: : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : |
: : : : : : : : : : : : : :: |
* : : :: : : : : : : : |
: : : : : : : : : : |
O O O *.O O O * O O : O O : O * O O O O O O O O |
O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+
turbostat.CPU%c1
---------------------------------------------*-------------+
* * .* .* * *. : *.*.*. .*.*.|
: : * : * : : : *.* : * *
: : : : : : : : : : : : |
: : : : :: : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : O : : : : : O |
O O : O : O: :O:O :: : : : : :: O O |
* : : : : O O : O : O : O O O * O O |
: : : : : : : : : |
*.* : * : : O : : O |
O---O---O-*---O---O-*-O------*-O---------------------------+
turbostat.CPU%c6
-----O---O-*-----------------------O-----------------------+
.* : * O * |
O O O O * O O O : O O O O : O O O * O O |
* : : : : O : O O O |
: : : : : :: : : |
: : : : : : : : : :: |
: : : : : : : : : : : : |
: : : : : : : : : : : : |
: : : : : : : : : : : : |
: : : : : : : : : : : : |
: : : : : : : : : : : : |
: : : : : : : : : : |
: : : : : : : : : |
: : *. .*. : :.*.: : .*. .*
-O-*-----*---*-O---O-*-O---*-*-O-*---*---*---*-*-*-*---*-*-+
turbostat.CoreTmp
-----------------------------------------------------------+
* * .* .* *.*.* .*.*. |
: : : : * : * : * : : *.* *.*.*.*
O : :O: O: :O: : : : O : : :O : : |
O: : : : : :O: :O : O : : O O: : O O:O O O O |
: : : : : : : : : :: : * |
* O.* O * : : : O: : O O |
* * : : : : * |
: : : : |
: : : : |
: : : : |
: : : : |
: : : : |
: : : : |
O-------------O---O-*-O------*-O---------------------------+
turbostat.PkgTmp
-----------------------------------------------------------+
* * .* .* *.*.* .*.*. .*
: : : : * : * : * : : * *.*.*.* |
O : : : O: :O: : : : O : : :O : :O |
O: :O: : : :O: :O : O : : O O: : O O: O O O |
: : : : : : : : : :: : * |
* O.* O * : : : O: : O O |
* * : : : : * |
: : : : |
: : : : |
: : : : |
: : : : |
: : : : |
: : : : |
O-------------O---O-*-O------*-O---------------------------+
turbostat.Pkg%pc2
-----------------------------------------------------------+
O |
O O O O |
O O O O O |
O O O O O |
O |
O |
O O * O |
.* : O * |
* * : : : * * : |
: : : : : :: : : : |
: : : : : : : : : : : : |
: : : : : : : : : : : : |
: : *. .*. : :.*.: : .*. |
O-*-----*---*-O---O-*-O----*-*-O-*---*---*---*-*-*-*---*-*-*
turbostat.PkgWatt
---------*---*-----*-------*-------------------*-------*---+
: * : : * : * : * *.*.* * *.*.* *.*
: : : : : : : : : : : : |
: : :: :: : : : : : : : : |
: : : : : : : : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
O O : O : O: O:O ::O :O :O :O :O: :O O O O O O O O |
* : : : : : : : : : * |
*.: : : : : O : : : |
O * O * * : : : : O |
: : : : |
-O-------------O---O-*-O-----*-O---------------------------+
turbostat.CorWatt
-----------------------------------------------------------+
* * .* .* * *. .* *.*.*.*. .*.*.|
: : : : * : * : : : * : : * *
: : : : : : : : : : : : |
: : : :: :: : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
O : O : O: O: : : :O : :O : O O O |
O * : : : O : O O : O : : O O O * O O |
*.: : * : : O : : * |
O * O * : : : : O |
-O-------------O---O-*-O-----*-O---------------------------+
netperf.Throughput_tps
-----------------------------------------------------------+
* * *.* *.* *.*.* *.*.*. .*.*. .*
: : : : : : : : * : : : * * |
: : : : : : : : : : : : |
: : : : : : : : : : : : |
O O: :O: O: :O:O: :O :O O :O O: :O O O:O:O O O O |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : |
: : : : : : : : : : : |
: : : : : : : : : : : |
--O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.user_time
-----------------------------------------------------------+
* *.*.*. .*.*.|
*. *. : : *.* *
: * * : * : * : : |
: : : : : : : :: : |
: : : : : : : : : *.*.* : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : :: : : |
: : : : : : : : : : :: |
* : : * : : : : * : |
O O O +O O O O O :O O : O O O O O O O O O |
--O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.system_time
++------------------------------------------------------------------+
* .* |
: * *.* *.* *.* : .*.*.*.*.*.*.*
+ +: : : : : : : * : : * |
* : : : : : : : : : : : |
: : : :: : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : :: |
: : : : : : : : : : : :: |
: : : : : : : : : : :: |
:: :: * : : : : : :: |
*: : : : : : * : |
O O O O O O O O O : O O : O O O OO O O O O |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
netperf.time.percent_of_cpu_this_job_got
-----------------------------------------*-----------------+
* * .* .* *. +: .*.*.*. .*.*.|
: : : : * : * : * : * : * * *
: : : : : : : : : : : : |
: : : : :: : : : : : : : : |
: : : : : : : : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : |
: : : : : :: : : : : : : : : |
* : : : : : : : : : :: |
: : : : : : : : : : |
O O O O *.O O O * O O :O O : O * O O O O O O O O |
--O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.voluntary_context_switches
*+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-*
: :: : : : : : : * : : : |
: :: : : : : : : : : : : |
* : : :: : : : : : : : : |
O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O |
: : : : : : : : : : : : : :O |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : :: |
: : : : : : : : : : : : :: |
: : : : : : : : : : :: |
: : : : : : : : : : :: |
:: : * : : : : * : |
*: : : : : : : |
: : : : : : : |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
netperf.time.involuntary_context_switches
++------------------------------------------------------------------+
*. |
+ * .* |
* * * : * : * *
: : *.* *.* : : : : + + +|
* + +: : :: : : : : * : : : * *.* |
* : : : : : : : : : : : : : |
: : : : : : : : : :: : : * |
: : : : : : : : : : : : : : |
: : : : :: : : : : : : :: |
: : : : : : : : : :: :: |
: : : : * : : : : : :: |
*: : : : : : * ::O |
O O O O O O O O O : O O : O O O OO O O O |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.user_time
-----------------------------------------------------------+
* *.*.*. .*.*.|
*. *. : : *.* *
: * * : * : * : : |
: : : : : : : :: : |
: : : : : : : : : *.*.* : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : :: : : |
: : : : : : : : : : :: |
* : : * : : : : * : |
O O O +O O O O O :O O : O O O O O O O O O |
--O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
time.system_time
++------------------------------------------------------------------+
* .* |
: * *.* *.* *.* : .*.*.*.*.*.*.*
+ +: : : : : : : * : : * |
* : : : : : : : : : : : |
: : : :: : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : :: |
: : : : : : : : : : : :: |
: : : : : : : : : : :: |
:: :: * : : : : : :: |
*: : : : : : * : |
O O O O O O O O O : O O : O O O OO O O O O |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.percent_of_cpu_this_job_got
-----------------------------------------*-----------------+
* * .* .* *. +: .*.*.*. .*.*.|
: : : : * : * : * : * : * * *
: : : : : : : : : : : : |
: : : : :: : : : : : : : : |
: : : : : : : : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : |
: : : : : :: : : : : : : : : |
* : : : : : : : : : :: |
: : : : : : : : : : |
O O O O *.O O O * O O :O O : O * O O O O O O O O |
--O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
time.voluntary_context_switches
*+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-*
: :: : : : : : : * : : : |
: :: : : : : : : : : : : |
* : : :: : : : : : : : : |
O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O |
: : : : : : : : : : : : : :O |
: : : : : : : : : : : : : : |
: : : : : : : : : : : : :: |
: : : : : : : : : : : : :: |
: : : : : : : : : : :: |
: : : : : : : : : : :: |
:: : * : : : : * : |
*: : : : : : : |
: : : : : : : |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.involuntary_context_switches
++------------------------------------------------------------------+
*. |
+ * .* |
* * * : * : * *
: : *.* *.* : : : : + + +|
* + +: : :: : : : : * : : : * *.* |
* : : : : : : : : : : : : : |
: : : : : : : : : :: : : * |
: : : : : : : : : : : : : : |
: : : : :: : : : : : : :: |
: : : : : : : : : :: :: |
: : : : * : : : : : :: |
*: : : : : : * ::O |
O O O O O O O O O : O O : O O O OO O O O |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
vmstat.procs.r
-----------------------------------------------------------+
* * * * *. .* .*.*.* *. |
: : : : +: +: * : * : * + + *.*
: : : : * : * : : : : : * |
: : : : :: : : : : : : : : |
: : : : : : : : : : : :: : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : |
: : : : : :: : : : : : : :: |
* : : : : : : : : : : |
: : : : : : : : : : |
O O O *.O O O * O O : O O : O * O O O O O O O O |
O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+
vmstat.system.in
++-----------------------------------------------------------------+
* * .* .* * *. .* *.**.*. .*.*.|
: : : : * : * : : : * : : * *
: : : : : : : : : : : : |
: : : : :: : : : : :: : : : |
: : : : : : : : : : : : : : : : |
:: : : : : : : : : : : : : : : |
:: : : : : : : : : : : : : : : |
:: : : : : : : : : : : : : : : |
: : : : : : : : : : : : : : : |
: : : : : :: : : : : :: : : |
* : : : : : : : : : : |
: : : : : : : : : : |
O O O *.O O O * O O :O O : O * O O O O O OO O |
++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+
numa-numastat.node0.numa_hit
++----------------------------------------------------------------+
O O O O |
O O O O O O O O O O O OO |
O |
|
|
|
|
* * |
:: : |
: : : : |
: *. *. .*. .*.*. |
** * : *. .* * : * * *.*.* *.*.** *.*
: + * + : + : + + |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
numa-numastat.node0.local_node
++----------------------------------------------------------------+
O O O O |
O O O O O O O O O O O OO |
O |
|
|
|
|
* * |
:: : |
: : : : |
: *. *. .*. .*.*. |
** * : *. .* * : * * *.*.* *.*.** *.*
: + * + : + : + + |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
numa-vmstat.node0.numa_hit
++-----------------------------------------------------------------+
O |
O O O OO |
O O O O O O O O O O O |
O |
|
|
* |
:: : : |
: : .*. .*. |
*.* *.*. .*. .* *.* *.* * *.*.* *.**.*.* *.*
* O * : : : : + + |
O : : O: : O * |
: : : : |
++---------O-------------O---O-*-O----*-O--------------------------+
numa-vmstat.node0.numa_local
++------------------------------------------------O----------------+
O O O OO |
O O O O O O O O |
O O O |
|
|
|
* * |
:: : : |
: : .* .*. |
*. .* *.* *.* * + .*.* *.**.*.* *.*
+ + * : : : : * + + |
O * O : : O: : O * |
: : : : |
++---------O-------------O---O-*-O----*-O--------------------------+
proc-vmstat.numa_hit
++----------------------------------------------------------------+
* |
: |
: : |
O O OO: :O O O O *O O O O O O O O O OO |
: : :: O O |
: : : : *. |
* * *.* *.* + *.*.* .*.* .*.*. .*
* : : : : : * : * * * |
: : : : : : : : : : |
: : : : : : : : : : |
: : : : : : : : : |
: * : : : : : : |
* O : : : : O : |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
proc-vmstat.numa_local
++----------------------------------------------------------------+
* |
: |
: : |
O O OO: :O O O O *O O O O O O O O O OO |
: : :: O O |
: : : : *. |
* * *.* *.* + *.*.* .*.* .*.*. .*
* : : : : : * : * * * |
: : : : : : : : : : |
: : : : : : : : : : |
: : : : : : : : : |
: * : : : : : : |
* O : : : : O : |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
proc-vmstat.pgalloc_dma32
++-----------------------------------------------------------------+
O O O O O O O O O |
O O O O O O O |
O |
|
|
|
* * .* .* * *.*.* .**. .*.*
: :: * : * : : : : * *.*.* |
: : : : : + : : : : : : : : |
: : : : * : : : : : : : : |
: : : : : : : : : : : |
* : : : : : : : * : : |
+: : : : : : : |
++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+
proc-vmstat.pgalloc_normal
++----------------------------------------------------------------+
O O O O O O O O O O O O O |
O O O O |
|
|
|
.*. |
* * *.* *.* *.*.* *.*.**.*.*.*.*
: : :: : : : : * : : : |
: : : : :: : : : :: : : : |
: : : : * : : : :: : : : |
: : : : : : : : * : : |
* : : : : : : : : : |
+: : : : : : : |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
proc-vmstat.pgfree
++------O------------O----------------O--O-----O-O-----O-O--------+
O O O O O O O O O O |
O |
|
|
|
|
* * *.* *.* * *.*.* *.*.**.*.*.*.*
: : :: : : : : :: : : : |
: : : : :: : : : :: : : : |
: : : : * : : : : :: : : |
* : : : : : : : * : : |
: : : : : : : : : |
:: * : : : : : |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
proc-vmstat.numa_hint_faults
++------------------------------------------------------------------+
|
* * * |
+: +: .*.* :+ .*. |
* : * : * : : *.* *.*.*
* * : : : : : : : |
* : :: : : : : * : : * |
: : :: : : : : :: : : : |
: : : : : : : : :: : : : |
: : : : * : : : : : ::O |
*O: :O:O : :O : : * O O OO: O O |
O : : : : O : O :O :: O |
: : O : : : : :: |
: : : : : : : |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
proc-vmstat.numa_hint_faults_local
++------------------------------------------------------------------+
* * * |
+: +: :: |
* : * : .*.* : : .*. |
* * : : : : * : : *.* *.*.*
* *.*.* : : : : : : : : : |
: : :: : : : : * : : * |
: : : : : : : : : : : : : |
: : : : : : : : : :: : : : |
: : : : : : : : : : : :: |
*O: :O:O * : :O : : : O O:O O O |
O : : : : O : : O :O * O O:: O |
: : O : : : :: |
: : : : : : : |
++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
job.yaml # the job file attached in this email
have been estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
configuration may affect actual performance.

Attachments:

job.yaml (3.30 kB)
reproduce (9.25 kB)
Download all attachments

2015-06-15 17:53:07

by Thomas Gleixner

[permalink] [raw]

Subject: Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps

On Mon, 15 Jun 2015, Huang Ying wrote:

> FYI, we noticed the below changes on
>
> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
>

I really appreciate this testing effort, but the information provided
is not really helpful.

I asked this before. Can you pretty please, upload ALL relevant
information (.config, full dmesg, below stats, /proc/interrupts ...)
to some place where everyone interested can download them?

Then themail contains a useful link instead of 200k waste of network bandwidth.

Thanks,

tglx

2015-06-16 02:09:41

by Jiang Liu

[permalink] [raw]

Subject: Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps

On 2015/6/16 1:52, Thomas Gleixner wrote:
> On Mon, 15 Jun 2015, Huang Ying wrote:
>
>> FYI, we noticed the below changes on
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
>> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
>>
>
> I really appreciate this testing effort, but the information provided
> is not really helpful.
>
> I asked this before. Can you pretty please, upload ALL relevant
> information (.config, full dmesg, below stats, /proc/interrupts ...)
> to some place where everyone interested can download them?
>
> Then themail contains a useful link instead of 200k waste of network bandwidth.

Hi Ying and Thomas,
I guess this report discloses a regression in hierarchy
irqdomain, and which should have been fixed by the patch posted at:
lkml.org/lkml/2015/6/1/80
The root cause is that, with hierarchy irqdomain enabled,
there are multiple irq_data associated with one irq. And function
irq_move_irq() on x86 uses a wrong copy of irq_data to check
whether there's pending irq migration operation. So all irq migration
/set_affinity operations will get pending for ever. This may
affect network performance due to interrupt load balance issue.
And the patch set posted at
http://www.gossamer-threads.com/lists/linux/kernel/2185533
should have solved all such regressions.

Thanks!
Gerry

>
> Thanks,
>
> tglx
>

2015-06-16 05:41:20

by Huang, Ying

[permalink] [raw]

Subject: Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps

Hi, Gerry,

On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote:
> On 2015/6/16 1:52, Thomas Gleixner wrote:
> > On Mon, 15 Jun 2015, Huang Ying wrote:
> >
> >> FYI, we noticed the below changes on
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
> >>
> >
> > I really appreciate this testing effort, but the information provided
> > is not really helpful.
> >
> > I asked this before. Can you pretty please, upload ALL relevant
> > information (.config, full dmesg, below stats, /proc/interrupts ...)
> > to some place where everyone interested can download them?
> >
> > Then themail contains a useful link instead of 200k waste of network bandwidth.
>
> Hi Ying and Thomas,
> I guess this report discloses a regression in hierarchy
> irqdomain, and which should have been fixed by the patch posted at:
> lkml.org/lkml/2015/6/1/80
> The root cause is that, with hierarchy irqdomain enabled,
> there are multiple irq_data associated with one irq. And function
> irq_move_irq() on x86 uses a wrong copy of irq_data to check
> whether there's pending irq migration operation. So all irq migration
> /set_affinity operations will get pending for ever. This may
> affect network performance due to interrupt load balance issue.
> And the patch set posted at
> http://www.gossamer-threads.com/lists/linux/kernel/2185533
> should have solved all such regressions.

Thanks for your information. Do you have a git tree for this patchset?

Best Regards,
Huang, Ying

2015-06-16 05:51:39

by Jiang Liu

[permalink] [raw]

Subject: Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps

On 2015/6/16 13:41, Huang Ying wrote:
> Hi, Gerry,
>
> On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote:
>> On 2015/6/16 1:52, Thomas Gleixner wrote:
>>> On Mon, 15 Jun 2015, Huang Ying wrote:
>> lkml.org/lkml/2015/6/1/80
>> The root cause is that, with hierarchy irqdomain enabled,
>> there are multiple irq_data associated with one irq. And function
>> irq_move_irq() on x86 uses a wrong copy of irq_data to check
>> whether there's pending irq migration operation. So all irq migration
>> /set_affinity operations will get pending for ever. This may
>> affect network performance due to interrupt load balance issue.
>> And the patch set posted at
>> http://www.gossamer-threads.com/lists/linux/kernel/2185533
>> should have solved all such regressions.
>
> Thanks for your information. Do you have a git tree for this patchset?

Hi Ying,
You may access the branch at:
https://github.com/jiangliu/linux test/irq_data_v3
or view at url:
https://github.com/jiangliu/linux/tree/test/irq_data_v3
Thanks!
Gerry

>
> Best Regards,
> Huang, Ying
>
>

2015-06-16 06:13:52

by Thomas Gleixner

[permalink] [raw]

Subject: Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps

On Tue, 16 Jun 2015, Jiang Liu wrote:

> On 2015/6/16 1:52, Thomas Gleixner wrote:
> > On Mon, 15 Jun 2015, Huang Ying wrote:
> >
> >> FYI, we noticed the below changes on
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
> >>
> >
> > I really appreciate this testing effort, but the information provided
> > is not really helpful.
> >
> > I asked this before. Can you pretty please, upload ALL relevant
> > information (.config, full dmesg, below stats, /proc/interrupts ...)
> > to some place where everyone interested can download them?
> >
> > Then themail contains a useful link instead of 200k waste of network bandwidth.
>
> Hi Ying and Thomas,
> I guess this report discloses a regression in hierarchy
> irqdomain, and which should have been fixed by the patch posted at:
> lkml.org/lkml/2015/6/1/80
> The root cause is that, with hierarchy irqdomain enabled,
> there are multiple irq_data associated with one irq. And function
> irq_move_irq() on x86 uses a wrong copy of irq_data to check
> whether there's pending irq migration operation. So all irq migration
> /set_affinity operations will get pending for ever. This may
> affect network performance due to interrupt load balance issue.
> And the patch set posted at
> http://www.gossamer-threads.com/lists/linux/kernel/2185533
> should have solved all such regressions.

Next time, please send fixes which need to be applied right away
seperate from patch series which touch the world and some more.

Thanks,

tglx