LinuxLists.cc - [btrfs] e076ab2a2c: fio.write

2021-01-12 15:23:26

Subject: [btrfs] e076ab2a2c: fio.write_iops -18.3% regression

Greeting,

FYI, we noticed a

commit: e076ab2a2ca70a02702

If you fix the issue, Reported-by: kernel

Details are as below:
---------------------------

To reproduce:

git clone bin/lkp install job.yaml bin/lkp run job.yaml

=========================== bs/compiler/cpufreq_governo 4k/gcc-9/performance/1SSD
commit:
50e31ef486 ("btrfs: e076ab2a2c ("btrfs:
50e31ef486afe60f e076ab2a2c ---------------- ---------- fail:runs %reproduction | | 0:4 1% 0:4 2% %stddev %change \ | 0.03 ? 10% +0.2 20.78 ? 4% -8.7 0.01 +0.0 0.29 ? 33% +1.6 72.15 -13.8 0.05 ? 4% +0.3 0.02 ? 7% +0.4 0.71 +3.9 5.28 ? 10% +15.1 0.01 -0.0 0.09 ? 7% +0.9 0.01 -0.0 1.488e+08 1247 ? 2% -17.5% 41164 ? 9% -20.4% 132.00 -14.8% 333.68 -15.4% 66.48 ? 4% -12.8% 18466787 18601377 241.84 -18.3% 155648 +47.2% 169984 +56.4% 242688 +204.8% 127782 +22.7% 3792729 ? 3% -77.2% 61910 -18.3% 3.55 -7.5% 0.01 ? 26% -0.0 2.07 -0.3 292190 -18.3% 46974505 364386 -19.1% 6.096e+08 ? 4% 33545458 40784386 ? 53% 9195081 ? 2% 1004 ? 6% -17.3% 141.47 ? 8% -16.4% 1026 ? 2% -16.4% 162.93 ? 8% -13.3% 1026 ? 2% -16.5% 162.88 ? 8% -13.4% 972.33 ? 2% -30.0% 116.75 ? 9% -33.0% 838.02 ? 6% 296238 -19.3% 1029425 ? 16% -27.4% 177688 ? 5% -13.8% 1665501 ? 33% -63.5% 1664327 ? 33% -63.6% 658720 ? 13% -54.1% 953109 ? 18% -35.6% 1146488 ? 20% -38.4% 1167 ? 33% +87.4% 318436 ? 10% +53.3% 1363 ? 32% +87.7% 504582 ? 3% -15.9% 112828 ? 35% -47.0% 32143 ? 5% -33.6% 555914 ? 2% -17.4% 1387 ? 15% +86.4% 5069900 ? 6% 509598 ? 2% -15.5% 4560301 ? 6% 330755 ? 2% -12.2% 46166951 1947190 40388694 38882003 51931186 10645 ? 4% -21.3% 3347589 4062206 4971 ? 7% +96.8% 312047 -18.6% 2607626 ? 27% 416310 ? 33% -63.6% 164589 ? 13% -54.1% 238399 ? 18% -35.6% 289.00 ? 34% +91.3% 416310 ? 33% -63.6% 164899 ? 13% -53.9% 79657 ? 10% +53.3% 317.00 ? 35% +101.9% 79961 ? 10% +53.3% 126199 ? 2% -16.0% 28308 ? 34% -46.6% 8212 ? 6% -31.7% 139120 ? 2% -17.4% 337.25 ? 20% +92.0% 126199 ? 2% -16.0% 28308 ? 34% -46.6% 7234521 141900 -14.0% 7236947 141900 -14.0% 22698 ? 4% -32.8% 628.50 ? 2% -36.4% 24535 ? 2% -36.4% 628.50 ? 2% -36.4% 227033 ? 5% -29.6% 3384 ? 5% -29.0% 230151 ? 5% -29.0% 3384 ? 5% -29.0% 6464 ? 4% -9.9% 6464 ? 4% -9.9% 36076 ? 5% +22.3% 37240 ? 5% +19.8% 12947 ? 6% +105.8% 361.50 ? 6% +107.2% 13038 ? 6% +106.9% 361.50 ? 6% +107.2% 16713593 298487 -17.2% 16715295 298487 -17.2% 126844 ? 2% -15.6% 1138477 ? 6% -28.7% 82750 ? 2% -12.2% 22860671 486577 -16.1% 11529365 36458632 377699 9709300 296711 2666 ? 4% -21.7% 421970 178635 836153 -15.2% 1247 ? 7% +93.7% 22124969 126844 ? 2% -15.6% 1138477 ? 6% -28.7% 377699 9709300 487874 -15.9% 67603 ? 4% -21.6% 43463 ? 18% -18.9% 23728440 23469490 28285 ? 21% -29.4% 193069 ? 5% -11.2% 2098562 ? 8% 24762066 1896167 1745632 28285 ? 21% -29.4% 88507956 2.407e+09 24.89 ? 8% +6.0 30551727 ? 2% 1.243e+08 ? 8% 367250 -19.2% 1.81 +5.0% 2.216e+10 214.03 755.49 -18.1% 4598831 ? 9% 3.314e+09 1.717e+09 7711379 ? 2% 14110623 1.249e+10 1663 ? 3% -5.9% 0.57 -5.0% 0.12 -9.8% 39.54 -13.4% 4274 -1.9% 84.46 -2.9 1840912 ? 6% 2069105 ? 3% 5197 -1.5% 24.75 ? 8% +5.7 1.78 +4.2% 725.94 -17.2% 1620 ? 2% -3.3% 0.56 -4.0% 84.57 -2.7 202639 2.399e+09 30439572 ? 2% 1.238e+08 ? 8% 365832 -19.2% 2.209e+10 213.36 4581044 ? 9% 3.303e+09 1.711e+09 7683689 ? 2% 14057332 1.245e+10 4265 -1.9% 1833617 ? 6% 2061479 ? 3% 5195 -1.5% 3.769e+12 0.01 ? 5% -25.2% 0.00 ? 25% +84.6% 0.00 ? 57% +353.3% 0.00 ?100% +1460.0% 0.00 ?173% +1000.0% 0.00 ?122% +480.0% 0.00 ? 31% -100.0% 0.01 ? 8% -100.0% 7.51 ? 30% -40.4% 0.02 ? 38% -55.8% 0.00 ?100% +520.0% 0.01 ? 22% -43.8% 0.01 ? 20% +126.7% 0.01 ? 46% -100.0% 0.01 ? 28% +182.2% 0.03 ? 68% -100.0% 2.06 ?166% -97.2% 2.05 +20.6% 1587968 ? 2% 2.05 +20.7% 0.12 ? 3% +32.3% 30.60 +23.5% 479.86 ? 8% -98.1% 381.33 -100.0% 0.98 ? 8% +18.6% 561469 ? 2% -21.8% 5.25 ?173% +399.0% 572.75 -17.8% 3.00 +7806.7% 20.75 ? 3% -100.0% 1010874 ? 2% -16.5% 5880 ? 35% -44.7% 1.00 ?173% +399.8% 534.64 ? 10% -46.9% 6.77 ? 29% -25.4% 600.79 ? 16% -100.0% 0.11 ? 3% +34.2% 0.16 ? 30% +5605.1% 0.02 ? 37% +1.9e+05% 0.03 ? 20% +3644.0% 0.05 ? 3% -58.6% 0.04 ? 62% -100.0% 30.60 +23.5% 479.86 ? 8% -98.1% 381.33 -100.0% 0.98 ? 8% +18.7% 2.89 ? 8% +4625.7% 5880 ? 35% -44.7% 0.72 ?139% +157.4% 1.57 ? 54% +4770.5% 1.84 ?166% -99.0% 0.07 ? 85% +2.5e+05% 0.14 ? 9% +494.4% 1.00 ?173% +399.3% 0.07 ? 29% +2916.3% 0.08 ? 20% -63.4% 0.14 ?105% -100.0% 534.63 ? 10% -46.9% 600.77 ? 16% -100.0% 495.50 ? 52% +6469.9% 950178 ? 3% 787.75 ? 76% -58.9% 11703 ? 28% -46.1% 26213 ? 42% -71.9% 15857 ? 59% -56.3% 17175 ? 27% -59.3% 20559 ? 34% -54.9% 18309 ? 22% -67.6% 6727 ? 12% -35.4% 18695 ? 35% -72.3% 20766 ? 50% -68.9% 495.50 ? 52% +6469.9% 16674 ? 39% -71.4% 10068 ? 42% -51.7% 14332 ? 63% -64.5% 8837 ? 22% -43.6% 11663 ? 31% -58.7% 697.50 ? 22% -54.6% 17787 ? 41% -61.0% 17306 ? 31% -64.5% 17198 ? 24% -74.6% 6750 ? 7% -33.5% 20260 ? 49% -77.4% 17544 ? 48% -74.6% 15840 ? 69% -73.1% 19961 ? 46% -80.6% 23099 ? 57% -75.2% 22411 ? 33% -74.2% 19423 ? 36% -67.1% 23502 ? 27% -71.9% 19845 ? 36% -67.0% 7302 ? 15% -32.2% 24689 ? 40% -71.7% 28704 ? 63% -78.7% 23226 ? 20% -73.9% 6991 ? 30% -43.8% 25077 ? 52% -84.1% 18334 ? 34% -68.0% 17484 ? 23% -60.5% 20088 ? 21% -77.7% 5407 ? 40% -37.0% 17273 ? 29% -67.2% 14848 ? 41% -68.6% 5144 ? 9% -27.4% 11588 ? 23% -75.6% 4427 ? 15% +53.0% 179.25 ? 37% +222.2% 179.25 ? 37% +222.2% 223.25 ? 62% +118.4% 223.25 ? 62% +118.4% 12367 ? 12% -37.1% 151.50 ? 42% +219.6% 151.50 ? 42% +219.6% 4527 ? 12% +45.5% 14973 ? 34% -39.4% 3958 ? 21% +47.2% 7593 ? 25% -41.9% 1452 ? 32% -64.2% 91.75 ? 53% -51.8% 5924 ? 9% -29.1% 1074 ? 22% -51.0% 7677 ? 33% -53.2% 1512 ? 35% -72.6% 1136 ? 36% -55.3% 8426 ? 68% -74.3% 480.25 ? 80% -60.7% 480.25 ? 80% -60.7% 9548 ? 31% -70.2% 4642 ? 26% -30.5% 13166 ? 63% -82.8% 558.75 ? 22% -60.7% 558.75 ? 22% -60.7% 190.50 ? 42% +139.4% 190.50 ? 42% +139.4% 216.25 ? 43% +149.3% 216.25 ? 43% +149.3% 3888 ? 14% -21.8% 5086 ? 25% -33.4% 816.00 ? 29% -56.5% 6125 ? 10% -13.9% 57.00 ? 43% +593.0% 31.50 ?110% +225.7% 6322 ? 43% -31.8% 201.25 ? 32% +200.5% 201.25 ? 32% +200.5% 192.00 ? 39% +124.1% 192.00 ? 39% +124.1% 38.00 ? 57% +172.1% 4627 ? 13% -26.5% 91.25 ? 58% -53.3% 56.50 ? 45% +149.9% 30.75 ? 99% +305.2% 379.25 ? 27% -43.3% 379.25 ? 27% -43.3% 6254 ? 37% -81.0% 43.25 ? 52% +159.0% 651.75 ? 2% -18.3% 71040 ? 3% +10.8% 71040 ? 3% +10.8% 1625554 ? 4% -50.9% 10832 ? 15% +98.1% 6865 ? 9% +158.2% 6568 ? 8% +158.9% 6168 ? 2% +180.2% 6432 ? 8% +165.9% 6279 +173.8% 6528 ? 2% +161.9% 7066 ? 14% +143.3% 6470 ? 5% +160.3% 6320 ? 10% +162.3% 6430 ? 11% +163.2% 6477 ? 5% +161.0% 6229 +168.1% 272.25 ? 45% +6267.9% 6123 ? 8% +185.7% 763.00 ?158% +564.1% 6332 ? 8% +169.9% 6086 ? 4% +176.8% 5664 ? 2% +169.4% 5828 ? 2% +166.0% 5648 ? 3% +175.8% 5637 ? 2% +176.9% 5596 ? 2% +176.1% 5624 ? 2% +174.9% 5722 +166.8% 5545 ? 5% +181.4% 7465 ? 16% +128.4% 5496 ? 5% +178.6% 5683 ? 11% +168.0% 5965 ? 21% +153.8% 5542 ? 6% +171.2% 5956 ? 10% +157.0% 5423 ? 4% +179.6% 5818 ? 10% +162.6% 5656 ? 6% +169.8% 6127 ? 6% +167.9% 6353 ? 10% +154.5% 6313 ? 2% +170.8% 6072 ? 6% +165.2% 6183 ? 7% +169.0% 6241 ? 7% +166.4% 6089 ? 6% +170.8% 6036 ? 6% +166.4% 6181 ? 7% +166.3% 6217 ? 8% +161.4% 6462 ? 10% +158.2% 6389 ? 10% +160.0% 7159 ? 29% +126.9% 7324 ? 9% +155.4% 6536 ? 13% +148.4% 5988 ? 6% +172.2% 6241 ? 8% +162.9% 6344 ? 7% +159.6% 5997 ? 8% +188.2% 5937 ? 7% +179.8% 5922 ? 5% +182.5% 5987 ? 8% +180.0% 6025 ? 6% +182.2% 5900 ? 11% +187.4% 6271 ? 3% +171.7% 5967 ? 6% +186.6% 5852 ? 7% +184.5% 5889 ? 5% +181.0% 5981 ? 6% +177.6% 5992 ? 5% +179.0% 5991 ? 6% +184.6% 5941 ? 6% +187.9% 5803 ? 5% +175.6% 5783 ? 4% +187.1% 5832 ? 7% +188.5% 6065 ? 6% +156.2% 5385 ? 5% +183.6% 5572 ? 7% +181.6% 5492 ? 5% +182.6% 5560 ? 5% +180.6% 5633 ? 4% +176.0% 5776 ? 11% +163.1% 5402 ? 5% +186.3% 5457 ? 5% +188.7% 5423 ? 6% +173.3% 5305 ? 6% +176.5% 6063 ? 4% +163.7% 5324 ? 6% +172.5% 5428 ? 6% +175.3% 5441 ? 10% +175.1% 5452 ? 8% +174.2% 5351 ? 6% +177.0% 6561 ? 28% +128.5% 5751 ? 6% +181.7% 5769 ? 5% +178.0% 5807 ? 7% +178.9% 6178 ? 16% +166.6% 5868 ? 7% +169.9% 5792 ? 6% +175.1% 5597 ? 8% +185.7% 5853 ? 7% +177.0% 5851 ? 5% +176.5% 5745 ? 8% +179.5% 5768 ? 6% +195.7% 5849 ? 5% +171.4% 5855 ? 5% +182.4% 5856 ? 6% +188.8% 5720 ? 5% +178.1% 5795 ? 4% +167.2% 5698 ? 6% +182.3% 6134 ? 7% +165.6% 6621 ? 4% +165.0% 6024 ? 6% +153.7% 5978 ? 5% +158.7% 5971 ? 2% +160.8% 5897 ? 6% +166.3% 6136 ? 3% +162.2% 5851 ? 4% +168.8% 5908 ? 6% +166.5% 5910 ? 4% +165.4% 5791 ? 5% +176.4% 5748 ? 7% +173.9% 6571 ? 5% +176.3% 5834 ? 5% +168.4% 5748 ? 3% +170.4% 6812 ? 7% +151.6% 6555 ? 9% +151.1% 6385 ? 3% +162.8% 6342 ? 4% +169.6% 6269 ? 4% +172.5% 6293 ? 6% +164.6% 6245 ? 5% +168.1% 6267 ? 6% +164.4% 6384 ? 4% +172.4% 6226 ? 6% +167.3% 6392 ? 6% +165.7% 6368 ? 7% +167.7% 6280 ? 4% +167.1% 6306 ? 7% +162.5% 6215 ? 3% +166.5% 6406 ? 7% +157.8% 6338 ? 7% +162.5% 6688 ? 5% +176.1% 6338 ? 5% +170.2% 6528 ? 4% +171.2% 6329 ? 6% +176.2% 6334 ? 5% +179.3% 6463 ? 4% +172.7% 6587 ? 11% +168.5% 6470 ? 7% +173.2% 6231 ? 6% +179.6% 6197 ? 7% +181.8% 6263 ? 7% +170.3% 6158 ? 7% +177.9% 6393 ? 6% +169.5% 6312 ? 3% +172.3% 6345 ? 8% +171.2% 6198 ? 6% +167.8% 6106 ? 6% +177.0% 6204 ? 8% +172.5% 6192 ? 11% +149.9% 5745 ? 9% +175.4% 5986 ? 8% +163.5% 5838 ? 8% +170.6% 5806 ? 8% +173.9% 5879 ? 8% +167.1% 6456 ? 2% +169.5% 5920 ? 6% +166.5% 5807 ? 7% +172.0% 6272 ? 4% +173.6% 6072 ? 4% +165.0% 5848 ? 5% +172.8% 5779 ? 4% +172.2% 5931 ? 8% +167.7% 5774 ? 5% +168.1% 5772 ? 5% +169.2% 5675 ? 4% +174.0% 6496 ? 5% +168.2% 5951 ? 5% +172.8% 5936 ? 5% +172.7% 6032 ? 4% +170.9% 5971 ? 6% +172.5% 6032 ? 3% +166.1% 5898 ? 6% +170.2% 5974 ? 3% +166.0% 5916 ? 4% +169.2% 5971 ? 5% +168.2% 5891 ? 6% +169.9% 6287 ? 2% +170.6% 5804 ? 4% +176.6% 5876 ? 6% +185.2% 6051 ? 4% +163.1% 5744 ? 5% +175.1% 5801 ? 5% +176.1% 5760 ? 6% +181.3% 5958 ? 3% +181.7% 6296 ? 4% +168.7% 6088 ? 4% +182.2% 6061 ? 3% +182.0% 1165257 ? 3% +169.6% 10.12 ? 11% -10.1 10.12 ? 11% -10.1 10.12 ? 11% -10.1 10.12 ? 11% -10.1 10.12 ? 11% -10.1 9.77 ? 11% -9.8 26.39 ? 3% -8.3 3.60 ? 2% -1.0 3.74 ? 2% -1.0 3.74 ? 2% -1.0 3.74 ? 2% -1.0 3.74 ? 2% -1.0 3.74 ? 2% -1.0 3.77 ? 3% -0.9 3.77 ? 3% -0.9 3.27 ? 2% -0.9 3.20 ? 2% -0.9 3.20 ? 2% -0.9 3.16 ? 2% -0.9 3.10 ? 2% -0.9 1.86 ? 8% -0.7 1.86 ? 8% -0.7 1.71 ? 8% -0.6 1.93 ? 12% -0.5 1.65 ? 2% -0.5 1.74 ? 12% -0.4 6.84 ? 3% -0.4 0.74 ? 16% -0.4 1.24 ? 13% -0.4 0.73 ? 5% -0.4 0.73 ? 5% -0.4 1.07 ? 10% -0.3 0.87 ? 4% -0.3 0.90 ? 12% -0.3 1.76 ? 6% -0.3 0.57 -0.3 0.69 ? 11% -0.2 1.48 ? 5% -0.2 1.69 ? 4% -0.2 1.71 ? 5% -0.2 0.93 ? 11% -0.2 0.81 ? 11% -0.2 0.82 ? 9% -0.2 0.83 ? 9% -0.1 0.83 ? 2% -0.1 0.72 ? 3% -0.1 1.42 ? 2% -0.1 1.26 ? 5% -0.1 0.65 ? 6% +0.1 0.72 ? 5% +0.2 0.71 ? 5% +0.2 1.49 ? 4% +0.2 0.44 ? 58% +0.8 0.83 ? 15% +0.9 1.32 ? 8% +0.9 1.33 ? 8% +0.9 0.49 ? 58% +0.9 1.47 ? 6% +0.9 0.29 ?100% +0.9 0.00 +0.9 0.29 ?100% +0.9 0.53 ? 59% +1.0 0.00 +1.0 5.94 ? 4% +1.2 5.88 ? 4% +1.2 5.93 ? 4% +1.2 0.00 +1.3 0.00 +1.3 0.00 +1.4 0.00 +1.4 2.87 ? 4% +1.8 15.13 ? 5% +1.9 0.00 +2.0 0.00 +2.0 0.00 +2.0 0.00 +2.0 0.00 +2.0 0.00 +2.0 0.00 +2.0 0.00 +2.0 6.20 ? 5% +2.0 1.11 ? 4% +7.1 0.00 +7.2 0.00 +7.2 0.00 +7.2 0.00 +7.2 0.00 +7.2 0.00 +7.2 0.00 +7.2 1.11 ? 4% +7.2 0.00 +8.9 13.86 ? 9% -11.1 10.12 ? 11% -10.1 26.40 ? 3% -8.3 3.60 ? 2% -1.0 4.17 ? 4% -1.0 3.74 ? 2% -1.0 3.74 ? 2% -1.0 3.74 ? 2% -1.0 3.77 ? 3% -0.9 3.77 ? 3% -0.9 3.85 ? 4% -0.9 3.21 ? 2% -0.8 3.17 ? 2% -0.8 3.14 ? 2% -0.8 2.99 ? 3% -0.6 1.69 ? 3% -0.4 7.07 ? 3% -0.4 1.86 ? 8% -0.4 1.86 ? 8% -0.4 1.78 ? 6% -0.3 1.35 ? 2% -0.2 1.07 ? 5% -0.2 1.50 ? 5% -0.2 1.72 ? 5% -0.2 1.21 ? 4% -0.2 1.07 ? 10% -0.2 0.75 ? 6% -0.2 0.83 ? 9% -0.2 1.09 ? 5% -0.2 1.92 ? 4% -0.2 1.58 ? 4% -0.2 1.04 ? 5% -0.1 0.43 ? 10% -0.1 0.84 ? 9% -0.1 1.03 ? 6% -0.1 0.43 ? 9% -0.1 0.41 ? 5% -0.1 0.28 ? 7% -0.1 0.58 ? 5% -0.1 1.28 ? 2% -0.1 0.58 ? 3% -0.1 1.14 ? 4% -0.1 0.71 ? 6% -0.1 0.39 ? 4% -0.1 0.48 ? 8% -0.1 0.23 ? 10% -0.1 1.33 ? 4% -0.1 0.24 ? 10% -0.1 0.22 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 8% -0.1 0.23 ? 13% -0.1 0.94 ? 4% -0.1 0.28 ? 14% -0.1 0.60 ? 4% -0.1 0.22 ? 10% -0.1 0.43 ? 5% -0.1 0.21 ? 10% -0.1 0.56 ? 3% -0.1 0.26 ? 5% -0.0 0.23 ? 7% -0.0 0.27 ? 5% -0.0 0.26 ? 9% -0.0 0.18 ? 4% -0.0 0.21 ? 7% -0.0 0.33 ? 10% -0.0 0.22 ? 11% -0.0 0.22 ? 6% -0.0 0.41 ? 3% -0.0 0.38 ? 4% -0.0 0.15 ? 5% -0.0 0.35 ? 3% -0.0 0.08 ? 10% -0.0 0.29 ? 4% -0.0 0.15 ? 7% -0.0 0.12 ? 8% -0.0 0.09 ? 11% -0.0 0.10 ? 10% -0.0 0.10 ? 10% -0.0 0.09 ? 9% -0.0 0.08 ? 10% -0.0 0.11 ? 3% +0.0 0.21 ? 5% +0.0 0.16 ? 4% +0.0 0.15 ? 10% +0.0 0.25 ? 11% +0.1 0.23 ? 14% +0.1 0.14 ? 18% +0.1 0.01 ?173% +0.1 0.27 ? 10% +0.1 0.01 ?173% +0.1 0.29 ? 8% +0.1 0.19 ? 13% +0.1 0.25 ? 11% +0.1 0.38 ? 12% +0.1 0.84 ? 5% +0.1 1.12 ? 5% +0.1 1.65 ? 7% +0.2 1.16 ? 7% +0.3 1.48 ? 8% +0.3 2.88 ? 6% +0.3 3.16 ? 5% +0.3 1.96 ? 4% +0.8 1.85 ? 5% +0.8 1.33 ? 8% +0.9 1.41 ? 7% +0.9 1.47 ? 6% +0.9 4.70 +1.0 3.96 ? 6% +1.0 1.16 ? 10% +1.0 0.56 ? 4% +1.0 1.80 ? 12% +1.1 6.04 ? 4% +1.2 5.96 ? 4% +1.2 6.34 ? 4% +1.2 6.30 ? 4% +1.2 6.03 ? 4% +1.2 5.88 ? 4% +1.2 5.93 ? 4% +1.2 1.46 ? 5% +1.4 10.11 ? 2% +1.5 15.13 ? 5% +1.9 0.00 +2.0 0.00 +2.0 0.00 +2.0 0.00 +2.0 6.21 ? 5% +2.0 1.11 ? 4% +7.1 0.00 +7.2 0.00 +7.2 0.00 +7.2 1.11 ? 4% +7.2 0.00 +9.2 2.94 ? 4% -0.6 1.04 ? 5% -0.2 0.70 ? 10% -0.1 1.05 ? 2% -0.1 0.83 ? 9% -0.1 0.43 ? 9% -0.1 0.57 ? 3% -0.1 0.23 ? 4% -0.1 0.29 ? 4% -0.1 0.27 ? 15% -0.1 0.14 ? 15% -0.1 0.91 ? 4% -0.1 0.28 ? 5% -0.1 0.27 ? 4% -0.0 0.18 ? 4% -0.0 0.34 ? 3% -0.0 0.27 ? 6% -0.0 0.15 ? 5% -0.0 0.11 ? 7% -0.0 0.14 ? 11% -0.0 0.10 ? 7% -0.0 0.08 ? 5% -0.0 0.12 ? 8% -0.0 0.07 ? 12% -0.0 0.11 ? 7% -0.0 0.10 ? 10% -0.0 0.10 ? 8% -0.0 0.11 ? 3% +0.0 0.25 ? 8% +0.1 0.01 ?173% +0.1 0.16 ? 13% +0.1 0.26 ? 7% +0.1 0.13 ? 9% +0.1 0.21 ? 13% +0.1 0.25 ? 5% +0.1 0.82 ? 5% +0.1 0.36 ? 8% +0.1 1.79 ? 12% +1.1 1.44 ? 5% +1.4

64000 +------------------ | 62000 |-+ | 60000 |-+ : | : 58000 |-+ : | .+... : 56000 |..+...+. + | 54000 |-+ | 52000 |-+ | O O O 50000 +------------------ 5e+06 +---------------- | +. 4.5e+06 |.. .. 4e+06 |-+ .+..+... | +. 3.5e+06 |-+ 3e+06 |-+ | 2.5e+06 |-+ 2e+06 |-+ | 1.5e+06 |-+ 1e+06 |-+ O | O O O 500000 +---------------- 240000 +----------------- | O 230000 |-+O O O 220000 |-+ | 210000 |-+ 200000 |-+ | 190000 |-+ 180000 |-+ |..+...+..+..+. 170000 |-+ .. 160000 |-+ | 150000 +----------------- 280000 +----------------- | O 260000 |-+O O O | | 240000 |-+ | 220000 |-+ | 200000 |-+ |.. ..+.. .+. | +. +. .. 180000 |-+ | 160000 +----------------- 800000 +----------------- | O O 700000 |-+O O | | 600000 |-+ | 500000 |-+ | 400000 |-+ | | 300000 |.. ..+.. .+... | +. +. 200000 +----------------- 2.2 +-------------------- 2 |-+ O | O O 1.8 |-+ O 1.6 |-+ 1.4 |-+ 1.2 |-+ | 1 |-+ 0.8 |-+ 0.6 |-+ 0.4 |-+ |... .+... 0.2 |-+ +. +..+...+.. 0 +-------------------- 22 +--------------------- | O O O 20 |-+ O 18 |-+ | 16 |-+ 14 |-+ | 12 |-+ 10 |-+ | 8 |-+ 6 |-..+..+... .+... .+...+.. |. +. +. 4 +--------------------- 76 +--------------------- |...+. . .. +.. ..+ 74 |-+ + 72 |-+ | 70 |-+ 68 |-+ | 66 |-+ 64 |-+ | 62 |-+ 60 |-+ | O 58 +--------------------- 5 +-------------------- | O O O 4.5 |-+ O 4 |-+ | 3.5 |-+ 3 |-+ | 2.5 |-+ 2 |-+ | 1.5 |-+ 1 |... .+...+..+... | +. 0.5 +-------------------- 1.1 +-------------------- 1 |-+ O O | O O 0.9 |-+ 0.8 |-+ 0.7 |-+ 0.6 |-+ | 0.5 |-+ 0.4 |-+ 0.3 |-+ 0.2 |-+ |... .+... .+... 0.1 |-+ +. +. 0 +-------------------- 0.24 +------------------- 0.22 |-+ O O | O O 0.2 |-+ 0.18 |-+ 0.16 |-+ 0.14 |-+ | 0.12 |-+ 0.1 |-+ 0.08 |-+ 0.06 |-+ |.. ..+.. ..+.. 0.04 |-++. +. 0.02 +------------------- 0.4 +------------------- | 0.35 |-+ O 0.3 |-+O O O | 0.25 |-+ | 0.2 |-+ | 0.15 |-+ 0.1 |-+ | 0.05 |.. ..+..+...+..+.. | +. 0 +------------------- 0.45 +------------------- | O O O 0.4 |-+ O 0.35 |-+ | 0.3 |-+ 0.25 |-+ | 0.2 |-+ 0.15 |-+ | 0.1 |-+ 0.05 |-+ |..+...+..+...+..+...+..+... 0 +------------------- 0.024 +------------------ | O O 0.022 |-+O | 0.02 |-+ O | 0.018 |-+ | 0.016 |-+ | 0.014 |-+ | 0.012 |-+ | 0.01 +------------------ 1.9e+07 +--------------- 1.85e+07 |-+ | 1.8e+07 |-+ : 1.75e+07 |-+ : | : 1.7e+07 |..+... .+.. : 1.65e+07 |-+ +. + 1.6e+07 |-+ | 1.55e+07 |-+ 1.5e+07 |-+ | O O O 1.45e+07 |-+ O 1.4e+07 +--------------- 2.2e+06 +---------------- 2.1e+06 |-+ + + | + : 2e+06 |-.+...+ : 1.9e+06 |.+ : | : 1.8e+06 |-+ 1.7e+06 |-+ 1.6e+06 |-+ | 1.5e+06 |-+ 1.4e+06 |-+ | O O O 1.3e+06 |-+ O 1.2e+06 +---------------- 0.16 +------------------- | O O O O 0.14 |-+ 0.12 |-+ | 0.1 |-+ : | : 0.08 |-+ : | : 0.06 |-+ : 0.04 |-+ : | : 0.02 |-+ : | : 0 +------------------- 0.16 +------------------- | O 0.14 |-+ 0.12 |-+ ..+.. | 0.1 |-+ : | : 0.08 |-+ : | : 0.06 |-+ : 0.04 |-+ : | : 0.02 |-+ : | : 0 +------------------- 40 +--------------------- | 38 |-+ O O O | O 36 |-+ | 34 |-+ | 32 |-+ | 30 |-+ : | : 28 |-+ : |... .+... : 26 +--------------------- 640 +-------------------- 620 |...+.. | 600 |-+ + +.. 580 |-+ + : . | + : +..+... 560 |-+ + : 540 |-+ + 520 |-+ | 500 |-+ 480 |-+ O | O O O 460 |-+ 440 +-------------------- 40 +--------------------- | 38 |-+ O O O | O 36 |-+ | 34 |-+ | 32 |-+ | 30 |-+ : | : 28 |-+ : |... .+... : 26 +--------------------- 600 +-------------------- | .. +. 500 |.+ .. | | + + 400 |-+ + + | + + 300 |-+ + | 200 |-+ | | 100 |-+ | 0 +-------------------- 300 +-------------------- | 250 |-+ O | O O | O 200 |-+ | 150 |-+ | 100 |-+ | | 50 |-+ | 0 +-------------------- 600 +-------------------- | .. +. 500 |.+ .. | | + + 400 |-+ + + | + + 300 |-+ + | 200 |-+ | | 100 |-+ | 0 +-------------------- 0.0101 +---------------- 0.01008 |-+ | 0.01006 |-+ 0.01004 |-+ | 0.01002 |-+ 0.01 |..+...+..+..+... 0.00998 |-+ | 0.00996 |-+ 0.00994 |-+ | 0.00992 |-+ 0.0099 +---------------- 0.0101 +---------------- 0.01008 |-+ | 0.01006 |-+ 0.01004 |-+ | 0.01002 |-+ 0.01 |..+...+..+..+... 0.00998 |-+ | 0.00996 |-+ 0.00994 |-+ | 0.00992 |-+ 0.0099 +---------------- 950 +-------------------- 900 |-+ + | + + 850 |-+ + : 800 |-+ + : | : : 750 |.. : : 700 |-+. : : 650 |-+ + : | : 600 |-+ : 550 |-+ | +.. : 500 |-+ 450 +-------------------- 500 +-------------------- | .+ 480 |-+ ..+. : |... .+. : 460 |-+ +. : | : 440 |-+ : | : 420 |-+ : | : 400 |-+ : | : 380 |-+ | 360 +-------------------- 0.018 +------------------ | 0.016 |-+ | 0.014 |-+ | 0.012 |-+ | 0.01 |-+ | 0.008 |.. | .+...+.. 0.006 |-++...+. | 0.004 +------------------ 23 +--------------------- | | 22 |-+ | | 21 |-+ + | : 20 |-+ : | : | : 19 |-+ + | + | + 18 +--------------------- 950 +-------------------- 900 |-+ + | + + 850 |-+ + : 800 |-+ + : | : : 750 |.. : : 700 |-+. : : 650 |-+ + : | : 600 |-+ : 550 |-+ | +.. : 500 |-+ 450 +-------------------- 500 +-------------------- | .+ 480 |-+ ..+. : |... .+. : 460 |-+ +. : | : 440 |-+ : | : 420 |-+ : | : 400 |-+ : | : 380 |-+ | 360 +-------------------- [*] bisect-good sample
[O] bisect-bad sample

Disclaimer:
Results have been for informational design or configuration

Thanks,
Oliver Sang

-18.3% regression of fio.write_iops due to commit:
32067cd49f76cd92efe64 ("btrfs: shrink delalloc pages instead of full inodes")
l.org/cgit/linux/kernel/git/torvalds/linux.git">https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
/github.com/axboe/fio">https://github.com/axboe/fio
kindly add following tag
test robot <[email protected]>
----------------------------------------------------------------------->
href="https://github.com/intel/lkp-tests.git">https://github.com/intel/lkp-tests.git
# job file is attached in this email
==============================================================
r/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
/btrfs/sync/x86_64-rhel-8.3/8/debian-10.4-x86_64-20200603.cgz/300s/randwrite/lkp-csl-2ap1/256g/fio-basic/0x4003003
reloc: fix wrong file extent type check to avoid false ENOENT")
shrink delalloc pages instead of full inodes")
a70a0270232067cd4
-----------------
fail:runs
|
0:5 perf-profile.children.cycles-pp.error_entry
0:5 perf-profile.self.cycles-pp.error_entry
%stddev
\
0.21 ? 3% fio.latency_1000us%
12.11 ? 3% fio.latency_100us%
0.02 ? 8% fio.latency_10ms%
1.85 ? 8% fio.latency_20us%
58.39 fio.latency_250us%
0.32 ? 6% fio.latency_2ms%
0.41 ? 3% fio.latency_4ms%
4.64 ? 2% fio.latency_500us%
20.37 fio.latency_50us%
0.00 fio.latency_750ms%
0.97 ? 3% fio.latency_750us%
0.00 fio.latency_>=2000ms%
-18.4% 1.215e+08 fio.time.file_system_outputs
1029 ? 7% fio.time.involuntary_context_switches
32749 ? 2% fio.time.minor_page_faults
112.40 fio.time.percent_of_cpu_this_job_got
282.44 fio.time.system_time
57.94 fio.time.user_time
-20.9% 14611125 fio.time.voluntary_context_switches
-18.4% 15183620 fio.workload
197.69 fio.write_bw_MBps
229171 fio.write_clat_90%_us
265830 fio.write_clat_95%_us
739737 fio.write_clat_99%_us
156730 fio.write_clat_mean_us
862957 ? 12% fio.write_clat_stddev
50610 fio.write_iops
3.28 iostat.cpu.system
0.00 ? 72% mpstat.cpu.all.iowait%
1.77 mpstat.cpu.all.sys%
238813 vmstat.io.bo
-18.8% 38163680 vmstat.memory.cache
294676 vmstat.system.cs
-10.3% 5.468e+08 cpuidle.C1.time
-9.4% 30387383 cpuidle.C1.usage
-46.6% 21771278 ?114% cpuidle.C6.usage
-27.3% 6681582 ? 4% cpuidle.POLL.usage
830.83 ? 2% sched_debug.cfs_rq:/.load_avg.max
118.27 ? 8% sched_debug.cfs_rq:/.load_avg.stddev
858.57 ? 5% sched_debug.cfs_rq:/.runnable_avg.max
141.23 ? 6% sched_debug.cfs_rq:/.runnable_avg.stddev
856.87 ? 4% sched_debug.cfs_rq:/.util_avg.max
141.07 ? 6% sched_debug.cfs_rq:/.util_avg.stddev
680.73 ? 14% sched_debug.cfs_rq:/.util_est_enqueued.max
78.23 ? 11% sched_debug.cfs_rq:/.util_est_enqueued.stddev
-9.6% 757.83 sched_debug.cpu.clock_task.stddev
238943 sched_debug.cpu.nr_switches.avg
747256 ? 8% sched_debug.cpu.nr_switches.max
153162 ? 7% sched_debug.cpu.nr_switches.stddev
607882 ? 33% numa-meminfo.node1.Active
606197 ? 33% numa-meminfo.node1.Active(file)
302300 ? 25% numa-meminfo.node1.Dirty
613863 ? 24% numa-meminfo.node1.SUnreclaim
706259 ? 26% numa-meminfo.node1.Slab
2188 ? 13% numa-meminfo.node1.Writeback
488289 ? 18% numa-meminfo.node2.Dirty
2559 ? 12% numa-meminfo.node2.Writeback
424150 ? 2% numa-meminfo.node3.Active(anon)
59786 ? 17% numa-meminfo.node3.Inactive(anon)
21333 ? 16% numa-meminfo.node3.Mapped
458995 numa-meminfo.node3.Shmem
2586 ? 5% numa-meminfo.node3.Writeback
-27.3% 3684543 ? 3% meminfo.Active
430816 ? 2% meminfo.Active(anon)
-28.7% 3253726 ? 3% meminfo.Active(file)
290338 meminfo.AnonPages
-19.0% 37407254 meminfo.Cached
-16.0% 1635058 meminfo.Dirty
-18.4% 32973910 meminfo.Inactive
-18.9% 31526207 meminfo.Inactive(file)
-17.9% 42619789 meminfo.Memused
8383 meminfo.PageTables
-15.2% 2839492 meminfo.SUnreclaim
-12.8% 3542533 meminfo.Slab
9782 ? 7% meminfo.Writeback
254068 meminfo.max_used_kB
-22.2% 2029903 ? 27% numa-vmstat.node0.nr_dirtied
151628 ? 33% numa-vmstat.node1.nr_active_file
75572 ? 25% numa-vmstat.node1.nr_dirty
153531 ? 24% numa-vmstat.node1.nr_slab_unreclaimable
552.80 ? 11% numa-vmstat.node1.nr_writeback
151628 ? 33% numa-vmstat.node1.nr_zone_active_file
76023 ? 25% numa-vmstat.node1.nr_zone_write_pending
122076 ? 18% numa-vmstat.node2.nr_dirty
640.00 ? 12% numa-vmstat.node2.nr_writeback
122608 ? 18% numa-vmstat.node2.nr_zone_write_pending
105998 ? 2% numa-vmstat.node3.nr_active_anon
15119 ? 19% numa-vmstat.node3.nr_inactive_anon
5613 ? 10% numa-vmstat.node3.nr_mapped
114874 numa-vmstat.node3.nr_shmem
647.60 ? 5% numa-vmstat.node3.nr_writeback
105998 ? 2% numa-vmstat.node3.nr_zone_active_anon
15119 ? 19% numa-vmstat.node3.nr_zone_inactive_anon
-14.0% 6225179 slabinfo.Acpi-State.active_objs
122065 slabinfo.Acpi-State.active_slabs
-14.0% 6225334 slabinfo.Acpi-State.num_objs
122065 slabinfo.Acpi-State.num_slabs
15253 ? 6% slabinfo.btrfs_delayed_tree_ref.active_objs
399.60 ? 6% slabinfo.btrfs_delayed_tree_ref.active_slabs
15599 ? 6% slabinfo.btrfs_delayed_tree_ref.num_objs
399.60 ? 6% slabinfo.btrfs_delayed_tree_ref.num_slabs
159878 ? 2% slabinfo.btrfs_extent_buffer.active_objs
2402 ? 2% slabinfo.btrfs_extent_buffer.active_slabs
163417 ? 2% slabinfo.btrfs_extent_buffer.num_objs
2402 ? 2% slabinfo.btrfs_extent_buffer.num_slabs
5825 ? 4% slabinfo.dmaengine-unmap-16.active_objs
5825 ? 4% slabinfo.dmaengine-unmap-16.num_objs
44114 ? 7% slabinfo.fsnotify_mark_connector.active_objs
44606 ? 7% slabinfo.fsnotify_mark_connector.num_objs
26643 ? 5% slabinfo.khugepaged_mm_slot.active_objs
749.00 ? 5% slabinfo.khugepaged_mm_slot.active_slabs
26981 ? 5% slabinfo.khugepaged_mm_slot.num_objs
749.00 ? 5% slabinfo.khugepaged_mm_slot.num_slabs
-17.2% 13835925 slabinfo.pid_namespace.active_objs
247071 slabinfo.pid_namespace.active_slabs
-17.2% 13836058 slabinfo.pid_namespace.num_objs
247071 slabinfo.pid_namespace.num_slabs
107108 ? 2% proc-vmstat.nr_active_anon
811851 ? 3% proc-vmstat.nr_active_file
72623 proc-vmstat.nr_anon_pages
-18.1% 18721182 proc-vmstat.nr_dirtied
408475 proc-vmstat.nr_dirty
-19.0% 9335345 proc-vmstat.nr_file_pages
+6.4% 38790804 proc-vmstat.nr_free_pages
-3.9% 363103 proc-vmstat.nr_inactive_anon
-19.0% 7866105 proc-vmstat.nr_inactive_file
-1.2% 293034 proc-vmstat.nr_mapped
2088 proc-vmstat.nr_page_table_pages
-5.7% 397736 proc-vmstat.nr_shmem
-1.6% 175728 proc-vmstat.nr_slab_reclaimable
708791 proc-vmstat.nr_slab_unreclaimable
2416 ? 3% proc-vmstat.nr_writeback
-18.0% 18143705 proc-vmstat.nr_written
107108 ? 2% proc-vmstat.nr_zone_active_anon
811851 ? 3% proc-vmstat.nr_zone_active_file
-3.9% 363103 proc-vmstat.nr_zone_inactive_anon
-19.0% 7866105 proc-vmstat.nr_zone_inactive_file
410477 proc-vmstat.nr_zone_write_pending
53019 ? 5% proc-vmstat.numa_hint_faults
35265 ? 15% proc-vmstat.numa_hint_faults_local
-18.2% 19413142 proc-vmstat.numa_hit
-18.4% 19153143 proc-vmstat.numa_local
19977 ? 28% proc-vmstat.numa_pages_migrated
171462 ? 6% proc-vmstat.numa_pte_updates
-34.3% 1378408 ? 8% proc-vmstat.pgactivate
-17.7% 20377643 proc-vmstat.pgalloc_normal
-1.6% 1865772 proc-vmstat.pgfault
+2.9% 1796898 proc-vmstat.pgfree
19977 ? 28% proc-vmstat.pgmigrate_success
-18.0% 72577472 proc-vmstat.pgpgout
-13.7% 2.077e+09 perf-stat.i.branch-instructions
30.84 ? 5% perf-stat.i.cache-miss-rate%
+8.8% 33242366 ? 2% perf-stat.i.cache-misses
-11.9% 1.095e+08 ? 4% perf-stat.i.cache-references
296734 perf-stat.i.context-switches
1.90 perf-stat.i.cpi
-9.9% 1.997e+10 ? 2% perf-stat.i.cpu-cycles
-1.5% 210.90 perf-stat.i.cpu-migrations
619.06 perf-stat.i.cycles-between-cache-misses
-20.1% 3676350 ? 7% perf-stat.i.dTLB-load-misses
-13.4% 2.87e+09 perf-stat.i.dTLB-loads
-13.3% 1.489e+09 perf-stat.i.dTLB-stores
-10.6% 6891530 ? 2% perf-stat.i.iTLB-load-misses
-12.5% 12341846 perf-stat.i.iTLB-loads
-13.5% 1.08e+10 perf-stat.i.instructions
1564 perf-stat.i.instructions-per-iTLB-miss
0.54 perf-stat.i.ipc
0.10 ? 2% perf-stat.i.metric.GHz
34.23 perf-stat.i.metric.M/sec
4195 perf-stat.i.minor-faults
81.57 perf-stat.i.node-load-miss-rate%
+25.1% 2303179 ? 3% perf-stat.i.node-loads
-17.7% 1702288 ? 3% perf-stat.i.node-store-misses
5119 perf-stat.i.page-faults
30.45 ? 6% perf-stat.overall.cache-miss-rate%
1.85 perf-stat.overall.cpi
600.93 perf-stat.overall.cycles-between-cache-misses
1567 perf-stat.overall.instructions-per-iTLB-miss
0.54 perf-stat.overall.ipc
81.87 perf-stat.overall.node-load-miss-rate%
+5.7% 214240 perf-stat.overall.path-length
-13.7% 2.071e+09 perf-stat.ps.branch-instructions
+8.8% 33128519 ? 2% perf-stat.ps.cache-misses
-11.9% 1.091e+08 ? 4% perf-stat.ps.cache-references
295738 perf-stat.ps.context-switches
-9.9% 1.991e+10 ? 2% perf-stat.ps.cpu-cycles
-1.5% 210.23 perf-stat.ps.cpu-migrations
-20.0% 3664346 ? 7% perf-stat.ps.dTLB-load-misses
-13.4% 2.861e+09 perf-stat.ps.dTLB-loads
-13.3% 1.484e+09 perf-stat.ps.dTLB-stores
-10.6% 6868645 ? 2% perf-stat.ps.iTLB-load-misses
-12.5% 12300074 perf-stat.ps.iTLB-loads
-13.5% 1.076e+10 perf-stat.ps.instructions
4183 perf-stat.ps.minor-faults
+25.2% 2295599 ? 3% perf-stat.ps.node-loads
-17.7% 1696720 ? 3% perf-stat.ps.node-store-misses
5116 perf-stat.ps.page-faults
-13.7% 3.253e+12 perf-stat.total.instructions
0.01 ? 16% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
0.01 ? 14% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.down_read.__btrfs_tree_read_lock.btrfs_search_slot
0.01 ? 52% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.alloc_extent_map.btrfs_drop_extent_cache
0.01 ? 26% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.btrfs_bio_alloc
0.01 ? 36% perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
0.01 ? 45% perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_csums_range
0.00 perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_csum
0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
4.48 ? 12% perf-sched.sch_delay.max.ms.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
0.01 ? 30% perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
0.01 ? 12% perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.btrfs_bio_alloc
0.01 ? 42% perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
0.03 ? 51% perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
0.00 perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_csum
0.03 ? 54% perf-sched.sch_delay.max.ms.schedule_timeout.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
0.06 ? 70% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
2.48 ? 4% perf-sched.total_wait_and_delay.average.ms
-18.1% 1301217 ? 2% perf-sched.total_wait_and_delay.count.ms
2.47 ? 4% perf-sched.total_wait_time.average.ms
0.16 ? 2% perf-sched.wait_and_delay.avg.ms.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
37.79 ? 2% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
9.25 ? 29% perf-sched.wait_and_delay.avg.ms.schedule_timeout.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
1.16 ? 8% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
438897 ? 2% perf-sched.wait_and_delay.count.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
26.20 ? 52% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork
470.80 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
237.20 ? 4% perf-sched.wait_and_delay.count.schedule_timeout.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 perf-sched.wait_and_delay.count.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
844244 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
3250 ? 45% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
5.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
284.07 ? 22% perf-sched.wait_and_delay.max.ms.schedule_timeout.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
5.05 perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
0.15 ? 2% perf-sched.wait_time.avg.ms.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
9.29 ?111% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.alloc_extent_state.set_extent_bit
41.54 ? 92% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.btrfs_bio_alloc
1.03 ? 73% perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_csums_range
0.02 ? 70% perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_file_extent
0.00 perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_csum
37.79 ? 2% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
9.24 ? 29% perf-sched.wait_time.avg.ms.schedule_timeout.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
1.16 ? 8% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
136.63 ? 97% perf-sched.wait_time.max.ms.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
3250 ? 45% perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
1.87 ? 74% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.alloc_extent_state.__clear_extent_bit
76.54 ?126% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.alloc_extent_state.set_extent_bit
0.02 ? 87% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.btrfs_add_delayed_data_ref.btrfs_inc_extent_ref
184.99 ? 95% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.btrfs_bio_alloc
0.86 ?123% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
4.99 perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
2.06 ? 48% perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_csums_range
0.03 ? 60% perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_file_extent
0.00 perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_csum
284.06 ? 22% perf-sched.wait_time.max.ms.schedule_timeout.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.start_delalloc_inodes.btrfs_start_delalloc_roots
32554 ? 75% interrupts.32:PCI-MSI.524290-edge.eth0-TxRx-1
-5.8% 894611 ? 4% interrupts.CAL:Function_call_interrupts
323.80 ? 40% interrupts.CPU10.RES:Rescheduling_interrupts
6313 ? 45% interrupts.CPU101.RES:Rescheduling_interrupts
7357 ? 43% interrupts.CPU102.RES:Rescheduling_interrupts
6937 ? 59% interrupts.CPU104.RES:Rescheduling_interrupts
6991 ? 55% interrupts.CPU105.RES:Rescheduling_interrupts
9281 ? 51% interrupts.CPU106.RES:Rescheduling_interrupts
5940 ? 58% interrupts.CPU107.RES:Rescheduling_interrupts
4348 ? 31% interrupts.CPU108.CAL:Function_call_interrupts
5177 ? 55% interrupts.CPU108.RES:Rescheduling_interrupts
6455 ? 57% interrupts.CPU109.RES:Rescheduling_interrupts
32554 ? 75% interrupts.CPU11.32:PCI-MSI.524290-edge.eth0-TxRx-1
4770 ? 64% interrupts.CPU111.RES:Rescheduling_interrupts
4863 ? 65% interrupts.CPU112.RES:Rescheduling_interrupts
5088 ? 97% interrupts.CPU114.RES:Rescheduling_interrupts
4988 ? 53% interrupts.CPU117.RES:Rescheduling_interrupts
4813 ? 42% interrupts.CPU118.RES:Rescheduling_interrupts
316.40 ? 61% interrupts.CPU12.RES:Rescheduling_interrupts
6944 ? 56% interrupts.CPU121.RES:Rescheduling_interrupts
6135 ? 61% interrupts.CPU122.RES:Rescheduling_interrupts
4374 ? 38% interrupts.CPU123.RES:Rescheduling_interrupts
4486 ? 20% interrupts.CPU124.CAL:Function_call_interrupts
4582 ? 53% interrupts.CPU124.RES:Rescheduling_interrupts
4450 ? 61% interrupts.CPU125.RES:Rescheduling_interrupts
4257 ? 54% interrupts.CPU126.RES:Rescheduling_interrupts
3877 ? 57% interrupts.CPU127.RES:Rescheduling_interrupts
5728 ? 27% interrupts.CPU128.RES:Rescheduling_interrupts
5773 ? 75% interrupts.CPU129.RES:Rescheduling_interrupts
6382 ? 50% interrupts.CPU130.RES:Rescheduling_interrupts
6613 ? 62% interrupts.CPU131.RES:Rescheduling_interrupts
6545 ? 36% interrupts.CPU132.RES:Rescheduling_interrupts
4950 ? 34% interrupts.CPU133.CAL:Function_call_interrupts
6986 ? 48% interrupts.CPU133.RES:Rescheduling_interrupts
6127 ? 52% interrupts.CPU134.RES:Rescheduling_interrupts
6069 ? 61% interrupts.CPU135.RES:Rescheduling_interrupts
3932 ? 33% interrupts.CPU136.CAL:Function_call_interrupts
3998 ? 72% interrupts.CPU136.RES:Rescheduling_interrupts
5871 ? 47% interrupts.CPU137.RES:Rescheduling_interrupts
6910 ? 42% interrupts.CPU138.RES:Rescheduling_interrupts
4475 ? 64% interrupts.CPU139.RES:Rescheduling_interrupts
3405 ? 41% interrupts.CPU14.CAL:Function_call_interrupts
5660 ? 64% interrupts.CPU141.RES:Rescheduling_interrupts
4666 ? 66% interrupts.CPU142.RES:Rescheduling_interrupts
3734 ? 24% interrupts.CPU143.CAL:Function_call_interrupts
2830 ? 65% interrupts.CPU143.RES:Rescheduling_interrupts
6773 ? 11% interrupts.CPU150.CAL:Function_call_interrupts
577.60 ? 18% interrupts.CPU154.NMI:Non-maskable_interrupts
577.60 ? 18% interrupts.CPU154.PMI:Performance_monitoring_interrupts
487.60 ? 26% interrupts.CPU155.NMI:Non-maskable_interrupts
487.60 ? 26% interrupts.CPU155.PMI:Performance_monitoring_interrupts
7781 ? 26% interrupts.CPU170.RES:Rescheduling_interrupts
484.20 ? 38% interrupts.CPU175.NMI:Non-maskable_interrupts
484.20 ? 38% interrupts.CPU175.PMI:Performance_monitoring_interrupts
6586 ? 21% interrupts.CPU178.CAL:Function_call_interrupts
9067 ? 44% interrupts.CPU186.RES:Rescheduling_interrupts
5827 ? 17% interrupts.CPU190.CAL:Function_call_interrupts
4415 ? 21% interrupts.CPU2.CAL:Function_call_interrupts
520.00 ? 48% interrupts.CPU2.RES:Rescheduling_interrupts
44.20 ?124% interrupts.CPU2.TLB:TLB_shootdowns
4203 ? 17% interrupts.CPU28.CAL:Function_call_interrupts
526.20 ? 46% interrupts.CPU28.RES:Rescheduling_interrupts
3590 ? 20% interrupts.CPU3.CAL:Function_call_interrupts
414.00 ? 50% interrupts.CPU3.RES:Rescheduling_interrupts
507.60 ? 60% interrupts.CPU4.RES:Rescheduling_interrupts
2168 ? 90% interrupts.CPU40.RES:Rescheduling_interrupts
188.60 ? 65% interrupts.CPU46.NMI:Non-maskable_interrupts
188.60 ? 65% interrupts.CPU46.PMI:Performance_monitoring_interrupts
2849 ? 92% interrupts.CPU46.RES:Rescheduling_interrupts
3225 ? 29% interrupts.CPU47.CAL:Function_call_interrupts
2266 ? 77% interrupts.CPU47.RES:Rescheduling_interrupts
219.60 ? 68% interrupts.CPU50.NMI:Non-maskable_interrupts
219.60 ? 68% interrupts.CPU50.PMI:Performance_monitoring_interrupts
456.00 ? 26% interrupts.CPU58.NMI:Non-maskable_interrupts
456.00 ? 26% interrupts.CPU58.PMI:Performance_monitoring_interrupts
539.20 ? 36% interrupts.CPU59.NMI:Non-maskable_interrupts
539.20 ? 36% interrupts.CPU59.PMI:Performance_monitoring_interrupts
3039 ? 21% interrupts.CPU64.CAL:Function_call_interrupts
3386 ? 18% interrupts.CPU7.CAL:Function_call_interrupts
354.60 ? 61% interrupts.CPU7.RES:Rescheduling_interrupts
5275 ? 7% interrupts.CPU72.CAL:Function_call_interrupts
395.00 ? 73% interrupts.CPU72.TLB:TLB_shootdowns
102.60 ? 41% interrupts.CPU73.TLB:TLB_shootdowns
4309 ? 8% interrupts.CPU77.CAL:Function_call_interrupts
604.80 ? 43% interrupts.CPU77.NMI:Non-maskable_interrupts
604.80 ? 43% interrupts.CPU77.PMI:Performance_monitoring_interrupts
430.20 ? 12% interrupts.CPU79.NMI:Non-maskable_interrupts
430.20 ? 12% interrupts.CPU79.PMI:Performance_monitoring_interrupts
103.40 ? 38% interrupts.CPU79.TLB:TLB_shootdowns
3402 ? 16% interrupts.CPU8.CAL:Function_call_interrupts
42.60 ?113% interrupts.CPU8.TLB:TLB_shootdowns
141.20 ? 57% interrupts.CPU81.TLB:TLB_shootdowns
124.60 ? 35% interrupts.CPU82.TLB:TLB_shootdowns
215.20 ? 35% interrupts.CPU9.NMI:Non-maskable_interrupts
215.20 ? 35% interrupts.CPU9.PMI:Performance_monitoring_interrupts
1187 ?154% interrupts.CPU91.RES:Rescheduling_interrupts
112.00 ? 40% interrupts.CPU94.TLB:TLB_shootdowns
532.60 ? 2% interrupts.IWI:IRQ_work_interrupts
78698 ? 4% interrupts.NMI:Non-maskable_interrupts
78698 ? 4% interrupts.PMI:Performance_monitoring_interrupts
798247 ? 3% interrupts.RES:Rescheduling_interrupts
21457 ? 11% softirqs.CPU0.RCU
17724 ? 4% softirqs.CPU1.RCU
17007 ? 2% softirqs.CPU10.RCU
17286 ? 6% softirqs.CPU100.RCU
17101 ? 3% softirqs.CPU101.RCU
17193 ? 3% softirqs.CPU102.RCU
17097 ? 3% softirqs.CPU103.RCU
17194 ? 5% softirqs.CPU104.RCU
16843 ? 3% softirqs.CPU105.RCU
16579 ? 6% softirqs.CPU106.RCU
16926 ? 4% softirqs.CPU107.RCU
16907 ? 4% softirqs.CPU108.RCU
16701 ? 2% softirqs.CPU109.RCU
17336 ? 75% softirqs.CPU11.NET_RX
17492 ? 3% softirqs.CPU11.RCU
5066 ? 77% softirqs.CPU11.TIMER
17091 ? 5% softirqs.CPU110.RCU
16849 ? 4% softirqs.CPU111.RCU
15258 ? 2% softirqs.CPU112.RCU
15501 ? 2% softirqs.CPU113.RCU
15576 ? 4% softirqs.CPU114.RCU
15608 ? 3% softirqs.CPU115.RCU
15450 ? 6% softirqs.CPU116.RCU
15459 ? 2% softirqs.CPU117.RCU
15267 ? 3% softirqs.CPU118.RCU
15607 ? 3% softirqs.CPU119.RCU
17048 ? 2% softirqs.CPU12.RCU
15312 ? 8% softirqs.CPU120.RCU
15229 ? 5% softirqs.CPU121.RCU
15139 ? 5% softirqs.CPU122.RCU
15031 ? 6% softirqs.CPU123.RCU
15308 ? 5% softirqs.CPU124.RCU
15163 ? 5% softirqs.CPU125.RCU
15279 ? 5% softirqs.CPU126.RCU
15261 ? 5% softirqs.CPU127.RCU
16411 ? 6% softirqs.CPU128.RCU
16169 ? 6% softirqs.CPU129.RCU
17098 ? 4% softirqs.CPU13.RCU
16103 ? 9% softirqs.CPU130.RCU
16635 ? 6% softirqs.CPU131.RCU
16627 ? 7% softirqs.CPU132.RCU
16490 ? 4% softirqs.CPU133.RCU
16083 ? 6% softirqs.CPU134.RCU
16461 ? 7% softirqs.CPU135.RCU
16252 ? 6% softirqs.CPU136.RCU
16685 ? 4% softirqs.CPU137.RCU
16616 ? 6% softirqs.CPU138.RCU
16242 ? 5% softirqs.CPU139.RCU
18706 ? 6% softirqs.CPU14.RCU
16238 ? 5% softirqs.CPU140.RCU
16298 ? 5% softirqs.CPU141.RCU
16408 ? 4% softirqs.CPU142.RCU
16470 ? 5% softirqs.CPU143.RCU
17286 ? 5% softirqs.CPU144.RCU
16611 ? 3% softirqs.CPU145.RCU
16732 ? 2% softirqs.CPU146.RCU
16764 ? 2% softirqs.CPU147.RCU
17001 ? 2% softirqs.CPU148.RCU
16957 ? 3% softirqs.CPU149.RCU
17039 ? 4% softirqs.CPU15.RCU
17104 ? 4% softirqs.CPU150.RCU
16649 ? 3% softirqs.CPU151.RCU
16550 ? 3% softirqs.CPU152.RCU
16605 softirqs.CPU153.RCU
16719 ? 3% softirqs.CPU154.RCU
17051 ? 3% softirqs.CPU155.RCU
17107 ? 3% softirqs.CPU156.RCU
15996 ? 6% softirqs.CPU157.RCU
16606 ? 2% softirqs.CPU158.RCU
16830 ? 4% softirqs.CPU159.RCU
15540 ? 3% softirqs.CPU16.RCU
15270 ? 2% softirqs.CPU160.RCU
15693 ? 2% softirqs.CPU161.RCU
15522 ? 3% softirqs.CPU162.RCU
15600 ? 3% softirqs.CPU163.RCU
15549 ? 4% softirqs.CPU164.RCU
15199 ? 3% softirqs.CPU165.RCU
15465 ? 2% softirqs.CPU166.RCU
15755 ? 4% softirqs.CPU167.RCU
14824 ? 2% softirqs.CPU168.RCU
14667 ? 4% softirqs.CPU169.RCU
15987 ? 3% softirqs.CPU17.RCU
14508 ? 3% softirqs.CPU170.RCU
14943 ? 3% softirqs.CPU171.RCU
14970 ? 4% softirqs.CPU172.RCU
14953 ? 3% softirqs.CPU173.RCU
14826 ? 2% softirqs.CPU174.RCU
14990 ? 3% softirqs.CPU175.RCU
16203 ? 3% softirqs.CPU176.RCU
16040 ? 3% softirqs.CPU177.RCU
16195 ? 4% softirqs.CPU178.RCU
16471 ? 4% softirqs.CPU179.RCU
15839 softirqs.CPU18.RCU
15936 ? 3% softirqs.CPU180.RCU
15993 ? 3% softirqs.CPU181.RCU
16216 ? 4% softirqs.CPU182.RCU
16180 ? 4% softirqs.CPU183.RCU
16058 ? 4% softirqs.CPU184.RCU
17055 ? 12% softirqs.CPU185.RCU
15876 ? 3% softirqs.CPU186.RCU
16536 ? 4% softirqs.CPU187.RCU
16915 ? 12% softirqs.CPU188.RCU
15911 ? 3% softirqs.CPU189.RCU
15485 ? 2% softirqs.CPU19.RCU
16086 ? 4% softirqs.CPU190.RCU
16292 ? 3% softirqs.CPU191.RCU
17548 ? 2% softirqs.CPU2.RCU
15283 ? 4% softirqs.CPU20.RCU
15465 ? 2% softirqs.CPU21.RCU
15574 softirqs.CPU22.RCU
15704 ? 2% softirqs.CPU23.RCU
16087 ? 8% softirqs.CPU24.RCU
15730 ? 4% softirqs.CPU25.RCU
15744 ? 4% softirqs.CPU26.RCU
15682 ? 7% softirqs.CPU27.RCU
16008 ? 5% softirqs.CPU28.RCU
15743 ? 5% softirqs.CPU29.RCU
18154 ? 7% softirqs.CPU3.RCU
15656 ? 5% softirqs.CPU30.RCU
15544 ? 4% softirqs.CPU31.RCU
17137 ? 7% softirqs.CPU32.RCU
16458 ? 6% softirqs.CPU33.RCU
16778 ? 10% softirqs.CPU34.RCU
17097 ? 6% softirqs.CPU35.RCU
17082 ? 6% softirqs.CPU36.RCU
16651 ? 6% softirqs.CPU37.RCU
16744 ? 7% softirqs.CPU38.RCU
16571 ? 6% softirqs.CPU39.RCU
17393 ? 5% softirqs.CPU4.RCU
16645 ? 5% softirqs.CPU40.RCU
16986 ? 5% softirqs.CPU41.RCU
17051 ? 6% softirqs.CPU42.RCU
16777 ? 6% softirqs.CPU43.RCU
16552 ? 5% softirqs.CPU44.RCU
16562 ? 5% softirqs.CPU45.RCU
16518 ? 6% softirqs.CPU46.RCU
16636 ? 5% softirqs.CPU47.RCU
18463 ? 3% softirqs.CPU48.RCU
17123 ? 3% softirqs.CPU49.RCU
17708 ? 7% softirqs.CPU5.RCU
17477 ? 3% softirqs.CPU50.RCU
17695 ? 3% softirqs.CPU51.RCU
17627 ? 4% softirqs.CPU52.RCU
17688 ? 3% softirqs.CPU53.RCU
17676 ? 3% softirqs.CPU54.RCU
17420 ? 3% softirqs.CPU55.RCU
17464 ? 4% softirqs.CPU56.RCU
16927 ? 2% softirqs.CPU57.RCU
17115 ? 3% softirqs.CPU58.RCU
17231 ? 4% softirqs.CPU59.RCU
17185 ? 4% softirqs.CPU6.RCU
17208 ? 4% softirqs.CPU60.RCU
16596 ? 7% softirqs.CPU61.RCU
16918 ? 2% softirqs.CPU62.RCU
16909 ? 3% softirqs.CPU63.RCU
15478 softirqs.CPU64.RCU
15823 ? 3% softirqs.CPU65.RCU
15773 ? 3% softirqs.CPU66.RCU
15796 ? 2% softirqs.CPU67.RCU
15907 ? 4% softirqs.CPU68.RCU
15701 ? 2% softirqs.CPU69.RCU
17398 ? 4% softirqs.CPU7.RCU
15778 ? 3% softirqs.CPU70.RCU
15799 softirqs.CPU71.RCU
17158 ? 13% softirqs.CPU72.RCU
16094 ? 5% softirqs.CPU73.RCU
15952 ? 5% softirqs.CPU74.RCU
15730 ? 3% softirqs.CPU75.RCU
15879 ? 4% softirqs.CPU76.RCU
15482 ? 2% softirqs.CPU77.RCU
15537 ? 4% softirqs.CPU78.RCU
15552 ? 3% softirqs.CPU79.RCU
17425 ? 4% softirqs.CPU8.RCU
16237 ? 3% softirqs.CPU80.RCU
16186 ? 3% softirqs.CPU81.RCU
16340 ? 5% softirqs.CPU82.RCU
16273 ? 4% softirqs.CPU83.RCU
16051 ? 4% softirqs.CPU84.RCU
15939 ? 3% softirqs.CPU85.RCU
15895 ? 4% softirqs.CPU86.RCU
15924 ? 3% softirqs.CPU87.RCU
16012 ? 2% softirqs.CPU88.RCU
15899 ? 4% softirqs.CPU89.RCU
17012 ? 3% softirqs.CPU9.RCU
16057 ? 3% softirqs.CPU90.RCU
16761 ? 7% softirqs.CPU91.RCU
15920 ? 4% softirqs.CPU92.RCU
15806 ? 2% softirqs.CPU93.RCU
16015 ? 4% softirqs.CPU94.RCU
16203 ? 3% softirqs.CPU95.RCU
16782 ? 2% softirqs.CPU96.RCU
16914 ? 7% softirqs.CPU97.RCU
17180 ? 5% softirqs.CPU98.RCU
17094 ? 5% softirqs.CPU99.RCU
3141962 ? 3% softirqs.RCU
0.00 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_run_delalloc_work.btrfs_work_helper.process_one_work.worker_thread
0.00 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_run_delalloc_work.btrfs_work_helper.process_one_work
0.00 perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_work.btrfs_work_helper.process_one_work.worker_thread.kthread
0.00 perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_run_delalloc_work.btrfs_work_helper
0.00 perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_run_delalloc_work
0.00 perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range
18.13 ? 2% perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
2.60 ? 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
2.76 ? 20% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction
2.76 ? 20% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.transaction_kthread.kthread
2.76 ? 20% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.transaction_kthread
2.76 ? 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction
2.77 ? 20% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
2.83 ? 19% perf-profile.calltrace.cycles-pp.transaction_kthread.kthread.ret_from_fork
2.83 ? 19% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork
2.38 ? 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
2.32 ? 20% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages
2.32 ? 20% perf-profile.calltrace.cycles-pp.btrfs_submit_metadata_bio.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages
2.29 ? 20% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btrfs_submit_metadata_bio.submit_one_bio.submit_extent_page.write_one_eb
2.24 ? 20% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btrfs_submit_metadata_bio.submit_one_bio.submit_extent_page
1.14 ? 5% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.14 ? 5% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
1.06 ? 5% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range
1.43 ? 6% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
1.18 ? 20% perf-profile.calltrace.cycles-pp.check_extent_data_item.check_leaf.btree_csum_one_bio.btrfs_submit_metadata_bio.submit_one_bio
1.30 ? 6% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
6.41 ? 3% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
0.31 ? 81% perf-profile.calltrace.cycles-pp.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range
0.82 ? 7% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
0.36 ? 82% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs_for_head.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space
0.36 ? 81% perf-profile.calltrace.cycles-pp.btrfs_get_32.check_leaf.btree_csum_one_bio.btrfs_submit_metadata_bio.submit_one_bio
0.73 ? 7% perf-profile.calltrace.cycles-pp.find_lock_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
0.55 ? 52% perf-profile.calltrace.cycles-pp.btrfs_get_64.check_extent_data_item.check_leaf.btree_csum_one_bio.btrfs_submit_metadata_bio
0.61 ? 5% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages
1.48 ? 5% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
0.31 ? 81% perf-profile.calltrace.cycles-pp.__btrfs_tree_lock.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper
0.44 ? 50% perf-profile.calltrace.cycles-pp.__etree_search.btrfs_find_delalloc_range.find_lock_delalloc_range.writepage_delalloc.__extent_writepage
1.25 ? 6% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
1.47 ? 5% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_finish_ordered_io.btrfs_work_helper
1.49 ? 5% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work
0.72 ? 6% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range
0.61 ? 7% perf-profile.calltrace.cycles-pp.btrfs_find_delalloc_range.find_lock_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
0.63 ? 8% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
0.69 ? 10% perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
0.70 ? 5% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_finish_ordered_io
0.59 ? 5% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
1.35 ? 3% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_finish_ordered_io.btrfs_work_helper
1.20 ? 3% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.76 ? 4% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.setup_items_for_insert.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_finish_ordered_io
0.91 ? 8% perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
0.90 ? 7% perf-profile.calltrace.cycles-pp.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
1.70 ? 2% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_finish_ordered_io.btrfs_work_helper
1.28 ? 10% perf-profile.calltrace.cycles-pp.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io
1.70 ? 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
2.19 ? 4% perf-profile.calltrace.cycles-pp.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter
2.20 ? 4% perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
1.36 ? 10% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper
2.35 ? 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
1.19 ? 11% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
0.92 ? 16% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node
1.23 ? 11% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written
1.50 ? 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__reserve_bytes.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata
1.03 ? 14% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node
7.18 ? 6% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.11 ? 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
7.16 ? 6% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
1.28 ? 12% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
1.33 ? 11% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written
1.36 ? 11% perf-profile.calltrace.cycles-pp.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io
1.38 ? 11% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper
4.63 ? 7% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work
16.98 ? 3% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread.kthread
2.04 ? 14% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
2.04 ? 14% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
2.04 ? 14% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
2.04 ? 14% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
2.04 ? 14% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
2.04 ? 14% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
2.04 ? 14% perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
2.04 ? 14% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
8.24 ? 5% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread
8.23 ? 6% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread
7.17 ? 5% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_single_inode.start_delalloc_inodes.btrfs_start_delalloc_roots
7.17 ? 5% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_single_inode.start_delalloc_inodes.btrfs_start_delalloc_roots.flush_space
7.17 ? 5% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_single_inode.start_delalloc_inodes
7.17 ? 5% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_single_inode
7.17 ? 5% perf-profile.calltrace.cycles-pp.writeback_single_inode.start_delalloc_inodes.btrfs_start_delalloc_roots.flush_space.btrfs_async_reclaim_metadata_space
7.17 ? 5% perf-profile.calltrace.cycles-pp.btrfs_start_delalloc_roots.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
7.17 ? 5% perf-profile.calltrace.cycles-pp.start_delalloc_inodes.btrfs_start_delalloc_roots.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
8.30 ? 6% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork
8.86 ? 4% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode
2.76 ? 20% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
0.00 perf-profile.children.cycles-pp.btrfs_run_delalloc_work
18.13 ? 2% perf-profile.children.cycles-pp.btrfs_work_helper
2.60 ? 20% perf-profile.children.cycles-pp.write_one_eb
3.18 ? 15% perf-profile.children.cycles-pp.submit_extent_page
2.76 ? 20% perf-profile.children.cycles-pp.btrfs_write_marked_extents
2.76 ? 20% perf-profile.children.cycles-pp.btree_write_cache_pages
2.77 ? 20% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction
2.83 ? 19% perf-profile.children.cycles-pp.transaction_kthread
2.83 ? 19% perf-profile.children.cycles-pp.btrfs_commit_transaction
2.96 ? 17% perf-profile.children.cycles-pp.submit_one_bio
2.40 ? 20% perf-profile.children.cycles-pp.btrfs_submit_metadata_bio
2.37 ? 20% perf-profile.children.cycles-pp.btree_csum_one_bio
2.34 ? 20% perf-profile.children.cycles-pp.check_leaf
2.42 ? 5% perf-profile.children.cycles-pp.rwsem_spin_on_owner
1.24 ? 21% perf-profile.children.cycles-pp.check_extent_data_item
6.64 ? 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.46 ? 7% perf-profile.children.cycles-pp.btrfs_lookup_csums_range
1.46 ? 7% perf-profile.children.cycles-pp.csum_exist_in_range
1.49 ? 5% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.12 ? 13% perf-profile.children.cycles-pp.btrfs_get_32
0.84 ? 18% perf-profile.children.cycles-pp.btrfs_get_64
1.28 ? 6% perf-profile.children.cycles-pp.tick_nohz_next_event
1.50 ? 5% perf-profile.children.cycles-pp.btrfs_lookup_csum
1.00 ? 9% perf-profile.children.cycles-pp.check_setget_bounds
0.87 ? 5% perf-profile.children.cycles-pp.find_lock_delalloc_range
0.55 ? 15% perf-profile.children.cycles-pp.btrfs_run_delayed_refs_for_head
0.64 ? 8% perf-profile.children.cycles-pp.poll_idle
0.91 ? 5% perf-profile.children.cycles-pp.__push_leaf_right
1.74 ? 3% perf-profile.children.cycles-pp.split_leaf
1.43 ? 4% perf-profile.children.cycles-pp.push_leaf_right
0.89 ? 10% perf-profile.children.cycles-pp.btrfs_cow_block
0.29 ? 14% perf-profile.children.cycles-pp.copy_extent_buffer_full
0.69 ? 10% perf-profile.children.cycles-pp.timekeeping_max_deferment
0.89 ? 11% perf-profile.children.cycles-pp.__btrfs_cow_block
0.29 ? 14% perf-profile.children.cycles-pp.copy_page
0.29 ? 15% perf-profile.children.cycles-pp.__btrfs_free_extent
0.16 ? 10% perf-profile.children.cycles-pp.btrfs_del_items
0.47 ? 2% perf-profile.children.cycles-pp.btrfs_extend_item
1.17 ? 2% perf-profile.children.cycles-pp.try_to_wake_up
0.47 ? 12% perf-profile.children.cycles-pp.read_extent_buffer
1.04 ? 3% perf-profile.children.cycles-pp.__clear_extent_bit
0.62 ? 4% perf-profile.children.cycles-pp.kmem_cache_alloc
0.30 ? 16% perf-profile.children.cycles-pp.btrfs_get_8
0.41 ? 4% perf-profile.children.cycles-pp.clear_state_bit
0.16 ? 9% perf-profile.children.cycles-pp.serial8250_console_write
1.27 ? 3% perf-profile.children.cycles-pp.irq_exit_rcu
0.18 ? 9% perf-profile.children.cycles-pp.irq_work_run_list
0.16 ? 9% perf-profile.children.cycles-pp.uart_console_write
0.17 ? 9% perf-profile.children.cycles-pp.asm_sysvec_irq_work
0.17 ? 9% perf-profile.children.cycles-pp.sysvec_irq_work
0.17 ? 9% perf-profile.children.cycles-pp.__sysvec_irq_work
0.17 ? 9% perf-profile.children.cycles-pp.irq_work_run
0.17 ? 9% perf-profile.children.cycles-pp.irq_work_single
0.17 ? 9% perf-profile.children.cycles-pp.printk
0.17 ? 9% perf-profile.children.cycles-pp.vprintk_emit
0.17 ? 9% perf-profile.children.cycles-pp.console_unlock
0.17 ? 6% perf-profile.children.cycles-pp.hrtimer_next_event_without
0.89 ? 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.22 ? 6% perf-profile.children.cycles-pp.queued_write_lock_slowpath
0.54 ? 4% perf-profile.children.cycles-pp.__wake_up_common
0.16 ? 9% perf-profile.children.cycles-pp.wait_for_xmitr
0.38 ? 5% perf-profile.children.cycles-pp.kmem_cache_free
0.16 ? 9% perf-profile.children.cycles-pp.serial8250_console_putchar
0.51 ? 5% perf-profile.children.cycles-pp.autoremove_wake_function
0.21 ? 6% perf-profile.children.cycles-pp.__slab_alloc
0.19 ? 7% perf-profile.children.cycles-pp.___slab_alloc
0.22 ? 5% perf-profile.children.cycles-pp.copy_pages
0.21 ? 6% perf-profile.children.cycles-pp.get_next_timer_interrupt
0.14 ? 15% perf-profile.children.cycles-pp.crc_128
0.17 ? 5% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.29 ? 5% perf-profile.children.cycles-pp.btrfs_free_path
0.18 ? 10% perf-profile.children.cycles-pp.merge_state
0.18 ? 7% perf-profile.children.cycles-pp.__push_leaf_left
0.37 ? 4% perf-profile.children.cycles-pp.sched_clock_cpu
0.34 ? 4% perf-profile.children.cycles-pp.sched_clock
0.11 ? 14% perf-profile.children.cycles-pp.io_serial_in
0.32 ? 4% perf-profile.children.cycles-pp.native_sched_clock
0.05 ? 53% perf-profile.children.cycles-pp.memzero_extent_buffer
0.26 ? 7% perf-profile.children.cycles-pp.alloc_extent_state
0.12 ? 9% perf-profile.children.cycles-pp.hrtimer_get_next_event
0.10 ? 10% perf-profile.children.cycles-pp._raw_write_lock
0.07 ? 11% perf-profile.children.cycles-pp.btrfs_delalloc_release_metadata
0.08 ? 9% perf-profile.children.cycles-pp.__smp_call_single_queue
0.08 ? 9% perf-profile.children.cycles-pp.llist_add_batch
0.07 ? 12% perf-profile.children.cycles-pp.allocate_slab
0.06 ? 7% perf-profile.children.cycles-pp.btrfs_free_tree_block
0.14 ? 7% perf-profile.children.cycles-pp.verify_parent_transid
0.24 ? 7% perf-profile.children.cycles-pp.btrfs_delayed_update_inode
0.19 ? 6% perf-profile.children.cycles-pp.leaf_space_used
0.19 ? 8% perf-profile.children.cycles-pp.btrfs_buffer_uptodate
0.31 ? 9% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref
0.30 ? 9% perf-profile.children.cycles-pp.btrfs_inc_extent_ref
0.21 ? 14% perf-profile.children.cycles-pp.xas_find_marked
0.08 ? 19% perf-profile.children.cycles-pp.rwsem_wake
0.34 ? 8% perf-profile.children.cycles-pp.down_read
0.09 ? 22% perf-profile.children.cycles-pp._find_next_bit
0.38 ? 10% perf-profile.children.cycles-pp.btrfs_delayed_refs_rsv_release
0.29 ? 4% perf-profile.children.cycles-pp.mark_page_accessed
0.35 ? 4% perf-profile.children.cycles-pp.mark_extent_buffer_accessed
0.49 ? 5% perf-profile.children.cycles-pp.__radix_tree_lookup
0.97 ? 6% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys
1.26 ? 5% perf-profile.children.cycles-pp.generic_bin_search
1.90 ? 5% perf-profile.children.cycles-pp.btrfs_block_rsv_release
1.42 ? 3% perf-profile.children.cycles-pp.find_extent_buffer
1.78 ? 4% perf-profile.children.cycles-pp.read_block_for_search
3.19 ? 3% perf-profile.children.cycles-pp.setup_items_for_insert
3.50 ? 5% perf-profile.children.cycles-pp.__btrfs_tree_read_lock
2.72 ? 4% perf-profile.children.cycles-pp.__btrfs_tree_lock
2.62 ? 5% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
2.20 ? 4% perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
2.28 ? 4% perf-profile.children.cycles-pp.__reserve_bytes
2.35 ? 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
5.69 ? 5% perf-profile.children.cycles-pp.rwsem_optimistic_spin
4.96 ? 4% perf-profile.children.cycles-pp._raw_spin_lock
2.20 ? 9% perf-profile.children.cycles-pp.btrfs_read_lock_root_node
1.60 ? 10% perf-profile.children.cycles-pp.btrfs_lock_root_node
2.92 ? 5% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
7.27 ? 6% perf-profile.children.cycles-pp.ksys_write
7.19 ? 6% perf-profile.children.cycles-pp.new_sync_write
7.58 ? 6% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
7.53 ? 6% perf-profile.children.cycles-pp.do_syscall_64
7.26 ? 6% perf-profile.children.cycles-pp.vfs_write
7.11 ? 6% perf-profile.children.cycles-pp.btrfs_buffered_write
7.16 ? 6% perf-profile.children.cycles-pp.btrfs_file_write_iter
2.88 ? 10% perf-profile.children.cycles-pp.osq_lock
11.61 ? 4% perf-profile.children.cycles-pp.btrfs_search_slot
16.99 ? 3% perf-profile.children.cycles-pp.btrfs_finish_ordered_io
2.04 ? 14% perf-profile.children.cycles-pp.wb_workfn
2.04 ? 14% perf-profile.children.cycles-pp.wb_writeback
2.04 ? 14% perf-profile.children.cycles-pp.__writeback_inodes_wb
2.04 ? 14% perf-profile.children.cycles-pp.writeback_sb_inodes
8.25 ? 5% perf-profile.children.cycles-pp.btrfs_mark_extent_written
8.23 ? 6% perf-profile.children.cycles-pp.flush_space
7.17 ? 5% perf-profile.children.cycles-pp.writeback_single_inode
7.17 ? 5% perf-profile.children.cycles-pp.btrfs_start_delalloc_roots
7.17 ? 5% perf-profile.children.cycles-pp.start_delalloc_inodes
8.30 ? 6% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space
9.21 ? 4% perf-profile.children.cycles-pp.__writeback_single_inode
2.38 ? 5% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.81 ? 10% perf-profile.self.cycles-pp.check_setget_bounds
0.55 ? 8% perf-profile.self.cycles-pp.poll_idle
0.91 ? 11% perf-profile.self.cycles-pp.btrfs_get_32
0.69 ? 11% perf-profile.self.cycles-pp.timekeeping_max_deferment
0.29 ? 14% perf-profile.self.cycles-pp.copy_page
0.47 ? 13% perf-profile.self.cycles-pp.read_extent_buffer
0.17 ? 27% perf-profile.self.cycles-pp.check_extent_data_item
0.23 ? 14% perf-profile.self.cycles-pp.btrfs_get_8
0.20 ? 7% perf-profile.self.cycles-pp.queued_write_lock_slowpath
0.08 ? 16% perf-profile.self.cycles-pp.find_get_pages_range_tag
0.85 ? 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.23 ? 18% perf-profile.self.cycles-pp.check_leaf
0.22 ? 5% perf-profile.self.cycles-pp.copy_pages
0.13 ? 15% perf-profile.self.cycles-pp.crc_128
0.30 ? 3% perf-profile.self.cycles-pp.native_sched_clock
0.23 ? 3% perf-profile.self.cycles-pp.kmem_cache_free
0.11 ? 14% perf-profile.self.cycles-pp.io_serial_in
0.08 ? 23% perf-profile.self.cycles-pp.clear_page_dirty_for_io
0.11 ? 10% perf-profile.self.cycles-pp.release_extent_buffer
0.07 ? 11% perf-profile.self.cycles-pp.__push_leaf_right
0.05 ? 52% perf-profile.self.cycles-pp.memzero_extent_buffer
0.10 ? 10% perf-profile.self.cycles-pp._raw_write_lock
0.04 ? 50% perf-profile.self.cycles-pp.allocate_slab
0.09 ? 7% perf-profile.self.cycles-pp.___might_sleep
0.08 ? 9% perf-profile.self.cycles-pp.llist_add_batch
0.08 ? 16% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
0.14 ? 7% perf-profile.self.cycles-pp.verify_parent_transid
0.31 ? 8% perf-profile.self.cycles-pp.btrfs_root_node
0.09 ? 29% perf-profile.self.cycles-pp._find_next_bit
0.23 ? 5% perf-profile.self.cycles-pp.mark_page_accessed
0.33 ? 10% perf-profile.self.cycles-pp.btrfs_search_slot
0.21 ? 14% perf-profile.self.cycles-pp.xas_find_marked
0.29 ? 10% perf-profile.self.cycles-pp.down_read
0.34 ? 5% perf-profile.self.cycles-pp.rwsem_optimistic_spin
0.93 ? 6% perf-profile.self.cycles-pp.btrfs_comp_cpu_keys
0.48 ? 5% perf-profile.self.cycles-pp.__radix_tree_lookup
2.88 ? 5% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
2.84 ? 10% perf-profile.self.cycles-pp.osq_lock

fio.write_iops

-------------------------------------------------+
+... |
+..+... .. +..+..+... ..+..+..+...+..+...+..|
: + +..+. |
|
|
|
|
|
|
|
|
|
|
-------------------------------------------------+

fio.write_clat_stddev

-------------------------------------------------+
|
|
+... ..+..|
+.. .. +.. .+... .+..+...+..+..+..+. |
+..+ +. +. |
|
|
|
|
|
|
|
|
-------------------------------------------------+

fio.write_clat_90__us

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+..+..+...+..+...+..+..+...+..+..+...+..+..+...+..|
-------------------------------------------------+

fio.write_clat_95__us

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+..+..+...+..+...+..+..+...+..+..+...+..+..+...+..|
-------------------------------------------------+

fio.write_clat_99__us

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+..+..+...+..+...+..+..+...+..+..+...+..+..+...+..|
-------------------------------------------------+

fio.latency_20us_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
+.. .+..|
..+.. .. . .. |
+...+..+...+..+. +...+..+...+ +..+ |
-------------------------------------------------+

fio.latency_50us_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
.+...+.. ..+.. |
+...+...+. +...+..+. +...+.. ..|
-------------------------------------------------+

fio.latency_250us_

-------------------------------------------------+
.+..+.. |
+. + .. . ..+..+.. |
+ ..+...+..+ +..+. . .+...|
+. +. |
|
|
|
|
|
|
|
|
|
-------------------------------------------------+

fio.latency_500us_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+..+...+..+...+..+...+..+...+..+...+..+...+..+...+..|
-------------------------------------------------+

fio.latency_750us_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+..+...+..+...+..+...+..+...+..+...+..+...+..+...+..|
-------------------------------------------------+

fio.latency_1000us_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
+...+..+...+..+...+..+..+...+..+...+..+...+..+...+..|
-------------------------------------------------+

fio.latency_2ms_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
.+..+...+..+...+..+..+...+..+...+..+...+..+...+..|
|
-------------------------------------------------+

fio.latency_4ms_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
.+...+..+..+...+..+...+..+...+..+...+..|
-------------------------------------------------+

fio.latency_10ms_

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
-------------------------------------------------+

fio.time.voluntary_context_switches

-------------------------------------------------+
+... .+.. ..+.. ..+.. ..+.. .|
: +..+. +. +..+..+. +..+..+. +. |
|
|
|
|
|
|
|
|
|
|
|
-------------------------------------------------+

perf-sched.total_wait_and_delay.count.ms

-------------------------------------------------+
|
|
|
|
+ |
: : + |
: : + +.. |
: : + .. .+... |
+..+..+ + +...+..+..+...+..+..+. +..|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
+...+..+.. +... .+..+...+..+...+..+...+.. ..+..|
. .. +. +. |
+ |
|
|
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
.+...+.. ..+..+...+.. ..+..|
+. +... .+...+..+. +. +. |
+. |
|
|
|
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
|
|
|
|
..+..+... ..+.. |
+..+...+..+... ..+..+. +..+. .+...|
+. +...+. |
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
+.. .+ .+.. |
.. .. + .+... .. +...+..|
+.. . + + ..+. +..+ |
+ +. |
|
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
|
|
|
|
..+..+... ..+.. |
+..+...+..+... ..+..+. +..+. .+...|
+. +...+. |
|
|
|
-------------------------------------------------+

-------------------------------------------------+
+.. |
.. . +.. +.. .|
+..+...+ + : +...+..+. : ..+. |
: : .. : +. |
: : : |
: : +.. : |
+ + |
|
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
+.. |
.. . +.. +.. .|
+..+...+ + : +...+..+. : ..+. |
: : .. : +. |
: : : |
: : +.. : |
+ + |
|
|
|
|
|
|
-------------------------------------------------+

fio.latency_750ms_

-------------------------------------------------+
|
|
|
|
|
|
+..+..+..+...+..+..+...+..+..+...+..+..+..+...+..|
|
|
|
|
|
|
-------------------------------------------------+

fio.latency__=2000ms_

-------------------------------------------------+
|
|
|
|
|
|
+..+..+..+...+..+..+...+..+..+...+..+..+..+...+..|
|
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
+ |
:: + |
: : :: |
: : : : |
+. : : : : |
: .. : : : : |
: : : : : |
: : + : .+.. .+ : .|
+.. .. .. +. |
+...+ +...+..+ + |
-------------------------------------------------+

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
.+.. +.. |
.. .. . .|
+..+...+.. .+ +... ..+ +.. ..+. |
+...+. +..+. +. |
-------------------------------------------------+

-------------------------------------------------+
+ |
: |
: : |
: : |
: : |
: : |
: : |
: : |
: : ..+..|
+... : +..+. |
.. +..+...+..+ |
+..+...+.. .+..+ |
.. |
-------------------------------------------------+

-------------------------------------------------+
.. : |
. : |
+...+ + : + +...+ + |
+ + .. : .. + + : : + |
+ + . :. + + : : + |
+ + +...+ : : +.. |
: : .|
+ |
|
|
|
|
|
-------------------------------------------------+

-------------------------------------------------+
|
|
|
+ |
:: + |
: : :: |
: : : : |
+. : : : : |
: .. : : : : |
: : : : : |
: : + : .+.. .+ : .|
+.. .. .. +. |
+...+ +...+..+ + |
-------------------------------------------------+

-------------------------------------------------+
|
|
|
|
|
|
|
|
|
.+.. +.. |
.. .. . .|
+..+...+.. .+ +... ..+ +.. ..+. |
+...+. +..+. +. |
-------------------------------------------------+

estimated based on internal Intel analysis and are provided
purposes only. Any difference in system hardware or software
may affect actual performance.

Attachments:

(No filename) (131.53 kB)
config-5.10.0-rc7-00216-ge076ab2a2ca7 (173.70 kB)
job-script (8.15 kB)
job.yaml (5.69 kB)
reproduce (772.00 B)
Download all attachments

2021-01-12 15:51:02

by David Sterba

[permalink] [raw]

Subject: Re: [btrfs] e076ab2a2c: fio.write_iops -18.3% regression

On Tue, Jan 12, 2021 at 11:36:14PM +0800, kernel test robot wrote:
>
> Greeting,
>
> FYI, we noticed a -18.3% regression of fio.write_iops due to commit:
>
>
> commit: e076ab2a2ca70a0270232067cd49f76cd92efe64 ("btrfs: shrink delalloc pages instead of full inodes")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
>
> in testcase: fio-basic
> on test machine: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
> with following parameters:
>
> disk: 1SSD
> fs: btrfs
> runtime: 300s
> nr_task: 8
> rw: randwrite
> bs: 4k
> ioengine: sync
> test_size: 256g

Though I do a similar test (emulating bit torrent workload), it's a bit
extreme as it's 4k synchronous on a huge file. It always takes a lot of
time but could point out some concurrency issues namely on faster
devices. There are 8 threads possibly competing for the same inode lock
or other locks related to it.

The mentioned commit fixed another perf regression on a much more common
workload (untgrring files), so at this point drop in this fio workload
is inevitable.

2021-01-13 06:01:13

by Xing Zhengjun

[permalink] [raw]

Subject: Re: [LKP] Re: [btrfs] e076ab2a2c: fio.write_iops -18.3% regression

On 1/12/2021 11:45 PM, David Sterba wrote:
> On Tue, Jan 12, 2021 at 11:36:14PM +0800, kernel test robot wrote:
>> Greeting,
>>
>> FYI, we noticed a -18.3% regression of fio.write_iops due to commit:
>>
>>
>> commit: e076ab2a2ca70a0270232067cd49f76cd92efe64 ("btrfs: shrink delalloc pages instead of full inodes")
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>
>>
>> in testcase: fio-basic
>> on test machine: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
>> with following parameters:
>>
>> disk: 1SSD
>> fs: btrfs
>> runtime: 300s
>> nr_task: 8
>> rw: randwrite
>> bs: 4k
>> ioengine: sync
>> test_size: 256g
> Though I do a similar test (emulating bit torrent workload), it's a bit
> extreme as it's 4k synchronous on a huge file. It always takes a lot of
> time but could point out some concurrency issues namely on faster
> devices. There are 8 threads possibly competing for the same inode lock
> or other locks related to it.
>
> The mentioned commit fixed another perf regression on a much more common
> workload (untgrring files), so at this point drop in this fio workload
> is inevitable.

Do you have a plan to fix it? Thanks.
> _______________________________________________
> LKP mailing list -- [email protected]
> To unsubscribe send an email to [email protected]

--
Zhengjun Xing

2021-01-14 22:56:49

by David Sterba

[permalink] [raw]

Subject: Re: [LKP] Re: [btrfs] e076ab2a2c: fio.write_iops -18.3% regression

On Wed, Jan 13, 2021 at 01:58:18PM +0800, Xing Zhengjun wrote:
>
>
> On 1/12/2021 11:45 PM, David Sterba wrote:
> > On Tue, Jan 12, 2021 at 11:36:14PM +0800, kernel test robot wrote:
> >> Greeting,
> >>
> >> FYI, we noticed a -18.3% regression of fio.write_iops due to commit:
> >>
> >>
> >> commit: e076ab2a2ca70a0270232067cd49f76cd92efe64 ("btrfs: shrink delalloc pages instead of full inodes")
> >> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >>
> >>
> >> in testcase: fio-basic
> >> on test machine: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
> >> with following parameters:
> >>
> >> disk: 1SSD
> >> fs: btrfs
> >> runtime: 300s
> >> nr_task: 8
> >> rw: randwrite
> >> bs: 4k
> >> ioengine: sync
> >> test_size: 256g
> > Though I do a similar test (emulating bit torrent workload), it's a bit
> > extreme as it's 4k synchronous on a huge file. It always takes a lot of
> > time but could point out some concurrency issues namely on faster
> > devices. There are 8 threads possibly competing for the same inode lock
> > or other locks related to it.
> >
> > The mentioned commit fixed another perf regression on a much more common
> > workload (untgrring files), so at this point drop in this fio workload
> > is inevitable.
>
> Do you have a plan to fix it? Thanks.

My plan is to find somebody who will fix it. Thanks.