With and without kmap_atomic() -based copy_*_user() patches from akpm.
Taken on a 16x/16GB box.
Take 1: total throughput:
before:
Throughput 32.3019 MB/sec (NB=40.3774 MB/sec 323.019 MBit/sec) 512 procs
after:
Throughput 46.9837 MB/sec (NB=58.7296 MB/sec 469.837 MBit/sec) 512 procs
In the follow-ups I'll include oprofile numbers for each and vmstat logs.
Cheers,
Bill
On Thu, Aug 15, 2002 at 04:21:26PM -0700, William Lee Irwin III wrote:
> In the follow-ups I'll include oprofile numbers for each and vmstat logs.
vmstat log before:
Script started on Thu Aug 15 18:06:12 2002
$ vmstat 10
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 16030500 0 17152 0 0 6 1 64 3 0 0 99
0 0 0 0 15958620 0 18220 0 0 106 0 1011 60 0 1 99
0 512 1 0 15653616 0 262312 0 0 48 158 1060 786 0 43 57
129 383 1 0 15089556 0 808580 0 0 8 44 1084 410 0 93 6
149 363 0 0 14426392 0 1451152 0 0 27 2 1194 679 0 98 2
401 111 1 0 13826876 0 2027240 0 0 26 357 1086 690 0 99 0
493 19 1 0 13209856 0 2631808 0 0 13 871 1132 789 0 100 0
411 101 1 0 12637416 0 3192400 0 0 32 0 1008 1121 0 100 0
421 91 1 0 11984456 0 3831656 0 0 19 0 1173 1202 0 100 0
0 512 1 0 11692000 0 4114760 0 0 29 1664 1059 618 0 52 48
507 5 1 0 11133820 0 4660992 0 0 19 2908 1165 646 0 96 3
460 52 1 0 10588860 0 5195184 0 0 54 2638 1110 1017 0 100 0
511 1 1 0 9983548 0 5787468 0 0 12 3038 1254 1203 0 100 0
456 56 1 0 9454660 0 6305104 0 0 46 2752 1115 1041 0 100 0
512 0 1 0 8850200 0 6896792 0 0 10 3439 1330 1002 0 100 0
475 37 1 0 8340844 0 7395540 0 0 30 2836 1115 914 0 100 0
501 11 1 0 7884784 0 7841860 0 0 44 2962 1119 663 1 99 0
511 1 1 0 7467560 0 8250548 0 0 12 2878 1110 448 1 99 0
504 8 1 0 7101380 0 8609880 0 0 51 2746 1122 333 1 99 0
49 463 1 0 6775144 0 8927748 0 0 17 2854 1119 293 1 99 0
1 511 1 0 6743308 0 8957548 0 0 1 2863 1106 26 0 9 91
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 512 1 0 6734056 0 8966760 0 0 0 2956 1104 9 0 2 98
1 511 1 0 6726252 0 8974336 0 0 0 2942 1103 8 0 2 98
0 512 1 0 6718468 0 8981972 0 0 0 2906 1101 8 0 2 98
0 512 1 0 6709704 0 8990412 0 0 0 2988 1105 9 0 2 98
0 512 1 0 6701000 0 8999160 0 0 0 2834 1101 8 0 2 98
1 511 1 0 6692044 0 9007868 0 0 0 2845 1101 8 0 1 98
0 512 1 0 6681136 0 9018320 0 0 0 2917 1101 8 0 2 98
0 512 1 0 6672436 0 9027192 0 0 0 2862 1102 8 0 2 98
1 511 1 0 6663200 0 9035896 0 0 0 2930 1104 9 0 2 98
0 512 1 0 6654712 0 9044048 0 0 0 2904 1102 9 0 1 99
1 511 1 0 6625628 0 9072692 0 0 21 2835 1103 9 0 3 96
1 511 1 0 6595384 0 9102340 0 0 0 2791 1101 10 0 4 96
1 511 1 0 6566500 0 9130468 0 0 1 2870 1103 8 0 3 97
1 511 1 0 6535240 0 9161088 0 0 1 2891 1104 7 0 2 97
1 511 1 0 6506408 0 9189280 0 0 1 2831 1102 8 0 3 97
2 510 1 0 6474892 0 9219692 0 0 1 2954 1104 10 0 4 96
1 511 1 0 6446944 0 9247368 0 0 2 2793 1101 9 0 2 98
0 512 1 0 6417712 0 9276400 0 0 3 2808 1104 15 0 3 97
0 512 1 0 6386148 0 9306724 0 0 3 2932 1103 13 0 3 97
0 512 1 0 6355680 0 9336752 0 0 2 2917 1106 10 0 4 96
3 509 1 0 6328132 0 9363816 0 0 4 2737 1103 17 0 3 97
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 513 1 0 6296992 0 9393708 0 0 3 2887 1104 14 0 4 96
0 512 1 0 6270080 0 9420724 0 0 2 2767 1106 11 0 3 97
0 512 1 0 6239632 0 9449904 0 0 3 2905 1103 17 0 4 96
0 512 1 0 6211120 0 9477884 0 0 3 2786 1101 10 0 3 97
0 512 1 0 6182544 0 9506248 0 0 4 2821 1103 11 0 3 97
0 512 1 0 6153224 0 9534520 0 0 2 2890 1102 11 0 3 97
0 512 1 0 6123908 0 9563556 0 0 39 2789 1106 18 0 5 95
0 512 1 0 6095808 0 9591008 0 0 4 2829 1104 12 0 3 97
0 512 1 0 6065320 0 9620528 0 0 2 2877 1103 11 0 3 97
0 512 1 0 6036204 0 9649112 0 0 2 2813 1105 18 0 4 95
0 512 1 0 6008404 0 9676656 0 0 1 2832 1105 11 0 4 96
0 512 1 0 5979352 0 9705188 0 0 0 2834 1102 11 0 3 97
0 512 1 0 5950212 0 9733380 0 0 0 2839 1103 12 0 3 96
0 512 1 0 5920924 0 9761812 0 0 1 2861 1106 11 0 3 96
2 510 1 0 5892504 0 9789956 0 0 2 2773 1105 11 0 3 97
1 511 1 0 5862944 0 9818284 0 0 0 2831 1105 14 0 4 96
0 512 1 0 5833700 0 9846684 0 0 2 2799 1107 21 0 4 95
1 511 1 0 5806112 0 9874076 0 0 0 2743 1105 17 0 4 96
0 512 1 0 5777160 0 9902324 0 0 2 2774 1106 31 1 6 94
0 512 1 0 5749864 0 9929472 0 0 0 2732 1102 11 0 3 97
1 511 1 0 5720448 0 9958276 0 0 1 2817 1106 15 0 4 96
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 511 1 0 5690996 0 9987028 0 0 0 2880 1107 16 0 4 95
0 512 1 0 5661268 0 10015812 0 0 2 2858 1104 12 0 4 96
0 512 1 0 5632096 0 10044776 0 0 2 2748 1103 11 0 3 97
7 505 1 0 5607828 0 10068312 0 0 21 2858 1129 30 1 6 93
0 512 1 0 5575800 0 10099712 0 0 34 2713 1108 33 0 13 87
6 506 1 0 5545128 0 10128844 0 0 2 2984 1114 35 0 5 94
0 512 1 0 5514480 0 10159328 0 0 112 2824 1115 46 1 9 91
0 512 1 0 5485404 0 10187644 0 0 3 2894 1111 37 1 6 94
2 510 1 0 5457668 0 10214952 0 0 3 2747 1114 35 1 7 93
0 512 1 0 5426568 0 10244836 0 0 2 2930 1113 34 1 6 94
0 512 1 0 5397912 0 10272996 0 0 2 2798 1106 19 0 4 96
0 512 1 0 5370080 0 10300712 0 0 4 2766 1109 58 1 6 93
0 512 1 0 5341324 0 10328744 0 0 1 2763 1107 27 0 5 95
1 511 1 0 5313236 0 10356208 0 0 2 2810 1108 15 0 4 96
0 512 1 0 5283084 0 10385644 0 0 2 2814 1107 31 0 5 95
0 512 1 0 5254348 0 10413472 0 0 2 2876 1107 19 0 4 96
0 512 1 0 5226876 0 10440616 0 0 5 2738 1113 88 1 24 76
1 511 1 0 5197164 0 10469304 0 0 2 2863 1113 40 1 6 93
0 512 1 0 5169448 0 10496036 0 0 1 2792 1104 24 0 5 94
0 512 1 0 5141960 0 10523728 0 0 26 2728 1107 27 0 4 96
1 511 1 0 5112580 0 10551664 0 0 0 2825 1112 67 1 21 78
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 512 1 0 5084232 0 10579604 0 0 0 2718 1110 37 1 9 91
0 512 1 0 5057348 0 10606140 0 0 1 2722 1113 63 1 29 70
1 511 1 0 5031844 0 10631100 0 0 1 2781 1110 40 1 6 93
0 512 1 0 5000676 0 10661360 0 0 1 2666 1111 63 1 24 75
1 511 1 0 4973048 0 10688612 0 0 0 2761 1106 13 0 3 96
1 511 1 0 4945132 0 10715932 0 0 1 2729 1108 36 1 8 92
1 511 1 0 4913792 0 10746284 0 0 3 2759 1113 70 1 25 74
17 495 1 0 4893276 0 10766836 0 0 0 2751 1130 33 1 10 89
0 512 1 0 4859448 0 10799772 0 0 21 2715 1111 51 1 28 71
1 511 1 0 4830860 0 10827544 0 0 31 2740 1112 61 1 15 84
0 512 1 0 4803540 0 10854056 0 0 2 2730 1112 55 1 23 76
46 466 1 0 4783896 0 10873464 0 0 3 2530 1126 186 2 52 46
0 512 1 0 4727848 0 10927944 0 0 2 2751 1113 87 1 40 59
1 511 1 0 4699816 0 10955220 0 0 2 2783 1118 67 1 10 89
2 510 1 0 4671984 0 10982724 0 0 1 2745 1109 19 0 4 96
1 511 1 0 4647732 0 11006304 0 0 2 2754 1113 43 1 8 91
0 512 1 0 4615392 0 11037568 0 0 1 2788 1109 36 1 8 91
0 512 1 0 4588104 0 11064532 0 0 2 2715 1108 36 1 6 94
5 507 1 0 4558404 0 11093124 0 0 4 2792 1119 100 1 20 79
0 512 1 0 4532236 0 11119008 0 0 2 2700 1112 35 1 6 93
1 511 1 0 4503824 0 11146548 0 0 0 2821 1109 26 1 5 94
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 512 1 0 4464900 0 11185036 0 0 4 2688 1121 145 1 55 43
15 497 1 0 4452708 0 11196928 0 0 1 2780 1134 77 1 18 82
21 491 1 0 4415616 0 11232624 0 0 4 2718 1123 170 2 77 21
1 511 1 0 4387976 0 11259524 0 0 1 2716 1107 49 1 13 87
0 512 1 0 4357528 0 11289292 0 0 2 2632 1116 157 2 50 48
18 494 1 0 4336624 0 11310296 0 0 2 2617 1116 137 1 20 79
1 511 1 0 4306500 0 11339308 0 0 1 2760 1110 66 1 22 77
51 461 1 0 4290888 0 11354840 0 0 14 2668 1119 115 1 40 59
30 482 1 0 4263016 0 11381860 0 0 2 2684 1116 108 1 49 50
146 366 2 0 4229972 0 11414624 0 0 2 2630 1121 185 2 98 0
0 512 1 0 4189560 0 11453328 0 0 3 2691 1117 157 1 64 35
0 512 1 0 4167640 0 11474876 0 0 2 2700 1113 83 1 16 83
10 502 1 0 4145668 0 11496432 0 0 0 2606 1118 105 2 56 42
74 438 1 0 4116664 0 11524812 0 0 10 2618 1122 161 2 96 2
187 325 1 0 4118092 0 11523044 0 0 1 2624 1123 211 2 98 0
304 208 2 0 4063200 0 11577608 0 0 2 2656 1147 194 2 98 0
37 475 1 0 4044028 0 11595504 0 0 2 2662 1130 185 2 70 28
157 355 1 0 4037396 0 11601984 0 0 2 2572 1119 179 2 98 0
263 249 2 0 4008540 0 11630764 0 0 2 2554 1121 195 2 98 0
375 137 1 0 3996592 0 11642044 0 0 1 2672 1120 187 2 98 0
484 28 2 0 3954796 0 11683672 0 0 2 2620 1121 179 2 98 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
512 0 1 0 3962612 0 11675468 0 0 2 1278 1144 203 2 98 0
512 0 2 0 3931376 0 11706564 0 0 0 1572 1143 183 2 98 0
511 0 2 0 3924356 0 11715812 0 0 3 1295 1144 218 2 98 0
511 0 1 0 3923176 0 11714456 0 0 2 1528 1144 199 2 98 0
510 1 2 0 3926740 0 11712952 0 0 2 2152 1127 208 2 98 0
511 0 2 0 3919160 0 11720444 0 0 0 1216 1143 204 2 98 0
509 2 2 0 3952668 0 11686496 0 0 0 1290 1151 206 2 98 0
511 0 2 0 3944192 0 11694232 0 0 0 1400 1142 193 2 98 0
511 0 1 0 3979192 0 11660256 0 0 0 1696 1135 191 2 98 0
509 0 2 0 3997656 0 11642040 0 0 1 775 1153 190 2 98 0
509 0 1 0 3999412 0 11640236 0 0 0 1667 1138 189 2 98 0
509 0 1 0 3988136 0 11648228 0 0 0 955 1157 195 2 98 0
508 1 1 0 4006224 0 11633260 0 0 1 1168 1140 226 2 98 0
508 0 1 0 4036952 0 11599396 0 0 0 1557 1146 214 2 98 0
507 0 2 0 4051844 0 11584960 0 0 0 1533 1144 206 2 98 0
503 0 2 0 4112112 0 11528108 0 0 0 2127 1124 181 2 98 0
501 2 1 0 4093032 0 11547088 0 0 0 1604 1144 203 2 98 0
503 0 1 0 4100296 0 11539728 0 0 1 1536 1141 192 2 98 0
503 0 1 0 4153784 0 11486160 0 0 0 1292 1143 228 2 98 0
500 1 1 0 4202116 0 11438076 0 0 0 722 1156 219 2 98 0
499 0 1 0 4232768 0 11407708 0 0 0 1074 1147 190 2 98 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
498 0 1 0 4245960 0 11394600 0 0 0 1031 1153 197 2 98 0
496 0 1 0 4317348 0 11323444 0 0 0 1301 1147 213 2 98 0
495 0 3 0 4349556 0 11289944 0 0 0 2063 1139 183 2 98 0
494 0 1 0 4374812 0 11263824 0 0 0 2022 1136 204 2 98 0
484 6 1 0 4455692 0 11184128 0 0 0 1586 1140 194 2 98 0
488 0 1 0 4519336 0 11119252 0 0 0 2726 1127 173 2 98 0
458 25 3 0 4604064 0 11036628 0 0 0 1864 1140 235 2 98 0
479 0 1 0 4727324 0 10914252 0 0 0 2126 1131 178 2 98 0
475 0 2 0 4785124 0 10855204 0 0 0 2500 1129 178 2 98 0
469 1 1 0 4870052 0 10771256 0 0 0 2451 1122 178 2 98 0
468 0 1 0 4894780 0 10746320 0 0 0 2528 1125 167 2 98 0
461 0 1 0 5098844 0 10543828 0 0 0 2067 1125 184 2 98 0
455 0 1 0 5200836 0 10442688 0 0 0 2351 1131 172 2 98 0
447 0 1 0 5355456 0 10289656 0 0 0 2373 1127 184 2 98 0
444 0 1 0 5522224 0 10122984 0 0 0 2478 1125 174 2 98 0
431 0 2 0 5718152 0 9929460 0 0 0 2605 1122 178 2 98 0
425 0 1 0 5927768 0 9720220 0 0 0 2769 1123 174 2 98 0
408 0 1 0 6235760 0 9414664 0 0 0 2754 1127 174 2 98 0
401 0 1 0 6417560 0 9234080 0 0 0 2753 1120 162 2 98 0
381 8 1 0 6679396 0 8974080 0 0 0 2613 1122 180 2 98 0
374 1 1 0 7014740 0 8640536 0 0 0 2711 1121 182 2 98 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
366 0 2 0 7226640 0 8429968 0 0 0 2670 1125 182 2 98 0
352 0 2 0 7540784 0 8118040 0 0 0 2644 1123 177 2 98 0
348 0 1 0 7635932 0 8023952 0 0 0 2611 1122 172 2 98 0
343 0 1 0 7744096 0 7916516 0 0 0 2706 1124 158 2 98 0
339 0 1 0 7799716 0 7861344 0 0 0 2628 1117 158 2 98 0
335 0 1 0 7896960 0 7764748 0 0 0 2652 1119 159 2 98 0
331 0 1 0 8022476 0 7639148 0 0 0 2694 1123 167 2 98 0
325 0 1 0 8140000 0 7522724 0 0 0 2628 1119 169 2 98 0
316 0 2 0 8306312 0 7357656 0 0 0 2605 1121 162 2 98 0
308 0 2 0 8510444 0 7154704 0 0 0 2617 1121 196 2 98 0
301 1 1 0 8651348 0 7014896 0 0 0 2601 1117 158 2 98 0
293 0 1 0 8832568 0 6835252 0 0 0 2580 1119 157 2 98 0
286 0 1 0 9002060 0 6666908 0 0 0 2717 1123 165 2 98 0
278 0 1 0 9158980 0 6511276 0 0 0 2668 1118 150 2 98 0
273 0 1 0 9280348 0 6390320 0 0 0 2685 1121 155 2 98 0
256 0 2 0 9620268 0 6053080 0 0 0 2656 1121 170 2 98 0
253 0 2 0 9684960 0 5988192 0 0 0 2698 1119 151 2 98 0
249 0 1 0 9764732 0 5909552 0 0 0 2599 1119 154 2 98 0
243 0 1 0 9900232 0 5774964 0 0 0 2648 1119 150 2 98 0
236 0 1 0 10040144 0 5635812 0 0 0 2643 1117 152 2 98 0
227 0 2 0 10264664 0 5413280 0 0 0 2614 1123 168 2 98 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
219 1 2 0 10438612 0 5239980 0 0 0 2752 1124 155 2 98 0
213 0 1 0 10621820 0 5057548 0 0 0 2649 1118 150 2 98 0
203 0 1 0 10914592 0 4766988 0 0 0 2594 1120 165 2 98 0
189 0 1 0 11160692 0 4522212 0 0 0 2727 1119 155 2 98 0
185 0 1 0 11303236 0 4380332 0 0 0 2677 1120 159 2 98 0
176 0 1 0 11492392 0 4192492 0 0 0 2744 1119 159 2 98 0
169 0 1 0 11630124 0 4055636 0 0 0 2772 1121 148 2 98 0
156 0 1 0 11889592 0 3798296 0 0 0 2682 1120 153 2 98 0
141 0 2 0 12219120 0 3470760 0 0 0 2584 1116 157 2 98 0
132 0 2 0 12410372 0 3280848 0 0 0 2520 1113 149 2 98 0
121 0 1 0 12632632 0 3060512 0 0 0 2676 1120 148 2 98 0
115 0 1 0 12802380 0 2891428 0 0 0 2601 1116 148 2 98 0
101 0 1 0 13116068 0 2580380 0 0 0 2716 1122 165 2 98 0
83 2 1 0 13489008 0 2209960 0 0 0 2719 1119 155 2 98 0
80 0 1 0 13639208 0 2060280 0 0 0 2803 1119 157 2 98 0
60 0 1 0 14028280 0 1674380 0 0 0 2750 1117 152 2 98 0
47 0 1 0 14323412 0 1380688 0 0 0 2802 1119 142 2 98 0
33 0 1 0 14667540 0 1039752 0 0 0 2701 1121 154 2 98 0
13 1 1 0 15073012 0 636504 0 0 0 2751 1118 132 2 97 1
0 1 2 0 15368508 0 342820 0 0 0 2741 1100 103 1 21 78
0 0 1 0 15451236 0 332436 0 0 0 744 1134 19 0 2 97
Script done on Thu Aug 15 18:41:37 2002
On Thu, Aug 15, 2002 at 04:21:26PM -0700, William Lee Irwin III wrote:
> In the follow-ups I'll include oprofile numbers for each and vmstat logs.
oprofile numbers before:
c013bf64 7205865 63.7904 .text.lock.highmem
c0105394 1392984 12.3315 default_idle
c013b5cc 1013797 8.97471 kmap_high
c013b7c0 864121 7.64969 kunmap_high
c0114820 144253 1.27701 scheduler_tick
c013bcac 103670 0.917746 blk_queue_bounce
c012f250 73437 0.650106 generic_file_write
c01143d8 50347 0.4457 load_balance
c012deb0 39488 0.34957 unlock_page
c013563c 35625 0.315373 rmqueue
c01432ac 32983 0.291984 __block_prepare_write
c013b548 28023 0.248076 flush_all_zero_pkmaps
c0141bb8 17749 0.157124 __find_get_block_slow
c012dca4 15073 0.133435 add_to_page_cache
c014e7ab 8010 0.0709091 .text.lock.namei
c013428c 7957 0.0704399 lru_cache_add
c016db38 7334 0.0649248 ext2_free_branches
c016d5b0 7030 0.0622336 ext2_get_block
c012e100 6578 0.0582322 find_lock_page
c01435e8 6559 0.058064 __block_commit_write
c012e52c 6368 0.0563732 file_read_actor
c013c34c 5913 0.0523453 mempool_alloc
On Thu, Aug 15, 2002 at 04:21:26PM -0700, William Lee Irwin III wrote:
> In the follow-ups I'll include oprofile numbers for each and vmstat logs.
vmstat log from after:
Script started on Thu Aug 15 18:59:14 2002
$ vmstat 10
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 16002532 0 42580 0 0 14 1 65 6 0 1 99
0 0 0 0 16002524 0 42588 0 0 1 0 1006 12 0 0 100
1 1 0 0 15930364 0 43740 0 0 116 17 1055 66 0 2 98
43 469 0 0 15685644 0 234880 0 0 100 3 1053 1276 0 15 85
23 489 0 0 14662044 0 1231044 0 0 87 0 1066 4888 1 73 26
20 492 1 0 13579932 0 2285476 0 0 55 175 1071 5795 1 81 18
26 486 1 0 12964732 0 2885648 0 0 26 586 1179 3185 1 46 53
26 486 1 0 11873320 0 3950852 0 0 90 537 1198 6224 2 86 12
17 495 1 0 10867228 0 4933188 0 0 63 532 1205 6209 3 86 12
21 491 1 0 10177340 0 5603044 0 0 42 1738 1191 4521 2 64 34
21 491 1 0 9294060 0 6464972 0 0 54 2562 1154 6687 3 84 12
21 491 1 0 8515076 0 7224868 0 0 56 2646 1172 7587 4 81 16
23 489 1 0 7827388 0 7895268 0 0 20 2805 1152 7307 4 81 16
28 484 1 0 7144504 0 8562912 0 0 49 2669 1172 7645 4 80 16
1 511 1 0 6947076 0 8753936 0 0 20 2867 1127 3413 2 31 67
1 511 1 0 6941276 0 8759728 0 0 0 2764 1103 8 0 2 98
0 512 1 0 6935196 0 8765788 0 0 0 2722 1102 9 0 2 98
1 511 1 0 6932872 0 8768024 0 0 0 2796 1102 9 0 2 98
1 511 1 0 6925744 0 8774612 0 0 0 2820 1107 10 0 2 98
0 512 1 0 6919924 0 8780860 0 0 0 2736 1103 10 0 2 98
0 512 1 0 6911104 0 8789408 0 0 0 2794 1106 11 0 2 98
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 511 1 0 6882308 0 8817992 0 0 81 2778 1114 19 0 4 96
0 512 1 0 6852444 0 8846940 0 0 1 2782 1111 33 1 4 95
0 512 1 0 6822936 0 8875476 0 0 1 2854 1112 27 1 5 95
0 512 1 0 6794624 0 8903744 0 0 1 2711 1111 23 0 4 96
1 511 1 0 6766444 0 8931032 0 0 1 2807 1112 19 0 4 96
2 510 1 0 6737392 0 8959148 0 0 2 2815 1131 211 1 7 92
0 512 1 0 6710268 0 8986204 0 0 1 2739 1114 16 0 3 97
2 510 1 0 6681840 0 9013796 0 0 1 2831 1113 17 0 3 96
0 512 1 0 6651576 0 9043280 0 0 1 2774 1113 37 0 4 95
0 512 1 0 6624220 0 9070548 0 0 2 2698 1114 31 1 5 95
1 511 1 0 6595480 0 9097904 0 0 0 2755 1110 12 0 3 97
0 512 1 0 6568812 0 9124480 0 0 2 2651 1108 10 0 3 97
2 510 1 0 6543284 0 9149532 0 0 1 2669 1110 26 0 3 97
4 508 1 0 6517608 0 9174580 0 0 2 2758 1119 24 0 4 96
0 512 1 0 6482120 0 9208992 0 0 35 2822 1113 25 1 4 95
1 511 1 0 6455904 0 9234968 0 0 2 2778 1114 21 0 4 96
0 512 1 0 6427148 0 9262844 0 0 2 2766 1113 73 0 4 95
0 512 1 0 6398152 0 9291056 0 0 3 2812 1113 25 0 4 96
0 512 1 0 6369636 0 9318848 0 0 2 2684 1110 30 0 4 96
4 508 1 0 6340556 0 9346788 0 0 2 2798 1113 78 0 4 95
0 512 1 0 6314476 0 9372824 0 0 3 2639 1112 60 1 5 94
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 510 1 0 6292692 0 9394500 0 0 2 2614 1114 25 0 4 96
1 511 1 0 6256836 0 9428792 0 0 33 2877 1112 27 0 4 96
5 507 1 0 6230128 0 9455580 0 0 3 2665 1122 78 1 5 95
0 512 1 0 6202904 0 9482552 0 0 1 2707 1111 11 0 3 97
1 511 1 0 6174600 0 9509992 0 0 2 2781 1111 15 0 3 96
1 511 1 0 6144816 0 9538540 0 0 34 2706 1113 19 0 4 96
0 512 1 0 6122140 0 9562136 0 0 0 2573 1109 10 0 2 97
0 512 1 0 6089840 0 9592612 0 0 2 2753 1121 404 1 10 89
1 511 1 0 6065520 0 9616676 0 0 2 2641 1116 34 1 5 95
0 512 1 0 6036664 0 9644620 0 0 2 2764 1120 105 1 6 93
1 511 1 0 6009552 0 9671124 0 0 1 2674 1107 16 0 3 96
0 512 1 0 5981040 0 9698748 0 0 1 2725 1114 38 1 6 94
0 512 1 0 5950392 0 9729124 0 0 2 2671 1113 34 1 5 95
1 511 1 0 5929216 0 9749660 0 0 1 2711 1117 31 1 5 94
3 509 1 0 5898468 0 9779472 0 0 36 2843 1127 341 2 10 89
0 512 1 0 5867756 0 9809476 0 0 1 2693 1111 20 0 4 95
1 511 1 0 5841120 0 9835296 0 0 1 2712 1120 89 1 6 93
0 512 1 0 5813404 0 9862876 0 0 2 2648 1112 28 0 4 95
0 512 1 0 5785760 0 9889380 0 0 1 2761 1111 16 0 4 96
1 511 1 0 5759816 0 9915052 0 0 2 2577 1119 68 1 5 94
2 510 1 0 5733596 0 9940948 0 0 1 2604 1117 87 1 7 92
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 512 1 0 5703988 0 9969072 0 0 2 2811 1119 50 1 5 94
1 511 1 0 5677608 0 9995452 0 0 0 2611 1113 38 0 4 95
0 512 1 0 5648928 0 10023256 0 0 1 2700 1110 14 0 3 96
0 512 1 0 5622476 0 10049120 0 0 2 2647 1114 105 1 6 93
0 512 1 0 5595304 0 10075880 0 0 2 2640 1114 97 1 5 94
11 501 1 0 5566652 0 10104140 0 0 29 2677 1131 157 1 8 91
0 512 1 0 5539044 0 10130500 0 0 44 2735 1115 64 1 6 93
1 511 1 0 5509824 0 10159292 0 0 1 2794 1112 18 1 4 95
1 511 1 0 5479264 0 10188748 0 0 4 2924 1115 92 1 6 93
0 512 1 0 5451112 0 10216728 0 0 1 2833 1109 20 0 4 96
3 509 1 0 5422476 0 10244456 0 0 4 2769 1116 104 1 7 92
0 512 1 0 5395264 0 10271476 0 0 3 2740 1115 88 1 6 93
0 512 1 0 5365012 0 10300472 0 0 3 2918 1114 64 1 6 93
0 512 1 0 5335636 0 10329120 0 0 3 2842 1113 68 1 6 93
0 512 1 0 5307068 0 10357588 0 0 1 2784 1112 80 1 6 93
1 511 1 0 5277820 0 10385904 0 0 2 2860 1118 391 1 9 89
1 511 1 0 5250884 0 10412244 0 0 2 2848 1111 34 1 5 94
1 510 1 0 5232856 0 10428864 0 0 3 2885 1158 416 2 12 86
0 511 1 0 5204232 0 10456380 0 0 2 2983 1132 93 1 5 95
1 510 1 0 5175584 0 10484564 0 0 1 2873 1113 54 0 4 96
0 511 1 0 5146928 0 10513124 0 0 1 2802 1116 196 1 6 93
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 511 1 0 5119044 0 10540228 0 0 1 2939 1128 165 1 5 94
2 509 1 0 5090140 0 10568236 0 0 0 2900 1144 225 1 6 93
1 510 1 0 5060380 0 10597712 0 0 1 2854 1123 183 1 5 95
0 511 1 0 5034576 0 10623048 0 0 4 2884 1131 233 1 6 93
0 511 1 0 5005072 0 10652112 0 0 0 2882 1119 226 1 5 94
1 510 1 0 4976928 0 10679624 0 0 2 2909 1124 440 1 9 90
3 508 1 0 4944776 0 10710280 0 0 2 3004 1128 478 1 9 90
1 511 1 0 4919448 0 10735704 0 0 2 2809 1124 969 1 13 86
1 510 1 0 4887604 0 10766520 0 0 2 3048 1131 285 1 7 92
2 509 1 0 4859396 0 10793948 0 0 2 2884 1124 280 1 6 93
12 499 1 0 4829296 0 10823572 0 0 0 2933 1131 287 1 6 93
0 511 1 0 4797996 0 10854836 0 0 2 2755 1122 356 1 6 93
3 508 1 0 4776392 0 10875784 0 0 5 2811 1129 769 1 11 87
0 511 1 0 4743864 0 10907520 0 0 0 2953 1133 1265 2 16 82
0 510 1 0 4721440 0 10929336 0 0 2 2931 1127 502 1 13 85
1 509 1 0 4694216 0 10955792 0 0 2 2870 1118 276 1 7 92
0 510 1 0 4666732 0 10982952 0 0 1 2876 1119 328 1 7 92
1 509 1 0 4638948 0 11009820 0 0 2 2957 1117 286 1 6 94
0 510 1 0 4610756 0 11037696 0 0 2 2852 1119 307 1 7 92
2 508 1 0 4583104 0 11064692 0 0 0 2906 1118 339 1 7 93
11 499 1 0 4560252 0 11087580 0 0 1 2845 1162 843 1 10 88
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
6 504 1 0 4521020 0 11125024 0 0 165 2869 1165 1538 2 14 84
8 500 1 0 4531608 0 11114800 0 0 597 2870 1219 2330 2 21 77
0 508 1 0 4487936 0 11157312 0 0 365 2976 1169 1824 2 18 80
0 506 1 0 4496284 0 11149280 0 0 3 2810 1146 2037 3 20 77
3 503 1 0 4468680 0 11176552 0 0 0 2907 1130 652 1 6 93
0 506 1 0 4438204 0 11206216 0 0 2 2952 1133 1979 2 14 84
0 506 1 0 4409364 0 11234576 0 0 0 2903 1122 370 1 8 91
29 474 1 0 4441240 0 11202836 0 0 3 3190 1291 2299 3 37 60
28 471 1 0 4477508 0 11167108 0 0 2 2930 1195 10508 7 79 14
19 477 1 0 4503452 0 11141696 0 0 4 2867 1209 9993 7 75 18
23 471 2 0 4483828 0 11161828 0 0 3 2976 1253 8403 6 76 17
18 475 1 0 4465124 0 11183244 0 0 4 2478 1193 9252 7 72 22
26 465 2 0 4460604 0 11187652 0 0 2 1678 1240 8358 7 73 20
16 473 1 0 4481356 0 11163940 0 0 1 1336 1238 9286 7 72 22
32 456 1 0 4472860 0 11171464 0 0 2 2210 1252 9309 7 71 22
22 466 1 0 4615392 0 11031960 0 0 0 1413 1212 10223 7 67 26
21 462 6 0 4774272 0 10871584 0 0 0 1874 1322 8649 6 76 18
17 461 5 0 4874708 0 10771480 0 0 0 1738 1202 9156 6 71 23
35 438 2 0 5011440 0 10638120 0 0 0 1246 1232 8524 7 74 20
18 446 4 0 5079604 0 10568312 0 0 0 2760 1213 8706 7 74 20
29 428 3 0 5274500 0 10374876 0 0 0 1940 1234 8993 6 74 20
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
15 434 4 0 5490296 0 10161236 0 0 0 2080 1225 8980 6 73 21
11 439 1 0 5616836 0 10035308 0 0 0 2126 1196 10325 7 67 26
18 423 4 0 5881364 0 9769676 0 0 0 932 1221 8675 6 75 19
12 415 4 0 6120552 0 9534052 0 0 0 1750 1218 9150 6 73 20
14 400 7 0 6486700 0 9169252 0 0 0 2546 1202 8861 6 73 21
23 375 3 0 6892552 0 8767868 0 0 0 1910 1210 8789 6 75 19
13 367 7 0 7189680 0 8473360 0 0 0 1978 1195 8727 6 74 20
19 345 9 0 7579952 0 8082436 0 0 0 2484 1180 8758 6 75 19
22 329 3 0 7990672 0 7673816 0 0 0 2966 1201 9331 6 72 22
15 321 2 0 8531100 0 7135212 0 0 0 2958 1172 9089 6 73 21
27 277 6 0 9068984 0 6602336 0 0 0 3030 1197 8460 5 75 19
28 246 2 0 9624396 0 6051168 0 0 0 2983 1170 8323 6 76 18
27 226 4 0 10193168 0 5485828 0 0 0 2852 1159 8433 6 74 20
44 177 5 0 10833884 0 4849916 0 0 0 2955 1174 7071 5 80 15
23 164 5 0 11593864 0 4095216 0 0 0 2956 1166 7079 5 80 15
19 123 7 0 12347016 0 3348140 0 0 0 3083 1168 5243 5 82 13
45 66 2 0 13234756 0 2465732 0 0 0 2824 1142 4539 4 82 14
21 4 1 0 15005744 0 707192 0 0 0 2997 1119 476 1 58 41
0 0 1 0 15411664 0 377924 0 0 0 1531 1131 52 0 14 86
Script done on Thu Aug 15 19:24:01 2002
On Thu, Aug 15, 2002 at 04:21:26PM -0700, William Lee Irwin III wrote:
> In the follow-ups I'll include oprofile numbers for each and vmstat logs.
after:
c0105394 2429647 69.5933 default_idle
c012f760 383024 10.9711 generic_file_write
c0114a40 139836 4.00538 scheduler_tick
c01145f8 77154 2.20995 load_balance
c014393c 76891 2.20242 __block_prepare_write
c01d54eb 32084 0.918994 .text.lock.ll_rw_blk
c0136732 20643 0.591285 .text.lock.page_alloc
c016e628 20171 0.577766 ext2_free_branches
c0135c1c 18996 0.54411 rmqueue
c012e96c 16750 0.479777 file_read_actor
c0140664 15094 0.432343 vfs_write
c01564d6 11752 0.336617 .text.lock.dcache
c014f1a2 11128 0.318744 .text.lock.namei
c012e4a8 11016 0.315536 find_get_page
c01d1f70 9011 0.258106 elevator_linus_merge
c01360e0 8451 0.242065 __alloc_pages
c01556c4 8283 0.237253 d_instantiate
c013e1b8 8014 0.229548 __set_page_dirty_nobuffers
c016e0a0 7710 0.220841 ext2_get_block
c0196c45 6642 0.190249 .text.lock.dec_and_lock
c0142248 6612 0.18939 __find_get_block_slow
c0134780 6507 0.186383 activate_page
On Thu, Aug 15, 2002 at 04:21:26PM -0700, William Lee Irwin III wrote:
> With and without kmap_atomic() -based copy_*_user() patches from akpm.
> Taken on a 16x/16GB box.
> Take 1: total throughput:
> before:
> Throughput 32.3019 MB/sec (NB=40.3774 MB/sec 323.019 MBit/sec) 512 procs
> after:
> Throughput 46.9837 MB/sec (NB=58.7296 MB/sec 469.837 MBit/sec) 512 procs
> In the follow-ups I'll include oprofile numbers for each and vmstat logs.
doh! the workload was dbench 512
Cheers,
Bill
William Lee Irwin III wrote:
> With and without kmap_atomic() -based copy_*_user() patches from akpm.
> Taken on a 16x/16GB box.
Have you seen any instability with these things applied? I seem to be
getting a fair amount of these BUG()s. But, I imagine that it could
be a race uncovered because of the serialization that highmem locks
caused.
kernel BUG at softirq.c:229!
invalid operand: 0000
CPU: 4
EIP: 0060:[<8011c8dd>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000 ebx: 80374f54 ecx: 8037e194 edx: ffffffff
esi: 00000000 edi: f7fba000 ebp: 80357560 esp: f7fbbf38
ds: 0068 es: 0068 ss: 0068
Stack: 00000001 80353960 fffffffe 00000080 8011c62a 80353960 00000000
8033b800
00000000 f7fbbf78 00000046 801092e1 f7fba000 80105334 00000000
802de388
00000000 80107d28 f7fba000 00000300 f7fba000 80105334 00000000
00000000
Call Trace: [<8011c62a>] [<801092e1>] [<80105334>] [<80107d28>]
[<80105334>]
[<8010535d>] [<801053b3>] [<801180fd>]
Code: 0f 0b e5 00 13 74 27 80 8b 43 10 50 8b 43 0c ff d0 83 c4 04
>>EIP; 8011c8dd <tasklet_hi_action+5d/c4> <=====
>>ebx; 80374f54 <bh_task_vec+14/280>
>>ecx; 8037e194 <tv1+14/804>
>>ebp; 80357560 <__bss_start+0/0>
Trace; 8011c62a <do_softirq+5a/ac>
Trace; 801092e1 <do_IRQ+f1/100>
Trace; 80105334 <poll_idle+0/48>
Trace; 80107d28 <common_interrupt+18/20>
Trace; 80105334 <poll_idle+0/48>
Trace; 8010535d <poll_idle+29/48>
Trace; 801053b3 <cpu_idle+37/48>
Trace; 801180fd <printk+125/140>
Code; 8011c8dd <tasklet_hi_action+5d/c4>
00000000 <_EIP>:
Code; 8011c8dd <tasklet_hi_action+5d/c4> <=====
0: 0f 0b ud2a <=====
Code; 8011c8df <tasklet_hi_action+5f/c4>
2: e5 00 in $0x0,%eax
Code; 8011c8e1 <tasklet_hi_action+61/c4>
4: 13 74 27 80 adc 0xffffff80(%edi,1),%esi
Code; 8011c8e5 <tasklet_hi_action+65/c4>
8: 8b 43 10 mov 0x10(%ebx),%eax
Code; 8011c8e8 <tasklet_hi_action+68/c4>
b: 50 push %eax
Code; 8011c8e9 <tasklet_hi_action+69/c4>
c: 8b 43 0c mov 0xc(%ebx),%eax
Code; 8011c8ec <tasklet_hi_action+6c/c4>
f: ff d0 call *%eax
Code; 8011c8ee <tasklet_hi_action+6e/c4>
11: 83 c4 04 add $0x4,%esp
--
Dave Hansen
[email protected]
William Lee Irwin III wrote:
>> With and without kmap_atomic() -based copy_*_user() patches from akpm.
>> Taken on a 16x/16GB box.
On Thu, Aug 15, 2002 at 07:10:13PM -0700, Dave Hansen wrote:
> Have you seen any instability with these things applied? I seem to be
> getting a fair amount of these BUG()s. But, I imagine that it could
> be a race uncovered because of the serialization that highmem locks
> caused.
> kernel BUG at softirq.c:229!
> invalid operand: 0000
Hmm, I didn't catch this one. OTOH I did use a fwd port of an earlier
version of the patch. Shall we kgdb? Which box/workload?
Cheers,
Bill
William Lee Irwin III wrote:
> Hmm, I didn't catch this one. OTOH I did use a fwd port of an
> earlier version of the patch. Shall we kgdb?
Yeah, that's the next step.
> Which box/workload?
3b96 the 8-way Xeon 16GB, during the warmup of the specweb file set.
It also happened once at the end of a Specweb run
--
Dave Hansen
[email protected]
Dave Hansen wrote:
>
> William Lee Irwin III wrote:
> > With and without kmap_atomic() -based copy_*_user() patches from akpm.
> > Taken on a 16x/16GB box.
>
> Have you seen any instability with these things applied? I seem to be
> getting a fair amount of these BUG()s. But, I imagine that it could
> be a race uncovered because of the serialization that highmem locks
> caused.
>
> kernel BUG at softirq.c:229!
I just hit it. No kmap patches though.
> ...
>
> >>EIP; 8011c8dd <tasklet_hi_action+5d/c4> <=====
>
> >>ebx; 80374f54 <bh_task_vec+14/280>
> >>ecx; 8037e194 <tv1+14/804>
> >>ebp; 80357560 <__bss_start+0/0>
>
> Trace; 8011c62a <do_softirq+5a/ac>
> Trace; 801092e1 <do_IRQ+f1/100>
> Trace; 80105334 <poll_idle+0/48>
> Trace; 80107d28 <common_interrupt+18/20>
> Trace; 80105334 <poll_idle+0/48>
> Trace; 8010535d <poll_idle+29/48>
> Trace; 801053b3 <cpu_idle+37/48>
#0 tasklet_hi_action (a=0xc03671a0) at softirq.c:230
#1 0xc011e74d in do_softirq () at softirq.c:89
#2 0xc0112b53 in smp_apic_timer_interrupt (regs=
{ebx = -1056148556, ecx = 8, edx = 1024, esi = 134587171, edi = -997773344, ebp = -870006964, eax = 0, xds = -870055832, xes = -1072562072, orig_eax = -17, eip = -1072501662, xcs = 96, eflags = 66054, esp = -965588688, xss = 63491})
at apic.c:1091
#3 0xc0107f7a in apic_timer_interrupt () at stats.c:204
#4 0xc012ee3f in generic_file_write (file=0xcb90d5e0, buf=0x804b340 '\001' <repeats 200 times>..., count=63491,
ppos=0xcb90d600) at filemap.c:2085
#5 0xc013e210 in vfs_write (file=0xcb90d5e0, buf=0x804b340 '\001' <repeats 200 times>..., count=63491, pos=0xcb90d600)
(gdb) p t->func
$1 = (void (*)()) 0xc011eb90 <bh_action>
(gdb) p t->data
$2 = 0
(gdb) p bh_base[0]
$3 = (void (*)()) 0xc0121c94 <timer_bh>
On Friday 16 August 2002 01:35, William Lee Irwin III wrote:
> On Thu, Aug 15, 2002 at 04:21:26PM -0700, William Lee Irwin III wrote:
> > With and without kmap_atomic() -based copy_*_user() patches from akpm.
> > Taken on a 16x/16GB box.
> > Take 1: total throughput:
> > before:
> > Throughput 32.3019 MB/sec (NB=40.3774 MB/sec 323.019 MBit/sec) 512 procs
> > after:
> > Throughput 46.9837 MB/sec (NB=58.7296 MB/sec 469.837 MBit/sec) 512 procs
> > In the follow-ups I'll include oprofile numbers for each and vmstat logs.
>
> doh! the workload was dbench 512
Of course you know about the butterfly effect with respect to dbench...
--
Daniel