Here follow the latest benchmarks with contest (http://contest.kolivas.net)
noload:
Kernel Time CPU Ratio
2.4.19 67.71 98% 1.00*
2.5.38 72.38 94% 1.07
2.5.38-mm3 73.00 93% 1.08
2.5.39 73.17 93% 1.08
process_load:
Kernel Time CPU Ratio
2.4.19 110.75 57% 1.64*
2.5.38 85.71 79% 1.27
2.5.38-mm3 96.32 72% 1.42*
2.5.39 88.18 77% 1.30
io_load:
Kernel Time CPU Ratio
2.4.19 216.05 33% 3.19
2.5.38 887.76 8% 13.11*
2.5.38-mm3 105.17 70% 1.55*
2.5.39 216.81 37% 3.20
mem_load:
Kernel Time CPU Ratio
2.4.19 105.40 70% 1.56
2.5.38 107.89 73% 1.59
2.5.38-mm3 117.09 63% 1.73*
2.5.39 103.72 72% 1.53
Things to note:
Despite the new deadline scheduler, performance under IO load is worse than
2.5.38-mm3 (something else?)
Asterisks are placed where the difference was statistically significant from 2.5.39
Hardware: 1133Mhz P3, 224Mb Ram, IDE ATA100 5400rpm drive, IO load on same drive
as compile.
*NOTE* New version of contest means results are not compatible with results of
0.3x. Process Load now more of a process load (and less of a cpu load), and
changed priming (memory flushing) prior to each test results in far greater
resolution of results - hence the notable change in even noload results.
Comments?
Con.
Con Kolivas wrote:
>
> Here follow the latest benchmarks with contest (http://contest.kolivas.net)
>
> noload:
> Kernel Time CPU Ratio
> 2.4.19 67.71 98% 1.00*
> 2.5.38 72.38 94% 1.07
> 2.5.38-mm3 73.00 93% 1.08
> 2.5.39 73.17 93% 1.08
>
> process_load:
> Kernel Time CPU Ratio
> 2.4.19 110.75 57% 1.64*
> 2.5.38 85.71 79% 1.27
> 2.5.38-mm3 96.32 72% 1.42*
> 2.5.39 88.18 77% 1.30
well that's funny.
> io_load:
> Kernel Time CPU Ratio
> 2.4.19 216.05 33% 3.19
> 2.5.38 887.76 8% 13.11*
> 2.5.38-mm3 105.17 70% 1.55*
> 2.5.39 216.81 37% 3.20
-mm3 has fifo_batch=16. 2.5.39 has fifo_batch=32.
> mem_load:
> Kernel Time CPU Ratio
> 2.4.19 105.40 70% 1.56
> 2.5.38 107.89 73% 1.59
> 2.5.38-mm3 117.09 63% 1.73*
> 2.5.39 103.72 72% 1.53
2.5's swapout is still fairly synchronously sucky. So low-latency
writeout could be advantageous there. This difference is probably
also the fifo_batch thing. Or maybe statistical?
I did some testing with your latest. 4xPIII, mem=512m, SCSI,
tag depth = 0, 2.5.39-mm1 candidate:
fifo_batch=32:
noload 2:34.53 291%
cpuload 2:36.20 286%
memload 2:19.44 333%
ioloadhalf 2:34.81 303%
ioloadfull 3:15.47 238%
(err. memload sped it up!)
fifo_batch=16:
noload 2:00.03 380%
cpuload 2:27.62 304%
memload 2:22.59 326%
ioloadhalf 2:33.75 306%
ioloadfull 2:59.18 259%
- Something went horridly wrong in the first `noload' test.
- fifo_batch=16 is better than 32.
- you see a 4x hit from io_load. I see a 1.5x hit.
This is all pretty wild. I'll go profile process_load a bit.
BTW, please change all the
#define dprintf(...) printf(__VA_ARGS__)
to
#define dprintf(x...) printf(x)
so people who use crufty old compilers can build it.
Quoting Andrew Morton <[email protected]>:
> Con Kolivas wrote:
> >
> > Here follow the latest benchmarks with contest
> (http://contest.kolivas.net)
> >
> > noload:
> > Kernel Time CPU Ratio
> > 2.4.19 67.71 98% 1.00*
> > 2.5.38 72.38 94% 1.07
> > 2.5.38-mm3 73.00 93% 1.08
> > 2.5.39 73.17 93% 1.08
> >
> > process_load:
> > Kernel Time CPU Ratio
> > 2.4.19 110.75 57% 1.64*
> > 2.5.38 85.71 79% 1.27
> > 2.5.38-mm3 96.32 72% 1.42*
> > 2.5.39 88.18 77% 1.30
>
> well that's funny.
>
> > io_load:
> > Kernel Time CPU Ratio
> > 2.4.19 216.05 33% 3.19
> > 2.5.38 887.76 8% 13.11*
> > 2.5.38-mm3 105.17 70% 1.55*
> > 2.5.39 216.81 37% 3.20
>
> -mm3 has fifo_batch=16. 2.5.39 has fifo_batch=32.
>
> > mem_load:
> > Kernel Time CPU Ratio
> > 2.4.19 105.40 70% 1.56
> > 2.5.38 107.89 73% 1.59
> > 2.5.38-mm3 117.09 63% 1.73*
> > 2.5.39 103.72 72% 1.53
>
> 2.5's swapout is still fairly synchronously sucky. So low-latency
> writeout could be advantageous there. This difference is probably
> also the fifo_batch thing. Or maybe statistical?
>
>
> I did some testing with your latest. 4xPIII, mem=512m, SCSI,
> tag depth = 0, 2.5.39-mm1 candidate:
>
> fifo_batch=32:
>
> noload 2:34.53 291%
> cpuload 2:36.20 286%
> memload 2:19.44 333%
> ioloadhalf 2:34.81 303%
> ioloadfull 3:15.47 238%
>
> (err. memload sped it up!)
>
> fifo_batch=16:
>
> noload 2:00.03 380%
> cpuload 2:27.62 304%
> memload 2:22.59 326%
> ioloadhalf 2:33.75 306%
> ioloadfull 2:59.18 259%
>
> - Something went horridly wrong in the first `noload' test.
> - fifo_batch=16 is better than 32.
> - you see a 4x hit from io_load. I see a 1.5x hit.
>
> This is all pretty wild. I'll go profile process_load a bit.
>
>
>
> BTW, please change all the
>
> #define dprintf(...) printf(__VA_ARGS__)
>
> to
>
> #define dprintf(x...) printf(x)
>
> so people who use crufty old compilers can build it.
>
Ok will fix. But please Andrew use version 0.41 of contest (posted only 2 hours
ago). The results from that are far more meaningful and reproducible.
Con
Con Kolivas wrote:
>
> ...
> Ok will fix. But please Andrew use version 0.41 of contest (posted only 2 hours
> ago). The results from that are far more meaningful and reproducible.
>
I am doing. I just tested for the "proces load" variation which
you saw (I assume that's what the test calls "CPU load"). With
and without the -mm patches:
cpuload 2:31.55 297%
cpuload 2:31.75 295%
So... Maybe some extra samples would be needed there.
On Sat, Sep 28 2002, Con Kolivas wrote:
> > > io_load:
> > > Kernel Time CPU Ratio
> > > 2.4.19 216.05 33% 3.19
> > > 2.5.38 887.76 8% 13.11*
> > > 2.5.38-mm3 105.17 70% 1.55*
> > > 2.5.39 216.81 37% 3.20
> >
> > -mm3 has fifo_batch=16. 2.5.39 has fifo_batch=32.
That's not the only difference, btw.
> > > mem_load:
> > > Kernel Time CPU Ratio
> > > 2.4.19 105.40 70% 1.56
> > > 2.5.38 107.89 73% 1.59
> > > 2.5.38-mm3 117.09 63% 1.73*
> > > 2.5.39 103.72 72% 1.53
> >
> > 2.5's swapout is still fairly synchronously sucky. So low-latency
> > writeout could be advantageous there. This difference is probably
> > also the fifo_batch thing. Or maybe statistical?
> >
> >
> > I did some testing with your latest. 4xPIII, mem=512m, SCSI,
> > tag depth = 0, 2.5.39-mm1 candidate:
> >
> > fifo_batch=32:
> >
> > noload 2:34.53 291%
> > cpuload 2:36.20 286%
> > memload 2:19.44 333%
> > ioloadhalf 2:34.81 303%
> > ioloadfull 3:15.47 238%
> >
> > (err. memload sped it up!)
> >
> > fifo_batch=16:
> >
> > noload 2:00.03 380%
> > cpuload 2:27.62 304%
> > memload 2:22.59 326%
> > ioloadhalf 2:33.75 306%
> > ioloadfull 2:59.18 259%
> >
> > - Something went horridly wrong in the first `noload' test.
> > - fifo_batch=16 is better than 32.
> > - you see a 4x hit from io_load. I see a 1.5x hit.
So far fifo_batch=16 looks pretty good. Doesn't quite make sense to me.
Need to bench/test some more :-)
--
Jens Axboe
Quoting Jens Axboe <[email protected]>:
> On Sat, Sep 28 2002, Con Kolivas wrote:
> > > > io_load:
> > > > Kernel Time CPU Ratio
> > > > 2.4.19 216.05 33% 3.19
> > > > 2.5.38 887.76 8% 13.11*
> > > > 2.5.38-mm3 105.17 70% 1.55*
> > > > 2.5.39 216.81 37% 3.20
> > >
> > > -mm3 has fifo_batch=16. 2.5.39 has fifo_batch=32.
>
> That's not the only difference, btw.
>
> > > > mem_load:
> > > > Kernel Time CPU Ratio
> > > > 2.4.19 105.40 70% 1.56
> > > > 2.5.38 107.89 73% 1.59
> > > > 2.5.38-mm3 117.09 63% 1.73*
> > > > 2.5.39 103.72 72% 1.53
> > >
> > > 2.5's swapout is still fairly synchronously sucky. So low-latency
> > > writeout could be advantageous there. This difference is probably
> > > also the fifo_batch thing. Or maybe statistical?
> > >
> > >
> > > I did some testing with your latest. 4xPIII, mem=512m, SCSI,
> > > tag depth = 0, 2.5.39-mm1 candidate:
> > >
> > > fifo_batch=32:
> > >
> > > noload 2:34.53 291%
> > > cpuload 2:36.20 286%
> > > memload 2:19.44 333%
> > > ioloadhalf 2:34.81 303%
> > > ioloadfull 3:15.47 238%
> > >
> > > (err. memload sped it up!)
> > >
> > > fifo_batch=16:
> > >
> > > noload 2:00.03 380%
> > > cpuload 2:27.62 304%
> > > memload 2:22.59 326%
> > > ioloadhalf 2:33.75 306%
> > > ioloadfull 2:59.18 259%
> > >
> > > - Something went horridly wrong in the first `noload' test.
> > > - fifo_batch=16 is better than 32.
> > > - you see a 4x hit from io_load. I see a 1.5x hit.
>
> So far fifo_batch=16 looks pretty good. Doesn't quite make sense to me.
> Need to bench/test some more :-)
Andrew was using an older version of contest which may have been misrepresenting
things as there were serious limitations in the older versions.
I've directed him to the new version which has worked around (most) of the
limitations. SMP on the older version was particularly bad.
Con.
HP Omnibook 6000 (laptop), 256 MiB of RAM, PIII@800.
Test against 2.4.19, 2.5.38-mm2, 2.5.39
What I did:
$ rebootin "kernel" apm=off single
$ contest -n 3
Results:
Administrator@OIVT444P /cygdrive/log
$ cat results.log
noload:
Kernel Time CPU Ratio
2.4.19 133.07 98% 1.00
2.4.19 133.16 98% 1.00
2.4.19 135.43 97% 1.02
2.5.38-mm2 138.19 97% 1.04
2.5.38-mm2 138.47 96% 1.04
2.5.38-mm2 139.54 96% 1.05
2.5.39 138.30 96% 1.04
2.5.39 138.63 96% 1.04
2.5.39 139.99 96% 1.05
process_load:
Kernel Time CPU Ratio
2.4.19 200.43 60% 1.51
2.4.19 203.11 60% 1.53
2.4.19 203.97 59% 1.53
2.5.38-mm2 194.42 69% 1.46
2.5.38-mm2 195.19 69% 1.47
2.5.38-mm2 207.36 64% 1.56
2.5.39 190.44 70% 1.43
2.5.39 191.37 70% 1.44
2.5.39 193.60 69% 1.45
io_load:
Kernel Time CPU Ratio
2.4.19 486.58 27% 3.66
2.4.19 593.72 22% 4.46
2.4.19 637.61 21% 4.79
2.5.38-mm2 232.35 61% 1.75
2.5.38-mm2 237.83 57% 1.79
2.5.38-mm2 274.39 50% 2.06
2.5.39 242.98 57% 1.83
2.5.39 294.52 50% 2.21
2.5.39 328.01 42% 2.46
mem_load:
Kernel Time CPU Ratio
2.4.19 172.24 78% 1.29
2.4.19 174.74 77% 1.31
2.4.19 174.87 77% 1.31
2.5.38-mm2 165.53 82% 1.24
2.5.38-mm2 170.00 80% 1.28
2.5.38-mm2 171.96 79% 1.29
2.5.39 167.92 81% 1.26
2.5.39 170.80 80% 1.28
2.5.39 172.68 79% 1.30
Ciao,
Paolo
--
Get your free email from http://www.linuxmail.org
Powered by Outblaze
Quoting Paolo Ciarrocchi <[email protected]>:
> noload:
> Kernel Time CPU Ratio
> 2.4.19 133.07 98% 1.00
> 2.4.19 133.16 98% 1.00
> 2.4.19 135.43 97% 1.02
> 2.5.38-mm2 138.19 97% 1.04
> 2.5.38-mm2 138.47 96% 1.04
> 2.5.38-mm2 139.54 96% 1.05
> 2.5.39 138.30 96% 1.04
> 2.5.39 138.63 96% 1.04
> 2.5.39 139.99 96% 1.05
>
> process_load:
> Kernel Time CPU Ratio
> 2.4.19 200.43 60% 1.51
> 2.4.19 203.11 60% 1.53
> 2.4.19 203.97 59% 1.53
> 2.5.38-mm2 194.42 69% 1.46
> 2.5.38-mm2 195.19 69% 1.47
> 2.5.38-mm2 207.36 64% 1.56
> 2.5.39 190.44 70% 1.43
> 2.5.39 191.37 70% 1.44
> 2.5.39 193.60 69% 1.45
>
> io_load:
> Kernel Time CPU Ratio
> 2.4.19 486.58 27% 3.66
> 2.4.19 593.72 22% 4.46
> 2.4.19 637.61 21% 4.79
> 2.5.38-mm2 232.35 61% 1.75
> 2.5.38-mm2 237.83 57% 1.79
> 2.5.38-mm2 274.39 50% 2.06
> 2.5.39 242.98 57% 1.83
> 2.5.39 294.52 50% 2.21
> 2.5.39 328.01 42% 2.46
>
> mem_load:
> Kernel Time CPU Ratio
> 2.4.19 172.24 78% 1.29
> 2.4.19 174.74 77% 1.31
> 2.4.19 174.87 77% 1.31
> 2.5.38-mm2 165.53 82% 1.24
> 2.5.38-mm2 170.00 80% 1.28
> 2.5.38-mm2 171.96 79% 1.29
> 2.5.39 167.92 81% 1.26
> 2.5.39 170.80 80% 1.28
> 2.5.39 172.68 79% 1.30
Quick statistical analysis:
Noload, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
ProcessLoad, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
IO Load, 2.5.39 is faster than 2.4.19 and _appears_ slower than 2.5.38-mm2 but
has no statistically significant difference; This is probably a type 2 error
(meaning more samples are required). Paolo if you could perform three more runs
on these two kernels it would help discriminate for those in the crowd who need
firm proof.
Mem Load, 2.5.39 is faster than 2.4.19 and same as 2.5.38-mm2
Note that for the results to be useful, they need to be run back to back on the
same system as you seem to have done. If you use your machine between runs for
something else, it can and probably will affect any further results.
Con
From: Con Kolivas <[email protected]>
[...]
> > process_load:
> > Kernel Time CPU Ratio
> > 2.4.19 200.43 60% 1.51
> > 2.4.19 203.11 60% 1.53
> > 2.4.19 203.97 59% 1.53
> > 2.5.38-mm2 194.42 69% 1.46
> > 2.5.38-mm2 195.19 69% 1.47
> > 2.5.38-mm2 207.36 64% 1.56
> > 2.5.39 190.44 70% 1.43
> > 2.5.39 191.37 70% 1.44
> > 2.5.39 193.60 69% 1.45
> >
> > io_load:
> > Kernel Time CPU Ratio
> > 2.4.19 486.58 27% 3.66
> > 2.4.19 593.72 22% 4.46
> > 2.4.19 637.61 21% 4.79
> > 2.5.38-mm2 232.35 61% 1.75
> > 2.5.38-mm2 237.83 57% 1.79
> > 2.5.38-mm2 274.39 50% 2.06
> > 2.5.39 242.98 57% 1.83
> > 2.5.39 294.52 50% 2.21
> > 2.5.39 328.01 42% 2.46
> >
> > mem_load:
> > Kernel Time CPU Ratio
> > 2.4.19 172.24 78% 1.29
> > 2.4.19 174.74 77% 1.31
> > 2.4.19 174.87 77% 1.31
> > 2.5.38-mm2 165.53 82% 1.24
> > 2.5.38-mm2 170.00 80% 1.28
> > 2.5.38-mm2 171.96 79% 1.29
> > 2.5.39 167.92 81% 1.26
> > 2.5.39 170.80 80% 1.28
> > 2.5.39 172.68 79% 1.30
>
> Quick statistical analysis:
> Noload, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
>
> ProcessLoad, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
Why ?
If look at the numbers I assume that 2.5.39 is faster then 2.4.19.
Am I missing something?
I'll run further test...
Ciao,
Paolo
--
Get your free email from http://www.linuxmail.org
Powered by Outblaze
Quoting Paolo Ciarrocchi <[email protected]>:
> From: Con Kolivas <[email protected]>
> [...]
> > > process_load:
> > > Kernel Time CPU Ratio
> > > 2.4.19 200.43 60% 1.51
> > > 2.4.19 203.11 60% 1.53
> > > 2.4.19 203.97 59% 1.53
> > > 2.5.38-mm2 194.42 69% 1.46
> > > 2.5.38-mm2 195.19 69% 1.47
> > > 2.5.38-mm2 207.36 64% 1.56
> > > 2.5.39 190.44 70% 1.43
> > > 2.5.39 191.37 70% 1.44
> > > 2.5.39 193.60 69% 1.45
> > >
> > > io_load:
> > > Kernel Time CPU Ratio
> > > 2.4.19 486.58 27% 3.66
> > > 2.4.19 593.72 22% 4.46
> > > 2.4.19 637.61 21% 4.79
> > > 2.5.38-mm2 232.35 61% 1.75
> > > 2.5.38-mm2 237.83 57% 1.79
> > > 2.5.38-mm2 274.39 50% 2.06
> > > 2.5.39 242.98 57% 1.83
> > > 2.5.39 294.52 50% 2.21
> > > 2.5.39 328.01 42% 2.46
> > >
> > > mem_load:
> > > Kernel Time CPU Ratio
> > > 2.4.19 172.24 78% 1.29
> > > 2.4.19 174.74 77% 1.31
> > > 2.4.19 174.87 77% 1.31
> > > 2.5.38-mm2 165.53 82% 1.24
> > > 2.5.38-mm2 170.00 80% 1.28
> > > 2.5.38-mm2 171.96 79% 1.29
> > > 2.5.39 167.92 81% 1.26
> > > 2.5.39 170.80 80% 1.28
> > > 2.5.39 172.68 79% 1.30
> >
> > Quick statistical analysis:
> > Noload, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
> >
> > ProcessLoad, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
> Why ?
> If look at the numbers I assume that 2.5.39 is faster then 2.4.19.
> Am I missing something?
Sorry, typo should read 2.5.39 is faster than 2.4.19 and same as 2.5.38-mm2
> I'll run further test...
Not really needed. I'm convinced the difference is there, and the people who can
act on the data probably will be happy with that much information too. Some are
less satisfied with the quality of the data unless there is firm statistical
data to support the hypothesis. Your time is better spent on other things.
Con
From: Con Kolivas <[email protected]>
[...]
> > > ProcessLoad, 2.5.39 is slower than 2.4.19 and same as 2.5.38-mm2
> > Why ?
> > If look at the numbers I assume that 2.5.39 is faster then 2.4.19.
> > Am I missing something?
>
> Sorry, typo should read 2.5.39 is faster than 2.4.19 and same as 2.5.38-mm2
Ok.
> > I'll run further test...
>
> Not really needed. I'm convinced the difference is there, and the people who can
> act on the data probably will be happy with that much information too. Some are
> less satisfied with the quality of the data unless there is firm statistical
> data to support the hypothesis. Your time is better spent on other things.
I've just ran further tests...
Administrator@OIVT444P ~
$ cat /cygdrive/log/results.log
noload:
Kernel Time CPU Ratio
2.4.19 133.07 98% 1.00
2.4.19 133.16 98% 1.00
2.4.19 135.43 97% 1.02
2.5.38-mm2 138.19 97% 1.04
2.5.38-mm2 138.47 96% 1.04
2.5.38-mm2 138.72 97% 1.04
2.5.38-mm2 139.54 96% 1.05
2.5.38-mm2 139.59 96% 1.05
2.5.38-mm2 139.88 96% 1.05
2.5.39 138.30 96% 1.04
2.5.39 138.63 96% 1.04
2.5.39 138.70 96% 1.04
2.5.39 138.70 96% 1.04
2.5.39 139.44 96% 1.05
2.5.39 139.99 96% 1.05
process_load:
Kernel Time CPU Ratio
2.4.19 200.43 60% 1.51
2.4.19 203.11 60% 1.53
2.4.19 203.97 59% 1.53
2.5.38-mm2 190.13 70% 1.43
2.5.38-mm2 194.06 69% 1.46
2.5.38-mm2 194.25 69% 1.46
2.5.38-mm2 194.42 69% 1.46
2.5.38-mm2 195.19 69% 1.47
2.5.38-mm2 207.36 64% 1.56
2.5.39 188.72 71% 1.42
2.5.39 190.44 70% 1.43
2.5.39 191.37 70% 1.44
2.5.39 191.48 70% 1.44
2.5.39 193.60 69% 1.45
2.5.39 199.50 67% 1.50
io_load:
Kernel Time CPU Ratio
2.4.19 486.58 27% 3.66
2.4.19 593.72 22% 4.46
2.4.19 637.61 21% 4.79
2.5.38-mm2 232.35 61% 1.75
2.5.38-mm2 237.83 57% 1.79
2.5.38-mm2 247.05 58% 1.86
2.5.38-mm2 274.39 50% 2.06
2.5.38-mm2 281.40 49% 2.11
2.5.38-mm2 295.87 47% 2.22
2.5.39 233.58 59% 1.76
2.5.39 242.98 57% 1.83
2.5.39 272.38 51% 2.05
2.5.39 294.52 50% 2.21
2.5.39 304.73 45% 2.29
2.5.39 328.01 42% 2.46
mem_load:
Kernel Time CPU Ratio
2.4.19 172.24 78% 1.29
2.4.19 174.74 77% 1.31
2.4.19 174.87 77% 1.31
2.5.38-mm2 165.53 82% 1.24
2.5.38-mm2 170.00 80% 1.28
2.5.38-mm2 170.89 79% 1.28
2.5.38-mm2 171.84 79% 1.29
2.5.38-mm2 171.96 79% 1.29
2.5.38-mm2 172.15 79% 1.29
2.5.39 167.92 81% 1.26
2.5.39 168.38 81% 1.27
2.5.39 170.16 80% 1.28
2.5.39 170.64 80% 1.28
2.5.39 170.80 80% 1.28
2.5.39 172.68 79% 1.30
Con, do you think is a good idea add the capability to the contest bechmark to provide an analisys of the results ? I have a few ideas, if you want we can contunue the discussion in pvt.
Ciao,
Paolo
--
Get your free email from http://www.linuxmail.org
Powered by Outblaze