Hello,
I finally did benchmark makedumpfile with mmap() on /proc/vmcore on
*2TB memory system*.
In summary, it tooks about 35 seconds to filter 2TB memory. This can be
compared to the two kernel-space filtering works:
- Cliff Wickman's 4 minutes on 8 TB memory system:
http://lists.infradead.org/pipermail/kexec/2012-November/007177.html
- Jingbai Ma's 17.50 seconds on 1TB memory system:
https://lkml.org/lkml/2013/3/7/275
= Machine spec
- System: PRIMEQUEST 1800E2
- CPU: Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz (8 sockets, 10 cores, 2 threads)
(*) only 1 lcpu is used in the 2nd kernel now.
- memory: 2TB
- kernel: 3.9-rc3 with the patch set in: https://lkml.org/lkml/2013/3/18/878
- kexec tools: v2.0.4
- makedumpfile
- v1.5.2-map: git map branch
- git://git.code.sf.net/p/makedumpfile/code
- To use mmap, specify --map-size <size in kilo-bytes> option.
= Perofrmance of filtering processing
== How to measure
I measured performance of filtering processing by reading time
contained in makedumpfile's report message. For example:
$ makedumpfile --message-level 31 -p -d 31 /proc/vmcore vmcore-pd31
...
STEP [Checking for memory holes ] : 0.163673 seconds
STEP [Excluding unnecessary pages] : 1.321702 seconds
STEP [Excluding free pages ] : 0.489022 seconds
STEP [Copying data ] : 26.221380 seconds
The message starting with "STEP [Excluding" corresponds to the message
of filtering processing.
- STEP [Excluding unnecessary pages] corresponds to the time for
mem_map array logic.
- STEP [Excluding free pages ] corresponds to the time for free list
logic.
The message is displayed multiple times in cyclic mode, exactly the
same number of cycles.
== Result
mmap
| map_size | unnecessay | unnecessary | free list |
| [KB] | cyclic | non-cyclic | non-cyclic |
|----------+------------+-------------+------------|
| 4 | 66.212 | 59.087 | 75.165 |
| 8 | 51.594 | 44.863 | 75.657 |
| 16 | 43.761 | 36.338 | 75.508 |
| 32 | 39.235 | 32.911 | 76.061 |
| 64 | 37.201 | 30.201 | 76.116 |
| 128 | 35.901 | 29.238 | 76.261 |
| 256 | 35.152 | 28.506 | 76.700 |
| 512 | 34.711 | 27.956 | 77.660 |
| 1024 | 34.432 | 27.746 | 79.319 |
| 2048 | 34.361 | 27.594 | 84.331 |
| 4096 | 34.236 | 27.474 | 91.517 |
| 8192 | 34.173 | 27.450 | 105.648 |
| 16384 | 34.240 | 27.448 | 133.099 |
| 32768 | 34.291 | 27.479 | 184.488 |
read
| unnecessary | unnecessary | free list |
| cyclic | non-cyclic | non-cyclic |
|-------------+-------------+------------|
| 100.859588 | 93.881849 | 80.367015 |
== Discussion
- The best case shows the performance close to the ones in the
kernel-space works by Cliff and Ma as mentioned first.
- The reason why times consumed for filtering unnecessary pages are
different between cyclic mode nad non-cyclic mode is that the former
does free pages filtering while the latter does not; in the latter,
page filtering is done in free list logic.
= Performance degradation in cyclic mode
Next benchmark case is to measure how performance is changed in
cyclic-mode if the number of cycles is increased.
== How to measure
Similarly to the above, but in this benchmark I also added
--cyclic-buffer as parameter.
The command I executed was like:
for buf_size in 4 8 16 ... 32768 ; do
time makedumpfile --cyclic-buffer ${buf_size} /proc/vmcore vmcore
rm -f ./vmcore
done
I choosed buffers sizes as the number of cycles ranged from 1 to 8
because current existing huge system memory size is up to 16TB and if
crashkernel=512MB, the number of cycles would be at most 8.
== Result
mmap
| buf size | nr cycles | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | total |
| [KB] | | | | | | | | | | |
|----------+-----------+--------+--------+--------+-------+-------+-------+-------+-------+--------|
| 8747 | 8 | 4.695 | 4.470 | 4.582 | 4.512 | 4.935 | 4.790 | 4.824 | 2.345 | 35.153 |
| 9371 | 8 | 5.010 | 4.782 | 4.891 | 4.996 | 5.280 | 5.108 | 4.986 | 0.007 | 35.059 |
| 10092 | 7 | 5.371 | 5.145 | 5.001 | 5.316 | 5.500 | 5.405 | 2.593 | - | 34.330 |
| 10933 | 7 | 5.816 | 5.581 | 5.533 | 6.169 | 6.163 | 5.882 | 0.007 | - | 35.152 |
| 11927 | 6 | 6.308 | 6.078 | 6.174 | 6.734 | 6.667 | 3.049 | - | - | 35.010 |
| 13120 | 5 | 6.967 | 6.641 | 6.973 | 7.427 | 6.899 | - | - | - | 34.907 |
| 14578 | 5 | 7.678 | 7.536 | 7.948 | 8.161 | 3.845 | - | - | - | 35.167 |
| 16400 | 4 | 8.942 | 8.697 | 9.529 | 9.276 | - | - | - | - | 36.445 |
| 18743 | 4 | 9.822 | 9.718 | 10.452 | 5.013 | - | - | - | - | 35.005 |
| 21867 | 3 | 11.413 | 11.550 | 11.923 | - | - | - | - | - | 34.886 |
| 26240 | 3 | 13.554 | 14.104 | 7.114 | - | - | - | - | - | 34.772 |
| 32800 | 2 | 16.693 | 17.809 | - | - | - | - | - | - | 34.502 |
| 43733 | 2 | 22.633 | 11.863 | - | - | - | - | - | - | 34.497 |
| 65600 | 1 | 34.245 | - | - | - | - | - | - | - | 34.245 |
| 131200 | 1 | 34.291 | - | - | - | - | - | - | - | 34.291 |
read
| buf size | nr cycles | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | total |
| [KB] | | | | | | | | | | |
|----------+-----------+---------+--------+--------+--------+--------+--------+--------+-------+---------|
| 8747 | 8 | 13.514 | 13.351 | 13.294 | 13.488 | 13.981 | 13.678 | 13.848 | 6.953 | 102.106 |
| 9371 | 8 | 14.429 | 14.279 | 14.484 | 14.624 | 14.929 | 14.649 | 14.620 | 0.001 | 102.017 |
| 10092 | 7 | 15.560 | 15.375 | 15.164 | 15.559 | 15.720 | 15.626 | 8.033 | - | 101.036 |
| 10933 | 7 | 16.906 | 16.724 | 16.650 | 17.474 | 17.440 | 17.127 | 0.002 | - | 102.319 |
| 11927 | 6 | 18.456 | 18.254 | 18.339 | 19.037 | 18.943 | 9.477 | - | - | 102.505 |
| 13120 | 5 | 20.162 | 20.222 | 20.287 | 20.779 | 20.149 | - | - | - | 101.599 |
| 14578 | 5 | 22.646 | 22.535 | 23.006 | 23.237 | 11.519 | - | - | - | 102.942 |
| 16400 | 4 | 25.228 | 25.033 | 26.016 | 25.660 | - | - | - | - | 101.936 |
| 18743 | 4 | 28.849 | 28.761 | 29.648 | 14.677 | - | - | - | - | 101.935 |
| 21867 | 3 | 33.720 | 33.877 | 34.344 | - | - | - | - | - | 101.941 |
| 26240 | 3 | 40.403 | 41.042 | 20.642 | - | - | - | - | - | 102.087 |
| 32800 | 2 | 50.393 | 51.895 | - | - | - | - | - | - | 102.288 |
| 43733 | 2 | 66.658 | 34.056 | - | - | - | - | - | - | 100.714 |
| 65600 | 1 | 100.975 | - | - | - | - | - | - | - | 100.975 |
| 131200 | 1 | 100.699 | - | - | - | - | - | - | - | 100.699 |
- As the result shows, there's very small degradation only; just a
second. Also, this small degradation depens on the number of cycles,
not IO size, so there seems no effect even if system memory becomes
larger.
Thanks.
HATAYAMA, Daisuke