Hi!
Using Linux 3.10-rc5 on an ThinkPad T520 with Intel Sandybridge i5-2620M,
8 GiB RAM and Intel SSD 320. Currently I have Zcache enabled to test the
effects of it but I observed similar figures on kernels without Zcache.
If I let the kernel swap out for example with
stress -m 1 --vm-keep --vm-bytes 5G
or so, then swapping out is pretty fast, I have seen values around
100-200 MiB/s
But on issuing a swapoff command to swap stuff in again, the swap in is
abysmally slow, just a few MiB/s (see below).
I wonder why is that so? The SSD is basically idling around on swap-in.
merkaba:~> stress -m 1 --vm-keep --vm-bytes 5G
stress: info: [10998] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
^C
merkaba:~> swapon -s
Filename Type Size Used Priority
/dev/mapper/merkaba-swap partition 12582908 2168872 -1
merkaba:~> free -m
total used free shared buffers cached
Mem: 7767 2342 5425 0 0 305
-/+ buffers/cache: 2036 5730
Swap: 12287 2182 10105
merkaba:~> vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 2234528 5554868 40 313056 5 14 532 53 225 309 6 2 91 0
0 0 2234508 5554736 40 313168 60 0 60 0 573 926 0 1 99 0
0 0 2234508 5554736 40 313168 0 0 0 0 708 1317 1 1 98 0
0 0 2234492 5554612 40 313168 52 0 52 0 530 946 0 1 99 0
1 0 2193204 5524412 372 340940 1908 0 2476 0 3857 6961 1 22 75 2
1 0 2190412 5522924 372 341044 0 0 0 0 2334 4606 0 26 74 0
1 0 2187116 5521188 372 341044 0 0 0 0 2212 4625 0 26 74 0
1 0 2182972 5517964 372 341072 1852 0 1852 0 3635 6901 0 25 73 2
1 0 2178332 5515112 372 341076 196 0 196 0 2460 4997 1 26 73 1
2 0 2172832 5511980 372 341076 104 0 104 0 2275 4742 0 26 74 0
1 0 2165116 5508136 372 341076 12 0 12 0 2320 4741 0 26 74 0
1 0 2154596 5501936 372 341080 92 0 92 0 2263 4768 0 26 74 0
1 0 2144784 5494620 372 341108 3048 0 3048 0 4491 8359 1 24 73 3
1 0 2135528 5486748 372 341092 5376 0 5376 0 5969 10706 1 23 72 4
1 0 2131528 5481984 372 341268 2160 0 4776 0 3841 7337 1 24 72 3
1 3 2128708 5478008 372 341544 3324 0 3648 0 3844 7318 0 24 72 4
2 0 2126884 5463440 372 353084 2956 0 95900 0 9990 19942 1 27 51 21
1 0 2124888 5461332 372 353084 1592 0 1592 0 3453 6590 0 25 73 2
2 1 2118644 5439740 372 360832 6364 0 86884 4 10292 20131 1 29 42 27
1 0 2113572 5415604 372 370908 4812 0 33340 0 9810 18000 5 33 52 10
1 0 2111736 5413872 372 370684 1596 0 1684 0 3475 6487 1 25 73 2
1 0 2109836 5411888 372 370728 1912 0 1912 0 3571 6776 0 25 73 2
1 0 2107804 5410028 372 370740 1360 0 1360 0 3162 6164 0 25 73 2
1 0 2106104 5407996 372 370740 1820 0 1820 0 3355 6493 0 25 73 2
1 0 2103756 5406136 372 370776 976 0 976 0 3057 5974 1 26 71 1
1 0 2101548 5403160 372 370776 2356 0 2356 0 3588 6939 1 26 71 2
1 0 2099000 5400804 372 370776 1948 0 1948 0 3666 6884 1 25 72 3
2 0 2096784 5398820 372 370776 1944 0 1944 0 3881 7383 1 27 70 2
1 0 2094976 5397208 372 370776 1584 0 1584 0 3541 6564 1 25 73 2
1 0 2092924 5394976 372 370776 1752 0 1752 0 3513 6649 0 24 73 2
1 0 2089460 5391752 372 370776 1072 0 1072 0 3051 5932 1 25 73 2
1 0 2084456 5389116 372 370776 496 0 496 0 2694 5272 0 26 74 1
1 1 2081160 5386256 372 370804 2116 0 2116 696 4039 7462 1 25 73 2
3 0 2077804 5383156 372 370816 3004 0 3004 0 4394 8157 1 24 73 3
1 0 2075284 5380800 372 370816 1856 0 1856 0 3570 6831 1 25 73 2
1 0 2073544 5379064 372 370824 1452 0 1452 0 3246 6270 0 25 73 2
According to atop the SSD is about 5-15% busy.
Swapoff is quite CPU busy:
PID TID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/7
11006 - root root 1 7.54s 0.00s 0K 0K 30100K 0K -- - R 0 76% swapoff
PID TID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/8
11006 - root root 1 9.65s 0.00s 0K 0K 7148K 0K -- - R 3 97% swapoff
PID TID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/6
11006 - root root 1 9.60s 0.00s 0K 0K 8752K 0K -- - R 3 96% swapoff
Anyway idea?
I can try this without Zcache as well. But I am pretty sure it was slow
as well. But I had ZCACHE compiled in, as far as I guess from hints on
the web, I bet its basically a no-op unless enabled with "zcache" kernel
commandline parameter.
Thanks,
--
Martin Steigerwald - teamix GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
On 06/11/2013 09:34:36 AM, Martin Steigerwald wrote:
> Hi!
>
> Using Linux 3.10-rc5 on an ThinkPad T520 with Intel Sandybridge
> i5-2620M,
> 8 GiB RAM and Intel SSD 320. Currently I have Zcache enabled to test
> the
> effects of it but I observed similar figures on kernels without
> Zcache.
>
> If I let the kernel swap out for example with
>
> stress -m 1 --vm-keep --vm-bytes 5G
>
> or so, then swapping out is pretty fast, I have seen values around
> 100-200 MiB/s
>
> But on issuing a swapoff command to swap stuff in again, the swap in
> is
> abysmally slow, just a few MiB/s (see below).
>
> I wonder why is that so? The SSD is basically idling around on
> swap-in.
Transaction granularity. Swapping out can queue up large batches of
pages because you can queue up more outgoing pages while the others are
still writing. Swapping _in_ you don't know what you need next until
you resume the process, so you fault in 4k, schedule DMA, resume the
process when it completes, fault on the next page, schedule more DMA,
rinse repeat. Programs don't really execute linearly, so you wind up
with round trip latency to and from device each time.
The problem with doing readahead on swapin is that programs jump around
randomly calling a function here and a function there, so you dunno
which other pages it'll need until it requests them. (Speculatively
faulting in pages when the system is starved of memory usually just
makes the memory shortage worse. This code only runs when there's a
shortage of physical pages.)
Having an ssd just exacerbates the problem, because with the device
itself sped up the round trip latency from all the tiny transactions
comes to dominate.
Rob-