Sorry for the long email. I pruned it as much as possible...
The problem:
When we upgraded one of our production boxes (details below) to 2.6.6,
we noticed an immediate loss of 5 - 15 percent efficiency. While these
boxes usually had less than 0.5% variation through out the day, this box
was consistently doing 10% fewer searches than the others.
Upon investigation, we saw that the 2.6 box was reading from the disk
about 5 times as much as 2.4. Iin 2.4 we can almost completely saturate
the CPUs; they'll get to 90% of the real CPU's, and 15% of the virtual
CPUs. With 2.6, they never get above 60/10 because they are in io-wait
state constantly (which, under 2.4, is reported as idle IIRC). I have
not done extensive testing of the anticipatory elevator, but it did
appear slower than deadline in early tests.
The vmstat runs at the bottom of this email were done in parallel on two
machines, receiving mostly identical amounts of real traffic. Traffic is
load balanced by mod_backhand to these machines, and is directly
responsive to system load with a granularity of 2 seconds, so really,
the 2.6.6 box was actually getting somewhat *less* traffic. Notice how
much higher the 'bi' numbers are, for blocks in. As I said before,
expected variation is less tham 0.5%.
This behavior is consistent and has been observed for over a month in
production now. I'm just looking for reasons this is happening and maybe
what needs to be profiled/tuned/fixed in order to find out.
If you're still interested by this point, here is the Background:
We have a bunch of identical Dual P4 Xeon 2.8Ghz machines. Each has an
Intel i865 chipset, 1GB of DDR RAM, and 2xWDC 40GB 8MB cache drives. All
are running RedHat 8.0 with security patches from fedoralegacy.org. All
are running vanilla kernel 2.4.23 or later, except one, that runs 2.6.6.
The 2.6.6 kernel was built directly from one of the other kernel's
.config with 'make oldconfig'. Two new options were selected..
CONFIG_4KSTACKS, and (on the cmdline) elevator=deadline. The two disks
are setup in software RAID1. There is no swap configured.
These machines run text searches from web requests using a proprietary
file-based database (called Texis, http://www.thunderstone.com). The
data access patterns are generally "search through index files in a
tree-walking type of manner, then seek to data records in data files."
The index files are less than 300MB, and constantly accessed. The data
files total 3GB, but the data being read is very small... at most 40kB
at a time.
And now for the real details:
---------------------VMSTAT 2.4.23-------------------------
$ free -m ; uptime ; vmstat 5 5
total used free shared buffers
cached
Mem: 1007 983 24 0 12
812
-/+ buffers/cache: 157 850
Swap: 0 0 0
14:51:06 up 113 days, 22:22, 1 user, load average: 0.70, 0.59, 0.64
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 0 0 25284 13284 832500 0 0 1 1 1 1 12
2 86 0
1 0 0 20748 13312 832688 0 0 37 39 139 92 3
0 97 0
3 0 0 16720 13348 833020 0 0 63 80 189 222 12
2 86 0
0 0 0 23672 13376 833184 0 0 31 57 166 142 7
1 92 0
1 0 0 16572 13412 833288 0 0 20 51 155 137 5
1 93 0
------------------END VMSTAT 2.4.23-------------------------
---------------------VMSTAT 2.6.6-------------------------
$ free -m ; uptime ; vmstat 5 5
total used free shared buffers
cached
Mem: 1010 990 20 0 27
867
-/+ buffers/cache: 95 915
Swap: 0 0 0
14:51:05 up 7 days, 1:17, 1 user, load average: 0.59, 0.66, 0.76
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 1 0 20732 28220 888556 0 0 11 12 11 14 9
2 88 2
0 0 0 27552 28260 883416 0 0 223 107 198 232 19
3 76 2
0 0 0 26452 28276 884420 0 0 226 33 192 217 6
1 87 6
0 0 0 26388 28308 884660 0 0 36 56 154 136 4
0 95 0
0 0 0 25536 28344 885236 0 0 114 62 173 186 8
2 89 1
------------------END VMSTAT 2.6.6-------------------------
Clint Byrum <[email protected]> wrote:
>
> When we upgraded one of our production boxes (details below) to 2.6.6,
> we noticed an immediate loss of 5 - 15 percent efficiency. While these
> boxes usually had less than 0.5% variation through out the day, this box
> was consistently doing 10% fewer searches than the others.
>
> Upon investigation, we saw that the 2.6 box was reading from the disk
> about 5 times as much as 2.4. Iin 2.4 we can almost completely saturate
> the CPUs; they'll get to 90% of the real CPU's, and 15% of the virtual
> CPUs. With 2.6, they never get above 60/10 because they are in io-wait
> state constantly (which, under 2.4, is reported as idle IIRC).
Possibly a memory zone problem. Could you try booting with "mem=896m" on
the kernel command line, see how that affects things?
> Upon investigation, we saw that the 2.6 box was reading from the disk
> about 5 times as much as 2.4.
I don't think this will account for the entire change in disk activity,
but 2.6.7-pre? contains a fix for the read ahead code to prevent it from
reading extra sectors when unneeded.
The fix applies to 'seeky database type loads' which...
> The data access patterns are generally "search through index files in
> a tree-walking type of manner, then seek to data records in data
> files."
...sounds like it may apply to you.
So, if you and your server have some time, you might try 2.6.7rc3 and
see if it changes any of the numbers.
Ray Lee
On Wednesday, June 9, 2004, at 06:03 PM, Ray Lee wrote:
>> Upon investigation, we saw that the 2.6 box was reading from the disk
>> about 5 times as much as 2.4.
>
> I don't think this will account for the entire change in disk activity,
> but 2.6.7-pre? contains a fix for the read ahead code to prevent it
> from
> reading extra sectors when unneeded.
>
> The fix applies to 'seeky database type loads' which...
>
>> The data access patterns are generally "search through index files in
>> a tree-walking type of manner, then seek to data records in data
>> files."
>
> ...sounds like it may apply to you.
>
> So, if you and your server have some time, you might try 2.6.7rc3 and
> see if it changes any of the numbers.
>
I updated my 2.6 box to 2.6.7-rc3 a few hours ago. While its not really
fair to pass judgement during such low-traffic times, things look about
the same. Where i see the 2.4 box not even touching the disks for a
minute or longer, the 2.6 box will hit it every 10 seconds at least,
and hit it with 40 blocks or more. 3 hours should be plenty of time to
get the indexes and data files mostly cached.
It almost seems to me like 2.6 is caching too much with each read, and
therefore having to free other pages that really should have been left
alone. This might even explain why 2.4 is maintaining a lot more free
memory (75-80MB versus 2.6 having only 15-20MB free) and less cache.
Oddly enough, I noticed that the blockdev command reports 256 as the
readahead in 2.6.x, but 1024 in 2.4.x. I tried mucking with that value
but it didn't make a whole lot of difference.
Might it help to have some swap available, if for nothing else but to
make the algorithms work better? Really the box should never use it;
there are no daemons that go unused for longer than 10 minutes ... and
when I have turned on swap in the past, it uses less than 2MB of it.
This machine, for all intents and purposes, should spend most of its
time searching an index and database that are already cached.. 90% of
the searches are on similar terms.
That brings up an interesting point... is there a system wide stat that
tells me how effective the file cache is? I guess majfaults/s fits that
bill to some degree.
On Wed, Jun 09, 2004 at 11:03:38PM -0700, Clint Byrum wrote:
> That brings up an interesting point... is there a system wide stat that
> tells me how effective the file cache is? I guess majfaults/s fits that
> bill to some degree.
/proc/vmstat should log global major/minor fault counters (actually
summed on the fly per-cpu counters). I fixed those to properly report
major and minor faults for 2.6. The analogous numbers where they are
present are completely and utterly meaningless gibberish in 2.4.
-- wli
On Wed, 2004-06-09 at 16:25, Andrew Morton wrote:
> Possibly a memory zone problem. Could you try booting with "mem=896m" on
> the kernel command line, see how that affects things?
This took a while longer to try, as I didn't want to unfairly test it
against a box with 1G of RAM. So I rebooted my 2.4.23 and 2.6.7-rc3 test
boxes with mem=896m. No change in the 5:1 ratio when comparing 2.6's
disk reads to 2.4's. Of course, both boxes ended up reading from the
disk more often, as they had less RAM for cache. I was unable to run a
long test as I did before, but I'm confident the 3 hours I did run tests
for show that this isn't a memory zone problem.
I still think this behavior is happening because useful pages are being
removed from the page cache too soon. Maybe this is happening because of
excessive readahead?
-cb