Hi,
recently I noticed that direct IO causes memory leaks with
linux-2.6.0-test9.
The program that causes memory leaks is "fsstress", which is
testcases/kernel/fs/fsstress in ltp-full-20031106.tgz (ftp from
http://sourceforge.net/projects/ltp/).
fsstress does various file operations, and I found that the problem is
with the combination of write and dread (O_DIRECT read).
You should be able to reproduce the bug with the following command
line.
$ while true; do ./fsstress -c -d /usr/src/test -z -f write=1 \
-f dread=1 -f creat=1 -S -n 1000 -p 32; done
The test machine is a quad P3 machine with the following file systems.
/usr/src is an ext3 file system but mounted as an ext2 using mount -t ext2.
$ mount -v
/dev/rd/c0d0p2 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/rd/c0d0p5 on /usr/src type ext2 (rw)
I didn't see memory leaks with the other three combinations of
read+write (dread+dwrite, read+dwrite, and read+write).
Any ideas?
I'll try to debug this next week, but rather like to see the fix
in the meanwhile. :)
--
IWAMOTO Toshihiro
IWAMOTO Toshihiro <[email protected]> wrote:
>
> recently I noticed that direct IO causes memory leaks with
> linux-2.6.0-test9.
> The program that causes memory leaks is "fsstress", which is
> testcases/kernel/fs/fsstress in ltp-full-20031106.tgz (ftp from
> http://sourceforge.net/projects/ltp/).
>
> fsstress does various file operations, and I found that the problem is
> with the combination of write and dread (O_DIRECT read).
> You should be able to reproduce the bug with the following command
> line.
>
> $ while true; do ./fsstress -c -d /usr/src/test -z -f write=1 \
> -f dread=1 -f creat=1 -S -n 1000 -p 32; done
It seems OK here. Please take a copy of /proc/meminfo and /proc/slabinfo.
At Thu, 20 Nov 2003 23:17:49 -0800,
Andrew Morton wrote:
>
> IWAMOTO Toshihiro <[email protected]> wrote:
> >
> > recently I noticed that direct IO causes memory leaks with
> > linux-2.6.0-test9.
> > The program that causes memory leaks is "fsstress", which is
> > testcases/kernel/fs/fsstress in ltp-full-20031106.tgz (ftp from
> > http://sourceforge.net/projects/ltp/).
> >
> > fsstress does various file operations, and I found that the problem is
> > with the combination of write and dread (O_DIRECT read).
> > You should be able to reproduce the bug with the following command
> > line.
> >
> > $ while true; do ./fsstress -c -d /usr/src/test -z -f write=1 \
> > -f dread=1 -f creat=1 -S -n 1000 -p 32; done
>
> It seems OK here. Please take a copy of /proc/meminfo and /proc/slabinfo.
It'll take a while to leak a noticable amount of memory. So I reduced
the amount of memory using a boot option.
I'll try the same test on another machine.
Here they are. slabinfo is in the attachment.
$ cat /proc/meminfo
MemTotal: 254592 kB
MemFree: 3952 kB
Buffers: 688 kB
Cached: 6184 kB
SwapCached: 2512 kB
Active: 8616 kB
Inactive: 212056 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 254592 kB
LowFree: 3952 kB
SwapTotal: 2097136 kB
SwapFree: 2090436 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 4924 kB
Slab: 15200 kB
Committed_AS: 20580 kB
PageTables: 344 kB
VmallocTotal: 770040 kB
VmallocUsed: 9460 kB
VmallocChunk: 760580 kB
IWAMOTO Toshihiro <[email protected]> wrote:
>
> It'll take a while to leak a noticable amount of memory. So I reduced
> the amount of memory using a boot option.
Well I'll be darned. I took a new version of fsstress and it happens here
too. We're leaking anonymous memory. -mm doesn't do any better, either.
On Friday 21 November 2003 02:55, Andrew Morton wrote:
>IWAMOTO Toshihiro <[email protected]> wrote:
>> It'll take a while to leak a noticable amount of memory. So I
>> reduced the amount of memory using a boot option.
>
>Well I'll be darned. I took a new version of fsstress and it
> happens here too. We're leaking anonymous memory. -mm doesn't do
> any better, either.
Running 2.6.0-test9-mm4, default as scheduler
That triggerd me to go look at ksysguard, and I've got 70 megs out in
swap in less than 24 hours uptime with my normal loading. Usually it
takes me a couple of weeks to get that much as I've half a gig of
main memory. Its also showing about 95 megs free. Would this leak
show up there (ksysguard), and if so, in what section?
T'would be nice if xosview were to be made operable, but this kernel
breaks it. I used to keep it running in the corner of one of my
screens.
--
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz 512M
99.27% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
Gene Heskett <[email protected]> wrote:
>
> On Friday 21 November 2003 02:55, Andrew Morton wrote:
> >IWAMOTO Toshihiro <[email protected]> wrote:
> >> It'll take a while to leak a noticable amount of memory. So I
> >> reduced the amount of memory using a boot option.
> >
> >Well I'll be darned. I took a new version of fsstress and it
> > happens here too. We're leaking anonymous memory. -mm doesn't do
> > any better, either.
>
> Running 2.6.0-test9-mm4, default as scheduler
>
> That triggerd me to go look at ksysguard, and I've got 70 megs out in
> swap in less than 24 hours uptime with my normal loading. Usually it
> takes me a couple of weeks to get that much as I've half a gig of
> main memory.
That's good. The kernel has given you 70 megs more memory.
> Its also showing about 95 megs free.
free memory isn't really relevant. If there's plenty of `buffers' and
pagecache around then it's mostly reclaimable.
> Would this leak
> show up there (ksysguard), and if so, in what section?
I don't know.
> T'would be nice if xosview were to be made operable, but this kernel
> breaks it. I used to keep it running in the corner of one of my
> screens.
I had a patch for that. Maybe it got merged. You should hunt down the
upstream source and try it out.
For diagnosing this sort of thing it is best to learn to read the /proc
files.
On Friday 21 November 2003 03:40, Andrew Morton wrote:
>Gene Heskett <[email protected]> wrote:
>> On Friday 21 November 2003 02:55, Andrew Morton wrote:
>> >IWAMOTO Toshihiro <[email protected]> wrote:
>> >> It'll take a while to leak a noticable amount of memory. So I
>> >> reduced the amount of memory using a boot option.
>> >
>> >Well I'll be darned. I took a new version of fsstress and it
>> > happens here too. We're leaking anonymous memory. -mm doesn't
>> > do any better, either.
>>
>> Running 2.6.0-test9-mm4, default as scheduler
>>
>> That triggerd me to go look at ksysguard, and I've got 70 megs out
>> in swap in less than 24 hours uptime with my normal loading.
>> Usually it takes me a couple of weeks to get that much as I've
>> half a gig of main memory.
>
>That's good. The kernel has given you 70 megs more memory.
>
>> Its also showing about 95 megs free.
>
Now its down to about 80, and the display for cache and buffers has
shrunk considerably. Normal after boot memory usage is in the 125
meg area, and that portion of the display is now around 350 megs,
having grown 100megs or so overnight.
>free memory isn't really relevant. If there's plenty of `buffers'
> and pagecache around then it's mostly reclaimable.
Those two colors in the ksysguard report window are getting mighty
thin.
>> Would this leak
>> show up there (ksysguard), and if so, in what section?
>
>I don't know.
>
>> T'would be nice if xosview were to be made operable, but this
>> kernel breaks it. I used to keep it running in the corner of one
>> of my screens.
>
>I had a patch for that. Maybe it got merged. You should hunt down
> the upstream source and try it out.
The srcs for xosview? I did a freshmeat search, and what I found
hadn't been touched in over a year. I didn't bother grabbing it as
it was the same version number as the copy I have.
>For diagnosing this sort of thing it is best to learn to read the
> /proc files.
cat /proc/meminfo:
MemTotal: 514964 kB
MemFree: 171100 kB
Buffers: 14256 kB
Cached: 80548 kB
SwapCached: 15912 kB
Active: 220456 kB
Inactive: 5124 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 514964 kB
LowFree: 171100 kB
SwapTotal: 3857104 kB
SwapFree: 3786316 kB
Dirty: 52 kB
Writeback: 0 kB
Mapped: 181676 kB
Slab: 111528 kB
Committed_AS: 540836 kB
PageTables: 2144 kB
VmallocTotal: 515896 kB
VmallocUsed: 51372 kB
VmallocChunk: 464524 kB
But in the various memory related things in /proc, I don't see a
reference to anonymous?
--
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz 512M
99.27% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
On Friday 21 November 2003 09:02, Gene Heskett wrote:
[...]
>>I had a patch for that. Maybe it got merged. You should hunt down
>> the upstream source and try it out.
>
>The srcs for xosview? I did a freshmeat search, and what I found
>hadn't been touched in over a year. I didn't bother grabbing it as
>it was the same version number as the copy I have.
I must have followed the wrong link, 1.8.1 is building now.
--
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz 512M
99.27% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.
On Friday 21 November 2003 09:25, Gene Heskett wrote:
>On Friday 21 November 2003 09:02, Gene Heskett wrote:
>[...]
>
>>>I had a patch for that. Maybe it got merged. You should hunt
>>> down the upstream source and try it out.
>>
>>The srcs for xosview? I did a freshmeat search, and what I found
>>hadn't been touched in over a year. I didn't bother grabbing it as
>>it was the same version number as the copy I have.
>
>I must have followed the wrong link, 1.8.1 is building now.
But it won't build the memsat.o module when enabled. Many pages of
parse errors if I softlink the newest version number of it to
memstat.c. Otherwise it cannot find it at all. Not much meat in the
README's either. That isn't the one with the newest date however,
wth?
I'll try without it.
And that, when built, then './xosview' to run it insitu, never shows
its window, and never shows up in the process table. A ctrl-c
returns the prompt instantly. So I'm not going to install that
one...
--
Cheers, Gene
AMD K6-III@500mhz 320M
Athlon1600XP@1400mhz 512M
99.27% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2003 by Maurice Eugene Heskett, all rights reserved.