Hello list,
on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a
backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local
ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card).
Even though earlier system load was minimal, free memory was plenty, the
system now is unresponsive and is thrashing the disk, but the swapfile
is rarely touched.
I managed to get some information by renicing a root console to -20, but
even then each keypress showed with a minimum 10s delay! In the attached
file you can see the dmesg output, SysRq+{W,T}, ps, vmstat, slabs,
meminfo.
I think I'm seeing paging of executable pages because ext4_inode_cache
is aggressively using all memory, evicting other pages. However under no
condition should realtime processes be unresponsive. What do you think?
Please note that I've set vm.swappiness to 0 and gradually increased
vm.vfs_cache_pressure to 1000000, but see no difference.
Thanks in advance,
Dimitris
P.S. Please CC me in all replies
On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote:
> on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a
> backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local
> ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card).
> Even though earlier system load was minimal, free memory was plenty, the
> system now is unresponsive and is thrashing the disk, but the swapfile
> is rarely touched.
I'm now having the same experience even though I replaced xz (which
needed ~50MB RAM) with gzip. Even though I feel the realtime root shell
is a bit more responsive than before, the OOM killer is out killing
small processes like syslog-ng and systemd-logind... The
ext4_inode_cache slab is taking almost all my memory (117MB). Please
advise!
Thanks,
Dimitris
Hello,
I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost
within 5 minutes of starting the backup process, I got:
[ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure
[ 486.887195] kmemleak: Automatic memory scanning thread ended
[ 486.906604] kmemleak: Kernel memory leak detector disabled
Any ideas?
Thanks,
Dimitris
On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote:
> On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote:
> > on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a
> > backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local
> > ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card).
> > Even though earlier system load was minimal, free memory was plenty, the
> > system now is unresponsive and is thrashing the disk, but the swapfile
> > is rarely touched.
>
> I'm now having the same experience even though I replaced xz (which
> needed ~50MB RAM) with gzip. Even though I feel the realtime root shell
> is a bit more responsive than before, the OOM killer is out killing
> small processes like syslog-ng and systemd-logind... The
> ext4_inode_cache slab is taking almost all my memory (117MB). Please
> advise!
Hello Dimitrios,
I would try a 2.6.27.* kernel, for following reasons:
(1) Kernel development since 2.6.27 achieved significant performance
improvements at the cost of exploding memory consumption by the kernel for
_internal_ data structures. I am currently using a 3.2.34 kernel on a Notebook
with 4 G RAM. 0,5 … 1 G RAM is usually occupied just by kernel slab [1]; this
memory cannot be swapped, it cannot be released by other means than rebooting,
and there seems to be _no_ adjustment to memory pressure. I am surprised,
that you have managed to boot a 3.6.* kernel at all with only 128 M RAM.
(2) At the time, when I used a PIII-Notebook, kernels 2.6.27 to 2.6.29 where
current. Thus chances are good, that a 2.6.27.* kernel will support chipset,
PCI bus and devices of your notebook. 2.6.27 got longterm maintainance, the
latest release in the linux-stable git repository is 2.6.27.62.
So Q @ LKML community:
Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
[1] Picture described in my LKML message
Date: Fri, 20 Jan 2012 01:08:00 +0100
Subject: Re: [kmemleak report 1/2] kernel 3.1.6, x86_64: mm, xfs ?, vfs ?
remained the same with _every_ 3.1.* and 3.2.* kernel tried so far.
--
Roland
On Mon, 2012-11-26 at 00:41 +0200, Dimitrios Apostolou wrote:
> Hello,
>
> I tried running a kmemleak and DEBUG_SLUB enabled kernel, but almost
> within 5 minutes of starting the backup process, I got:
>
> [ 486.863124] kmemleak: Cannot allocate a kmemleak_object structure
> [ 486.887195] kmemleak: Automatic memory scanning thread ended
> [ 486.906604] kmemleak: Kernel memory leak detector disabled
Important detail: I decided to try kmemleak after suspecting a leak
because 'echo 3>/proc/sys/vm/drop_caches' left about 90MB out of 118MB
of ext4_inode_cache, and I found no way to get rid of that (via
remounting filesystems for example).
> Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
Probably not a good idea, there are known exploitable holes in 2.6.27 era
kernels and nobody maintains anything that old.
I guess RHEL/Centos might work for you.
I've not had any problems with leaks in 3.6 but there have been a few
reports and its clear that some obscure configurations trigger something
bad.
Gnome 3 on the other hand leaks like a sieve.
On 2012-11-25 Sunday at 23:56 +0000 Alan Cox wrote:
> > Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
>
> Probably not a good idea, there are known exploitable holes in 2.6.27 era
> kernels and nobody maintains anything that old.
“old” is relative …
cd git/linux-stable && git log -1 --date=iso v2.6.27.62
........................................................
commit bc4e1a77b06519a01e7aed1125695598e27ddeb2
Author: Willy Tarreau <[email protected]>
Date: 2012-03-17 14:03:53 +0100
Linux 2.6.27.62
Signed-off-by: Willy Tarreau <[email protected]>
… and surely less exploitable than a system suffering oom-killer actions.
Google "linux-2.6.27 download" gives some Ubuntu hits …
> I guess RHEL/Centos might work for you.
>
> I've not had any problems with leaks in 3.6 but there have been a few
> reports and its clear that some obscure configurations trigger something
> bad.
>
> Gnome 3 on the other hand leaks like a sieve.
For a system with 128 M total RAM, Gnome is obviously off topic.
--
Roland