Hi,
The setup:
is 16-core opteron node, diskless with NFS root, swapless, 64GB of RAM. Operating under OpenSUSE 11.2. With kernel version 2.6.31. Although it isn't vanilla, I think probably more right is to submit this into LKML.
The problem:
On this node user run MPI job with 16 processes, local job by using shared memory communication.
At some point this processes are trying to use more memory that available.
Normally, all of them or part of them would be killed by OOM killer, and it use to work for years over many versions of kernel.
Now, with fresh setup I got something new. OOM tried to kill, but didn't succeed, and even more, brought system in unusable state. All those processes are locked and un-killable. some of other processes are also locked and un-killable/inaccessible. kswapd consume 100% CPU (which I think is expected behavior when there is no free memory).
No free memory obviously, cause all original processes are still in memory.
I tried to test OOM behavior and it always happens like that now.
Here I attach full gzipped log of all related information captured by logserver (sent by logserver and netconsole, so it can be partly doubled). Sorry that it is too big, but I didn't know what information can be important.
Anton.
On Mon, 15 Feb 2010 00:43:02 +0100
Anton Starikov <[email protected]> wrote:
> Hi,
>
> The setup:
> is 16-core opteron node, diskless with NFS root, swapless, 64GB of RAM. Operating under OpenSUSE 11.2. With kernel version 2.6.31. Although it isn't vanilla, I think probably more right is to submit this into LKML.
>
At first, what is the version of kernel you are comparing with ? 2.6.22?(If OpenSuse10)
If so, many changes since that..
> The problem:
> On this node user run MPI job with 16 processes, local job by using shared memory communication.
> At some point this processes are trying to use more memory that available.
> Normally, all of them or part of them would be killed by OOM killer, and it use to work for years over many versions of kernel.
>
> Now, with fresh setup I got something new. OOM tried to kill, but didn't succeed, and even more, brought system in unusable state. All those processes are locked and un-killable. some of other processes are also locked and un-killable/inaccessible. kswapd consume 100% CPU (which I think is expected behavior when there is no free memory).
> No free memory obviously, cause all original processes are still in memory.
>
> I tried to test OOM behavior and it always happens like that now.
>
> Here I attach full gzipped log of all related information captured by logserver (sent by logserver and netconsole, so it can be partly doubled). Sorry that it is too big, but I didn't know what information can be important.
>
Anyway, I think it's not appreciated to depend on OOM-Kill on swapless-system.
I recommend you to use cgroup "memory" to encapsulate your apps (but please check
the performance regression can be seen or not..)
Thanks,
-Kame
On Feb 15, 2010, at 2:19 AM, KAMEZAWA Hiroyuki wrote:
> At first, what is the version of kernel you are comparing with ? 2.6.22?(If OpenSuse10)
> If so, many changes since that..
Latest kernel version where OOM killer worked as it should in our setup was 2.6.29.
> Anyway, I think it's not appreciated to depend on OOM-Kill on swapless-system.
> I recommend you to use cgroup "memory" to encapsulate your apps (but please check
> the performance regression can be seen or not..)
OK, I will check it.
Thanks,
Anton.-