When OOM happen, will dump buddy free areas info, hugetlb pages info,
memory state of all eligible tasks, per-cpu memory info.
But do not dump slab/vmalloc info, sometime, it's not enough to figure out the
reason OOM happened.
So, my questions are:
1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file,
but usually we do not monitor the logs and check proc file immediately when OOM happened.
2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped?
3. Without these info, usually how to figure out OOM reason?
On Mon, 20 Jan 2014, Jianguo Wu wrote:
> When OOM happen, will dump buddy free areas info, hugetlb pages info,
> memory state of all eligible tasks, per-cpu memory info.
> But do not dump slab/vmalloc info, sometime, it's not enough to figure out the
> reason OOM happened.
>
> So, my questions are:
> 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file,
> but usually we do not monitor the logs and check proc file immediately when OOM happened.
>
The problem is that slabinfo becomes excessively verbose and dumping it
all to the kernel log often times causes important messages to be lost.
This is why we control things like the tasklist dump with a VM sysctl. It
would be possible to dump, say, the top ten slab caches with the highest
memory usage, but it will only be helpful for slab leaks. Typically there
are better debugging tools available than analyzing the kernel log; if you
see unusually high slab memory in the meminfo dump, you can enable it.
> 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped?
>
Also very verbose and would cause important messages to be lost, we try to
avoid spamming the kernel log with all of this information as much as
possible.
> 3. Without these info, usually how to figure out OOM reason?
>
Analyze the memory usage in the meminfo and determine what is unusually
high; if it's mostly anonymous memory, you can usually correlate it back
to a high rss for a process in the tasklist that you didn't suspect to be
using that much memory, for example.
On 2014/1/21 13:34, David Rientjes wrote:
> On Mon, 20 Jan 2014, Jianguo Wu wrote:
>
>> When OOM happen, will dump buddy free areas info, hugetlb pages info,
>> memory state of all eligible tasks, per-cpu memory info.
>> But do not dump slab/vmalloc info, sometime, it's not enough to figure out the
>> reason OOM happened.
>>
>> So, my questions are:
>> 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file,
>> but usually we do not monitor the logs and check proc file immediately when OOM happened.
>>
>
Hi David,
Thank you for your patience to answer!
> The problem is that slabinfo becomes excessively verbose and dumping it
> all to the kernel log often times causes important messages to be lost.
> This is why we control things like the tasklist dump with a VM sysctl. It
> would be possible to dump, say, the top ten slab caches with the highest
> memory usage, but it will only be helpful for slab leaks. Typically there
> are better debugging tools available than analyzing the kernel log; if you
> see unusually high slab memory in the meminfo dump, you can enable it.
>
But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc
is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys?
Thanks.
>> 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped?
>>
>
> Also very verbose and would cause important messages to be lost, we try to
> avoid spamming the kernel log with all of this information as much as
> possible.
>
>> 3. Without these info, usually how to figure out OOM reason?
>>
>
> Analyze the memory usage in the meminfo and determine what is unusually
> high; if it's mostly anonymous memory, you can usually correlate it back
> to a high rss for a process in the tasklist that you didn't suspect to be
> using that much memory, for example.
>
>
On Tue, 21 Jan 2014, Jianguo Wu wrote:
> > The problem is that slabinfo becomes excessively verbose and dumping it
> > all to the kernel log often times causes important messages to be lost.
> > This is why we control things like the tasklist dump with a VM sysctl. It
> > would be possible to dump, say, the top ten slab caches with the highest
> > memory usage, but it will only be helpful for slab leaks. Typically there
> > are better debugging tools available than analyzing the kernel log; if you
> > see unusually high slab memory in the meminfo dump, you can enable it.
> >
>
> But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc
> is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys?
>
You could, but it's a tradeoff between how much to dump to a general
resource such as the kernel log and how many sysctls we add that control
every possible thing. Slab leaks would definitely be a minority of oom
conditions and you should normally be able to reproduce them by running
the same workload; just use slabtop(1) or manually inspect /proc/slabinfo
while such a workload is running for indicators. I don't think we want to
add the information by default, though, nor do we want to add sysctls to
control the behavior (you'd still need to reproduce the issue after
enabling it).
We are currently discussing userspace oom handlers, though, that would
allow you to run a process that would be notified and allowed to allocate
a small amount of memory on oom conditions. It would then be trivial to
dump any information you feel pertinent in userspace prior to killing
something. I like to inspect heap profiles for memory hogs while
debugging our malloc() issues, for example, and you could look more
closely at kernel memory.
I'll cc you on future discussions of that feature.
On 2014/1/22 4:41, David Rientjes wrote:
> On Tue, 21 Jan 2014, Jianguo Wu wrote:
>
>>> The problem is that slabinfo becomes excessively verbose and dumping it
>>> all to the kernel log often times causes important messages to be lost.
>>> This is why we control things like the tasklist dump with a VM sysctl. It
>>> would be possible to dump, say, the top ten slab caches with the highest
>>> memory usage, but it will only be helpful for slab leaks. Typically there
>>> are better debugging tools available than analyzing the kernel log; if you
>>> see unusually high slab memory in the meminfo dump, you can enable it.
>>>
>>
>> But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc
>> is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys?
>>
>
> You could, but it's a tradeoff between how much to dump to a general
> resource such as the kernel log and how many sysctls we add that control
> every possible thing. Slab leaks would definitely be a minority of oom
> conditions and you should normally be able to reproduce them by running
> the same workload; just use slabtop(1) or manually inspect /proc/slabinfo
> while such a workload is running for indicators. I don't think we want to
> add the information by default, though, nor do we want to add sysctls to
> control the behavior (you'd still need to reproduce the issue after
> enabling it).
>
> We are currently discussing userspace oom handlers, though, that would
> allow you to run a process that would be notified and allowed to allocate
> a small amount of memory on oom conditions. It would then be trivial to
> dump any information you feel pertinent in userspace prior to killing
> something. I like to inspect heap profiles for memory hogs while
> debugging our malloc() issues, for example, and you could look more
> closely at kernel memory.
>
> I'll cc you on future discussions of that feature.
>
Hi David,
Thanks for your kindly explanation, do you have any specific plans on this?
Thanks,
Jianguo Wu.
>
On Tue, 11 Feb 2014, Jianguo Wu wrote:
> Thanks for your kindly explanation, do you have any specific plans on this?
>
We're going to be discussing it at the LSF/mm conference at the end of
March.