Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH 00/10] OOM Debug print selection and additional
 information
From:   Qian Cai <cai@lca.pw>
In-Reply-To: <CAM3twVSdxJaEpmWXu2m_F1MxFMB58C6=LWWCDYNn5yT3Ns+0sQ@mail.gmail.com>
Date:   Tue, 27 Aug 2019 21:32:41 -0400
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        Michal Hocko <mhocko@suse.com>, Roman Gushchin <guro@fb.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        David Rientjes <rientjes@google.com>,
        Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
        Shakeel Butt <shakeelb@google.com>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, Ivan Delalande <colona@arista.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <2A1D8FFC-9E9E-4D86-9A0E-28F8263CC508@lca.pw>
References: <20190826193638.6638-1-echron@arista.com>
 <1566909632.5576.14.camel@lca.pw>
 <CAM3twVQEMGWMQEC0dduri0JWt3gH6F2YsSqOmk55VQz+CZDVKg@mail.gmail.com>
 <79FC3DA1-47F0-4FFC-A92B-9A7EBCE3F15F@lca.pw>
 <CAM3twVSdxJaEpmWXu2m_F1MxFMB58C6=LWWCDYNn5yT3Ns+0sQ@mail.gmail.com>
To:     Edward Chron <echron@arista.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Aug 27, 2019, at 9:13 PM, Edward Chron <echron@arista.com> wrote:
>=20
> On Tue, Aug 27, 2019 at 5:50 PM Qian Cai <cai@lca.pw> wrote:
>>=20
>>=20
>>=20
>>> On Aug 27, 2019, at 8:23 PM, Edward Chron <echron@arista.com> wrote:
>>>=20
>>>=20
>>>=20
>>> On Tue, Aug 27, 2019 at 5:40 AM Qian Cai <cai@lca.pw> wrote:
>>> On Mon, 2019-08-26 at 12:36 -0700, Edward Chron wrote:
>>>> This patch series provides code that works as a debug option =
through
>>>> debugfs to provide additional controls to limit how much =
information
>>>> gets printed when an OOM event occurs and or optionally print =
additional
>>>> information about slab usage, vmalloc allocations, user process =
memory
>>>> usage, the number of processes / tasks and some summary information
>>>> about these tasks (number runable, i/o wait), system information
>>>> (#CPUs, Kernel Version and other useful state of the system),
>>>> ARP and ND Cache entry information.
>>>>=20
>>>> Linux OOM can optionally provide a lot of information, what's =
missing?
>>>> =
----------------------------------------------------------------------
>>>> Linux provides a variety of detailed information when an OOM event =
occurs
>>>> but has limited options to control how much output is produced. The
>>>> system related information is produced unconditionally and limited =
per
>>>> user process information is produced as a default enabled option. =
The
>>>> per user process information may be disabled.
>>>>=20
>>>> Slab usage information was recently added and is output only if =
slab
>>>> usage exceeds user memory usage.
>>>>=20
>>>> Many OOM events are due to user application memory usage sometimes =
in
>>>> combination with the use of kernel resource usage that exceeds what =
is
>>>> expected memory usage. Detailed information about how memory was =
being
>>>> used when the event occurred may be required to identify the root =
cause
>>>> of the OOM event.
>>>>=20
>>>> However, some environments are very large and printing all of the
>>>> information about processes, slabs and or vmalloc allocations may
>>>> not be feasible. For other environments printing as much =
information
>>>> about these as possible may be needed to root cause OOM events.
>>>>=20
>>>=20
>>> For more in-depth analysis of OOM events, people could use kdump to =
save a
>>> vmcore by setting "panic_on_oom", and then use the crash utility to =
analysis the
>>> vmcore which contains pretty much all the information you need.
>>>=20
>>> Certainly, this is the ideal. A full system dump would give you the =
maximum amount of
>>> information.
>>>=20
>>> Unfortunately some environments may lack space to store the dump,
>>=20
>> Kdump usually also support dumping to a remote target via NFS, SSH =
etc
>>=20
>>> let alone the time to dump the storage contents and restart the =
system. Some
>>=20
>> There is also =E2=80=9Cmakedumpfile=E2=80=9D that could compress and =
filter unwanted memory to reduce
>> the vmcore size and speed up the dumping process by utilizing =
multi-threads.
>>=20
>>> systems can take many minutes to fully boot up, to reset and =
reinitialize all the
>>> devices. So unfortunately this is not always an option, and we need =
an OOM Report.
>>=20
>> I am not sure how the system needs some minutes to reboot would be =
relevant  for the
>> discussion here. The idea is to save a vmcore and it can be analyzed =
offline even on
>> another system as long as it having a matching =E2=80=9Cvmlinux.".
>>=20
>>=20
>=20
> If selecting a dump on an OOM event doesn't reboot the system and if
> it runs fast enough such
> that it doesn't slow processing enough to appreciably effect the
> system's responsiveness then
> then it would be ideal solution. For some it would be over kill but
> since it is an option it is a
> choice to consider or not.

It sounds like you are looking for more of this,

https://github.com/iovisor/bcc/blob/master/tools/oomkill.py