Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH 00/10] OOM Debug print selection and additional
 information
From:   Qian Cai <cai@lca.pw>
In-Reply-To: <CAM3twVQEMGWMQEC0dduri0JWt3gH6F2YsSqOmk55VQz+CZDVKg@mail.gmail.com>
Date:   Tue, 27 Aug 2019 20:50:00 -0400
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        Michal Hocko <mhocko@suse.com>, Roman Gushchin <guro@fb.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        David Rientjes <rientjes@google.com>,
        Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
        Shakeel Butt <shakeelb@google.com>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, Ivan Delalande <colona@arista.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <79FC3DA1-47F0-4FFC-A92B-9A7EBCE3F15F@lca.pw>
References: <20190826193638.6638-1-echron@arista.com>
 <1566909632.5576.14.camel@lca.pw>
 <CAM3twVQEMGWMQEC0dduri0JWt3gH6F2YsSqOmk55VQz+CZDVKg@mail.gmail.com>
To:     Edward Chron <echron@arista.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Aug 27, 2019, at 8:23 PM, Edward Chron <echron@arista.com> wrote:
>=20
>=20
>=20
> On Tue, Aug 27, 2019 at 5:40 AM Qian Cai <cai@lca.pw> wrote:
> On Mon, 2019-08-26 at 12:36 -0700, Edward Chron wrote:
> > This patch series provides code that works as a debug option through
> > debugfs to provide additional controls to limit how much information
> > gets printed when an OOM event occurs and or optionally print =
additional
> > information about slab usage, vmalloc allocations, user process =
memory
> > usage, the number of processes / tasks and some summary information
> > about these tasks (number runable, i/o wait), system information
> > (#CPUs, Kernel Version and other useful state of the system),
> > ARP and ND Cache entry information.
> >=20
> > Linux OOM can optionally provide a lot of information, what's =
missing?
> > =
----------------------------------------------------------------------
> > Linux provides a variety of detailed information when an OOM event =
occurs
> > but has limited options to control how much output is produced. The
> > system related information is produced unconditionally and limited =
per
> > user process information is produced as a default enabled option. =
The
> > per user process information may be disabled.
> >=20
> > Slab usage information was recently added and is output only if slab
> > usage exceeds user memory usage.
> >=20
> > Many OOM events are due to user application memory usage sometimes =
in
> > combination with the use of kernel resource usage that exceeds what =
is
> > expected memory usage. Detailed information about how memory was =
being
> > used when the event occurred may be required to identify the root =
cause
> > of the OOM event.
> >=20
> > However, some environments are very large and printing all of the
> > information about processes, slabs and or vmalloc allocations may
> > not be feasible. For other environments printing as much information
> > about these as possible may be needed to root cause OOM events.
> >=20
>=20
> For more in-depth analysis of OOM events, people could use kdump to =
save a
> vmcore by setting "panic_on_oom", and then use the crash utility to =
analysis the
>  vmcore which contains pretty much all the information you need.
>=20
> Certainly, this is the ideal. A full system dump would give you the =
maximum amount of
> information.=20
>=20
> Unfortunately some environments may lack space to store the dump,

Kdump usually also support dumping to a remote target via NFS, SSH etc=20=


> let alone the time to dump the storage contents and restart the =
system. Some

There is also =E2=80=9Cmakedumpfile=E2=80=9D that could compress and =
filter unwanted memory to reduce
the vmcore size and speed up the dumping process by utilizing =
multi-threads.

> systems can take many minutes to fully boot up, to reset and =
reinitialize all the
> devices. So unfortunately this is not always an option, and we need an =
OOM Report.

I am not sure how the system needs some minutes to reboot would be =
relevant  for the
discussion here. The idea is to save a vmcore and it can be analyzed =
offline even on=20
another system as long as it having a matching =E2=80=9Cvmlinux.".=20