Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <cover.1599072725.git.nachukannan@gmail.com>
In-Reply-To: <cover.1599072725.git.nachukannan@gmail.com>
From:   Joel Fernandes <joel@joelfernandes.org>
Date:   Wed, 2 Sep 2020 17:47:44 -0400
Message-ID: <CAEXW_YSt49yGH+Wc63zfm4i7n1M_YxETbAA8nxmfDH1B3WFQXg@mail.gmail.com>
Subject: Re: [RFC PATCH 0/7] Trace events to pstore
To:     Nachammai Karuppiah <nachukannan@gmail.com>
Cc:     Steven Rostedt <rostedt@goodmis.org>,
        Ingo Molnar <mingo@redhat.com>,
        Rob Herring <robh+dt@kernel.org>,
        Frank Rowand <frowand.list@gmail.com>,
        Kees Cook <keescook@chromium.org>,
        Anton Vorontsov <anton@enomsg.org>,
        Colin Cross <ccross@android.com>,
        Tony Luck <tony.luck@intel.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org>,
        computersforpeace@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Sep 2, 2020 at 4:01 PM Nachammai Karuppiah
<nachukannan@gmail.com> wrote:
>
> Hi,
>
> This patch series adds support to store trace events in pstore.
>
> Storing trace entries in persistent RAM would help in understanding what
> happened just before the system went down. The trace events that led to t=
he
> crash can be retrieved from the pstore after a warm reboot. This will hel=
p
> debug what happened before machine=E2=80=99s last breath. This has to be =
done in a
> scalable way so that tracing a live system does not impact the performanc=
e
> of the system.

Just to add, Nachammai was my intern in the recent outreachy program
and we designed together a way for trace events to be written to
pstore backed memory directory instead of regular memory. The basic
idea is to allocate frace's ring buffer on pstore memory and have it
right there. Then recover it on reboot. Nachammai wrote the code with
some guidance :) . I talked to Steve as well in the past about the
basic of idea of this. Steve is on vacation this week though.

This is similar to what +Sai Prakash Ranjan was trying to do sometime
ago: https://lkml.org/lkml/2018/9/8/221 . But that approach involved
higher overhead due to synchronization of writing to the otherwise
lockless ring buffer.

+Brian Norris has also expressed interest for this feature.

thanks,

 - Joel

>
> This requires a new backend - ramtrace that allocates pages from
> persistent storage for the tracing utility. This feature can be enabled
> using TRACE_EVENTS_TO_PSTORE.
> In this feature, the new backend is used only as a page allocator and
> once the  users chooses to use pstore to record trace entries, the ring
> buffer pages are freed and allocated in pstore. Once this switch is done,
> ring_buffer continues to operate just as before without much overhead.
> Since the ring buffer uses the persistent RAM buffer directly to record
> trace entries, all tracers would also persist across reboot.
>
> To test this feature, I used a simple module that would call panic during
> a write operation to file in tracefs directory. Before writing to the fil=
e,
> the ring buffer is moved to persistent RAM buffer through command line
> as shown below,
>
> $echo 1 > /sys/kernel/tracing/options/persist
>
> Writing to the file,
> $echo 1 > /sys/kernel/tracing/crash/panic_on_write
>
> The above write operation results in system crash. After reboot, once the
> pstore is mounted, the trace entries from previous boot are available in =
file,
> /sys/fs/pstore/trace-ramtrace-0
>
> Looking through this file, gives us the stack trace that led to the crash=
.
>
>            <...>-1     [001] ....    49.083909: __vfs_write <-vfs_write
>            <...>-1     [001] ....    49.083933: panic <-panic_on_write
>            <...>-1     [001] d...    49.084195: printk <-panic
>            <...>-1     [001] d...    49.084201: vprintk_func <-printk
>            <...>-1     [001] d...    49.084207: vprintk_default <-printk
>            <...>-1     [001] d...    49.084211: vprintk_emit <-printk
>            <...>-1     [001] d...    49.084216: __printk_safe_enter <-vpr=
intk_emit
>            <...>-1     [001] d...    49.084219: _raw_spin_lock <-vprintk_=
emit
>            <...>-1     [001] d...    49.084223: vprintk_store <-vprintk_e=
mit
>
> Patchwise oneline description is given below:
>
> Patch 1 adds support to allocate ring buffer pages from persistent RAM bu=
ffer.
>
> Patch 2 introduces a new backend, ramtrace.
>
> Patch 3 adds methods to read previous boot pages from pstore.
>
> Patch 4 adds the functionality to allocate page-sized memory from pstore.
>
> Patch 5 adds the seq_operation methods to iterate through trace entries.
>
> Patch 6 modifies ring_buffer to allocate from ramtrace when pstore is use=
d.
>
> Patch 7 adds ramtrace DT node as child-node of /reserved-memory.
>
> Nachammai Karuppiah (7):
>   tracing: Add support to allocate pages from persistent memory
>   pstore: Support a new backend, ramtrace
>   pstore: Read and iterate through trace entries in PSTORE
>   pstore: Allocate and free page-sized memory in persistent RAM buffer
>   tracing: Add support to iterate through pages retrieved from pstore
>   tracing: Use ramtrace alloc and free methods while using persistent
>     RAM
>   dt-bindings: ramtrace: Add ramtrace DT node
>
>  .../bindings/reserved-memory/ramtrace.txt          |  13 +
>  drivers/of/platform.c                              |   1 +
>  fs/pstore/Makefile                                 |   2 +
>  fs/pstore/inode.c                                  |  46 +-
>  fs/pstore/platform.c                               |   1 +
>  fs/pstore/ramtrace.c                               | 821 +++++++++++++++=
++++++
>  include/linux/pstore.h                             |   3 +
>  include/linux/ramtrace.h                           |  28 +
>  include/linux/ring_buffer.h                        |  19 +
>  include/linux/trace.h                              |  13 +
>  kernel/trace/Kconfig                               |  10 +
>  kernel/trace/ring_buffer.c                         | 663 +++++++++++++++=
+-
>  kernel/trace/trace.c                               | 312 +++++++-
>  kernel/trace/trace.h                               |   5 +-
>  14 files changed, 1924 insertions(+), 13 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/reserved-memory/ram=
trace.txt
>  create mode 100644 fs/pstore/ramtrace.c
>  create mode 100644 include/linux/ramtrace.h
>
> --
> 2.7.4
>