Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Nachammai Karuppiah <nachukannan@gmail.com>
To:     Steven Rostedt <rostedt@goodmis.org>,
        Ingo Molnar <mingo@redhat.com>,
        Rob Herring <robh+dt@kernel.org>,
        Frank Rowand <frowand.list@gmail.com>,
        Kees Cook <keescook@chromium.org>,
        Anton Vorontsov <anton@enomsg.org>,
        Colin Cross <ccross@android.com>,
        Tony Luck <tony.luck@intel.com>
Cc:     joel@joelfernandes.org, linux-kernel@vger.kernel.org,
        Nachammai Karuppiah <nachukannan@gmail.com>
Subject: [RFC PATCH 0/7] Trace events to pstore
Date:   Wed,  2 Sep 2020 13:00:15 -0700
Message-Id: <cover.1599072725.git.nachukannan@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi,

This patch series adds support to store trace events in pstore.

Storing trace entries in persistent RAM would help in understanding what 
happened just before the system went down. The trace events that led to the 
crash can be retrieved from the pstore after a warm reboot. This will help 
debug what happened before machine’s last breath. This has to be done in a 
scalable way so that tracing a live system does not impact the performance 
of the system.

This requires a new backend - ramtrace that allocates pages from 
persistent storage for the tracing utility. This feature can be enabled
using TRACE_EVENTS_TO_PSTORE.
In this feature, the new backend is used only as a page allocator and 
once the  users chooses to use pstore to record trace entries, the ring 
buffer pages are freed and allocated in pstore. Once this switch is done,
ring_buffer continues to operate just as before without much overhead.
Since the ring buffer uses the persistent RAM buffer directly to record 
trace entries, all tracers would also persist across reboot.

To test this feature, I used a simple module that would call panic during 
a write operation to file in tracefs directory. Before writing to the file, 
the ring buffer is moved to persistent RAM buffer through command line 
as shown below,

$echo 1 > /sys/kernel/tracing/options/persist 

Writing to the file,
$echo 1 > /sys/kernel/tracing/crash/panic_on_write

The above write operation results in system crash. After reboot, once the
pstore is mounted, the trace entries from previous boot are available in file,
/sys/fs/pstore/trace-ramtrace-0

Looking through this file, gives us the stack trace that led to the crash. 

           <...>-1     [001] ....    49.083909: __vfs_write <-vfs_write                         
           <...>-1     [001] ....    49.083933: panic <-panic_on_write                   
           <...>-1     [001] d...    49.084195: printk <-panic                                 
           <...>-1     [001] d...    49.084201: vprintk_func <-printk                          
           <...>-1     [001] d...    49.084207: vprintk_default <-printk                          
           <...>-1     [001] d...    49.084211: vprintk_emit <-printk                          
           <...>-1     [001] d...    49.084216: __printk_safe_enter <-vprintk_emit         
           <...>-1     [001] d...    49.084219: _raw_spin_lock <-vprintk_emit       
           <...>-1     [001] d...    49.084223: vprintk_store <-vprintk_emit                    

Patchwise oneline description is given below:

Patch 1 adds support to allocate ring buffer pages from persistent RAM buffer.

Patch 2 introduces a new backend, ramtrace.

Patch 3 adds methods to read previous boot pages from pstore.

Patch 4 adds the functionality to allocate page-sized memory from pstore.

Patch 5 adds the seq_operation methods to iterate through trace entries.

Patch 6 modifies ring_buffer to allocate from ramtrace when pstore is used.

Patch 7 adds ramtrace DT node as child-node of /reserved-memory. 

Nachammai Karuppiah (7):
  tracing: Add support to allocate pages from persistent memory
  pstore: Support a new backend, ramtrace
  pstore: Read and iterate through trace entries in PSTORE
  pstore: Allocate and free page-sized memory in persistent RAM buffer
  tracing: Add support to iterate through pages retrieved from pstore
  tracing: Use ramtrace alloc and free methods while using persistent
    RAM
  dt-bindings: ramtrace: Add ramtrace DT node

 .../bindings/reserved-memory/ramtrace.txt          |  13 +
 drivers/of/platform.c                              |   1 +
 fs/pstore/Makefile                                 |   2 +
 fs/pstore/inode.c                                  |  46 +-
 fs/pstore/platform.c                               |   1 +
 fs/pstore/ramtrace.c                               | 821 +++++++++++++++++++++
 include/linux/pstore.h                             |   3 +
 include/linux/ramtrace.h                           |  28 +
 include/linux/ring_buffer.h                        |  19 +
 include/linux/trace.h                              |  13 +
 kernel/trace/Kconfig                               |  10 +
 kernel/trace/ring_buffer.c                         | 663 ++++++++++++++++-
 kernel/trace/trace.c                               | 312 +++++++-
 kernel/trace/trace.h                               |   5 +-
 14 files changed, 1924 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/reserved-memory/ramtrace.txt
 create mode 100644 fs/pstore/ramtrace.c
 create mode 100644 include/linux/ramtrace.h

-- 
2.7.4