Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Kent Overstreet <kent.overstreet@gmail.com>
To:     linux-kernel@vger.kernel.org, linux-mm@kvack.org, pmladek@suse.com
Cc:     Kent Overstreet <kent.overstreet@gmail.com>, rostedt@goodmis.org,
        enozhatsky@chromium.org, linux@rasmusvillemoes.dk,
        willy@infradead.org
Subject: [PATCH v4 00/34] Printbufs - new data structure for building strings
Date:   Sun, 19 Jun 2022 20:41:59 -0400
Message-Id: <20220620004233.3805-1-kent.overstreet@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Previous discussions:
https://lore.kernel.org/all/20220419203202.2670193-1-kent.overstreet@gmail.com/
https://lore.kernel.org/all/20220519172421.162394-1-kent.overstreet@gmail.com/
https://lore.kernel.org/all/20220604193042.1674951-1-kent.overstreet@gmail.com/

Git repo:
https://evilpiepirate.org/git/bcachefs.git/log/?h=printbuf_v4

Changes since v3:
  Bugfixes and performance improvements, the latest iteration of this patch
  series has been baking in the bcachefs tree and that shook out some bugs.

  Rasmus pointed out that -fno-strict-aliasing is going to cause gcc to generate
  nasty code, and indeed it unfortunately does but according to worst case
  scenario microbenchmarks it's not a problem for actual performance. Using
  memcpy() and memset() in the printbuf helpers _was_ a problem for performance,
  so that's been fixed.

-----------

Core idea: Wouldn't it be nice if we had a common data structure and calling
convention for outputting strings?

The core concept this patch series is aimed at cleaning up and standardizing is
that of a "pretty-printer", which is now a function like prt_foo() or
foo_to_text():

    void foo_to_text(struct printbuf *out, struct foo)

What this patch series does or enables:

 - It becomes quite a bit easier to write composable pretty printers! This is
   huge.

 - A ton of code that works in terms of raw char * pointers and lengths
   (snprintf style, and many weird variations) gets cleaned up, with error prone
   raw pointers arithmetic replaced by proper helpers

 - A ton of code that emits either directly via printk() or to other places
   (sysfs, debugfs) can now output to printbufs, and becomes more reusable and
   composable

 - Countesy of Matthew Wilcox, the new and very cool %pf() format string, which
   allows passing a pretty printer function and its arguments to sprintf() and
   family. This means we can now call type specific pretty-printers without
   adding them to lib/vsprintf.c and writing a bunch of crazy
   parsing-and-dispatch code. For example,

     printk("%pd", dentry);

   becomes

     printk("%pf(%p)", prt_dentry, dentry);

   My OOM debugging & reporting patch series that builds off of this uses this
   to solve a very real problem that Michal Hocko brought up at LSF - with this
   we write shrinkers_to_text(), slab_to_text() which can _also_ now be used for
   reporting in debugfs (which Roman has been working on), as well as in the
   show_mem() report - the "%pf()" syntax lets us print the output of those
   functions without allocating (and having to preallocate) a separate buffer.

 - Some new formatting helpers:
   
   Nicely aligned text is much easier to read, and something that we want a
   _lot, but outputting nicely aligned text with printf() is a pain in the ass.
   Printbufs add tabstops, which can be used for right or left justification -
   simple, easy. prt_tab() emits spaces up to the next tabstop, prt_tab_rjust()
   advances to the next tabstop right justifying text since the previous
   tabstop.

   Printbufs also add an indent level, obeyed by prt_newline() which can be very
   useful for multi line output.

   In the future, \n and \t in format strings may learn to obey these as well.

 - Optional heap allocation - no need to statically allocate buffers on the
   stack and guess at the output size.

 - Lots of consolidating and refactoring
   
   This series replaces seq_buf, which does basically what an earlier version of
   printbufs did.

   A good chunk of lib/string_helpers.c, as well as lib/hexdump.c are converted
   (and simplified!).

   Pretty printers in lib/vsprintf.c previously outputted to buffers on the
   stack and then copied _that_ to the actual output buffer, that's all gone
   (replaced by proper helpers for outputting chars and strings), and they also
   used printf_spec for argument passing in ad-hoc ways. This patch series does
   a lot towards converting them to more standard pretty printers that can be
   called via %pf() instead of having to live in lib/vsprintf.c. Still to do:
   format string decoding for argument passing is a mess that's scattered all
   over the place.

In the course of working on this patch series, I've spotted a _lot_ more
consolidation and refactoring that needs to be done - we've got a ton of API
fragmentation leading to lots of code duplication.

But I'm already really excited about what this patch series enables.

Cheers!

Kent Overstreet (34):
  lib/printbuf: New data structure for printing strings
  lib/string_helpers: Convert string_escape_mem() to printbuf
  vsprintf: Convert to printbuf
  lib/hexdump: Convert to printbuf
  vsprintf: %pf(%p)
  lib/string_helpers: string_get_size() now returns characters wrote
  lib/printbuf: Heap allocation
  lib/printbuf: Tabstops, indenting
  lib/printbuf: Unit specifiers
  lib/pretty-printers: prt_string_option(), prt_bitflags()
  vsprintf: Improve number()
  vsprintf: prt_u64_minwidth(), prt_u64()
  test_printf: Drop requirement that sprintf not write past nul
  vsprintf: Start consolidating printf_spec handling
  vsprintf: Refactor resource_string()
  vsprintf: Refactor fourcc_string()
  vsprintf: Refactor ip_addr_string()
  vsprintf: Refactor mac_address_string()
  vsprintf: time_and_date() no longer takes printf_spec
  vsprintf: flags_string() no longer takes printf_spec
  vsprintf: Refactor device_node_string, fwnode_string
  vsprintf: Refactor hex_string, bitmap_string_list, bitmap_string
  Input/joystick/analog: Convert from seq_buf -> printbuf
  mm/memcontrol.c: Convert to printbuf
  clk: tegra: bpmp: Convert to printbuf
  tools/testing/nvdimm: Convert to printbuf
  powerpc: Convert to printbuf
  x86/resctrl: Convert to printbuf
  PCI/P2PDMA: Convert to printbuf
  tracing: trace_events_synth: Convert to printbuf
  d_path: prt_path()
  ACPI/APEI: Add missing include
  tracing: Convert to printbuf
  Delete seq_buf

 Documentation/core-api/printk-formats.rst |   22 +
 arch/powerpc/kernel/process.c             |   16 +-
 arch/powerpc/kernel/security.c            |   75 +-
 arch/powerpc/platforms/pseries/papr_scm.c |   34 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |   16 +-
 drivers/acpi/apei/erst-dbg.c              |    1 +
 drivers/clk/tegra/clk-bpmp.c              |   21 +-
 drivers/input/joystick/analog.c           |   23 +-
 drivers/pci/p2pdma.c                      |   21 +-
 fs/d_path.c                               |   35 +
 include/linux/dcache.h                    |    1 +
 include/linux/kernel.h                    |   12 +
 include/linux/pretty-printers.h           |   10 +
 include/linux/printbuf.h                  |  253 +++
 include/linux/seq_buf.h                   |  162 --
 include/linux/string.h                    |    5 +
 include/linux/string_helpers.h            |    8 +-
 include/linux/trace_events.h              |    2 +-
 include/linux/trace_seq.h                 |   17 +-
 kernel/trace/trace.c                      |   45 +-
 kernel/trace/trace_dynevent.c             |   34 +-
 kernel/trace/trace_events_filter.c        |    2 +-
 kernel/trace/trace_events_synth.c         |   32 +-
 kernel/trace/trace_functions_graph.c      |    6 +-
 kernel/trace/trace_kprobe.c               |    2 +-
 kernel/trace/trace_seq.c                  |  111 +-
 lib/Makefile                              |    4 +-
 lib/hexdump.c                             |  246 +--
 lib/pretty-printers.c                     |   60 +
 lib/printbuf.c                            |  253 +++
 lib/seq_buf.c                             |  397 -----
 lib/string_helpers.c                      |  224 +--
 lib/test_hexdump.c                        |   30 +-
 lib/test_printf.c                         |   33 +-
 lib/vsprintf.c                            | 1723 ++++++++++-----------
 mm/memcontrol.c                           |   68 +-
 tools/testing/nvdimm/test/ndtest.c        |   22 +-
 37 files changed, 2050 insertions(+), 1976 deletions(-)
 create mode 100644 include/linux/pretty-printers.h
 create mode 100644 include/linux/printbuf.h
 delete mode 100644 include/linux/seq_buf.h
 create mode 100644 lib/pretty-printers.c
 create mode 100644 lib/printbuf.c
 delete mode 100644 lib/seq_buf.c

-- 
2.36.1