Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5;
From:   Nick Terrell <nickrterrell@gmail.com>
To:     linux-kernel@vger.kernel.org
Cc:     Yann Collet <cyan@meta.com>, Nick Terrell <terrelln@meta.com>,
        Nick Terrell <nickrterrell@gmail.com>,
        Kernel Team <Kernel-team@fb.com>,
        Giovanni Cabiddu <giovanni.cabiddu@intel.com>,
        Linus Torvalds <torvalds@linux-foundation.org>
Subject: [PATCH 0/2] zstd: import upstream v1.5.5
Date:   Mon, 20 Nov 2023 17:03:16 -0800
Message-ID: <20231121010318.524570-1-nickrterrell@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

From: Nick Terrell <terrelln@meta.com>

Import upstream zstd v1.5.5 to expose upstream's QAT integration.

The following changes since commit 77618db346455129424fadbbaec596a09feaf3bb:

  zstd: Fix array-index-out-of-bounds UBSAN warning (2023-11-14 17:12:52 -0800)

are also available in the Git repository at:

  https://github.com/terrelln/linux.git tags/2023-11-20-zstd-v1.5.5-v1

In addition to keeping the kernel's copy of zstd up to date, this update
was requested by Intel to expose upstream zstd's external match provider
API to the kernel, which allows QAT to accelerate the LZ match finding
stage.

The result of the series is that zstd is cleanly imported from the
tag v1.5.5-kernel [0], which is signed with upstream's signing key
EF8FE99528B52FFD [1]. Meaning that running this script creates no diff:

  export ZSTD=/path/to/repo/zstd/
  export LINUX=/path/to/repo/linux/
  cd "$ZSTD/contrib/linux-kernel"
  git checkout v1.5.5-kernel
  make import LINUX="$LINUX"

The first patch updates to v1.5.5 with three cherry picked backports
from the Linux kernel and the upstream dev branch. The second patch
negates a regression in decompression speed caused by the update.

The series has been tested on x86-64, and has been boot tested with a
zstd compressed kernel on i386 and aarch64. I benchmarked the series
on x86-64 with gcc-13.2.1 on an Intel i9-9900K by measuring the
performance of compressed filesystems read & writes. 

Component,	Level,	C. time delta,	size delta,	D. time delta
Btrfs    ,	    1,	        -1.5%,	     +0.0%,	        +1.8%
Btrfs    ,	    3,	        -5.8%,	     +0.0%,	        +3.2%
Btrfs    ,	    5,	        -5.7%,	     +0.0%,	        +0.0%
Btrfs    ,	    7,	        -6.5%,	     +0.0%,	        +0.4%
Btrfs    ,	    9,	        -6.9%,	     +0.0%,	        +1.5%
Squashfs ,	    1,	          N/A,	      0.0%,	        +1.0%

For levels 3 and above, there is a ~6% reduction in write+compression time.
There is also a small increase in read+decompression time.

If everything looks good, I will submit this patch set with the rest of my
tree in the v6.8 merge window.

[0] https://github.com/facebook/zstd/tree/v1.5.5-kernel
[1] https://keyserver.ubuntu.com/pks/lookup?search=EF8FE99528B52FFD&fingerprint=on&op=index

Nick Terrell (2):
  zstd: import upstream v1.5.5
  zstd: Backport Huffman speed improvement from upstream

 include/linux/zstd.h                          |    2 +-
 include/linux/zstd_errors.h                   |   23 +-
 include/linux/zstd_lib.h                      |  697 +++++--
 lib/zstd/Makefile                             |    2 +-
 lib/zstd/common/allocations.h                 |   56 +
 lib/zstd/common/bits.h                        |  149 ++
 lib/zstd/common/bitstream.h                   |   53 +-
 lib/zstd/common/compiler.h                    |   14 +-
 lib/zstd/common/cpu.h                         |    3 +-
 lib/zstd/common/debug.c                       |    3 +-
 lib/zstd/common/debug.h                       |    3 +-
 lib/zstd/common/entropy_common.c              |   42 +-
 lib/zstd/common/error_private.c               |   12 +-
 lib/zstd/common/error_private.h               |    3 +-
 lib/zstd/common/fse.h                         |   89 +-
 lib/zstd/common/fse_decompress.c              |   94 +-
 lib/zstd/common/huf.h                         |  222 +--
 lib/zstd/common/mem.h                         |    2 +-
 lib/zstd/common/portability_macros.h          |   26 +-
 lib/zstd/common/zstd_common.c                 |   38 +-
 lib/zstd/common/zstd_deps.h                   |   16 +-
 lib/zstd/common/zstd_internal.h               |   99 +-
 lib/zstd/compress/clevels.h                   |    3 +-
 lib/zstd/compress/fse_compress.c              |   59 +-
 lib/zstd/compress/hist.c                      |    3 +-
 lib/zstd/compress/hist.h                      |    3 +-
 lib/zstd/compress/huf_compress.c              |  372 ++--
 lib/zstd/compress/zstd_compress.c             | 1762 ++++++++++++-----
 lib/zstd/compress/zstd_compress_internal.h    |  333 +++-
 lib/zstd/compress/zstd_compress_literals.c    |  155 +-
 lib/zstd/compress/zstd_compress_literals.h    |   25 +-
 lib/zstd/compress/zstd_compress_sequences.c   |    7 +-
 lib/zstd/compress/zstd_compress_sequences.h   |    3 +-
 lib/zstd/compress/zstd_compress_superblock.c  |   47 +-
 lib/zstd/compress/zstd_compress_superblock.h  |    3 +-
 lib/zstd/compress/zstd_cwksp.h                |  149 +-
 lib/zstd/compress/zstd_double_fast.c          |  129 +-
 lib/zstd/compress/zstd_double_fast.h          |    6 +-
 lib/zstd/compress/zstd_fast.c                 |  582 ++++--
 lib/zstd/compress/zstd_fast.h                 |    6 +-
 lib/zstd/compress/zstd_lazy.c                 |  518 ++---
 lib/zstd/compress/zstd_lazy.h                 |    7 +-
 lib/zstd/compress/zstd_ldm.c                  |   11 +-
 lib/zstd/compress/zstd_ldm.h                  |    3 +-
 lib/zstd/compress/zstd_ldm_geartab.h          |    3 +-
 lib/zstd/compress/zstd_opt.c                  |  187 +-
 lib/zstd/compress/zstd_opt.h                  |    3 +-
 lib/zstd/decompress/huf_decompress.c          |  770 ++++---
 lib/zstd/decompress/zstd_ddict.c              |    9 +-
 lib/zstd/decompress/zstd_ddict.h              |    3 +-
 lib/zstd/decompress/zstd_decompress.c         |  261 ++-
 lib/zstd/decompress/zstd_decompress_block.c   |  283 ++-
 lib/zstd/decompress/zstd_decompress_block.h   |    8 +-
 .../decompress/zstd_decompress_internal.h     |    7 +-
 lib/zstd/decompress_sources.h                 |    2 +-
 lib/zstd/zstd_common_module.c                 |    5 +-
 lib/zstd/zstd_compress_module.c               |    2 +-
 lib/zstd/zstd_decompress_module.c             |    4 +-
 58 files changed, 4787 insertions(+), 2594 deletions(-)
 create mode 100644 lib/zstd/common/allocations.h
 create mode 100644 lib/zstd/common/bits.h

-- 
2.42.1