Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of cern.ch designates
 188.184.36.48 as permitted sender) receiver=protection.outlook.com;
 client-ip=188.184.36.48; helo=cernmxgwlb4.cern.ch; pr=C
From:   Federico Vaga <federico.vaga@cern.ch>
To:     Masahiro Yamada <masahiroy@kernel.org>,
        Michal Marek <michal.lkml@markovi.net>,
        Nick Desaulniers <ndesaulniers@google.com>
CC:     <linux-kbuild@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        "Federico Vaga" <federico.vaga@cern.ch>
Subject: [PATCH 1/2] package: add tar development package for 3rd party modules
Date:   Fri, 21 Oct 2022 12:14:52 +0200
Message-ID: <20221021101452.1367745-2-federico.vaga@cern.ch>
In-Reply-To: <20221021101452.1367745-1-federico.vaga@cern.ch>
References: <20221021101452.1367745-1-federico.vaga@cern.ch>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2022 10:15:33.6879
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 977344f1-935a-4831-ea4d-08dab34d30ce
X-MS-Exchange-CrossTenant-Id: c80d3499-4a40-4a8c-986e-abce017d6b19
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=c80d3499-4a40-4a8c-986e-abce017d6b19;Ip=[188.184.36.48];Helo=[cernmxgwlb4.cern.ch]
X-MS-Exchange-CrossTenant-AuthSource: AM0EUR02FT035.eop-EUR02.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV0P278MB0049
X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Most, if not all, Linux distributions provides a Linux development
package which purpose is to support the building of out-of-tree modules
without providing the entire source tree.

What ends up in this development directory is a mixture of source
files (mainly headers) and generated ones (headers, and tools produced
by `make modules_prepare`).

This patch is an attempt to generate a tarball archive containing all
required files to build external modules. It could be than reused by
packagers.

Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
---
 Makefile                       |   2 +-
 scripts/Makefile.package       |  13 +++
 scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
 3 files changed, 221 insertions(+), 1 deletion(-)
 create mode 100644 scripts/package/buildtar-devel

diff --git a/Makefile b/Makefile
index cfbe6a7de640..36a58394ce16 100644
--- a/Makefile
+++ b/Makefile
@@ -1578,7 +1578,7 @@ CLEAN_FILES += include/ksym vmlinux.symvers modules-only.symvers \
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
 		  arch/$(SRCARCH)/include/generated .objdiff \
-		  debian snap tar-install \
+		  debian snap tar-install* \
 		  .config .config.old .version \
 		  Module.symvers \
 		  certs/signing_key.pem \
diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 8bbcced67c22..9523a4dfaee5 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -112,6 +112,13 @@ $(tar-pkgs):
 	$(MAKE) -f $(srctree)/Makefile
 	+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@
 
+tar-dev-pkgs := dir-dev-pkg tar-dev-pkg targz-dev-pkg tarbz2-dev-pkg
+tar-dev-pkgs += tarxz-dev-pkg tarzst-dev-pkg
+PHONY += $(tar-dev-pkgs)
+$(tar-dev-pkgs):
+	$(MAKE) -f $(srctree)/Makefile
+	+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar-devel $@
+
 # perf-pkg - generate a source tarball with perf source
 # ---------------------------------------------------------------------------
 
@@ -159,6 +166,12 @@ help:
 	@echo '  tarbz2-pkg          - Build the kernel as a bzip2 compressed tarball'
 	@echo '  tarxz-pkg           - Build the kernel as a xz compressed tarball'
 	@echo '  tarzst-pkg          - Build the kernel as a zstd compressed tarball'
+	@echo '  dir-dev-pkg         - Module development as a plain directory structure'
+	@echo '  tar-dev-pkg         - Module development as an uncompressed tarball'
+	@echo '  targz-dev-pkg       - Module development as a gzip compressed tarball'
+	@echo '  tarbz2-dev-pkg      - Module development as a bzip2 compressed tarball'
+	@echo '  tarxz-dev-pkg       - Module development as a xz compressed tarball'
+	@echo '  tarzst-dev-pkg      - Module development as a zstd compressed tarball'
 	@echo '  perf-tar-src-pkg    - Build $(perf-tar).tar source tarball'
 	@echo '  perf-targz-src-pkg  - Build $(perf-tar).tar.gz source tarball'
 	@echo '  perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball'
diff --git a/scripts/package/buildtar-devel b/scripts/package/buildtar-devel
new file mode 100644
index 000000000000..87706d50a302
--- /dev/null
+++ b/scripts/package/buildtar-devel
@@ -0,0 +1,207 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# (C) 2022 CERN (home.cern)
+# Author Federico Vaga <federico.vaga@cern.ch>
+#
+# This script is used to build a 3rd party kernel module development tarball
+# from the currently prepared kernel.
+#
+
+set -ex
+
+#
+# Some variables and settings used throughout the script
+#
+tmpdir="${abs_objtree}/tar-install-dev"
+tarball="${abs_objtree}/linux-${KERNELRELEASE}-${ARCH}-dev.tar"
+
+#
+# Figure out how to compress, if requested at all
+#
+case "${1}" in
+    dir-pkg-dev|tar-pkg-dev)
+        opts=
+		;;
+	targz-pkg-dev)
+		opts="-I ${KGZIP}"
+		tarball=${tarball}.gz
+		;;
+	tarbz2-pkg-dev)
+		opts="-I ${KBZIP2}"
+		tarball=${tarball}.bz2
+		;;
+	tarxz-pkg-dev)
+		opts="-I ${XZ}"
+		tarball=${tarball}.xz
+		;;
+	tarzst-pkg-dev)
+		opts="-I ${ZSTD}"
+		tarball=${tarball}.zst
+		;;
+	*)
+		echo "Unknown tarball target \"${1}\" requested, please add it to ${0}." >&2
+		exit 1
+		;;
+esac
+
+#
+# Clean-up and re-create the temporary directory
+#
+
+rm -rf -- "${tmpdir}"
+mkdir -p -- "${tmpdir}"
+
+#
+# Copy required files
+#
+FILTER_FILE=$(mktemp)
+cat <<EOF > ${FILTER_FILE}
+
+#
+# Include the following files and directories required to build external modules
+#
+
++ /arch/
++ /arch/${ARCH}/
++ /arch/${ARCH}/**.h
++ /arch/${ARCH}/**/Makefile*
++ /.config
++ /drivers/
++ /drivers/**/Kbuild*
++ /drivers/**/Kconfig*
++ /drivers/**/Makefile*
++ /include/*
++ /include/**/*.h
++ /Kbuild
++ /Kconfig
++ /Makefile
++ /Module.symvers
++ /scripts/Makefile.*
++ /scripts/basic
++ /scripts/basic/fixdep
++ /scripts/bin2c
++ /scripts/checkincludes.pl
++ /scripts/checkstack.pl
++ /scripts/checkversion.pl
++ /scripts/check-local-export
++ /scripts/depmod.sh
++ /scripts/extract-cert
++ /scripts/gcc-goto.sh
++ /scripts/gcc-version.sh
++ /scripts/gcc-x86_32-has-stack-protector.sh
++ /scripts/gcc-x86_64-has-stack-protector.sh
++ /scripts/genksyms
++ /scripts/genksyms/genksyms
++ /scripts/headers_install.sh
++ /scripts/kallsyms
++ /scripts/Kbuild.include
++ /scripts/kconfig
++ /scripts/kconfig/conf
++ /scripts/kernel-doc
++ /scripts/ld-version.sh
++ /scripts/Lindent
++ /scripts/makelst
++ /scripts/mksysmap
++ /scripts/mkuboot.sh
++ /scripts/mod/
++ /scripts/mod/modpost
++ /scripts/module.lds
++ /scripts/patch-kernel
++ /scripts/pahole-flags.sh
++ /scripts/recordmcount
++ /scripts/recordmcount.pl
++ /scripts/setlocalversion
++ /scripts/sign-file
++ /scripts/subarch.include
++ /scripts/unifdef
++ /scripts/ver_linux
++ /tools
++ /tools/objtool/
++ /tools/objtool/objtool
++ /.version
+
+#
+# Completly ignore the following directories
+#
+- /arch/*
+- /Documentation
+- /LICENSES
+- /block
+- /certs
+- /crypto
+- /drivers/*
+- /fs
+- /init
+- /io_uring
+- /ipc
+- /kernel
+- /lib
+- /MAINTAINERS
+- /mm
+- /modules.*
+- /net
+- /README
+- /rust
+- /samples
+- /scripts/*
+- /scripts/basic/Makefile
+- /scripts/genksyms/*
+- /scripts/kconfig/*
+- /scripts/mod/*
+- /security
+- /sound
+- /System.map
+- /tools/*
+- /tools/objtool/*
+- /usr
+- /virt
+
+#
+# Completly ignore intermediate directories
+#
+- /source
+- /tar-install*
+- /${O}
+#
+# Completly ignore the following files
+#
+
+- .*
+- *.a
+- *.c
+- *.o
+- *.S
+- *vmlinux*
+EOF
+
+#
+# Copy from object directory as well when it is not the same as the source one
+#
+if [ "${abs_objtree}" != "${abs_srctree}" ]
+then
+    rsync -a --filter="merge ${FILTER_FILE}" ${abs_objtree}/ ${tmpdir}
+fi
+rsync -a --filter="merge ${FILTER_FILE}" ${abs_srctree}/ ${tmpdir}
+rm ${FILTER_FILE}
+
+if [ "${1}" = dir-pkg-dev ]; then
+	echo "Kernel tree successfully created in $tmpdir"
+	exit 0
+fi
+
+#
+# Create the tarball
+#
+if tar --owner=root --group=root --help >/dev/null 2>&1; then
+	opts="$opts --owner=root --group=root"
+fi
+
+opts="$opts"
+
+tar cf $tarball -C $tmpdir $opts .
+
+echo "Tarball successfully created in $tarball"
+
+exit 0
-- 
2.27.0