This patch series provides the option to omit exported symbols from
the kernel and modules that are never referenced by any of the selected
modules in the current kernel configuration. this allows for optimizing
the compiled code and reducing final binaries' size. When using LTO the
binary size reduction is even more effective. It could also be argued
that this could bring some security advantages.
The original cover letter with lots of test results can be found here:
https://lkml.org/lkml/2016/2/8/813
Please consider for merging into your tree. Alternately, the following
branch can be pulled:
http://git.linaro.org/people/nicolas.pitre/linux.git autoksyms
Thanks.
Changes from v5:
- Disable automatic dependency file generation during the preprocessor
pass when gathering exported symbol names. Not only it is redundant
but in some conditions this crashes fixdep that might be reading the
previous dependency file at the same time. Reported by Michal Marek.
- Redirect error messages to stderr rather than pass it as a symbol name
dependency.
Changes from v4:
- Correctness changes plus small cleanup to adjust_autoksyms.sh as
suggested by Michal Marek.
- Changed ksym build dependency generation by re-running the preprocessor
rather than collecting those dependencies as warnings through stderr
which was too fragile.
Changes from v3:
- Shell portability changes to adjust_autoksyms.sh, partly from
suggestions by Zev Weiss.
- Fix sample modules by building them before adjust_autoksyms.sh is run.
Changes from v2:
- Generating the build dependencies by parsing the source with fixdep
turned out to be unreliable due to all the EXPORT_SYMBOL() variants,
and especially their use within macros where the actual symbol name
is known only after running the preprocessor. This list of symbol names
is now obtained from the preprocessor directly, fixing allmodconfig
builds.
Changes from v1:
- Replaced "exp" that doesn't convey the right meaning as noted by
Sam Ravnborg. The "ksym" identifier is actually what the kernel
already uses for this. Therefore:
- CONFIG_TRIM_UNUSED_EXPSYMS --> CONFIG_TRIM_UNUSED_KSYMS
- include/generated/expsyms.h --> include/generated/autoksyms.h
- #define __EXPSYM_* --> #define __KSYM_*
- Some sed regexp improvements as suggested by Al Viro.
- Renamed vmlinux_recursive target to autoksyms_recursive.
- Accept EXPORT_SYMBOL variants with a prefix, e.g. ACPI_EXPORT_SYMBOL.
- Minor commit log clarifications.
- Added Rusty's ACK.
diffstat:
Makefile | 23 ++++++--
include/linux/export.h | 33 +++++++++++-
init/Kconfig | 16 ++++++
scripts/Kbuild.include | 32 +++++++++++-
scripts/Makefile.build | 22 ++++----
scripts/adjust_autoksyms.sh | 101 ++++++++++++++++++++++++++++++++++++
scripts/basic/fixdep.c | 61 ++++++++++++++++------
7 files changed, 258 insertions(+), 30 deletions(-)
Similar to include/generated/autoconf.h, include/generated/autoksyms.h
will contain a list of defines for each EXPORT_SYMBOL() that we want
active. The format is:
#define __KSYM_<symbol_name> 1
This list will be auto-generated with another patch. For now we only
include the preprocessor magic to automatically create or omit the
corresponding struct kernel_symbol declaration.
Given the content of include/generated/autoksyms.h may not be known in
advance, an empty file is created early on to let the build proceed.
Signed-off-by: Nicolas Pitre <[email protected]>
Acked-by: Rusty Russell <[email protected]>
---
Makefile | 2 ++
include/linux/export.h | 22 ++++++++++++++++++++--
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 6c1a3c2479..e916428cf7 100644
--- a/Makefile
+++ b/Makefile
@@ -986,6 +986,8 @@ prepare2: prepare3 outputmakefile asm-generic
prepare1: prepare2 $(version_h) include/generated/utsrelease.h \
include/config/auto.conf
$(cmd_crmodverdir)
+ $(Q)test -e include/generated/autoksyms.h || \
+ touch include/generated/autoksyms.h
archprepare: archheaders archscripts prepare1 scripts_basic
diff --git a/include/linux/export.h b/include/linux/export.h
index 96e45ea463..77afdb2a25 100644
--- a/include/linux/export.h
+++ b/include/linux/export.h
@@ -38,7 +38,7 @@ extern struct module __this_module;
#ifdef CONFIG_MODULES
-#ifndef __GENKSYMS__
+#if defined(__KERNEL__) && !defined(__GENKSYMS__)
#ifdef CONFIG_MODVERSIONS
/* Mark the CRC weak since genksyms apparently decides not to
* generate a checksums for some symbols */
@@ -53,7 +53,7 @@ extern struct module __this_module;
#endif
/* For every exported symbol, place a struct in the __ksymtab section */
-#define __EXPORT_SYMBOL(sym, sec) \
+#define ___EXPORT_SYMBOL(sym, sec) \
extern typeof(sym) sym; \
__CRC_SYMBOL(sym, sec) \
static const char __kstrtab_##sym[] \
@@ -65,6 +65,24 @@ extern struct module __this_module;
__attribute__((section("___ksymtab" sec "+" #sym), unused)) \
= { (unsigned long)&sym, __kstrtab_##sym }
+#ifdef CONFIG_TRIM_UNUSED_KSYMS
+
+#include <linux/kconfig.h>
+#include <generated/autoksyms.h>
+
+#define __EXPORT_SYMBOL(sym, sec) \
+ __cond_export_sym(sym, sec, config_enabled(__KSYM_##sym))
+#define __cond_export_sym(sym, sec, conf) \
+ ___cond_export_sym(sym, sec, conf)
+#define ___cond_export_sym(sym, sec, enabled) \
+ __cond_export_sym_##enabled(sym, sec)
+#define __cond_export_sym_1(sym, sec) ___EXPORT_SYMBOL(sym, sec)
+#define __cond_export_sym_0(sym, sec) /* nothing */
+
+#else
+#define __EXPORT_SYMBOL ___EXPORT_SYMBOL
+#endif
+
#define EXPORT_SYMBOL(sym) \
__EXPORT_SYMBOL(sym, "")
--
2.5.0
Make sample modules in parallel with the rest of the kernel rather
than having them built from the vmlinux target. This makes the build
slightly faster, and those modules are properly considered when
adjust_autoksyms.sh is executed.
Signed-off-by: Nicolas Pitre <[email protected]>
---
Makefile | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index bb865095ca..f5daa4bbf3 100644
--- a/Makefile
+++ b/Makefile
@@ -928,9 +928,6 @@ endif
ifdef CONFIG_HEADERS_CHECK
$(Q)$(MAKE) -f $(srctree)/Makefile headers_check
endif
-ifdef CONFIG_SAMPLES
- $(Q)$(MAKE) $(build)=samples
-endif
ifdef CONFIG_BUILD_DOCSRC
$(Q)$(MAKE) $(build)=Documentation
endif
@@ -948,6 +945,11 @@ PHONY += autoksyms_recursive
include/generated/autoksyms.h: FORCE
$(Q)$(CONFIG_SHELL) scripts/adjust_autoksyms.sh true
+# Build samples along the rest of the kernel
+ifdef CONFIG_SAMPLES
+vmlinux-dirs += samples
+endif
+
# The actual objects are generated when descending,
# make sure no implicit rule kicks in
$(sort $(vmlinux-deps)): $(vmlinux-dirs) ;
--
2.5.0
Like with kconfig options, we now have the ability to compile in and
out individual EXPORT_SYMBOL() declarations based on the content of
include/generated/autoksyms.h. However we don't want the entire
world to be rebuilt whenever that file is touched.
Let's apply the same build dependency trick used for CONFIG_* symbols
where the time stamp of empty files whose paths matching those symbols
is used to trigger fine grained rebuilds. In our case the key is the
symbol name passed to EXPORT_SYMBOL().
However, unlike config options, we cannot just use fixdep to parse
the source code for EXPORT_SYMBOL(ksym) because several variants exist
and parsing them all in a separate tool, and keeping it in synch, is
not trivially maintainable. Furthermore, there are variants such as
EXPORT_SYMBOL_GPL(pci_user_read_config_##size);
that are instanciated via a macro for which we can't easily determine
the actual exported symbol name(s) short of actually running the
preprocessor on them.
Storing the symbol name string in a special ELF section doesn't work
for targets that output assembly or preprocessed source.
So the best way is really to leverage the preprocessor by having it
output actual symbol names anchored by a special sequence that can be
easily filtered out. Then the list of symbols is simply fed to fixdep
to be merged with the other dependencies.
That implies the preprocessor is executed twice for each source file.
A previous attempt relied on a warning pragma for each EXPORT_SYMBOL()
instance that was filtered apart from stderr by the build system with
a sed script during the actual compilation pass. Unfortunately the
preprocessor/compiler diagnostic output isn't stable between versions
and this solution, although more efficient, was deemed too fragile.
Because of the lowercasing performed by fixdep, there might be name
collisions triggering spurious rebuilds for similar symbols. But this
shouldn't be a big issue in practice. (This is the case for CONFIG_*
symbols and I didn't want to be different here, whatever the original
reason for doing so.)
To avoid needless build overhead, the exported symbol name gathering is
performed only when CONFIG_TRIM_UNUSED_KSYMS is selected.
Signed-off-by: Nicolas Pitre <[email protected]>
Acked-by: Rusty Russell <[email protected]>
---
include/linux/export.h | 13 ++++++++++++-
scripts/Kbuild.include | 27 +++++++++++++++++++++++++++
scripts/basic/fixdep.c | 1 +
3 files changed, 40 insertions(+), 1 deletion(-)
diff --git a/include/linux/export.h b/include/linux/export.h
index 77afdb2a25..2f9ccbe6a6 100644
--- a/include/linux/export.h
+++ b/include/linux/export.h
@@ -65,7 +65,18 @@ extern struct module __this_module;
__attribute__((section("___ksymtab" sec "+" #sym), unused)) \
= { (unsigned long)&sym, __kstrtab_##sym }
-#ifdef CONFIG_TRIM_UNUSED_KSYMS
+#if defined(__KSYM_DEPS__)
+
+/*
+ * For fine grained build dependencies, we want to tell the build system
+ * about each possible exported symbol even if they're not actually exported.
+ * We use a string pattern that is unlikely to be valid code that the build
+ * system filters out from the preprocessor output (see ksym_dep_filter
+ * in scripts/Kbuild.include).
+ */
+#define __EXPORT_SYMBOL(sym, sec) === __KSYM_##sym ===
+
+#elif defined(CONFIG_TRIM_UNUSED_KSYMS)
#include <linux/kconfig.h>
#include <generated/autoksyms.h>
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 8a257fa663..2c14a27e39 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -258,12 +258,39 @@ if_changed_dep = $(if $(strip $(any-prereq) $(arg-check) ), \
@set -e; \
$(cmd_and_fixdep))
+ifndef CONFIG_TRIM_UNUSED_KSYMS
+
cmd_and_fixdep = \
$(echo-cmd) $(cmd_$(1)); \
scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).tmp;\
rm -f $(depfile); \
mv -f $(dot-target).tmp $(dot-target).cmd;
+else
+
+# Filter out exported kernel symbol names from the preprocessor output.
+# See also __KSYM_DEPS__ in include/linux/export.h.
+# We disable the depfile generation here, so as not to overwrite the existing
+# depfile while fixdep is parsing it
+flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
+ksym_dep_filter = \
+ case "$(1)" in \
+ cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
+ as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
+ cpp_lds_S) : ;; \
+ *) echo "Don't know how to preprocess $(1)" >&2; false ;; \
+ esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'
+
+cmd_and_fixdep = \
+ $(echo-cmd) $(cmd_$(1)); \
+ $(ksym_dep_filter) | \
+ scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \
+ > $(dot-target).tmp; \
+ rm -f $(depfile); \
+ mv -f $(dot-target).tmp $(dot-target).cmd;
+
+endif
+
# Usage: $(call if_changed_rule,foo)
# Will check if $(cmd_foo) or any of the prerequisites changed,
# and if so will execute $(rule_foo).
diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
index d984deb120..2dc6bf754a 100644
--- a/scripts/basic/fixdep.c
+++ b/scripts/basic/fixdep.c
@@ -354,6 +354,7 @@ static void parse_dep_file(void *map, size_t len)
/* Ignore certain dependencies */
if (strrcmp(s, "include/generated/autoconf.h") &&
+ strrcmp(s, "include/generated/autoksyms.h") &&
strrcmp(s, "arch/um/include/uml-config.h") &&
strrcmp(s, "include/linux/kconfig.h") &&
strrcmp(s, ".ver")) {
--
2.5.0
The config option to enable it all.
Signed-off-by: Nicolas Pitre <[email protected]>
Acked-by: Rusty Russell <[email protected]>
---
init/Kconfig | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/init/Kconfig b/init/Kconfig
index 22320804fb..e6f666331b 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1990,6 +1990,22 @@ config MODULE_COMPRESS_XZ
endchoice
+config TRIM_UNUSED_KSYMS
+ bool "Trim unused exported kernel symbols"
+ depends on MODULES && !UNUSED_SYMBOLS
+ help
+ The kernel and some modules make many symbols available for
+ other modules to use via EXPORT_SYMBOL() and variants. Depending
+ on the set of modules being selected in your kernel configuration,
+ many of those exported symbols might never be used.
+
+ This option allows for unused exported symbols to be dropped from
+ the build. In turn, this provides the compiler more opportunities
+ (especially when using LTO) for optimizing the code and reducing
+ binary size. This might have some security advantages as well.
+
+ If unsure say N.
+
endif # MODULES
config MODULES_TREE_LOOKUP
--
2.5.0
Given the list of exported symbols needed by all modules, we can create
a header file containing preprocessor defines for each of those symbols.
Also, when some symbols are added and/or removed from the list, we can
update the time on the corresponding files used as build dependencies for
those symbols. And finally, if any symbol did change state, the
corresponding source files must be rebuilt.
The insertion or removal of an EXPORT_SYMBOL() entry within a module may
create or remove the need for another exported symbol. This is why this
operation has to be repeated until the list of needed exported symbols
becomes stable. Only then the final kernel and modules link take place.
Signed-off-by: Nicolas Pitre <[email protected]>
Acked-by: Rusty Russell <[email protected]>
---
Makefile | 13 ++++++
scripts/adjust_autoksyms.sh | 101 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 114 insertions(+)
create mode 100755 scripts/adjust_autoksyms.sh
diff --git a/Makefile b/Makefile
index e916428cf7..bb865095ca 100644
--- a/Makefile
+++ b/Makefile
@@ -921,6 +921,10 @@ quiet_cmd_link-vmlinux = LINK $@
# Include targets which we want to
# execute if the rest of the kernel build went well.
vmlinux: scripts/link-vmlinux.sh $(vmlinux-deps) FORCE
+ifdef CONFIG_TRIM_UNUSED_KSYMS
+ $(Q)$(CONFIG_SHELL) scripts/adjust_autoksyms.sh \
+ "$(MAKE) KBUILD_MODULES=1 -f $(srctree)/Makefile autoksyms_recursive"
+endif
ifdef CONFIG_HEADERS_CHECK
$(Q)$(MAKE) -f $(srctree)/Makefile headers_check
endif
@@ -935,6 +939,15 @@ ifdef CONFIG_GDB_SCRIPTS
endif
+$(call if_changed,link-vmlinux)
+autoksyms_recursive: $(vmlinux-deps)
+ $(Q)$(CONFIG_SHELL) scripts/adjust_autoksyms.sh \
+ "$(MAKE) KBUILD_MODULES=1 -f $(srctree)/Makefile autoksyms_recursive"
+PHONY += autoksyms_recursive
+
+# standalone target for easier testing
+include/generated/autoksyms.h: FORCE
+ $(Q)$(CONFIG_SHELL) scripts/adjust_autoksyms.sh true
+
# The actual objects are generated when descending,
# make sure no implicit rule kicks in
$(sort $(vmlinux-deps)): $(vmlinux-dirs) ;
diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh
new file mode 100755
index 0000000000..5bf538f1ed
--- /dev/null
+++ b/scripts/adjust_autoksyms.sh
@@ -0,0 +1,101 @@
+#!/bin/sh
+
+# Script to create/update include/generated/autoksyms.h and dependency files
+#
+# Copyright: (C) 2016 Linaro Limited
+# Created by: Nicolas Pitre, January 2016
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+
+# Create/update the include/generated/autoksyms.h file from the list
+# of all module's needed symbols as recorded on the third line of
+# .tmp_versions/*.mod files.
+#
+# For each symbol being added or removed, the corresponding dependency
+# file's timestamp is updated to force a rebuild of the affected source
+# file. All arguments passed to this script are assumed to be a command
+# to be exec'd to trigger a rebuild of those files.
+
+set -e
+
+cur_ksyms_file="include/generated/autoksyms.h"
+new_ksyms_file="include/generated/autoksyms.h.tmpnew"
+
+info() {
+ if [ "$quiet" != "silent_" ]; then
+ printf " %-7s %s\n" "$1" "$2"
+ fi
+}
+
+info "CHK" "$cur_ksyms_file"
+
+# Use "make V=1" to debug this script.
+case "$KBUILD_VERBOSE" in
+*1*)
+ set -x
+ ;;
+esac
+
+# We need access to CONFIG_ symbols
+case "${KCONFIG_CONFIG}" in
+*/*)
+ . "${KCONFIG_CONFIG}"
+ ;;
+*)
+ # Force using a file from the current directory
+ . "./${KCONFIG_CONFIG}"
+esac
+
+# In case it doesn't exist yet...
+if [ -e "$cur_ksyms_file" ]; then touch "$cur_ksyms_file"; fi
+
+# Generate a new ksym list file with symbols needed by the current
+# set of modules.
+cat > "$new_ksyms_file" << EOT
+/*
+ * Automatically generated file; DO NOT EDIT.
+ */
+
+EOT
+sed -ns -e '3s/ /\n/gp' "$MODVERDIR"/*.mod | sort -u |
+while read sym; do
+ if [ -n "$CONFIG_HAVE_UNDERSCORE_SYMBOL_PREFIX" ]; then
+ sym="${sym#_}"
+ fi
+ echo "#define __KSYM_${sym} 1"
+done >> "$new_ksyms_file"
+
+# Special case for modversions (see modpost.c)
+if [ -n "$CONFIG_MODVERSIONS" ]; then
+ echo "#define __KSYM_module_layout 1" >> "$new_ksyms_file"
+fi
+
+# Extract changes between old and new list and touch corresponding
+# dependency files.
+changed=$(
+count=0
+sort "$cur_ksyms_file" "$new_ksyms_file" | uniq -u |
+sed -n 's/^#define __KSYM_\(.*\) 1/\1/p' | tr "A-Z_" "a-z/" |
+while read sympath; do
+ if [ -z "$sympath" ]; then continue; fi
+ depfile="include/config/ksym/${sympath}.h"
+ mkdir -p "$(dirname "$depfile")"
+ touch "$depfile"
+ echo $((count += 1))
+done | tail -1 )
+changed=${changed:-0}
+
+if [ $changed -gt 0 ]; then
+ # Replace the old list with tne new one
+ old=$(grep -c "^#define __KSYM_" "$cur_ksyms_file" || true)
+ new=$(grep -c "^#define __KSYM_" "$new_ksyms_file" || true)
+ info "KSYMS" "symbols: before=$old, after=$new, changed=$changed"
+ info "UPD" "$cur_ksyms_file"
+ mv -f "$new_ksyms_file" "$cur_ksyms_file"
+ # Then trigger a rebuild of affected source files
+ exec $@
+else
+ rm -f "$new_ksyms_file"
+fi
--
2.5.0
... and merge them in the list of parsed dependencies.
Signed-off-by: Nicolas Pitre <[email protected]>
---
scripts/basic/fixdep.c | 60 +++++++++++++++++++++++++++++++++++++-------------
1 file changed, 45 insertions(+), 15 deletions(-)
diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c
index 5b327c67a8..d984deb120 100644
--- a/scripts/basic/fixdep.c
+++ b/scripts/basic/fixdep.c
@@ -120,13 +120,15 @@
#define INT_NFIG ntohl(0x4e464947)
#define INT_FIG_ ntohl(0x4649475f)
+int insert_extra_deps;
char *target;
char *depfile;
char *cmdline;
static void usage(void)
{
- fprintf(stderr, "Usage: fixdep <depfile> <target> <cmdline>\n");
+ fprintf(stderr, "Usage: fixdep [-e] <depfile> <target> <cmdline>\n");
+ fprintf(stderr, " -e insert extra dependencies given on stdin\n");
exit(1);
}
@@ -138,6 +140,40 @@ static void print_cmdline(void)
printf("cmd_%s := %s\n\n", target, cmdline);
}
+/*
+ * Print out a dependency path from a symbol name
+ */
+static void print_config(const char *m, int slen)
+{
+ int c, i;
+
+ printf(" $(wildcard include/config/");
+ for (i = 0; i < slen; i++) {
+ c = m[i];
+ if (c == '_')
+ c = '/';
+ else
+ c = tolower(c);
+ putchar(c);
+ }
+ printf(".h) \\\n");
+}
+
+static void do_extra_deps(void)
+{
+ if (insert_extra_deps) {
+ char buf[80];
+ while(fgets(buf, sizeof(buf), stdin)) {
+ int len = strlen(buf);
+ if (len < 2 || buf[len-1] != '\n') {
+ fprintf(stderr, "fixdep: bad data on stdin\n");
+ exit(1);
+ }
+ print_config(buf, len-1);
+ }
+ }
+}
+
struct item {
struct item *next;
unsigned int len;
@@ -197,23 +233,12 @@ static void define_config(const char *name, int len, unsigned int hash)
static void use_config(const char *m, int slen)
{
unsigned int hash = strhash(m, slen);
- int c, i;
if (is_defined_config(m, slen, hash))
return;
define_config(m, slen, hash);
-
- printf(" $(wildcard include/config/");
- for (i = 0; i < slen; i++) {
- c = m[i];
- if (c == '_')
- c = '/';
- else
- c = tolower(c);
- putchar(c);
- }
- printf(".h) \\\n");
+ print_config(m, slen);
}
static void parse_config_file(const char *map, size_t len)
@@ -250,7 +275,7 @@ static void parse_config_file(const char *map, size_t len)
}
}
-/* test is s ends in sub */
+/* test if s ends in sub */
static int strrcmp(const char *s, const char *sub)
{
int slen = strlen(s);
@@ -374,6 +399,8 @@ static void parse_dep_file(void *map, size_t len)
exit(1);
}
+ do_extra_deps();
+
printf("\n%s: $(deps_%s)\n\n", target, target);
printf("$(deps_%s):\n", target);
}
@@ -430,7 +457,10 @@ int main(int argc, char *argv[])
{
traps();
- if (argc != 4)
+ if (argc == 5 && !strcmp(argv[1], "-e")) {
+ insert_extra_deps = 1;
+ argv++;
+ } else if (argc != 4)
usage();
depfile = argv[1];
--
2.5.0
The generation and postprocessing of automatic dependency rules is
duplicated in rule_cc_o_c and if_changed_dep. Since this is not a
trivial one-liner action, it is now abstracted under cmd_and_fixdep
to simplify things and make future changes easier.
In the rule_cc_o_c case that means the order of some commands has been
altered, namely fixdep and related file manipulations are executed
earlier, but they didn't depend on those commands that now execute later.
Signed-off-by: Nicolas Pitre <[email protected]>
---
scripts/Kbuild.include | 5 ++++-
scripts/Makefile.build | 9 ++-------
2 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 1db6d73c8d..8a257fa663 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -256,10 +256,13 @@ if_changed = $(if $(strip $(any-prereq) $(arg-check)), \
# Execute the command and also postprocess generated .d dependencies file.
if_changed_dep = $(if $(strip $(any-prereq) $(arg-check) ), \
@set -e; \
+ $(cmd_and_fixdep))
+
+cmd_and_fixdep = \
$(echo-cmd) $(cmd_$(1)); \
scripts/basic/fixdep $(depfile) $@ '$(make-cmd)' > $(dot-target).tmp;\
rm -f $(depfile); \
- mv -f $(dot-target).tmp $(dot-target).cmd)
+ mv -f $(dot-target).tmp $(dot-target).cmd;
# Usage: $(call if_changed_rule,foo)
# Will check if $(cmd_foo) or any of the prerequisites changed,
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index f4b4320e0d..8134ee81ad 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -243,14 +243,9 @@ endif
define rule_cc_o_c
$(call echo-cmd,checksrc) $(cmd_checksrc) \
- $(call echo-cmd,cc_o_c) $(cmd_cc_o_c); \
+ $(call cmd_and_fixdep,cc_o_c) \
$(cmd_modversions) \
- $(call echo-cmd,record_mcount) \
- $(cmd_record_mcount) \
- scripts/basic/fixdep $(depfile) $@ '$(call make-cmd,cc_o_c)' > \
- $(dot-target).tmp; \
- rm -f $(depfile); \
- mv -f $(dot-target).tmp $(dot-target).cmd
+ $(call echo-cmd,record_mcount) $(cmd_record_mcount)
endef
# List module undefined symbols (or empty line if not enabled)
--
2.5.0
Kernel modules are partially linked object files with some undefined
symbols that are expected to be matched with EXPORT_SYMBOL() entries
from elsewhere.
Each .tmp_versions/*.mod file currently contains two line of text
separated by a newline character. The first line has the actual module
file name while the second line has a list of object files constituting
that module. Those files are parsed by modpost (scripts/mod/sumversion.c),
scripts/Makefile.modpost, scripts/Makefile.modsign, etc. Only the
modpost utility cares about the second line while the others retrieve
only the first line.
Therefore we can add a third line to record the list of undefined symbols
aka required EXPORT_SYMBOL() entries for each module into that file
without breaking anything. Like for the second line, symbols are separated
by a blank and the list is terminated with a newline character.
To avoid needless build overhead, the undefined symbols extraction is
performed only when CONFIG_TRIM_UNUSED_KSYMS is selected.
Signed-off-by: Nicolas Pitre <[email protected]>
Acked-by: Rusty Russell <[email protected]>
---
scripts/Makefile.build | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 2c47f9c305..f4b4320e0d 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -253,6 +253,13 @@ define rule_cc_o_c
mv -f $(dot-target).tmp $(dot-target).cmd
endef
+# List module undefined symbols (or empty line if not enabled)
+ifdef CONFIG_TRIM_UNUSED_KSYMS
+cmd_undef_syms = $(NM) $@ | sed -n 's/^ \+U //p' | xargs echo
+else
+cmd_undef_syms = echo
+endif
+
# Built-in and composite module parts
$(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
$(call cmd,force_checksrc)
@@ -263,7 +270,8 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
$(single-used-m): $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
$(call cmd,force_checksrc)
$(call if_changed_rule,cc_o_c)
- @{ echo $(@:.o=.ko); echo $@; } > $(MODVERDIR)/$(@F:.o=.mod)
+ @{ echo $(@:.o=.ko); echo $@; \
+ $(cmd_undef_syms); } > $(MODVERDIR)/$(@F:.o=.mod)
quiet_cmd_cc_lst_c = MKLST $@
cmd_cc_lst_c = $(CC) $(c_flags) -g -c -o $*.o $< && \
@@ -393,7 +401,8 @@ $(call multi_depend, $(multi-used-y), .o, -objs -y)
$(multi-used-m): FORCE
$(call if_changed,link_multi-m)
- @{ echo $(@:.o=.ko); echo $(link_multi_deps); } > $(MODVERDIR)/$(@F:.o=.mod)
+ @{ echo $(@:.o=.ko); echo $(link_multi_deps); \
+ $(cmd_undef_syms); } > $(MODVERDIR)/$(@F:.o=.mod)
$(call multi_depend, $(multi-used-m), .o, -objs -y -m)
targets += $(multi-used-y) $(multi-used-m)
--
2.5.0
Dne 14.3.2016 v 03:42 Nicolas Pitre napsal(a):
> +# Filter out exported kernel symbol names from the preprocessor output.
> +# See also __KSYM_DEPS__ in include/linux/export.h.
> +# We disable the depfile generation here, so as not to overwrite the existing
> +# depfile while fixdep is parsing it
> +flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
> +ksym_dep_filter = \
> + case "$(1)" in \
> + cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
> + as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
> + cpp_lds_S) : ;; \
> + *) echo "Don't know how to preprocess $(1)" >&2; false ;; \
> + esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'
> +
> +cmd_and_fixdep = \
> + $(echo-cmd) $(cmd_$(1)); \
> + $(ksym_dep_filter) | \
> + scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \
> + > $(dot-target).tmp; \
> + rm -f $(depfile); \
> + mv -f $(dot-target).tmp $(dot-target).cmd;
> +
> +endif
Not sure what happened this time, but I got
drivers/md/.dm-round-robin.mod.o.cmd:5: *** unterminated call to
function 'wildcard': missing ')'. Stop.
The depfile is indeed corrupt (truncated):
tail drivers/md/.dm-round-robin.mod.o.cmd
arch/x86/include/asm/disabled-features.h \
$(wildcard include/config/x86/intel/mpx.h) \
arch/x86/include/asm/rmwcc.h \
arch/x86/include/asm/barrier.h \
$(wildcard include/config/x86/ppro/fence.h) \
arch/x86/include/asm/nops.h \
$(wildcard include/config/mk7.h) \
include/asm-generic/barrier.h \
include/asm-generic/bitops/find.h \
$(wildcard include/config/generic/
The file is exactly 8kB long:
$ ls -l drivers/md/.dm-round-robin.mod.o.cmd
-rw-r--r-- 1 mmarek users 8192 Mar 8 13:33
drivers/md/.dm-round-robin.mod.o.cmd
Michal
Dne 15.3.2016 v 21:33 Michal Marek napsal(a):
> Dne 14.3.2016 v 03:42 Nicolas Pitre napsal(a):
>> +# Filter out exported kernel symbol names from the preprocessor output.
>> +# See also __KSYM_DEPS__ in include/linux/export.h.
>> +# We disable the depfile generation here, so as not to overwrite the existing
>> +# depfile while fixdep is parsing it
>> +flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
>> +ksym_dep_filter = \
>> + case "$(1)" in \
>> + cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
>> + as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
>> + cpp_lds_S) : ;; \
>> + *) echo "Don't know how to preprocess $(1)" >&2; false ;; \
>> + esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'
>> +
>> +cmd_and_fixdep = \
>> + $(echo-cmd) $(cmd_$(1)); \
>> + $(ksym_dep_filter) | \
>> + scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \
>> + > $(dot-target).tmp; \
>> + rm -f $(depfile); \
>> + mv -f $(dot-target).tmp $(dot-target).cmd;
>> +
>> +endif
>
> Not sure what happened this time, but I got
>
> drivers/md/.dm-round-robin.mod.o.cmd:5: *** unterminated call to
> function 'wildcard': missing ')'. Stop.
Forgot to add: This was an allmodconfig build without
CONFIG_TRIM_UNUSED_SYMS.
Michal
On Tue, 15 Mar 2016, Michal Marek wrote:
> Dne 14.3.2016 v 03:42 Nicolas Pitre napsal(a):
> > +# Filter out exported kernel symbol names from the preprocessor output.
> > +# See also __KSYM_DEPS__ in include/linux/export.h.
> > +# We disable the depfile generation here, so as not to overwrite the existing
> > +# depfile while fixdep is parsing it
> > +flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
> > +ksym_dep_filter = \
> > + case "$(1)" in \
> > + cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
> > + as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
> > + cpp_lds_S) : ;; \
> > + *) echo "Don't know how to preprocess $(1)" >&2; false ;; \
> > + esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'
> > +
> > +cmd_and_fixdep = \
> > + $(echo-cmd) $(cmd_$(1)); \
> > + $(ksym_dep_filter) | \
> > + scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \
> > + > $(dot-target).tmp; \
> > + rm -f $(depfile); \
> > + mv -f $(dot-target).tmp $(dot-target).cmd;
> > +
> > +endif
>
> Not sure what happened this time, but I got
>
> drivers/md/.dm-round-robin.mod.o.cmd:5: *** unterminated call to
> function 'wildcard': missing ')'. Stop.
>
> The depfile is indeed corrupt (truncated):
> tail drivers/md/.dm-round-robin.mod.o.cmd
> arch/x86/include/asm/disabled-features.h \
> $(wildcard include/config/x86/intel/mpx.h) \
> arch/x86/include/asm/rmwcc.h \
> arch/x86/include/asm/barrier.h \
> $(wildcard include/config/x86/ppro/fence.h) \
> arch/x86/include/asm/nops.h \
> $(wildcard include/config/mk7.h) \
> include/asm-generic/barrier.h \
> include/asm-generic/bitops/find.h \
> $(wildcard include/config/generic/
>
> The file is exactly 8kB long:
> $ ls -l drivers/md/.dm-round-robin.mod.o.cmd
> -rw-r--r-- 1 mmarek users 8192 Mar 8 13:33
> drivers/md/.dm-round-robin.mod.o.cmd
[...]
> Forgot to add: This was an allmodconfig build without
> CONFIG_TRIM_UNUSED_SYMS.
If CONFIG_TRIM_UNUSED_SYMS is actually not selected, then I really don't
see how my changes can be involved in this failure.
And I cannot reproduce here so far. I didn't receive anything from
kbuild test robot since February 15 either. Is this something that you
are able to reproduce on your side?
If so, do you get suspicious output on stderr during the build? Maybe
fixdep exits prematurelywith "fixdep: bad data on stdin"? However, that
cannot happen if CONFIG_TRIM_UNUSED_SYMS=n.
Did you test my git branch, or did you apply the posted patches to your
tree? If the later, could you publish a branch for it so I could test
the same tree as you do?
Nicolas
On 2016-03-16 00:14, Nicolas Pitre wrote:
> On Tue, 15 Mar 2016, Michal Marek wrote:
>
>> Dne 14.3.2016 v 03:42 Nicolas Pitre napsal(a):
>>> +# Filter out exported kernel symbol names from the preprocessor output.
>>> +# See also __KSYM_DEPS__ in include/linux/export.h.
>>> +# We disable the depfile generation here, so as not to overwrite the existing
>>> +# depfile while fixdep is parsing it
>>> +flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
>>> +ksym_dep_filter = \
>>> + case "$(1)" in \
>>> + cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
>>> + as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
>>> + cpp_lds_S) : ;; \
>>> + *) echo "Don't know how to preprocess $(1)" >&2; false ;; \
>>> + esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'
>>> +
>>> +cmd_and_fixdep = \
>>> + $(echo-cmd) $(cmd_$(1)); \
>>> + $(ksym_dep_filter) | \
>>> + scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \
>>> + > $(dot-target).tmp; \
>>> + rm -f $(depfile); \
>>> + mv -f $(dot-target).tmp $(dot-target).cmd;
>>> +
>>> +endif
>>
>> Not sure what happened this time, but I got
>>
>> drivers/md/.dm-round-robin.mod.o.cmd:5: *** unterminated call to
>> function 'wildcard': missing ')'. Stop.
>>
>> The depfile is indeed corrupt (truncated):
>> tail drivers/md/.dm-round-robin.mod.o.cmd
>> arch/x86/include/asm/disabled-features.h \
>> $(wildcard include/config/x86/intel/mpx.h) \
>> arch/x86/include/asm/rmwcc.h \
>> arch/x86/include/asm/barrier.h \
>> $(wildcard include/config/x86/ppro/fence.h) \
>> arch/x86/include/asm/nops.h \
>> $(wildcard include/config/mk7.h) \
>> include/asm-generic/barrier.h \
>> include/asm-generic/bitops/find.h \
>> $(wildcard include/config/generic/
>>
>> The file is exactly 8kB long:
>> $ ls -l drivers/md/.dm-round-robin.mod.o.cmd
>> -rw-r--r-- 1 mmarek users 8192 Mar 8 13:33
>> drivers/md/.dm-round-robin.mod.o.cmd
>
> [...]
>
>> Forgot to add: This was an allmodconfig build without
>> CONFIG_TRIM_UNUSED_SYMS.
>
> If CONFIG_TRIM_UNUSED_SYMS is actually not selected, then I really don't
> see how my changes can be involved in this failure.
I have no idea either.
> And I cannot reproduce here so far. I didn't receive anything from
> kbuild test robot since February 15 either. Is this something that you
> are able to reproduce on your side?
I started a build loop now.
> If so, do you get suspicious output on stderr during the build? Maybe
> fixdep exits prematurelywith "fixdep: bad data on stdin"? However, that
> cannot happen if CONFIG_TRIM_UNUSED_SYMS=n.
>
> Did you test my git branch, or did you apply the posted patches to your
> tree? If the later, could you publish a branch for it so I could test
> the same tree as you do?
I applied the patches to my -rc1 based kbuild branch and yes, there was
a conflict with 2aedcd098a94 ("kbuild: suppress annoying "... is up to
date." message"). So it's possible that I screwed it up. I pushed the
branch to
git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild.git
tmp.autoksyms6
Michal
On Wed, 16 Mar 2016, Michal Marek wrote:
> On 2016-03-16 00:14, Nicolas Pitre wrote:
> > On Tue, 15 Mar 2016, Michal Marek wrote:
> >
> >> Dne 14.3.2016 v 03:42 Nicolas Pitre napsal(a):
> >>> +# Filter out exported kernel symbol names from the preprocessor output.
> >>> +# See also __KSYM_DEPS__ in include/linux/export.h.
> >>> +# We disable the depfile generation here, so as not to overwrite the existing
> >>> +# depfile while fixdep is parsing it
> >>> +flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
> >>> +ksym_dep_filter = \
> >>> + case "$(1)" in \
> >>> + cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
> >>> + as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
> >>> + cpp_lds_S) : ;; \
> >>> + *) echo "Don't know how to preprocess $(1)" >&2; false ;; \
> >>> + esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'
> >>> +
> >>> +cmd_and_fixdep = \
> >>> + $(echo-cmd) $(cmd_$(1)); \
> >>> + $(ksym_dep_filter) | \
> >>> + scripts/basic/fixdep -e $(depfile) $@ '$(make-cmd)' \
> >>> + > $(dot-target).tmp; \
> >>> + rm -f $(depfile); \
> >>> + mv -f $(dot-target).tmp $(dot-target).cmd;
> >>> +
> >>> +endif
> >>
> >> Not sure what happened this time, but I got
> >>
> >> drivers/md/.dm-round-robin.mod.o.cmd:5: *** unterminated call to
> >> function 'wildcard': missing ')'. Stop.
> >>
> >> The depfile is indeed corrupt (truncated):
> >> tail drivers/md/.dm-round-robin.mod.o.cmd
> >> arch/x86/include/asm/disabled-features.h \
> >> $(wildcard include/config/x86/intel/mpx.h) \
> >> arch/x86/include/asm/rmwcc.h \
> >> arch/x86/include/asm/barrier.h \
> >> $(wildcard include/config/x86/ppro/fence.h) \
> >> arch/x86/include/asm/nops.h \
> >> $(wildcard include/config/mk7.h) \
> >> include/asm-generic/barrier.h \
> >> include/asm-generic/bitops/find.h \
> >> $(wildcard include/config/generic/
> >>
> >> The file is exactly 8kB long:
> >> $ ls -l drivers/md/.dm-round-robin.mod.o.cmd
> >> -rw-r--r-- 1 mmarek users 8192 Mar 8 13:33
> >> drivers/md/.dm-round-robin.mod.o.cmd
> >
> > [...]
> >
> >> Forgot to add: This was an allmodconfig build without
> >> CONFIG_TRIM_UNUSED_SYMS.
> >
> > If CONFIG_TRIM_UNUSED_SYMS is actually not selected, then I really don't
> > see how my changes can be involved in this failure.
>
> I have no idea either.
>
>
> > And I cannot reproduce here so far. I didn't receive anything from
> > kbuild test robot since February 15 either. Is this something that you
> > are able to reproduce on your side?
>
> I started a build loop now.
>
>
> > If so, do you get suspicious output on stderr during the build? Maybe
> > fixdep exits prematurelywith "fixdep: bad data on stdin"? However, that
> > cannot happen if CONFIG_TRIM_UNUSED_SYMS=n.
> >
> > Did you test my git branch, or did you apply the posted patches to your
> > tree? If the later, could you publish a branch for it so I could test
> > the same tree as you do?
>
> I applied the patches to my -rc1 based kbuild branch and yes, there was
> a conflict with 2aedcd098a94 ("kbuild: suppress annoying "... is up to
> date." message"). So it's possible that I screwed it up. I pushed the
> branch to
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild.git
> tmp.autoksyms6
As far as I can see, the conflict resolution is correct.
Testing your branch now. No failure so far.
I did notice spurious error messages after cleaning my build tree.
They're unrelated to your failure case though. Regardless, here's a
patch to silence them.
----- >8
From: Nicolas Pitre <[email protected]>
Subject: kbuild: don't complain about existing if_changed_dep targets
There are more if_changed_dep cases for which we know there is no
ksyms to parse. Add them to the list.
Also add a full stop to the comment.
Signed-off-by: Nicolas Pitre <[email protected]>
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index f8631a935e..a09927e027 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -271,13 +271,13 @@ else
# Filter out exported kernel symbol names from the preprocessor output.
# See also __KSYM_DEPS__ in include/linux/export.h.
# We disable the depfile generation here, so as not to overwrite the existing
-# depfile while fixdep is parsing it
+# depfile while fixdep is parsing it.
flags_nodeps = $(filter-out -Wp$(comma)-M%, $($(1)))
ksym_dep_filter = \
case "$(1)" in \
cc_*_c) $(CPP) $(call flags_nodeps,c_flags) -D__KSYM_DEPS__ $< ;; \
as_*_S) $(CPP) $(call flags_nodeps,a_flags) -D__KSYM_DEPS__ $< ;; \
- cpp_lds_S) : ;; \
+ boot*|build*|*cpp_lds_S|dtc|host*|vdso*) : ;; \
*) echo "Don't know how to preprocess $(1)" >&2; false ;; \
esac | sed -rn 's/^.*=== __KSYM_(.*) ===.*$$/KSYM_\1/p'