From: Luis R. Rodriguez <[email protected]>
Hauke reported that parallel building was borked with the latest
autoconf fixes on compat. This series addresses that but also optimizes
things a bit further now now that I've gone down the rabbit hole of
reviewing make -d and also studying GNU Make's jobserver [0]. The thing
that impresses me was the inherent expectation and support by GNU Make
on recursion when you'd expect it to be supported. Unfortunately none
of this was documented well but I hope this series and my rants helps
with spreading some of the documentation love on this.
[0] http://mcgrof.blogspot.com/2012/02/gnu-things-are-gnu-simplicity-of-gnu.html
Luis R. Rodriguez (7):
compat: fix parallel builds
compat: move definition of COMPAT_CONFIG and COMPAT_AUTOCONF
compat: explicitly define paths for local configs
compat: remove dubious clean at install target
compat: split COMPAT_CONFIG and COMPAT_AUTOCONF targets
compat: avoid unnecessary recursion to include a file
compat: optimize building by extending .PHONY
Makefile | 27 +++++++++++----------------
1 files changed, 11 insertions(+), 16 deletions(-)
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
The Makefile for external modules are read twice, once during
the initial make command, and then later to build the modules
target *by the kernel*. This ensures that we define the variables
COMPAT_CONFIG and COMPAT_AUTOCONF are only defined once.
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/Makefile b/Makefile
index d11da17..287f35f 100644
--- a/Makefile
+++ b/Makefile
@@ -19,6 +19,11 @@ export COMPAT_BASE_TREE := "linux-next.git"
export COMPAT_BASE_TREE_VERSION := "next-20100517"
export COMPAT_VERSION := $(shell git describe)
+# to check config and compat autoconf
+export COMPAT_CONFIG=.config
+export COMPAT_AUTOCONF=include/linux/compat_autoconf.h
+export MAKE
+
else
# By stuffing this hear we avoid using
# this hackery on modpost, the 2nd section of module building.
@@ -36,11 +41,6 @@ NOSTDINC_FLAGS := -I$(M)/include/ \
endif
-# to check config and compat autoconf
-export COMPAT_CONFIG=.config
-export COMPAT_AUTOCONF=include/linux/compat_autoconf.h
-export MAKE
-
# Recursion lets us ensure we get this file included.
# Trick is to run make -C $(PWD) modules later.
-include $(PWD)/$(COMPAT_CONFIG)
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
I've started reading make -d output to realize that GNU Make
is pretty simple but darn stupid and needs a little help in
direction. Traditionally GNU Make will try to see if it needs
Makefile* generation. I can count 18 variations of Makefile it
looks for:
Makefile
Makefile.c
Makefile.C
Makefile.cc
Makefile.cpp
Makefile.f
Makefile.F
Makefile.l
Makefile.mod
Makefile.o
Makefile.p
Makefile.r
Makefile.s
Makefile.S
Makefile.sh
Makefile.w
Makefile.web
Makefile.y
GNU Make will look for targets in the Makefile for each of these...
GNU Make will also look for these 18 variations of files for each
target that it does not a target rule for. There are two ways to
help GNU Make do the right thing, write a simple empty target rule [0]
or just extend the target into the .PHONY target. Lets just extend
the .PHONY target with the items we see in make -d are not needed.
This simple change forces GNU Make to skip 36 lookups of false
targets.
[0] Makefile : ;
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index 9e676ed..82d4cbb 100644
--- a/Makefile
+++ b/Makefile
@@ -69,4 +69,4 @@ clean:
clean-files := Module.symvers modules.order Module.markers compat/modules.order
clean-files += modules $(COMPAT_CONFIG) $(COMPAT_AUTOCONF)
-.PHONY: all install clean
+.PHONY: all install clean modules Makefile
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
Not sure why this was added, this is a big typo.
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index 423d1ed..2f47e3f 100644
--- a/Makefile
+++ b/Makefile
@@ -57,7 +57,6 @@ install: modules
$(MAKE) -C $(KLIB_BUILD) M=$(PWD) $(KMODDIR_ARG) $(KMODPATH_ARG) \
modules_install
depmod -a
- $(MAKE) -C $(KLIB_BUILD) M=$(PWD) clean
$(COMPAT_AUTOCONF): ;
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
I added recursion call to the same Makefile after noticing that
GNU Make *will* fail if a file that is being included does not
exist. I also added the option that the file *may* not exist by
prepending the inclusion of the file with "-". It turns out that
GNU Make is smart enough to look for targets for header files that
are included and *will not fail* if it can successfully build that
file and include it. I will note that this target file *does not
need* to be a dependency to any of the final targets, GNU Make
will just assume and add it. It is important to highlight that
GNU Make *will* run make against itself again after it builds the
file it needs to include. If GNU Make runs into this situation,
where it can build the target file it needs to include, it will
not fail but you will see something like this pesky warning:
Makefile:16: /home/mcgrof/compat/.config: No such file or directory
Under new found knowledge of how GNU Make works we simplify the
reading and running of the compat Makefile by ensuring that the
file we need to include is defined as a target but for sanity and
reader's sake (although technically not necessary) we also add the
file as a dependency to the modules target building.
Furthermore the pesky warning can confuse developers / users and
as it turns out we only really need it at build time. We take
advantage of the fact that the kernel will use the same Makefile
later upon building the external module and that we can identify
when this happens in the Makefile [0] and only *require* including
the header file upon module building time.
[0] the part where ifeq ($(KERNELRELEASE),) is false
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 8 ++------
1 files changed, 2 insertions(+), 6 deletions(-)
diff --git a/Makefile b/Makefile
index 537d759..9e676ed 100644
--- a/Makefile
+++ b/Makefile
@@ -39,15 +39,12 @@ NOSTDINC_FLAGS := -I$(M)/include/ \
-DCOMPAT_PROJECT="\"Generic kernel\"" \
-DCOMPAT_VERSION="\"$(COMPAT_VERSION)\""
+include $(COMPAT_CONFIG)
endif
-# Recursion lets us ensure we get this file included.
-# Trick is to run make -C $(PWD) modules later.
--include $(COMPAT_CONFIG)
-
obj-y += compat/
-all: $(COMPAT_CONFIG)
+all: modules
modules: $(COMPAT_CONFIG) $(COMPAT_AUTOCONF)
$(MAKE) -C $(KLIB_BUILD) M=$(PWD) modules
@@ -63,7 +60,6 @@ $(COMPAT_AUTOCONF): $(COMPAT_CONFIG)
$(COMPAT_CONFIG):
+@$(PWD)/scripts/gen-compat-config.sh > $(COMPAT_CONFIG)
- @$(MAKE) -C $(PWD) modules
install: modules
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
Let GNU Make take care of the dependency map.
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 2f47e3f..537d759 100644
--- a/Makefile
+++ b/Makefile
@@ -58,11 +58,11 @@ install: modules
modules_install
depmod -a
-$(COMPAT_AUTOCONF): ;
+$(COMPAT_AUTOCONF): $(COMPAT_CONFIG)
+ +@$(PWD)/scripts/gen-compat-autoconf.sh $(COMPAT_CONFIG) > $(COMPAT_AUTOCONF)
$(COMPAT_CONFIG):
+@$(PWD)/scripts/gen-compat-config.sh > $(COMPAT_CONFIG)
- +@$(PWD)/scripts/gen-compat-autoconf.sh $(COMPAT_CONFIG) > $(COMPAT_AUTOCONF)
@$(MAKE) -C $(PWD) modules
install: modules
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
We broke parallel builds by breaking GNU Make's heuristics.
GNU Make does not whether or not target commands are sub-make
commands or not, these must be explicitly annotated. To tell
GNU Make that target command is going to be a sub-process
we can use the $(MAKE) variable for a command. Another option
is to tell GNU Make that a target command is *not* a sub-make
command instead explicitly by prepending a command with a plus,
"+" prior to the command. This will force GNU Make to avoid
supporting certain flags for those targets but more importantly,
it will also ensure that during execution of commands that are
*not* sub-make commands it will yield the jobserver to ensure
file descriptor / job sanity.
Annotate to GNU Make two commands that we issue are *not* sub-make
commands. This avoids stop the job server for these commands but
also allows GNU Make to figure out the heuristics to run the job
server for further sub-make commands -- in this case the recursive
call to make.
Reported-by: Hauke Mehrtens <[email protected]>
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index d96d098..d11da17 100644
--- a/Makefile
+++ b/Makefile
@@ -62,8 +62,8 @@ install: modules
$(COMPAT_AUTOCONF): ;
$(COMPAT_CONFIG):
- @$(PWD)/scripts/gen-compat-config.sh > $(PWD)/$(COMPAT_CONFIG)
- @$(PWD)/scripts/gen-compat-autoconf.sh $(COMPAT_CONFIG) > $(PWD)/$(COMPAT_AUTOCONF)
+ +@$(PWD)/scripts/gen-compat-config.sh > $(PWD)/$(COMPAT_CONFIG)
+ +@$(PWD)/scripts/gen-compat-autoconf.sh $(COMPAT_CONFIG) > $(PWD)/$(COMPAT_AUTOCONF)
@$(MAKE) -C $(PWD) modules
install: modules
--
1.7.4.15.g7811d
From: Luis R. Rodriguez <[email protected]>
As noted in the previous commit a Linux kernel module's
Makefile will be read twice, during initial GNU make run,
and later for when the kernel will build the external module.
Variables that require path consideration must address in
which context it needs to be defined and do so by detecting
at which run time case it wants to instantiate variables. In
our case we want to ensure that both COMPAT_CONFIG and
COMPAT_AUTOCONF are defined within the context of the
directory of the external module we are building as otherwise
GNU Make will try to treat it as part of the kernel's build
directory's files.
We already have COMPAT_CONFIG and COMPAT_AUTOCONF defined
under a ifeq ($(KERNELRELEASE),) check, we now just need
to add the directory context and then remove the other
now superfluous uses of the $(PWD).
Signed-off-by: Luis R. Rodriguez <[email protected]>
---
Makefile | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/Makefile b/Makefile
index 287f35f..423d1ed 100644
--- a/Makefile
+++ b/Makefile
@@ -20,8 +20,8 @@ export COMPAT_BASE_TREE_VERSION := "next-20100517"
export COMPAT_VERSION := $(shell git describe)
# to check config and compat autoconf
-export COMPAT_CONFIG=.config
-export COMPAT_AUTOCONF=include/linux/compat_autoconf.h
+export COMPAT_CONFIG=$(PWD)/.config
+export COMPAT_AUTOCONF=$(PWD)/include/linux/compat_autoconf.h
export MAKE
else
@@ -43,7 +43,7 @@ endif
# Recursion lets us ensure we get this file included.
# Trick is to run make -C $(PWD) modules later.
--include $(PWD)/$(COMPAT_CONFIG)
+-include $(COMPAT_CONFIG)
obj-y += compat/
@@ -62,8 +62,8 @@ install: modules
$(COMPAT_AUTOCONF): ;
$(COMPAT_CONFIG):
- +@$(PWD)/scripts/gen-compat-config.sh > $(PWD)/$(COMPAT_CONFIG)
- +@$(PWD)/scripts/gen-compat-autoconf.sh $(COMPAT_CONFIG) > $(PWD)/$(COMPAT_AUTOCONF)
+ +@$(PWD)/scripts/gen-compat-config.sh > $(COMPAT_CONFIG)
+ +@$(PWD)/scripts/gen-compat-autoconf.sh $(COMPAT_CONFIG) > $(COMPAT_AUTOCONF)
@$(MAKE) -C $(PWD) modules
install: modules
--
1.7.4.15.g7811d