These two patches should speed up the make deb-pkg build process by utilizing
the make jobserver properly, and using 'find | xargs' to parallelize debug
module installation.
I've tested with the resulting packages to ensure the symbols work as intended
by crash dumping a machine and using the dbg package symbols.
Using an Ubuntu distro conf + make olddefconfig on a 56-core machine, I can see
the following improvements (make clean && make deb-pkg -j`nproc`):
* Before
real 40m56.240s
user 182m17.292s
sys 140m18.812s
* After
real 36m22.633s
user 182m28.028s
sys 148m17.724s
Chris J Arges (2):
package: Makefile: ensure $MAKE can use jobserver
builddeb: parallelize debug module installation
scripts/package/Makefile | 2 +-
scripts/package/builddeb | 15 ++++++++-------
2 files changed, 9 insertions(+), 8 deletions(-)
--
1.9.1
When using make deb-pkg, builddeb is called without proper MAKEFLAGS due to the
script being invoked without '+'. This results in the following message when
building:
warning: jobserver unavailable: using -j1. Add `+' to parent make rule
Add the '+' so the make operations can be parallelized.
Signed-off-by: Chris J Arges <[email protected]>
---
scripts/package/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 99ca6e7..0dbfae7 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -88,7 +88,7 @@ quiet_cmd_builddeb = BUILDDEB
deb-pkg: FORCE
$(MAKE) KBUILD_SRC=
- $(call cmd,builddeb)
+ +$(call cmd,builddeb)
clean-dirs += $(objtree)/debian/
--
1.9.1
When building the dbg package, we use a large 'for module in $(find' loop that
can be easily parallelized by using 'find | xargs'. This patch modifies this
loop to use the later paradigm.
Signed-off-by: Chris J Arges <[email protected]>
---
scripts/package/builddeb | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 88dbf23..d12d062 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
rmdir "$tmpdir/lib/modules/$version"
fi
if [ -n "$BUILD_DEBUG" ] ; then
- for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
- module=lib/modules/$module
- mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
+ find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -I {} sh -c '
+ mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
# only keep debug symbols in the debug file
- $OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
+ $OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
+ $dbg_dir/usr/lib/debug/lib/modules/{};
# strip original module from debug symbols
- $OBJCOPY --strip-debug $tmpdir/$module
+ $OBJCOPY --strip-debug $tmpdir/lib/modules/{};
# then add a link to those
- $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
- done
+ $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
+ $tmpdir/lib/modules/{};
+ " -- {}
fi
fi
--
1.9.1
On 2015-04-24 20:25, Chris J Arges wrote:
> When building the dbg package, we use a large 'for module in $(find' loop that
> can be easily parallelized by using 'find | xargs'. This patch modifies this
> loop to use the later paradigm.
>
> Signed-off-by: Chris J Arges <[email protected]>
> ---
> scripts/package/builddeb | 15 ++++++++-------
> 1 file changed, 8 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index 88dbf23..d12d062 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
> rmdir "$tmpdir/lib/modules/$version"
> fi
> if [ -n "$BUILD_DEBUG" ] ; then
> - for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> - module=lib/modules/$module
> - mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> + find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -I {} sh -c '
I guess you want to use the -P option here.
Michal
When building the dbg package, we use a large 'for module in $(find' loop that
can be easily parallelized by using 'find | xargs'. This patch modifies this
loop to use the later paradigm.
In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
possible.
Signed-off-by: Chris J Arges <[email protected]>
---
scripts/package/builddeb | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 88dbf23..538f829 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
rmdir "$tmpdir/lib/modules/$version"
fi
if [ -n "$BUILD_DEBUG" ] ; then
- for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
- module=lib/modules/$module
- mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
+ find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
+ mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
# only keep debug symbols in the debug file
- $OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
+ $OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
+ $dbg_dir/usr/lib/debug/lib/modules/{};
# strip original module from debug symbols
- $OBJCOPY --strip-debug $tmpdir/$module
+ $OBJCOPY --strip-debug $tmpdir/lib/modules/{};
# then add a link to those
- $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
- done
+ $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
+ $tmpdir/lib/modules/{};
+ " -- {}
fi
fi
--
1.9.1
On 24 April 2015 at 21:25, Chris J Arges <[email protected]> wrote:
> When using make deb-pkg, builddeb is called without proper MAKEFLAGS due to the
> script being invoked without '+'. This results in the following message when
> building:
> warning: jobserver unavailable: using -j1. Add `+' to parent make rule
>
> Add the '+' so the make operations can be parallelized.
Looks good, and works fine in my tests.
Reviewed-by: Riku Voipio <[email protected]>
> Signed-off-by: Chris J Arges <[email protected]>
> ---
> scripts/package/Makefile | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/package/Makefile b/scripts/package/Makefile
> index 99ca6e7..0dbfae7 100644
> --- a/scripts/package/Makefile
> +++ b/scripts/package/Makefile
> @@ -88,7 +88,7 @@ quiet_cmd_builddeb = BUILDDEB
>
> deb-pkg: FORCE
> $(MAKE) KBUILD_SRC=
> - $(call cmd,builddeb)
> + +$(call cmd,builddeb)
>
> clean-dirs += $(objtree)/debian/
>
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On 27 April 2015 at 19:43, Chris J Arges <[email protected]> wrote:
> When building the dbg package, we use a large 'for module in $(find' loop that
> can be easily parallelized by using 'find | xargs'. This patch modifies this
> loop to use the later paradigm.
>
> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
> possible.
>
> Signed-off-by: Chris J Arges <[email protected]>
> ---
> scripts/package/builddeb | 15 ++++++++-------
> 1 file changed, 8 insertions(+), 7 deletions(-)
>
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index 88dbf23..538f829 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
> rmdir "$tmpdir/lib/modules/$version"
> fi
> if [ -n "$BUILD_DEBUG" ] ; then
> - for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> - module=lib/modules/$module
> - mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> + find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
of modules (allmodconfig will make 4500).
> + mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
> # only keep debug symbols in the debug file
> - $OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
> + $OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
> + $dbg_dir/usr/lib/debug/lib/modules/{};
> # strip original module from debug symbols
> - $OBJCOPY --strip-debug $tmpdir/$module
> + $OBJCOPY --strip-debug $tmpdir/lib/modules/{};
> # then add a link to those
> - $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
> - done
> + $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
> + $tmpdir/lib/modules/{};
> + " -- {}
> fi
> fi
>
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2015-04-28 10:57, Riku Voipio wrote:
> On 27 April 2015 at 19:43, Chris J Arges <[email protected]> wrote:
>> When building the dbg package, we use a large 'for module in $(find' loop that
>> can be easily parallelized by using 'find | xargs'. This patch modifies this
>> loop to use the later paradigm.
>>
>> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
>> possible.
>>
>> Signed-off-by: Chris J Arges <[email protected]>
>> ---
>> scripts/package/builddeb | 15 ++++++++-------
>> 1 file changed, 8 insertions(+), 7 deletions(-)
>>
>> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
>> index 88dbf23..538f829 100755
>> --- a/scripts/package/builddeb
>> +++ b/scripts/package/builddeb
>> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
>> rmdir "$tmpdir/lib/modules/$version"
>> fi
>> if [ -n "$BUILD_DEBUG" ] ; then
>> - for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
>> - module=lib/modules/$module
>> - mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
>> + find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
>
> I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
> of modules (allmodconfig will make 4500).
Yep. I was thinking about retrieving the value of the make -j argument
somehow, but this is not possible and we certainly do not want to
implement the jobserver protocol in shell :-). So using the number of
processors is a sensible choice. What can be done is to detect whether
the -j option is in MAKEFLAGS and only then use multiple instances.
Michal
On Tue, Apr 28, 2015 at 12:05:19PM +0200, Michal Marek wrote:
> On 2015-04-28 10:57, Riku Voipio wrote:
> > On 27 April 2015 at 19:43, Chris J Arges <[email protected]> wrote:
> >> When building the dbg package, we use a large 'for module in $(find' loop that
> >> can be easily parallelized by using 'find | xargs'. This patch modifies this
> >> loop to use the later paradigm.
> >>
> >> In addition, ensure we add '-n1 -P0' to xargs to run as many processes as
> >> possible.
> >>
> >> Signed-off-by: Chris J Arges <[email protected]>
> >> ---
> >> scripts/package/builddeb | 15 ++++++++-------
> >> 1 file changed, 8 insertions(+), 7 deletions(-)
> >>
> >> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> >> index 88dbf23..538f829 100755
> >> --- a/scripts/package/builddeb
> >> +++ b/scripts/package/builddeb
> >> @@ -152,16 +152,17 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
> >> rmdir "$tmpdir/lib/modules/$version"
> >> fi
> >> if [ -n "$BUILD_DEBUG" ] ; then
> >> - for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
> >> - module=lib/modules/$module
> >> - mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
> >> + find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P0 -I {} sh -c '
> >
> > I would go with -P`getconf _NPROCESSORS_ONLN`. There can be thousands
> > of modules (allmodconfig will make 4500).
>
> Yep. I was thinking about retrieving the value of the make -j argument
> somehow, but this is not possible and we certainly do not want to
> implement the jobserver protocol in shell :-). So using the number of
> processors is a sensible choice. What can be done is to detect whether
> the -j option is in MAKEFLAGS and only then use multiple instances.
>
> Michal
>
Michal,
The MAKEFLAGS variable will have the jobserver fd's in it, if we are building
in parallel with -j. I could use this to detect if we are building in parallel
and adjust the '-P' flag in xargs accordingly. I'll work on v3 of this patch,
thanks for the reviews!
--chris
When building the dbg package, we use a large 'for module in $(find' loop that
can be easily parallelized by using 'find | xargs'. This patch modifies this
loop to use the later paradigm.
In addition, check if the user has requested a parallel build with make. If so,
add the appropriate flags to xargs to set MAXPROCS (-P) equal to the number of
processing units on the system.
Signed-off-by: Chris J Arges <[email protected]>
---
scripts/package/builddeb | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 88dbf23..5849f0c 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -152,16 +152,21 @@ if grep -q '^CONFIG_MODULES=y' $KCONFIG_CONFIG ; then
rmdir "$tmpdir/lib/modules/$version"
fi
if [ -n "$BUILD_DEBUG" ] ; then
- for module in $(find $tmpdir/lib/modules/ -name *.ko -printf '%P\n'); do
- module=lib/modules/$module
- mkdir -p $(dirname $dbg_dir/usr/lib/debug/$module)
+ # If we've invoked make with -j, then parallelize; otherwise
+ # just use a single process.
+ procs=1
+ test "${MAKEFLAGS#*-j}" != "${MAKEFLAGS}" && procs=`getconf _NPROCESSORS_ONLN`
+ find $tmpdir/lib/modules/ -name *.ko -printf '%P\n' | xargs -n1 -P${procs} -I {} sh -c '
+ mkdir -p $(dirname '"$dbg_dir"'/usr/lib/debug/lib/modules/$1);'"
# only keep debug symbols in the debug file
- $OBJCOPY --only-keep-debug $tmpdir/$module $dbg_dir/usr/lib/debug/$module
+ $OBJCOPY --only-keep-debug $tmpdir/lib/modules/{} \
+ $dbg_dir/usr/lib/debug/lib/modules/{};
# strip original module from debug symbols
- $OBJCOPY --strip-debug $tmpdir/$module
+ $OBJCOPY --strip-debug $tmpdir/lib/modules/{};
# then add a link to those
- $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/$module $tmpdir/$module
- done
+ $OBJCOPY --add-gnu-debuglink=$dbg_dir/usr/lib/debug/lib/modules/{} \
+ $tmpdir/lib/modules/{};
+ " -- {}
fi
fi
--
1.9.1