The following commit, ad7a953c, "kbuild: strip generated symbols from
*.ko" which was merged to the mainline last week (December 28th) has the
unfortunate side effct of making ccache useless. That's because all
files are now built using a two step process. First they are compiled
to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a
separate step. Unfortunately, ccache doesn't seem to be able to cache
either the compilation or the assembly step (and it may not be worth it
once the two steps are separated).
It would be nice if there was a CONFIG option which compiles files the
old-fashioned way, even if we lose the ability to strip the__crc_
symbols, just so that ccache can function again, since it can really
speed up the kernel development process. Alternatively, maybe ccache
could somehow be taught a specialized way take the .c file, some kind of
kbuild-specified version identifier, and then output of the gcc -S,
assembly frobnication, gcc -c, objcopy pipeline that kbuild uses to
build an object file, so we can have cache the entire kbuild object
building process.
Or, if that's too complicated, maybe it would be worthwhile to have
kbuild create its own specialized ccache system? Note that the last two
solutions rule out using distcc, unless we can encapsulate the build
process from a series of Makefile macros to a shell or C program, which
could then be injected to the remote host system to be executed by
distcc. One value of doing that is the CRC or MD5 of the shell script
could be used as the version tag for the cache system.
In the short term, though, it would be nice if we could get back a
simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?
Thanks,
- Ted
>>> "Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>
>In the short term, though, it would be nice if we could get back a
>simple way of making a kernel object file using just cc, so that ccache
>and distcc could be functional again. Does that seem reasonable?
Making the new logic dependent on a config option would seem reasonable
to me - of course at the expense of the respective Makefile becoming
even less readable.
Jan
From: "Theodore Ts'o" <[email protected]>
Date: Tue, 06 Jan 2009 10:15:26 -0500
> The following commit, ad7a953c, "kbuild: strip generated symbols from
> *.ko" which was merged to the mainline last week (December 28th) has the
> unfortunate side effct of making ccache useless. That's because all
> files are now built using a two step process. First they are compiled
> to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a
> separate step. Unfortunately, ccache doesn't seem to be able to cache
> either the compilation or the assembly step (and it may not be worth it
> once the two steps are separated).
I think the problem might be that the assembler source always
"changes" because a different date or timestamp string is put in there
by gcc every time the assembler source is built.
On Tue, Jan 06, 2009 at 03:29:35PM +0000, Jan Beulich wrote:
> >>> "Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>
> >In the short term, though, it would be nice if we could get back a
> >simple way of making a kernel object file using just cc, so that ccache
> >and distcc could be functional again. Does that seem reasonable?
>
> Making the new logic dependent on a config option would seem reasonable
> to me - of course at the expense of the respective Makefile becoming
> even less readable.
Too late. :-) It's pretty unreadable already.... as a result, I'm not
at all confident that I could make such a patch. Is this something
you could perhaps whip up? I'd really appreicate it, as it would
seriously speed up by kernel development efforts.
Regards,
- Ted
On Tue, Jan 06, 2009 at 12:33:00PM -0500, Theodore Tso wrote:
> On Tue, Jan 06, 2009 at 03:29:35PM +0000, Jan Beulich wrote:
> > >>> "Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>
> > >In the short term, though, it would be nice if we could get back a
> > >simple way of making a kernel object file using just cc, so that ccache
> > >and distcc could be functional again. Does that seem reasonable?
> >
> > Making the new logic dependent on a config option would seem reasonable
> > to me - of course at the expense of the respective Makefile becoming
> > even less readable.
>
> Too late. :-) It's pretty unreadable already....
I'm all ears for inputs how to make them even a little more readable.
We actually used several iteratiosn of Jan's patch just to improve
readability in areas the patch touched.
I think at least half of the unreadability comes from the fact that
people do not realise the steps needed to actually build the kernel.
It is a little bit more involved than doing a simple sequence
of gcc; ld.
We have btw. 1140 Makefiles in the kernel of a total of 28000 lines.
Yes - exactly 28000 lines as reported by:
find -name 'Makefile*' | xargs cat | wc -l
But numbers aside - any inputs appreciated.
Sam
On Tue, Jan 06, 2009 at 03:29:35PM +0000, Jan Beulich wrote:
> >>> "Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>
> >In the short term, though, it would be nice if we could get back a
> >simple way of making a kernel object file using just cc, so that ccache
> >and distcc could be functional again. Does that seem reasonable?
>
> Making the new logic dependent on a config option would seem reasonable
> to me - of course at the expense of the respective Makefile becoming
> even less readable.
We need to do this I think - sigh..
The Makefile.build bits should be easiest.
Maybe I should revisit the patch to move vmlinux link to a shell script
to make the toplevel Makefile simpler??
Sam
On Tue, Jan 06, 2009 at 06:48:25PM +0100, Sam Ravnborg wrote:
>
> I'm all ears for inputs how to make them even a little more readable.
> We actually used several iteratiosn of Jan's patch just to improve
> readability in areas the patch touched.
>
> I think at least half of the unreadability comes from the fact that
> people do not realise the steps needed to actually build the kernel.
Yeah, I thought about that a lot, but I couldn't come up with a
constructive suggestion. The best I could come up with to have kbuild
generate a shell script helper or use a shell script helper to
actually build each .o file. However, the speed hit probably makes
that unacceptable, although it would make it easier, I suspect, to use
that abstractionas part of making something which ccache or distcc
could use to cache (or distribute to a cluster) all of the steps
needed to build each object file. Maybe a helper C program instead?
- Ted
On Tue, Jan 06, 2009 at 08:26:19AM -0800, David Miller wrote:
> From: "Theodore Ts'o" <[email protected]>
> Date: Tue, 06 Jan 2009 10:15:26 -0500
>
> I think the problem might be that the assembler source always
> "changes" because a different date or timestamp string is put in there
> by gcc every time the assembler source is built.
I'm not sure that's the problem, actually. Running "ccache -s" doesn't
show any additional cache misses *or* hits. Instead the only
statistic which gets bumped is the "called for link" counter. So I
suspect it's because ccache doesn't attempt to cache "gcc -S" or "gcc
-c foo.s" operations. So it might be the case that a change to ccache
to cache compile-to-assemble and assemble-to-object code
transformation would help, but then we would still end up calling
ccache twice, and double the number of things that would be cached in
the ccache directories, not to mention doubling the number of calls to
remote hosts of distcc is involved.
So a full, long-term solution that gets back all of the efficiency
wins of ccache and distcc probably requires somehow passing a script
or program to ccache and distcc which does all of the steps which
kbuild needs to do to transform a kernel .c file to a kernel .o file,
which seems to involve interposing multiple steps between the foo.c to
foo.s and foo.s and foo.o steps, as well as post-processing on the
resulting .o file using objcopy. If we can somehow get ccache to
cache all of that work, and distribute all of the steps of going from
foo.c to foo.o to other machines when distcc is involved, that would
be really cool. I suspect this is not trivial to do, however....
- Ted
On Tue, Jan 06, 2009 at 10:15:26AM -0500, Theodore Ts'o wrote:
> The following commit, ad7a953c, "kbuild: strip generated symbols from
> *.ko" which was merged to the mainline last week (December 28th) has the
> unfortunate side effct of making ccache useless. That's because all
> files are now built using a two step process. First they are compiled
> to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a
> separate step. Unfortunately, ccache doesn't seem to be able to cache
> either the compilation or the assembly step (and it may not be worth it
> once the two steps are separated).
Hi Ted.
How about this simple patch.
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 5d90030..3b9b8f5 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -208,7 +208,7 @@ cmd_modversions = \
fi; \
else \
rm -f $(v_file); \
- $(CC) $(no_g_c_flags) -c -o $@ $(s_file); \
+ $(CC) $(no_g_c_flags) -c -o $@ $< ; \
fi;
endif
It basically starts all over again with the C file if the
file did not export any symbols (typical case).
This I think should allow ccache to be functional again.
I do not propose this as a final suggestion - more so
that I'm sure I understand the problem.
Sam
On Tue, Jan 06, 2009 at 11:09:50PM +0100, Sam Ravnborg wrote:
> Hi Ted.
>
> How about this simple patch.
>
> It basically starts all over again with the C file if the
> file did not export any symbols (typical case).
I tried the patch. It does cause ccache to be functional again, at
least as far as causing the "cache hit" stats to get bumped. However,
the ccache "called for link" stats is still going up (which is I guess
how ccache misleadingly categorizes "cc -S" requests to compile to
assembly). However, it doesn't actually save time; quite the reverse,
it makes things worse. Doing a 32-bit compile, I tried compiling a
kernel where after each compile, I remove the ext4 object files via
"/bin/rm ../ext4-32/fs/ext4/*.o". Without the patch, these were the
results after getting the cache warm, running the compile three tims,
and taking the average:
real 36.238 // user 44.143 // sys 11.375
Applying the patch, it actually took longer to recompile fs/ext4/*.o
and relink vmlinuz: (again average of three compiles)
real 38.401 // user 46.438 // sys 11.838
That's because ccache has to do the ccache -E and then calculate the
checksum to look up the results of the cache. Before the patch, we
are doing this:
preprocessor, compile, check for exported sysmbols, assemble
after the patch, we are doing this:
preprocessor, compile, check for exported symbols, then (in
ccache), preprocessor, locate the cached object file, write
cached objectfile.
What this experiment shows is that even with a completely warm cache,
it's faster to run the assembler than to run ccache and have it re-run
the preprocessor and look up the cached object file. In fact, it's
quite a bit faster.
So basically, in order to make things work well with ccache, we really
need to avoid redoing the compile unconditionally. If the goal is
just to check to see if there were any exported symbols, we would be
better off doing a grep for EXPORT_SYMBOL.
In contrast, with 2.6.28, the numbers for just compiling fs/ext4/*.o
and relinking vmlinuz is.
real 28.836 // user 30.949 // sys 10.562
Note that the bulk of this time is doing the kbuild setup and
relinking vmlinuz. The amount of time to do setup and relink is
18.677 / 21.639 / 8.253. So if you subtract that out, and simply
compare the time to compile fs/ext4/*.o, what you get is:
2.6.28: real 10.159 user 9.310 sys 2.309
2.6.28-git7: real 17.561 user 22.503 sys 3.122
2.6.28-git7 w/your patch real 19.724 user 24.799 sys 3.585
Bottom line, I suspect if we want to make builds fast, we really need
to cache the full kbuild pipeline. I hope you find these measurements
useful.
- Ted
On Tue, Jan 06, 2009 at 10:15:26AM -0500, Theodore Ts'o wrote:
> Or, if that's too complicated, maybe it would be worthwhile to have
> kbuild create its own specialized ccache system? Note that the last two
> solutions rule out using distcc, unless we can encapsulate the build
> process from a series of Makefile macros to a shell or C program, which
> could then be injected to the remote host system to be executed by
> distcc. One value of doing that is the CRC or MD5 of the shell script
> could be used as the version tag for the cache system.
Ho-hum... Could somebody explain why the hell had we switched to this
"intermediate .s" approach, anyway? It's not as if we couldn't run
objcopy after what we used to do...
BTW, do we care about non-ELF targets at all?
>>> Al Viro <[email protected]> 07.01.09 06:10 >>>
>On Tue, Jan 06, 2009 at 10:15:26AM -0500, Theodore Ts'o wrote:
>> Or, if that's too complicated, maybe it would be worthwhile to have
>> kbuild create its own specialized ccache system? Note that the last two
>> solutions rule out using distcc, unless we can encapsulate the build
>> process from a series of Makefile macros to a shell or C program, which
>> could then be injected to the remote host system to be executed by
>> distcc. One value of doing that is the CRC or MD5 of the shell script
>> could be used as the version tag for the cache system.
>
>Ho-hum... Could somebody explain why the hell had we switched to this
>"intermediate .s" approach, anyway? It's not as if we couldn't run
>objcopy after what we used to do...
Because objcopy wouldn't remove the __crc_* symbols the way the
object files were generated previously. I explored all possibilities I could
think of, with that two step process being the last, but the only one that
had the intended effect. In the end I queried the binutils list, and got
confirmation that there's no (existing) way to get rid of symbols used in
relocations in an already existing object file without corrupting it.
Jan
>>> Theodore Tso <[email protected]> 06.01.09 18:33 >>>
>On Tue, Jan 06, 2009 at 03:29:35PM +0000, Jan Beulich wrote:
>> >>> "Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>
>> >In the short term, though, it would be nice if we could get back a
>> >simple way of making a kernel object file using just cc, so that ccache
>> >and distcc could be functional again. Does that seem reasonable?
>>
>> Making the new logic dependent on a config option would seem reasonable
>> to me - of course at the expense of the respective Makefile becoming
>> even less readable.
>
>Too late. :-) It's pretty unreadable already.... as a result, I'm not
>at all confident that I could make such a patch. Is this something
>you could perhaps whip up? I'd really appreicate it, as it would
>seriously speed up by kernel development efforts.
Yes, I think I could (and in fact I already put it on my to-do list), but I can't
give a good prediction on when I'd be able to get to it.
Jan
On Wed, Jan 07, 2009 at 08:50:56AM +0000, Jan Beulich wrote:
> >>> Theodore Tso <[email protected]> 06.01.09 18:33 >>>
> >On Tue, Jan 06, 2009 at 03:29:35PM +0000, Jan Beulich wrote:
> >> >>> "Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>
> >> >In the short term, though, it would be nice if we could get back a
> >> >simple way of making a kernel object file using just cc, so that ccache
> >> >and distcc could be functional again. Does that seem reasonable?
> >>
> >> Making the new logic dependent on a config option would seem reasonable
> >> to me - of course at the expense of the respective Makefile becoming
> >> even less readable.
> >
> >Too late. :-) It's pretty unreadable already.... as a result, I'm not
> >at all confident that I could make such a patch. Is this something
> >you could perhaps whip up? I'd really appreicate it, as it would
> >seriously speed up by kernel development efforts.
>
> Yes, I think I could (and in fact I already put it on my to-do list), but I can't
> give a good prediction on when I'd be able to get to it.
We only see the ccache/distcc issue if we have MODVERSIONS
enabled.
So if we introduce a CONFIG option to disable strip of modules
then we will have the double amount of configuration
possibilities.
Today we have with or without MODVERSION.
If we make the stripping configurable then we will have
in addition two different configurations.
The reason to do the .c -> .s -> .o step is to make the
__crc_ symbols local so we can strip them off.
What is the gain/pain ratio here?
Would it be a possibility to drop stripping off the
crc symbols and go back to the ld method that
is more ccache friendly?
We would still benefir from all the other stripping done - no?
Sam
>>> Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>
>What is the gain/pain ratio here?
Certainly depends on the pov - it reduces the kernel module (disk) image
sizes quite a bit, so from a distro perspective its a move against the ever
growing package sizes (and the disk space) hundreds or even thousands
of modules require.
>Would it be a possibility to drop stripping off the
>crc symbols and go back to the ld method that
>is more ccache friendly?
That's certainly a possibility, but I have to admit that I'd like to keep the
option - a respective config option could default to 'no' if the build issues
are considered more significant. I'm really not that much worried about
adding another config parameter here...
>We would still benefir from all the other stripping done - no?
Yes.
Jan
On Tue, Jan 06, 2009 at 10:15:26AM -0500, Theodore Ts'o wrote:
> The following commit, ad7a953c, "kbuild: strip generated symbols from
> *.ko" which was merged to the mainline last week (December 28th) has the
> unfortunate side effct of making ccache useless.
I've just discovered another problem with this commit.
% du -s /lib/modules/2.6.28
63072 /lib/modules/2.6.28/
du -s /lib/modules/2.6.28-git7
221556 /lib/modules/2.6.28-git7/
The two configs are largely equal. I'm building with
CONFIG_DEBUG_KERNEL=y, but then installing with "make
INSTALL_MOD_STRIP=1 modules_install".
But look:
% ls /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko
10460 /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko
% ls /lib/modules/2.6.28/kernel/fs/xfs/xfs.ko
564 /lib/modules/2.6.28/kernel/fs/xfs/xfs.ko
The reason? The ad7a953c commit uses this procedure to do the installation:
# objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko || cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
So because it can't safely strip __ksymtab_strings it returns a
non-zero exit status, and doesn't do anything at all, and then the cp
command doesn't do any stripping whatso ever.
I could fix this by patching Makefile and scripts/Makefile.modist to
do this instead:
# objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko || (cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko ; strip --strip-debug /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko)
... but at this point, is there going to be any massive downside if I
just revert commit ad7a953c? Quite frankly, it's causing me a huge
amount of trouble, and I'm still a bit unclear what the upside of this
patch is. As near as I can tell there is *single* __crc_ symbol in
xfs.ko which all of this rigamorale is doing to strip out. From what
I can tell, not quite doubling the compile time when fully cached by
ccache, causing INSTALL_MOD_STRIP to fail randomly so that the
installed modules are a factor of 4 larger, compromising the amount of
space in my root partition, is all to remove a handful of __crc_*
symbols from the generated .ko file?
What am I missing? Why is stripping the __crc_* symbols so
gosh-darned important?
- Ted
On Wed, Jan 07, 2009 at 12:35:09PM +0000, Jan Beulich wrote:
> >>> Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>
> >What is the gain/pain ratio here?
>
> Certainly depends on the pov - it reduces the kernel module (disk) image
> sizes quite a bit, so from a distro perspective its a move against the ever
> growing package sizes (and the disk space) hundreds or even thousands
> of modules require.
The kernel modules seldom export anything so the benefit is minimal here.
I really think we are on a wrong track here.
I did a 15 minutes hack to try ripping the .c -> .s -> .o stuff out.
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 5d90030..8c93dc3 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -153,7 +153,7 @@ $(obj)/%.i: $(src)/%.c FORCE
cmd_genksyms = \
$(CPP) -D__GENKSYMS__ $(c_flags) $< | \
- $(GENKSYMS) -T $@ -A -a $(ARCH) \
+ $(GENKSYMS) -T $@ $(if $(CONFIG_KBUILD_STRIP_CRC), -A) -a $(ARCH) \
$(if $(KBUILD_PRESERVE),-p) \
$(if $(1),-r $(firstword $(wildcard $(@:.symtypes=.symref) /dev/null)))
@@ -192,6 +192,7 @@ v_file = $(@D)/.tmp_$(@F:.o=.v)
tmp_o_file = $(@D)/.tmp_$(@F)
no_g_c_flags = $(filter-out -g%,$(c_flags))
+ifdef CONFIG_KBUILD_STRIP_CRC
cmd_cc_o_c = $(CC) $(c_flags) -S -o $(s_file) $<
cmd_modversions = \
@@ -208,8 +209,20 @@ cmd_modversions = \
fi; \
else \
rm -f $(v_file); \
- $(CC) $(no_g_c_flags) -c -o $@ $(s_file); \
+ $(CC) $(no_g_c_flags) -c -o $@ $< ; \
fi;
+else
+cmd_cc_o_c = $(CC) $(c_flags) -c -o $(tmp_o_file) $<
+
+cmd_modversions = \
+ if $(OBJDUMP) -h $(tmp_o_file) | grep -q __ksymtab; then \
+ $(call cmd_genksyms, $(KBUILD_SYMTYPES)) > $(v_file); \
+ $(LD) $(LDFLAGS) -r -o $@ $(tmp_o_file) -T $(v_file); \
+ else \
+ rm -f $(v_file); \
+ fi;
+
+endif
endif
ifdef CONFIG_FTRACE_MCOUNT_RECORD
Jan - I need some hard numbers to convince me that stripping the
__crc symbols is worth it.
We can make it a configuration option as I showed above but we should
at least know what we are gaining for this.
I am very positive about stripping of the modules in general,
but the __crc_ symbols seems to trigger a few things:
1) ccahce, distcc issues in the MODVERSION case (that is allmodconfig builds)
2) It does not increase readability of the Makefile machinery and I like more
people to be able to hack on this - not less.
Thanks,
Sam
>>> Theodore Tso <[email protected]> 07.01.09 14:12 >>>
>objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
Another thing I've never seen in any of my builds - these strings always
get relocations against the corresponding section symbol.
>So because it can't safely strip __ksymtab_strings it returns a
>non-zero exit status, and doesn't do anything at all, and then the cp
>command doesn't do any stripping whatso ever.
>
>I could fix this by patching Makefile and scripts/Makefile.modist to
>do this instead:
>
># objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko || (cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko ; strip --strip-debug /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko)
I'd rather insert a second objcopy attempt here with just --strip-debug.
>... but at this point, is there going to be any massive downside if I
>just revert commit ad7a953c? Quite frankly, it's causing me a huge
>amount of trouble, and I'm still a bit unclear what the upside of this
>patch is. As near as I can tell there is *single* __crc_ symbol in
>xfs.ko which all of this rigamorale is doing to strip out. From what
>I can tell, not quite doubling the compile time when fully cached by
>ccache, causing INSTALL_MOD_STRIP to fail randomly so that the
>installed modules are a factor of 4 larger, compromising the amount of
>space in my root partition, is all to remove a handful of __crc_*
>symbols from the generated .ko file?
With this and the previous issue I certainly wouldn't mind this and the
subsequent vmlinux stripping patches to be reverted, and I'd aim at
providing an improved version for .30.
>What am I missing? Why is stripping the __crc_* symbols so
>gosh-darned important?
In a separate response I already explained what the goal here is - size
reduction (i.e. making distro package sizes as well as foot print on disk
smaller, thus improving the already existing stripping). And it's not only
about __crc_*, those are just the ones that when removed have the
biggest overall effect. And, as pointed out above, the vmlinux stripping
patch, which has an even more significant effect as it reduces the in-
memory kernel size when using KALLSYMS_ALL, depends on this one.
I have to admit that I generally dislike the tendency for everything to
always only grow, and hence I'm trying to find opportunities to shrink
stuff where possible. I've got a couple of other patches in my local
queue that also aim in that direction, plus there are a few thoughts on
improvements beyond that.
Jan
>>> Sam Ravnborg <[email protected]> 07.01.09 14:23 >>>
>On Wed, Jan 07, 2009 at 12:35:09PM +0000, Jan Beulich wrote:
>> >>> Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>
>> >What is the gain/pain ratio here?
>>
>> Certainly depends on the pov - it reduces the kernel module (disk) image
>> sizes quite a bit, so from a distro perspective its a move against the ever
>> growing package sizes (and the disk space) hundreds or even thousands
>> of modules require.
>
>The kernel modules seldom export anything so the benefit is minimal here.
Not really, for highly modular distro kernels.
>I did a 15 minutes hack to try ripping the .c -> .s -> .o stuff out.
Looks good to me. In case you decide to revert the stuff for .29, I'll
merge this into my patch.
>Jan - I need some hard numbers to convince me that stripping the
>__crc symbols is worth it.
The on-disk size of the modules in my builds is reduced by about 5% with
this patch (but I can't immediately tell how much of this is due to __crc_*
vs. other symbol stripping).
Jan
On Wed, Jan 07, 2009 at 08:49:18AM +0000, Jan Beulich wrote:
> >>> Al Viro <[email protected]> 07.01.09 06:10 >>>
> >On Tue, Jan 06, 2009 at 10:15:26AM -0500, Theodore Ts'o wrote:
> >> Or, if that's too complicated, maybe it would be worthwhile to have
> >> kbuild create its own specialized ccache system? Note that the last two
> >> solutions rule out using distcc, unless we can encapsulate the build
> >> process from a series of Makefile macros to a shell or C program, which
> >> could then be injected to the remote host system to be executed by
> >> distcc. One value of doing that is the CRC or MD5 of the shell script
> >> could be used as the version tag for the cache system.
> >
> >Ho-hum... Could somebody explain why the hell had we switched to this
> >"intermediate .s" approach, anyway? It's not as if we couldn't run
> >objcopy after what we used to do...
>
> Because objcopy wouldn't remove the __crc_* symbols the way the
> object files were generated previously. I explored all possibilities I could
> think of, with that two step process being the last, but the only one that
> had the intended effect. In the end I queried the binutils list, and got
> confirmation that there's no (existing) way to get rid of symbols used in
> relocations in an already existing object file without corrupting it.
I do not understand two things.
a) WTF would we care about ones in vmlinux, when kallsyms drops the
absolute ones anyway?
b) is there any reason why we can't copy symbol table out of module
ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of
it? Note that __crc_... is not the only junk in there - you don't need
(or want) to keep the things like undefs for /proc/kallsyms purposes.
And modules *using* an exported symbol are more common that modules exporting
one... Why not copy it at add_kallsyms() time, skipping the junk we don't
want anyway?
>>> Al Viro <[email protected]> 07.01.09 15:03 >>>
>I do not understand two things.
> a) WTF would we care about ones in vmlinux, when kallsyms drops the
>absolute ones anyway?
The patch is not only about __crc_* - for vmlinux, this part really doesn't
matter, but there's a significant amount of other symbols that get stripped
from vmlinux with the second patch.
> b) is there any reason why we can't copy symbol table out of module
>ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of
>it? Note that __crc_... is not the only junk in there - you don't need
>(or want) to keep the things like undefs for /proc/kallsyms purposes.
>And modules *using* an exported symbol are more common that modules exporting
>one... Why not copy it at add_kallsyms() time, skipping the junk we don't
>want anyway?
Would certainly be doable, but would seem reasonable only if it's a small
set of easily recognizable symbols. If you look at scripts/strip-symbols,
you may agree that it's better to do this at build time...
Jan
On Wed, Jan 07, 2009 at 01:39:38PM +0000, Jan Beulich wrote:
> >>> Theodore Tso <[email protected]> 07.01.09 14:12 >>>
> >objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
>
> Another thing I've never seen in any of my builds - these strings always
> get relocations against the corresponding section symbol.
Just to be clear; it doesn't always fail; it seems to be random (or at
least, I haven't had time to try to suss out why sometimes
__ksymtab_strings is being used in a relocation):
INSTALL fs/cifs/cifs.ko
INSTALL fs/coda/coda.ko
INSTALL fs/configfs/configfs.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
INSTALL fs/dlm/dlm.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
INSTALL fs/ecryptfs/ecryptfs.ko
INSTALL fs/efs/efs.ko
INSTALL fs/ext2/ext2.ko
INSTALL fs/fat/fat.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
INSTALL fs/fat/msdos.ko
INSTALL fs/fat/vfat.ko
INSTALL fs/freevxfs/freevxfs.ko
This is on an Ubuntu Hardy system, if this helps. I'm using gcc
4.2.4, and binutils version 2.18.1~cvs20080103-0ubuntu1.
> ># objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko || (cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko ; strip --strip-debug /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko)
>
> I'd rather insert a second objcopy attempt here with just --strip-debug.
Yes, that makes sense, of course.
> In a separate response I already explained what the goal here is - size
> reduction (i.e. making distro package sizes as well as foot print on disk
> smaller, thus improving the already existing stripping). And it's not only
> about __crc_*, those are just the ones that when removed have the
> biggest overall effect. And, as pointed out above, the vmlinux stripping
> patch, which has an even more significant effect as it reduces the in-
> memory kernel size when using KALLSYMS_ALL, depends on this one.
I'm all for size reduction; the problem is that I don't have one of
Linus's fancy machines that can do 10 second full kernel compiles. So
I'm very much reliant on ccache, and a 70% increase in kernel
compilation time, when a full compile already takes order of magnitude
5-10 minutes (this is for my personal no-modules, stripped-down
development build), is very painful and easily noticed.
Maybe we need to modify ccache, or pull ccache into the kernel sources
and then hack it for our unique, and apparently , constantly
growing-in-complexity build complexity. One thing that would really
help is some documentation and some way of seeing *all* of the steps
that are getting executed when we compile a .c to .o file. One of the
things I noticed is that make V=1 only showed the cc -S step, but it
seemd to mask out the other steps of what was going on, which forced
me to go diving into the kbuild scripts --- and for someone who isn't
daily hacking the kbuild scripts, and who needed to trace all of the
macros, and who might not know at their fingertips what $(filter-out
...), $(wildcard ...), and $(notdir ...) do --- it's highly painful.
I'm sure this is just my own ignorance, but a tutorial in how the
kbuild system works, which files are involved, where to look for
various things, and a short summary to all of the gmake magic which
gets used would be very helpful.
> I have to admit that I generally dislike the tendency for everything to
> always only grow, and hence I'm trying to find opportunities to shrink
> stuff where possible. I've got a couple of other patches in my local
> queue that also aim in that direction, plus there are a few thoughts on
> improvements beyond that.
I'm certainly sympathetic to this. I just also have this desire to
make sure that it is possible to do kernel hacking on a X61-class
laptop, and that mongo-expensive 8 socket, quad-core systems with vast
amounts of memories aren't necessary in order to do compile-edit-debug
cycles in reasonable amounts of time. :-/
(And BTW, I consider the pre-2.6.28 compile times only barely
acceptable; most of this isn't kbuild's fault, I know, it includes the
fact that link steps are still slow, and gcc is constantly getting
increasingly slower over time. Sigh... I will just note wryly that
it used to be that I could compile 0.9x kernels on a 40 MHz 386
machine in 10 minutes. Some 15 years later, it still takes roughly
the same amount of time to compile a kernel, even though computers
have gotten vastly faster since then. Something seems wrong with
that.... :-)
- Ted
On Wed, Jan 07, 2009 at 02:28:10PM +0000, Jan Beulich wrote:
> > b) is there any reason why we can't copy symbol table out of module
> >ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of
> >it? Note that __crc_... is not the only junk in there - you don't need
> >(or want) to keep the things like undefs for /proc/kallsyms purposes.
> >And modules *using* an exported symbol are more common that modules exporting
> >one... Why not copy it at add_kallsyms() time, skipping the junk we don't
> >want anyway?
>
> Would certainly be doable, but would seem reasonable only if it's a small
> set of easily recognizable symbols. If you look at scripts/strip-symbols,
> you may agree that it's better to do this at build time...
I will not. Undefined ones are far more common than any of those...
>>> Theodore Tso <[email protected]> 07.01.09 15:28 >>>
>On Wed, Jan 07, 2009 at 01:39:38PM +0000, Jan Beulich wrote:
>> >>> Theodore Tso <[email protected]> 07.01.09 14:12 >>>
>> >objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
>>
>> Another thing I've never seen in any of my builds - these strings always
>> get relocations against the corresponding section symbol.
>
>Just to be clear; it doesn't always fail; it seems to be random (or at
>least, I haven't had time to try to suss out why sometimes
>__ksymtab_strings is being used in a relocation):
It's apparently happening only for modules which export symbols.
Jan
>>> Al Viro <[email protected]> 07.01.09 15:37 >>>
>On Wed, Jan 07, 2009 at 02:28:10PM +0000, Jan Beulich wrote:
>> > b) is there any reason why we can't copy symbol table out of module
>> >ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of
>> >it? Note that __crc_... is not the only junk in there - you don't need
>> >(or want) to keep the things like undefs for /proc/kallsyms purposes.
>> >And modules *using* an exported symbol are more common that modules exporting
>> >one... Why not copy it at add_kallsyms() time, skipping the junk we don't
>> >want anyway?
>>
>> Would certainly be doable, but would seem reasonable only if it's a small
>> set of easily recognizable symbols. If you look at scripts/strip-symbols,
>> you may agree that it's better to do this at build time...
>
>I will not. Undefined ones are far more common than any of those...
Undefined ones could certainly be stripped out there (while they definitely
can't be at build time).
Jan
On Wed, Jan 07, 2009 at 01:48:03PM +0000, Jan Beulich wrote:
> >>> Sam Ravnborg <[email protected]> 07.01.09 14:23 >>>
> >On Wed, Jan 07, 2009 at 12:35:09PM +0000, Jan Beulich wrote:
> >> >>> Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>
> >> >What is the gain/pain ratio here?
> >>
> >> Certainly depends on the pov - it reduces the kernel module (disk) image
> >> sizes quite a bit, so from a distro perspective its a move against the ever
> >> growing package sizes (and the disk space) hundreds or even thousands
> >> of modules require.
> >
> >The kernel modules seldom export anything so the benefit is minimal here.
>
> Not really, for highly modular distro kernels.
>
> >I did a 15 minutes hack to try ripping the .c -> .s -> .o stuff out.
>
> Looks good to me. In case you decide to revert the stuff for .29, I'll
> merge this into my patch.
As it stand now it:
1) breaks sparc32 build
2) causes regression with distcc/ccache use when MODVERSIONS are enabled
3) the concept is under discussion
So I will have to revert it somehow.
I will look into during a partial revert though as your patches had some stuff
in them I like to keep or which does not harm (genksyms change for instance).
>
> >Jan - I need some hard numbers to convince me that stripping the
> >__crc symbols is worth it.
>
> The on-disk size of the modules in my builds is reduced by about 5% with
> this patch (but I can't immediately tell how much of this is due to __crc_*
> vs. other symbol stripping).
But if only on-disk size is your primary concern then another patch
of Jan Engelhardt would be a bigger win:
build: install modules compressed
---
scripts/Makefile.modinst | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..c3421a1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -17,7 +17,7 @@ __modinst: $(modules)
@:
quiet_cmd_modules_install = INSTALL $@
- cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
$(2)/$(notdir $@)
+ cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
$(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
# Modules built outside the kernel source tree go into extra by default
INSTALL_MOD_DIR ?= extra
(cut'n'paste so will not apply direct)
It does not decrease the size of vmlinux but the > 1000 modules are compressed.
The patch will in some form be present in kbuild-next.git when I open up
for that.
Sam
On Wed, Jan 07, 2009 at 09:06:46PM +0100, Sam Ravnborg wrote:
> > Looks good to me. In case you decide to revert the stuff for .29, I'll
> > merge this into my patch.
> As it stand now it:
> 1) breaks sparc32 build
> 2) causes regression with distcc/ccache use when MODVERSIONS are enabled
> 3) the concept is under discussion
>
4) Bloats distro modules from 48M to >1GB when stripping fails. :\
regards, Kyle
On Wed, Jan 07, 2009 at 08:12:21AM -0500, Theodore Tso wrote:
> On Tue, Jan 06, 2009 at 10:15:26AM -0500, Theodore Ts'o wrote:
> > The following commit, ad7a953c, "kbuild: strip generated symbols from
> > *.ko" which was merged to the mainline last week (December 28th) has the
> > unfortunate side effct of making ccache useless.
>
> I've just discovered another problem with this commit.
>
> % du -s /lib/modules/2.6.28
> 63072 /lib/modules/2.6.28/
> du -s /lib/modules/2.6.28-git7
> 221556 /lib/modules/2.6.28-git7/
>
> The two configs are largely equal. I'm building with
> CONFIG_DEBUG_KERNEL=y, but then installing with "make
> INSTALL_MOD_STRIP=1 modules_install".
Fedora rawhide users noticed this same problem this morning too,
when I pushed out a build that eats up a gig of diskspace thanks
to the number of modules we have, that are all afflicted by this problem.
I'll have to revert ad7a953c for our builds to continue tracking mainline
without monopolising the diskspace in the Fedora build system.
Dave
--
http://www.codemonkey.org.uk
On Thu, Jan 08, 2009 at 02:17:03PM -0500, Dave Jones wrote:
>
> Fedora rawhide users noticed this same problem this morning too,
> when I pushed out a build that eats up a gig of diskspace thanks
> to the number of modules we have, that are all afflicted by this problem.
>
> I'll have to revert ad7a953c for our builds to continue tracking mainline
> without monopolising the diskspace in the Fedora build system.
Note: you'll probably also have revert 9bb48247 (allow stripping of
generated symbols under CONFIG_KALLSYMS_ALL) since I believe it
depends on ad7a953c.
- Ted
On Thu, Jan 08, 2009 at 04:31:58PM -0500, Theodore Tso wrote:
> On Thu, Jan 08, 2009 at 02:17:03PM -0500, Dave Jones wrote:
> >
> > Fedora rawhide users noticed this same problem this morning too,
> > when I pushed out a build that eats up a gig of diskspace thanks
> > to the number of modules we have, that are all afflicted by this problem.
> >
> > I'll have to revert ad7a953c for our builds to continue tracking mainline
> > without monopolising the diskspace in the Fedora build system.
>
> Note: you'll probably also have revert 9bb48247 (allow stripping of
> generated symbols under CONFIG_KALLSYMS_ALL) since I believe it
> depends on ad7a953c.
>
This doesn't appear to be the culprit. Or at least, doesn't fix the
problem.
make INSTALL_MOD_PATH=mods/ modules_install
with a distro kernel config (ie: Fedora's[1]) still results in
1.8GB of modules (willy checked that it wasn't a Fedora toolchain
bug by reproducing on a Debian system.)
As near as I can ascertain, this is limited to x86_64, at least on
Fedora:
kyle@localhost ~/misc $ for i in *.rpm; do echo $i; rpm
-qip $i | grep Size; done
kernel-2.6.29-0.25.rc0.git14.fc11.ppc64.rpm
Size : 124858478 License:
kernel-2.6.29-0.25.rc0.git14.fc11.ppc.rpm
Size : 66313693 License:
kernel-2.6.29-0.25.rc0.git14.fc11.x86_64.rpm
Size : 1041480317 License:
kernel-PAE-2.6.29-0.25.rc0.git14.fc11.i686.rpm
Size : 56252360 License:
x86_64 is an order of magnitude bigger than the rest, which have
identical debug options enabled.
regards, Kyle
On Sun, Jan 11, 2009 at 04:48:37PM -0500, Kyle McMartin wrote:
> > > I'll have to revert ad7a953c for our builds to continue tracking mainline
> > > without monopolising the diskspace in the Fedora build system.
> >
> > Note: you'll probably also have revert 9bb48247 (allow stripping of
> > generated symbols under CONFIG_KALLSYMS_ALL) since I believe it
> > depends on ad7a953c.
>
> This doesn't appear to be the culprit. Or at least, doesn't fix the
> problem.
>
> make INSTALL_MOD_PATH=mods/ modules_install
>
> with a distro kernel config (ie: Fedora's[1]) still results in
> 1.8GB of modules (willy checked that it wasn't a Fedora toolchain
> bug by reproducing on a Debian system.)
>
> As near as I can ascertain, this is limited to x86_64, at least on
> Fedora:
It shows up for me with x86_32, and it is fixed by reverting those two
commits. Pre-2.6.28-rc6:
74864 -rw-r--r-- 1 tytso tytso 76657780 2008-12-12 14:41 linux-image-2.6.28-rc8-00066-gc810663_tytso.SMP.1_i386.deb
Post 2.6.28:
146380 -rw-r--r-- 1 tytso tytso 149888992 2009-01-06 02:01 linux-image-2.6.28-05784-gd5b9778_tytso.SMP.1_i386.deb
After I revert the above two commits:
76340 -rw-r--r-- 1 tytso tytso 78167848 2009-01-09 08:45 linux-image-2.6.28-07956-g2e69ab5_tytso.SMP.1_i386.deb
- Ted
On Sun, Jan 11, 2009 at 04:48:37PM -0500, Kyle McMartin wrote:
>
> with a distro kernel config (ie: Fedora's[1]) still results in
> 1.8GB of modules (willy checked that it wasn't a Fedora toolchain
> bug by reproducing on a Debian system.)
P.S. I have a hacked make-kpkg which installs the modules via:
make INSTALL_MOD_STRIP=1 modules_install
Unfortunately, I'm not using kbuild's make install builddeb since it
doesn't create the headers and debuginfo files; and the debian's
official kernel building scripts (make-kpkg has been deprecated)
doesn't correctly deal with creating debuginfo files, either.
In any case, its make INSTALL_MOD_STRIP=1 which is getting busted by
the two commits, and which also spike out ccache and distcc.
- Ted
On Sun, Jan 11, 2009 at 05:51:38PM -0500, Theodore Tso wrote:
> On Sun, Jan 11, 2009 at 04:48:37PM -0500, Kyle McMartin wrote:
> >
> > with a distro kernel config (ie: Fedora's[1]) still results in
> > 1.8GB of modules (willy checked that it wasn't a Fedora toolchain
> > bug by reproducing on a Debian system.)
>
> P.S. I have a hacked make-kpkg which installs the modules via:
>
> make INSTALL_MOD_STRIP=1 modules_install
>
> Unfortunately, I'm not using kbuild's make install builddeb since it
> doesn't create the headers and debuginfo files; and the debian's
> official kernel building scripts (make-kpkg has been deprecated)
> doesn't correctly deal with creating debuginfo files, either.
>
> In any case, its make INSTALL_MOD_STRIP=1 which is getting busted by
> the two commits, and which also spike out ccache and distcc.
>
Right, I had a braino and forgot that having 1.8GB of modules is
actually expected. The broken stage is the stripping of the debuginfo
into a seperate set of files, so it's likely Fedora-specific.
regards, Kyle
Selon Sam Ravnborg <[email protected]>:
...
> But if only on-disk size is your primary concern then another patch
> of Jan Engelhardt would be a bigger win:
>
Would be better to add -n to gzip flags for 2 reasons
- compressed module is smaller without file name and date include
- it make paranoid people happier. At each compilation of the same source within
same environnement, using gzip without timestamp produce exactly the same
binary. With timestamp include, you are no more able to track a change to the
compressed module file with md5|sha1...
Even gzip-1.2.4 from 1993 support -n flag
build: install modules compressed
---
scripts/Makefile.modinst | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..c3421a1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -17,7 +17,7 @@ __modinst: $(modules)
@:
quiet_cmd_modules_install = INSTALL $@
- cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
$(2)/$(notdir $@)
+ cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
$(mod_strip_cmd) $(2)/$(notdir $@); gzip -9nf $(2)/$(notdir $@)
# Modules built outside the kernel source tree go into extra by default
INSTALL_MOD_DIR ?= extra
(cut'n'paste so will not apply direct)
Gilles
On Tue, 2009-01-06 at 10:15 -0500, Theodore Ts'o wrote:
> The following commit, ad7a953c, "kbuild: strip generated symbols from
> *.ko" which was merged to the mainline last week (December 28th) has the
> unfortunate side effct of making ccache useless. That's because all
> files are now built using a two step process. First they are compiled
> to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a
> separate step. Unfortunately, ccache doesn't seem to be able to cache
> either the compilation or the assembly step (and it may not be worth it
> once the two steps are separated).
>
> It would be nice if there was a CONFIG option which compiles files the
> old-fashioned way, even if we lose the ability to strip the__crc_
> symbols, just so that ccache can function again, since it can really
> speed up the kernel development process. Alternatively, maybe ccache
> could somehow be taught a specialized way take the .c file, some kind of
> kbuild-specified version identifier, and then output of the gcc -S,
> assembly frobnication, gcc -c, objcopy pipeline that kbuild uses to
> build an object file, so we can have cache the entire kbuild object
> building process.
>
> Or, if that's too complicated, maybe it would be worthwhile to have
> kbuild create its own specialized ccache system? Note that the last two
> solutions rule out using distcc, unless we can encapsulate the build
> process from a series of Makefile macros to a shell or C program, which
> could then be injected to the remote host system to be executed by
> distcc. One value of doing that is the CRC or MD5 of the shell script
> could be used as the version tag for the cache system.
>
> In the short term, though, it would be nice if we could get back a
> simple way of making a kernel object file using just cc, so that ccache
> and distcc could be functional again. Does that seem reasonable?
So what's the current status on this, could we get it reverted ASAP or
add that CONFIG_ switch (default N)?
Leaving the build system broken for so long just isn't cool.
>>> Peter Zijlstra <[email protected]> 14.01.09 18:16 >>>
>So what's the current status on this, could we get it reverted ASAP or
>add that CONFIG_ switch (default N)?
>
>Leaving the build system broken for so long just isn't cool.
It is my understanding that Sam was looking at doing a partial revert first.
If that doesn't work out, doing the CONFIG_* thing would be pretty trivial
(I merged it into my local patch, and hence would simply have to extract it).
Sam?
Jan
On Thu, Jan 15, 2009 at 09:11:14AM +0000, Jan Beulich wrote:
> >>> Peter Zijlstra <[email protected]> 14.01.09 18:16 >>>
> >So what's the current status on this, could we get it reverted ASAP or
> >add that CONFIG_ switch (default N)?
> >
> >Leaving the build system broken for so long just isn't cool.
>
> It is my understanding that Sam was looking at doing a partial revert first.
> If that doesn't work out, doing the CONFIG_* thing would be pretty trivial
> (I merged it into my local patch, and hence would simply have to extract it).
I just have a local branch, "emergency-fixes" which just reverts
commits ad7a953c and 9bb48247, which I pull in before doing full
builds.
You can also work around the bug by disabling CONFIG_MODVERSIONS; one
of my build configurations has almost no modules at all but it still
had CONFIG_MODVERSIONS disabled, so for my sniff tests build of
mainline where I don't want to pull in the emergency-fixes branch,
I've also dealt with the situation by disabling CONFIG_MODVERSIONS.
These are both hacks, but they do work around the problem...
- Ted