`-combine'
If you are compiling multiple source files, this option tells the
driver to pass all the source files to the compiler at once (for
those languages for which the compiler can handle this). This
will allow intermodule analysis (IMA) to be performed by the
compiler. Currently the only language for which this is supported
is C. If you pass source files for multiple languages to the
driver, using this option, the driver will invoke the compiler(s)
that support IMA once each, passing each compiler all the source
files appropriate for it. For those languages that do not support
IMA this option will be ignored, and the compiler will be invoked
once for each source file in that language. If you use this
option in conjunction with `-save-temps', the compiler will
generate multiple pre-processed files (one for each source file),
but only one (combined) `.o' or `.s' file.
`-fwhole-program'
Assume that the current compilation unit represents whole program
being compiled. All public functions and variables with the
exception of `main' and those merged by attribute
`externally_visible' become static functions and in a affect gets
more aggressively optimized by interprocedural optimizers. While
this option is equivalent to proper use of `static' keyword for
programs consisting of single file, in combination with option
`--combine' this flag can be used to compile most of smaller scale
C programs since the functions and variables become local for the
whole combined compilation unit, not for the single source file
itself.
Using a combination of these two compiler options for building kernel
code leads to some useful optimisation -- especially with modules which
are made up of a bunch of incestuous C files, where none of the global
symbols actually _need_ to be visible outside the directory they reside
in. File systems are a prime example of this -- on PPC64 I see a
reduction in size of ext3.ko by 2.6%, jffs2.ko by 5%, cifs.ko by 8% and
befs.ko by a scary 14%. Strangely, udf.ko seems to have _grown_ by 6.6%
-- that'll probably be another optimisation bug like GCC PR28755.
The same benefits can be extended to the vmlinux too, although there are
caveats with making _everything_ static. However, it's relatively simple
to make EXPORT_SYMBOL() automatically set the 'externally_visible'
attribute on the symbol in question, and to introduce a new '__global'
tag which does the same for those symbols which aren't exported to
modules but which _are_ needed as a global symbol in vmlinux.
Size results from a test build on ppc64 are shown at
http://david.woodhou.se/combine/sizes.csv -- the format is
<old size>,<new size>,<delta>,<percentage * 100>,<object name>
The same file with objects where the size didn't change omitted, and
sorted on percentage is http://david.woodhou.se/combine/sizes-sorted.csv
There are a bunch of GCC bugs which make this interesting:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=27898
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=27889
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28706
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28712
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28744
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28755
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28779
Fixes (or workarounds) for some of these are at
http://david.woodhou.se/combine/gcc-patches/
(Actual patches will follow, to [email protected] only)
--
dwmw2
> If you are compiling multiple source files, this option tells the
> driver to pass all the source files to the compiler at once (for
> those languages for which the compiler can handle this). This
> will allow intermodule analysis (IMA) to be performed by the
> compiler. Currently the only language for which this is supported
> is C. If you pass source files for multiple languages to the
> driver, using this option, the driver will invoke the compiler(s)
> that support IMA once each, passing each compiler all the source
> files appropriate for it.
Compiling files on their own (`make drivers/foo/bar.o`) seems to make
the optimization void. Sure, most people don't stop compiling in
between. Just a note
> For those languages that do not support
> IMA this option will be ignored, and the compiler will be invoked
> once for each source file in that language. If you use this
> option in conjunction with `-save-temps', the compiler will
> generate multiple pre-processed files (one for each source file),
> but only one (combined) `.o' or `.s' file.
There should be an option (in the kernel's makefile system) to disable
its use, just in case `gcc *.c` gobbles up a little more RAM
than is present.
>Using a combination of these two compiler options for building kernel
>code leads to some useful optimisation -- especially with modules which
>are made up of a bunch of incestuous C files, where none of the global
>symbols actually _need_ to be visible outside the directory they reside
>in.
For modules, we have EXPORT_SYMBOL() and any other symbols that are
'extern' but not exported are not visible to other modules.
>The same benefits can be extended to the vmlinux too, although there are
>caveats with making _everything_ static. However, it's relatively simple
>to make EXPORT_SYMBOL() automatically set the 'externally_visible'
>attribute on the symbol in question, and to introduce a new '__global'
>tag which does the same for those symbols which aren't exported to
>modules but which _are_ needed as a global symbol in vmlinux.
Does the kernel (at least for modules) really use ELF symbol visibility
(read: __attribute__((visibility(xyz))) or -fvisibility=xyz) for
lookups? I do not think so, since EXPORT_SYMBOL explicitly puts something
into __ksymtab/__kstrtab.
If visibility supports had been in GCC a long time ago, I am sure we would
not need EXPORT_SYMBOL today, or rather, would do it by use of
__attribute__() rather than a macro that ksymtabs it. Or am I possibly
misunderstanding something?
Jan Engelhardt
--
On Thu, 2006-08-24 at 18:48 +0200, Jan Engelhardt wrote:
> > If you are compiling multiple source files, this option tells the
> > driver to pass all the source files to the compiler at once (for
> > those languages for which the compiler can handle this). This
> > will allow intermodule analysis (IMA) to be performed by the
> > compiler. Currently the only language for which this is supported
> > is C. If you pass source files for multiple languages to the
> > driver, using this option, the driver will invoke the compiler(s)
> > that support IMA once each, passing each compiler all the source
> > files appropriate for it.
>
> Compiling files on their own (`make drivers/foo/bar.o`) seems to make
> the optimization void. Sure, most people don't stop compiling in
> between. Just a note
Yeah, that's largely because my support for this in the makefiles is an
evil hack. I'm sure Sam can come up with something better :)
Actually I'm not entirely sure what you write is true. It'll _build_
fs/jffs2/read.o, for example, but it still won't then use it when I make
the kernel -- it'll just use fs/jffs2/jffs2.o which is built from all
the C files with --combine. So the optimisation isn't lost.
> There should be an option (in the kernel's makefile system) to disable
> its use, just in case `gcc *.c` gobbles up a little more RAM
> than is present.
Maybe, yeah. If you look in my 'hacks' patch you'll see I actually did
this for drivers/net/e1000.ko because of a known compiler bug. In an
ideal world it shouldn't be necessary though -- and there is always the
option of just turning off CONFIG_COMBINED_COMPILE if your compiler is
buggy or your system doesn't have the balls.
> >Using a combination of these two compiler options for building kernel
> >code leads to some useful optimisation -- especially with modules which
> >are made up of a bunch of incestuous C files, where none of the global
> >symbols actually _need_ to be visible outside the directory they reside
> >in.
>
> For modules, we have EXPORT_SYMBOL() and any other symbols that are
> 'extern' but not exported are not visible to other modules.
Indeed.
> >The same benefits can be extended to the vmlinux too, although there are
> >caveats with making _everything_ static. However, it's relatively simple
> >to make EXPORT_SYMBOL() automatically set the 'externally_visible'
> >attribute on the symbol in question, and to introduce a new '__global'
> >tag which does the same for those symbols which aren't exported to
> >modules but which _are_ needed as a global symbol in vmlinux.
>
> Does the kernel (at least for modules) really use ELF symbol visibility
> (read: __attribute__((visibility(xyz))) or -fvisibility=xyz) for
> lookups?
No, I think that's something entirely different. This is just about
whether something is static or not.
The -fwhole-program option makes _everything_ static -- not visible
outside the object file it's compiled into. For modules that's fine,
since they really don't need to provide _any_ global symbols.
For stuff compiled into the kernel, however, this can be a problem,
since sometimes we _need_ a symbol to actually be global, and accessible
from somewhere outside the directory it's provided by.
So to overcome this, we use GCC's __attribute__((externally_visible))
which, as documented, just makes it global again -- undoing the effect
of -fwhole-program just for this _one_ symbol.
For anything which is exported by EXPORT_SYMBOL or marked ASMLINKAGE, we
put the externally_visible attribute on it automatically, by modifying
the EXPORT_SYMBOL* and ASMLINKAGE macros to do so. That covers a large
number of the symbols which actually need to be global within vmlinux.
The new '__global' tag covers the rest.
--
dwmw2
On Thursday 24 August 2006 18:48, Jan Engelhardt wrote:
> If visibility supports had been in GCC a long time ago, I am sure we would
> not need EXPORT_SYMBOL today, or rather, would do it by use of
> __attribute__() rather than a macro that ksymtabs it. Or am I possibly
> misunderstanding something?
>
It's probably true, but the way it's done today gave us CONFIG_MODVERSIONS
and EXPORT_SYMBOL_GPL, which would break when turning EXPORT_SYMBOL into a
simple __attribute__().
Arnd <><
On Thu, 2006-08-24 at 19:15 +0200, Arnd Bergmann wrote:
> It's probably true, but the way it's done today gave us CONFIG_MODVERSIONS
> and EXPORT_SYMBOL_GPL, which would break when turning EXPORT_SYMBOL into a
> simple __attribute__().
ELF can do symbol versioning too. But I don't really see a simple way to
make normal ELF linking work with a kernel like that. The vmlinux ELF
file is long gone by the time the kernel boots -- it needs its own
mechanism for managing its symbols anyway; its own internal dynamic
linker. I suppose you _could_ make it work from the standard ELF linkage
but I don't think it'd be fun.
ObMODVERSIONS: Forgot to mention that my makefile hacks for --combine
break that completely. It tries to preprocess all the files together
with 'gcc -E -D__GENKSYMS__ *.c' -- and you can't do that.
--
dwmw2
On Thu, Aug 24, 2006 at 04:24:28PM +0100, David Woodhouse wrote:
>...
> Using a combination of these two compiler options for building kernel
> code leads to some useful optimisation -- especially with modules which
> are made up of a bunch of incestuous C files, where none of the global
> symbols actually _need_ to be visible outside the directory they reside
> in. File systems are a prime example of this -- on PPC64 I see a
> reduction in size of ext3.ko by 2.6%, jffs2.ko by 5%, cifs.ko by 8% and
> befs.ko by a scary 14%. Strangely, udf.ko seems to have _grown_ by 6.6%
> -- that'll probably be another optimisation bug like GCC PR28755.
>...
Looks good. :-)
What kernel are your patches against?
2.6.18-rc4 gives me:
<-- snip -->
...
CHK include/linux/version.h
CHK include/linux/utsrelease.h
CC arch/i386/kernel/asm-offsets.s
GEN include/asm-i386/asm-offsets.h
HOSTCC scripts/kallsyms
HOSTCC scripts/conmakehash
HOSTCC scripts/bin2c
make[1]: *** No rule to make target `/home/bunk/linux/linux-2.6.18-rc4/dummy.c', needed by `init/init.o'. Stop.
make: *** [init] Error 2
<-- snip -->
> dwmw2
cu
Adrian
--
Gentoo kernels are 42 times more popular than SUSE kernels among
KLive users (a service by SUSE contractor Andrea Arcangeli that
gathers data about kernels from many users worldwide).
There are three kinds of lies: Lies, Damn Lies, and Statistics.
Benjamin Disraeli
On Thu, 2006-08-24 at 23:49 +0200, Adrian Bunk wrote:
>
> make[1]: *** No rule to make target
> `/home/bunk/linux/linux-2.6.18-rc4/dummy.c', needed by `init/init.o'.
> Stop.
oops, sorry. touch dummy.c -- evil makefile hack ;)
--
dwmw2
>> Compiling files on their own (`make drivers/foo/bar.o`) seems to make
>> the optimization void. Sure, most people don't stop compiling in
>> between. Just a note
>
>Actually I'm not entirely sure what you write is true. It'll _build_
>fs/jffs2/read.o, for example, but it still won't then use it when I make
>the kernel -- it'll just use fs/jffs2/jffs2.o which is built from all
>the C files with --combine. So the optimisation isn't lost.
Umm then it spends double the time in compilation, doing:
read.o <- read.c
foo.o <- foo.c
bar.o <- bar.c
built-in.o <- read.c foo.c bar.c
(cf. default current:
built-in.o <- read.o foo.o bar.o)
>So to overcome this, we use GCC's __attribute__((externally_visible))
>which, as documented, just makes it global again -- undoing the effect
>of -fwhole-program just for this _one_ symbol.
Interesting. __attribute__((visibility("default"))) does the same?
Jan Engelhardt
--
On Fri, Aug 25, 2006 at 08:01:27AM +0200, Jan Engelhardt wrote:
> >> Compiling files on their own (`make drivers/foo/bar.o`) seems to make
> >> the optimization void. Sure, most people don't stop compiling in
> >> between. Just a note
> >
> >Actually I'm not entirely sure what you write is true. It'll _build_
> >fs/jffs2/read.o, for example, but it still won't then use it when I make
> >the kernel -- it'll just use fs/jffs2/jffs2.o which is built from all
> >the C files with --combine. So the optimisation isn't lost.
>
> Umm then it spends double the time in compilation, doing:
>
> read.o <- read.c
It will only do this if you ask for it.
The question was what happened when you did make read.o
> foo.o <- foo.c
> bar.o <- bar.c
> built-in.o <- read.c foo.c bar.c
>
> (cf. default current:
> built-in.o <- read.o foo.o bar.o)
And this discussion is btw. mood. If the general opinion is that we shall
include the -combine support all the kbuild infrastructure will anyway
be redone.
There are several small things that are not addressed in todays
implementation and that will be fixed one way or the other.
Sam
On Fri, 2006-08-25 at 08:01 +0200, Jan Engelhardt wrote:
> >> Compiling files on their own (`make drivers/foo/bar.o`) seems to make
> >> the optimization void. Sure, most people don't stop compiling in
> >> between. Just a note
> >
> >Actually I'm not entirely sure what you write is true. It'll _build_
> >fs/jffs2/read.o, for example, but it still won't then use it when I make
> >the kernel -- it'll just use fs/jffs2/jffs2.o which is built from all
> >the C files with --combine. So the optimisation isn't lost.
>
> Umm then it spends double the time in compilation, doing:
>
> read.o <- read.c
> foo.o <- foo.c
> bar.o <- bar.c
> built-in.o <- read.c foo.c bar.c
Only if you invoke make explicitly for read.o, foo.o and bar.o. If you
just type 'make' then it won't build those.
> >So to overcome this, we use GCC's __attribute__((externally_visible))
> >which, as documented, just makes it global again -- undoing the effect
> >of -fwhole-program just for this _one_ symbol.
>
> Interesting. __attribute__((visibility("default"))) does the same?
That much you can test for yourself without a fixed compiler. No, it
doesn't.
--
dwmw2
>> >> Compiling files on their own (`make drivers/foo/bar.o`) seems to make
>> >> the optimization void. Sure, most people don't stop compiling in
>> >> between. Just a note
>> >
>> >Actually I'm not entirely sure what you write is true. It'll _build_
>> >fs/jffs2/read.o, for example, but it still won't then use it when I make
>> >the kernel -- it'll just use fs/jffs2/jffs2.o which is built from all
>> >the C files with --combine. So the optimisation isn't lost.
>>
>> Umm then it spends double the time in compilation, doing:
>>
>> read.o <- read.c
>> foo.o <- foo.c
>> bar.o <- bar.c
>> built-in.o <- read.c foo.c bar.c
>
>Only if you invoke make explicitly for read.o, foo.o and bar.o. If you
>just type 'make' then it won't build those.
That's what I meant. Assume I explicitly built read.o foo.o and bar.o.
If I then run the regular make, it will rerun gcc for read.c foo.c and
bar.c rather than using the already-created .o files for linking.
Jan Engelhardt
--
On Fri, 2006-08-25 at 11:11 +0200, Jan Engelhardt wrote:
> That's what I meant. Assume I explicitly built read.o foo.o and bar.o.
> If I then run the regular make, it will rerun gcc for read.c foo.c and
> bar.c rather than using the already-created .o files for linking.
Yes. Just as if I run 'make fs/ntfs/inode.o' and then build my kernel,
my build of fs/ntfs/inode.o isn't used -- because I don't have
CONFIG_NTFS set.
You built something manually that wasn't needed, and then it wasn't
used. Is there a problem here?
--
dwmw2
>> That's what I meant. Assume I explicitly built read.o foo.o and bar.o.
>> If I then run the regular make, it will rerun gcc for read.c foo.c and
>> bar.c rather than using the already-created .o files for linking.
>
>You built something manually that wasn't needed, and then it wasn't
>used. Is there a problem here?
Although I cannot give you any logical reasons, there are possibly reasons
why building some things manually might help. Oh well, I think in that case
I just pass a make option CONFIG_NOCOMBINE, as you said.
Jan Engelhardt
--
On Fri, 2006-08-25 at 11:51 +0200, Jan Engelhardt wrote:
> Although I cannot give you any logical reasons, there are possibly reasons
> why building some things manually might help. Oh well, I think in that case
> I just pass a make option CONFIG_NOCOMBINE, as you said.
Of course. Sometimes when I'm debugging I make stuff manually with
EXTRA_CFLAGS='-O1 -g', and sometimes I'll probably want do the build
with CONFIG_COMBINED_COMPILE= as well.
That should certainly be possible -- and it _is_ possible with my
makefile hack but mostly by chance. It's a feature which we should
probably try to keep though.
--
dwmw2
On Fri, 2006-08-25 at 09:26 +0200, Sam Ravnborg wrote:
> And this discussion is btw. mood. If the general opinion is that we shall
> include the -combine support all the kbuild infrastructure will anyway
> be redone.
> There are several small things that are not addressed in todays
> implementation and that will be fixed one way or the other.
There's a few _big_ things that aren't addressed in my makefile hack
either. It's very much a proof of concept.
However, if we limit the use of --combine to files within the same
directory, the modifications to the kbuild infrastructure shouldn't be
too intrusive. I strongly suspect that files within the same directory
are where the majority of the benefit is anyway.
Whole-kernel optimisation is likely to be prohibitively expensive, and
can wait for LTO¹.
--
dwmw2
¹ http://www.gelato.org/pdf/apr2006/gelato_ICE06apr_lto_mitchell_codesourcery.pdf
On Thursday 24 August 2006 11:24 am, David Woodhouse wrote:
> Using a combination of these two compiler options for building kernel
> code leads to some useful optimisation
BusyBox has been doing this for months now: "build at once" is one of our
config options. I'd like to point out that gcc eats needs several hundred
megabytes of ram to do this and you have no useful progress indicator between
starting and ending. But the result is definitely smaller.
We also tell the linker "--sort-common --gc-sections" which may or may not
apply here...
Rob
--
Never bet against the cheap plastic solution.
On Fri, 2006-08-25 at 16:11 -0400, Rob Landley wrote:
> BusyBox has been doing this for months now: "build at once" is one of our
> config options. I'd like to point out that gcc eats needs several hundred
> megabytes of ram to do this and you have no useful progress indicator between
> starting and ending. But the result is definitely smaller.
It isn't that bad when you're only building a few files at a time -- I
wouldn't suggest doing it for the whole kernel.
And you get a nice progress indicator at the moment -- a new warning
about global register variables every time it eats a new file, due to
GCC PR27899 :)
> We also tell the linker "--sort-common --gc-sections" which may or may not
> apply here...
We want --gc-sections (which requires -ffunction-sections
-fdata-sections to be useful) but that's a separate issue. We were doing
it a long time ago for the FR-V kernel; Marcelo has been looking at it
recently for other architectures.
--
dwmw2
> And you get a nice progress indicator at the moment -- a new warning
> about global register variables every time it eats a new file, due to
> GCC PR27899 :)
Or pass -Q to the compiler, it'll print the name of every function
it compiles, and a nice timing report at the end.
Segher
David Woodhouse wrote:
> On Fri, 2006-08-25 at 16:11 -0400, Rob Landley wrote:
>
>> BusyBox has been doing this for months now: "build at once" is one of our
>> config options. I'd like to point out that gcc eats needs several hundred
>> megabytes of ram to do this and you have no useful progress indicator between
>> starting and ending. But the result is definitely smaller.
>>
>
> It isn't that bad when you're only building a few files at a time -- I
> wouldn't suggest doing it for the whole kernel.
>
I suggest a new makefile target for this.
I.e. "make bzImage" as always for those who do development and
recompile after small changes/patches.
And a "make optImage" (optimized image) when building a
kernel for production use, when you believe compiling every file
and spending lots of extra time is worth it.
Helge Hafting
>>
> I suggest a new makefile target for this.
>
> I.e. "make bzImage" as always for those who do development and
> recompile after small changes/patches.
> And a "make optImage" (optimized image) when building a
> kernel for production use, when you believe compiling every file
> and spending lots of extra time is worth it.
Good idea. :)
Jan Engelhardt
--
On Mon, 2006-08-28 at 12:52 +0200, Helge Hafting wrote:
> And a "make optImage" (optimized image) when building a
> kernel for production use, when you believe compiling every file
> and spending lots of extra time is worth it.
If we revamp the entire kbuild infrastructure to allow building the
_whole_ kernel with --combine then I might be inclined to agree -- we
could do that instead of a CONFIG_COMBINED_COMPILE option.
But if, as I suggest, we're doing the simple option which combines only
the files which tend to get most benefit from it -- those which are in
the same directory -- then there's not a lot of point in the separate
target. It really doesn't take that much extra time.
--
dwmw2
David Woodhouse wrote:
> On Mon, 2006-08-28 at 12:52 +0200, Helge Hafting wrote:
>> And a "make optImage" (optimized image) when building a
>> kernel for production use, when you believe compiling every file
>> and spending lots of extra time is worth it.
<snip>
> But if, as I suggest, we're doing the simple option which combines only
> the files which tend to get most benefit from it -- those which are in
> the same directory -- then there's not a lot of point in the separate
> target. It really doesn't take that much extra time.
I thought that it used rather a lot more RAM. I still often(ish) compile
a kernel on my PII/300/128M. It'd be moderately annoying if it got
slower, and there was no way to turn it off.
On Fri, 2006-09-01 at 20:35 +0100, Ian Stirling wrote:
> I thought that it used rather a lot more RAM. I still often(ish)
> compile a kernel on my PII/300/128M. It'd be moderately annoying if it
> got slower, and there was no way to turn it off.
There will definitely be a way to turn it off. Compilers with all the
relevant bugs fixed are not common yet -- in fact I think I may have the
_only_ existing builds with all the patches collected together.
--
dwmw2
--
VGER BF report: H 0