In reading include/linux/init.h, I was surprised to discover
that __init{,data} expands to nothing when compiling a module.
I was wondering if anyone is contemplating adding support for
__init{,data} in module loading, to reduce the memory footprints
of modules after they have been loaded.
Adam J. Richter __ ______________ 4880 Stevens Creek Blvd, Suite 104
[email protected] \ / San Jose, California 95129-1034
+1 408 261-6630 | g g d r a s i l United States of America
fax +1 408 261-6631 "Free Software For The Rest Of Us."
On Sun, 26 Nov 2000 07:30:44 -0800,
"Adam J. Richter" <[email protected]> wrote:
> In reading include/linux/init.h, I was surprised to discover
>that __init{,data} expands to nothing when compiling a module.
>I was wondering if anyone is contemplating adding support for
>__init{,data} in module loading, to reduce the memory footprints
>of modules after they have been loaded.
It has been discussed a few times but nothing was ever done about it.
AFAIK the savings were not seen to be that important because modules
occupy complete pages. __init would have to be stored in a separate
page which was then discarded. It would complicate insmod, rmmod,
ksymoops and gdb. gdb against the kernel already gets confused by
vmlinux data that is discarded and gdb has problems with modules at the
best of times. Definitely 2.5 material, if at all.
On Mon, Nov 27, 2000 at 09:54:57AM +1100, Keith Owens wrote:
> On Sun, 26 Nov 2000 07:30:44 -0800,
> "Adam J. Richter" <[email protected]> wrote:
> > In reading include/linux/init.h, I was surprised to discover
> >that __init{,data} expands to nothing when compiling a module.
> >I was wondering if anyone is contemplating adding support for
> >__init{,data} in module loading, to reduce the memory footprints
> >of modules after they have been loaded.
>
> It has been discussed a few times but nothing was ever done about it.
> AFAIK the savings were not seen to be that important because modules
> occupy complete pages. __init would have to be stored in a separate
> page which was then discarded. It would complicate insmod, rmmod,
> ksymoops and gdb. gdb against the kernel already gets confused by
> vmlinux data that is discarded and gdb has problems with modules at the
> best of times. Definitely 2.5 material, if at all.
One really nice addition to the modutils would be to allow module
dependencies to be stored in the headers of modules. i.e. NetWare
NLMs have a "module depends" field where you can list modules you
have dependencies on, and so does NT with DLL's, and the loader auto
loads these modules when you load any module. What's troublesome
about modutils in their current form is if you don't have a
modules.dep file, insmod isn't smart enough to read the depenency
modules from a binary header, and load them without doing a lot
of needless checking with modprobe. Having to depend on an external
text file for this looks tacky to customers and is error prone.
I understand that it's nice to be able to change modules around
and have the flexibilty to do this dynamically, but since modules
that use this are complied from the kernel, and the kernel
build always knows "who owns the APIs" there's no reason it could
or should not be done this way. i.e.
when loading ppp and it's modules:
slhc
ppp
ppp_deflate
bsd_comp
There's no reason that ppp could not have in it's binary header the
name "slhc" and any other modules it is dependent on so insmod
will load all of them if someone types
insmod ppp_deflate (should trigger load of all these modules). I
know it's works this way if there's a modules.dep file laying
around, but it would be nice for it to work this way without
needing the external text file.
Jeff
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> Please read the FAQ at http://www.tux.org/lkml/
On Sun, 26 Nov 2000 17:01:35 -0700,
"Jeff V. Merkey" <[email protected]> wrote:
>insmod ppp_deflate (should trigger load of all these modules). I
>know it's works this way if there's a modules.dep file laying
>around, but it would be nice for it to work this way without
>needing the external text file.
There is a clean split between modprobe and insmod, modprobe is the
high level command that does all the fancy checking for inter module
dependencies, handling aliases and extracting options from
modules.conf. insmod is the low level command that does exactly what
you tell it to do, no more, no less. The only smarts that insmod has
is the ability to take a module name without '/' and find it using the
patchs in modules.conf. That split between high and low level commands
is too useful to contaminate.
modules.conf already supports "above" and "below" commands for
non-standard dependencies. The problem of not having a module.dep on
the first boot of a new kernel was addressed in kernel 2.4.0-test5 or
thereabouts, make modules_install runs depmod to build modules.dep
ready for the first boot.
Keith Owens <[email protected]> wrote:
>"Adam J. Richter" <[email protected]> wrote:
>> In reading include/linux/init.h, I was surprised to discover
>>that __init{,data} expands to nothing when compiling a module.
>>I was wondering if anyone is contemplating adding support for
>>__init{,data} in module loading, to reduce the memory footprints
>>of modules after they have been loaded.
>It has been discussed a few times but nothing was ever done about it.
>AFAIK the savings were not seen to be that important because modules
>occupy complete pages. __init would have to be stored in a separate
>page which was then discarded. [...]
No, you could just discard the part after the next page
boundary. The expected savings would be about the same, since
the cases where the original code had just creeped over a page
boundary in many cases would result in dropping more memory savings
that the actual init size, from dropping those unused bytes
between the very end of the init section and the end of that page.
I say "about" the same becuase the distribution of text and data
sizes is not uniformly random within some fixed interval.
Since you would not have to bump the start address of a
section to the next page boundary, I wonder if it would still
complicate insmod et al.
In case there is any confusion, I am not suggesting that
this should go into the stock 2.4.0 releases.
However, I do find it helpful in allocating my time to
cosider that saving one page by something like this or by enhancing
gcc's variable placement saves as much space as 1024 eliminations
of "= 0" or "= NULL" static variable initializations.
Adam J. Richter __ ______________ 4880 Stevens Creek Blvd, Suite 104
[email protected] \ / San Jose, California 95129-1034
+1 408 261-6630 | g g d r a s i l United States of America
fax +1 408 261-6631 "Free Software For The Rest Of Us."
On Mon, Nov 27, 2000 at 10:45:34AM +1100, Keith Owens wrote:
> On Sun, 26 Nov 2000 17:01:35 -0700,
> "Jeff V. Merkey" <[email protected]> wrote:
> >insmod ppp_deflate (should trigger load of all these modules). I
> >know it's works this way if there's a modules.dep file laying
> >around, but it would be nice for it to work this way without
> >needing the external text file.
>
> There is a clean split between modprobe and insmod, modprobe is the
> high level command that does all the fancy checking for inter module
> dependencies, handling aliases and extracting options from
> modules.conf. insmod is the low level command that does exactly what
> you tell it to do, no more, no less. The only smarts that insmod has
> is the ability to take a module name without '/' and find it using the
> patchs in modules.conf. That split between high and low level commands
> is too useful to contaminate.
>
> modules.conf already supports "above" and "below" commands for
> non-standard dependencies. The problem of not having a module.dep on
> the first boot of a new kernel was addressed in kernel 2.4.0-test5 or
> thereabouts, make modules_install runs depmod to build modules.dep
> ready for the first boot.
Good. I am glad this is being addressed.
Jeff
On Sun, Nov 26, 2000 at 03:47:30PM -0800, Adam J. Richter wrote:
> Keith Owens <[email protected]> wrote:
> >"Adam J. Richter" <[email protected]> wrote:
> >> In reading include/linux/init.h, I was surprised to discover
> >>that __init{,data} expands to nothing when compiling a module.
> >>I was wondering if anyone is contemplating adding support for
> >>__init{,data} in module loading, to reduce the memory footprints
> >>of modules after they have been loaded.
>
> >It has been discussed a few times but nothing was ever done about it.
> >AFAIK the savings were not seen to be that important because modules
> >occupy complete pages. __init would have to be stored in a separate
> >page which was then discarded. [...]
>
> No, you could just discard the part after the next page
> boundary. The expected savings would be about the same, since
> the cases where the original code had just creeped over a page
> boundary in many cases would result in dropping more memory savings
> that the actual init size, from dropping those unused bytes
> between the very end of the init section and the end of that page.
> I say "about" the same becuase the distribution of text and data
> sizes is not uniformly random within some fixed interval.
>
> Since you would not have to bump the start address of a
> section to the next page boundary, I wonder if it would still
> complicate insmod et al.
>
> In case there is any confusion, I am not suggesting that
> this should go into the stock 2.4.0 releases.
>
> However, I do find it helpful in allocating my time to
> cosider that saving one page by something like this or by enhancing
> gcc's variable placement saves as much space as 1024 eliminations
> of "= 0" or "= NULL" static variable initializations.
Microsoft drivers have an .INIT code section that is initialization
ccode that get's chunked after it's loaded. Their model allows
memory segments to be defined as DISCARDABLE, which tells the loader
to chunk them after they get loaded in portable executable format.
If ELF has something like it, perhaps it should go in the loader?
Jeff
>
> Adam J. Richter __ ______________ 4880 Stevens Creek Blvd, Suite 104
> [email protected] \ / San Jose, California 95129-1034
> +1 408 261-6630 | g g d r a s i l United States of America
> fax +1 408 261-6631 "Free Software For The Rest Of Us."
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> Please read the FAQ at http://www.tux.org/lkml/
On Sun, 26 Nov 2000 19:49:43 -0700,
"Jeff V. Merkey" <[email protected]> wrote:
>Microsoft drivers have an .INIT code section that is initialization
>ccode that get's chunked after it's loaded. Their model allows
>memory segments to be defined as DISCARDABLE, which tells the loader
>to chunk them after they get loaded in portable executable format.
The loader is insmod, which does all its own reloaction and loading.
The problem is that ancillary tools like ksymoops, gdb, kdb and
possibly others do not expect sections to be discarded after load.
Adding the feature to insmod is fairly easy, fixing the ancillary tools
to understand that some sections are discarded after load is a bit
harder. Debugging is particularly messy, when an oops occurs how do we
tell if the __init data been discarded yet or not?
I have added this to my investigation list for modutils, ksymoops and
kdb 2.5, no promises.
On Mon, Nov 27, 2000 at 01:06:04PM +1100, Keith Owens wrote:
> On Sun, 26 Nov 2000 19:49:43 -0700,
> "Jeff V. Merkey" <[email protected]> wrote:
> >Microsoft drivers have an .INIT code section that is initialization
> >ccode that get's chunked after it's loaded. Their model allows
> >memory segments to be defined as DISCARDABLE, which tells the loader
> >to chunk them after they get loaded in portable executable format.
>
> The loader is insmod, which does all its own reloaction and loading.
> The problem is that ancillary tools like ksymoops, gdb, kdb and
> possibly others do not expect sections to be discarded after load.
> Adding the feature to insmod is fairly easy, fixing the ancillary tools
> to understand that some sections are discarded after load is a bit
> harder. Debugging is particularly messy, when an oops occurs how do we
> tell if the __init data been discarded yet or not?
>
> I have added this to my investigation list for modutils, ksymoops and
> kdb 2.5, no promises.
:-)
Jeff
On Mon, Nov 27, 2000 at 09:54:57AM +1100, Keith Owens wrote:
> On Sun, 26 Nov 2000 07:30:44 -0800,
> "Adam J. Richter" <[email protected]> wrote:
> > In reading include/linux/init.h, I was surprised to discover
> >that __init{,data} expands to nothing when compiling a module.
> >I was wondering if anyone is contemplating adding support for
> >__init{,data} in module loading, to reduce the memory footprints
> >of modules after they have been loaded.
>
> It has been discussed a few times but nothing was ever done about it.
Well, I've actually implemented it few years ago and even current modutils
you maintain support that already (see runsize member of struct module and
how is it assigned). __init stuff was not stored in a separate page and was
initially vmalloced together with the whole module, the only vm addition was
a shrink for a vmalloc area where it would free some pages from the end of
the area.
It lived in sparclinux-cvs for quite some time, but Linus have not accepted
it (I've posted several times).
I can dig the patch out of sparclinux CVS if anyone is interested.
Jakub