Enough people have asked for persistent module storage to at least
justify me writing the code. The design is simple.
MODULE_PARM(var,type) currently defines type as [min[-max]]{b,h,i,l,s}.
For persistent data support, type is now [min[-max]]{b,h,i,l,s}{p}, the
trailing 'p' for persistent is optional. Existing modutils only checks
one character after [min[-max]] so this is backwards compatible, no
need to upgrade modutils unless you want persistent data.
insmod takes parameters from modules.conf, from the saved persistent
data (see below) and from the command line, in that order. The last
value for a parameter takes precedence.
rmmod locates the object for the module using the __insmod_xxx_O ksyms
entry. If the object cannot be found or its timestamp has changed
since it was loaded then rmmod silently skips the persistent data.
Otherwise rmmod uses the .modinfo data in that object to determine the
address and type of the persistent parameters. Each persistent
parameter is extracted from the module being unloaded, formatted as a
module parameter (e.g. "irq=17") and written to /somewhere/module_name
which is a text file (vi is the ultimate configuration tool).
Almost all support is in user space. The only kernel change is to add
'p' to the end of module parameters that are to be persistent. Module
variables that are to be persistent and are not currently module
parameters need to be defined as MODULE_PARM(). The same kernel code
should work on 2.2 and 2.4 kernels, it should even work with modutils
2.1.121.
I have not decided where to save the persistent module parameters. It
could be under /lib/modules/<version>/persist or it could be under
/var/log or /var/run. I am tending towards /var/run/module_persist, in
any case it will be a modules.conf parameter.
On Tue, 07 Nov 2000 15:00:11 +1100,
Keith Owens <[email protected]> wrote:
>insmod takes parameters from modules.conf, from the saved persistent
>data (see below) and from the command line, in that order. The last
>value for a parameter takes precedence.
Correction: modprobe takes parameters from ...
insmod only does exactly what you tell it to. It does not get parameter
values from modules.conf or anywhere else, only from the command line.
Keith Owens <[email protected]> said:
[...]
> I have not decided where to save the persistent module parameters. It
> could be under /lib/modules/<version>/persist or it could be under
> /var/log or /var/run. I am tending towards /var/run/module_persist, in
> any case it will be a modules.conf parameter.
/var/lib/persist/<version>/<wherever-the-module-is-in-/lib/modules/<version>/>
or some such. It has to match the kernel version somewhere (in case module
interfaces change), and it also should mirror the tree under
/lib/modules/<version> if for no other reason that there might show up
several modules named <foo>.
--
Dr. Horst H. von Brand mailto:[email protected]
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Horst von Brand <[email protected]> said:
[Yes, I know this is bad taste...]
> Keith Owens <[email protected]> said:
>
> [...]
>
> > I have not decided where to save the persistent module parameters. It
> > could be under /lib/modules/<version>/persist or it could be under
> > /var/log or /var/run. I am tending towards /var/run/module_persist, in
> > any case it will be a modules.conf parameter.
>
> /var/lib/persist/<version>/<wherever-the-module-is-in-/lib/modules/<version>/>
>
> or some such. It has to match the kernel version somewhere (in case module
> interfaces change), and it also should mirror the tree under
> /lib/modules/<version> if for no other reason that there might show up
> several modules named <foo>.
Note! This _has_ to be in the / filesystem so it works before mounting the
rest of the stuff (if ever). This would rule out /var, and leave just
/lib/modules/<version>. Makes me quite unhappy...
--
Dr. Horst H. von Brand mailto:[email protected]
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
On Tue, 07 Nov 2000 09:45:42 -0300,
Horst von Brand <[email protected]> wrote:
>Keith Owens <[email protected]> said:
>> I have not decided where to save the persistent module parameters. It
>> could be under /lib/modules/<version>/persist or it could be under
>> /var/log or /var/run. I am tending towards /var/run/module_persist, in
>> any case it will be a modules.conf parameter.
>
>/var/lib/persist/<version>/<wherever-the-module-is-in-/lib/modules/<version>/>
>
>or some such. It has to match the kernel version somewhere (in case module
>interfaces change),
But then you get the problem that upgrading from one kernel release to
the next loses the persistent data; either option has potential
problems. I am tending towards no version number and have a flag on
insmod that says "the following parameters may not exist in the module,
be silent about any unknown symbols", the flag is automatically set by
modprobe for persistent parameters.
Going forward is not a problem, going backwards silently ignores
unknown parameters.
>and it also should mirror the tree under
>/lib/modules/<version> if for no other reason that there might show up
>several modules named <foo>.
It makes no sense to allow duplicate module names in the same kernel
tree. "modprobe foo" - which one gets loaded?
On Tue, 07 Nov 2000 10:30:39 -0300,
Horst von Brand <[email protected]> wrote:
>> Keith Owens <[email protected]> said:
>> > I have not decided where to save the persistent module parameters. It
>> > could be under /lib/modules/<version>/persist or it could be under
>> > /var/log or /var/run. I am tending towards /var/run/module_persist, in
>> > any case it will be a modules.conf parameter.
>
>Note! This _has_ to be in the / filesystem so it works before mounting the
>rest of the stuff (if ever). This would rule out /var, and leave just
>/lib/modules/<version>. Makes me quite unhappy...
Modules are loaded before non-root file systems are mounted, damn!
Looks like persistent data has to be stored in /lib/modules/persist (no
<version>, see earlier mail). If somebody wants both a read only /lib
and persistent data then /lib/modules/persist must be a symlink and
they must mount the target file system before loading modules with
persistent data.
> Note! This _has_ to be in the / filesystem so it works before mounting the
> rest of the stuff (if ever). This would rule out /var, and leave just
> /lib/modules/<version>. Makes me quite unhappy...
The /lib filesystem is likely not writable so /var is the right default.
Any reason it cant be overridden in modules.conf ?
Keith Owens <[email protected]> said:
[...]
> It makes no sense to allow duplicate module names in the same kernel
> tree. "modprobe foo" - which one gets loaded?
Why the tree then?
--
Dr. Horst H. von Brand mailto:[email protected]
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
On Tue, 07 Nov 2000 11:47:57 -0300,
Horst von Brand <[email protected]> wrote:
>Keith Owens <[email protected]> said:
>> It makes no sense to allow duplicate module names in the same kernel
>> tree. "modprobe foo" - which one gets loaded?
>
>Why the tree then?
Mainly so you can "modprobe -t net \*" and load all modules with /net/
in their pathname. pcmcia modules need to be identified and linked.
mkinitrd also needs paths to identify scsi modules. The old system of
assigning pathnames was manual, fragile and error prone, copying the
kernel tree works. But whether modules are a flat directory, a
manually built two level directory or a copy of the kernel tree, it
still makes no sense to have duplicate names in the same kernel tree.
Keith Owens <[email protected]>:
> Enough people have asked for persistent module storage to at least
> justify me writing the code. The design is simple.
>
> MODULE_PARM(var,type) currently defines type as [min[-max]]{b,h,i,l,s}.
> For persistent data support, type is now [min[-max]]{b,h,i,l,s}{p}, the
> trailing 'p' for persistent is optional. Existing modutils only checks
> one character after [min[-max]] so this is backwards compatible, no
> need to upgrade modutils unless you want persistent data.
>
> insmod takes parameters from modules.conf, from the saved persistent
> data (see below) and from the command line, in that order. The last
> value for a parameter takes precedence.
>
> rmmod locates the object for the module using the __insmod_xxx_O ksyms
> entry. If the object cannot be found or its timestamp has changed
> since it was loaded then rmmod silently skips the persistent data.
> Otherwise rmmod uses the .modinfo data in that object to determine the
> address and type of the persistent parameters. Each persistent
> parameter is extracted from the module being unloaded, formatted as a
> module parameter (e.g. "irq=17") and written to /somewhere/module_name
> which is a text file (vi is the ultimate configuration tool).
How about including the posibility for some binary data. If something
like devfs were to store permanent data, then it would likely contain
a list of security labels (rwx/owner/group/(future mac)). This could
be a sizable block to store in ascii (but hex might be reasonable).
Some of this data (resource allocation control) would have no reason
to exist on a boot command line, as well as being too large.
This would shift the "MODULE_PARM" definition to something like:
MODULE_PARM(var,type,size)
where size is only used when the type would be binary.
This could hold a lot of data, as well as allow for parameters that
are configured in user space, but don't take effect until next device
load or boot (buffer size settings, kernel scheduling options, memory
resource controls...). An additional option would be an IOCTL (or something)
on the resource file to indicate a userspace update had been made (including
the parameter identifier) so that the appropriate driver/module could
be requested to perform an update. This would be most usefull for being
able to atomicly change kernel parameters (scheduling or resource controls).
> Almost all support is in user space. The only kernel change is to add
> 'p' to the end of module parameters that are to be persistent. Module
> variables that are to be persistent and are not currently module
> parameters need to be defined as MODULE_PARM(). The same kernel code
> should work on 2.2 and 2.4 kernels, it should even work with modutils
> 2.1.121.
>
> I have not decided where to save the persistent module parameters. It
> could be under /lib/modules/<version>/persist or it could be under
> /var/log or /var/run. I am tending towards /var/run/module_persist, in
> any case it will be a modules.conf parameter.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
On Tue, 7 Nov 2000 10:01:02 -0600 (CST),
Jesse Pollard <[email protected]> wrote:
>Keith Owens <[email protected]>:
>> Enough people have asked for persistent module storage to at least
>> justify me writing the code. The design is simple.
>>
>> MODULE_PARM(var,type) currently defines type as [min[-max]]{b,h,i,l,s}.
>> For persistent data support, type is now [min[-max]]{b,h,i,l,s}{p}, the
>
>How about including the posibility for some binary data. If something
>like devfs were to store permanent data, then it would likely contain
>a list of security labels (rwx/owner/group/(future mac)). This could
>be a sizable block to store in ascii (but hex might be reasonable).
>
>This would shift the "MODULE_PARM" definition to something like:
> MODULE_PARM(var,type,size)
>where size is only used when the type would be binary.
#define MAX_PERSIST 100
int mode[MAX_PERSIST];
int owner[MAX_PERSIST];
int group[MAX_PERSIST];
MODULE_PARM(mode, "1-" __MODULE_STRING(MAX_PERSIST) "ip");
MODULE_PARM(owner, "1-" __MODULE_STRING(MAX_PERSIST) "ip");
MODULE_PARM(group, "1-" __MODULE_STRING(MAX_PERSIST) "ip");
Resulting data looks like this, trailing all zero values are removed.
mode=420,420,493
owner=0,0,1
group=0,0,2
No need for a separate size field. Note that MODULE_PARM is built at
compile time so all persistent data must have a fixed compile time
size.
Pure binary immediately runs into kernel version skew problems. If the
data in kernel 2.4.n is
struct { int mode; int owner; int group; } persistent[MAX_PERSIST];
but kernel 2.4.n+1 has
struct { int mode; int owner; int group; long acl; } persistent[MAX_PERSIST];
Then saving binary data from kernel 2.4.n and loading it into 2.4.n+1
or vice versa results in garbage because the structure format has
changed. Separating individual fields and treating them as text has no
such problem. I have already decided that persistent data will not be
passed to insmod on the command line, instead insmod will read directly
from the saved data file, that removes any worries about command line
limitations.
>This could hold a lot of data, as well as allow for parameters that
>are configured in user space, but don't take effect until next device
>load or boot (buffer size settings, kernel scheduling options, memory
>resource controls...).
If the values are implemented via modules then make them persistent.
OTOH if they are implemented via code that is built into the kernel
then insmod is far too late.
>An additional option would be an IOCTL (or something)
>on the resource file to indicate a userspace update had been made (including
>the parameter identifier) so that the appropriate driver/module could
>be requested to perform an update. This would be most usefull for being
>able to atomicly change kernel parameters (scheduling or resource controls).
Now you are getting into the area of configuration utilities and the
interface between such utilities and the kernel as a whole, not just
modules. That is a seperate problem and is best left to the
configuration tools that already exist. Module persistent data is
intended for values in individual modules that the user changes daily
or even hourly (volume, TV tuner), not for overall system control.
From: Keith Owens <[email protected]>
> On Tue, 7 Nov 2000 10:01:02 -0600 (CST),
> Jesse Pollard <[email protected]> wrote:
> >Keith Owens <[email protected]>:
> >> Enough people have asked for persistent module storage to at least
> >> justify me writing the code. The design is simple.
> >>
> >> MODULE_PARM(var,type) currently defines type as [min[-max]]{b,h,i,l,s}.
> >> For persistent data support, type is now [min[-max]]{b,h,i,l,s}{p}, the
> >
> >How about including the posibility for some binary data. If something
> >like devfs were to store permanent data, then it would likely contain
> >a list of security labels (rwx/owner/group/(future mac)). This could
> >be a sizable block to store in ascii (but hex might be reasonable).
> >
> >This would shift the "MODULE_PARM" definition to something like:
> > MODULE_PARM(var,type,size)
> >where size is only used when the type would be binary.
>
> #define MAX_PERSIST 100
> int mode[MAX_PERSIST];
> int owner[MAX_PERSIST];
> int group[MAX_PERSIST];
> MODULE_PARM(mode, "1-" __MODULE_STRING(MAX_PERSIST) "ip");
> MODULE_PARM(owner, "1-" __MODULE_STRING(MAX_PERSIST) "ip");
> MODULE_PARM(group, "1-" __MODULE_STRING(MAX_PERSIST) "ip");
>
> Resulting data looks like this, trailing all zero values are removed.
>
> mode=420,420,493
> owner=0,0,1
> group=0,0,2
>
> No need for a separate size field. Note that MODULE_PARM is built at
> compile time so all persistent data must have a fixed compile time
> size.
I'll buy that - but it does mean that if there are 300 items then it
will get LONG... but then, that can be handled. I was thinking of the
"parameter" to possibly being a full data structure that could be updated
in an atomic manner, with a minimum of overhead (no number conversions
in the kernel).
> Pure binary immediately runs into kernel version skew problems. If the
> data in kernel 2.4.n is
>
> struct { int mode; int owner; int group; } persistent[MAX_PERSIST];
>
> but kernel 2.4.n+1 has
>
> struct { int mode; int owner; int group; long acl; } persistent[MAX_PERSIST];
>
> Then saving binary data from kernel 2.4.n and loading it into 2.4.n+1
> or vice versa results in garbage because the structure format has
> changed. Separating individual fields and treating them as text has no
> such problem. I have already decided that persistent data will not be
> passed to insmod on the command line, instead insmod will read directly
> from the saved data file, that removes any worries about command line
> limitations.
The identification of data version should be left up to the userspace
utility that retrieves the data. That way different versions wouldn't have
a skew. If the utility used a file format like "ar", then the access key
could contain the kernel version, the module, and the parameter. If not the
kernel version, then the module version - with the kernel version represented
by a file identifier (/var/persist/`uname -r` ?).
> >This could hold a lot of data, as well as allow for parameters that
> >are configured in user space, but don't take effect until next device
> >load or boot (buffer size settings, kernel scheduling options, memory
> >resource controls...).
>
> If the values are implemented via modules then make them persistent.
> OTOH if they are implemented via code that is built into the kernel
> then insmod is far too late.
For some things, yes. I was thinking of things like automatically changing
the scheduling priorities for batch+interactive use. Also things like
fair-share scheduler parameters, resident set size/swap resource control,
(other large system capabilities, I admit).
The only large (data sized) thing I can think of right now would be
re-loading default sounds into a audio devices wave table. The other
things are closer to having only 20-30 values at a time.
>
> >An additional option would be an IOCTL (or something)
> >on the resource file to indicate a userspace update had been made (including
> >the parameter identifier) so that the appropriate driver/module could
> >be requested to perform an update. This would be most usefull for being
> >able to atomicly change kernel parameters (scheduling or resource controls).
>
> Now you are getting into the area of configuration utilities and the
> interface between such utilities and the kernel as a whole, not just
> modules. That is a seperate problem and is best left to the
> configuration tools that already exist. Module persistent data is
> intended for values in individual modules that the user changes daily
> or even hourly (volume, TV tuner), not for overall system control.
It looked to be closely related.
I know it wasn't considered, but batch schedulers may have their parameters
changed hourly. My site currently works with one that has parameters changed
to reflect available resources for future scheduling cycles that use updated
job priorities to determine how the system should respond.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
On Tue, 7 Nov 2000 16:30:22 -0600 (CST),
Jesse Pollard <[email protected]> wrote:
> From: Keith Owens <[email protected]>
>> No need for a separate size field. Note that MODULE_PARM is built at
>> compile time so all persistent data must have a fixed compile time
>> size.
>
>I'll buy that - but it does mean that if there are 300 items then it
>will get LONG... but then, that can be handled. I was thinking of the
>"parameter" to possibly being a full data structure that could be updated
>in an atomic manner, with a minimum of overhead (no number conversions
>in the kernel).
Irrelevant. Remember that persistent module data is only set when the
module is loaded and retrieved when it removed. Atomicity and speed
are not an issue at those points.
>> Pure binary immediately runs into kernel version skew problems.
>
>The identification of data version should be left up to the userspace
>utility that retrieves the data.
The userspace utilities are insmod and rmmod. No, I am not going to
put version numbers into the saved data.
>For some things, yes. I was thinking of things like automatically changing
>the scheduling priorities for batch+interactive use. Also things like
>fair-share scheduler parameters, resident set size/swap resource control,
>(other large system capabilities, I admit).
All of which are system level tuning parameters that vary based on time
and/or load. My MVS sysprog hat says that there is no point in saving
these parameters across a reboot. Instead you have global targets
which rarely change and let the system tue to meet the targets - like
MVS Work Load Manager. These settings are nothing to do with modules.
>I know it wasn't considered, but batch schedulers may have their parameters
>changed hourly. My site currently works with one that has parameters changed
>to reflect available resources for future scheduling cycles that use updated
>job priorities to determine how the system should respond.
And what is the point of saving those parameters? Firstly they will
not be implemented via modules. Secondly the values just before
shutdown and at startup will not be useful for the peak hour load, nor
for the overnight backup window. Again, set targets and let the system
auto tune. The only thing you save are the overall targets, those are
already in text files with their own specific format.
In message <[email protected]> you write:
> On Tue, 07 Nov 2000 10:30:39 -0300,
> Horst von Brand <[email protected]> wrote:
> >Note! This _has_ to be in the / filesystem so it works before mounting the
> >rest of the stuff (if ever). This would rule out /var, and leave just
> >/lib/modules/<version>. Makes me quite unhappy...
>
> Modules are loaded before non-root file systems are mounted, damn!
modules.conf already breaks FHS lib/ badly enough. Modules loaded
before /var is mounted won't get persistant data. Too bad; they
have to do something sane when it doesn't exist anyway.
> Looks like persistent data has to be stored in /lib/modules/persist (no
> <version>, see earlier mail).
You need versions: binary data is too prone to change (proven kernel
history). It's the kernel installer's duty to know which ones can be
safely linked/copied to the new version.
Otherwise every data change requires a new symbol name: and this will
happen all the time.
Rusty.
--
Hacking time.
Followup to: <[email protected]>
By author: Rusty Russell <[email protected]>
In newsgroup: linux.dev.kernel
>
> > Modules are loaded before non-root file systems are mounted, damn!
>
> modules.conf already breaks FHS lib/ badly enough. Modules loaded
> before /var is mounted won't get persistant data. Too bad; they
> have to do something sane when it doesn't exist anyway.
>
Last I checked modules.conf was in /etc, not in /lib.
>
> > Looks like persistent data has to be stored in /lib/modules/persist (no
> > <version>, see earlier mail).
>
> You need versions: binary data is too prone to change (proven kernel
> history). It's the kernel installer's duty to know which ones can be
> safely linked/copied to the new version.
>
> Otherwise every data change requires a new symbol name: and this will
> happen all the time.
>
Remember that we cannot rely on ANY form of persistent storage to be
available in the beginning; / may very well be readonly (on a ROM,
say.) Since that means that we can't rely on writable storage being
available until at least one other filesystem has been mounted, it
might as well be the standard for variable data, i.e. /var.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt
On Thu, 09 Nov 2000 15:52:47 +1100,
Rusty Russell <[email protected]> wrote:
>In message <[email protected]> you write:
>> Looks like persistent data has to be stored in /lib/modules/persist (no
>> <version>, see earlier mail).
>
>You need versions: binary data is too prone to change (proven kernel
>history). It's the kernel installer's duty to know which ones can be
>safely linked/copied to the new version.
Not saving persistent data in binary format, so no problem.
On 8 Nov 2000 21:09:49 -0800,
"H. Peter Anvin" <[email protected]> wrote:
>Remember that we cannot rely on ANY form of persistent storage to be
>available in the beginning; / may very well be readonly (on a ROM,
>say.) Since that means that we can't rely on writable storage being
>available until at least one other filesystem has been mounted, it
>might as well be the standard for variable data, i.e. /var.
Agreed. Default is /var/lib/modules-persist. People who have a
separate /var partition that is mounted after modules are loaded can
use / and specify
persist /lib/modules/persist
in /etc/modules.conf. If you have a separate /var and you do not want
to write to /, change your initialization scripts to load modules after
mounting /var.
On Tue, Nov 07, 2000 at 01:55:59PM +0000, Alan Cox wrote:
> > Note! This _has_ to be in the / filesystem so it works before mounting the
> > rest of the stuff (if ever). This would rule out /var, and leave just
> > /lib/modules/<version>. Makes me quite unhappy...
>
> The /lib filesystem is likely not writable so /var is the right default.
In theory yes. In practice /var is often a mounted filesyste so for
modules loaded before /var is mounted this solution gets somewhat ugly.
Ralf