LinuxLists.cc - Quick and dirty miniconfig howto, with feature suggestions.

2005-11-17 13:10:26

Subject: Quick and dirty miniconfig howto, with feature suggestions.

--- What is a miniconfig?

A new feature of 2.6.15 lets you use miniature configuration files, listing
just the symbols you want to enable and letting the configurator enable any
dependencies to give you a valid configuration.

To make it work, create a mini.config file and run allnoconfig (to create
a .config file with all unspecified symbols switched off) with the extra
argument "KCONFIG_ALLCONFIG=mini.config".

--- Advantages of miniconfigs

Miniconfigs have several advantages over conventional configuration files:

* They're portable between versions. A miniconfig from linux 2.6.15 will
probably build an equivalent 2.6.16 kernel.

* It's easy to see exactly what features have been specified.

* Miniconfigs are flexible and human editable, while generated .config files
are brittle and easy to break if you try to edit them. (And if you
redirect the output > /dev/null so you just see stderr, it'll show you
any unrecognized symbols it couldn't set due to typos.)

--- Real-world example.

Here's the mini.config I use to build User Mode Linux:

CONFIG_MODE_SKAS=y
CONFIG_BINFMT_ELF=y
CONFIG_HOSTFS=y
CONFIG_SYSCTL=y
CONFIG_STDERR_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_UBD=y
CONFIG_TMPFS=y
CONFIG_SWAP=y
CONFIG_LBD=y
CONFIG_EXT2_FS=y
CONFIG_PROC_FS=y

And here's how I build and test it (as a normal user, not as root):

# Configure, building in an external directory and using a mini.config file in
# my home directory.

make KCONFIG_ALLCONFIG=~/mini.config ARCH=um O=../linux-umlbuild allnoconfig

# change to build directory and build User Mode Linux

cd ../linux-umlbuild
make ARCH=um

# Test run

./linux rootfstype=hostfs rw init=/bin/sh
$ whoami
$ mount -t proc /proc /proc
$ cat /proc/cpuinfo
$ exit

--- Suggestions.

1) Add a "make miniconfig" which works like allnoconfig but A) takes
mini.config as its' default name, B) redirects stdout to /dev/null to make it
easier to spot typoed symbols, C) aborts (exits with an error, does not write
new .config) if mini.config isn't found or if it contains an unrecognized
symbol.

2) Fix the interaction with O= so that it looks for the mini.config file in
the O= directory and not the source directory, so people don't _have_ to
specify KCONFIG_ALLCONFIG when building out of tree.

2005-11-20 03:09:06

by Roman Zippel

[permalink] [raw]

Subject: Re: Quick and dirty miniconfig howto, with feature suggestions.

Hi,

On Thu, 17 Nov 2005, Rob Landley wrote:

> 1) Add a "make miniconfig" which works like allnoconfig but A) takes
> mini.config as its' default name, B) redirects stdout to /dev/null to make it
> easier to spot typoed symbols, C) aborts (exits with an error, does not write
> new .config) if mini.config isn't found or if it contains an unrecognized
> symbol.

I think I better make allnoconfig silent (unless with V=1 or something),
which makes it your miniconfig already almost like allnoconfig.
I'm not quite sure about aborting there are other error possibilities
(e.g. new dependencies), so you never quite can trust the error value
anyway.

> 2) Fix the interaction with O= so that it looks for the mini.config file in
> the O= directory and not the source directory, so people don't _have_ to
> specify KCONFIG_ALLCONFIG when building out of tree.

I indeed need to fix this.

bye, Roman

2005-11-20 03:25:57

Hi!

> P.S. I don't use git. Poked at it a few times, but I made the mistake of
> reading largeish chunks of the git man page and the git glossary in an
> attempt to get up to speed, and got a headache. Anything that can define
> "clean" in such a way that I'm _less_ sure of the definition afterwards:
> http://www.kernel.org/pub/software/scm/git/docs/glossary.html
> Learning git went back on the to-do list somewhere between cleaning behind the
> refrigerator and sorting my book collection by author...

Getting latest try via git is _really_ easy:

mkdir clean-cg; cd clean-cg
cg-init
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

... Do cg-update origin to pickup latest changes from Linus. You can
do cg-diff to see what changes you done in your local tree. cg-cancel
will kill any such changes, and cg-commit will make them permanent.

Sorry, I did not have time to look what's wrong with miniconfig, yet.

Pavel

--
Thanks, Sharp!

2005-11-23 08:59:00

Hi!

> > Ok, this one applied okay for me. It still does not seem to work:
> >
> > pavel@amd:/data/l/linux$ scripts/miniconfig.sh config.ok
> > Calculating mini.config...
> > pavel@amd:/data/l/linux$ cat mini.config
> > CONFIG_PM=y
> > pavel@amd:/data/l/linux$
> >
> > ...and yes, my config is definitely more complex than that, I
> > handselected only relevant PCI cards, for example.
>
> This is a wild hunch, but try changing the #!/bin/sh at the top of
> miniconfig.sh to #!/bin/bash and see if that makes a difference? (I vaguely
> remember an ancient email where you're using a funky shell?)

No, not it:

pavel@amd:~/sf/routeplanner$ ls -al /bin/sh
lrwxrwxrwx 1 root root 4 Nov 22 15:01 /bin/sh -> bash

Yes, I'm using some strange shell on zaurus system, but this is just
plain PC.
Pavel
--
Thanks, Sharp!

2005-11-25 21:12:44

by Rob Landley

[permalink] [raw]

On Sunday 27 November 2005 17:20, Roman Zippel wrote:
> Hi,
>
> On Fri, 25 Nov 2005, Rob Landley wrote:
> > Ok, what's the best thing I can do to help get this implemented, working
> > _with_ you rather than against?
>
> It's not exactly simple, as it requires some kconfig hacking.
> Something relatively simply would be to change the miniconfig.sh script
> into a C program, where it would have to access to all the information to
> do the job fast and correctly.

We are in agreement that miniconfig.sh sucks. (I always saw it as a temporary
measure, to be replaced with kconfig being able to write it out directly
someday.) Unfortunately, while I've sort of poked at kconfig before, I don't
understand its dependency mechanism well enough to calculate the result I
want reliably.

The current miniconfig.sh is based on a very simple (and slow) procedure, and
even then I still haven't figured out why miniconfig.sh run on a straight
allnoconfig insists that CONFIG_PM should be set. (It correctly eliminates
everything else...)

I can take a stab at making conf.c do this if you'd like. (Or possibly
confdata.c...) Is the (admittedly sucky) shell script your main objection?

> I think it can even be done in a single pass over all the symbols, where
> boolean/tristate symbols are checked if they are already at the minimum
> value and string/hex/int values are compared with their default values.

Minimum value?

> Next step could be to add a variation of allnoconfig with better error
> checking (e.g. checking that all requested symbols have been set),

Um, I thought my patch did that. If any unrecognized symbols were
encountered, my miniconfig patch would report it and exit with an error by
the simple expedient of making the warning count a global and checking it
afterwards. (I did a sort of -Werror for kconfig.) If it attempts to set an
unrecognized symbol, it would already generate a warning, and if the warning
count is nonzero it bails out with an error at the end. Seemed to work quite
well, for me anyway...

What cases would that not catch?

(In theory, I could just has easily have done that part in a pure makefile
change, copying mini.config to allno.config, running the existing allnoconfig
suppressing stdout and capturing stderr to an environment variable, printing
the content of the environment variable, and then exiting with an error if
the environment variable isn't blank. I just thought patching the .c code
was the correct way to do it.)

> the basic allnoconfig functionality is just a few lines of code, the fun is
> in the extras.
>
> To further reduce the config size one could look at the dependecies, e.g.:
>
> config FOO
> depends on BAR && BAZ1 || BAZ2
>
> In this case FOO could also set BAR, but not BAZ1/BAZ2.

Good point, but the existing format is 90% of the gain for 10% of the effort.
Going from .config to miniconfig for my laptop's kernel, for example, goes
from 1370 lines to 138 lines, almost exactly a 10x reduction. And that can
be done (admittedly badly) today, with the patch I posted.

Dropping that 138 down to 120, or even to 100, is a polishing step in
comparison. Do you think there are another 30 lines that could be trimmed
out of that 138? (Attached.)

Looking through it, I remember setting most if not all of of those symbols in
menuconfig...

> But this also requires a new frontend to read such a minimized config
> file and is quite a bit more complex.

Right now, I have mini.configs that the existing infrastructure can read.
Currently, creating a miniconfig by hand involves setting exactly the set of
symbols that you would modify in menuconfig. And conceptually, to me, that's
what a miniconfig is. Only the symbols indicating exactly the features the
human wants to include. This has the advantage of being understandable to
said human.

This miniconfig is using the same symbol names as the existing config system
(which was originally designed to be human readable, long ago), it's just
eliminating unnecessary information (almost all dependency information,
except for some menu-opening symbols), and being a lot more forgiving to
human editing (while the existing .config explicitly says "don't edit" in a
comment at the top).

I don't understand what advantage the new frontend that's quite a bit more
complex would give here. Which symbols would we then not have to specify?
(Can you give me an example from any currently existing .config?)

> bye, Roman

Rob
--
Steve Ballmer: Innovation! Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

Attachments:

(No filename) (4.52 kB)
mini.config (2.79 kB)
Download all attachments

2005-11-29 01:00:39

by Roman Zippel

[permalink] [raw]

Subject: Re: [PATCH] make miniconfig (take 2)

Hi,

On Sun, 27 Nov 2005, Rob Landley wrote:

> The current miniconfig.sh is based on a very simple (and slow) procedure, and
> even then I still haven't figured out why miniconfig.sh run on a straight
> allnoconfig insists that CONFIG_PM should be set. (It correctly eliminates
> everything else...)

It seems because ACPI takes itself too important and sets the default to y
and this selects PM and prevents it from setting to n, until ACPI is
reset. default and select is a really bad combination. :-(

> > I think it can even be done in a single pass over all the symbols, where
> > boolean/tristate symbols are checked if they are already at the minimum
> > value and string/hex/int values are compared with their default values.
>
> Minimum value?

This is actually documented. :)
n - 0, m - 1, y - 2

> > Next step could be to add a variation of allnoconfig with better error
> > checking (e.g. checking that all requested symbols have been set),
>
> Um, I thought my patch did that. If any unrecognized symbols were
> encountered, my miniconfig patch would report it and exit with an error by
> the simple expedient of making the warning count a global and checking it
> afterwards. (I did a sort of -Werror for kconfig.) If it attempts to set an
> unrecognized symbol, it would already generate a warning, and if the warning
> count is nonzero it bails out with an error at the end. Seemed to work quite
> well, for me anyway...
>
> What cases would that not catch?

Symbols can be hidden by new dependencies.

> Good point, but the existing format is 90% of the gain for 10% of the effort.
> Going from .config to miniconfig for my laptop's kernel, for example, goes
> from 1370 lines to 138 lines, almost exactly a 10x reduction. And that can
> be done (admittedly badly) today, with the patch I posted.
>
> Dropping that 138 down to 120, or even to 100, is a polishing step in
> comparison. Do you think there are another 30 lines that could be trimmed
> out of that 138? (Attached.)

Yes, some symbols are hidden behind a lot of dependencies, if a user wants
to enable a new option, he only adds one new option and kconfig can try to
figure out the missing options.

bye, Roman

2005-11-29 02:26:29

by Rob Landley

[permalink] [raw]

Subject: Re: [PATCH] make miniconfig (take 2)

On Monday 28 November 2005 19:00, Roman Zippel wrote:

> > > I think it can even be done in a single pass over all the symbols,
> > > where boolean/tristate symbols are checked if they are already at the
> > > minimum value and string/hex/int values are compared with their default
> > > values.
> >
> > Minimum value?
>
> This is actually documented. :)
> n - 0, m - 1, y - 2

I understand that bit, but the problem with the single pass you suggested is
that symbols can be set by dependencies (like the one afflicting CONFIG_PM),
and those are neither off nor at their default, yet they're not actually
required.

So a third criteria would be that this symbol's value is occluded by some
other symbol. (And "occluded" could either be "in a menu that's not open" or
"set by somebody else's dependency".)

This starts sounding like a directed graph, and I am _not_ familiar enough
with the kconfig structure to try to work it out yet.

I think, however, that "at default" (for allnoconfig's default, anyway) or
"set by a dependency from another symbol" are the only two instances where we
don't care about this value because it can be regenerated. There's a corner
case where the directed graph has cycles in it (two things that each switch
the other on if they're enabled), but A) this is probably illegal already, B)
we don't care which one we skip as long as there's no _redundant_
information. Just be consistent (keep the first one).

How far off am I?

> > > Next step could be to add a variation of allnoconfig with better error
> > > checking (e.g. checking that all requested symbols have been set),
> >
> > Um, I thought my patch did that. If any unrecognized symbols were
> > encountered, my miniconfig patch would report it and exit with an error
> > by the simple expedient of making the warning count a global and checking
> > it afterwards. (I did a sort of -Werror for kconfig.) If it attempts to
> > set an unrecognized symbol, it would already generate a warning, and if
> > the warning count is nonzero it bails out with an error at the end.
> > Seemed to work quite well, for me anyway...
> >
> > What cases would that not catch?
>
> Symbols can be hidden by new dependencies.

Something gets enabled that then disables other things? Hmmm...

I've been thinking of things in terms of visibility. The menu this symbol
lives in either is or isn't open, and I can't set the symbol unless its menu
is open. If two different symbols control visibility for the same menu, it's
still really that the menu has one guard symbol and that dependencies of
other things are messing with that guard symbol. I think.

The way I've been thinking about miniconfigs is that each symbol in a
miniconfig is an action changing the default state. When you read in a
miniconfig, you start with all symbols at default allnoconfig values, and
then make this list of changes to that state (in order) letting the
dependencies do their thing with each change. (This maps to what a user
would do in menuconfig, selecting X, Y, and Z in this order. I could write
it down on a piece of paper for the user, so why can't the machine do it for
me?)

So really a miniconfig can't specify an inconsistent state, it can only be
redundant. (It can switch on things that later get switched off again by
dependencies on later symbols.) And all that means is the miniconfig is
longer than it needs to be, which could easily be the case if a human made it
by hand. Perhaps some kind of warning would be a good idea ("symbol
CONFIG_BLAH reverted by CONFIG_BLAH"), if it was easy to implement (perhaps
by tagging the "manually" selected symbols, ones which the miniconfig had
already explicitly specified a value for). But not a show-stopper to me.

> > Good point, but the existing format is 90% of the gain for 10% of the
> > effort. Going from .config to miniconfig for my laptop's kernel, for
> > example, goes from 1370 lines to 138 lines, almost exactly a 10x
> > reduction. And that can be done (admittedly badly) today, with the patch
> > I posted.
> >
> > Dropping that 138 down to 120, or even to 100, is a polishing step in
> > comparison. Do you think there are another 30 lines that could be
> > trimmed out of that 138? (Attached.)
>
> Yes, some symbols are hidden behind a lot of dependencies, if a user wants
> to enable a new option, he only adds one new option and kconfig can try to
> figure out the missing options.

You mean enabling a symbol in a closed menu would open the menu (but leave
everything else in it disabled, rather than at default values)? So
miniconfig wouldn't have to specify guard symbols at all anymore?

That would be nice. I suspect there are guard symbols that are also
functional (CONFIG_SCSI comes to mind), so the rule would have to be
something like guard symbols are only discardable when something in the menu
they protect _is_ specified.

However, that could go in as a future enhancement, because it falls under
"specifying redundant symbols". A miniconfig that did switch on the menus
(which would then be selected by a later dependency anyway, once the config
reading logic was upgraded) would still work in to configure a newer kernel
with a smarter configurator, wouldn't it?

> bye, Roman

Rob
--
Steve Ballmer: Innovation! Inigo Montoya: You keep using that word.
I do not think it means what you think it means.

2005-11-30 02:30:06

by Roman Zippel

[permalink] [raw]

Subject: Re: [PATCH] make miniconfig (take 2)

Hi,

On Mon, 28 Nov 2005, Rob Landley wrote:

> I understand that bit, but the problem with the single pass you suggested is
> that symbols can be set by dependencies (like the one afflicting CONFIG_PM),
> and those are neither off nor at their default, yet they're not actually
> required.

Setting everything to n can still be done in single pass, but that
requires working around the normal API and setting the values directly,
e.g.:

for_all_symbols(i, sym) {
if (!(sym->flags & SYMBOL_NEW))
continue;
switch (sym->type) {
case S_BOOLEAN:
case S_TRISTATE:
sym->user.tri = no;
sym->flags &= ~SYMBOL_NEW;
}
}

Now you only need to flush the old precomputed values
(sym_clear_all_valid()) and you're done.

> This starts sounding like a directed graph, and I am _not_ familiar enough
> with the kconfig structure to try to work it out yet.

The kconfig data is a directed acyclic graph, which allows to do most
operation deterministically in a single pass.
The problem is to transform the data from a valid state to another valid
state. The normal API allows only changes which keep the data in a valid
state (e.g. you can't set a not visible symbol), but it's possible to
bypass this and set multiple symbols, flush the old state and a new state
will generated which is consistent with the Kconfig rules.
Currently only conf_read() does this, i.e. you can throw pretty much any
.config at it and it will generate a valid (but not necessarily useful)
config from it.

> > Symbols can be hidden by new dependencies.
>
> Something gets enabled that then disables other things? Hmmm...
>
> I've been thinking of things in terms of visibility. The menu this symbol
> lives in either is or isn't open, and I can't set the symbol unless its menu
> is open. If two different symbols control visibility for the same menu, it's
> still really that the menu has one guard symbol and that dependencies of
> other things are messing with that guard symbol. I think.
>
> The way I've been thinking about miniconfigs is that each symbol in a
> miniconfig is an action changing the default state. When you read in a
> miniconfig, you start with all symbols at default allnoconfig values, and
> then make this list of changes to that state (in order) letting the
> dependencies do their thing with each change. (This maps to what a user
> would do in menuconfig, selecting X, Y, and Z in this order. I could write
> it down on a piece of paper for the user, so why can't the machine do it for
> me?)

This is not how it works (see above), reading the .config and user
interface changes are two different things.
After the config has been read, the data is brought into a consistent
state, which may mean the actual symbol value may be different, as they
are limited by their dependencies.

> > Yes, some symbols are hidden behind a lot of dependencies, if a user wants
> > to enable a new option, he only adds one new option and kconfig can try to
> > figure out the missing options.
>
> You mean enabling a symbol in a closed menu would open the menu (but leave
> everything else in it disabled, rather than at default values)? So
> miniconfig wouldn't have to specify guard symbols at all anymore?
>
> That would be nice. I suspect there are guard symbols that are also
> functional (CONFIG_SCSI comes to mind), so the rule would have to be
> something like guard symbols are only discardable when something in the menu
> they protect _is_ specified.

There are no special guard symbols, so if a SCSI driver is requested, this
means CONFIG_SCSI needs to be set.

bye, Roman