LinuxLists.cc - ISA hardware discovery -- the elegant solution

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Eric W. Biederman <[email protected]>:
> ISA is not a software enumerable bus especially not for unprivledged
> users. And no amount of complaining will change that. That is why we
> have PNP ISA and PCI.

But the kernel itself has to know how to probe and initialize these devices
at boot time, correct? That information is implicitly exported via
/var/log/dmesg -- I'm simply suggesting that it be a little more explicit.

> > But suppose the format of boot-time driver messages were standardized in a
> > format that included their config symbol in a discoverable form?
>
> If there was an ISA device in your example it might be interesting.

Some of the on-board devices on my Tyan Thunder are ISA.

> > With this change, generating a report on ISA hardware and other
> > facilities configured in at boot time would be trivial. This would
> > make the autoconfigurator much more capable. Best of all, the only
> > change required to accomplish this would be safe edits of print format
> > strings.
>
> It sounds like what you want is an lsmod that lists compiled in
> modules.

Would that be feasible without root privileges in order to read kmem?
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

[The disarming of citizens] has a double effect, it palsies the hand
and brutalizes the mind: a habitual disuse of physical forces totally
destroys the moral [force]; and men lose at once the power of
protecting themselves, and of discerning the cause of their
oppression.
-- Joel Barlow, "Advice to the Privileged Orders", 1792-93

2002-01-14 09:18:40

by H. Peter Anvin

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Followup to: <[email protected]>
By author: "Eric S. Raymond" <[email protected]>
In newsgroup: linux.dev.kernel
>
> But the kernel itself has to know how to probe and initialize these devices
> at boot time, correct? That information is implicitly exported via
> /var/log/dmesg -- I'm simply suggesting that it be a little more explicit.
>

dmesg is lossy. Don't assume it is complete.

-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>

2002-01-14 09:33:42

by Alan

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

> You haven't taken into consideration that not many distributions have
> drivers in kernel, and in particular ISA device drivers. Namely because
> ISA probes are ugly and require frobbing of memory in the vague hopes of

Red Hat for one basically avoids ISA probing in favour of user guidance. We
also use a standard kernel build and the more I think about this the more
I think Erik's tool is trying to be too clever and should simply build
a complete kernel set for the right cpu with the root fs and root fs block
device built into it

2002-01-14 10:16:32

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Eric S. Raymond wrote:

> With this change, generating a report on ISA hardware and other
> facilities configured in at boot time would be trivial. This would
> make the autoconfigurator much more capable. Best of all, the only
> change required to accomplish this would be safe edits of print format
> strings.

Better: create a /proc/driver and every driver will register in it.
This file can help some bug report (and not only autoconfigurator).

BTW, my new tests for:
memory (request_mem_region)
io port (request_region)
irq (request_irq)
dma (request_dma)
are nearly completes.
I think every ISA card should registers one of
these resources.
With the check of register_blkdev, register_chrdev
and miscdevices we should have a complete list of the
old ISA devices.
(this would detect only already detected devices,
but autoconfigure is not yet designed for bootfloppies
makers).

With such new test: no patch to kernel and nearly automatic
generation of probes.

Some patch are still welcome. I.e.:
some people with copy+paste have not changes the driver
string. A kernel patch will help distinguish the two drivers.
watchdog: to many driver register as 'watchdog' (or 'serial',
or 'ps2mouse'. This will create some difficulties to
autoconfigurators. A patch will help us (but it would break
other userspace tools?)

giacomo

2002-01-14 11:18:47

by Michael Lazarou (ETL)

[permalink] [raw]

Subject: RE: ISA hardware discovery -- the elegant solution

> -----Original Message-----
> From: Eric S. Raymond [mailto:[email protected]]
> Subject: ISA hardware discovery -- the elegant solution
>
>
> I've been thinking about the hardware-discovery problem for
> ISA devices,
> and there may be an elegant solution. It will take a number
> of small changes
> to the kernel sources, however.
>
> The kernel's device drivers have, of course, to include probe
> routines, and those hard-compiled in typically log the presence of
> their hardware to /var/log/mesg when it loads. By scanning that
> file, we in effect get to use those probes.

Doesn't this mean that you would need a fully functional kernel
before you get to run the autoconfigurator?

Michael

2002-01-14 12:33:11

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Michael Lazarou (ETL) wrote:

>>
>>The kernel's device drivers have, of course, to include probe
>>routines, and those hard-compiled in typically log the presence of
>>their hardware to /var/log/mesg when it loads. By scanning that
>>file, we in effect get to use those probes.
>>
>
> Doesn't this mean that you would need a fully functional kernel
> before you get to run the autoconfigurator?

Not a problem. Autoconfiguration is made to help configuring
the kernel, before to compile it. So you need a linux working
machine (actually you can cross-compile).

Our task is to allow user to compile a kernel, with the
needed drivers, without the non used drivers.

If you want a good running kernel image:
check the kernels in your distribution.
If you want to compile a general running kernel:
use the kernel sources in your distribution (
with your distribution's .config), or compile
the std kernel with your distribution's .config

If you want a working kernel to boot Linux in your

old/new/non-x86 machine: check the installation
note of your distribution. Use special kernel (from
your distribution,...)

You see: different task needs different tools.
Maybe we can merge some problems, but how? why?

[I still have a working version of autoconfigure in
shell, if some "boot-floppies" people need some
of our detection.]

But for old ISA cards (and some newer laptop) there is
only one method to find the right *kernel*:
boot and try (and changing kernels, parameters,...)
[check: installation document and internet resources].

We don't want a boot and retry for our configuration,
so let use the (incomplete) infos from kernel.

giacomo

2002-01-14 14:02:53

by Zwane Mwaikambo

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

On Mon, 14 Jan 2002, Alan Cox wrote:
> Red Hat for one basically avoids ISA probing in favour of user guidance. We
> also use a standard kernel build and the more I think about this the more
> I think Erik's tool is trying to be too clever and should simply build
> a complete kernel set for the right cpu with the root fs and root fs block
> device built into it

I'm definately not a fan of automagic kernel configurations, there are way
too many variables. Frankly the way i see it distro kernels do a damn good
job as it is right now and i don't see anything wrong with going the RH
errata kernel style for upgrading kernels (ditto for Debian and their apt
kernel upgrades). All Aunt Tillie has to do is click on the update button.

Are ISA device questions really that much of a big problem? The way i see
it, the user would get flooded more with the plethora of PCI and USB
devices more than anything. Then again i might be missing the "Big
Picture" ;)

Regards,
Zwane Mwaikambo

2002-01-14 14:27:05

by Michael Lazarou (ETL)

[permalink] [raw]

Subject: RE: ISA hardware discovery -- the elegant solution

> -----Original Message-----
> From: Giacomo Catenazzi [mailto:[email protected]]
> >>
> >>The kernel's device drivers have, of course, to include probe
> >>routines, and those hard-compiled in typically log the presence of
> >>their hardware to /var/log/mesg when it loads. By scanning that
> >>file, we in effect get to use those probes.
> >>
> >
> > Doesn't this mean that you would need a fully functional kernel
> > before you get to run the autoconfigurator?
>
>
> Not a problem. Autoconfiguration is made to help configuring
> the kernel, before to compile it. So you need a linux working
> machine (actually you can cross-compile).
>
> Our task is to allow user to compile a kernel, with the
> needed drivers, without the non used drivers.

OK, well I guess I am a little confused.

If I hit an autoconfigurator button then I would expect a kernel that
will boot and know everything there is to know about my machine.

Without probing the hardware how will the autoconfigurator cope with
the hardware changing underneath it?

Michael

2002-01-14 14:49:55

by Giacomo Catenazzi

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Michael Lazarou (ETL) wrote:

>>Not a problem. Autoconfiguration is made to help configuring
>>the kernel, before to compile it. So you need a linux working
>>machine (actually you can cross-compile).
>>
>>Our task is to allow user to compile a kernel, with the
>>needed drivers, without the non used drivers.
>>
>
> OK, well I guess I am a little confused.
>
> If I hit an autoconfigurator button then I would expect a kernel that
> will boot and know everything there is to know about my machine.

Actually there is no yet 'autoconfigurator button'.
I recommend to run a std configuration tool and to check
the configuration before the kernel build phase.

> Without probing the hardware how will the autoconfigurator cope with
> the hardware changing underneath it?

We probe the hardware (but in a soft manner).
Better: we probe nothing, we ask kernel to give us the results
of already done kernel probes. Thus we never hang, we never crash
machine, no 10-15 reboots to install a new kernel.
The good news: this is nearly enought.

Linux is magic: it can do infinite loops in 5 sec, but also
it can configure automatically a new kernel without real hardware
probes!.

giacomo

2002-01-14 16:27:27

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Michael Lazarou (ETL) <[email protected]>:
> Doesn't this mean that you would need a fully functional kernel
> before you get to run the autoconfigurator?

Yes, but this was always true.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

You need only reflect that one of the best ways to get yourself a
reputation as a dangerous citizen these days is to go about repeating
the very phrases which our founding fathers used in the great struggle
for independence. -- Attributed to Charles Austin Beard (1874-1948)

2002-01-14 16:44:22

by Giacomo Catenazzi

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Eric S. Raymond wrote:

> Giacomo Catenazzi <[email protected]>:
>
>>>With this change, generating a report on ISA hardware and other
>>>facilities configured in at boot time would be trivial. This would
>>>make the autoconfigurator much more capable. Best of all, the only
>>>change required to accomplish this would be safe edits of print format
>>>strings.
>>>
>>
>>Better: create a /proc/driver and every driver will register in it.
>>This file can help some bug report (and not only autoconfigurator).
>>
>
> That would be fine with me. But wouldn't it involve adding a new
> initialization-time call to every driver.

I think we can add it automatically, modifing one of the
the modules macros. (this works automatically only for
trit symbols).
Hmm. IIRC there is a big module/driver API (for modules) change
in 2.5 programs (driver registration,...). In this case we should
only modify the API before it will be effectivelly used, and
the changes will go automatically in kernel...

giacomo

2002-01-14 16:52:42

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Giacomo Catenazzi <[email protected]>:
> >>Better: create a /proc/driver and every driver will register in it.
> >>This file can help some bug report (and not only autoconfigurator).
> >
> > That would be fine with me. But wouldn't it involve adding a new
> > initialization-time call to every driver.
>
> I think we can add it automatically, modifing one of the
> the modules macros. (this works automatically only for
> trit symbols).

That would be OK. I think there are *very* few drivers left that
aren't modular.

> Hmm. IIRC there is a big module/driver API (for modules) change
> in 2.5 programs (driver registration,...). In this case we should
> only modify the API before it will be effectivelly used, and
> the changes will go automatically in kernel...

This sounds encouraging.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Ideology, politics and journalism, which luxuriate in failure, are
impotent in the face of hope and joy.
-- P. J. O'Rourke

2002-01-14 17:00:32

by Eli Carter

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

"Eric S. Raymond" wrote:
>
> Michael Lazarou (ETL) <[email protected]>:
> > Doesn't this mean that you would need a fully functional kernel
> > before you get to run the autoconfigurator?
>
> Yes, but this was always true.

And if the reason you are building a new kernel is that the old one is
mis-identifying some of your hardware? ;)

Could you maybe describe the problem you are trying to solve a bit more
clearly than "the hardware-discovery problem for ISA devices"? If you
are trying to discover the ISA devices, but rely upon them having
already been discovered, what are you accomplishing?

Puzzled,

Eli
--------------------. Real Users find the one combination of bizarre
Eli Carter \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

2002-01-14 17:11:52

by Wichert Akkerman

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

In article <[email protected]>,
Eli Carter <[email protected]> wrote:
>Could you maybe describe the problem you are trying to solve a bit more
>clearly than "the hardware-discovery problem for ISA devices"? If you
>are trying to discover the ISA devices, but rely upon them having
>already been discovered, what are you accomplishing?

The problem is that it is simply not possible to identify ISA devices
if they aren't isapnp devices. The only thing you can do is try to
probe for them by poking at different addresses and checking what happens.
Unfortunately this can do any of three things: show that a piece of
hardware exists, show that it is not there or completely crash your
machine if another unpexpected piece of hardware happens to be at the
place you are poking.

The best approach to doing ISA detection is ask the user which
devices he thinks he has installed and try looking for them while
praying bad things won't happen.

Wichert.

--
_________________________________________________________________
/ Nothing is fool-proof to a sufficiently talented fool \
| [email protected] http://www.liacs.nl/~wichert/ |
| 1024D/2FA3BC2D 576E 100B 518D 2F16 36B0 2805 3CB8 9250 2FA3 BC2D |

2002-01-14 17:26:32

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Giacomo Catenazzi <[email protected]>:
> > With this change, generating a report on ISA hardware and other
> > facilities configured in at boot time would be trivial. This would
> > make the autoconfigurator much more capable. Best of all, the only
> > change required to accomplish this would be safe edits of print format
> > strings.
>
> Better: create a /proc/driver and every driver will register in it.
> This file can help some bug report (and not only autoconfigurator).

That would be fine with me. But wouldn't it involve adding a new
initialization-time call to every driver.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

The whole of the Bill [of Rights] is a declaration of the right of the
people at large or considered as individuals... It establishes some
rights of the individual as unalienable and which consequently, no
majority has a right to deprive them of.
-- Albert Gallatin, Oct 7 1789

2002-01-14 17:37:32

by Alan

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

> That would be fine with me. But wouldn't it involve adding a new
> initialization-time call to every driver.

man lsmod

2002-01-14 17:37:22

by Eric W. Biederman

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

"Eric S. Raymond" <[email protected]> writes:

> Eric W. Biederman <[email protected]>:
> > ISA is not a software enumerable bus especially not for unprivledged
> > users. And no amount of complaining will change that. That is why we
> > have PNP ISA and PCI.
>
> But the kernel itself has to know how to probe and initialize these devices
> at boot time, correct? That information is implicitly exported via
> /var/log/dmesg -- I'm simply suggesting that it be a little more explicit.

For ISA not necessarily. It could simply be told that it was there.
/etc/modules.conf or it's compile time equivalent.

> > > But suppose the format of boot-time driver messages were standardized in a
> > > format that included their config symbol in a discoverable form?
> >
> > If there was an ISA device in your example it might be interesting.
>
> Some of the on-board devices on my Tyan Thunder are ISA.

I agree that ISA is a case. I simply didn't see it. And that was
my real point. Usually we are pretty silent about most ISA devices in
dmesg, because of most of them are mandated by the PC architecture and
we just expect them to be there.

> > > With this change, generating a report on ISA hardware and other
> > > facilities configured in at boot time would be trivial. This would
> > > make the autoconfigurator much more capable. Best of all, the only
> > > change required to accomplish this would be safe edits of print format
> > > strings.
> >
> > It sounds like what you want is an lsmod that lists compiled in
> > modules.
>
> Would that be feasible without root privileges in order to read kmem?

Given that nothing in the linux world reads kmem...

Honestly you have two separate problems.
Problem 1: How do you find out all of the hardware you have, as root.
Problem 2: How do you get that information as non-root.

Please try solving these problems separately.

Eric

2002-01-14 17:56:32

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Hello All , And what mechanism is going to be used for an -all-
compiled in kernel ? Everyone and there brother is so enamoured
of Modules . Not everyone uses nor will use modules .
Tia , JimL

On Mon, 14 Jan 2002, Alan Cox wrote:

> > That would be fine with me. But wouldn't it involve adding a new
> > initialization-time call to every driver.
>
> man lsmod
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-14 17:58:12

by Alan

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

> Hello All , And what mechanism is going to be used for an -all-
> compiled in kernel ? Everyone and there brother is so enamoured
> of Modules . Not everyone uses nor will use modules .

For 2.5 if things go to plan there will be no such thing as a "compiled in"
driver. They simply are not needed with initramfs holding what were once the
"compiled in" modules.

Alan

2002-01-14 18:08:13

[permalink] [raw]

Subject: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Eli Carter <[email protected]>:
> Could you maybe describe the problem you are trying to solve a bit more
> clearly than "the hardware-discovery problem for ISA devices"? If you
> are trying to discover the ISA devices, but rely upon them having
> already been discovered, what are you accomplishing?

Sure. Let's say Aunt Tillie needs a kernel update.

Aunt Tillie is running a distro kernel (modular everything) on the
machine her nephew Melvin bought her last year. The distro kernel
knows about the ISA devices on the machine.

She complains of occasional lockups, and that she gets skips when
playing her Guy Lombardo MP3s. Melvin says, over the phone: "Yup,
that version had some VM problems. And you need the low-latency stuff
that went in three releases ago. But never mind the technical stuff.
Just click on the 'kernel update' icon on your desktop." Melvin is
500 miles away at college.

Aunt Tillie clicks. She sees a message saying "Downloading kernel
sources" and a progress bar. Under the hood, the mchine is downloading
the tip of the stable branch from a kernel.org mirror site.

So why doesn't she use Red Hat or Mandrake's RPM update? Maybe she's
running something else. (You ain't going to tell me Aunt Tillie is ready
for Debian apt-get, either.) Maybe she wants a kernel that's compiled
for her AMD Athlon K6 rather than a 386. OK, so she doesn't know what
processor she has, she just remembers that Melvin told her you get a
faster kernel when you build it yourself.

(Aunt Tillie probably thinks having a faster kernel will mean she can
download images from her favorite knitting-pattern website more
quickly. Aunt Tillie is a little confused about the difference
between processor and network speed. That's OK; well-designed
technology should allow people the luxury of ignorance.)

Then the progress window changes to a message that says
"Configuring..." and some information about her hardware. This is the
autoconfigurator running inside a GUI shell.

Then it says "Building..." and another progress bar. Finally it says
"Please enter your root password so I can install the new kernel".
And, once she's done that, it tells her "Your new kernel will be named
'Sapphire' on your boot screen. Shall I reboot now?" She clicks "Yes".

Her machine reboots. She selects "Sapphire" on her boot screen. The
new kernel boots. She logs in. She surfs to her knitting-pattern
website. She logs out.

When she clicks "Shutdown", a popup says "This is not the same kernel
that will come up by default when you next boot. Should I make it the
default?" She clicks "Yes". Her kernel upgrade is done. It required
just four mouse clicks and one password entry. At no point did she
have to retain mental state about previous stages of the upgrade.
(Aunt Tillie is getting on in years; she isn't as good at retaining
abstract facts as she used to be.)

In fact, at a pinch we could have done away with the password entry,
presuming that anyone with physical access to the console is allowed
to perform canned administrative functions (though not to a root
shell).

We have the technology to do all of this now; the autoconfigurator is
the last nontrivial missing piece. And if we truly want world
domination, this is what it's going to take -- ease of use that is
Macintosh-like or better at *every* level of system use and
administration.

It takes a different way of thinking than most hackers are used to.
We're proud of our mad programming skillz and our ability to wrestle
with arcana. That pride isn't a bad thing -- except when it gets in
the way of designing systems that Aunt Tillie can use.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Sometimes the law defends plunder and participates in it. Sometimes the law
places the whole apparatus of judges, police, prisons and gendarmes at the
service of the plunderers, and treats the victim -- when he defends himself --
as a criminal. -- Frederic Bastiat, "The Law"

2002-01-14 18:16:22

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Alan Cox <[email protected]>:
> > That would be fine with me. But wouldn't it involve adding a new
> > initialization-time call to every driver.
>
> man lsmod

I couldn't make any sense at all out of this until your later posting
saying that compiled-in drivers are going away when initramfs comes in.
You might try being a *little* less terse...
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Love your country, but never trust its government.
-- Robert A. Heinlein.

2002-01-14 18:23:12

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> So why doesn't she use Red Hat or Mandrake's RPM update? Maybe she's
> running something else. (You ain't going to tell me Aunt Tillie is ready
> for Debian apt-get, either.) Maybe she wants a kernel that's compiled

I've seen apt-get used successfully by people whom I wouldnt trust to have
a glass front door in case they forgot to open it. The trick being that
"upgrade" is a desktop item someone put there that runs apt for them (or
gnome-apt or kpackage)

Now to do everything you describe does not need her to configure a custom
kernel tree. Not one bit. You think apt or up2date build each user a custom
kernel tree ?

You basically build everything modular as the distro kernel did, you install
it as the distro kernel would. You build an initrd as the distro kernel did.

Not only does it work, but it needs no configuration because when the box
was installed someone already configured it

2002-01-14 18:24:02

by Alan

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

> I couldn't make any sense at all out of this until your later posting
> saying that compiled-in drivers are going away when initramfs comes in.
> You might try being a *little* less terse...

why ? 8)

(and yes that was meant to be a joke)

Alan

2002-01-14 18:27:42

[permalink] [raw]

Subject: Hardwired drivers are going away?

Alan Cox <[email protected]>:
> For 2.5 if things go to plan there will be no such thing as a "compiled in"
> driver. They simply are not needed with initramfs holding what were once the
> "compiled in" modules.

This is something of a bombshell. Not necessarily a bad one, but...

Alan, do you have *any* *freakin'* *idea* how much more complicated
the CML2 deduction engine had to be because the basic logical entity
was a tristate rather than a bool? If this plan goes through, I'm
going to be able to drop out at least 20% of the code, with most of
that 20% being in the nasty complicated bits where the maintainability
improvement will be greatest. And I can get rid of the nasty "vitality"
flag, which probably the worst wart on the language.

Yowza...so how soon is this supposed to happen?
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Government should be weak, amateurish and ridiculous. At present, it
fulfills only a third of the role. -- Edward Abbey

2002-01-14 18:33:52

by Oliver Xymoron

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Michael Lazarou (ETL) <[email protected]>:
> > Doesn't this mean that you would need a fully functional kernel
> > before you get to run the autoconfigurator?
>
> Yes, but this was always true.

No it's not. You only need a kernel that can run your compiler.

--
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."

2002-01-14 18:37:22

by diamond

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

Hi,
If, as Alan described, you can depend on the distro's installation
program to have automatically identified all the hardware possible,
and for the user to have specified and additional (i.e. isa etc)
devices, then could not the autoconfigurator simply see what drivers
are currently in use (via /proc/devices etc), check
/etc/modules.conf for any that don't happen to be loaded at the
time, and use that info to configure the new kernel. Unless there is
some new piece of hardware that the new kernel supports that is
present (and isn't supported by the old one), if linux was installed
properly, you now have all the info you need, no? This removes the
need for the configurator to do any sort of probing, neither are
root proviledges required. Anyway, that's all probably blatantly
obvious etc, so I'll just be quiet now.

Steve
--
"My mom had Windows at work and it hurt her eyes real bad"

2002-01-14 18:38:52

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 12:52:28PM -0500, Eric S. Raymond wrote:
> She complains of occasional lockups, and that she gets skips when
> playing her Guy Lombardo MP3s. Melvin says, over the phone: "Yup,
> that version had some VM problems. And you need the low-latency stuff
> that went in three releases ago.

... and the 200 patches that the vendor added that she's become
so used to just being there...

> Just click on the 'kernel update' icon on your desktop."

*sigh*, and the package updater to get a new kernel for $distro
is insufficient because ?
distro kernel update has the following advantages.
- Comes complete with the 200 patches already applied.
- Is _tested_ by $distrovendor.
- If it screws up, and Aunt Tillie shelled out for support
(which of course, she did being the 'needing support' type)
$distrovendor will help. Ringing support and saying "Melvin
told me to install a new kernel, and now my box doesn't boot"
may not be a supportable scenario for all vendors.

> So why doesn't she use Red Hat or Mandrake's RPM update? Maybe she's
> running something else.

Red Hat & Mandrake are not the only distros with online update,
in fact, it's probably considered a must-have feature for most
distros these days.

> (You ain't going to tell me Aunt Tillie is ready
> for Debian apt-get, either.)

Wait a minute. Not ready for 'apt-get', but ready to build & run a
kernel made up of a collection of random patches on Melvin's say-so ?

> Maybe she wants a kernel that's compiled
> for her AMD Athlon K6 rather than a 386.

Various distro vendors update facilities give you this option.

> OK, so she doesn't know what processor she has

Some even autodetect.

> We have the technology to do all of this now;

Indeed. It's called YaST, Red Carpet, Mandrake Update, apt-get,
apt-rpm, and a plethora of other such tools.

> It takes a different way of thinking than most hackers are used to.

Yup. One where reinventing the wheel seems appropriate.

> We're proud of our mad programming skillz and our ability to wrestle
> with arcana. That pride isn't a bad thing -- except when it gets in
> the way of designing systems that Aunt Tillie can use.

The systems are designed, and the punchline is, that they work,
and they're being used out there today.

Dave.

--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs

2002-01-14 18:43:12

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox <[email protected]>:
> Now to do everything you describe does not need her to configure a custom
> kernel tree. Not one bit. You think apt or up2date build each user a custom
> kernel tree ?

Is it OK in your world that Aunt Tillie is dependent on a distro maker? Is
it OK that she never gets to have a kernel compiled for anything above the
least-common-denominator chip?

Not that I'm running down distro makers. They do a valuable job, and in fact
my approach relies on Aunt Tillie's machine starting life with an all-modular
distro kernel.

But the point of this game is for Aunt Tillie to have more and better
choices. Isn't that what we're supposed to be about?
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Government should be weak, amateurish and ridiculous. At present, it
fulfills only a third of the role. -- Edward Abbey

2002-01-14 18:46:04

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> that 20% being in the nasty complicated bits where the maintainability
> improvement will be greatest. And I can get rid of the nasty "vitality"
> flag, which probably the worst wart on the language.
>
> Yowza...so how soon is this supposed to happen?

Subject to availability. No contract implied or etc.. Its something to
tackle after the rest of initramfs works. Even if not then the lsmod case
can be made to work since its just a matter of putting the names in a segment
for the linker to collate.

We already have module_init/module_exit which is a big chunk of the cleanup.

2002-01-14 18:50:32

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> Is it OK in your world that Aunt Tillie is dependent on a distro maker? Is
> it OK that she never gets to have a kernel compiled for anything above the
> least-common-denominator chip?

The two don't follow.

The goal of a typical end user is "make it work, make it go away and do what
it did last week". Random mechanics hating car owners don't do engine tuning
jobs or fit turbochargers.

Secondly we've established we can pick the right CPU for the kernel reliably
that is seperate to modules.

Thirdly building a lot of stuff modular is the right choice anyway - in the
world of hot plugging and USB Grandma is not going to want to recompile her
kernel because she bought a new trackball to boost her quake score.

Alan

2002-01-14 18:54:15

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> 1. security, if you don't need any modules you can disable modules entirly
> and then it's impossible to add a module without patching the kernel first
> (the module load system calls aren't there)

Urban legend.

> 2. speed, there was a discussion a few weeks ago pointing out that there
> is some overhead for useing modules (far calls need to be used just in
> case becouse the system can't know where the module will be located IIRC)

I defy you to measure it on x86

> 3. simplicity in building kernels for other machines. with a monolithic
> kernel you have one file to move (and a bootloader to run) with modules
> you have to move quite a few more files.

tar or nfs mount; make modules_install.

2002-01-14 18:51:24

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

I can see a couple reasons for building a kernel without useing modules.

1. security, if you don't need any modules you can disable modules entirly
and then it's impossible to add a module without patching the kernel first
(the module load system calls aren't there)

2. speed, there was a discussion a few weeks ago pointing out that there
is some overhead for useing modules (far calls need to be used just in
case becouse the system can't know where the module will be located IIRC)

3. simplicity in building kernels for other machines. with a monolithic
kernel you have one file to move (and a bootloader to run) with modules
you have to move quite a few more files.

modules are good for distros and for machines that have to worry about
hotpluged equipment (but even that is less nessasary then it used to be)
but for servers that don't change their hardware they aren't nessasary,
why should they have to deal with them?

David Lang

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Date: Mon, 14 Jan 2002 13:10:50 -0500
> From: Eric S. Raymond <[email protected]>
> To: Alan Cox <[email protected]>
> Cc: Mr. James W. Laferriere <[email protected]>,
> Giacomo Catenazzi <[email protected]>,
> Linux Kernel List <[email protected]>
> Subject: Hardwired drivers are going away?
>
> Alan Cox <[email protected]>:
> > For 2.5 if things go to plan there will be no such thing as a "compiled in"
> > driver. They simply are not needed with initramfs holding what were once the
> > "compiled in" modules.
>
> This is something of a bombshell. Not necessarily a bad one, but...
>
> Alan, do you have *any* *freakin'* *idea* how much more complicated
> the CML2 deduction engine had to be because the basic logical entity
> was a tristate rather than a bool? If this plan goes through, I'm
> going to be able to drop out at least 20% of the code, with most of
> that 20% being in the nasty complicated bits where the maintainability
> improvement will be greatest. And I can get rid of the nasty "vitality"
> flag, which probably the worst wart on the language.
>
> Yowza...so how soon is this supposed to happen?
> --
> <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>
>
> Government should be weak, amateurish and ridiculous. At present, it
> fulfills only a third of the role. -- Edward Abbey
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2002-01-14 18:57:06

by Charles Cazabon

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Eric S. Raymond <[email protected]> wrote:
> Alan Cox <[email protected]>:
> > Now to do everything you describe does not need her to configure a custom
> > kernel tree. Not one bit. You think apt or up2date build each user a custom
> > kernel tree ?
>
> Is it OK in your world that Aunt Tillie is dependent on a distro maker? Is
> it OK that she never gets to have a kernel compiled for anything above the
> least-common-denominator chip?

Yes, and yes. Aunt Tillie is running Linux because someone installed a
distribution for her.

She is never going to need anything out of her kernel that her vendor-shipped
update kernels do not provide. She is never going to need the 1% performance
difference she might she if she custom-compiled a kernel for her architecture
rather than using the closest one shipped by her vendor.

> But the point of this game is for Aunt Tillie to have more and better
> choices. Isn't that what we're supposed to be about?

No. We're supposed to be about stuff that works. Vendor-shipped kernels work
for 99.9% of people. The remaining 0.1% have no need for an
"auto-configurator".

Charles
--
-----------------------------------------------------------------------
Charles Cazabon <[email protected]>
GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

2002-01-14 18:57:06

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Alan Cox <[email protected]>:
> > Yowza...so how soon is [the end of hard-compiled drivers] supposed to
> > happen?
>
> Subject to availability. No contract implied or etc.. Its something to
> tackle after the rest of initramfs works.

Dang. This will make the CML2 inference engine work better in some
funky corner cases, too. And its behavior will be easier to
understand all around.

Sign me up. This will be a good change; I like it when I can make things
better by taking features *out* of my code.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Every Communist must grasp the truth, 'Political power grows out of
the barrel of a gun.'
-- Mao Tse-tung, 1938, inadvertently endorsing the Second Amendment.

2002-01-14 19:01:35

by Larry McVoy

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 07:00:13PM +0000, Alan Cox wrote:
> Secondly we've established we can pick the right CPU for the kernel reliably
> that is seperate to modules.

Yup, I noticed this when I did a RH 7.2 install recently, pretty spiffy.
And it seems to work, 7.2 "feels" faster, especially groff for some reason
I don't understand.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2002-01-14 19:02:52

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Hello Alan ,

On Mon, 14 Jan 2002, Alan Cox wrote:
> > 1. security, if you don't need any modules you can disable modules entirly
> > and then it's impossible to add a module without patching the kernel first
> > (the module load system calls aren't there)
>
> Urban legend.
I do not agree . Got proof ? Yes that is a valid question .

> > 2. speed, there was a discussion a few weeks ago pointing out that there
> > is some overhead for useing modules (far calls need to be used just in
> > case becouse the system can't know where the module will be located IIRC)
> I defy you to measure it on x86
OK ,How about sparc-64/alpha/ia64/... ?

> > 3. simplicity in building kernels for other machines. with a monolithic
> > kernel you have one file to move (and a bootloader to run) with modules
> > you have to move quite a few more files.
> tar or nfs mount; make modules_install.
Please my laugh'o meter is stuck already . Sorry . JimL

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-14 19:07:12

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, Alan Cox wrote:

> > 1. security, if you don't need any modules you can disable modules entirly
> > and then it's impossible to add a module without patching the kernel first
> > (the module load system calls aren't there)
>
> Urban legend.
>

If this is the case then why do I get systemcall undefined error messages
when I make a mistake and attempt to load a module on a kernel without
modules enabled?

> > 2. speed, there was a discussion a few weeks ago pointing out that there
> > is some overhead for useing modules (far calls need to be used just in
> > case becouse the system can't know where the module will be located IIRC)
>
> I defy you to measure it on x86

during the discussion a few weeks ago there were people pointing out cases
where this overhead would be a problem.

> > 3. simplicity in building kernels for other machines. with a monolithic
> > kernel you have one file to move (and a bootloader to run) with modules
> > you have to move quite a few more files.
>
> tar or nfs mount; make modules_install.
>
not on my firewalls thank you.

David Lang

2002-01-14 19:05:35

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> > it did last week". Random mechanics hating car owners don't do engine tuning
> > jobs or fit turbochargers.
>
> No...but they do change their own oil and antifreeze. Upgrading your
> kernel should be as simple as changing your oil.

Yeah. I go to the garage versus I click "up 2 date". I don't mix custom
oils.

> > Secondly we've established we can pick the right CPU for the kernel reliably
> > that is seperate to modules.
>
> Right, but that doesn't get you a recompiled binary with extended instructions
> in it.

It does. Once you have picked the required processor type you will get the
right instructions. Except for the Athlon, Winchip and maybe the PIV
I've seen little evidence it matters. The Debian folk claim its worth < 1%
and I don't doubt them.

2002-01-14 19:08:32

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> > Urban legend.
> I do not agree . Got proof ? Yes that is a valid question .

Most of the rootkit type stuff I see nowdays includes code for loading
patches into module free kernels. Its a real no win. The better ones support
regexp scanning so they can patch kernels where the sysadmin thinks he/she
is cool and has hidden or crapped in System.map

> > > case becouse the system can't know where the module will be located IIRC)
> > I defy you to measure it on x86
> OK ,How about sparc-64/alpha/ia64/... ?

Not generally found in your grandmothers PC

> > > 3. simplicity in building kernels for other machines. with a monolithic
> > > kernel you have one file to move (and a bootloader to run) with modules
> > > you have to move quite a few more files.
> > tar or nfs mount; make modules_install.
> Please my laugh'o meter is stuck already . Sorry . JimL

Then fix it, because the above works well. Also remember that autoconfig
tools won't be able to guess remote machines very well 8)

2002-01-14 19:11:02

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> > Urban legend.
>
> If this is the case then why do I get systemcall undefined error messages
> when I make a mistake and attempt to load a module on a kernel without
> modules enabled?

Because you are using insmod. The guys who break into your computer probably
are not in their own language "that lame".

> > I defy you to measure it on x86
>
> during the discussion a few weeks ago there were people pointing out cases
> where this overhead would be a problem.

TLB miss overhead for kernel modules is very very hard to measure, at least
on the x86 platform. It can be an issue in userspace as you address much
larger objects and in certain cases like scanning an element in an array
of objects hit a TLB reload each scan or two (think about vertical line draw
unaccelerated in X11)

> > tar or nfs mount; make modules_install.
> >
> not on my firewalls thank you.

If you are that worried just burn a new root file system on CD when you
upgrade your firewall. At least I hope if you are going to pretend to be
so secure that tar is a bad idea you have no writable file store on the
machine ?

Alan

2002-01-14 19:10:54

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Alan Cox <[email protected]>:
> > For 2.5 if things go to plan there will be no such thing as a "compiled in"
> > driver. They simply are not needed with initramfs holding what were once the
> > "compiled in" modules.
>
> This is something of a bombshell. Not necessarily a bad one, but...
>
> Alan, do you have *any* *freakin'* *idea* how much more complicated
> the CML2 deduction engine had to be because the basic logical entity
> was a tristate rather than a bool? If this plan goes through, I'm
> going to be able to drop out at least 20% of the code, with most of
> that 20% being in the nasty complicated bits where the maintainability
> improvement will be greatest. And I can get rid of the nasty "vitality"
> flag, which probably the worst wart on the language.
>
> Yowza...so how soon is this supposed to happen?

Two technical obstacles:
a) on some architecures modular code is slower (IIRC, the problem is
with medium-range calls being faster than far ones and usable only in the
kernel proper). We probaly want to leave a gap after the .text and remap
.text of module in there - if I understand the problem that should be
enough, but that's really a question to folks dealing with these ports (PPC64
and Itanic?)
b) current differences between options parsing in case of built-in
and modular drivers.

The rest is trivial and should be done in 2.5.4-pre* or so.

But it still leaves you with tristate - instead of yes/module/no it's
yes/yes, but don't put it on initramfs/no. However, dependencies become
simpler - all you need is "I want this, that and that on initramfs" and
the rest can be found by depmod (i.e. configurator doesn't have to deal
with "FOO goes on initramfs (== old Y), so BAR and BAZ must go there
(== can't be M)").

2002-01-14 19:12:32

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Charles Cazabon <[email protected]>:
> Yes, and yes. Aunt Tillie is running Linux because someone installed a
> distribution for her.

You don't know that. Maybe she installed it herself.

> She is never going to need anything out of her kernel that her vendor-shipped
> update kernels do not provide.

*You can't know that.*

And your belief that you *can* know it is a key part of the elitist
developer psychology and implicit assumptions that keeps Linux mostly
inaccessible to the Aunt Tillies of the world.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

According to the National Crime Survey administered by the Bureau of
the Census and the National Institute of Justice, it was found that
only 12 percent of those who use a gun to resist assault are injured,
as are 17 percent of those who use a gun to resist robbery. These
percentages are 27 and 25 percent, respectively, if they passively
comply with the felon's demands. Three times as many were injured if
they used other means of resistance.
-- G. Kleck, "Policy Lessons from Recent Gun Control Research,"
Law and Contemporary Problems 49, no. 1. (Winter 1986.): 35-62.

2002-01-14 19:16:34

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Hello Alan ,

On Mon, 14 Jan 2002, Alan Cox wrote:
> > > Urban legend.
> > I do not agree . Got proof ? Yes that is a valid question .
> Most of the rootkit type stuff I see nowdays includes code for loading
> patches into module free kernels. Its a real no win. The better ones support
> regexp scanning so they can patch kernels where the sysadmin thinks he/she
> is cool and has hidden or crapped in System.map
Same as I am seeing . This does not eleviate the fact that
modules can & are a still a security issue . Yes , sloppy
administration is still the biggest culprit in 'most' compromises .

> > > > case becouse the system can't know where the module will be located IIRC)
> > > I defy you to measure it on x86
> > OK ,How about sparc-64/alpha/ia64/... ?
> Not generally found in your grandmothers PC
No, But it is -mine- ! If I can not have complete control over
my system due to the way the Fathers of this operating system
are taking it I will have no choice but to use another or be stuck
at a non-maintained version . Now mind you I LIKE LINUX !

> > > > 3. simplicity in building kernels for other machines. with a monolithic
> > > > kernel you have one file to move (and a bootloader to run) with modules
> > > > you have to move quite a few more files.
> > > tar or nfs mount; make modules_install.
> > Please my laugh'o meter is stuck already . Sorry . JimL
> Then fix it, because the above works well. Also remember that autoconfig
> tools won't be able to guess remote machines very well 8)
Show me , Please . I am not convinced . I shall try out the new
level of kernel without a doubt . But , I find the concept that
-everything- is dynamic to be wrong . Mind you My opinion .
Twyl , JimL

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-14 19:15:24

by Andrew Pimlott

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

On Mon, Jan 14, 2002 at 11:11:41AM -0500, Eric S. Raymond wrote:
> Michael Lazarou (ETL) <[email protected]>:
> > Doesn't this mean that you would need a fully functional kernel
> > before you get to run the autoconfigurator?
>
> Yes, but this was always true.

I think the point people are getting at is: If you're re-detecting
things that have already been detected, doesn't it seem you're going
about the problem the wrong way?

Most distributors need to solve essentially the same problem you're
solving (detect hardware and install drivers), but without compiling
a kernel (if only to spare the user the wait). To the extent that
they are successful (and they will presumably put considerable
effort into it), your compile-time probes are superfluous--you're
better off piggy-backing on the distribution's hardware detection.
Eg, derive your .config from the modules the distribution has
decided to load, or--even simpler--just compile a kernel with the
same .config as the distributor's kernel, and let the boot-time
scripts take care of the rest. The drawback is that this differs
across distributions--but see below.

Even if you can usually do a better job, you have two major
handicaps: One, you may have to repeat questions that the
distribution already asked, annoying the user. Two, if you ever
screw up something the distribution had working, the user will curse
you.

Which leads me to conclude that your compile-time autodetection,
while cool, is a dead-end as far as helping Cousin Billie (though I
do support the ultimate goal of letting him compile a kernel).
There are some scenarios where you win, but not enough IMO.

To correct this, retarget the project to solve the distributors'
problem. Ie, add a back-end that specifies modules and module
options, instead of a kernel .config. Assuming you do a good enough
job, and meet the distributor's other requirements (eg, running in
an install environment), then they will use your system for their
install-time hardware detection. You can store the results in a
standard (across distributions!) format, which administrators will
love (and which can be used by various to-be-written utilities).
Finally, you can later use the same database for compiling a
"stripped-down" custom kernel.

Andrew

2002-01-14 19:20:33

by Andrew Pimlott

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 14, 2002 at 10:57:19AM -0800, David Lang wrote:
> On Mon, 14 Jan 2002, Alan Cox wrote:
>
> > > 1. security, if you don't need any modules you can disable modules entirly
> > > and then it's impossible to add a module without patching the kernel first
> > > (the module load system calls aren't there)
> >
> > Urban legend.
>
> If this is the case then why do I get systemcall undefined error messages
> when I make a mistake and attempt to load a module on a kernel without
> modules enabled?

It's an urban legend that this is a security benefit. grep the
hacker zines for injecting code into a non-modular kernel.

> > > 2. speed, there was a discussion a few weeks ago pointing out that there
> > > is some overhead for useing modules (far calls need to be used just in
> > > case becouse the system can't know where the module will be located IIRC)
> >
> > I defy you to measure it on x86
^^^^^^^^^^
> during the discussion a few weeks ago there were people pointing out cases
> where this overhead would be a problem.

Above.

> > > 3. simplicity in building kernels for other machines. with a monolithic
> > > kernel you have one file to move (and a bootloader to run) with modules
> > > you have to move quite a few more files.
> >
> > tar or nfs mount; make modules_install.
> >
> not on my firewalls thank you.

You won't install tar on your firewalls? Weird :-)

Andrew

2002-01-14 19:05:34

by Arjan van de Ven

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

In article <[email protected]> you wrote:
> Not that I'm running down distro makers. They do a valuable job, and in fact
> my approach relies on Aunt Tillie's machine starting life with an all-modular
> distro kernel.

> But the point of this game is for Aunt Tillie to have more and better
> choices. Isn't that what we're supposed to be about?

While I'm absolutely not disputing the right of anyone to compile their own
kernel (I'm not trying to lock customers into my special platform like some
unnamed industrial giant *cough*), I just wonder why an AUTOMATIC generated
kernel with all the _relevant_ drivers modules will be "more and better"
than a (distro) kernel with _all_ drivers modules. That's the POINT of
modules -> be there or not, nothing else cares. If that's not the case
right now it's something we ought to fix first I'd say.

Of course there are other settings that do have impact (CPU type mostly,
maybe memory layout) but other than that... distros already ship several
binary versions (last I counted Red Hat ships 11 or so with RHL72) to
account for CPU type and amount etc.

Greetings,
Arjan van d Ven

2002-01-14 19:15:22

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> a) on some architecures modular code is slower (IIRC, the problem is
> with medium-range calls being faster than far ones and usable only in the
> kernel proper). We probaly want to leave a gap after the .text and remap

Also with TLB mapping sizes (4Mb versus 4K)

> .text of module in there - if I understand the problem that should be
> enough, but that's really a question to folks dealing with these ports (PPC64
> and Itanic?)

For x86 you just need to leave a nice chunk of physical memory that is there
to copy the module text/data into as you insmod them during boot then give
the rest of the pool back to the paging system. This also solves the problem
with "can't be a module because I need 1Mb of linear space" drivers.

Alan

2002-01-14 19:27:24

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> No, But it is -mine- ! If I can not have complete control over
> my system due to the way the Fathers of this operating system
> are taking it I will have no choice but to use another or be stuck
> at a non-maintained version . Now mind you I LIKE LINUX !

Right but you aren't going to go and use autoconfig except maybe to get
an initial Configuration to frob with ?

Alan

2002-01-14 19:02:44

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox <[email protected]>:
> The goal of a typical end user is "make it work, make it go away and do what
> it did last week". Random mechanics hating car owners don't do engine tuning
> jobs or fit turbochargers.

No...but they do change their own oil and antifreeze. Upgrading your
kernel should be as simple as changing your oil.

> Secondly we've established we can pick the right CPU for the kernel reliably
> that is seperate to modules.

Right, but that doesn't get you a recompiled binary with extended instructions
in it.

> Thirdly building a lot of stuff modular is the right choice anyway - in the
> world of hot plugging and USB Grandma is not going to want to recompile her
> kernel because she bought a new trackball to boost her quake score.

I'm not arguing with building a lot of stuff modular. The autoconfiugurator
does exactly that for hot-plug buses.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"Are we to understand," asked the judge, "that you hold your own interests
above the interests of the public?"

"I hold that such a question can never arise except in a society of cannibals."
-- Ayn Rand

2002-01-14 19:30:26

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, David Lang wrote:

> I can see a couple reasons for building a kernel without useing modules.
>
> 1. security, if you don't need any modules you can disable modules entirly
> and then it's impossible to add a module without patching the kernel first
> (the module load system calls aren't there)

Oh, please... Give me permissions sufficient to make these syscalls
and in a couple of minutes your kernel will be replaced with ELIZA.
As a bonus - ELIZA running under TOPS-10 on PDP-10 emulator. And talking
to PARRY.

Anyway, it's trivial to disable said system calls just before doing
execve("/sbin/init",...). It won't buy you any security, but if you
insist...

> 2. speed, there was a discussion a few weeks ago pointing out that there
> is some overhead for useing modules (far calls need to be used just in
> case becouse the system can't know where the module will be located IIRC)

_That_ has to be addressed - regardless of anything else, if that suckitude
can be fixed it should be.

> 3. simplicity in building kernels for other machines. with a monolithic
> kernel you have one file to move (and a bootloader to run) with modules
> you have to move quite a few more files.

FVO"quite a few" equal to 2. Kernel and initramfs.cpio.gz

2002-01-14 19:30:26

by Ben Collins

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 01:26:18PM -0500, Eric S. Raymond wrote:
> Alan Cox <[email protected]>:
> > Now to do everything you describe does not need her to configure a custom
> > kernel tree. Not one bit. You think apt or up2date build each user a custom
> > kernel tree ?
>
> Is it OK in your world that Aunt Tillie is dependent on a distro maker? Is
> it OK that she never gets to have a kernel compiled for anything above the
> least-common-denominator chip?
>
> Not that I'm running down distro makers. They do a valuable job, and in fact
> my approach relies on Aunt Tillie's machine starting life with an all-modular
> distro kernel.
>
> But the point of this game is for Aunt Tillie to have more and better
> choices. Isn't that what we're supposed to be about?

Wouldn't it be better for Aunt Tillie to continue using an all modular
kernel (from a distro) so she doesn't have to "update" (in your example,
that means recompile) a new one everytime she adds new hardware? Does
she have to connect all of here 1394 and USB devices to the computer
during this phase in order to have them all supported?

Ben

--
.----------=======-=-======-=========-----------=====------------=-=-----.
/ Ben Collins -- Debian GNU/Linux \
` [email protected] -- [email protected] -- [email protected] '
`---=========------=======-------------=-=-----=-===-======-------=--=---'

2002-01-14 19:31:52

by Eli Carter

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" wrote:
> Charles Cazabon <[email protected]>:
> > Yes, and yes. Aunt Tillie is running Linux because someone installed a
> > distribution for her.
>
> You don't know that. Maybe she installed it herself.

>From a box she bought at CompUSA (or equiv.). She is dependant upon the
distro maker for the service they provide to her... that's what she's
paying them for.

> > She is never going to need anything out of her kernel that her vendor-shipped
> > update kernels do not provide.
>
> *You can't know that.*
>
> And your belief that you *can* know it is a key part of the elitist
> developer psychology and implicit assumptions that keeps Linux mostly
> inaccessible to the Aunt Tillies of the world.

Then she needs a better distribution, or at least one that caters to her
needs.
If there isn't such a distribution for her, she is no longer "Aunt
Tillie", but has become "Aun7 71LL13" ;)

Ahem, sorry, couldn't resist. :) I'll be quiet now.

Eli
--------------------. Real Users find the one combination of bizarre
Eli Carter \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

2002-01-14 19:37:13

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 07:14:40PM +0000, Alan Cox wrote:
> > > it did last week". Random mechanics hating car owners don't do engine tuning
> > > jobs or fit turbochargers.
> >
> > No...but they do change their own oil and antifreeze. Upgrading your
> > kernel should be as simple as changing your oil.
>
> Yeah. I go to the garage versus I click "up 2 date". I don't mix custom
> oils.

And people that do change their own oil usually know something about
their cars too.

To carry this anaology out a bit further, the people who change their
own oil know about as much about their car(s) as the people who go and
recompile their own kernel, but don't claim to know much about the
kernel internals.

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

2002-01-14 18:57:05

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Hello All , And on that day linux will loose one small person .
Taf , JimL

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Alan Cox <[email protected]>:
> > For 2.5 if things go to plan there will be no such thing as a "compiled in"
> > driver. They simply are not needed with initramfs holding what were once the
> > "compiled in" modules.
>
> This is something of a bombshell. Not necessarily a bad one, but...
>
> Alan, do you have *any* *freakin'* *idea* how much more complicated
> the CML2 deduction engine had to be because the basic logical entity
> was a tristate rather than a bool? If this plan goes through, I'm
> going to be able to drop out at least 20% of the code, with most of
> that 20% being in the nasty complicated bits where the maintainability
> improvement will be greatest. And I can get rid of the nasty "vitality"
> flag, which probably the worst wart on the language.
>
> Yowza...so how soon is this supposed to happen?
> --
> <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>
>
> Government should be weak, amateurish and ridiculous. At present, it
> fulfills only a third of the role. -- Edward Abbey
>

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-14 19:41:04

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 01:54:12PM -0500, Eric S. Raymond wrote:
> Charles Cazabon <[email protected]>:
> > Yes, and yes. Aunt Tillie is running Linux because someone installed a
> > distribution for her.
>
> You don't know that. Maybe she installed it herself.

So she installed it herself, why isn't $distrovender kernel good enough?
Keep in mind that just because $kernelversion has a bug doesn't mean
that $distrovender-$kernelversion does. So yes, relying on an update
from $distrovendor is a good thing. What if your automagic tool existed
and Aunt Tillie clicked when 2.4.15 was current stable?

> > She is never going to need anything out of her kernel that her vendor-shipped
> > update kernels do not provide.
>
> *You can't know that.*

Would you accept She'll almost never need something thats not in her
current $distrovednor kernel or the one $distrovendor is working on?

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

2002-01-14 18:57:05

by Larry McVoy

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 07:38:24PM +0100, Dave Jones wrote:
> On Mon, Jan 14, 2002 at 12:52:28PM -0500, Eric S. Raymond wrote:
> > She complains of occasional lockups, and that she gets skips when
> > playing her Guy Lombardo MP3s. Melvin says, over the phone: "Yup,
> > that version had some VM problems. And you need the low-latency stuff
> > that went in three releases ago.
>
> ... and the 200 patches that the vendor added that she's become
> so used to just being there...

Yeah, what Dave said. Eric, your approach is pushing Aunt Tillie towards
more variations and what the Aunt Tillie needs is less. Ditto for the
distro vendors. They want as few as possible different kernels running
out there, the more variations there the greater the support load. It's
the *opposite* of what the Linux kernel community wants, they want the
broadest coverage of combinations they can get, that shows up more bugs.
If a distro vendor could get away with exactly one kernel config, they
would. Even if it made for a 20MB kernel image, the support tradeoff
is a win.

Supporting products for Aunt Tillie is very different than supporting
products for a bunch of hackers. One wants "it works" and the other
wants "the source".
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

2002-01-14 19:48:12

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, Alan Cox wrote:

> Date: Mon, 14 Jan 2002 19:17:46 +0000 (GMT)
> From: Alan Cox <[email protected]>
> To: [email protected]
> Cc: [email protected], [email protected]
> Subject: Re: Hardwired drivers are going away?
>
> > > Urban legend.
> > I do not agree . Got proof ? Yes that is a valid question .
>
> Most of the rootkit type stuff I see nowdays includes code for loading
> patches into module free kernels. Its a real no win. The better ones support
> regexp scanning so they can patch kernels where the sysadmin thinks he/she
> is cool and has hidden or crapped in System.map
>
> > > > case becouse the system can't know where the module will be located IIRC)
> > > I defy you to measure it on x86
> > OK ,How about sparc-64/alpha/ia64/... ?
>
> Not generally found in your grandmothers PC

doesn't matter, they are likly to be found on dedicated servers where
the flexibility of modules is not needed and the slight performance
advantage is desired.

making everything modular is fine for desktops/laptops but why should
dedicated servers pay the price?

> > > > 3. simplicity in building kernels for other machines. with a monolithic
> > > > kernel you have one file to move (and a bootloader to run) with modules
> > > > you have to move quite a few more files.
> > > tar or nfs mount; make modules_install.
> > Please my laugh'o meter is stuck already . Sorry . JimL
>
> Then fix it, because the above works well. Also remember that autoconfig
> tools won't be able to guess remote machines very well 8)

I don't want autoconfig to detect my remote machines I want to specify
them directly. (there are likly to be things on the motherboards of
machines that I don't want to compile into the kernel, sound cards for
servers are one example, video drivers are another, and USB is a third)

autoconfig is supposed to be an option, not the only way to compile a
kernel (or are you saying that you don't want to be able to use your 1.2G
athlon to build the kernel for your 486?)

David Lang

2002-01-14 19:48:13

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 01:54:12PM -0500, Eric S. Raymond wrote:

> > She is never going to need anything out of her kernel that her vendor-shipped
> > update kernels do not provide.
> *You can't know that.*
> And your belief that you *can* know it is a key part of the elitist
> developer psychology and implicit assumptions that keeps Linux mostly
> inaccessible to the Aunt Tillies of the world.

nonsense. if theres some whizzy-new-feature that Aunt Tillie really
believes she'll need, there's more chance she'll find it in a
prepackaged tested vendor kernel than stock kernel. And if not,
then there's usually a good reason for non-inclusion.

--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs

2002-01-14 19:57:14

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

note: this discussion has moved away from autoconfig to requiring
everything to be modules.

autoconfig is a useful tool in some cases, but is not going to be used
everywhere.

I view modules as being in the same catagory.

David Lang

On Mon, 14 Jan 2002, Alan Cox wrote:

> Date: Mon, 14 Jan 2002 19:17:46 +0000 (GMT)
> From: Alan Cox <[email protected]>
> To: [email protected]
> Cc: [email protected], [email protected]
> Subject: Re: Hardwired drivers are going away?
>
> > > Urban legend.
> > I do not agree . Got proof ? Yes that is a valid question .
>
> Most of the rootkit type stuff I see nowdays includes code for loading
> patches into module free kernels. Its a real no win. The better ones support
> regexp scanning so they can patch kernels where the sysadmin thinks he/she
> is cool and has hidden or crapped in System.map
>
> > > > case becouse the system can't know where the module will be located IIRC)
> > > I defy you to measure it on x86
> > OK ,How about sparc-64/alpha/ia64/... ?
>
> Not generally found in your grandmothers PC
>
> > > > 3. simplicity in building kernels for other machines. with a monolithic
> > > > kernel you have one file to move (and a bootloader to run) with modules
> > > > you have to move quite a few more files.
> > > tar or nfs mount; make modules_install.
> > Please my laugh'o meter is stuck already . Sorry . JimL
>
> Then fix it, because the above works well. Also remember that autoconfig
> tools won't be able to guess remote machines very well 8)
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2002-01-14 19:55:03

by Charles Cazabon

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Eric S. Raymond <[email protected]> wrote:
> Charles Cazabon <[email protected]>:
> > Yes, and yes. Aunt Tillie is running Linux because someone installed a
> > distribution for her.
>
> You don't know that. Maybe she installed it herself.

If she installed Linux herself, today, then she's not Aunt Tillie. She's "my
Aunt Tillie who's something of a computer hobbyist, and not afraid to install
a Unix-like OS on her PC alongside (or instead of) Windows". That's a _whole_
different ballgame -- and if she can do that, then she's fully capable of
reading a little bit and answering some yes/no questions to compile a kernel.

> > She is never going to need anything out of her kernel that her
> > vendor-shipped update kernels do not provide.
>
> *You can't know that.*

Sure I can -- the same way I know that Ma & Pa Kettle are never going to use
their computer for more than email, web surfing, and playing Solitaire.

> And your belief that you *can* know it is a key part of the elitist
> developer psychology and implicit assumptions that keeps Linux mostly
> inaccessible to the Aunt Tillies of the world.

No, my beliefs are based on twenty years of observing people using computers
and technoogy. I have yet to see a single person who is simultaneously (i)
technical enough to need something from their kernel that their distribution
vendor doesn't provide, and (ii) not technical enough to be able to compile it
themselves with today's tools.

I think your auto-configurator will be something of a white elephant --
equivalent to a better guide for setting the clock in your VCR. Those people
who can't figure out how to set their VCR clock from the current instructions
are the same people who don't care whether it flashes "12:00" or not.

Charles
--
-----------------------------------------------------------------------
Charles Cazabon <[email protected]>
GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

2002-01-14 20:07:26

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

[email protected] <[email protected]>:
> Of course there are other settings that do have impact (CPU type mostly,
> maybe memory layout) but other than that... distros already ship several
> binary versions (last I counted Red Hat ships 11 or so with RHL72) to
> account for CPU type and amount etc.

OK. Scenario #2:

Tillie's nephew Melvin is a junior-grade geek. He's working his way
through college doing website administration for small businesses. He
doesn't know C, but he can hack his way around Perl and a little PHP,
and he can type "configure; make". He's been known to wear a penguin
T-shirt.

Some time back he set up a Linux box for Joe Foonly over at Joe's
Garage. Joe calls him back and says "Hey, kid, I gotta problem here.
Lot of hits on that website and the machine's getting sluggish when
I'm doing my books with GnuCash on it at the same time. But what with
the recession and all, I don't want to go buying new hardware if I can
help it."

Melvin thinks to himself "OK, let's see if we can't tune this sucker a
bit." He runs top(1) and sees a bad shortage of free RAM; the box is
an older machine, a 586-based PCI/ISA hybrid from around 1995, and
only has 32MiB of memory in it. But Joe doesn't want to spend money
on hardware and, since money is tight all over and Joe took care of
the state inspection for Melvin's car a few weeks ago without getting
around to billing him yet, Melvin kind of needs to make the best of
the situation.

Melvin thinks this is no problem, he'll start by building a new kernel
with some stuff trimmed out to leave more RAM for userspace. But...
uh oh! He nuked that source tree because free disk was getting kind
of tight, and the .config went with it. Looks like Melvin's going to
have to reconstruct his configuration by hand.

"Crap." Melvin thinks. "I don't remember what kind of network card I
compiled in. Am I going to have to open this puppy up just to eyeball
the hardware?" Doing that would take time Melvin was planning to spend
chatting up a girl geek he's noticed over at the computer lab.

Autoconfigure saves the day. Possibly it even helps Melvin get laid.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Freedom, morality, and the human dignity of the individual consists
precisely in this; that he does good not because he is forced to do
so, but because he freely conceives it, wants it, and loves it.
-- Mikhail Bakunin

2002-01-14 20:11:32

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, David Lang wrote:

> doesn't matter, they are likly to be found on dedicated servers where
> the flexibility of modules is not needed and the slight performance
> advantage is desired.
>
> making everything modular is fine for desktops/laptops but why should
> dedicated servers pay the price?

There should be no price. And AFAICS that's doable.

> autoconfig is supposed to be an option, not the only way to compile a
> kernel (or are you saying that you don't want to be able to use your 1.2G
> athlon to build the kernel for your 486?)

autoconfig is completely separate story. There are very good reasons to
avoid built-in/modular crap, regardless of the autoconf. Every #ifdef MODULE
is a bug waiting to happen - simply because the stuff gets out of sync.
Every place that consists of
#ifdef CONFIG_FOO
init_foo();
#endif
....
is a permanent source of PITA for out-of-tree drivers/patch merging/etc.
And that's aside of the fun with ordering, etc.

If we can get rid of performance hit on modular code - we should go
for that variant simply because it reduces the amount of bug sources.

Autoconfig and other details of ESR's Aunt Tillie fetish belong to
alt.sex.* - from time to time they may be amusing, but let's not
mix them with the technical arguments, OK?

2002-01-14 20:29:34

by Charles Cazabon

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Eric S. Raymond <[email protected]> wrote:
> [email protected] <[email protected]>:
> > Of course there are other settings that do have impact (CPU type mostly,
> > maybe memory layout) but other than that... distros already ship several
> > binary versions (last I counted Red Hat ships 11 or so with RHL72) to
> > account for CPU type and amount etc.
>
> OK. Scenario #2:

Hmm. This scenario seems totally bogus.

[...]
> Some time back he set up a Linux box for Joe Foonly over at Joe's
> Garage. Joe calls him back and says "Hey, kid, I gotta problem here.
> Lot of hits on that website and the machine's getting sluggish when
> I'm doing my books with GnuCash on it at the same time.
[...]
> the box is
> an older machine, a 586-based PCI/ISA hybrid from around 1995, and
> only has 32MiB of memory in it.
[...]

Hmmm, 32MiB of RAM on a 586-class machine, and its doing useful work as both a
webserver and running GnuCash? Care to construct something more real-world?

Even at that:

> Melvin thinks this is no problem, he'll start by building a new kernel
> with some stuff trimmed out to leave more RAM for userspace. But...
> uh oh! He nuked that source tree because free disk was getting kind
> of tight, and the .config went with it. Looks like Melvin's going to
> have to reconstruct his configuration by hand.
>
> "Crap." Melvin thinks. "I don't remember what kind of network card I
> compiled in. Am I going to have to open this puppy up just to eyeball
> the hardware?"

Uh, no. Try `lsmod`.

> Autoconfigure saves the day.

Autoconfigure not necessary; read the output of lsmod, or read modules.conf.
Problem solved.

Charles
--
-----------------------------------------------------------------------
Charles Cazabon <[email protected]>
GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

2002-01-14 20:31:05

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Hello Alan ,

On Mon, 14 Jan 2002, Alan Cox wrote:
> > No, But it is -mine- ! If I can not have complete control over
> > my system due to the way the Fathers of this operating system
> > are taking it I will have no choice but to use another or be stuck
> > at a non-maintained version . Now mind you I LIKE LINUX !
> Right but you aren't going to go and use autoconfig except maybe to get
> an initial Configuration to frob with ?
Ok , Now I am confused . I guess it's not hard to do .
Is the Kernel headed toward Soft drivers or not ?

I get my initial Config (say from autoconf.) how will that keep
the Drivers in the Kernel ? Tia , JimL

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-14 20:35:13

by Justin Carlson

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

There are many times I would have found (and, I anticipate, many times I
will find) an autoconfigurator useful in my work, if only to get a
rough-cut kernel configuration done in minimal time. For my purposes,
it doesn't have to be 100% correct to save me some significant amount of
work.

Does this mean I'm incapable of configuring a kernel with the available
tools? No. And it doesn't mean I think there needs to be some grand
shift in the way distro vendors provide kernels, but this kind of a
facility would be useful for *me*. I don't want to get into whether
this is an appropriate thing to make easily accessible to good ol' Aunt
Tillie.

>From the other side, how does having the ability to probe local hardware
hurt? It should be cleanly seperable from the classical build process
for the purists, and helpful to some (I think) significant portion of
the userbase, particularly those folks who like to test bleeding edge
stuff on a variety of hardware. I don't really understand the
resistance to the idea of someone going out and implementing this.

my $.02.

-Justin

Attachments:

(No filename) (232.00 B)

2002-01-14 20:31:04

by Gerhard Mack

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Date: Mon, 14 Jan 2002 14:50:35 -0500
> From: Eric S. Raymond <[email protected]>
> To: [email protected]
> Cc: [email protected]
> Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery
> -- the elegant solution)
> "Crap." Melvin thinks. "I don't remember what kind of network card I
> compiled in. Am I going to have to open this puppy up just to eyeball
> the hardware?" Doing that would take time Melvin was planning to spend
> chatting up a girl geek he's noticed over at the computer lab.

BTDT queried the current kernel for the info I needed.

ISA doesn't look like it was designed to be autodetected at least not the
really old stuff.. if it's on a 586 it's likely to be at least PnP and
therefore more easilly detectable.

IMO ISA was designed for techies and not for J Random User.

--
Gerhard Mack

[email protected]

<>< As a computer I find your faith in technology amusing.

2002-01-14 20:38:03

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Charles Cazabon <[email protected]>:
> > "Crap." Melvin thinks. "I don't remember what kind of network card I
> > compiled in. Am I going to have to open this puppy up just to eyeball
> > the hardware?"
>
> Uh, no. Try `lsmod`.

He hard-compiled in that driver. lsmod(1) can't see it.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

No one is bound to obey an unconstitutional law and no courts are bound
to enforce it. -- 16 Am. Jur. Sec. 177 late 2d, Sec 256

2002-01-14 20:38:53

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 02:50:35PM -0500, Eric S. Raymond wrote:
> "Crap." Melvin thinks. "I don't remember what kind of network card I
> compiled in. Am I going to have to open this puppy up just to eyeball
> the hardware?" Doing that would take time Melvin was planning to spend
> chatting up a girl geek he's noticed over at the computer lab.
> Autoconfigure saves the day. Possibly it even helps Melvin get laid.

Unless of course, geekgirl realises what a dork Melvin was for
not doing something like lsmod / looking in /var/log/messages for
boot messages showing not only what card he has, but what driver
he needs.

Common sense saves the day. Possibly it even saves Melvin making an ass
of himself. 8-)

--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs

2002-01-14 20:36:42

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Alexander Viro <[email protected]>:
> But it still leaves you with tristate - instead of yes/module/no it's
> yes/yes, but don't put it on initramfs/no. However, dependencies become
> simpler - all you need is "I want this, that and that on initramfs" and
> the rest can be found by depmod (i.e. configurator doesn't have to deal
> with "FOO goes on initramfs (== old Y), so BAR and BAZ must go there
> (== can't be M)").

Actually I think we may no longer be in tristate-land. Instead, some
devices have the property "This belongs in initramfs if it's configured
at all" -- specifically, drivers for potential boot devices. Everything
else can dynamic-load after boot time.

In CML2 you can assign a symbol properties, which are written into
trailing comments in the config file on the same line as the symbol
value assignment. One such property, "PRIVATE", is already
interpreted by the postprocessor that generates autoconf.h from the
configuration output; it prevents the symbol from being written to
autoconf.h.

The critical-for-boot property could be interpreted by the same
postprocessor script and turned into a manifest for initramfs. There
would be no need for the inference engine or configurator to know
about this property at all, just as it doesn't need to know anything
about PRIVATE.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"To disarm the people... was the best and most effectual way to enslave them."
-- George Mason, speech of June 14, 1788

2002-01-14 20:44:42

by Chris Friesen

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Alexander Viro wrote:
>
> On Mon, 14 Jan 2002, David Lang wrote:
>
> > doesn't matter, they are likly to be found on dedicated servers where
> > the flexibility of modules is not needed and the slight performance
> > advantage is desired.
> >
> > making everything modular is fine for desktops/laptops but why should
> > dedicated servers pay the price?
>
> There should be no price. And AFAICS that's doable.

I haven't been following the initramfs stuff, but now I have a question.
Currently we're using initrd to store a kernel and compressed ramdisk bundled
together as an ~7MB single file that gets netbooted by firmware in a card. Will
it be possible to bundle initramfs together with the kernel into a single file
in this same manner?

Thanks,

Chris

--
Chris Friesen | MailStop: 043/33/F10
Nortel Networks | work: (613) 765-0557
3500 Carling Avenue | fax: (613) 765-2986
Nepean, ON K2H 8E9 Canada | email: [email protected]

2002-01-14 20:49:32

by Henrique de Moraes Holschuh

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 03:19:42PM -0500, Eric S. Raymond wrote:
> > > compiled in. Am I going to have to open this puppy up just to eyeball
> > > the hardware?"
> > Uh, no. Try `lsmod`.
> He hard-compiled in that driver. lsmod(1) can't see it.

$ grep eth0 /var/log/boot.msg
<6>eth0: PCnet/FAST+ 79C972 at 0x9400, 00 a0 d2 18 c0 2c

(Ok, it's not ISA, but it serves a point).

--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs

2002-01-14 20:51:14

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 14 Jan 2002, Larry McVoy wrote:
> On Mon, Jan 14, 2002 at 07:38:24PM +0100, Dave Jones wrote:
> If a distro vendor could get away with exactly one kernel config, they
> would. Even if it made for a 20MB kernel image, the support tradeoff
> is a win.

Looks like Debian's look-mama-I-can-beat-godzila-hands-down 2.4 kernel
packages with everything AND the kitchen sink... :) They eat about 20Mb of
diskspace, I think.

Package: kernel-image-2.4.16-k7
Installed-Size: 22976

--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh

2002-01-14 20:52:54

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

some systems don't want modules to be loaded after boot (yes it's security
that can be bypassed, but as long as it's not your only security it's
still worth doing)

also with hardwired drivers going away does this mean that other things
that are currently compiled in will become modules, like say the IP stack?

David Lang

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Date: Mon, 14 Jan 2002 15:17:48 -0500
> From: Eric S. Raymond <[email protected]>
> To: Alexander Viro <[email protected]>
> Cc: Alan Cox <[email protected]>,
> Mr. James W. Laferriere <[email protected]>,
> Giacomo Catenazzi <[email protected]>,
> Linux Kernel List <[email protected]>
> Subject: Re: Hardwired drivers are going away?
>
> Alexander Viro <[email protected]>:
> > But it still leaves you with tristate - instead of yes/module/no it's
> > yes/yes, but don't put it on initramfs/no. However, dependencies become
> > simpler - all you need is "I want this, that and that on initramfs" and
> > the rest can be found by depmod (i.e. configurator doesn't have to deal
> > with "FOO goes on initramfs (== old Y), so BAR and BAZ must go there
> > (== can't be M)").
>
> Actually I think we may no longer be in tristate-land. Instead, some
> devices have the property "This belongs in initramfs if it's configured
> at all" -- specifically, drivers for potential boot devices. Everything
> else can dynamic-load after boot time.
>
> In CML2 you can assign a symbol properties, which are written into
> trailing comments in the config file on the same line as the symbol
> value assignment. One such property, "PRIVATE", is already
> interpreted by the postprocessor that generates autoconf.h from the
> configuration output; it prevents the symbol from being written to
> autoconf.h.
>
> The critical-for-boot property could be interpreted by the same
> postprocessor script and turned into a manifest for initramfs. There
> would be no need for the inference engine or configurator to know
> about this property at all, just as it doesn't need to know anything
> about PRIVATE.
> --
> <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>
>
> "To disarm the people... was the best and most effectual way to enslave them."
> -- George Mason, speech of June 14, 1788
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2002-01-14 20:54:22

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Dave Jones <[email protected]>:
> On Mon, Jan 14, 2002 at 02:50:35PM -0500, Eric S. Raymond wrote:
> > "Crap." Melvin thinks. "I don't remember what kind of network card I
> > compiled in. Am I going to have to open this puppy up just to eyeball
> > the hardware?" Doing that would take time Melvin was planning to spend
> > chatting up a girl geek he's noticed over at the computer lab.
> > Autoconfigure saves the day. Possibly it even helps Melvin get laid.
>
> Unless of course, geekgirl realises what a dork Melvin was for
> not doing something like lsmod / looking in /var/log/messages for
> boot messages showing not only what card he has, but what driver
> he needs.

Right now, neither lsmod nor the boot time messages necessarily give you that
information. lsmod only works if the driver is in fact a module. My
/var/log/dmesg contains no message from the NIC on my motherboard. And
going from the driver to the config symbol isn't trivial even if you *have*
the lsmod or dmesg information.

And anyway there are settings you can't even recover by looking at the
hardware, such as whether KHTTPD or BSD process accounting were turned
on.

Sure, Melvin could remember a whole bunch of state, or a whole bunch
of rules for reconstructing it. But isn't sweating that kind of detail
exactly what *computers* are for?

Melvin, you may be sure, has more important things on his mind...
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

That the said Constitution shall never be construed to authorize
Congress to infringe the just liberty of the press or the rights of
conscience; or to prevent the people of the United states who are
peaceable citizens from keeping their own arms...
-- Samuel Adams, in "Phila. Independent Gazetteer", August 20, 1789

2002-01-14 20:56:03

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, Chris Friesen wrote:

> I haven't been following the initramfs stuff, but now I have a question.
> Currently we're using initrd to store a kernel and compressed ramdisk bundled
> together as an ~7MB single file that gets netbooted by firmware in a card. Will
> it be possible to bundle initramfs together with the kernel into a single file
> in this same manner?

<shrug> kernel unpacks to rootfs after init_vfs_caches(). It does
populate_root(archive_start, archive_size);
followed by releasing that area. That's it. If you have these variables
set - it will work, no matter how did the archive get there.

2002-01-14 20:56:22

by Brian Gerst

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" wrote:
>
> Charles Cazabon <[email protected]>:
> > > "Crap." Melvin thinks. "I don't remember what kind of network card I
> > > compiled in. Am I going to have to open this puppy up just to eyeball
> > > the hardware?"
> >
> > Uh, no. Try `lsmod`.
>
> He hard-compiled in that driver. lsmod(1) can't see it.

cat /proc/ioports. If the card is in use, it will show up there.

--

Brian Gerst

2002-01-14 21:03:52

by Hugh Dickins

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 14 Jan 2002, Eric S. Raymond wrote:
>
> Tillie's nephew Melvin is a junior-grade geek. He's working his way
...
> Some time back he set up a Linux box for Joe Foonly over at Joe's
> Garage. Joe calls him back and says "Hey, kid, I gotta problem here.
...
> Autoconfigure saves the day. Possibly it even helps Melvin get laid.
...
> so, but because he freely conceives it, wants it, and loves it.

This is terrific stuff! "The Bold & the Bazaarre"?

Hugh

2002-01-14 21:04:12

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> "Crap." Melvin thinks. "I don't remember what kind of network card I
> compiled in. Am I going to have to open this puppy up just to eyeball
> the hardware?" Doing that would take time Melvin was planning to spend

So he builds a kernel with modular setups just like the vendor kernel. The
existing module configuration and setup data that worked with the old
kernel keeps working and he has time to read up on his technique and get
laid.

I just don't buy your examples. Eric all I can think is that you have seriously
weird relatives.

Alan

2002-01-14 21:10:42

by Ian Molton

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On a sunny Mon, 14 Jan 2002 11:44:59 -0800 (PST) David Lang gathered a
sheaf of electrons and etched in their motions the following immortal
words:

> doesn't matter, they are likly to be found on dedicated servers where
> the flexibility of modules is not needed and the slight performance
> advantage is desired.

Exactly WHAT performance advantage? once the module is loaded, its loaded.
most modules use the same code to handle modular and non-modular builds
anyhow (look at the ide drivers, for example)

2002-01-14 21:10:42

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 03:38:44PM -0500, Eric S. Raymond wrote:

> Right now, neither lsmod nor the boot time messages necessarily give you that
> information.

One of the great things about Linux (or at least I think so) kernel
is it's incredibly verbose startup.
If you have a configured network, boot messages WILL tell you
what driver is controlling that card. If built as a module
lsmod WILL tell you.

> /var/log/dmesg contains no message from the NIC on my motherboard.

Then that's a driver issue. What NIC ?

> And going from the driver to the config symbol isn't trivial even
> if you *have* the lsmod or dmesg information.

Then we need better descriptions in the CML2 rules.

> And anyway there are settings you can't even recover by looking at the
> hardware, such as whether KHTTPD or BSD process accounting were turned
> on.

ls /proc/sys/net/khttpd
ls /proc/sys/kernel/acct

> Sure, Melvin could remember a whole bunch of state, or a whole bunch
> of rules for reconstructing it. But isn't sweating that kind of detail
> exactly what *computers* are for?

If Melvin really does have a mind like a sieve,he'd put .config
somewhere sensible after building a kernel.

--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs

2002-01-14 21:18:24

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> > > "Crap." Melvin thinks. "I don't remember what kind of network card I
> > > compiled in. Am I going to have to open this puppy up just to eyeball
> > > the hardware?"
> >
> > Uh, no. Try `lsmod`.
>
> He hard-compiled in that driver. lsmod(1) can't see it.

You mean he broke the carefully designed vendor set up by running a poorly
designed autoconfig script ? See if he'd run a sensible modular one he would
be fine

2002-01-14 21:35:12

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Monday 14 January 2002 01:55 pm, Charles Cazabon wrote:
> Eric S. Raymond <[email protected]> wrote:
> > Alan Cox <[email protected]>:
> > > Now to do everything you describe does not need her to configure a
> > > custom kernel tree. Not one bit. You think apt or up2date build each
> > > user a custom kernel tree ?
> >
> > Is it OK in your world that Aunt Tillie is dependent on a distro maker?
> > Is it OK that she never gets to have a kernel compiled for anything above
> > the least-common-denominator chip?
>
> Yes, and yes. Aunt Tillie is running Linux because someone installed a
> distribution for her.

Or, some glorious day in the future, it came preinstalled on her hardware.

> She is never going to need anything out of her kernel that her
> vendor-shipped update kernels do not provide.

I wouldn't go THAT far, but when she does she'll have someone else upgrade it
for her. (Just because you CAN learn to change the oil in your car doesn't
mean you want to. Can aunt tillie actually unscrew her case and insert a PCI
card? More to the point, WOULD she?)

> > But the point of this game is for Aunt Tillie to have more and better
> > choices. Isn't that what we're supposed to be about?

No. The point is to offer EVERYBODY more and better choices. Whether
they'll be any use to aunt tillie specifically is secondary.

> No. We're supposed to be about stuff that works. Vendor-shipped kernels
> work for 99.9% of people. The remaining 0.1% have no need for an
> "auto-configurator".

I like the auto-configurator. I build custom kernels for all sorts of
different machines and it can take me half an hour to walk through the menus
with a command line open in a second window doing cat /proc/pci, cat
/proc/bus/usb, and whatever replaced isapnp (cat /proc/bus/isapnp probably)
to figure out and properly select everything that's in this box.

If the autoprobe saves me that half an hour, I'm all for it. People who
don't see much use in an autoprober probably don't work with other people's
hardware very often.

It's also a nice educational tool for newbies learning the linux kernel,
which I suspect is the real reason some people actively object to it. :)

> Charles

Rob

2002-01-14 22:08:04

by Jeff Garzik

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" wrote:
> Right now, neither lsmod nor the boot time messages necessarily give you that
> information. lsmod only works if the driver is in fact a module. My
> /var/log/dmesg contains no message from the NIC on my motherboard. And
> going from the driver to the config symbol isn't trivial even if you *have*
> the lsmod or dmesg information.

For network cards one needs only to issue the ETHTOOL_GDRVINFO ioctl to
find out what hardware is associated with an ethernet interface.

Jeff

--
Jeff Garzik | Alternate titles for LOTR:
Building 1024 | Fast Times at Uruk-Hai
MandrakeSoft | The Took, the Elf, His Daughter and Her Lover
| Samwise Gamgee: International Hobbit of Mystery

2002-01-14 22:12:24

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

the impact is in all calls to the module, if they are far calls instead of
near calls each and every call is (a hair) slower.

so the code can be the same and still be slower to get to.

you can argue that it's not enough slower to matter, but even Alan admits
there is some impact.

David Lang

On Mon, 14 Jan 2002, Ian Molton wrote:

> Date: Mon, 14 Jan 2002 20:51:24 +0000
> From: Ian Molton <[email protected]>
> To: [email protected]
> Subject: Re: Hardwired drivers are going away?
>
> On a sunny Mon, 14 Jan 2002 11:44:59 -0800 (PST) David Lang gathered a
> sheaf of electrons and etched in their motions the following immortal
> words:
>
> > doesn't matter, they are likly to be found on dedicated servers where
> > the flexibility of modules is not needed and the slight performance
> > advantage is desired.
>
> Exactly WHAT performance advantage? once the module is loaded, its loaded.
> most modules use the same code to handle modular and non-modular builds
> anyhow (look at the ide drivers, for example)
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

2002-01-14 22:19:14

by Oliver Xymoron

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002, Eric S. Raymond wrote:

> Alexander Viro <[email protected]>:
> > But it still leaves you with tristate - instead of yes/module/no it's
> > yes/yes, but don't put it on initramfs/no. However, dependencies become
> > simpler - all you need is "I want this, that and that on initramfs" and
> > the rest can be found by depmod (i.e. configurator doesn't have to deal
> > with "FOO goes on initramfs (== old Y), so BAR and BAZ must go there
> > (== can't be M)").
>
> Actually I think we may no longer be in tristate-land. Instead, some
> devices have the property "This belongs in initramfs if it's configured
> at all" -- specifically, drivers for potential boot devices. Everything
> else can dynamic-load after boot time.

The knowledge of what's required to boot can and should be moved out of
the kernel to distro land. 'make install' should call the distro's install
script that sez /boot and /lib are on ext3 on /dev/sd?, ergo I need to
include the ext3 and relevant SCSI modules.

--
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."

2002-01-14 22:19:14

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 05:07:37PM -0500, Jeff Garzik wrote:
>
> For network cards one needs only to issue the ETHTOOL_GDRVINFO ioctl to
> find out what hardware is associated with an ethernet interface.

but not all the net drivers support this yet do they?

--
| Dave Jones. http://www.codemonkey.org.uk
| SuSE Labs

2002-01-14 22:20:55

by Reid Hekman

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 2002-01-14 at 14:38, Eric S. Raymond wrote:
> Right now, neither lsmod nor the boot time messages necessarily give you that
> information. lsmod only works if the driver is in fact a module. My
> /var/log/dmesg contains no message from the NIC on my motherboard. And
> going from the driver to the config symbol isn't trivial even if you *have*
> the lsmod or dmesg information.

Yes, getting the current used environment from the running kernel in a
simple way would be nice. Actually, I'm interested in trying out your
autoconfigurator, though I'll be one to go back to the config and look
it over. The Aunt Tillie scenario doesn't preclude the usefulness of an
autoconfig tool.

> Sure, Melvin could remember a whole bunch of state, or a whole bunch
> of rules for reconstructing it. But isn't sweating that kind of detail
> exactly what *computers* are for?

Precisely. The Real Problem(TM) is there are more issues than initial
kernel configuration that effect the details the user installing a
kernel or device is required to know. For a real example of this -- Grip
fails to see an audio CD in my CDRW when ide-scsi is loaded, DVD
playback is also slower but I need ide-scsi so I can backup CD's from
one drive to another. Right now, if I want to do one or another task, I
need to reboot using different kernel command lines, with a script to
alter my /dev links so the relevant apps continue to work. For another,
some of my apps need to be poked and prodded when my USB webcam hops
from /dev/video1 to /dev/video0 depending on whether bttv was or not.

Fixing those bits will allow vendors to make better and easier choices
with supplied kernels, at which point initial kernel (auto)configuration
becomes less important. I guess it's part of what Linus calls growing to
have a "middle-aged" kernel.

Anyway, I know it's not a zero sum game. We can't make Eric stop work on
one thing and make him do another. In what ways could an
autoconfigurator better work in today's environment? What could the
kernel do to make that job easier while not offending people too much?

Regards,
Reid

2002-01-14 22:21:14

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Monday 14 January 2002 02:09 pm, Alexander Viro wrote:

> But it still leaves you with tristate - instead of yes/module/no it's
> yes/yes, but don't put it on initramfs/no. However, dependencies become
> simpler - all you need is "I want this, that and that on initramfs" and
> the rest can be found by depmod (i.e. configurator doesn't have to deal
> with "FOO goes on initramfs (== old Y), so BAR and BAZ must go there
> (== can't be M)").

This is something I've wondered about and would like to ask for clarification
on: the relationship between the initramfs image and the kernel, build
process-wise.

How much of the build process for the initramfs will be integrated with the
kernel build? Since the kernel won't boot without a matching initramfs, I
take it that some kind of initramfs will be a kernel build target now?

There's been a lot of talk about having the source for a mini-libc (uclibc,
dietlibc, some combo) in the kernel tree, and other people saying we should
just grab the binary for build purposes. The most obvious model I can think
of for klibc staying seperate from the kernel is the user-space
pcmcia/cardbus hotplug stuff, but that DID get integrated into the kernel.

The klibc source/binary debate still seems to be ongoing, but apart from
that, will the build process for initramfs be part of the kernel build or not?

Rob.

2002-01-14 22:23:14

by Chris Adams

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

In article <[email protected]>, David Lang wrote:
>I can see a couple reasons for building a kernel without useing modules.
>
>1. security, if you don't need any modules you can disable modules entirly
>and then it's impossible to add a module without patching the kernel first
>(the module load system calls aren't there)

There is no security gain from disabling module loading. If someone has
the level of access needed to load modules, they also have access to
/dev/mem. Run-time patching of the kernel is something that can be done
(and probably is done by some rootkits). For bonus points, patch in the
system call(s) required by insmod and voila: module loading now works
(instead of having to patch all your rootkit code into the running
kernel, patch in insmod and let the kernel load the code for you).
--
Chris Adams <[email protected]>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

2002-01-14 22:26:14

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox <[email protected]>:
> > "Crap." Melvin thinks. "I don't remember what kind of network card I
> > compiled in. Am I going to have to open this puppy up just to eyeball
> > the hardware?" Doing that would take time Melvin was planning to spend
>
> So he builds a kernel with modular setups just like the vendor kernel. The
> existing module configuration and setup data that worked with the old
> kernel keeps working and he has time to read up on his technique and get
> laid.

Where did your assumption that Melvin has the canned config files to
do that still available come from? More fundamentally: which should
he *have* to have those configs available?

> I just don't buy your examples. Eric all I can think is that you
> have seriously weird relatives.

Correct. My relatives are *much* weirder than Melvin and Tillie :-).
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Faith may be defined briefly as an illogical belief in the occurrence
of the improbable...A man full of faith is simply one who has lost (or
never had) the capacity for clear and realistic thought. He is not a
mere ass: he is actually ill. -- H. L. Mencken

2002-01-14 22:31:24

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Monday 14 January 2002 01:54 pm, Eric S. Raymond wrote:
> Charles Cazabon <[email protected]>:
> > Yes, and yes. Aunt Tillie is running Linux because someone installed a
> > distribution for her.
>
> You don't know that. Maybe she installed it herself.
>
> > She is never going to need anything out of her kernel that her
> > vendor-shipped update kernels do not provide.
>
> *You can't know that.*
>
> And your belief that you *can* know it is a key part of the elitist
> developer psychology and implicit assumptions that keeps Linux mostly
> inaccessible to the Aunt Tillies of the world.

No need to get defensive, Eric.

If make autoprobe is to become useful to an aunt tillie, it's something the
distribution is going to have to provide a wrapper for, which strikes me as
unlikely in the short term. (QA issues: there's 8 zillion oddball hardware
combos out there, and autodetect is guaranteed to either miss something or
configure something wrong on at least some of them. It'll need rather a lot
of shakedown. More than a year in mainstream.)

As for aunt tillie moving from red hat's kernel to linus's most recent
kernel, is that advisable? Vendors provide outside patches long before linus
does (yes, Linus is more conservative than the distributions are). And
sometimes you get brown paper bag releases (2.4.11-dontuse). Even the
non-brown-paper-bag ones tend to have hardware combinations that won't build
due to easily patchable compilation errors, which aunt tillie ain't up to.

A vendor that provides faster updates than Red Hat (like Kevin's Red Hat Uber
Distribution) is a market opportunity. But a tool is not the same thing as a
service. It just makes providing the service easier.

Make autoconfigure expands the pool of people who can build kernels, but it's
not going to saturate the populace to the point where everybody immediately
SHOULD. Not even close. Over time, the pool will grow. But aiming for Aunt
Tillie in the short term is probably overreaching.

Rob

2002-01-14 22:37:34

by Greg KH

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 14, 2002 at 09:19:01AM -0500, Rob Landley wrote:
>
> The klibc source/binary debate still seems to be ongoing, but apart from
> that, will the build process for initramfs be part of the kernel build or not?

I would vote for yes. I would like this, if for no other reason than
the number of kernel builds most kernel developers do :)

I was going to wait until kbuild 2.5 went into the tree before looking
at what would be needed to add this to the build process.

thanks,

greg k-h

2002-01-14 22:37:34

by Jeff Garzik

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Dave Jones wrote:
>
> On Mon, Jan 14, 2002 at 05:07:37PM -0500, Jeff Garzik wrote:
> >
> > For network cards one needs only to issue the ETHTOOL_GDRVINFO ioctl to
> > find out what hardware is associated with an ethernet interface.
>
> but not all the net drivers support this yet do they?

correct. But it is the proper path one should take on this particular
train of thought ;-)

--
Jeff Garzik | Alternate titles for LOTR:
Building 1024 | Fast Times at Uruk-Hai
MandrakeSoft | The Took, the Elf, His Daughter and Her Lover
| Samwise Gamgee: International Hobbit of Mystery

2002-01-14 22:43:34

by Tom Gilbert

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

* Eric S. Raymond ([email protected]) wrote:
> Alan Cox <[email protected]>:
> > Now to do everything you describe does not need her to configure a custom
> > kernel tree. Not one bit. You think apt or up2date build each user a custom
> > kernel tree ?
>
> Is it OK in your world that Aunt Tillie is dependent on a distro maker? Is
> it OK that she never gets to have a kernel compiled for anything above the
> least-common-denominator chip?

$ apt-cache search kernel-image | grep 2.4.17
kernel-image-2.4.17-386 - Linux kernel image for version 2.4.17 on 386.
kernel-image-2.4.17-586tsc - Linux kernel image for version 2.4.17 on
Pentium-Classic.
kernel-image-2.4.17-686 - Linux kernel image for version 2.4.17 on
PPro/Celeron/PII/PIII.
kernel-image-2.4.17-686-smp - Linux kernel image 2.4.17 on
PPro/Celeron/PII/PIII SMP.
kernel-image-2.4.17-k6 - Linux kernel image for version 2.4.17 on AMD
K6/K6-II/K6-III
kernel-image-2.4.17-k7 - Linux kernel image for version 2.4.17 on AMD K7
pcmcia-modules-2.4.17-386 - PCMCIA Modules for Linux (kernel
2.4.17-386).

Aunt Tillie's distro autodetects her processor type and installs the
appropriate image. Aunt Tillie's distro provides precompiled,
preconfigured, modular kernels which require no configuration. Aunt
Tillie clicks on the "check for software updates" icon on her desktop
and her distro installs security fixes, new software, and a new kernel.

Aunt Tillie doesn't even have to _know_ she needs a better VM.

> But the point of this game is for Aunt Tillie to have more and better
> choices. Isn't that what we're supposed to be about?

$ apt-cache search kernel-image | wc -l
49

Tom.
--
.^. .-------------------------------------------------------.
/V\ | Tom Gilbert, London, England | http://linuxbrit.co.uk |
/( )\ | Open Source/UNIX consultant | [email protected] |
^^-^^ `-------------------------------------------------------'

2002-01-14 22:46:06

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Monday 14 January 2002 02:50 pm, Eric S. Raymond wrote:

> the hardware?" Doing that would take time Melvin was planning to spend
> chatting up a girl geek he's noticed over at the computer lab.
>
> Autoconfigure saves the day. Possibly it even helps Melvin get laid.

You had me with you right up until the end there, then suddenly my disbelief
precipitated out of solution... :)

Rob

2002-01-14 22:51:14

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Rob Landley <[email protected]>:
> Make autoconfigure expands the pool of people who can build kernels,
> but it's not going to saturate the populace to the point where
> everybody immediately SHOULD. Not even close. Over time, the pool
> will grow. But aiming for Aunt Tillie in the short term is probably
> overreaching.

No, it's not. Because the second we stop thinking about Aunt Tillie,
we start making excuses for badly-designed interfaces and excessive
complexity. We tend to fall back into insular, elitist assumptions
that limit both the useability of our software and its potential user
population. We get lazy and stop checking our assumptions. When we
do this, Bill Gates laughs at us, and is right to do so.

I've seen it happen in this thread. Many lkml people clearly have the
attitude that if you aren't willing to sweat the arcana, you
shouldn't be building kernels. Whst they don't realize is that with
that attitude, we not only lose the Aunt Tillies of the world, we
inflict a lot of unnecessary hassle on *ourselves*, too. We're
sweating details with think time we could be spending creatively.

Therefore I try to stay focused on Aunt Tillie even though I know
that you are objectively correct and her class of user is likely
not to build kernels regularly for some years yet.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

No matter how one approaches the figures, one is forced to the rather
startling conclusion that the use of firearms in crime was very much
less when there were no controls of any sort and when anyone,
convicted criminal or lunatic, could buy any type of firearm without
restriction. Half a century of strict controls on pistols has ended,
perversely, with a far greater use of this weapon in crime than ever
before.
-- Colin Greenwood, in the study "Firearms Control", 1972

2002-01-14 22:52:05

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> Where did your assumption that Melvin has the canned config files to
> do that still available come from? More fundamentally: which should
> he *have* to have those configs available?

Because the GPL says he's entitled to them ?

2002-01-14 22:57:54

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox <[email protected]>:
> > Where did your assumption that Melvin has the canned config files to
> > do that still available come from? More fundamentally: which should
> > he *have* to have those configs available?
>
> Because the GPL says he's entitled to them ?

You miss my point. Sure he's entitled to them. But why should he
*have to have them*? They're extra state which, in the presence
of a proper autoconfigurator, he doesn't need.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Militias, when properly formed, are in fact the people themselves and
include all men capable of bearing arms. [...] To preserve liberty it is
essential that the whole body of the people always possess arms and be
taught alike, especially when young, how to use them.
-- Senator Richard Henry Lee, 1788, on "militia" in the 2nd Amendment

2002-01-14 22:59:44

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Monday 14 January 2002 03:17 pm, Eric S. Raymond wrote:
> Alexander Viro <[email protected]>:
> > But it still leaves you with tristate - instead of yes/module/no it's
> > yes/yes, but don't put it on initramfs/no. However, dependencies become
> > simpler - all you need is "I want this, that and that on initramfs" and
> > the rest can be found by depmod (i.e. configurator doesn't have to deal
> > with "FOO goes on initramfs (== old Y), so BAR and BAZ must go there
> > (== can't be M)").
>
> Actually I think we may no longer be in tristate-land. Instead, some
> devices have the property "This belongs in initramfs if it's configured
> at all" -- specifically, drivers for potential boot devices. Everything
> else can dynamic-load after boot time.

No. Just about every block device there is is a POTENTIAL root device. Boot
and root devices are seperate things (just as boot and root partitions are
seperate things.) Your kernel can decompress from an enternet boot prom if
you're clever enough about. You could theoretically have your root device on
a USB storage device. Booting from CD-ROM is fairly common, yet not
something you normally want your installed kernel to have compiled in.

I've even seen kernels that load via tftp, including an initial ram disk, and
either never have local storage of any kind or use it purely as swap space.
(Why? Really really really easy system administration of multiple terminals
in libraries and kiosks and such. It breaks, you drop in a new one (don't
even have to image it). No local hard drive liminates the #1 cause of system
failure. And there's no way an end-user can screw up its configuration in a
way a reboot won't fix.)

What's interesting is what devices the root partition IS going to live on and
this has NOTHING to do with the device itself. It can be any device any
normal partition lives on. You can have IDE and SCSI in the same system.
You can have hard drives, CD-ROM drives, and floppies. The fact you can boot
off of an IDE cd-rom doesn't mean you will, or that's how you want to
configure your kernel.

When manually configuring, you select this. And if the autoprober isn't
smart enough to figure out what your root device is, it needs to be fixed.

Rob

2002-01-14 23:02:54

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 14 Jan 2002 17:34:23 -0500
"Eric S. Raymond" <[email protected]> wrote:

> Therefore I try to stay focused on Aunt Tillie even though I know
> that you are objectively correct and her class of user is likely
> not to build kernels regularly for some years yet.

Change that last line to read "her class of user will never build kernels ever,
and would be aggressively disinterested in the possibility of doing so", and
you might be closer to the truth.

Aunt Tillie just DOESN'T CARE, OK? She can talk to her vendor if she gets
worried about whether her kernel supports the Flangelistic2000 SuperDoodad.

2002-01-14 23:05:35

by Tom Gilbert

[permalink] [raw]

Subject: Re: ISA hardware discovery -- the elegant solution

* Oliver Xymoron ([email protected]) wrote:
> On Mon, 14 Jan 2002, Eric S. Raymond wrote:
>
> > Michael Lazarou (ETL) <[email protected]>:
> > > Doesn't this mean that you would need a fully functional kernel
> > > before you get to run the autoconfigurator?
> >
> > Yes, but this was always true.
>
> No it's not. You only need a kernel that can run your compiler.

It's already been pointed out that the autconfigurator requires a great
many in-kernel features and userspace utilities.

Tom.
--
.^. .-------------------------------------------------------.
/V\ | Tom Gilbert, London, England | http://linuxbrit.co.uk |
/( )\ | Open Source/UNIX consultant | [email protected] |
^^-^^ `-------------------------------------------------------'

2002-01-14 23:13:14

by Rusty Russell

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002 14:09:16 -0500 (EST)
Alexander Viro <[email protected]> wrote:
> Two technical obstacles:
> a) on some architecures modular code is slower (IIRC, the problem is
> with medium-range calls being faster than far ones and usable only in the
> kernel proper). We probaly want to leave a gap after the .text and remap
> .text of module in there - if I understand the problem that should be
> enough, but that's really a question to folks dealing with these ports (PPC64
> and Itanic?)

Sparc64 has a special allocator. PPC32 is not so lucky, and if the module
doesn't end up under 32MB we use trampolines (PLT in the ppc parlance).
PPC64 has similar issues.

> b) current differences between options parsing in case of built-in
> and modular drivers.

I have a patch for this, of course, but it's tied to my replacement module
loader:

http://www.kernel.org/pub/linux/kernel/people/rusty/patches/Module/param.patch.bz2

Cheers!
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

2002-01-14 23:13:24

by Brian Gerst

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

David Lang wrote:
>
> the impact is in all calls to the module, if they are far calls instead of
> near calls each and every call is (a hair) slower.
>
> so the code can be the same and still be slower to get to.
>
> you can argue that it's not enough slower to matter, but even Alan admits
> there is some impact.
>
> David Lang

Let's get the terminology right here (for x86 at least):
Far jump: Changes to a new code segment, absolute address
Near jump: Same code segment, 4-byte relative offset
Short jump: Same code segment, 1-byte signed offset

The kernel never uses far jumps except for some BIOS calls and during
booting. The difference betwen near and short jumps is very minute.
Short jumps are limited to +/- 128 bytes, so are really only applicable
for small loops. All function calls between object files must be near
jumps, as the assembler does not not know the distance of an unresolved
symbol and must assume the largest possible offset.

--

Brian Gerst

2002-01-14 23:17:14

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> No, it's not. Because the second we stop thinking about Aunt Tillie,
> we start making excuses for badly-designed interfaces and excessive
> complexity. We tend to fall back into insular, elitist assumptions
> that limit both the useability of our software and its potential user
> population. We get lazy and stop checking our assumptions. When we
> do this, Bill Gates laughs at us, and is right to do so.

That I doubt. Its good to see you've spent a day or two reading about
customer expectations and software relationship management, but you are
missing some key bits of UI design here

1. You don't need to ask the poor user any questions
2. You can be proactive
[Shall I update your computer ?]
3. You can be automated
[x] update at 4am every day

This is why we have things like DHCP, plug and play, EDID, and so on.
When you ask a user a question they are not sure of you make them feel
inadequate. People don't like that so they dislike the result. Phrasing the
question in child speak tends to make them feel more not less embarassed.
"Fronicate the Bazoom Hex value" people can justifiably feel isnt their
fault and just hit return. When faced with a two paragraph explanation of the
option, 3 choices and unsure of the answer - that upsets people.

Good design you simply don't notice. Nobody who ever used the gnome-lokkit
firewall configuration ever wondered why their DNS happened to still work for
example, while asking them about DNS would have confused many.

If you want Aunt Tillie to be happy your dialog should probably look like this

Update my computer
[yes] [no] [advanced] (and this one wants to go)

No hardware dialogs, no extra questions needed. Not even an "optimise my
computer to get the best performance", because that has only one answer in
reality "Duh of course, why are you asking me"

The Red Hat kernel is a modular kernel tuned to the CPU in question, and
with an initrd built for the root fs as needed. All the logic is built into
the existing GPL tools. It figures out what should be in the initrd, which
kernel cpu type to use (and that while now for install could easily be
for build).

I simply don't understand what you are trying to build and why it is hard.

Your new driver model isnt ideal too. Think bigger - unknown driver
PCMCIA [ident strings]. Fine - trigger an apt-get of dev-pcmcia-blah-blah-blah
and see if there is one yet.

Alan

2002-01-14 23:17:24

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> > Because the GPL says he's entitled to them ?
>
> You miss my point. Sure he's entitled to them. But why should he
> *have to have them*? They're extra state which, in the presence
> of a proper autoconfigurator, he doesn't need.

You have it backwards. The _autoconfigurator_ is extra state which in the
presence of the config he doesn't need

2002-01-14 23:19:56

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

my apologies for the terminology errors, I was trying to go from memory
from the thread here late last year.

David Lang

On Mon, 14 Jan 2002, Brian Gerst wrote:

> Date: Mon, 14 Jan 2002 18:12:43 -0500
> From: Brian Gerst <[email protected]>
> To: David Lang <[email protected]>
> Cc: Ian Molton <[email protected]>, [email protected]
> Subject: Re: Hardwired drivers are going away?
>
> David Lang wrote:
> >
> > the impact is in all calls to the module, if they are far calls instead of
> > near calls each and every call is (a hair) slower.
> >
> > so the code can be the same and still be slower to get to.
> >
> > you can argue that it's not enough slower to matter, but even Alan admits
> > there is some impact.
> >
> > David Lang
>
> Let's get the terminology right here (for x86 at least):
> Far jump: Changes to a new code segment, absolute address
> Near jump: Same code segment, 4-byte relative offset
> Short jump: Same code segment, 1-byte signed offset
>
> The kernel never uses far jumps except for some BIOS calls and during
> booting. The difference betwen near and short jumps is very minute.
> Short jumps are limited to +/- 128 bytes, so are really only applicable
> for small loops. All function calls between object files must be near
> jumps, as the assembler does not not know the distance of an unresolved
> symbol and must assume the largest possible offset.
>
> --
>
> Brian Gerst
>

2002-01-14 23:20:14

by Ian Molton

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On a sunny Mon, 14 Jan 2002 14:11:48 -0800 (PST) David Lang gathered a
sheaf of electrons and etched in their motions the following immortal
words:

> the impact is in all calls to the module, if they are far calls instead
> of near calls each and every call is (a hair) slower.
>
> so the code can be the same and still be slower to get to.
>
> you can argue that it's not enough slower to matter, but even Alan admits
> there is some impact.

Ok, #1 please dont send me courtesy copies without indicating so.

#2 Not all architectures have a problem with 'far' or 'near' calls, and
frankly, I'm glad the kernels design isnt being crippled just to serve the
fundamentally CRAP x86 architecture, for once.

2002-01-14 23:34:04

by Pete Zaitcev

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> 3. simplicity in building kernels for other machines. with a monolithic
> kernel you have one file to move (and a bootloader to run) with modules
> you have to move quite a few more files.

It is easy to automate. Simply build RPMs and install them.

Die-hard RPM haters may use some ad-hoc tools as well, see:
http://people.redhat.com/zaitcev/linux/linux-collect

-- Pete

2002-01-14 23:35:44

by Timothy Covell

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

OK, drop what you're holding and prepare to load up on rotten
tomatoes, but I know that there is a solution. Inasmuch as I
am a fan of Linux and BeOS, I think that the way to go is
with just about everything (sans networking perhaps) built
as a kernel module just like what is being discussed. Nothing
is simpler in BeOS than downloading a driver and plopping
it in $HOME/config/add-ons/kernel/busses/ieee1394 (most install
scripts do this for the user or provides a "drag this driver to here"
link. And with BeOS, the boot menu allows one to disable
"user add-ons" so that if there is a problem with the module, one
can disable it.

Yes, this doesn't please the server folks. For them,
I think that we should either leave the monolithic build as an
option or find a way to get rid off any penalties to using modules.

BeOS, autoconfigures and loads all drivers in the
15 seconds or so that it takes to achieve a complete hard
boot. That much said, BeOS only supports good hardware
and left ISA stuff to user to play with (via a GUI tool). I really
do not see the value of any more discussion of how to
autoconfigure ISA/MCA/EISA/VLB cards. The vast majority
of people using those systems are:

1. Still stuck in DOS World
2. Use M$ anyway
3. Would be afraid to upgrade their kernel
4. Think it's cool to have a Beowulf cluster of 386s
5. Forgot to turn off their computers before they died
6. Ancient sattelite based missile silos which we
wouldn't want to touch anyhow.....

--
[email protected].

2002-01-14 23:41:54

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Monday 14 January 2002 06:02 pm, Bruce Harada wrote:
> On Mon, 14 Jan 2002 17:34:23 -0500
>
> "Eric S. Raymond" <[email protected]> wrote:
> > Therefore I try to stay focused on Aunt Tillie even though I know
> > that you are objectively correct and her class of user is likely
> > not to build kernels regularly for some years yet.
>
> Change that last line to read "her class of user will never build kernels
> ever, and would be aggressively disinterested in the possibility of doing
> so", and you might be closer to the truth.
>
> Aunt Tillie just DOESN'T CARE, OK? She can talk to her vendor if she gets
> worried about whether her kernel supports the Flangelistic2000 SuperDoodad.

I think what Eric's REALLY going for is converting some of the Minesweeper
Certified Solitaire Experts down at the corner store (and yes there are still
corner computer stores in mini-malls around the country) over to The Penguin.
(And providing them enough coffee to sober up, and making sure that their
minimal training is slightly more than teaching to the test. Give them some
stimulus-response answers that might actually address reality in some small
way.)

Anyway, if aunt tillie calls for her neighborhood computer mechanic, he's
probably not going to be a particularly high powered geek. He may have
aspirations of geekdom, but basically we're talking glorified tech support.

These guys might build a kernel when they install her new cutting edge USB
Salad Shooter that is only supported by a kernel newer than the distribution
vendor has yet shipped. And most of them WOULD be lost without auto-probe.

The above maps even more strongly into corporate space, to the point of being
a cliche even. But I still think Aunt Tillie is a couple stepping stones
beyond a realistic next jump, and a distraction away from whether or not
auto-probe is a cool hack and useful toy. Not being useful for aunt tillie
is not the same as not being useful at all.

Rob

2002-01-15 00:08:25

by Mark Hahn

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> complexity. We tend to fall back into insular, elitist assumptions

it's elitist to claim that someone is elitist,
especially when using the royal 'we'. ;)

2002-01-15 00:40:50

by H. Peter Anvin

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Followup to: <[email protected]>
By author: Ian Molton <[email protected]>
In newsgroup: linux.dev.kernel
>
> #2 Not all architectures have a problem with 'far' or 'near' calls, and
> frankly, I'm glad the kernels design isnt being crippled just to serve the
> fundamentally CRAP x86 architecture, for once.
>

Especially since the x86 *ISN'T* one of the architectures with this
particular problem. 'far' and 'near' as used in this thread don't
refer to the x86 segmentation crud.

-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>

2002-01-15 00:47:31

by H. Peter Anvin

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Followup to: <[email protected]>
By author: Bruce Harada <[email protected]>
In newsgroup: linux.dev.kernel
>
> On Mon, 14 Jan 2002 17:34:23 -0500
> "Eric S. Raymond" <[email protected]> wrote:
>
> > Therefore I try to stay focused on Aunt Tillie even though I know
> > that you are objectively correct and her class of user is likely
> > not to build kernels regularly for some years yet.
>
> Change that last line to read "her class of user will never build kernels ever,
> and would be aggressively disinterested in the possibility of doing so", and
> you might be closer to the truth.
>
> Aunt Tillie just DOESN'T CARE, OK? She can talk to her vendor if she gets
> worried about whether her kernel supports the Flangelistic2000 SuperDoodad.
>

I would make this an even stronger statement:

We (yes, we) should make sure Aunt Tillie doesn't ever have to build a
kernel, ever. If we have designed our kernels so that:

a) A distributor needs more than a handful of kernels (UP, SMP,
SMP+PAE, perhaps CMOV or not) on their install CD, or;

b) It's not possible to add a driver without rebuilding the kernel, or;

c) It's not possible to autodetect the module set needed AT RUNTIME;

then we have screwed up. Kernel compile autoconfiguration is a red
herring in that respect; I would argue if anything it hides the real
issue. We're currently crappy at both (b) and (c) -- a monolithic
kernel does (c) a lot better, and that is quite frankly unacceptable.

-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>

2002-01-15 01:31:45

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 03:30:46PM -0500, Justin Carlson wrote:

> From the other side, how does having the ability to probe local hardware
> hurt? It should be cleanly seperable from the classical build process
> for the purists, and helpful to some (I think) significant portion of
> the userbase, particularly those folks who like to test bleeding edge
> stuff on a variety of hardware. I don't really understand the
> resistance to the idea of someone going out and implementing this.

Right, and this is 95% possible even. Doing PCI stuff is rather easy
(since we've got it all mapped out even). The problem is the 100%
point-click-run goal that Eric has.

The original sticking point was doing ISA (and other buses that are
_not_ autodetect friendly in a safe way).

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

2002-01-15 01:38:15

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox <[email protected]>:
> > You miss my point. Sure he's entitled to them. But why should he
> > *have to have them*? They're extra state which, in the presence
> > of a proper autoconfigurator, he doesn't need.
>
> You have it backwards. The _autoconfigurator_ is extra state which in the
> presence of the config he doesn't need

Oh, come on Alan. You can do better than that.

If Melvin loses the autoconfigurator, there's no state he has to reconstruct.
There will be one exactly like it in the next tarball.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

[The disarming of citizens] has a double effect, it palsies the hand
and brutalizes the mind: a habitual disuse of physical forces totally
destroys the moral [force]; and men lose at once the power of
protecting themselves, and of discerning the cause of their
oppression.
-- Joel Barlow, "Advice to the Privileged Orders", 1792-93

2002-01-15 02:23:17

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 14 Jan 2002 10:39:25 -0500
Rob Landley <[email protected]> wrote:

> On Monday 14 January 2002 06:02 pm, Bruce Harada wrote:
> > On Mon, 14 Jan 2002 17:34:23 -0500
> >
> > Aunt Tillie just DOESN'T CARE, OK? She can talk to her vendor if she gets
> > worried about whether her kernel supports the Flangelistic2000
SuperDoodad.
>
> I think what Eric's REALLY going for is converting some of the Minesweeper
> Certified Solitaire Experts down at the corner store (and yes there are still
> corner computer stores in mini-malls around the country) over to The Penguin.

Er... if Eric were REALLY going for that, why didn't he use it as an example?
That might actually be a possible real-world situation, whereas all the ones
I've seen so far are so far removed from reality as to be pointless.

Let me put it this way: Requiring Aunt Tillie to configure/compile her kernel
is a *failure* on the part of the vendor. It doesn't matter whether Aunt
Tillie is really Aunt Tillie or your local MSCE. They should not have to do it.

As for adding a driver that's not included in the vendor's kernel, do you
mean that having a Microsoft-trained drone rebuild a kernel specifically for a
certain PC (thus requiring further compilation for any hardware added later)
and including a no-doubt beta driver and then giving it to Aunt Tillie without
any testing beyond the MCSE's PC is a *good* idea?

(I've trimmed the cc line a bit, BTW.)

2002-01-15 02:24:10

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox <[email protected]>:
> I simply don't understand what you are trying to build and why it is hard.

Don't understand it? Download it, follow the directions, and see.

It's not that hard. Given Giacomo's table of probes it only took me about
two days to get it 85% of the way there. The remaining 15% is partly
issues with rulebase bugs that it exposes, and partly issues with what
to do about the possibility of ISA hardware that is not directly probeable.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

[W]hat country can preserve its liberties, if its rulers are not
warned from time to time that [the] people preserve the spirit of
resistance? Let them take arms...The tree of liberty must be
refreshed from time to time, with the blood of patriots and tyrants.
-- Thomas Jefferson, letter to Col. William S. Smith, 1787

2002-01-15 02:59:10

by peter

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

I don't mind how people do it, but there are currently two significant
penalties associated with modules on x86. I think other architectures
have analogues.

1) The main kernel is contiguous in physical memory and is mapped with
large (4 MB) pages. This reduces pressure on the TLB. Modules are
loaded in vmalloc memory, which uses small pages, and therefore
competes for TLB space. This is a performance penalty, especially
as most current machines have undersized TLBs already. (A 64-entry
TLB with 4K pages maps 256K at a time. On-chip L2 caches are this
large or larger. Thus, as a crude approximation, every L2 miss also
causes a TLB miss.)

Also, vmalloc space is limited, and contributes to the "why isn't
the kernel seeing all of my 1 GB?" question.

2) Space for module code is allocated in page units. Thus, each module
wastes an average of 2K. If I'm going to have dozens of modules
loaded, small machines are going to notice.

On a related subject, I'd like a kernel image to consist of a series
of concatenated "chunks". The first one is the kernel proper, and then
come various modules and initrd image(s). (There may be an "EOF" chunk,
or just the absence of a magic number.)

All the boot loader has to do is copy the image into contiguous memory
and jump to it. The start of the image (ideally, it'd be PIC) then
relocates itself to wherever it wants and starts assembling the pieces.

Whether this is done by a sort of linker before the kernel boots, or
if the chunks are all ramdisk components and form a userland that the
kernel boots to which assemble the rest of the kernel is not particularly
important. I just want the unnecessary bits to get thrown away once
we've booted. (Kernel bloat should NOT be designed in.)

That would make assembling a custom boot image simple enough for a fancy
boot loader (or fancy tftp server) to do.

2002-01-15 09:21:43

by Sean Hunter

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, Jan 14, 2002 at 05:34:23PM -0500, Eric S. Raymond wrote:
> Because the second we stop thinking about Aunt Tillie,
> we start making excuses for badly-designed interfaces and excessive
> complexity.

Bollocks. The second we (including you) stop thinking about the _user_ of the
technology, we make bad decisions. This is not the same thing.

We don't expect Aunt Tillie to write kernel drivers for her knitting machine.
She (and we) expect(s) someone else to do that for her.

The Aunt Tillies of this world don't install of update Windows (or Mac O/S) for
themselves except perhaps via "Windows Update" or "Apple Update", which (guess
what) supplies a prebuilt binary and DOESN'T BUILD THEM A KERNEL.

Besides any other factor, the download/install/reboot time is less than the
download-full-tarball/untar/configure/compile/install/reboot cycle.

Sean

2002-01-15 10:05:59

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" <[email protected]> said:
> Eli Carter <[email protected]>:
> > Could you maybe describe the problem you are trying to solve a bit more
> > clearly than "the hardware-discovery problem for ISA devices"? If you
> > are trying to discover the ISA devices, but rely upon them having
> > already been discovered, what are you accomplishing?

> Sure. Let's say Aunt Tillie needs a kernel update.

So Aunt Tillie goes and clicks on "Update distribution", which gets her Red
Hat's up2date or Debian's apt-get or whatever. A reboot later she is happy.

All the nonsense about "faster kernel for K6" and "Nephew Melvin" is just
nonsense. Please do remember the horrors of some of the "latest stable"
kernels, and think thrice before you inflict same on poor Aunt Tillie.

If she wants a kernel for her machine "latest revision", give her a
.configure that builds a fully modular kernel distribution style.

Sure, it would be _way_ cool to have your autodetection stuff. Just:

- ISA is impossible to do right. And on the way out, so irrelevant.
- PCI is rather easy.
- PCMCIA, USB, ... are on the rise. How is your autoconfigurator to detect
the USB Zip drive Nephew Melvin carries around and uses to backup Aunties
account from time to time? Or the PCMCIA network card Aunt Tillie is
keeping in the top drawer for a rainy day? Or the CD burner she plans to
buy tomorrow?

What would she gain? A smaller kernel (mostly irrelevant, Aunt Tillie sure
hasn't got an 8MB RAM machine), less modules on disk (what, you are
worrying about a few MB on a multi-GB disk?)

As I said before, you are trying to solve a non-problem.

In any case, it's your time. Good luck!
--
Horst von Brand http://counter.li.org # 22616

2002-01-15 10:46:13

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> On Mon, 14 Jan 2002, Alan Cox wrote:

> > > 1. security, if you don't need any modules you can disable modules
> > > entirly and then it's impossible to add a module without patching
> > > the kernel first (the module load system calls aren't there)

> > Urban legend.

> If this is the case then why do I get systemcall undefined error messages
> when I make a mistake and attempt to load a module on a kernel without
> modules enabled?

AFAIU the security improvement of no-modules are way overrated.

[...]

> > > 3. simplicity in building kernels for other machines. with a monolithic
> > > kernel you have one file to move (and a bootloader to run) with modules
> > > you have to move quite a few more files.

> > tar or nfs mount; make modules_install.

> not on my firewalls thank you.

You keep compilers and stuff on your firewalls then?!

I get them over via ssh or by floppy/CD. Same idea.
--
Horst von Brand http://counter.li.org # 22616

2002-01-15 10:57:24

by crispin

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Tue, Jan 15, 2002 at 11:45:39AM +0100, Horst von Brand wrote:
> > On Mon, 14 Jan 2002, Alan Cox wrote:
>
> > > > 1. security, if you don't need any modules you can disable modules
> > > > entirly and then it's impossible to add a module without patching
> > > > the kernel first (the module load system calls aren't there)
>
> > > Urban legend.
>
> > If this is the case then why do I get systemcall undefined error messages
> > when I make a mistake and attempt to load a module on a kernel without
> > modules enabled?
>
> AFAIU the security improvement of no-modules are way overrated.

insmod knark.o

Crispin

2002-01-15 11:00:55

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" <[email protected]> said:
> Charles Cazabon <[email protected]>:
> > Yes, and yes. Aunt Tillie is running Linux because someone installed a
> > distribution for her.

> You don't know that. Maybe she installed it herself.

Linux-from-scratch or some such? In which case she'd better be able to
configure a custom kernel without help...

> > She is never going to need anything out of her kernel that her
> > vendor-shipped update kernels do not provide.

> *You can't know that.*

Right. But 99.9% of Aunt Tillies won't need anything else. To have
_everybody_ go through a lot of pain for the sake of 0.02% of Linux users
is silly. Better let Nephew Mervin keep the junkheap, and give Tilly a new
machine for her birthday.

> And your belief that you *can* know it is a key part of the elitist
> developer psychology and implicit assumptions that keeps Linux mostly
> inaccessible to the Aunt Tillies of the world.

Have you ever tried f.ex. Red Hat's installer and updater for the latest
versions (Just because it is the one I know best; other distribuctions have
similar facilities)? Have you tried a machine with Linux preinstalled?

Have you ever battled with Windows "autoconfiguration"?! _It doesn't
work_. At all. _Ever_. And that hasn't been an impediment to Aunt Tilly to
get a WinPC...

I just think that your idea is cool. You are just wasting effort on trying
to solve a non-problem, when there are lots of problems that could use your
talents.

But again, it's your own time.
--
Horst von Brand http://counter.li.org # 22616

2002-01-15 11:35:38

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 14 Jan 2002 09:19:01 -0500,
Rob Landley <[email protected]> wrote:
>How much of the build process for the initramfs will be integrated with the
>kernel build? Since the kernel won't boot without a matching initramfs, I
>take it that some kind of initramfs will be a kernel build target now?
>
>There's been a lot of talk about having the source for a mini-libc (uclibc,
>dietlibc, some combo) in the kernel tree, and other people saying we should
>just grab the binary for build purposes. The most obvious model I can think
>of for klibc staying seperate from the kernel is the user-space
>pcmcia/cardbus hotplug stuff, but that DID get integrated into the kernel.
>
>The klibc source/binary debate still seems to be ongoing, but apart from
>that, will the build process for initramfs be part of the kernel build or not?

I am in two minds about this (but I am at one with my duality). Part
of me says that users will want to tweak the initramfs process and they
may want to use different versions of klibc, so klibc and building
initramfs should be outside the kernel build. Another part says that
users have to build initramfs before doing the install so initramfs and
klibc should be part of kbuild.

It will probably end up with klibc as part of the kernel tarball
supported by kbuild. Generating initramfs will be done by a script,
kbuild will supply a sample but users can copy and change the sample to
suit their own requirements.

2002-01-15 11:40:48

by David Woodhouse

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

[email protected] said:
> He hard-compiled in that driver. lsmod(1) can't see it.

man dmesg.

Others have asserted that this kind of autoconfigure facility for
non-technical people isn't necessary.

I assert that it is actually harmful, and will make their life more
difficult.

My father's computer runs Linux. He doesn't need to recompile his kernels
or do any maintenance - I don't even trust him to run up2date for himself.
It's all he can manage to dial up and look at a web page so I can grab his
current IP address out of my logs, log in and do the rest.

If I get a bug report from him, it's not particularly coherent or useful.
Yet because he has a kernel binary which is identical to the one used by
many other more technical users out there, I can often match what he
complains about with the more useful bugreports in Bugzilla.

If he (or even I) compiled a custom kernel for him rather than using the
distro one, I wouldn't have a whelk's chance in a supernova of working out
WTF he was on about.

--
dwmw2

2002-01-15 11:54:18

by T. A.

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

----- Original Message -----
From: "Eric S. Raymond" <[email protected]>
To: <[email protected]>
Cc: <[email protected]>
Sent: Monday, January 14, 2002 2:50 PM
Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery --
the elegant solution)

> [email protected] <[email protected]>:
> > Of course there are other settings that do have impact (CPU type mostly,
> > maybe memory layout) but other than that... distros already ship several
> > binary versions (last I counted Red Hat ships 11 or so with RHL72) to
> > account for CPU type and amount etc.
>
> OK. Scenario #2:
>
> Tillie's nephew Melvin is a junior-grade geek. He's working his way
> through college doing website administration for small businesses. He
> doesn't know C, but he can hack his way around Perl and a little PHP,
> and he can type "configure; make". He's been known to wear a penguin
> T-shirt.
>>> SNIP <<<
> Autoconfigure saves the day. Possibly it even helps Melvin get laid.
> --
> <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>
>

I see!!! I see the light now!!! I see it all now!!! I have been
saved!!! If your super doper kernel autoconfigurator goes into the kernel
source tree then all of geekdom will have a possibility of getting laid.
Marvelous. Absolutely marvelous. I always knew coding would be able to
overcome any kind of obstacle for us, even this one. Horrah. 8-D

8-D Of course wantabe geek (if there is such a thing) Melvin could
always do it the hard way. The experience might come in handy for the next
"internet" bubble to arise. With the knowledge he gains he can make enough
paper money to take care of the getting laid problem once and for all (or at
least until the next crash). Besides his current target seams way too easy
if he has a chance at her just because he entered "configure". 8-D

Well anyway seams CML2 is becoming a literal everything and the kitchen
sink. Wow. 8-D So far my new kernel is going to require Python to
configure it, will include an super doper hardware detector so that it can
customize my spiffy new kernel specifically to my hardware, will include
special kernel access for regular users to be able to read my hardware
configuration because its so evil to configure my kernel as root even though
I will have to be root to install it and probably be root to untar/ungzip it
into /usr/src and not to mention removing the old /usr/src/linux, did I read
new driver model for autoconfiguring kernel somewhere too?, and the kernel
will even included a spiffy X Windows, KDE, or gnome (hey why skimp now)
program that installs itself as an icon on the user desktop (no matter what
the distribution, linux-kernel will support all) which the user can then
click on to fire up the kernel self-configuration, self-compiler,
self-installer, and auto boot loader configurator so that Aunt Tilly or the
slutty girl El Geeko has his eyes on can see a "Click here to launch new
kernel" button on their screen after their first reboot, oh and of course we
can't forget that the kernel must know that its either succeeded or failed
so that it could tell the user that and either restore the previous boot
configuration or install itself as the user's new default kernel thus
voiding the plebeian's support contract. But no problem, I'm sure the new
kernel can take care of that too. Just give it time. 8-D

Hilarities aside, I wouldn't mind having a program to automatically
configure a kernel for a system via Windows-like hardware autoconfiguration.
However do have a couple of quick points.

1. Don't see any reason for the kernel hardware autoconfigurator to be
included in the kernel.

2. Don't see any reason the kernel hardware autoconfigurator cannot be
run as root. Actually see one very good reason why it shouldn't be able to
be run as a regular user. Probing certain hardware is inherently dangerous.
Machine can hang. Hardware could be probed to death. Heck a clever coder
could even make use of the user level access required to allow user hardware
autoconfiguration. Wiping disks, destroying flash roms, finding system
backdoors, etc, etc.

3. ISA is pretty much dead outside of certain standard PC equipment.
And of the remaining ISA out there, most in any machine than can still run a
Linux kernel effectively is most likely PNP ISA. Plus there are a few
fairly common ISA cards that can also be found easily. It seams that the
vast majority of Aunt Tillies will be served with just PCI autoconfiguration
and maybe PNP ISA configuration.

2002-01-15 12:16:21

by Andi Kleen

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

[email protected] writes:

> 1) The main kernel is contiguous in physical memory and is mapped with
> large (4 MB) pages. This reduces pressure on the TLB. Modules are
> loaded in vmalloc memory, which uses small pages, and therefore
> competes for TLB space. This is a performance penalty, especially
> as most current machines have undersized TLBs already. (A 64-entry
> TLB with 4K pages maps 256K at a time. On-chip L2 caches are this
> large or larger. Thus, as a crude approximation, every L2 miss also
> causes a TLB miss.)

-aa tries to load modules into the linear mapping when possible.
That usually works when you load the modules early when the memory
isn't that fragmented yet.

I agree on that trying to put everything into modules isn't a good idea,
especially because of your second point.

-Andi

2002-01-15 12:34:01

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> 1) The main kernel is contiguous in physical memory and is mapped with
> large (4 MB) pages. This reduces pressure on the TLB. Modules are
> 2) Space for module code is allocated in page units. Thus, each module
> wastes an average of 2K. If I'm going to have dozens of modules
> loaded, small machines are going to notice.

If at boot time we keep a big chunk of ram free at the kernel end and just
load modules one after each other into that space until we get into real
paging that problem goes away

2002-01-15 12:31:41

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Mon, 14 Jan 2002, Alan Cox wrote:

> > > Because the GPL says he's entitled to them ?
> >
> > You miss my point. Sure he's entitled to them. But why should he
> > *have to have them*? They're extra state which, in the presence
> > of a proper autoconfigurator, he doesn't need.
>
> You have it backwards. The _autoconfigurator_ is extra state which in the
> presence of the config he doesn't need
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

Alan, Eric (and others, too), please.
Of course the autoconfigurator is an useful piece of software.
And of course Eric is posting to the wrong list here. Kernel developers
don't need any autoconfigurator at all (yes, it's just "extra state").

Eric, Aunt Tillie doesn't need any custom-made, untested, probably not
working kernel. QA comes at a price. The lastest VM fix may take a while
to reach mainstream kernels. That's life.

Face it, you're under the wrong assumption that Alan's 2.2 or Marcelo's 2.4
vanilla kernels are "better" than their patched counterparts shipped by
distro vendors. This is far from the truth[*]. Only very recent 2.2 kernels
can be installed on a Red Hat 6.x with little pain. Call it Red Hat's fault,
or mine, for choosing RH, (it's not Alan's of course), but I wanted
RAID 0.90 from the beginning.
Maybe today you can upgrade most 2.4-based distros to the latest Marcelo's,
but I bet it's going to change as soon as vendors start patching their own
kernels with random backports from 2.5.

No matter how you put it, 99% of the times your autoconfigurator will
produce a previuosly untested configuration. We can discuss about release
policies forever, but Marcelo isn't expected to replicate all the QA job
vendors do before releasing a kernel. Now, the kernel is modular
enough not to turn the issue into a nightmare for the average developer.
Most of the time you DO keep your old .config around. And you know your HW.
And the vanilla kernel you've just compiled happens to work. But you'd
better do some testing before putting it on any production box.
And of course, I consider Aunt Tillie's PC definitely "production".

In the end, I think you're just pushing the right piece of software
on the wrong list. IMHO, endusers compiling the kernel it's not an l-k
issue, it's a distro one.

[*] That is, from Aunt Tillie's standpoint.

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]

2002-01-15 12:41:11

by David Miller

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

From: Alan Cox <[email protected]>
Date: Tue, 15 Jan 2002 12:44:38 +0000 (GMT)

If at boot time we keep a big chunk of ram free at the kernel end and just
load modules one after each other into that space until we get into real
paging that problem goes away

And we do have module_map/module_unmap interfaces already so it's
easy to toy with this.

I've always been meaning to do a "alloc 4MB page at boot, lock into
TLB, carve module pages out of that, vmalloc when that runs out" on
sparc64.

2002-01-15 12:46:21

by Andrew Pimlott

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, Jan 15, 2002 at 08:02:18AM +0900, Bruce Harada wrote:
> On Mon, 14 Jan 2002 17:34:23 -0500
> "Eric S. Raymond" <[email protected]> wrote:
>
> > Therefore I try to stay focused on Aunt Tillie even though I know
> > that you are objectively correct and her class of user is likely
> > not to build kernels regularly for some years yet.
>
> Change that last line to read "her class of user will never build
> kernels ever, and would be aggressively disinterested in the
> possibility of doing so", and you might be closer to the truth.
>
> Aunt Tillie just DOESN'T CARE, OK? She can talk to her vendor if she gets
> worried about whether her kernel supports the Flangelistic2000 SuperDoodad.

Ok, Grandpa Willie only cares about support for his doodad. Why do
you conclude that he should never build a kernel?

It's just as easy in principle to write a friendly front-end that
downloads sources and compiles them, as one that downloads binaries.
The obstacle is reliability, because there are more things that can
go wrong. But imagine for a moment that this is overcome. What
benefits might accrue from Willie compiling his own kernels (even if
he doesn't realize it)?

- It's easier for third-parties to provide kernel software in source
form than in binary form (because binaries must be in the correct
package format, and be compiled with the right config options, and
adhere to the particular distribution's conventionts; whereas
source is relatively neutral). Why should Willie be limited to
getting his kernels from his vendor? What if his vendor doesn't
support the Flangelistic2000 SuperDoodad yet, but there's a solid
driver available from a volunteer? What if he hears the hype
(sorry) about the low latency patch, and decides he wants to try
it (maybe his MP3's skip when Netscape thrashes)? Why take the
choice out of Willie's hands? And why keep a willing tester and a
developer apart? (If you claim that novice users don't want to
install random beta software--that contradicts my experience with
lots of Windows users!)

- It's a system that experts are likely to use as well, because
there's a lot of overlap between this system and what experts
want. A nice front-end to browse and manage kernel versions,
patches, and drivers; to download, configure, compile and install
them? I might use that. Such a system helps more people, and
thus attracts more developers. It's more likely to become common
infrastructure, instead of a distribution-specific one-off.

- It makes it easier for Willie's hacker grandson to help him.
Hackers know all about compiling kernels, but aren't as likely to
be familiar with the distribution's binary packaging. The more we
all do things the same way, the more we can help each other; when
different groups use different tools, the community is fragmented.

- It can support a graceful transition from beginner to expert.
Suppose one day, for whatever reason, Willie really does need to
change a compile-time option. Or, heaven forbid, he gets curious
about what his computer is doing when the status line says
"compiling". He's already got all the pieces he needs. Ideally,
he just needs to click on that scary "Advanced options" button.

- Building from source is good karma.

You might think these are trifles and < 1% cases. My intuition
tells me that they add up in the long run. At least it's worth
considering.

Andrew

2002-01-15 13:03:23

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

T. A. wrote:

> 1. Don't see any reason for the kernel hardware autoconfigurator to be
> included in the kernel.

Linus and Alan last year (end Dec 2000), told that it would nice to have
some kind of autoconfiguration.
The problem was the bug report about non running kernel, because of
false configuration (CPU configuration).
How many people try new kernel with the wrong CPU configuration?
(and mornally user know the name of own CPU, with netcards this is
more difficult).

[ We don't reduce trafic on lkml, because the fewer bug reports
are hidden by this huge flamewar]

>
> 2. Don't see any reason the kernel hardware autoconfigurator cannot be
> run as root. Actually see one very good reason why it shouldn't be able to
> be run as a regular user. Probing certain hardware is inherently dangerous.
> Machine can hang. Hardware could be probed to death. Heck a clever coder
> could even make use of the user level access required to allow user hardware
> autoconfiguration. Wiping disks, destroying flash roms, finding system
> backdoors, etc, etc.

Root will help, but AFAIK, not needed. Forget DMI.
Out detection is 'hang' free. (so a soft detection, but with some
tests, I can say that with this soft detection we can detect nearly
all, without the difficult to write hard probes).

>
> 3. ISA is pretty much dead outside of certain standard PC equipment.
> And of the remaining ISA out there, most in any machine than can still run a
> Linux kernel effectively is most likely PNP ISA. Plus there are a few
> fairly common ISA cards that can also be found easily. It seams that the
> vast majority of Aunt Tillies will be served with just PCI autoconfiguration
> and maybe PNP ISA configuration.

PCI, USB and ISAPNP detection works well.
ISA is a further step.
I will send Eric the new detections and database for new probes (for ISA
and others) drivers. So I hope also the ISA thread will end.

giacomo

2002-01-15 13:46:28

by Alan Cox

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, Jan 15, 2002 at 02:00:38PM +0100, Giacomo Catenazzi wrote:
> How many people try new kernel with the wrong CPU configuration?
> (and mornally user know the name of own CPU, with netcards this is
> more difficult).

All of us get the CPU wrong. By using modules however I don't have to guess
the PCI devices. My system already did that. I just need the configurator
to hit M a lot and to work out which root devices are for the initrd.

The code for that exists

> PCI, USB and ISAPNP detection works well.
> ISA is a further step.
> I will send Eric the new detections and database for new probes (for ISA
> and others) drivers. So I hope also the ISA thread will end.

I suspect ISA is a dead loss - but again build all the modules, the user
system already has the right ones to load configured.

Alan

2002-01-15 13:53:59

by David Woodhouse

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

[email protected] said:
> If we have designed our kernels so that:
<...>
> b) It's not possible to add a driver without rebuilding the kernel,
> or;
<...>
> then we have screwed up.

Oops. In that case, we screwed up (130 - delta) times.

find /usr/src/linux/ -name \*.[ch] | xargs egrep \#if.*CONFIG_.*_MODULE | cut -f2- -d: | sort | uniq | wc -l

--
dwmw2

2002-01-15 14:05:23

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alan Cox wrote:

> All of us get the CPU wrong. By using modules however I don't have to guess
> the PCI devices. My system already did that. I just need the configurator
> to hit M a lot and to work out which root devices are for the initrd.
>
> The code for that exists

It is easier to get autoconfigure in linux sources, than
modify the default (and broken) configuration from Linus.
(Sorry Linus :-) )

giacomo

2002-01-15 14:25:42

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

one other thing re: simplicity.

bootloaders

I know that many of you who are advocating this new approach despise lilo,
but for many people it does the job just fine. with the new approach when
switching between kernels you now not only need to switch the kernel
images, but also the initrd (or equivalent) images.

also don't forget that the current way of storing modules can have
problems when some kernel comile options change (SMP for example, but IIRC
there are others) if you make modules mandatory you will have to fix this
first.

David Lang

>
> > > > 3. simplicity in building kernels for other machines. with a monolithic
> > > > kernel you have one file to move (and a bootloader to run) with modules
> > > > you have to move quite a few more files.
>

2002-01-15 14:29:03

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

This topic is well past the point of being done to death - it's not so much
whipping a dead horse as trying to ride the gravestone, but ah well...

On Tue, 15 Jan 2002 07:29:58 -0500
Andrew Pimlott <[email protected]> wrote:

> On Tue, Jan 15, 2002 at 08:02:18AM +0900, Bruce Harada wrote:
> > On Mon, 14 Jan 2002 17:34:23 -0500
> > "Eric S. Raymond" <[email protected]> wrote:
> >
> > Aunt Tillie just DOESN'T CARE, OK? She can talk to her vendor if she gets
> > worried about whether her kernel supports the Flangelistic2000 SuperDoodad.
>
> Ok, Grandpa Willie only cares about support for his doodad. Why do
> you conclude that he should never build a kernel?

Because it adds extra complexity for very little gain and forces the vendor to
support 10,000 variations on a kernel that would normally only have a dozen or
so at most?

> It's just as easy in principle to write a friendly front-end that
> downloads sources and compiles them, as one that downloads binaries.

We're not talking about a front-end for kernel compilation - the topic is an
autoconfigurator that tries to decide what hardware is available, no matter
what might *need* to be available tomorrow, or next week, or next month.

> - It's easier for third-parties to provide kernel software in source
> form than in binary form (because binaries must be in the correct
> package format, and be compiled with the right config options, and
> adhere to the particular distribution's conventionts; whereas
> source is relatively neutral).

Except when it requires another source package to compile, which requires
another, and another... and God forbid that some patches conflict. Compiling
from source is a *bad* idea for someone who wants things to just work.

> Why should Willie be limited to
> getting his kernels from his vendor?

If he wants ongoing support from his vendor, he damn well better.

> What if his vendor doesn't
> support the Flangelistic2000 SuperDoodad yet, but there's a solid
> driver available from a volunteer?

"Solid" as in "will not eat his hard drive and spit it out as metal shavings
when used in combination with whatever other patches have been applied to the
kernel by the vendor"? Or "solid" as in "works for me, should work OK for you,
and if it doesn't there's always the next version, and hey, don't forget to
submit a bug report"?

> What if he hears the hype
> (sorry) about the low latency patch, and decides he wants to try
> it (maybe his MP3's skip when Netscape thrashes)?

I would say the chance of that level of user hearing about the low-latency
patches is about the same as me becoming an astronaut and flying to the Moon.
And if he does hear about them, the correct place for him to ask about them is
his vendor.

> Why take the
> choice out of Willie's hands?

Choice to do what? Break his box beyond hope of repair (by himself, at least)
and lose him any chance of support from the people he paid to do so?

> And why keep a willing tester and a
> developer apart? (If you claim that novice users don't want to
> install random beta software--that contradicts my experience with
> lots of Windows users!)

...and how many of those Windows users submitted meaningful bug reports? Or,
indeed, *any* bug reports?

> - It's a system that experts are likely to use as well, because
> there's a lot of overlap between this system and what experts
> want.

Er... from the response on the list so far, I somehow doubt that...

> A nice front-end to browse and manage kernel versions,
> patches, and drivers; to download, configure, compile and install
> them? I might use that.

Again, I believe the topic of this thread was an autoconfiguration system,
not a version management utility.

> - It makes it easier for Willie's hacker grandson to help him.
> Hackers know all about compiling kernels, but aren't as likely to
> be familiar with the distribution's binary packaging.

Really?? There aren't that many - RPM, deb, tgz. Anyone who's that familiar
with compiling kernels is rather likely to be able to handle 'man rpm'.

> The more we
> all do things the same way, the more we can help each other; when
> different groups use different tools, the community is fragmented.

Er, so you're one of those "let's all run Red Hat because it'll make things
simpler" people are you? Having a variety of tools that do similar things is
*good*.

> - It can support a graceful transition from beginner to expert.
> Suppose one day, for whatever reason, Willie really does need to
> change a compile-time option. Or, heaven forbid, he gets curious
> about what his computer is doing when the status line says
> "compiling". He's already got all the pieces he needs. Ideally,
> he just needs to click on that scary "Advanced options" button.

Who's to say that gcc is even installed on his box? Or Python? Or the userland
utilities required for him to actually make use of that compiletime option?
It's an awfully big jump from "I just want to balance my checkbook" to "let's
see, how do I activate and configure a new QoS algorithm". That's the VENDOR's
job, not Willie's or Penelope's or whoever the random user of the week may be.
If they want to do that, they are by definition no longer an average user.

> You might think these are trifles and < 1% cases. My intuition
> tells me that they add up in the long run. At least it's worth
> considering.

What is? A patch and version management utility? Sure - but it doesn't belong
in the kernel tree. An autoconfigurator? Not as it's been described so far by
Eric, and even if his idea wasn't broken it still shouldn't be in the kernel
tree.

Bruce

2002-01-15 15:35:29

by Oliver Xymoron

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Tue, 15 Jan 2002, Keith Owens wrote:

> On Mon, 14 Jan 2002 09:19:01 -0500,
> Rob Landley <[email protected]> wrote:
> >How much of the build process for the initramfs will be integrated with the
> >kernel build? Since the kernel won't boot without a matching initramfs, I
> >take it that some kind of initramfs will be a kernel build target now?
> >
> >There's been a lot of talk about having the source for a mini-libc (uclibc,
> >dietlibc, some combo) in the kernel tree, and other people saying we should
> >just grab the binary for build purposes. The most obvious model I can think
> >of for klibc staying seperate from the kernel is the user-space
> >pcmcia/cardbus hotplug stuff, but that DID get integrated into the kernel.
> >
> >The klibc source/binary debate still seems to be ongoing, but apart from
> >that, will the build process for initramfs be part of the kernel build or not?
>
> I am in two minds about this (but I am at one with my duality). Part
> of me says that users will want to tweak the initramfs process and they
> may want to use different versions of klibc, so klibc and building
> initramfs should be outside the kernel build. Another part says that
> users have to build initramfs before doing the install so initramfs and
> klibc should be part of kbuild.
>
> It will probably end up with klibc as part of the kernel tarball
> supported by kbuild. Generating initramfs will be done by a script,
> kbuild will supply a sample but users can copy and change the sample to
> suit their own requirements.

Better yet, we should do away with all install targets beyond assembling
the bootable images and provide sample install scripts in /scripts. Most
of it is already at odds with the distro's install and the remainder
becomes a maintenance issue with initramfs. Distro folks are going to be
working on initramfs-libc for their boot disks anyway and will undoubted
want to tweak such stuff greatly. Let's include Viro's
minimal-userspace-boot.c in scripts along with a
build-minimal-initramfs.sh and leave it at that.

--
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."

2002-01-15 17:05:24

by Tom Duffy

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 2002-01-15 at 04:29, Andrew Pimlott wrote:

> - Building from source is good karma.
>
> You might think these are trifles and < 1% cases. My intuition
> tells me that they add up in the long run. At least it's worth
> considering.

- Someday, a stupid government or court decides that there is a strict
separation between source and binary. Source is protected speech, but
binaries are not. Linux decides it wants a really fast DVD decryption
in the kernel, so it adds it in drivers. But now, distro's cannot
compile and distribute a binary kernel package and the end user will
need to compile the source code in order to watch their DVD.

Why is it unrealistic for everybody to compile their kernel when they do
an install? If it is rather automated, then it just becomes another
step on the progress bar.

-tduffy

2002-01-15 17:24:37

by Aaron Lehmann

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 14, 2002 at 10:50:20AM -0800, David Lang wrote:
> I can see a couple reasons for building a kernel without useing modules.

I agree with all of yours. IMHO, the proposed scheme for 2.5 is plain
bad. It will require an initrd (initramfs, or whatever), and force
kernel installation to be more difficult. The performance overhead
sounds like a major downside, epsecially when currently people know
what hardware they have and build things into the kernel accordingly.

Between this and CML2, my mental image of 2.5+ is starting to look
very grim.

2002-01-15 17:27:57

by Aaron Lehmann

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 14, 2002 at 07:17:46PM +0000, Alan Cox wrote:
> Not generally found in your grandmothers PC

What kind of an argument is this?!? Linux developers used to care
about portability to and performance on non-x86 platforms.

2002-01-15 17:47:27

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Tue, 15 Jan 2002, Aaron Lehmann wrote:

> On Mon, Jan 14, 2002 at 07:17:46PM +0000, Alan Cox wrote:
> > Not generally found in your grandmothers PC
>
> What kind of an argument is this?!? Linux developers used to care
> about portability to and performance on non-x86 platforms.

<rant>

Guys, could you lay off whatever the fuck you are smoking? And
I mean everybody involved.

* autoconfig is part of ESR's wet dream. Ignore him and he'll go away.
* Aunt Tillie and her magical mistery whorehouse <<--->>
* performance problems with modules are real, need to be solved and
can be solved.
* disabling module loading doesn't buy any security.
* it's easy to make irrevertible disabling modules _after_ boot.
* initramfs is independent from modules
* it is _not_ initrd or an analog
* it doesn't involve a shitload of extra files
* it removes tons of crap from kernel
* it allows to simplify build process and get rid of quite a few sources
of bugs, _if_ performance problems with modules are solved

Now could we fscking drop these threads? Or at least take care to separate
myths from reality before posting more crap.

</rant>

2002-01-15 18:18:07

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

CML2 is a tradeoff in the kernel config process. it (currently) trades
time for correctness. It has no effect after you have built the kernel.

I like this becouse there have been several times where I have had to go
on a scavanger hunt to figure out what option I need to turn on before I
can turn on another option (something that CML2 will fix)

as for the kbuild 2.5 performance 'penalty', is it really slower then
always doign make dep; make clean; make bzImage? Yes I know I don't always
have to do that, but I don't understand the rules as to when I do and when
I don't (they sound simple at first but there keep being exceptions
raised). having the new kernel build process always get this right is
valuble.

also the time penalties associated with these two options is something
that can (and will) be dealt with over time as they get optimized and
rewritten to be faster.

these are VERY different from the modules thing which is permanently
introducing additional penalties into the system that can affect it for
as long as it's running.

David Lang

On Tue, 15 Jan 2002, Aaron Lehmann wrote:

> On Mon, Jan 14, 2002 at 10:50:20AM -0800, David Lang wrote:
> > I can see a couple reasons for building a kernel without useing modules.
>
> I agree with all of yours. IMHO, the proposed scheme for 2.5 is plain
> bad. It will require an initrd (initramfs, or whatever), and force
> kernel installation to be more difficult. The performance overhead
> sounds like a major downside, epsecially when currently people know
> what hardware they have and build things into the kernel accordingly.
>
> Between this and CML2, my mental image of 2.5+ is starting to look
> very grim.
>

2002-01-15 18:20:07

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On 15 Jan 2002, Thomas Duffy wrote:

> On Tue, 2002-01-15 at 04:29, Andrew Pimlott wrote:
>
> > - Building from source is good karma.
> >
> > You might think these are trifles and < 1% cases. My intuition
> > tells me that they add up in the long run. At least it's worth
> > considering.
>
> - Someday, a stupid government or court decides that there is a strict
> separation between source and binary. Source is protected speech, but
> binaries are not. Linux decides it wants a really fast DVD decryption
> in the kernel, so it adds it in drivers. But now, distro's cannot
> compile and distribute a binary kernel package and the end user will
> need to compile the source code in order to watch their DVD.
>
> Why is it unrealistic for everybody to compile their kernel when they do
> an install? If it is rather automated, then it just becomes another
> step on the progress bar.

Every distro supplies a package with the source used to build their own
kernel. Just recomplile it. You can do it today. Yes, it takes longer
than building an autoconfigured kernel, since you're compiling a lot
of unused stuff. Yet the autoconfigurator belongs to the 'Kernel compiling
sybsystem' of that distribution. Don't forget that vanilla kernels can
even be incompatible with the one provided by the distro maker.

Doing it at install time is somewhat unrelated (it's even more distro-
dependant).

>
> -tduffy
>

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]

2002-01-15 18:52:57

by Richard B. Johnson

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 15 Jan 2002, Marco Colombo wrote:

> On 15 Jan 2002, Thomas Duffy wrote:
>
> > On Tue, 2002-01-15 at 04:29, Andrew Pimlott wrote:
> >
> > > - Building from source is good karma.

[SNIPPED...]

>
> Every distro supplies a package with the source used to build their own
> kernel. Just recomplile it.

Really??? Have you ever tried this? RedHat provides a directory
of random patches that won't patch regardless of the order in
which you attempt patches (based upon date-stamps on patches or
date-stamps on files). It's like somebody just copied in some
junk, thinking nobody would ever bother.

Some distributions don't even provide source. They provide
copies of /usr/src/linux/include/asm and /usr/src/linux/include/linux
but nothing else. You have to "find" source on the internet.

Some distributions don't even provide that, instead they provide
copies of /usr/src/linux/include/linux and /usr/src/linux/include/asm
under /usr/include.

The "good-ol-days" where you could get 72 floppies from Yggdrasil,
install Linux, and spend the next 48 hours watching it compile
are long gone.

I have never found a distribution that uses modules, in which is
was even remotely possible to duplicate the kernel supplied.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).

I was going to compile a list of innovations that could be
attributed to Microsoft. Once I realized that Ctrl-Alt-Del
was handled in the BIOS, I found that there aren't any.

2002-01-15 19:11:27

by Manuel McLure

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Red Hat provides a source RPM for their kernel - install the source RPM and
use rpm to build it. If you want to figure out what order the patches were
applied in, just look at the RPM spec file. I myself have applied patches to
a Red Hat kernel RPM in this fashion.

--
Manuel A. McLure KE6TAW | ...for in Ulthar, according to an ancient
<[email protected]> | and significant law, no man may kill a cat.
<http://www.mclure.org> | -- H.P. Lovecraft

----- Original Message -----
From: "Richard B. Johnson" <[email protected]>
To: "Marco Colombo" <[email protected]>
Cc: "Thomas Duffy" <[email protected]>; "Linux Mailing List"
<[email protected]>
Sent: Tuesday, January 15, 2002 10:52 AM
Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery --
the elegant solution)

> On Tue, 15 Jan 2002, Marco Colombo wrote:
>
> > On 15 Jan 2002, Thomas Duffy wrote:
> >
> > > On Tue, 2002-01-15 at 04:29, Andrew Pimlott wrote:
> > >
> > > > - Building from source is good karma.
>
> [SNIPPED...]
>
> >
> > Every distro supplies a package with the source used to build their own
> > kernel. Just recomplile it.
>
> Really??? Have you ever tried this? RedHat provides a directory
> of random patches that won't patch regardless of the order in
> which you attempt patches (based upon date-stamps on patches or
> date-stamps on files). It's like somebody just copied in some
> junk, thinking nobody would ever bother.
>
> Some distributions don't even provide source. They provide
> copies of /usr/src/linux/include/asm and /usr/src/linux/include/linux
> but nothing else. You have to "find" source on the internet.
>
> Some distributions don't even provide that, instead they provide
> copies of /usr/src/linux/include/linux and /usr/src/linux/include/asm
> under /usr/include.
>
> The "good-ol-days" where you could get 72 floppies from Yggdrasil,
> install Linux, and spend the next 48 hours watching it compile
> are long gone.
>
> I have never found a distribution that uses modules, in which is
> was even remotely possible to duplicate the kernel supplied.
>
> Cheers,
> Dick Johnson
>
> Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).
>
> I was going to compile a list of innovations that could be
> attributed to Microsoft. Once I realized that Ctrl-Alt-Del
> was handled in the BIOS, I found that there aren't any.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>

2002-01-15 19:25:59

by Aaron Lehmann

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Tue, Jan 15, 2002 at 10:17:29AM -0800, David Lang wrote:
> CML2 is a tradeoff in the kernel config process. it (currently) trades
> time for correctness. It has no effect after you have built the kernel.

My gripe is more that I find it ugly. I'm very used to the menuconfig
appearance, and I do not appreciate the interface change.

And (we've been through this before) I am *NOT* going to install yet
another language runtime just to configure a kernel.

2002-01-15 19:28:49

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 15 Jan 2002, Richard B. Johnson wrote:

> On Tue, 15 Jan 2002, Marco Colombo wrote:
>
> > On 15 Jan 2002, Thomas Duffy wrote:
> >
> > > On Tue, 2002-01-15 at 04:29, Andrew Pimlott wrote:
> > >
> > > > - Building from source is good karma.
>
> [SNIPPED...]
>
> >
> > Every distro supplies a package with the source used to build their own
> > kernel. Just recomplile it.
>
> Really??? Have you ever tried this? RedHat provides a directory
> of random patches that won't patch regardless of the order in
> which you attempt patches (based upon date-stamps on patches or
> date-stamps on files). It's like somebody just copied in some
> junk, thinking nobody would ever bother.

Uh?

# cd /usr/src/linux-2.4
# make xconfig
(note that the autoconfigurator would provide a good starting point
here for a stripped down kernel, reducing compile time a lot!)
# make bzImage
# make modules
# make install
# make modules_install

# ls -al /boot/vmlinuz-2.4.9-13custom
/boot/vmlinuz-2.4.9-13custom

I've done that (with minimal variations) hundreds of times. It worked like
that since 4.1, IIRC. What Red Hat are you talking about? And it's no
different from what you do with a standard tree.

(you need the kernel-source RPMS, of course)

You encounter troubles when you apply some random patch to the RH tree.
It happens with almost every non vanilla tree out there...

> Some distributions don't even provide source. They provide
> copies of /usr/src/linux/include/asm and /usr/src/linux/include/linux
> but nothing else. You have to "find" source on the internet.

Name one, please. I can't believe it.

> Some distributions don't even provide that, instead they provide
> copies of /usr/src/linux/include/linux and /usr/src/linux/include/asm
> under /usr/include.

Name one, please. Really.

> The "good-ol-days" where you could get 72 floppies from Yggdrasil,
> install Linux, and spend the next 48 hours watching it compile
> are long gone.

Uh. SLS was the one. Less than 50 floppies. And it never took
48 to compile a kernel. Before 1.0 the kernel was so small than my
386/40, equipped with 4MB of RAM, managed to complile it in reasonable
time (<30mins) (not with X running). 0.96 (or was it 0.98) ate my disk
so many times during compile (HD Timeout...) that I must confess that
I've been a 386BSDer and a NetBSDer at times... I can't remeber when
the problem has been fixed, BTW. Maybe when I switched to Slackware.

Kernel compile time has been nearly constant in time (since I was
upgrading my HW meanwhile), in my experience.

> I have never found a distribution that uses modules, in which is
> was even remotely possible to duplicate the kernel supplied.

s|make xconfig|cp configs/kernel-2.4.9-athlon.config .config|

in the above. Works like a charm (ok, the kernel is named -custom, but
I like it this way).

Anyway, rpm --rebuild kernel-xxxx.src.rpm does work, even if it's
obviously slow (I've never tried in on 7.x, though. I'm reminiscent of
6.x days), since it produces all the binary packages...

Go get a better internet connection: the one you're using is corrupting
the packages you're downloading. Or it's corrupting the messages you're
sending.

>
> Cheers,
> Dick Johnson
>
> Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).
>
> I was going to compile a list of innovations that could be
> attributed to Microsoft. Once I realized that Ctrl-Alt-Del
> was handled in the BIOS, I found that there aren't any.
>
>
>

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]

2002-01-15 19:37:10

by Alan

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

> On Mon, Jan 14, 2002 at 07:17:46PM +0000, Alan Cox wrote:
> > Not generally found in your grandmothers PC
>
> What kind of an argument is this?!? Linux developers used to care
> about portability to and performance on non-x86 platforms.

Read the entire thread (I know thats like asking you to read war and peace)
but if you've not read it in context you will misunderstand arguments (as
you do above).

2002-01-15 19:44:20

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> Really??? Have you ever tried this? RedHat provides a directory
> of random patches that won't patch regardless of the order in
> which you attempt patches (based upon date-stamps on patches or
> date-stamps on files). It's like somebody just copied in some
> junk, thinking nobody would ever bother.

They apply nicely and the spec file defines which to apply and when. The
srpm and rpm are generated together.

I was wondering who would actually need Aunt Tillie's autoconfigurator, now
I know

2002-01-15 20:14:00

by Richard B. Johnson

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 15 Jan 2002, Marco Colombo wrote:

> On Tue, 15 Jan 2002, Richard B. Johnson wrote:
>
> > On Tue, 15 Jan 2002, Marco Colombo wrote:
> >
> > > On 15 Jan 2002, Thomas Duffy wrote:
> > >
> > > > On Tue, 2002-01-15 at 04:29, Andrew Pimlott wrote:
> > > >
> > > > > - Building from source is good karma.
> >
> > [SNIPPED...]
> >
> > >
> > > Every distro supplies a package with the source used to build their own
> > > kernel. Just recomplile it.
> >
> > Really??? Have you ever tried this? RedHat provides a directory
> > of random patches that won't patch regardless of the order in
> > which you attempt patches (based upon date-stamps on patches or
> > date-stamps on files). It's like somebody just copied in some
> > junk, thinking nobody would ever bother.
>
> Uh?
>
> # cd /usr/src/linux-2.4
> # make xconfig

[NO, No, NO....]

I'm not talking about making a kernel that will `work` on your
machine. I'm talking about making __the__ kernel that they supplied
with all its modules, etc.

RedHat 7 is a prime example. I put it on a box in the other room.
/usr/src didn't contain ANYTHING after an installation.
However, /usr/include/asm and /usr/include/linux existed, not
as symlinks, but as files that would-have-existed within a
kernel distribution.

So... I did RPM install for the kernel after I found what was
alleged to have been the kernel. Now I had a /usr/src/linux/..., but
of course not /usr/src/linux-2.2.16-22, the binary kernel supplied.
The stuff in /usr/include was not fixed or changed to sym-links and it was
incompatible with what existed in the kernel. These were 2.2 files
with so much incompatible stuff; a 447,099 byte diff if you are truly
interested.

The usr/src/linux/.config was the .config obtained off from Linus`
tree, not something provided by RedHat so `make oldconfig` would have
made a "standard kernel" like you download from ftp.kernel.org.

Now, looking in /usr/src/redhat/../.., I find some patches that are
impossible to use to patch the kernel to bring it up (or down) to
the configuration used to build the distribution. The default
configuration, before I "installed" the kernel sources was some
empty directories of /usr/src/redhat/BUILD, /usr/src/redhat/RPMS,
/usr/src/redhat/SOURCES, /usr/src/redhat/SPECS, and /usr/src/redhat/SRPMS.
Now there were some patches and other files with no scripts and no
way to actually use them to modify the kernel. I spent hours, putting
them in order, based upon the time/date stamp within the files, not
the file time which was something more or less random. I made a script
and tried, over a period of weeks, to patch the supplied kernel with
the supplied patches. Forget it. If anything in this universe is truly
impossible, then making a Red Hat distribution kernel from the provided
tools, patches, and sources is a definitive example.

Then, to add insult to injury, the 'C' compiler provided would
not create a bootable kernel. It was egcs-2.91.66. To make
a bootable kernel, I had to install gcc-2.96. The list goes on.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).

I was going to compile a list of innovations that could be
attributed to Microsoft. Once I realized that Ctrl-Alt-Del
was handled in the BIOS, I found that there aren't any.

2002-01-15 20:26:42

by Manuel McLure

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

----- Original Message -----
From: "Richard B. Johnson" <[email protected]>
To: "Marco Colombo" <[email protected]>
Cc: "Linux Mailing List" <[email protected]>
Sent: Tuesday, January 15, 2002 12:13 PM
Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery --
the elegant solution)

[SNIP]
> The usr/src/linux/.config was the .config obtained off from Linus`
> tree, not something provided by RedHat so `make oldconfig` would have
> made a "standard kernel" like you download from ftp.kernel.org.
>
> Now, looking in /usr/src/redhat/../.., I find some patches that are
> impossible to use to patch the kernel to bring it up (or down) to
> the configuration used to build the distribution. The default
> configuration, before I "installed" the kernel sources was some
> empty directories of /usr/src/redhat/BUILD, /usr/src/redhat/RPMS,
> /usr/src/redhat/SOURCES, /usr/src/redhat/SPECS, and /usr/src/redhat/SRPMS.
> Now there were some patches and other files with no scripts and no
> way to actually use them to modify the kernel. I spent hours, putting
> them in order, based upon the time/date stamp within the files, not
> the file time which was something more or less random. I made a script
> and tried, over a period of weeks, to patch the supplied kernel with
> the supplied patches. Forget it. If anything in this universe is truly
> impossible, then making a Red Hat distribution kernel from the provided
> tools, patches, and sources is a definitive example.
>
> Then, to add insult to injury, the 'C' compiler provided would
> not create a bootable kernel. It was egcs-2.91.66. To make
> a bootable kernel, I had to install gcc-2.96. The list goes on.

Use the RPM - all of the instructions on what patches go in what order are
in the spec file. Or, you can simply copy the appropriate configuration file
from /usr/src/redhat/SOURCES to /usr/src/linux/.config, and do a "make
oldconfig; make dep; make clean; make install; make modules; make
modules_install". Voila, new kernel. Don't tell me this doesn't work because
I've done it myself.

--
Manuel A. McLure KE6TAW | ...for in Ulthar, according to an ancient
<[email protected]> | and significant law, no man may kill a cat.
<http://www.mclure.org> | -- H.P. Lovecraft

2002-01-15 20:42:40

by Arjan van de Ven

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

In article <[email protected]> you wrote:

Ok I shouldn't but...

> RedHat 7 is a prime example. I put it on a box in the other room.
> /usr/src didn't contain ANYTHING after an installation.
> However, /usr/include/asm and /usr/include/linux existed, not
> as symlinks, but as files that would-have-existed within a
> kernel distribution.

Exactly and 100% correct. Those are GLIBC headers and have NOTHING to do
with the kerenl.

> So... I did RPM install for the kernel after I found what was

Well you should have installed the kernel-source rpm if you wanted the full
expanded source... we make that one for a reason you know...

> alleged to have been the kernel. Now I had a /usr/src/linux/..., but

> The usr/src/linux/.config was the .config obtained off from Linus`
> tree, not something provided by RedHat so `make oldconfig` would have
> made a "standard kernel" like you download from ftp.kernel.org.

Ehm yes it does if you use the kernel-source RPM, also we ship about a dozen
different configs (differing in cpu type and up/smp mostly), ALL those
.config files are neatly available in the configs/ subdirectory.

> Now, looking in /usr/src/redhat/../.., I find some patches that are
> impossible to use to patch the kernel to bring it up (or down) to
> the configuration used to build the distribution. The default
> configuration, before I "installed" the kernel sources was some
> empty directories of /usr/src/redhat/BUILD, /usr/src/redhat/RPMS,
> /usr/src/redhat/SOURCES, /usr/src/redhat/SPECS, and /usr/src/redhat/SRPMS.
> Now there were some patches and other files with no scripts

Ehm the .spec file is the script!
rpm -bp kernel-2.2.spec to "run" it to create a source, or rpm -bb to make
rpm -i'able rpm binaries files from it.

> and no
> way to actually use them to modify the kernel. I spent hours, putting
> them in order, based upon the time/date stamp within the files, not
> the file time which was something more or less random. I made a script
> and tried, over a period of weeks, to patch the supplied kernel with
> the supplied patches. Forget it. If anything in this universe is truly
> impossible, then making a Red Hat distribution kernel from the provided
> tools, patches, and sources is a definitive example.

Ok now you offend me. I spent quite a bit of time making the .spec file easy
to read, AND we provide a convenient kernel-source rpm which installs
/usr/src/linux (for RHL7.0) or /usr/src/linux-2.4 for 2.4 kernels (7.1/7.2)
which contains the full source AND all configs we used. AND if you type
"make oldconfig" it picks the one you are currently running. Heck I even put
a (ok partial) description of each patch (in addition to the brief
description in the spec file) for the 7.1 kernel on
http://people.redhat.com/arjanv/patches.html for people who were interested
in why a patch existed.

Now what more would you want ?

Greetings,
Arjan van de Ven
Red Hat Linux kernel maintainer

2002-01-15 21:00:30

by David Woodhouse

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

[email protected] said:
> > Really??? Have you ever tried this? RedHat provides a directory
> > of random patches that won't patch regardless of the order in
> > which you attempt patches (based upon date-stamps on patches or
> > date-stamps on files). It's like somebody just copied in some
> > junk, thinking nobody would ever bother.

> They apply nicely and the spec file defines which to apply and when.
> The srpm and rpm are generated together.

And just in case you're too incompetent or lazy to manage this, there's also
a kernel-source package which contains the resulting source tree, with all
the patches already applied to it.

--
dwmw2

2002-01-15 20:56:40

by Richard B. Johnson

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 15 Jan 2002 [email protected] wrote:

> In article <[email protected]> you wrote:
>
> Ok I shouldn't but...
>
> > RedHat 7 is a prime example. I put it on a box in the other room.
> > /usr/src didn't contain ANYTHING after an installation.
> > However, /usr/include/asm and /usr/include/linux existed, not
> > as symlinks, but as files that would-have-existed within a
> > kernel distribution.
>
> Exactly and 100% correct. Those are GLIBC headers and have NOTHING to do
> with the kerenl.
>
> > So... I did RPM install for the kernel after I found what was
>
> Well you should have installed the kernel-source rpm if you wanted the full
> expanded source... we make that one for a reason you know...
>
> > alleged to have been the kernel. Now I had a /usr/src/linux/..., but
>
> > The usr/src/linux/.config was the .config obtained off from Linus`
> > tree, not something provided by RedHat so `make oldconfig` would have
> > made a "standard kernel" like you download from ftp.kernel.org.
>
> Ehm yes it does if you use the kernel-source RPM, also we ship about a dozen
> different configs (differing in cpu type and up/smp mostly), ALL those
> .config files are neatly available in the configs/ subdirectory.
>
> > Now, looking in /usr/src/redhat/../.., I find some patches that are
> > impossible to use to patch the kernel to bring it up (or down) to
> > the configuration used to build the distribution. The default
> > configuration, before I "installed" the kernel sources was some
> > empty directories of /usr/src/redhat/BUILD, /usr/src/redhat/RPMS,
> > /usr/src/redhat/SOURCES, /usr/src/redhat/SPECS, and /usr/src/redhat/SRPMS.
> > Now there were some patches and other files with no scripts
>
> Ehm the .spec file is the script!
> rpm -bp kernel-2.2.spec to "run" it to create a source, or rpm -bb to make
> rpm -i'able rpm binaries files from it.
>
> > and no
> > way to actually use them to modify the kernel. I spent hours, putting
> > them in order, based upon the time/date stamp within the files, not
> > the file time which was something more or less random. I made a script
> > and tried, over a period of weeks, to patch the supplied kernel with
> > the supplied patches. Forget it. If anything in this universe is truly
> > impossible, then making a Red Hat distribution kernel from the provided
> > tools, patches, and sources is a definitive example.
>
> Ok now you offend me. I spent quite a bit of time making the .spec file easy
> to read, AND we provide a convenient kernel-source rpm which installs
> /usr/src/linux (for RHL7.0) or /usr/src/linux-2.4 for 2.4 kernels (7.1/7.2)
> which contains the full source AND all configs we used. AND if you type
> "make oldconfig" it picks the one you are currently running. Heck I even put
> a (ok partial) description of each patch (in addition to the brief
> description in the spec file) for the 7.1 kernel on
> http://people.redhat.com/arjanv/patches.html for people who were interested
> in why a patch existed.
>
> Now what more would you want ?
>
> Greetings,
> Arjan van de Ven
> Red Hat Linux kernel maintainer
>
>

Well you SHOULD and you did. I want to thank you for your definitive
explaination of how to make a kernel that is an exact functional
duplicate of the distributed kernel. Hopefully this will work, and I
will, in fact, make it worth your company's time by getting the
"latest-and-greatest" Red Hat distribution and use your Email as
a reference.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).

I was going to compile a list of innovations that could be
attributed to Microsoft. Once I realized that Ctrl-Alt-Del
was handled in the BIOS, I found that there aren't any.

2002-01-15 21:17:41

by Alan

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

> However, /usr/include/asm and /usr/include/linux existed, not
> as symlinks, but as files that would-have-existed within a
> kernel distribution.

This is correct behaviour, please read the glibc installation documentation

Alan

2002-01-15 21:27:40

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 15 Jan 2002 [email protected] wrote:

[...]
> > and no
> > way to actually use them to modify the kernel. I spent hours, putting
> > them in order, based upon the time/date stamp within the files, not
> > the file time which was something more or less random. I made a script
> > and tried, over a period of weeks, to patch the supplied kernel with
> > the supplied patches. Forget it. If anything in this universe is truly
> > impossible, then making a Red Hat distribution kernel from the provided
> > tools, patches, and sources is a definitive example.
>
> Ok now you offend me. I spent quite a bit of time making the .spec file easy
> to read, AND we provide a convenient kernel-source rpm which installs
> /usr/src/linux (for RHL7.0) or /usr/src/linux-2.4 for 2.4 kernels (7.1/7.2)
> which contains the full source AND all configs we used. AND if you type
> "make oldconfig" it picks the one you are currently running. Heck I even put
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
that one I didn't know, I've always done a manual cp from configs/*.
Thanks for the hint. B-) (And for the good job)

> Greetings,
> Arjan van de Ven
> Red Hat Linux kernel maintainer
>

.TM.
--
____/ ____/ /
/ / / Marco Colombo
___/ ___ / / Technical Manager
/ / / ESI s.r.l.
_____/ _____/ _/ [email protected]

2002-01-15 21:29:30

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Monday 14 January 2002 09:22 pm, Bruce Harada wrote:

> As for adding a driver that's not included in the vendor's kernel, do you
> mean that having a Microsoft-trained drone rebuild a kernel specifically
> for a certain PC (thus requiring further compilation for any hardware added
> later) and including a no-doubt beta driver and then giving it to Aunt
> Tillie without any testing beyond the MCSE's PC is a *good* idea?

No, it would be tested on Aunt Tillie's PC, which she brought in for
upgrading (she's not going to unscrew the case and put a new card in herself,
is she), and which the drone has never seen before and didn't necessarily
sell her in the first place. (Maybe if Aunt Tillie's lucky the store did,
but this is the new guy and aunt tillie's had the PC for two years, they
don't sell that model anymore...)

I don't think having a microsoft-trained drone TOUCH a computer is a good
idea. But SAIR and Linux+ are going to turn out their own drones with a
"USDA approved" stamp on their forehead who are NOT hackers, and who intend
to make a living off of glorified tech support. It happened in the mainframe
world, it happened in the minicomputer world, it happened under DOS and
Windows, and it'll happen under Linux. And nine to five workers are not
actually a BAD thing. Very few actual hackers WANT to babysit database
schema for Fortune 500 corporations...

The "I summon the vast power of certification" crowd will always be with us.
Would you rather that the hundreds of thousands of people who get a four year
degree in computing every year (because they think there's money in it, not
because they actually LIKE computers) work on windows boxes, or on Linux?
(Hint: as long as they work on windows boxes, they and their bosses are
paying money to microsoft, and they are enabling windows-only shops where
employees send *.doc files to everybody they know, and advancing the dot-net
flag.)

You can't have total world domination without bringing along the drones.
Hoping for a world where there ARE no drones is a utopian view. Hoping for a
PLATFORM where there are no drones is easily achievable, it's called a "tiny
niche market".

The drones -ARE- the service industry through which you get aunt tillie. I
just think Eric's skipping a step...

Rob

2002-01-15 21:31:25

by kaih

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

[email protected] (Aaron Lehmann) wrote on 15.01.02 in <[email protected]>:

> I agree with all of yours. IMHO, the proposed scheme for 2.5 is plain
> bad. It will require an initrd (initramfs, or whatever), and force
> kernel installation to be more difficult. The performance overhead
> sounds like a major downside, epsecially when currently people know
> what hardware they have and build things into the kernel accordingly.

Installation more difficult? Significant performance overhead?

What *are* you smoking?

MfG Kai

2002-01-15 21:39:10

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

--On Monday, 14 January, 2002 10:53 AM -0800 Larry McVoy <[email protected]>
wrote:

> Eric, your approach is pushing Aunt Tillie towards
> more variations and what the Aunt Tillie needs is less. Ditto for the
> distro vendors.

Not entirely.

If Aunt Tillie has (say) a laptop, I think she is likely to find that no
distribution kernel actually supports all features (sound, APM, etc.)
if the laptop is even moderately new. This from experience with
Redhat & Debian (perhaps the others are miles better). So she
does indeed have a reasonable need to compile a kernel.

However, Eric's approach (dmesg) is still flawed as normally
the way these distros fail is either (a) hanging on boot, or
(b) failing to detect the relevant hardware. Needless to say,
neither failure mode is going to give much use to a configurator
tool which looks at dmesg.

Eric: I think you'd be far better off trying to identify the
machine (and hence get a working .config) rather than the
hardware.

Example: put in some wget based thingy, which goes to some (fixed) web
site, searches for (some extracted or Tillie composed string) which
describes the hardware (bound to have been bought as-is and never opened),
pulls down a set of config files and heuristics to determine between them
(look at BIOS, or 'that model will always show this or that in the PCI
table') and guesses the correct (initial) config as tested by some other
user. This is the automated equivalent of going to http://www.google.com/linux,
typing your machine name followed by 'kernel .config'. If the site
it contacted was configurable by the distro, you'd then have
the distros praising you in that once they have solved the problem
for one IBM T23, they've solved it for all of them, without doing
a new release. And Aunt Tillie (apart from the module changes whatever)
can be using the kernel version etc. from their distro (recompiled),
rather than the latest 2.[2468].xx with lots of new bugs^Wunwanted
fixes in.

--
Alex Bligh

2002-01-15 22:18:54

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

--On Monday, 14 January, 2002 9:15 PM +0000 Alan Cox
<[email protected]> wrote:

>> "Crap." Melvin thinks. "I don't remember what kind of network card I
>> compiled in. Am I going to have to open this puppy up just to eyeball
>> the hardware?" Doing that would take time Melvin was planning to spend
>
> So he builds a kernel with modular setups just like the vendor kernel.

& if this is the sole aim, just (configurably no doubt) stick
.config somewhere in initramfs as part of the build process
and you have no parsing to do whatsoever.

For added contraversiality, optionally stick it (gzipped at source)
inside some module that registers an entry into /proc. This
means it only occupies its gzipped space in RAM. & yes I know
this has been discussed before.

However, I'd submit that recovering the previous .config file
is the least of your problems. If I'm the only one who tries
to beg/borrow/steal config fies form other machines I've built,
the net at large, etc. etc., then tweak then, rather than
mis-answer a seemingly endless sequence of questions, I'll eat
hat.

Besides if the user is clued up enough to go buy (for example)
a GigE card, they will know it's possible they have to turn
the option on in the driver, and making small incremental
changes like that are relatively easy. What's hard is (for
instance) when kernel builds with particular .config files
hang hard on boot. Normally once you have something 'nearly
working' tweaking it to suit your taste is relatively
easy (as you change only one thing at a time). But my
practice, and I'm sure I'm not alone, is to start off
either with a config I've used elsewhere, or one someone
else has developed for that machine (the latter being
the only option for Aunt Tillie). [I then do a diff between
what I used, and the other option, and start integrating
changes - being able to show Aunt Tillie the difference
between vendor options and ones 'for her machine' would
doubtless be useful]

--
Alex Bligh

2002-01-15 22:46:37

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

>> From the other side, how does having the ability to probe local hardware
>> hurt? It should be cleanly seperable from the classical build process
>> for the purists, and helpful to some (I think) significant portion of
>> the userbase, particularly those folks who like to test bleeding edge
>> stuff on a variety of hardware. I don't really understand the
>> resistance to the idea of someone going out and implementing this.
>
> Right, and this is 95% possible even. Doing PCI stuff is rather easy
> (since we've got it all mapped out even). The problem is the 100%
> point-click-run goal that Eric has.
>
> The original sticking point was doing ISA (and other buses that are
> _not_ autodetect friendly in a safe way).

& this has a seemingly obvious solution, which is, if the autoprobe
stuff is selected, and, after presentation of the initial list
of drivers, plus comments like 'Network card: none', 'Sound card: none',
say 'We may have missed some stuff if you have an old computer, press
Y if what we've detected doesn't find all your hardware', and if
they press Y (only), select as modules every ISA driver except
those, which when loaded on a system not containing the relevant
card, can cause a hangup; thus deferring the autoprobing until
boot time.

--
Alex Bligh

2002-01-15 22:57:37

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, Jan 15, 2002 at 10:46:11PM -0000, Alex Bligh - linux-kernel wrote:
> >>From the other side, how does having the ability to probe local hardware
> >>hurt? It should be cleanly seperable from the classical build process
> >>for the purists, and helpful to some (I think) significant portion of
> >>the userbase, particularly those folks who like to test bleeding edge
> >>stuff on a variety of hardware. I don't really understand the
> >>resistance to the idea of someone going out and implementing this.
> >
> >Right, and this is 95% possible even. Doing PCI stuff is rather easy
> >(since we've got it all mapped out even). The problem is the 100%
> >point-click-run goal that Eric has.
> >
> >The original sticking point was doing ISA (and other buses that are
> >_not_ autodetect friendly in a safe way).
>
> & this has a seemingly obvious solution,
[snip]

> those, which when loaded on a system not containing the relevant
> card, can cause a hangup; thus deferring the autoprobing until
> boot time.

If everything worked before as a module, this works still. If not, you
need to have a for loop or something that attempts to load every XXX
subsystem driver.

But the point I was getting at is the 100% solution isn't possible.
Even if we narrow the scope of the problem to just x86 hardware, theres
still things which can't be solved. Talking to random memory locations
will bite you at somepoint on some hw. Old busses weren't designed with
any sort of autoconfig in mind.

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

2002-01-15 23:23:47

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, Jan 15, 2002 at 10:17:13PM -0000, Alex Bligh - linux-kernel wrote:

> & if this is the sole aim, just (configurably no doubt) stick
> .config somewhere in initramfs as part of the build process
> and you have no parsing to do whatsoever.

initramfs goes away, I believe. But most vendors stick their config in
/boot/config-`uname -r`, and last I looked at kbuild-2.5, it asked if
you wanted to stick your .config in /lib/modules/`uname -r` (which is
default loc for System.map too..) Or maybe it just did it.

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

2002-01-16 03:41:52

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Tuesday 15 January 2002 07:39 am, David S. Miller wrote:
> From: Alan Cox <[email protected]>
> Date: Tue, 15 Jan 2002 12:44:38 +0000 (GMT)
>
> If at boot time we keep a big chunk of ram free at the kernel end and
> just load modules one after each other into that space until we get into
> real paging that problem goes away
>
> And we do have module_map/module_unmap interfaces already so it's
> easy to toy with this.
>
> I've always been meaning to do a "alloc 4MB page at boot, lock into
> TLB, carve module pages out of that, vmalloc when that runs out" on
> sparc64.

How would this handle modules being unloaded? Are you going to try to pack
the space, try to re-use the holes (allowing fragmentation), handle unloading
the last module only (high water mark), or just ignore it and basically not
allow modules to ever be unloaded?

Just curious...

Rob

2002-01-16 03:41:42

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tuesday 15 January 2002 04:14 am, Sean Hunter wrote:
> On Mon, Jan 14, 2002 at 05:34:23PM -0500, Eric S. Raymond wrote:
> > Because the second we stop thinking about Aunt Tillie,
> > we start making excuses for badly-designed interfaces and excessive
> > complexity.
>
> Bollocks. The second we (including you) stop thinking about the _user_ of
> the technology, we make bad decisions. This is not the same thing.
>
> We don't expect Aunt Tillie to write kernel drivers for her knitting
> machine. She (and we) expect(s) someone else to do that for her.
>
> The Aunt Tillies of this world don't install of update Windows (or Mac O/S)
> for themselves except perhaps via "Windows Update" or "Apple Update", which
> (guess what) supplies a prebuilt binary and DOESN'T BUILD THEM A KERNEL.
>
> Besides any other factor, the download/install/reboot time is less than the
> download-full-tarball/untar/configure/compile/install/reboot cycle.
>
> Sean

Far down on my to-do list is breaking out the "linux from scratch" project,
reading through it, and making a small, simple distribution that doesn't have
ANY precompiled binaries except the initial boot disk. It should compile and
install everything from source code. "configure; make; make install" (I
wouldn't be suprised if somebody's already done this. It would save me a lot
of work, actually. But I haven't found it yet.)

The reason for this isn't that I expect end-users to deal with it, it's that
whenever I'm trying to debug a problem in X or Konqueror or some such, it's
ten times as much work to get the darn thing to compile and install from
source than it is to track down and patch the actual BUG. (Especially
getting the home-built version and the RPM-installed version not to fight to
the death configuration-wise, either overwriting OR installing them in
seperate directories.) Most of the time I just dump the problem on my "to
do" list and never get around to it. (I admit X and KDE are a bit worse than
average here, and I've only really wanted to patch a bug in glibc once. But
still, it would be nice to have the option of following my nose into the
source without doing four hours worth of preparatory work first...)

Possibly I've just had more than my share of bad experiences in this area...

Rob

2002-01-16 04:26:06

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alex Bligh - linux-kernel <[email protected]>:
> & this has a seemingly obvious solution, which is, if the autoprobe
> stuff is selected, and, after presentation of the initial list
> of drivers, plus comments like 'Network card: none', 'Sound card: none',
> say 'We may have missed some stuff if you have an old computer, press
> Y if what we've detected doesn't find all your hardware', and if
> they press Y (only), select as modules every ISA driver except
> those, which when loaded on a system not containing the relevant
> card, can cause a hangup; thus deferring the autoprobing until
> boot time.

Not a bad idea, but it conflicts with one of the goals of
`make autoconfigurator', which is completely hands-off opration.

Giacomo Catenazzi thinks he has collected enough "safe" probes to
find effectively all ISA devices, by grovelling through various bits
of /proc. So this problem may get solved directly.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Probably fewer than 2% of handguns and well under 1% of all guns will
ever be involved in a violent crime. Thus, the problem of criminal gun
violence is concentrated within a very small subset of gun owners,
indicating that gun control aimed at the general population faces a
serious needle-in-the-haystack problem.
-- Gary Kleck, "Point Blank: Handgun Violence In America"

2002-01-16 07:47:17

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tuesday 15 January 2002 04:38 pm, Alex Bligh - linux-kernel wrote:

> However, Eric's approach (dmesg) is still flawed as normally
> the way these distros fail is either (a) hanging on boot, or
> (b) failing to detect the relevant hardware. Needless to say,
> neither failure mode is going to give much use to a configurator
> tool which looks at dmesg.

"make autoconfigure" looks at a bunch of things, one of which is dmesg. (It
also looks at the PCI bus, isapnp, enumerates filesystems in use out of the
mounted partitions, checks /proc/cpuinfo to see what to optimize for...)

It's actually doing a fairly decent job, although it's not quite ready for
prime time yet. (Improving rapidly, of course, as we continue to thump on
it. :)

> Eric: I think you'd be far better off trying to identify the
> machine (and hence get a working .config) rather than the
> hardware.
>
> Example: put in some wget based thingy, which goes to some (fixed) web
> site, searches for (some extracted or Tillie composed string) which
> describes the hardware (bound to have been bought as-is and never opened),
> pulls down a set of config files and heuristics to determine between them
> (look at BIOS, or 'that model will always show this or that in the PCI
> table') and guesses the correct (initial) config as tested by some other
> user.

Meaning you'll continue to be six months behind the curve, and fail every
time Dell tweaks its laptop layout. (Dell does things like switch sound
chips without switching model numbers ALL THE TIME.)

Are you volunteering to maintain this database?

So no-name assembled white boxes from e-machines and stuff wouldn't be
supported?

Have you TRIED the current auto-configurator?

> This is the automated equivalent of going to http://www.google.com/linux,
> typing your machine name followed by 'kernel .config'. If the site
> it contacted was configurable by the distro, you'd then have
> the distros praising you in that once they have solved the problem
> for one IBM T23, they've solved it for all of them, without doing
> a new release.

Assuming every IBM T23 has the same hardware in it, which oddly enough is a
bit of a gamble. (OK, IBM is better at this than Dell, largely due to
inventory management reasons.) And assuming the finite number of database
maintainers has yet bought an IBM T23, and that the rest of the world can
wait until then.

Requiring live network access for the autoconfigurator to work is one heck of
an extra requirement, though. Most of the world is still using dialup, you
know...

> And Aunt Tillie (apart from the module changes whatever)
> can be using the kernel version etc. from their distro (recompiled),
> rather than the latest 2.[2468].xx with lots of new bugs^Wunwanted
> fixes in.

You want to write some other tool.

In order to compile a new kernel and use it on a new machine, you need to
configure it, which is time consuming and tedious, and can require a bit of
detective work. This is a problem that Giacamo and Eric decided to address.

This is NOT the problem you're trying to address.

Aunt Tillie is a side issue. She's going to continue to run Windows until
Linux comes preinstalled on her new computer, or until somebody ELSE installs
it for her and does an awful lot of hand holding. And what she probably
really WANTS is an iMac. :)

Autoprobing PCI is -EASY-. Almost trivial. USB and PCMCIA/Cardbus were
DESIGNED to be autoprobed. Finding out your CPU type and chipset aren't too
hard either.

It's really the old nasty ISA devices that are a pain to auto-probe, and they
are finally, mercifully, dying off. The newer and more naieve the user, the
less likely they are to have lashed together an old 486 with VESA local bus,
three different SCSI adapters, a CD-ROM hanging off the sound blaster, and a
ham radio interface plugged into the parallel port. Autoprobe really should
become EASIER as time goes on.

Giacamo and Eric started work on the autoprobe as a way to reduce the number
of questions the configurator showed people by eliminating hardware that they
provably do not have, and defaulting the stuff they DO have to on. But it
turns out that on any relatively recent machine, it's an easy enough problem
that you can autoprobe EVERYTHING and build straight from that. So the Linux
kernel could finally do "configure; make; make install".

I consider that a neat hack.

Rob

2002-01-16 09:10:16

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" <[email protected]> said:

[...]

> It's not that hard. Given Giacomo's table of probes it only took me about
> two days to get it 85% of the way there. The remaining 15% is partly
> issues with rulebase bugs that it exposes, and partly issues with what
> to do about the possibility of ISA hardware that is not directly
> probeable.

Ever heard of "90% done, the remaining 10% will take the other 90% of the
effort"? Just as predicted, a long way back... then comes a two or three
year shakedown to get it reliable enough for real use.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 09:10:16

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Eric S. Raymond" <[email protected]> said:
> [email protected] <[email protected]>:
> > Of course there are other settings that do have impact (CPU type mostly,
> > maybe memory layout) but other than that... distros already ship several
> > binary versions (last I counted Red Hat ships 11 or so with RHL72) to
> > account for CPU type and amount etc.
>
> OK. Scenario #2:
>
> Tillie's nephew Melvin is a junior-grade geek. He's working his way
> through college doing website administration for small businesses. He
> doesn't know C, but he can hack his way around Perl and a little PHP,
> and he can type "configure; make". He's been known to wear a penguin
> T-shirt.

> [Melvin needs a hand-tailored kernel for a crappy webserver. Source is
> _long_ gone]

Melvin is a html-only geek, but not stupid. He knows where he can get the
sources for the distribution's kernel, and he knows very well that it is
advisable to stick to the distribution's software. He understands enough of
the process so he can tailor a kernel for _webserving_, not optimal use of
all hardware on Ye Olde Coffepot. He leaves out sound, serial support, and
acceleration support for the video card (What for in a webserver?),
configures floppy and CD as modules (to save the odd byte), just ext3 as
builtin filesystem (no need for all the others, iso9660 is module for some
savings). He includes iptables support, but no connection tracking or NAT
modules. All things an autoconfigurator wouldn't dream of doing on its own.

[Again, building a tailored kernel for a specific application on a crappy
machine is no place for automatism; for everything else an Aunt Tillie
might need a "get nearest size" is enough]

[...]

> "Crap." Melvin thinks. "I don't remember what kind of network card I
> compiled in. Am I going to have to open this puppy up just to eyeball
> the hardware?" Doing that would take time Melvin was planning to spend
> chatting up a girl geek he's noticed over at the computer lab.

If Melvin did compile the kernel in the first place, he is smart enough to
keep .configure around. If not, he doesn't deserve getting laid.

For terminally stupid Melvins, keep .configure around somewhere.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 09:10:16

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

"Eric S. Raymond" <[email protected]> said:

[...]

> Actually I think we may no longer be in tristate-land. Instead, some
> devices have the property "This belongs in initramfs if it's configured
> at all" -- specifically, drivers for potential boot devices. Everything
> else can dynamic-load after boot time.

Then all SCSI drivers end up in the initramfs for the install kernel for
a distro? There might be _many_ devices configured that don't need to
reside on the initramfs.

OTOH, the initramfs will get built and populated by separate tools (just
like initrd is done today), so the alternatives would boil down to M/N.
_If_ the grand scheme of getting rid of builtins flies. I for one wouldn't
hold my breath.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 09:10:26

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

[email protected] (Henrique de Moraes Holschuh) said:

[...]

> Looks like Debian's look-mama-I-can-beat-godzila-hands-down 2.4 kernel
> packages with everything AND the kitchen sink... :) They eat about 20Mb of
> diskspace, I think.
>
> Package: kernel-image-2.4.16-k7
> Installed-Size: 22976

Red Hat's 2.4.6-13 for i686 adds up to 50297287 bytes.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 09:17:08

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Horst von Brand <[email protected]>:
> > Actually I think we may no longer be in tristate-land. Instead, some
> > devices have the property "This belongs in initramfs if it's configured
> > at all" -- specifically, drivers for potential boot devices. Everything
> > else can dynamic-load after boot time.
>
> Then all SCSI drivers end up in the initramfs for the install kernel for
> a distro? There might be _many_ devices configured that don't need to
> reside on the initramfs.

No worries. I have working code for a better approach now. It turns
out not to be that hard to discover the root device, controller, and bus.
--
<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

The end move in politics is always to pick up a gun.
-- R. Buckminster Fuller

2002-01-16 10:44:00

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Andrew Pimlott <[email protected]> said:

[...]

> It's just as easy in principle to write a friendly front-end that
> downloads sources and compiles them, as one that downloads binaries.

That it is _possible_ do do so doesn't mean it is _worthwhile_. I do trust
Red Hat's binaries (because of QA), there is reasonable support for them to
correlate bug reports, etc. I _don't_ trust kernel.org sources, at least
not for production use. Sure, I fool around with them on my machines. There
they have eaten a few filesystems, some crashed on boot, others were
useless.

Aunt Tillie (and Nephew Mervin, and the whole bunch of relatives) present a
support (i.e., distribution and people) problem. They are _production_, not
_experiments_. Sure, we could mandate that kernels from the stable series
go through rigurous QA before being posted. Progress would slow to a crawl,
the fun of hacking Linux would soon be over, and so would be Linux. Or
there would be trhee branches: Production quality, stable, and
experimental. Next you know, Uncle Eric would then demand autobuilding for
Aunt Tillie from the stable (or even experimantal) branch for updated
driver for the Foomatic 37.

Autobuilding *is here*, *right now*. It is called "distribution's kernel",
aided by something hacked up a time ago called "modules" and "autoloading".

As techies we always try to find a technical solution to problems, but
there are problems where the best solution isn't technical at all. It is
organizational, education, setting up a support network, ... There are even
problems that _have_ no technical solutions. Or even problems that have
perfectly adequate non-technical solutions right now, which we overlook.

> The obstacle is reliability, because there are more things that can
> go wrong.

Bingo! With a binary, the vendor _knows_ what was shipped, and has checked
that configuration to death. With autoconfiguration this is probably the
_only_ copy of that particular configuration on the whole world. And
Point&Drool Nephew Mervin is supposed to find out why it doesn't boot
today. Forget about eyeing Penelope, he won't have time for that.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 10:50:20

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Tue, 15 Jan 2002 16:22:52 -0700,
Tom Rini <[email protected]> wrote:
>... and last I looked at kbuild-2.5, it asked if
>you wanted to stick your .config in /lib/modules/`uname -r` (which is
>default loc for System.map too..) Or maybe it just did it.

The first time you make oldconfig kbuild 2.5 asks if you want to
install System.map and .config, separate questions. If you say y then
it asks where you want to install the files, with a default of
/lib/modules/`uname -r`/. Once set, the values propagate to subsequent
builds that use the same .config.

2002-01-16 10:57:30

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Giacomo Catenazzi <[email protected]> said:
> It is easier to get autoconfigure in linux sources, than
> modify the default (and broken) configuration from Linus.
> (Sorry Linus :-) )

Say again? Then get HPA to give you a space on kernel.org for "Giacomo
Standard Configure Entries" which you update. _Much_ less pain going around
that way.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 14:34:26

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Rob,

>> Example: put in some wget based thingy, which goes to some (fixed) web
>> site, searches for (some extracted or Tillie composed string) which
>> describes the hardware (bound to have been bought as-is and never
>> opened), pulls down a set of config files and heuristics to determine
>> between them (look at BIOS, or 'that model will always show this or that
>> in the PCI table') and guesses the correct (initial) config as tested by
>> some other user.
>
> Meaning you'll continue to be six months behind the curve, and fail every
> time Dell tweaks its laptop layout. (Dell does things like switch sound
> chips without switching model numbers ALL THE TIME.)
>
> Are you volunteering to maintain this database?

No, noone 'maintains' it; I actually /meant/ search.
So if you run it on a Redhat distribution, it goes queries
redhat first, then lists
IBM T23 - Alex Bligh
IBM T23 new version - Rob Landley
...
etc.

IE is the minimum amount of automation to simulate typing the thing
into google. (the advantage of doing this over using google
alone is that X may not be working at this stage and lynx
might daunt Aunt Tillie).

Obviously you will need some magic tag at the top of a config
file to make it easilly identifiable to search engines. And no
doubt some fools will fill files with crap.

> So no-name assembled white boxes from e-machines and stuff wouldn't be
> supported?

Correct; I'm sure the probing configurator has a place too.

> Have you TRIED the current auto-configurator?

No, to be honest. However, now you have set me the challenge, I'll
report back on how well or otherwise it works.

> Assuming every IBM T23 has the same hardware in it, which oddly enough is
> a bit of a gamble. (OK, IBM is better at this than Dell, largely due to
> inventory management reasons.) And assuming the finite number of
> database maintainers has yet bought an IBM T23, and that the rest of the
> world can wait until then.

Well, you'd wait until either your distro vendor had done one, or
someone else had posted one and it had reached search-engine
du-jour.

> Requiring live network access for the autoconfigurator to work is one
> heck of an extra requirement, though. Most of the world is still using
> dialup, you know...

True.

>> And Aunt Tillie (apart from the module changes whatever)
>> can be using the kernel version etc. from their distro (recompiled),
>> rather than the latest 2.[2468].xx with lots of new bugs^Wunwanted
>> fixes in.
>
> You want to write some other tool.

Perhaps you are right.

> Giacamo and Eric started work on the autoprobe as a way to reduce the
> number of questions the configurator showed people by eliminating
> hardware that they provably do not have, and defaulting the stuff they
> DO have to on. But it turns out that on any relatively recent machine,
> it's an easy enough problem that you can autoprobe EVERYTHING and build
> straight from that. So the Linux kernel could finally do "configure;
> make; make install".
>
> I consider that a neat hack.

Sure I agree, if it works; my speculation is that it doesn't, if one
boots with a default vendor kernel, on many machines. I would love
to be proved wrong, so rather than debate further here, I have another
T23 to set up so I'll boot it up from scratch, run the configurator,
and see what happens.

--
Alex Bligh

2002-01-16 21:09:12

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

On Wednesday 16 January 2002 09:33 am, Alex Bligh - linux-kernel wrote:

> IE is the minimum amount of automation to simulate typing the thing
> into google. (the advantage of doing this over using google
> alone is that X may not be working at this stage and lynx
> might daunt Aunt Tillie).

So you're advocating feeding random stuff from the internet directly to end
users without any sturgeon's law filter at all, and this is an improvement
over automated probing of mostly standards-compliant busses for known
hardware?

> Obviously you will need some magic tag at the top of a config
> file to make it easilly identifiable to search engines. And no
> doubt some fools will fill files with crap.

I was specifically thinking Microsoft, actually. And every script kiddie in
existence who thinks screwing up the linux automated install gives them 31337
status. (A new reason to crack webservers: add links to poison the google
cache. Sigh...)

> > So no-name assembled white boxes from e-machines and stuff wouldn't be
> > supported?
>
> Correct; I'm sure the probing configurator has a place too.

I'm all for having a "known strange" list. Eric volunteered to maintain one
in regards to DMI, which is sort of per-motherboard anyway.

But I tend to lean in favor of a having a maintainer for a database people
contribute exceptions to. Which basically means we might as well just make
the autoprober smarter via an exception list...

> > Have you TRIED the current auto-configurator?
>
> No, to be honest. However, now you have set me the challenge, I'll
> report back on how well or otherwise it works.

At the moment, "make autoprobe" is a really nice tool for setting 99% of the
questions in "make menuconfig" to the correct answers.

However, it's also a great tool for squeezing obscure bugs out of the
database like the fact if you DO switch on a PCMCIA network card that doesn't
automatically switch on CONFIG_NET. (Missing dependency, not yet in 2.1.3.
One of the four questions it gets wrong about my laptop. Out of a hundred
and something, that's not bad...)

As a result "make autoconfigure" generally needs a couple of switches
manually flipped before you get the config you want. (But we're working on
it. More boxes to probe is always a good thing... Send the results to Eric.
Or me, since Eric will be on the road this weekend. Or that Gicacamo
Catanazi dude, whose name I am highly unlikely to have spelled correctly
because the deck is stacked against me here.)

> > Assuming every IBM T23 has the same hardware in it, which oddly enough is
> > a bit of a gamble. (OK, IBM is better at this than Dell, largely due to
> > inventory management reasons.) And assuming the finite number of
> > database maintainers has yet bought an IBM T23, and that the rest of the
> > world can wait until then.
>
> Well, you'd wait until either your distro vendor had done one, or
> someone else had posted one and it had reached search-engine
> du-jour.

Sure. If you want your distribution vendor to build your kernel for you,
that's great. We're just trying to give you another option.

> > Giacamo and Eric started work on the autoprobe as a way to reduce the
> > number of questions the configurator showed people by eliminating
> > hardware that they provably do not have, and defaulting the stuff they
> > DO have to on. But it turns out that on any relatively recent machine,
> > it's an easy enough problem that you can autoprobe EVERYTHING and build
> > straight from that. So the Linux kernel could finally do "configure;
> > make; make install".
> >
> > I consider that a neat hack.
>
> Sure I agree, if it works; my speculation is that it doesn't, if one
> boots with a default vendor kernel, on many machines.

Right now, you're right. It ALMOST works, but the bugs are still being
squeezed out. And about half the bugs are in the configuration rulebase
rather than in the autoprober. (It's amazing how many bad combinations CML1
would allow you to do. Did you know if you accidentally turn off the
networking menu in the old menuconfig, there's no way to get it back without
editing the config file by hand? Sigh...)

> I would love
> to be proved wrong, so rather than debate further here, I have another
> T23 to set up so I'll boot it up from scratch, run the configurator,
> and see what happens.

Please let me know how it breaks. (No, I'm not going to say "if", I'm not
that optimistic yet. 2.2.0 perhaps...)

Rob

2002-01-16 21:40:20

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Thomas Duffy <[email protected]> said:

[...]

> - Someday, a stupid government or court decides that there is a strict
> separation between source and binary. Source is protected speech, but
> binaries are not. Linux decides it wants a really fast DVD decryption
> in the kernel, so it adds it in drivers. But now, distro's cannot
> compile and distribute a binary kernel package and the end user will
> need to compile the source code in order to watch their DVD.

No need for a full compile, just the offending driver. IF and WHEN it
happens. Till then, why bother?

> Why is it unrealistic for everybody to compile their kernel when they do
> an install? If it is rather automated, then it just becomes another
> step on the progress bar.

If each time you install $distro you have to get compiler, linker, Python,
and whatnot up and running; and then have to wait a half hour or so for the
standard distributed kernel to compile, I don't see the point. Its much
easier/faster/secure to just distribute and install binaries.
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 21:37:29

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

"Richard B. Johnson" <[email protected]> said:

[...]

> Really??? Have you ever tried this? RedHat provides a directory
> of random patches that won't patch regardless of the order in
> which you attempt patches (based upon date-stamps on patches or
> date-stamps on files). It's like somebody just copied in some
> junk, thinking nobody would ever bother.

rpm(1), particularly "rpm -p" on the SRPM. Or use the kernel-source RPM.

Last time I looked, they carried around patches that weren't applied in the
SRPM.

> Some distributions don't even provide source. They provide
> copies of /usr/src/linux/include/asm and /usr/src/linux/include/linux
> but nothing else. You have to "find" source on the internet.

They are in violation of GPL then. Make it clear to them.

> The "good-ol-days" where you could get 72 floppies from Yggdrasil,
> install Linux, and spend the next 48 hours watching it compile
> are long gone.

What stops you from getting Red Hat, and doing a massive "rpm --rebuild"
today?

> I have never found a distribution that uses modules, in which is
> was even remotely possible to duplicate the kernel supplied.

Why do they then bother to distribute sources?
--
Horst von Brand http://counter.li.org # 22616

2002-01-16 21:37:19

[permalink] [raw]

Subject: Re: Aunt Tillie builds a kernel (was Re: ISA hardware discovery -- the elegant solution)

Alex Bligh - linux-kernel <[email protected]> said:
> Eric: I think you'd be far better off trying to identify the
> machine (and hence get a working .config) rather than the
> hardware.

I've seen machines bought from the very same batch with _completely_
different innards. BTW, around in Chile you mostly get some store-build
machine, for years all in the same box with the same logo outside. Good
luck at finding CornerStoreComputer on your database... and if you do, may
it help find out which mobo was the cheapest around when it was built.
--
Horst von Brand http://counter.li.org # 22616

2002-01-18 12:41:26

by Florian Weimer

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

"Mr. James W. Laferriere" <[email protected]> writes:

> Hello Alan ,
>
> On Mon, 14 Jan 2002, Alan Cox wrote:
>> > 1. security, if you don't need any modules you can disable modules entirly
>> > and then it's impossible to add a module without patching the kernel first
>> > (the module load system calls aren't there)
>>
>> Urban legend.
> I do not agree . Got proof ? Yes that is a valid question .

http://www.phrack.org/phrack/58/p58-0x07

Globally preloading a shared library in user space is almost as
effective, BTW, unless your critical binaries are linked statically
(which is unusual on most systems nowadays).

--
Florian Weimer [email protected]
University of Stuttgart http://CERT.Uni-Stuttgart.DE/people/fw/
RUS-CERT +49-711-685-5973/fax +49-711-685-5898

2002-01-18 22:21:08

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Hello Florian ,

On Fri, 18 Jan 2002, Florian Weimer wrote:
> "Mr. James W. Laferriere" <[email protected]> writes:
> > Hello Alan ,
> > On Mon, 14 Jan 2002, Alan Cox wrote:
> >> > 1. security, if you don't need any modules you can disable modules entirly
> >> > and then it's impossible to add a module without patching the kernel first
> >> > (the module load system calls aren't there)
> >> Urban legend.
> > I do not agree . Got proof ? Yes that is a valid question .
> http://www.phrack.org/phrack/58/p58-0x07
Thank you for the pointer . Fine you do not need to allow modules
in order for a hacker to insert their code . It is still another
thing to allow modules & not put & use signatures on them . AFAIK
Linux doesn't have a method to load encrypted & signed modules at
this time . Please , someone prove me wrong . I -personally-
like statically compiled kernels . The method being pushed forth
at present doesn't allow that , Unless I am completely mistaken
about what Alan & the rest have been discussing . Again PLEASE
someone prove me wrong about this also .

> Globally preloading a shared library in user space is almost as
> effective, BTW, unless your critical binaries are linked statically
> (which is unusual on most systems nowadays).
I can beleive that 8-} . Statically or written to readonly media
or the drive hard set to readonly ;-) . I have followed most of
Alan's suggestions security concerns over the years & a few that
I thought of along the way . Which later I found had been being
done alot longer than I would have thought . Tia , JimL

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-18 23:23:19

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Fri, 18 Jan 2002 17:20:02 -0500 (EST),
"Mr. James W. Laferriere" <[email protected]> wrote:
> Linux doesn't have a method to load encrypted & signed modules at
> this time .

And never will. Who loads the module - root. Who maintains the list
of signatures - root. Who controls the code that verifies the
signature - root.

Your task Jim, should you choose to accept it, is to make the kernel
distinguish between a good use of root and a malicious use by some who
has broken in and got root privileges. When you can do that, then we
can add signed modules.

2002-01-21 00:44:01

by Frank van de Pol

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Sat, Jan 19, 2002 at 10:22:43AM +1100, Keith Owens wrote:
> On Fri, 18 Jan 2002 17:20:02 -0500 (EST),
> "Mr. James W. Laferriere" <[email protected]> wrote:
> > Linux doesn't have a method to load encrypted & signed modules at
> > this time .
>
> And never will. Who loads the module - root. Who maintains the list
> of signatures - root. Who controls the code that verifies the
> signature - root.
>
> Your task Jim, should you choose to accept it, is to make the kernel
> distinguish between a good use of root and a malicious use by some who
> has broken in and got root privileges. When you can do that, then we
> can add signed modules.
>

If you want to secure your box, why don't you simply put a lock on it and
throw away the key? Really, what might help the paranoid admins in this case
is a setting in the kernel which basically disables the ability to load or
unload modules. Of course once set this setting can not been turned with
rebooting the box.

Frank.

--
+---- --- -- - - - -
| Frank van de Pol -o) A-L-S-A
| [email protected] /\\ Sounds good!
| http://www.alsa-project.org _\_v
| Linux - Why use Windows if we have doors available?

2002-01-21 00:55:41

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 21 Jan 2002 00:20:41 +0100
Frank van de Pol <[email protected]> wrote:

> If you want to secure your box, why don't you simply put a lock on it and
> throw away the key? Really, what might help the paranoid admins in this case
> is a setting in the kernel which basically disables the ability to load or
> unload modules. Of course once set this setting can not been turned with
> rebooting the box.

...and how would you guarantee that this setting remains set, in the face of
some nasty little cracker screwing around with /dev/kmem?

2002-01-21 01:05:02

by Anton Altaparmakov

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

At 23:20 20/01/02, Frank van de Pol wrote:

>On Sat, Jan 19, 2002 at 10:22:43AM +1100, Keith Owens wrote:
> > On Fri, 18 Jan 2002 17:20:02 -0500 (EST),
> > "Mr. James W. Laferriere" <[email protected]> wrote:
> > > Linux doesn't have a method to load encrypted & signed modules at
> > > this time .
> >
> > And never will. Who loads the module - root. Who maintains the list
> > of signatures - root. Who controls the code that verifies the
> > signature - root.
> >
> > Your task Jim, should you choose to accept it, is to make the kernel
> > distinguish between a good use of root and a malicious use by some who
> > has broken in and got root privileges. When you can do that, then we
> > can add signed modules.
>
>If you want to secure your box, why don't you simply put a lock on it and
>throw away the key? Really, what might help the paranoid admins in this case
>is a setting in the kernel which basically disables the ability to load or
>unload modules. Of course once set this setting can not been turned with
>rebooting the box.

Er that sounds like just disabling modules in the kernel altogether (kernel
compile option exists for this since the beginning of time)... I do that on
all servers I control. Not only for security reasons but also because I
suspect it produces smaller and probably faster kernels (I haven't tested
this in any way, just a guess).

Best regards,

Anton

--
"I've not lost my mind. It's backed up on tape somewhere." - Unknown
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

2002-01-21 01:08:32

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Hello Anton ,

On Mon, 21 Jan 2002, Anton Altaparmakov wrote:
> At 23:20 20/01/02, Frank van de Pol wrote:
> >On Sat, Jan 19, 2002 at 10:22:43AM +1100, Keith Owens wrote:
> > > On Fri, 18 Jan 2002 17:20:02 -0500 (EST),
> > > "Mr. James W. Laferriere" <[email protected]> wrote:
> > > > Linux doesn't have a method to load encrypted & signed modules at
> > > > this time .
> > > And never will. Who loads the module - root. Who maintains the list
> > > of signatures - root. Who controls the code that verifies the
> > > signature - root.
> > > Your task Jim, should you choose to accept it, is to make the kernel
> > > distinguish between a good use of root and a malicious use by some who
> > > has broken in and got root privileges. When you can do that, then we
> > > can add signed modules.
> >If you want to secure your box, why don't you simply put a lock on it and
> >throw away the key? Really, what might help the paranoid admins in this case
> >is a setting in the kernel which basically disables the ability to load or
> >unload modules. Of course once set this setting can not been turned with
> >rebooting the box.

> Er that sounds like just disabling modules in the kernel altogether (kernel
> compile option exists for this since the beginning of time)... I do that on
> all servers I control. Not only for security reasons but also because I
> suspect it produces smaller and probably faster kernels (I haven't tested
> this in any way, just a guess).
This is just what the Heads are trying to do away with . There
will only be module enabled kernels . JimL

+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| [email protected] | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+

2002-01-21 01:29:14

by Anton Altaparmakov

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

At 01:07 21/01/02, Mr. James W. Laferriere wrote:
>On Mon, 21 Jan 2002, Anton Altaparmakov wrote:
> > At 23:20 20/01/02, Frank van de Pol wrote:
> > >On Sat, Jan 19, 2002 at 10:22:43AM +1100, Keith Owens wrote:
> > > > On Fri, 18 Jan 2002 17:20:02 -0500 (EST),
> > > > "Mr. James W. Laferriere" <[email protected]> wrote:
> > > > > Linux doesn't have a method to load encrypted & signed modules at
> > > > > this time .
> > > > And never will. Who loads the module - root. Who maintains the list
> > > > of signatures - root. Who controls the code that verifies the
> > > > signature - root.
> > > > Your task Jim, should you choose to accept it, is to make the kernel
> > > > distinguish between a good use of root and a malicious use by some who
> > > > has broken in and got root privileges. When you can do that, then we
> > > > can add signed modules.
> > >If you want to secure your box, why don't you simply put a lock on it and
> > >throw away the key? Really, what might help the paranoid admins in
> this case
> > >is a setting in the kernel which basically disables the ability to load or
> > >unload modules. Of course once set this setting can not been turned with
> > >rebooting the box.
>
> > Er that sounds like just disabling modules in the kernel altogether (kernel
> > compile option exists for this since the beginning of time)... I do that on
> > all servers I control. Not only for security reasons but also because I
> > suspect it produces smaller and probably faster kernels (I haven't tested
> > this in any way, just a guess).
> This is just what the Heads are trying to do away with . There
> will only be module enabled kernels . JimL

Ah! Serves me right for jumping into a thread without reading the previous
posts... I better shut up as I have no idea why this is being done. From my
unknowledgeable point of view it seems daft but there probably are good
reasons... I suppose even non-modular kernels can be modified by the root
user to be exploited, it's just a little bit harder...

Best regards,

Anton

--
"I've not lost my mind. It's backed up on tape somewhere." - Unknown
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

2002-01-21 01:37:05

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

"Mr. James W. Laferriere" wrote:
>
> This is just what the Heads are trying to do away with . There
> will only be module enabled kernels . JimL

I suspect none of these "Heads" spend much time in protracted
email debug sessions. Because the *first* thing you do is
ask the tester to compile the relevant driver into the
kernel.

The problems which the removal of this option will cause include:

1: Inability to look up symbols in the kernel elf image.
2: Breaks the kernel profiler
3: breaks kgdb
4: breaks ksymoops.

How often have we seen nonsensical backtraces here because
modules were involved? Possibly we can include a table
of module base addresses in the Oops output and teach ksymoops
about it.

This proposal is, frankly, brain-damaged. It will significantly
impeded kernel developers in remote problem diagnosis and it will
weaken the kernel development toolchain.

There's a lot of work to be done to overcome this damage, and
given the difficulty of getting debug tools into the mainstream
kernel, the damage may well be permanent.

-

2002-01-21 01:54:11

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Sun, 20 Jan 2002 17:30:12 -0800,
Andrew Morton <[email protected]> wrote:
>I suspect none of these "Heads" spend much time in protracted
>email debug sessions. Because the *first* thing you do is
>ask the tester to compile the relevant driver into the
>kernel.
>
>The problems which the removal of this option will cause include:
>
>1: Inability to look up symbols in the kernel elf image.
>2: Breaks the kernel profiler
>3: breaks kgdb
>4: breaks ksymoops.
>
>How often have we seen nonsensical backtraces here because
>modules were involved? Possibly we can include a table
>of module base addresses in the Oops output and teach ksymoops
>about it.

You see nonsensical backtraces because people persist in using the oops
decode option of klogd which is broken when faced with modules. Turn
off klogd oops (klogd -x) and you get a raw backtrace which ksymoops
can handle. Guess why these entries are in /proc/ksyms?

c48a2300 __insmod_3c589_cs_S.bss_L4 [3c589_cs]
c48a0000 __insmod_3c589_cs_O/lib/modules/2.4.17-xfs/kernel/drivers/net/pcmcia/3c589_cs.o_M3C332CFF_V132113 [3c589_cs]
c48a22a0 __insmod_3c589_cs_S.data_L96 [3c589_cs]
c48a1820 __insmod_3c589_cs_S.rodata_L1152 [3c589_cs]
c48a0060 __insmod_3c589_cs_S.text_L6064 [3c589_cs]

ksymoops uses the __insmod entries to work out exactly where each
module is, it gives an accurate backtrace with modules. man insmod for
details.

Kernel debuggers like kgdb and kdb use kallsyms which has full support
for modules. kgdb can also use the __insmod entries in /proc/ksyms to
tell gdb where each module was loaded.

ksymoops has a save map option (-s) which writes out the combined
system map, including the kernel and all symbols from all modules.

Sure, a dynamic system requires a little more work, but it has all been
done. Just kill the broken klogd code so it stops corrupting log data.

2002-01-21 02:10:36

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Keith Owens wrote:
>
> On Sun, 20 Jan 2002 17:30:12 -0800,
> Andrew Morton <[email protected]> wrote:
> >I suspect none of these "Heads" spend much time in protracted
> >email debug sessions. Because the *first* thing you do is
> >ask the tester to compile the relevant driver into the
> >kernel.
> >
> >The problems which the removal of this option will cause include:
> >
> >1: Inability to look up symbols in the kernel elf image.
> >2: Breaks the kernel profiler
> >3: breaks kgdb
> >4: breaks ksymoops.
> >
> >How often have we seen nonsensical backtraces here because
> >modules were involved? Possibly we can include a table
> >of module base addresses in the Oops output and teach ksymoops
> >about it.
>
> You see nonsensical backtraces because people persist in using the oops
> decode option of klogd which is broken when faced with modules. Turn
> off klogd oops (klogd -x) and you get a raw backtrace which ksymoops
> can handle.

ksymoops doesn't know what modules were loaded at the time
of the crash, and it doesn't know where they were loaded.

The `klogd -x' problem has been with us for *years* and
distributors still persist in turning it on.

> Guess why these entries are in /proc/ksyms?
>
> c48a2300 __insmod_3c589_cs_S.bss_L4 [3c589_cs]
> c48a0000 __insmod_3c589_cs_O/lib/modules/2.4.17-xfs/kernel/drivers/net/pcmcia/3c589_cs.o_M3C332CFF_V132113 [3c589_cs]
> c48a22a0 __insmod_3c589_cs_S.data_L96 [3c589_cs]
> c48a1820 __insmod_3c589_cs_S.rodata_L1152 [3c589_cs]
> c48a0060 __insmod_3c589_cs_S.text_L6064 [3c589_cs]
>
> ksymoops uses the __insmod entries to work out exactly where each
> module is, it gives an accurate backtrace with modules. man insmod for
> details.

It assumes too much. Arjan has a kksymoops thingy which does the symbol
resolution at crash-time. This seems much more reliable to me. It also
handles the common case where the running vmlinux/System.map/etc no longer
exist.

> Kernel debuggers like kgdb and kdb use kallsyms which has full support
> for modules. kgdb can also use the __insmod entries in /proc/ksyms to
> tell gdb where each module was loaded.
>
> ksymoops has a save map option (-s) which writes out the combined
> system map, including the kernel and all symbols from all modules.
>
> Sure, a dynamic system requires a little more work, but it has all been
> done. Just kill the broken klogd code so it stops corrupting log data.

Keith, I have spent a *lot* of time working weird kernel bugs
with people via email. Usually, things peter out simply because
the other party lacks the expertise or time to keep on doing
things. So the bug doesn't get fixed.

I would prefer that all this become easier, simpler and more reliable.
We need a damn good reason for deprecating statically linked kernels
and certainly none has been presented yet.

-

2002-01-21 02:11:46

by John Levon

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 21, 2002 at 12:53:28PM +1100, Keith Owens wrote:

> >How often have we seen nonsensical backtraces here because
> >modules were involved? Possibly we can include a table
> >of module base addresses in the Oops output and teach ksymoops
> >about it.
>
> You see nonsensical backtraces because people persist in using the oops
> decode option of klogd which is broken when faced with modules. Turn
> off klogd oops (klogd -x) and you get a raw backtrace which ksymoops
> can handle. Guess why these entries are in /proc/ksyms?
>
> c48a2300 __insmod_3c589_cs_S.bss_L4 [3c589_cs]

and quite often the user has unloaded / loaded modules in the meantime
and the oops is useless.

It would be nice if klogd's oops detection just passed everything to ksymoops
untouched, and stored everything somewhere using -m

I don't see any reason why the internal profiler can't have an extended EIP
range to catch module samples either. Perhaps I'm missing something. Perhaps
no one cares enough ...

regards
john

--
"I hope you will find the courage to keep on living
despite the existence of this feature."
- Richard Stallman

2002-01-21 02:25:50

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Sun, 20 Jan 2002 18:04:00 -0800,
Andrew Morton <[email protected]> wrote:
>ksymoops doesn't know what modules were loaded at the time
>of the crash, and it doesn't know where they were loaded.

/var/log/ksymoops. man insmod and look for ksymoops assistance.

>The `klogd -x' problem has been with us for *years* and
>distributors still persist in turning it on.

Tell me about it :(.

>It assumes too much. Arjan has a kksymoops thingy which does the symbol
>resolution at crash-time.

It can only get symbols that are in /proc/ksyms, i.e. the exported
symbols. Proper crash analysis needs the full symbol table.

>It also
>handles the common case where the running vmlinux/System.map/etc no longer
>exist.

So does ksymoops, with reduced detail because it only has exported
symbols.

>I would prefer that all this become easier, simpler and more reliable.
>We need a damn good reason for deprecating statically linked kernels
>and certainly none has been presented yet.

That is a different problem. Saying that modular kernels cause
problems for debugging is not a good enough reason to deprecate modular
kernels, all the problems have been solved.

2002-01-21 02:31:30

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 21 Jan 2002 02:13:55 +0000,
John Levon <[email protected]> wrote:
>On Mon, Jan 21, 2002 at 12:53:28PM +1100, Keith Owens wrote:
>> Guess why these entries are in /proc/ksyms?
>>
>> c48a2300 __insmod_3c589_cs_S.bss_L4 [3c589_cs]
>
>and quite often the user has unloaded / loaded modules in the meantime
>and the oops is useless.

/var/log/ksymoops. I added the code and documented it nicely, man
insmod or ksymoops. It's not my fault if nobody reads the docs!

>It would be nice if klogd's oops detection just passed everything to ksymoops
>untouched, and stored everything somewhere using -m

It would be better if klogd got out of the way completely. Everything
is stored, just created /var/log/ksymoops.

>I don't see any reason why the internal profiler can't have an extended EIP
>range to catch module samples either. Perhaps I'm missing something. Perhaps
>no one cares enough ...

IMNSHO the profiler should use kallsyms. Then it gets every symbol
from the kernel and from all modules, instead of only being able to
report on exported symbols.

int kallsyms_address_to_symbol(
unsigned long address, /* Address to lookup */
const char **mod_name, /* Set to module name */
unsigned long *mod_start, /* Set to start address of module */
unsigned long *mod_end, /* Set to end address of module */
const char **sec_name, /* Set to section name */
unsigned long *sec_start, /* Set to start address of section */
unsigned long *sec_end, /* Set to end address of section */
const char **sym_name, /* Set to full symbol name */
unsigned long *sym_start, /* Set to start address of symbol */
unsigned long *sym_end /* Set to end address of symbol */
)

We have the technology.

2002-01-21 03:03:22

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Keith Owens wrote:
>
> On Sun, 20 Jan 2002 18:04:00 -0800,
> Andrew Morton <[email protected]> wrote:
> >ksymoops doesn't know what modules were loaded at the time
> >of the crash, and it doesn't know where they were loaded.
>
> /var/log/ksymoops. man insmod and look for ksymoops assistance.

Oh I have, extensively, when trying to get kgdb to play with
modules. I did what any sensible person would do: gave up and went
for a static link.

I'm not aware of anyone getting kgdb working fully with modules.

> >The `klogd -x' problem has been with us for *years* and
> >distributors still persist in turning it on.
>
> Tell me about it :(.
>
> >It assumes too much. Arjan has a kksymoops thingy which does the symbol
> >resolution at crash-time.
>
> It can only get symbols that are in /proc/ksyms, i.e. the exported
> symbols. Proper crash analysis needs the full symbol table.

Proper crash analysis needs to know the load address of each module
at the time of the crash. We should print them out at Oops time.

> >It also
> >handles the common case where the running vmlinux/System.map/etc no longer
> >exist.
>
> So does ksymoops, with reduced detail because it only has exported
> symbols.
>
> >I would prefer that all this become easier, simpler and more reliable.
> >We need a damn good reason for deprecating statically linked kernels
> >and certainly none has been presented yet.
>
> That is a different problem. Saying that modular kernels cause
> problems for debugging is not a good enough reason to deprecate modular
> kernels, all the problems have been solved.

They are patently *not* solved, because we continue to get a
stream of partially and competely useless oops reports. It
is my empirical observation that this problem is worse when
modules are involved. It is my personal experience that
modular code is harder to debug.

It is also my personal experience that the refusal of sbp2.o to
work correctly when statically linked was a complete pain in the
ass when trying to diagnose its deadlock and module refcount
problems. It was only when I worked out that I could link it
statically and then get it to play via rescan-scsi-bus.sh that
I was able to make reasonably-paced progress in finding the bugs.

We don't want this to happen to every damn driver in the kernel.

-

2002-01-21 03:43:42

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Sun, 20 Jan 2002 18:56:34 -0800,
Andrew Morton <[email protected]> wrote:
>I'm not aware of anyone getting kgdb working fully with modules.

kgdb has a script that tells gdb where each module is loaded. AFAIK it
uses add-symbol-file FILENAME -sSECTION ADDRESS, the __insmod entries
contain enough data to tell gdb what is going on.

>Proper crash analysis needs to know the load address of each module
>at the time of the crash. We should print them out at Oops time.

You need 13 bits of data per module. Where each of text, rodata, data
and bss sections were loaded, you cannot assume they are contiguous.
The length of those four sections. Where the module is on disk, you
cannot assume the object name is the same as the module name, insmod
-o. The timestamp and kernel version of the module when it was loaded,
to detect updates after the event. All of that is encoded in the
__insmod entries in /proc/ksyms, 5 lines per module.

>> That is a different problem. Saying that modular kernels cause
>> problems for debugging is not a good enough reason to deprecate modular
>> kernels, all the problems have been solved.
>
>They are patently *not* solved, because we continue to get a
>stream of partially and competely useless oops reports.

Rule 1. Users do not read documentation.
Rule 2. You can't do anything about rule 1.

If you want complete bug reports, write a script that forces the user
to submit all the required data. Why do I feel the "Linux should have
a bug reporting system" thread starting again?

2002-01-21 03:45:12

by Victor Yodaiken

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 21, 2002 at 01:25:18PM +1100, Keith Owens wrote:
> >I would prefer that all this become easier, simpler and more reliable.
> >We need a damn good reason for deprecating statically linked kernels
> >and certainly none has been presented yet.
>
> That is a different problem. Saying that modular kernels cause
> problems for debugging is not a good enough reason to deprecate modular
> kernels, all the problems have been solved.

Interesting definition of "solved".

2002-01-21 03:50:13

by Andreas Ferber

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 21, 2002 at 09:54:58AM +0900, Bruce Harada wrote:
>
> ...and how would you guarantee that this setting remains set, in the face of
> some nasty little cracker screwing around with /dev/kmem?

If the attacker gained the ability to play with /dev/kmem, he can
already load modules into the kernel, regardless if the kernel is
actually compiled with module support or not. You can find various
papers describing how to do it via google, and AFAIK some rootkits are
already using those techniques, so it's even "scriptkiddy-ready".

Face it, there is absolutely /no/ security gain in disabling module
support.

Andreas
--
Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
---------------------------------------------------------
+49 521 1365800 - [email protected] - http://www.devcon.net

2002-01-21 04:30:59

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

Keith Owens wrote:
>
> On Sun, 20 Jan 2002 18:56:34 -0800,
> Andrew Morton <[email protected]> wrote:
> >I'm not aware of anyone getting kgdb working fully with modules.
>
> kgdb has a script that tells gdb where each module is loaded. AFAIK it
> uses add-symbol-file FILENAME -sSECTION ADDRESS, the __insmod entries
> contain enough data to tell gdb what is going on.
>
> >Proper crash analysis needs to know the load address of each module
> >at the time of the crash. We should print them out at Oops time.
>
> You need 13 bits of data per module. Where each of text, rodata, data
> and bss sections were loaded, you cannot assume they are contiguous.
> The length of those four sections. Where the module is on disk, you
> cannot assume the object name is the same as the module name, insmod
> -o. The timestamp and kernel version of the module when it was loaded,
> to detect updates after the event. All of that is encoded in the
> __insmod entries in /proc/ksyms, 5 lines per module.

That sounds OK. Just print it first. If it hits the logs then
great. If it just scrolls off the screen then no loss.

> >> That is a different problem. Saying that modular kernels cause
> >> problems for debugging is not a good enough reason to deprecate modular
> >> kernels, all the problems have been solved.
> >
> >They are patently *not* solved, because we continue to get a
> >stream of partially and competely useless oops reports.
>
> Rule 1. Users do not read documentation.
> Rule 2. You can't do anything about rule 1.

Often the users know nothing about elf internals, kernel tools
and build processes and all of that. They just want to get their
box back up and reliable, and they consider that any work they do
in problem disgnosis is a favour for the developers (and I have no
problem with that attitude).

Often, we only get one shot at it.

> If you want complete bug reports, write a script that forces the user
> to submit all the required data.

It's in our interest to make it as easy as possible for the reporter.
Cut-and-paste is easy, so we should err on the side of overkill when
choosing which data to dump out.

> Why do I feel the "Linux should have
> a bug reporting system" thread starting again?

Yeah, let's not do that. (But I'd be on the "no" side :-))

-

2002-01-21 04:45:50

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, 21 Jan 2002 04:49:52 +0100
Andreas Ferber <[email protected]> wrote:

> On Mon, Jan 21, 2002 at 09:54:58AM +0900, Bruce Harada wrote:
> >
> > ...and how would you guarantee that this setting remains set, in the face
> > of some nasty little cracker screwing around with /dev/kmem?
>
> If the attacker gained the ability to play with /dev/kmem, he can
> already load modules into the kernel, regardless if the kernel is
> actually compiled with module support or not. You can find various
> papers describing how to do it via google, and AFAIK some rootkits are
> already using those techniques, so it's even "scriptkiddy-ready".

Er... that was exactly what I meant - perhaps you meant to reply to the parent
post?

2002-01-21 05:12:34

by Andreas Ferber

[permalink] [raw]

Subject: Re: Hardwired drivers are going away?

On Mon, Jan 21, 2002 at 01:45:16PM +0900, Bruce Harada wrote:
>
> Er... that was exactly what I meant - perhaps you meant to reply to the parent
> post?

Hmm, I understood your post as if you consider a simple flag disabling
module loading a step-back regarding security. I stand corrected.

Andreas
--
Andreas Ferber - dev/consulting GmbH - Bielefeld, FRG
---------------------------------------------------------
+49 521 1365800 - [email protected] - http://www.devcon.net

2002-01-21 09:34:37