I have finished testing parallel module loading. It looks like a small
kernel with a minimal initramfs and running several instances of insmod
or modprobe in parallel has the best complexity to performance ratio.
Testing shows that a megamodule is slightly slower than parallel
insmods, so it's not really an option anymore.
A monolithic kernel with parallelized initcalls is better - about 200 ms
faster than parallel insmods on my test system. However, it comes with a
fairly large set of changes:
* First, you need a 200-line patch in init/main.c (do_initcalls() and
friends)
* Then the built-in module dependencies must be calculated properly, eg
with a modified depmod, and added to the build process.
* Finally "soft" dependencies, i.e dependencies that are not implied by
symbol use, have to be formalized and moved into the kernel somehow.
Right now they're only defined in "install" commands in modprobe.conf.
So, what do you think, should I keep going? IMHO, the slower userspace
implementation is acceptable since it's so much simpler.
Thanks,
Andreas
------------------------
Here are the test results:
Setup:
HP Pavillion DV6300 laptop. AMD Turion TL-56 @ 1.8GHz CPU. In a
benchmark at Tomshardware, the CPU have scores similar to an Intel core
duo T2300.
http://www.tomshardware.com/reviews/dual-core-notebook-cpus-explored,1553-11.html
Results:
Configuration | T | stddev (n = 10)
------------------------------------+--------+--------
[1] Serial monolithic kernel | 3.08 s | 0.08
[2] Megamodule, serial initcalls | 3.26 s | 0.05
[3] Megamodule, parallel initcalls | 2.27 s | 0.01
[4] Parallel insmods w/ mutex patch | 2.20 s | 0.01 <- best choice
[5] Parallel monolithic kernel | 2.02 s | <- estimate only
T = Time from kernel startup until all module initcalls have
executed, i.e when init can mount the root filesystem.
[1] Monolithic 2.6.29-rc5 with fastboot cmdline option. No initramfs.
[2] 94 modules linked into one megamodule. The megamodule executed
module initcalls sequentially. All files were on a minimal
initramfs. The kernel had fastboot enabled.
[3] 94 modules inserted with custom insmod, in parallel.
(Dependencies were accounted for.) Minimal initramfs, fastboot.
[4] Like [2], but initcalls ran in parallel.
(Dependencies were accounted for.) Minimal initramfs, fastboot.
Rusty's module loader mutex-patch is applied.
[5] This is an estimation based on how much faster [3] would be if
load_module() took no time at all. T5 = T3 - (T2 - T1).
That is, I assume T2-T1 is the time spent in load_module().
Note:
By minimal initramfs, I mean one such that it, plus small kernel, is
roughly the same size as the equivalent monolithic kernel. The only
executable on it is init, written in C. There are no shell scripts,
busybox, progress bars or unused modules.
On Mon, Mar 2, 2009 at 15:32, Andreas Robinson <[email protected]> wrote:
> I have finished testing parallel module loading. It looks like a small
> kernel with a minimal initramfs and running several instances of insmod
> or modprobe in parallel has the best complexity to performance ratio.
>
> Testing shows that a megamodule is slightly slower than parallel
> insmods, so it's not really an option anymore.
That's what I expected, that we would need more changes, to make a
mega-module working, because we need parallel code execution here.
> A monolithic kernel with parallelized initcalls is better - about 200 ms
> faster than parallel insmods on my test system. However, it comes with a
> fairly large set of changes:
>
> * First, you need a 200-line patch in init/main.c (do_initcalls() and
> friends)
>
> * Then the built-in module dependencies must be calculated properly, eg
> with a modified depmod, and added to the build process.
>
> * Finally "soft" dependencies, i.e dependencies that are not implied by
> symbol use, have to be formalized and moved into the kernel somehow.
> Right now they're only defined in "install" commands in modprobe.conf.
>
> So, what do you think, should I keep going? IMHO, the slower userspace
> implementation is acceptable since it's so much simpler.
I think so too, and think we are fine using the current standard
tools. Also the recent modprobe changes make modprobe very fast by
using a binary index to resolve aliases. I think, we can leave the
stuff as it is.
Seems with the queued stop-machine()-removal patch, and possibly with
the minimized-mutex patch Rusty did, we are fine for now -- at least
with the current numbers, and from a distro's point of view.
Most of the "module load delay" distros see, are likely caused by I/O
loading the module from disk into the kernel, at least that's what my
profiles show. I did all the recent tests (which is a real world udev
coldplug) with a ramfs mount containing a copy of the module
directory, before running the module autoloader. I can link 40 modules
in 0.26 seconds into the kernel with the recent kernel patches.
Thanks for all your testing, it was really helpful, and your patches
have been the reason we've found the stop-machine problem. I'm glad
that we have real numbers now to point people to. It seems, that we
don't need to start fiddling around with hacks in userspace or
initramfs tools.
Thanks a lot,
Kay
> A monolithic kernel with parallelized initcalls is better - about 200
> ms faster than parallel insmods on my test system. However, it comes
> with a fairly large set of changes:
>
> * First, you need a 200-line patch in init/main.c (do_initcalls() and
> friends)
why?
We already have async function calls; and those speed up my boot (when
enabled) significantly, by doing much of the kernel/driver init in
parallel.
My server box boots the whole kernel (including all drivers; I build
verything in) in 0.56 seconds, and my net books do it in around 1.0
seconds.
>
> * Then the built-in module dependencies must be calculated properly,
> eg with a modified depmod, and added to the build process.
nope not if done right
> So, what do you think, should I keep going? IMHO, the slower userspace
> implementation is acceptable since it's so much simpler.
I would strongly suggest that you turn on the async function calls and
look at the boot graph of the resulting kernel boot... if you send
that to me I can also take a look and make suggestions....
On Mon, Mar 2, 2009 at 17:20, Arjan van de Ven <[email protected]> wrote:
>> A monolithic kernel with parallelized initcalls is better - about 200
>> ms faster than parallel insmods on my test system. However, it comes
>> with a fairly large set of changes:
>>
>> * First, you need a 200-line patch in init/main.c (do_initcalls() and
>> friends)
>
> why?
> We already have async function calls; and those speed up my boot (when
> enabled) significantly, by doing much of the kernel/driver init in
> parallel.
>
> My server box boots the whole kernel (including all drivers; I build
> verything in) in 0.56 seconds, and my net books do it in around 1.0
> seconds.
>
>>
>> * Then the built-in module dependencies must be calculated properly,
>> eg with a modified depmod, and added to the build process.
>
> nope not if done right
>
>> So, what do you think, should I keep going? IMHO, the slower userspace
>> implementation is acceptable since it's so much simpler.
>
> I would strongly suggest that you turn on the async function calls and
> look at the boot graph of the resulting kernel boot... if you send
> that to me I can also take a look and make suggestions....
The "fastboot" kernel commandline option was used, as mentioned in the
mail. Is there anything else?
Thanks,
Kay
On Mon, 2 Mar 2009 17:29:57 +0100
Kay Sievers <[email protected]> wrote:
> On Mon, Mar 2, 2009 at 17:20, Arjan van de Ven <[email protected]>
> wrote:
> >> A monolithic kernel with parallelized initcalls is better - about
> >> 200 ms faster than parallel insmods on my test system. However, it
> >> comes with a fairly large set of changes:
> >>
> >> * First, you need a 200-line patch in init/main.c (do_initcalls()
> >> and friends)
> >
> > why?
> > We already have async function calls; and those speed up my boot
> > (when enabled) significantly, by doing much of the kernel/driver
> > init in parallel.
> >
> > My server box boots the whole kernel (including all drivers; I build
> > verything in) in 0.56 seconds, and my net books do it in around 1.0
> > seconds.
> >
> >>
> >> * Then the built-in module dependencies must be calculated
> >> properly, eg with a modified depmod, and added to the build
> >> process.
> >
> > nope not if done right
> >
> >> So, what do you think, should I keep going? IMHO, the slower
> >> userspace implementation is acceptable since it's so much simpler.
> >
> > I would strongly suggest that you turn on the async function calls
> > and look at the boot graph of the resulting kernel boot... if you
> > send that to me I can also take a look and make suggestions....
>
> The "fastboot" kernel commandline option was used, as mentioned in the
> mail. Is there anything else?
there is some sata level enabling needed depending on the system;
I'd love to see the bootgraph (scripts/bootgraph.pl) for the boot;
that shows exactly what happens when and for how long.
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
> there is some sata level enabling needed depending on the system;
> I'd love to see the bootgraph (scripts/bootgraph.pl) for the boot;
> that shows exactly what happens when and for how long.
Hmm, I was going to send you a reply tonight, but it seems I've run out
of time. I'll get back to you as soon as possible, probably tomorrow.
Cheers,
Andreas
On Mon, 2009-03-02 at 10:27 -0800, Arjan van de Ven wrote:
> On Mon, 2 Mar 2009 17:29:57 +0100
> Kay Sievers <[email protected]> wrote:
>
> > On Mon, Mar 2, 2009 at 17:20, Arjan van de Ven <[email protected]>
> > wrote:
[...]
> > > I would strongly suggest that you turn on the async function calls
> > > and look at the boot graph of the resulting kernel boot... if you
> > > send that to me I can also take a look and make suggestions....
> >
> > The "fastboot" kernel commandline option was used, as mentioned in the
> > mail. Is there anything else?
>
> there is some sata level enabling needed depending on the system;
> I'd love to see the bootgraph (scripts/bootgraph.pl) for the boot;
> that shows exactly what happens when and for how long.
>
Well, this was certainly an eye opener. I can only say thanks for
pointing out bootgraph!
You asked for one graph, but Rusty suggested that I compare insmod and
modprobe (though in another context), so I did more than one.
The (somewhat messy) combined result from booting four initramfs images
with slightly different setups is attached.
All of them install the 94 modules needed on the test system and all use
2.6.29-rc6 configured like a Ubuntu 8.10 generic kernel. The module
mutex minimizing patch is added, but not the stop_machine patch. (The
merge failed due to my own patches. I need to have a look at it again.)
Fastboot is enabled.
bootgraph.pl was tweaked to, among other things, not stop until when a
dummy module was inserted. This happened after udev had settled and just
before chaining to the file system on the hard drive.
1: (top row, light blue). This installed modules using udev 124 and
modprobe 3.3pre11 that are shipped with Ubuntu 8.10.
2: (second row, light green). This installed modules using the latest
versions of udev (139) and modprobe (3.7pre6) from the git repos.
3: (third row, light yellow). This used a multithreaded insmod. It
employs a very simple method to spawn child processes, so it looks
different from the others and had to be compressed vertically to fit.
(And four processes hang until all initcalls are done and the parent
process dies, so it's kind of buggy too.)
4: (bottom row, pink). This is the megamodule, only added out of
curiosity. It really is multithreaded, but it doesn't use
do_one_initcall() so you can't see anything.
On Wed, 04 Mar 2009 19:47:10 +0100
Andreas Robinson <[email protected]> wrote:
the only thing I can say (the graphs are a bit complex to read) is that
you are loading a LOT of very expensive modules that bring you no
functionality.
Also I'd love to have seen the scenario where the modules were just
build into the kernel; that gives usually a more reliable indication of
what is going on...
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Mon, Mar 02, 2009 at 08:20:03AM -0800, Arjan van de Ven wrote:
>
> I would strongly suggest that you turn on the async function calls and
Are async calls only turned on if you boot with the "fastboot" commandline
parameter? If so could a mention of it be added to
kernel-parameters as I don't see anything about it when I glanced at
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/kernel-parameters.txt;h=54f21a5c262b02ef6cb4718f72b8357693197cf3;hb=HEAD
...
--
Sitsofe | http://sucs.org/~sits/
On Fri, 6 Mar 2009 07:05:23 +0000
Sitsofe Wheeler <[email protected]> wrote:
> On Mon, Mar 02, 2009 at 08:20:03AM -0800, Arjan van de Ven wrote:
> >
> > I would strongly suggest that you turn on the async function calls
> > and
>
> Are async calls only turned on if you boot with the "fastboot"
> commandline parameter? If so could a mention of it be added to
> kernel-parameters as I don't see anything about it when I glanced at
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/kernel-parameters.txt;h=54f21a5c262b02ef6cb4718f72b8357693197cf3;hb=HEAD
> ...
>
that's because it's going away again in 2.6.30....
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Thu, 2009-03-05 at 16:18 -0800, Arjan van de Ven wrote:
> On Wed, 04 Mar 2009 19:47:10 +0100
> Andreas Robinson <[email protected]> wrote:
>
> the only thing I can say (the graphs are a bit complex to read) is that
> you are loading a LOT of very expensive modules that bring you no
> functionality.
Well, yes. It is the generic Ubuntu kernel booting in all its bloaty
glory. :) But this is what most users are stuck with. The distro vendors
won't change their ways because they want their stuff to run on
anything.
So I'm not trying to build a custom kernel just for me. I want to find
ways to have any distro optimize itself to boot in 5-10 seconds on any
computer it has been installed on, without manual configuration.
> Also I'd love to have seen the scenario where the modules were just
> build into the kernel; that gives usually a more reliable indication of
> what is going on...
Alright, I'm on it.
Cheers,
Andreas
On Fri, 06 Mar 2009 16:15:43 +0100
Andreas Robinson <[email protected]> wrote:
> On Thu, 2009-03-05 at 16:18 -0800, Arjan van de Ven wrote:
> > On Wed, 04 Mar 2009 19:47:10 +0100
> > Andreas Robinson <[email protected]> wrote:
> >
> > the only thing I can say (the graphs are a bit complex to read) is
> > that you are loading a LOT of very expensive modules that bring you
> > no functionality.
>
> Well, yes. It is the generic Ubuntu kernel booting in all its bloaty
> glory. :) But this is what most users are stuck with. The distro
> vendors won't change their ways because they want their stuff to run
> on anything.
the problem is that you are also having things that just don't make
sense; even for the generalized case...
but I guess that is for Ubuntu to resolve. I don't see the machines I
run (with Fedora) exhibit this same issue; while there are a bunch of
modules, none of the ones that are expensive-and-useless in your charts
are there.
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Fri, 2009-03-06 at 07:45 -0800, Arjan van de Ven wrote:
> On Fri, 06 Mar 2009 16:15:43 +0100
> Andreas Robinson <[email protected]> wrote:
>
> > On Thu, 2009-03-05 at 16:18 -0800, Arjan van de Ven wrote:
> > > On Wed, 04 Mar 2009 19:47:10 +0100
> > > Andreas Robinson <[email protected]> wrote:
> > >
> > > the only thing I can say (the graphs are a bit complex to read) is
> > > that you are loading a LOT of very expensive modules that bring you
> > > no functionality.
> >
> > Well, yes. It is the generic Ubuntu kernel booting in all its bloaty
> > glory. :) But this is what most users are stuck with. The distro
> > vendors won't change their ways because they want their stuff to run
> > on anything.
>
> the problem is that you are also having things that just don't make
> sense; even for the generalized case...
>
> but I guess that is for Ubuntu to resolve. I don't see the machines I
> run (with Fedora) exhibit this same issue; while there are a bunch of
> modules, none of the ones that are expensive-and-useless in your charts
> are there.
I think they will. Fast boot is one of the "essential" goals for Jaunty
and Karmic.
Anyway, attached is a bootchart for a cleaned up kernel with the
important (AFAIK) drivers built in.
>From this, I've found some tasks that I can set myself to. They would
obviously be the topic of other threads:
* Make the nforce2 ethernet driver (init_nic) initialize itself
asynchronously, if possible.
* New config option: Separate initramfs decompression and
file-system calls. Decompress as early as possible in a
separate thread.
* Don't unpack initramfs twice. If CONFIG_BLK_DEV_RAM is
set, the kernel decompresses the ramdisk just to see if
it's a cpio archive or not, and then throws away the result.
* See what can be done about pci_init(), if anything.
If there's anything else regarding fastboot you think would make more
sense to work on first, please let me know.
Thanks,
Andreas
On Sun, 08 Mar 2009 11:47:17 +0100
Andreas Robinson <[email protected]> wrote:
> > but I guess that is for Ubuntu to resolve. I don't see the machines
> > I run (with Fedora) exhibit this same issue; while there are a
> > bunch of modules, none of the ones that are expensive-and-useless
> > in your charts are there.
>
> I think they will. Fast boot is one of the "essential" goals for
> Jaunty and Karmic.
>
> Anyway, attached is a bootchart for a cleaned up kernel with the
> important (AFAIK) drivers built in.
thanks; very useful overview!
>
> >From this, I've found some tasks that I can set myself to. They would
> obviously be the topic of other threads:
>
> * Make the nforce2 ethernet driver (init_nic) initialize itself
> asynchronously, if possible.
I looked at doing this generically and it is really hard. BUT..
one of the options is to move work from the init to the "ifconfig up"
code...
In addition, for 2.6.30, I have a patch pending to move sata init
before the network init, which can help hide it some
>
> * New config option: Separate initramfs decompression and
> file-system calls. Decompress as early as possible in a
> separate thread.
interesting idea!
>
> * Don't unpack initramfs twice. If CONFIG_BLK_DEV_RAM is
> set, the kernel decompresses the ramdisk just to see if
> it's a cpio archive or not, and then throws away the result.
I actually have a patch for it stuck away somewhere; I thought it was
sent upstream but apparently it was not. I'll dig it out and dust it
off.
Note: Having a smaller initrd (like Fedora does, they make the initrd
only have the modules the specific machine needs) will save a bunch of
time... what isn't there you don't have to decompress either.
>
> * See what can be done about pci_init(), if anything.
>
> If there's anything else regarding fastboot you think would make more
> sense to work on first, please let me know.
One thing to mention is that fastinit for sata... I only did AHCI
(because all my machines have that). Your machine looks like it has
something different than AHCI, so maybe the sata controller can use
some work ;-)
(a sata driver needs to opt into parallel scanning via a flag)
SATA init tends to be one of the things that are fix time per drive;
and so working on that first to at least ONLY get that is worth it,
it's a huge chunk of time (more than half the total time in your
bootchart)
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
On Sun, 2009-03-08 at 09:01 -0700, Arjan van de Ven wrote:
> One thing to mention is that fastinit for sata... I only did AHCI
> (because all my machines have that). Your machine looks like it has
> something different than AHCI, so maybe the sata controller can use
> some work ;-)
> (a sata driver needs to opt into parallel scanning via a flag)
>
> SATA init tends to be one of the things that are fix time per drive;
> and so working on that first to at least ONLY get that is worth it,
> it's a huge chunk of time (more than half the total time in your
> bootchart)
Something like the patch below, mabye? :-)
Before: [ 3.056308] Freeing unused kernel memory: 364k freed
After: [ 2.411417] Freeing unused kernel memory: 364k freed
The chipset is nforce2 (drivers/ata/sata_nv.c). I couldn't find
anything like the SSS flag documented in the code, and register
specs from nvidia are as rare as hen's teeth... So I only added
the module parameter to force parallel scans.
I named it parallel_scan instead of ignore_sss though, since
there might not be an SSS-flag to ignore.
/Andreas
>From a5b090856173782308c8f61de84085be9a9b73ba Mon Sep 17 00:00:00 2001
From: Andreas Robinson <[email protected]>
Date: Sun, 8 Mar 2009 20:44:16 +0100
Subject: [PATCH] sata_nv: add a module parameter to enable async scanning
This parameter (parallel_scan) forces the OS to scan for
disks asynchronously/in parallel, to reduce boot time.
It might cause problems (brown-outs, blown fuses) in
multi-drive systems, if the PSU is unable to handle several
drives spinning up simultaneously.
It ought to be safe in single-drive systems.
Signed-off-by: Andreas Robinson <[email protected]>
---
drivers/ata/sata_nv.c | 6 ++++++
1 files changed, 6 insertions(+), 0 deletions(-)
diff --git a/drivers/ata/sata_nv.c b/drivers/ata/sata_nv.c
index 55a8eed..e8aba2f 100644
--- a/drivers/ata/sata_nv.c
+++ b/drivers/ata/sata_nv.c
@@ -579,6 +579,7 @@ MODULE_VERSION(DRV_VERSION);
static int adma_enabled;
static int swncq_enabled = 1;
+static int parallel_scan = 0;
static void nv_adma_register_mode(struct ata_port *ap)
{
@@ -2427,6 +2428,9 @@ static int nv_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
} else if (type == SWNCQ)
nv_swncq_host_init(host);
+ if (parallel_scan)
+ host->flags |= ATA_HOST_PARALLEL_SCAN;
+
pci_set_master(pdev);
return ata_host_activate(host, pdev->irq, ipriv->irq_handler,
IRQF_SHARED, ipriv->sht);
@@ -2526,4 +2530,6 @@ module_param_named(adma, adma_enabled, bool, 0444);
MODULE_PARM_DESC(adma, "Enable use of ADMA (Default: true)");
module_param_named(swncq, swncq_enabled, bool, 0444);
MODULE_PARM_DESC(swncq, "Enable use of SWNCQ (Default: true)");
+module_param_named(parallel_scan, parallel_scan, bool, 0444);
+MODULE_PARM_DESC(parallel_scan, "Force parallel host scan (Default: false)");
--
1.5.6.3
On Sun, Mar 8, 2009 at 4:01 PM, Arjan van de Ven <[email protected]> wrote:
>> * Don't unpack initramfs twice. If CONFIG_BLK_DEV_RAM is
>> ? set, the kernel decompresses the ramdisk just to see if
>> ? it's a cpio archive or not, and then throws away the result.
>
> I actually have a patch for it stuck away somewhere; I thought it was
> sent upstream but apparently it was not. I'll dig it out and dust it
> off.
Please do. It's a big win in relative terms compared to the rest of
the boot for small systems.