-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
2.6.25-rc4 invokes the oom-killer repeatedly when attempting to boot,
eventually panicing with "Out of memory and no killable processes."
This happens since at least 2.6.25-rc3, but 2.6.24 boots just fine,
The system is a Dell Inspiron E1705 running Fedora 8 (x86_64).
My .config is at http://tuxrocks.com/tmp/config-2.6.25-rc4, and a syslog
of the system up until the point where it oom-killed syslog (just before
the panic) is at http://tuxrocks.com/tmp/oom-2.6.25-rc4.txt
Frank
- --
Frank Sorenson - KD7TZK
Linux Systems Engineer, DSS Engineering, UBS AG
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFH0Ck9aI0dwg4A47wRAiE2AJ9fw0tRLtdnDrG7TdaR0721wj69owCgitR/
4XphvHdePCivYnu+gXFAgG8=
=shGl
-----END PGP SIGNATURE-----
* Frank Sorenson <[email protected]> wrote:
> 2.6.25-rc4 invokes the oom-killer repeatedly when attempting to boot,
> eventually panicing with "Out of memory and no killable processes."
> This happens since at least 2.6.25-rc3, but 2.6.24 boots just fine,
>
> The system is a Dell Inspiron E1705 running Fedora 8 (x86_64).
>
> My .config is at http://tuxrocks.com/tmp/config-2.6.25-rc4, and a
> syslog of the system up until the point where it oom-killed syslog
> (just before the panic) is at
> http://tuxrocks.com/tmp/oom-2.6.25-rc4.txt
i've picked up your .config and enabled a few drivers in it to make it
boot on a testsystem of mine and it doesnt OOM:
20:47:17 up 10 min, 1 user, load average: 0.01, 0.07, 0.03
total used free shared buffers cached
Mem: 1025768 258544 767224 0 12956 169556
-/+ buffers/cache: 76032 949736
Swap: 3911816 0 3911816
(config and bootlog from my box attached.)
So it's probably not a .config dependent generic kernel problem, but
probably something specific to your hardware.
Since the first oom happens about 9 minutes into the bootup:
[ 569.755853] sh invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
do you have any chance to log in and capture MM statistics? The
following script will capture a bunch of statistics:
http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
and takes less than a minute to run - should be enough time in theory.
If it's possible to run it then send us the output file it produces.
Ingo
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Ingo Molnar wrote:
> * Frank Sorenson <[email protected]> wrote:
>
>> 2.6.25-rc4 invokes the oom-killer repeatedly when attempting to boot,
>> eventually panicing with "Out of memory and no killable processes."
>> This happens since at least 2.6.25-rc3, but 2.6.24 boots just fine,
>>
>> The system is a Dell Inspiron E1705 running Fedora 8 (x86_64).
>>
>> My .config is at http://tuxrocks.com/tmp/config-2.6.25-rc4, and a
>> syslog of the system up until the point where it oom-killed syslog
>> (just before the panic) is at
>> http://tuxrocks.com/tmp/oom-2.6.25-rc4.txt
>
> i've picked up your .config and enabled a few drivers in it to make it
> boot on a testsystem of mine and it doesnt OOM:
>
> 20:47:17 up 10 min, 1 user, load average: 0.01, 0.07, 0.03
> total used free shared buffers cached
> Mem: 1025768 258544 767224 0 12956 169556
> -/+ buffers/cache: 76032 949736
> Swap: 3911816 0 3911816
>
> (config and bootlog from my box attached.)
>
> So it's probably not a .config dependent generic kernel problem, but
> probably something specific to your hardware.
>
> Since the first oom happens about 9 minutes into the bootup:
>
> [ 569.755853] sh invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
>
> do you have any chance to log in and capture MM statistics? The
> following script will capture a bunch of statistics:
>
> http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh
>
> and takes less than a minute to run - should be enough time in theory.
> If it's possible to run it then send us the output file it produces.
>
> Ingo
Thank you for the help, and for the time.
I did some additional debugging, and I believe you're correct about it
being specific to my system. The system seems to run fine until some
time during the boot. I booted with "init=/bin/sh" (that's how the
system stayed up for 9 minutes), then it died when I tried starting
things up. I've further narrowed the OOM down to udev (though it's not
entirely udev's fault, since 2.6.24 runs fine).
I ran your debug info tool before killing the box by running
/sbin/start_udev. The output of the tool is at
http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24
Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
causes udev (or something it calls) to behave very badly.
I'll keep looking further into the cause. Thanks again for the help.
Frank
- --
Frank Sorenson - KD7TZK
Linux Systems Engineer, DSS Engineering, UBS AG
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFH0ZcXaI0dwg4A47wRAoy7AJ9ILIlACjitvOpNghRNxmOgiygk1QCfb3Oi
8Drhxc4Tvu0K+1KD0U6XUOE=
=SmQj
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Frank Sorenson wrote:
> I did some additional debugging, and I believe you're correct about it
> being specific to my system. The system seems to run fine until some
> time during the boot. I booted with "init=/bin/sh" (that's how the
> system stayed up for 9 minutes), then it died when I tried starting
> things up. I've further narrowed the OOM down to udev (though it's not
> entirely udev's fault, since 2.6.24 runs fine).
>
> I ran your debug info tool before killing the box by running
> /sbin/start_udev. The output of the tool is at
> http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24
>
> Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
> causes udev (or something it calls) to behave very badly.
Found it. The culprit is 8f47f0b688bba7642dac4e979896e4692177670b
dcdbas: add DMI-based module autloading
DMI autoload dcdbas on all Dell systems.
This looks for BIOS Vendor or System Vendor == Dell, so this should
work for systems both Dell-branded and those Dell builds but brands
for others. It causes udev to load the dcdbas module at startup,
which is used by tools called by HAL for wireless control and
backlight control, among other uses.
What actually happens is that when udev loads the dcdbas module at
startup, modprobe apparently calls "modprobe dcdbas" itself, repeating
until the system runs out of resources (in this case, it OOMs).
# ps axf
...
506 ? S 0:00 /bin/bash /sbin/start_udev
590 ? S 0:00 \_ /sbin/udevsettle
533 ? S<s 0:00 /sbin/udevd -d
629 ? S< 0:00 \_ /sbin/udevd -d
630 ? S< 0:00 | \_ /sbin/modprobe
dmi:bvnDellInc.:bvrA08:bd04/02/2007:svnDellInc.:pnMP061:pvr:rvnDellInc.:rn0YD479:rvr:cvnDellInc.:ct8:cvr:
949 ? S< 0:00 | \_ /sbin/modprobe dcdbas
950 ? S< 0:00 | \_ /sbin/modprobe dcdbas
951 ? S< 0:00 | \_ /sbin/modprobe dcdbas
953 ? S< 0:00 | \_ /sbin/modprobe dcdbas
955 ? S< 0:00 | \_ /sbin/modprobe dcdbas
958 ? S< 0:00 | \_
/sbin/modprobe dcdbas
...repeat...
When the system crashed, there were at least 11,600 instances of
"/sbin/modprobe dcdbas", each calling the next.
Reverting 8f47f0b lets the system boot up just fine again. Note that a
manual "modprobe dcdbas" also causes this recursive behavior, it's just
not forced on the system by udev.
So dcdbas is a regression from 2.6.24, as well as being broken in other
ways.
Frank
- --
Frank Sorenson - KD7TZK
Linux Systems Engineer, DSS Engineering, UBS AG
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFH0jt7aI0dwg4A47wRAuXtAJ9kAXuT3hRCw8KJqs1e4SIwzXDYFACgqR8Q
gwV6NcjPq5x6Nt16V7Z/eVc=
=0rIk
-----END PGP SIGNATURE-----
* Frank Sorenson <[email protected]> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Frank Sorenson wrote:
> > I did some additional debugging, and I believe you're correct about it
> > being specific to my system. The system seems to run fine until some
> > time during the boot. I booted with "init=/bin/sh" (that's how the
> > system stayed up for 9 minutes), then it died when I tried starting
> > things up. I've further narrowed the OOM down to udev (though it's not
> > entirely udev's fault, since 2.6.24 runs fine).
> >
> > I ran your debug info tool before killing the box by running
> > /sbin/start_udev. The output of the tool is at
> > http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24
> >
> > Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
> > causes udev (or something it calls) to behave very badly.
>
> Found it. The culprit is 8f47f0b688bba7642dac4e979896e4692177670b
> dcdbas: add DMI-based module autloading
>
> DMI autoload dcdbas on all Dell systems.
>
> This looks for BIOS Vendor or System Vendor == Dell, so this should
> work for systems both Dell-branded and those Dell builds but brands
> for others. It causes udev to load the dcdbas module at startup,
> which is used by tools called by HAL for wireless control and
> backlight control, among other uses.
>
> What actually happens is that when udev loads the dcdbas module at
> startup, modprobe apparently calls "modprobe dcdbas" itself, repeating
> until the system runs out of resources (in this case, it OOMs).
nice work! I've attached the revert below against latest -git - just in
case no-one can think of an obvious fix to this bug.
Ingo
--------------------->
Subject: revert "dcdbas: add DMI-based module autloading"
From: Ingo Molnar <[email protected]>
Date: Sat Mar 08 09:09:16 CET 2008
Frank Sorenson reported that 2.6.25-rc OOMs on his box and
tracked it down to commit 8f47f0b688bba7642dac4e979896e4692177670b,
"dcdbas: add DMI-based module autloading". Frank says:
> What actually happens is that when udev loads the dcdbas module at
> startup, modprobe apparently calls "modprobe dcdbas" itself, repeating
> until the system runs out of resources (in this case, it OOMs).
revert the commit.
Bisected-by: Frank Sorenson <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
drivers/firmware/dcdbas.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
Index: linux/drivers/firmware/dcdbas.c
===================================================================
--- linux.orig/drivers/firmware/dcdbas.c
+++ linux/drivers/firmware/dcdbas.c
@@ -658,5 +658,4 @@ MODULE_DESCRIPTION(DRIVER_DESCRIPTION "
MODULE_VERSION(DRIVER_VERSION);
MODULE_AUTHOR("Dell Inc.");
MODULE_LICENSE("GPL");
-/* Any System or BIOS claiming to be by Dell */
-MODULE_ALIAS("dmi:*:[bs]vnD[Ee][Ll][Ll]*:*");
+
On Sat, Mar 08, 2008 at 01:08:46AM -0600, Frank Sorenson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Frank Sorenson wrote:
> > I did some additional debugging, and I believe you're correct about it
> > being specific to my system. The system seems to run fine until some
> > time during the boot. I booted with "init=/bin/sh" (that's how the
> > system stayed up for 9 minutes), then it died when I tried starting
> > things up. I've further narrowed the OOM down to udev (though it's not
> > entirely udev's fault, since 2.6.24 runs fine).
> >
> > I ran your debug info tool before killing the box by running
> > /sbin/start_udev. The output of the tool is at
> > http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24
> >
> > Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
> > causes udev (or something it calls) to behave very badly.
>
> Found it. The culprit is 8f47f0b688bba7642dac4e979896e4692177670b
> dcdbas: add DMI-based module autloading
>
> DMI autoload dcdbas on all Dell systems.
>
> This looks for BIOS Vendor or System Vendor == Dell, so this should
> work for systems both Dell-branded and those Dell builds but brands
> for others. It causes udev to load the dcdbas module at startup,
> which is used by tools called by HAL for wireless control and
> backlight control, among other uses.
>
> What actually happens is that when udev loads the dcdbas module at
> startup, modprobe apparently calls "modprobe dcdbas" itself, repeating
> until the system runs out of resources (in this case, it OOMs).
>
> # ps axf
> ...
> 506 ? S 0:00 /bin/bash /sbin/start_udev
> 590 ? S 0:00 \_ /sbin/udevsettle
> 533 ? S<s 0:00 /sbin/udevd -d
> 629 ? S< 0:00 \_ /sbin/udevd -d
> 630 ? S< 0:00 | \_ /sbin/modprobe
> dmi:bvnDellInc.:bvrA08:bd04/02/2007:svnDellInc.:pnMP061:pvr:rvnDellInc.:rn0YD479:rvr:cvnDellInc.:ct8:cvr:
> 949 ? S< 0:00 | \_ /sbin/modprobe dcdbas
> 950 ? S< 0:00 | \_ /sbin/modprobe dcdbas
> 951 ? S< 0:00 | \_ /sbin/modprobe dcdbas
> 953 ? S< 0:00 | \_ /sbin/modprobe dcdbas
> 955 ? S< 0:00 | \_ /sbin/modprobe dcdbas
> 958 ? S< 0:00 | \_
> /sbin/modprobe dcdbas
> ...repeat...
>
> When the system crashed, there were at least 11,600 instances of
> "/sbin/modprobe dcdbas", each calling the next.
>
> Reverting 8f47f0b lets the system boot up just fine again. Note that a
> manual "modprobe dcdbas" also causes this recursive behavior, it's just
> not forced on the system by udev.
>
> So dcdbas is a regression from 2.6.24, as well as being broken in other
> ways.
>
> Frank
> - --
> Frank Sorenson - KD7TZK
> Linux Systems Engineer, DSS Engineering, UBS AG
> [email protected]
Frank, what version of module-init-tools do you have? This has been
in use in Fedora 8 for a few months, and this is the first failure
report I've seen.
I'm fine with reverting the patch for now, but really do want to get
to root cause, because module autoloading is a really good idea, and
it would be a shame if we couldn't keep that feature enabled because
some systems have incompatible module-init-tools, and the kernel can't
know that... (Perhaps udev could know and not invoke modprobe in
those instances?)
-Matt
--
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & http://www.dell.com/linux
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Matt Domsch wrote:
> On Sat, Mar 08, 2008 at 01:08:46AM -0600, Frank Sorenson wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Frank Sorenson wrote:
>>> I did some additional debugging, and I believe you're correct about it
>>> being specific to my system. The system seems to run fine until some
>>> time during the boot. I booted with "init=/bin/sh" (that's how the
>>> system stayed up for 9 minutes), then it died when I tried starting
>>> things up. I've further narrowed the OOM down to udev (though it's not
>>> entirely udev's fault, since 2.6.24 runs fine).
>>>
>>> I ran your debug info tool before killing the box by running
>>> /sbin/start_udev. The output of the tool is at
>>> http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24
>>>
>>> Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
>>> causes udev (or something it calls) to behave very badly.
>> Found it. The culprit is 8f47f0b688bba7642dac4e979896e4692177670b
>> dcdbas: add DMI-based module autloading
>>
>> DMI autoload dcdbas on all Dell systems.
>>
>> This looks for BIOS Vendor or System Vendor == Dell, so this should
>> work for systems both Dell-branded and those Dell builds but brands
>> for others. It causes udev to load the dcdbas module at startup,
>> which is used by tools called by HAL for wireless control and
>> backlight control, among other uses.
>>
>> What actually happens is that when udev loads the dcdbas module at
>> startup, modprobe apparently calls "modprobe dcdbas" itself, repeating
>> until the system runs out of resources (in this case, it OOMs).
>>
>> # ps axf
>> ...
>> 506 ? S 0:00 /bin/bash /sbin/start_udev
>> 590 ? S 0:00 \_ /sbin/udevsettle
>> 533 ? S<s 0:00 /sbin/udevd -d
>> 629 ? S< 0:00 \_ /sbin/udevd -d
>> 630 ? S< 0:00 | \_ /sbin/modprobe
>> dmi:bvnDellInc.:bvrA08:bd04/02/2007:svnDellInc.:pnMP061:pvr:rvnDellInc.:rn0YD479:rvr:cvnDellInc.:ct8:cvr:
>> 949 ? S< 0:00 | \_ /sbin/modprobe dcdbas
>> 950 ? S< 0:00 | \_ /sbin/modprobe dcdbas
>> 951 ? S< 0:00 | \_ /sbin/modprobe dcdbas
>> 953 ? S< 0:00 | \_ /sbin/modprobe dcdbas
>> 955 ? S< 0:00 | \_ /sbin/modprobe dcdbas
>> 958 ? S< 0:00 | \_
>> /sbin/modprobe dcdbas
>> ...repeat...
>>
>> When the system crashed, there were at least 11,600 instances of
>> "/sbin/modprobe dcdbas", each calling the next.
>>
>> Reverting 8f47f0b lets the system boot up just fine again. Note that a
>> manual "modprobe dcdbas" also causes this recursive behavior, it's just
>> not forced on the system by udev.
>>
>> So dcdbas is a regression from 2.6.24, as well as being broken in other
>> ways.
>>
>> Frank
>> - --
>> Frank Sorenson - KD7TZK
>> Linux Systems Engineer, DSS Engineering, UBS AG
>> [email protected]
>
>
> Frank, what version of module-init-tools do you have? This has been
> in use in Fedora 8 for a few months, and this is the first failure
> report I've seen.
>
> I'm fine with reverting the patch for now, but really do want to get
> to root cause, because module autoloading is a really good idea, and
> it would be a shame if we couldn't keep that feature enabled because
> some systems have incompatible module-init-tools, and the kernel can't
> know that... (Perhaps udev could know and not invoke modprobe in
> those instances?)
>
> -Matt
It's module-init-tools-3.4-2.fc8.x86_64 (most recent Fedora rpm available).
Frank
- --
Frank Sorenson - KD7TZK
Linux Systems Engineer, DSS Engineering, UBS AG
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFH0pyoaI0dwg4A47wRAq9rAKCVbg5ngSyHVORpLAcD4WY4vNMQlQCdGtr1
9CiHmom5Vopsqukc8e+D1RU=
=GPMU
-----END PGP SIGNATURE-----
On Sat, 2008-03-08 at 09:22 +0100, Ingo Molnar wrote:
> * Frank Sorenson <[email protected]> wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Frank Sorenson wrote:
> > > I did some additional debugging, and I believe you're correct about it
> > > being specific to my system. The system seems to run fine until some
> > > time during the boot. I booted with "init=/bin/sh" (that's how the
> > > system stayed up for 9 minutes), then it died when I tried starting
> > > things up. I've further narrowed the OOM down to udev (though it's not
> > > entirely udev's fault, since 2.6.24 runs fine).
> > >
> > > I ran your debug info tool before killing the box by running
> > > /sbin/start_udev. The output of the tool is at
> > > http://tuxrocks.com/tmp/cfs-debug-info-2008.03.06-14.11.24
> > >
> > > Something is apparently happening between 2.6.24 and 2.6.25-rc[34] which
> > > causes udev (or something it calls) to behave very badly.
> >
> > Found it. The culprit is 8f47f0b688bba7642dac4e979896e4692177670b
> > dcdbas: add DMI-based module autloading
> >
> > DMI autoload dcdbas on all Dell systems.
> >
> > This looks for BIOS Vendor or System Vendor == Dell, so this should
> > work for systems both Dell-branded and those Dell builds but brands
> > for others. It causes udev to load the dcdbas module at startup,
> > which is used by tools called by HAL for wireless control and
> > backlight control, among other uses.
> >
> > What actually happens is that when udev loads the dcdbas module at
> > startup, modprobe apparently calls "modprobe dcdbas" itself, repeating
> > until the system runs out of resources (in this case, it OOMs).
>
> nice work! I've attached the revert below against latest -git - just in
> case no-one can think of an obvious fix to this bug.
Frank, can you grep for 'dcdbas' in the modprobe config files:
modprobe -c | grep dcdbas
?
I wonder what's going on here, that modprobe calls itself.
Thanks,
Kay
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kay Sievers wrote:
> Frank, can you grep for 'dcdbas' in the modprobe config files:
> modprobe -c | grep dcdbas
> ?
>
> I wonder what's going on here, that modprobe calls itself.
>
> Thanks,
> Kay
Aha. This is indeed where the problem was. A line in the modprobe
config files was supposed to cause dcdbas to load, but was instead
causing modprobe to call itself repeatedly for dcdbas. I don't know why
an incorrect line was there, but removing it from the config allows
dcdbas to load without problem manually, and the autoload patch loads it
automatically.
Sincere apologies to everyone for causing the fire drill on a false
alarm. Since fixing my config makes things work again, and nobody else
sees the problem, the autoload patch should stay.
Thanks for the help tracking down the issue.
Frank (off to hide in the corner)
- --
Frank Sorenson - KD7TZK
Linux Systems Engineer, DSS Engineering, UBS AG
[email protected]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFH0uGTaI0dwg4A47wRApD/AKDsYtoatp/mJShgdHVDj5RKOH8GsgCg4w8D
WA8R+ZpjHPManfxvIuqD+lY=
=4c2w
-----END PGP SIGNATURE-----
On Sat, 2008-03-08 at 08:03 -0600, Frank Sorenson wrote:
> It's module-init-tools-3.4-2.fc8.x86_64 (most recent Fedora rpm available).
Ok, so I'll see if I can find a Dell system in the office to reproduce
this on Monday. Any Dell should do, right Matt?
Jon.
On Sat, 2008-03-08 at 16:53 -0500, Jon Masters wrote:
> On Sat, 2008-03-08 at 08:03 -0600, Frank Sorenson wrote:
>
> > It's module-init-tools-3.4-2.fc8.x86_64 (most recent Fedora rpm available).
>
> Ok, so I'll see if I can find a Dell system in the office to reproduce
> this on Monday. Any Dell should do, right Matt?
Should not be needed:
http://lkml.org/lkml/2008/3/8/91
Cheers,
Kay
On Sat, 2008-03-08 at 23:54 +0100, Kay Sievers wrote:
> On Sat, 2008-03-08 at 16:53 -0500, Jon Masters wrote:
> > On Sat, 2008-03-08 at 08:03 -0600, Frank Sorenson wrote:
> >
> > > It's module-init-tools-3.4-2.fc8.x86_64 (most recent Fedora rpm available).
> >
> > Ok, so I'll see if I can find a Dell system in the office to reproduce
> > this on Monday. Any Dell should do, right Matt?
>
> Should not be needed:
> http://lkml.org/lkml/2008/3/8/91
Oh, all's well that ends well :)
Jon.