LinuxLists.cc - USB storage no-boot regression (bisected)

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Tue, 14 Apr 2009 17:06:14 -0400
Jeff Garzik <[email protected]> wrote:

>
> Once of the x86-64 machines I use for testing runs off of two 2GB USB
> flash drives, one for Fedora 10 userland, and one for kernel
> repository
> + builds.
>
> It boots correctly in 2.6.27, but fails with the same symptoms in
> 2.6.28, 2.6.29 and 2.6.30-rc1:
>
> 1) The kernel boots
> 2) After time passes, kernel begins executing initramfs
> userland
> 3) the kernel prints out probe messages for the USB keyboard,
> SCSI probe messages for the two USB flash drives
>
> Or IOW, the keyboard and two SCSI drives appear after initramfs
> begins booting. And this is for drivers built into the kernel
> (though same behavior with modules).
>
> This no-boot regression is 100% reproducible, and neatly bisects down
> to
>
> > commit 8520f38099ccfdac2147a0852f84ee7a8ee5e197
> > Author: Alan Stern <[email protected]>
> > Date: Mon Sep 22 14:44:26 2008 -0400
> >
> > USB: change hub initialization sleeps to delayed_work
> >
> > This patch (as1137) changes the hub_activate() routine,
> > replacing the power-power-up and debounce delays with delayed_work
> > calls. The idea is that on systems where the USB stack is compiled
> > into the kernel rather than built as modules, these delays will no
> > longer block the boot thread. At least 100 ms is saved for each
> > root hub, which can add up to a significant savings in total boot
> > time.
> > Arjan van de Ven was very pleased to see that this shaved 700
> > ms off his computer's boot time. Since his total boot time is on
> > the order of two seconds, the improvement is considerable.
> >
> > Signed-off-by: Alan Stern <[email protected]>
> > Tested-by: Arjan van de Ven <[email protected]>
> > Signed-off-by: Greg Kroah-Hartman <[email protected]>
>
>
> My preliminary guess is that this made things --too-- asynchronous,
> and for some reason userland begins executing before the SCSI core
> initializes the USB storage as Linux block devices.
>
> In any case, I cannot boot because of the above commit :)

the fundamental problem with USB is that you don'tm know when you're
done probing ... devices show up when they feel like or more or less.

This change just made it go faster enough for you to be out of luck;
fundamentally your userland needs to wait if the device it wants is not
there.

Or you pass a kernel boot parameter to wait.. there is one (root_delay
or something, haven't used it in a while).

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 01:53:30

by Greg KH

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Tue, Apr 14, 2009 at 05:06:14PM -0400, Jeff Garzik wrote:
>
> Once of the x86-64 machines I use for testing runs off of two 2GB USB
> flash drives, one for Fedora 10 userland, and one for kernel repository
> + builds.
>
> It boots correctly in 2.6.27, but fails with the same symptoms in
> 2.6.28, 2.6.29 and 2.6.30-rc1:
>
> 1) The kernel boots
> 2) After time passes, kernel begins executing initramfs
> userland
> 3) the kernel prints out probe messages for the USB keyboard,
> SCSI probe messages for the two USB flash drives
>
> Or IOW, the keyboard and two SCSI drives appear after initramfs begins
> booting. And this is for drivers built into the kernel (though same
> behavior with modules).
>
> This no-boot regression is 100% reproducible, and neatly bisects down to
>
> > commit 8520f38099ccfdac2147a0852f84ee7a8ee5e197
> > Author: Alan Stern <[email protected]>
> > Date: Mon Sep 22 14:44:26 2008 -0400
> >
> > USB: change hub initialization sleeps to delayed_work
> >
> > This patch (as1137) changes the hub_activate() routine, replacing the
> > power-power-up and debounce delays with delayed_work calls. The idea
> > is that on systems where the USB stack is compiled into the kernel
> > rather than built as modules, these delays will no longer block the
> > boot thread. At least 100 ms is saved for each root hub, which can
> > add up to a significant savings in total boot time.
> >
> > Arjan van de Ven was very pleased to see that this shaved 700 ms off
> > his computer's boot time. Since his total boot time is on the order
> > of two seconds, the improvement is considerable.
> >
> > Signed-off-by: Alan Stern <[email protected]>
> > Tested-by: Arjan van de Ven <[email protected]>
> > Signed-off-by: Greg Kroah-Hartman <[email protected]>
>
>
> My preliminary guess is that this made things --too-- asynchronous, and
> for some reason userland begins executing before the SCSI core
> initializes the USB storage as Linux block devices.
>
> In any case, I cannot boot because of the above commit :)

Like Arjan said, this is because we are initializing faster now, and
things are a bit more asynchronous. Use the root_delay boot option,
that's what I use for my USB-based systems, and have not had a problem
with that at all.

thanks,

greg k-h

2009-04-15 02:30:44

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Arjan van de Ven wrote:
> This change just made it go faster enough for you to be out of luck;
> fundamentally your userland needs to wait if the device it wants is not
> there.

All these drivers are in-kernel, and the root device is passed via
command line. There is no userland at that point, that needs to wait.

If this change added an implicit initramfs requirement, that is a pretty
major regression.

Jeff

2009-04-15 02:36:24

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Greg KH wrote:
> On Tue, Apr 14, 2009 at 05:06:14PM -0400, Jeff Garzik wrote:
>> Once of the x86-64 machines I use for testing runs off of two 2GB USB
>> flash drives, one for Fedora 10 userland, and one for kernel repository
>> + builds.
>>
>> It boots correctly in 2.6.27, but fails with the same symptoms in
>> 2.6.28, 2.6.29 and 2.6.30-rc1:
>>
>> 1) The kernel boots
>> 2) After time passes, kernel begins executing initramfs
>> userland
>> 3) the kernel prints out probe messages for the USB keyboard,
>> SCSI probe messages for the two USB flash drives
>>
>> Or IOW, the keyboard and two SCSI drives appear after initramfs begins
>> booting. And this is for drivers built into the kernel (though same
>> behavior with modules).
>>
>> This no-boot regression is 100% reproducible, and neatly bisects down to
>>
>>> commit 8520f38099ccfdac2147a0852f84ee7a8ee5e197
>>> Author: Alan Stern <[email protected]>
>>> Date: Mon Sep 22 14:44:26 2008 -0400
>>>
>>> USB: change hub initialization sleeps to delayed_work
>>>
>>> This patch (as1137) changes the hub_activate() routine, replacing the
>>> power-power-up and debounce delays with delayed_work calls. The idea
>>> is that on systems where the USB stack is compiled into the kernel
>>> rather than built as modules, these delays will no longer block the
>>> boot thread. At least 100 ms is saved for each root hub, which can
>>> add up to a significant savings in total boot time.
>>>
>>> Arjan van de Ven was very pleased to see that this shaved 700 ms off
>>> his computer's boot time. Since his total boot time is on the order
>>> of two seconds, the improvement is considerable.
>>>
>>> Signed-off-by: Alan Stern <[email protected]>
>>> Tested-by: Arjan van de Ven <[email protected]>
>>> Signed-off-by: Greg Kroah-Hartman <[email protected]>
>>
>> My preliminary guess is that this made things --too-- asynchronous, and
>> for some reason userland begins executing before the SCSI core
>> initializes the USB storage as Linux block devices.
>>
>> In any case, I cannot boot because of the above commit :)
>
> Like Arjan said, this is because we are initializing faster now, and
> things are a bit more asynchronous. Use the root_delay boot option,
> that's what I use for my USB-based systems, and have not had a problem
> with that at all.

Is that solution really scalable to every user with a regression severe
enough it prevents them from booting?

When did regressions become an acceptable tradeoff for speed?

This system boots just fine under kernel 2.6.27, 2.6.26, 2.6.25, and so
on. Switch the kernel to 2.6.28, and it no longer boots. A regression
cannot get more clear than that.

Maybe this commit should have been accompanied by one that checks "root=" ?

Jeff

2009-04-15 02:42:39

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Tue, 14 Apr 2009 22:30:28 -0400
Jeff Garzik <[email protected]> wrote:

> Arjan van de Ven wrote:
> > This change just made it go faster enough for you to be out of luck;
> > fundamentally your userland needs to wait if the device it wants is
> > not there.
>
> All these drivers are in-kernel, and the root device is passed via
> command line. There is no userland at that point, that needs to wait.

ok fair; but that does not change that the kernel does not know if a
device is coming.
Yes that sucks; sadly USB is just this way, you don't know when no new
devices will come from a certain bus.

>
> If this change added an implicit initramfs requirement, that is a
> pretty major regression.

it didn't; this is what root_wait is for.

Having said that, I have a patch that basically retries this stuff
inside the kernel if the mount-of-rootfs fails....

(but it's independent of the usb improvement)

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 02:45:12

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Tue, 14 Apr 2009 22:35:59 -0400
Jeff Garzik <[email protected]> wrote:

> Greg KH wrote:
> > On Tue, Apr 14, 2009 at 05:06:14PM -0400, Jeff Garzik wrote:
> >> Once of the x86-64 machines I use for testing runs off of two 2GB
> >> USB flash drives, one for Fedora 10 userland, and one for kernel
> >> repository
> >> + builds.
> >>
> >> It boots correctly in 2.6.27, but fails with the same symptoms in
> >> 2.6.28, 2.6.29 and 2.6.30-rc1:
> >>
> >> 1) The kernel boots
> >> 2) After time passes, kernel begins executing initramfs
> >> userland
> >> 3) the kernel prints out probe messages for the USB
> >> keyboard, SCSI probe messages for the two USB flash drives
> >>
> >> Or IOW, the keyboard and two SCSI drives appear after initramfs
> >> begins booting. And this is for drivers built into the kernel
> >> (though same behavior with modules).
> >>
> >> This no-boot regression is 100% reproducible, and neatly bisects
> >> down to
> >>
> >>> commit 8520f38099ccfdac2147a0852f84ee7a8ee5e197
> >>> Author: Alan Stern <[email protected]>
> >>> Date: Mon Sep 22 14:44:26 2008 -0400
> >>>
> >>> USB: change hub initialization sleeps to delayed_work
> >>>
> >>> This patch (as1137) changes the hub_activate() routine,
> >>> replacing the power-power-up and debounce delays with
> >>> delayed_work calls. The idea is that on systems where the USB
> >>> stack is compiled into the kernel rather than built as modules,
> >>> these delays will no longer block the boot thread. At least 100
> >>> ms is saved for each root hub, which can add up to a significant
> >>> savings in total boot time.
> >>> Arjan van de Ven was very pleased to see that this shaved 700
> >>> ms off his computer's boot time. Since his total boot time is on
> >>> the order of two seconds, the improvement is considerable.
> >>>
> >>> Signed-off-by: Alan Stern <[email protected]>
> >>> Tested-by: Arjan van de Ven <[email protected]>
> >>> Signed-off-by: Greg Kroah-Hartman <[email protected]>
> >>
> >> My preliminary guess is that this made things --too--
> >> asynchronous, and for some reason userland begins executing before
> >> the SCSI core initializes the USB storage as Linux block devices.
> >>
> >> In any case, I cannot boot because of the above commit :)
> >
> > Like Arjan said, this is because we are initializing faster now, and
> > things are a bit more asynchronous. Use the root_delay boot option,
> > that's what I use for my USB-based systems, and have not had a
> > problem with that at all.
>
> Is that solution really scalable to every user with a regression
> severe enough it prevents them from booting?
>
> When did regressions become an acceptable tradeoff for speed?
>
> This system boots just fine under kernel 2.6.27, 2.6.26, 2.6.25, and
> so on. Switch the kernel to 2.6.28, and it no longer boots. A
> regression cannot get more clear than that.

You had pure luck though.

We used to wait 100 msec per USB bus.
A normal laptop has like 5 of these.
if your usb storage was in the first one, basically you got a "free"
500msec delay there. You are/were happy.

Now.. if you stuck your disk in the last port you would get a 100msec
delay. Probably not enough for what you want. But you didn't stick
your disk there....

In the new code all ports get their power turned on and THEN things
wait... so all ports get the 100 msec treatment, not the
500/400/300/200/100 staggering.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 02:49:11

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Arjan van de Ven wrote:
> On Tue, 14 Apr 2009 22:30:28 -0400
> Jeff Garzik <[email protected]> wrote:
>
>> Arjan van de Ven wrote:
>>> This change just made it go faster enough for you to be out of luck;
>>> fundamentally your userland needs to wait if the device it wants is
>>> not there.
>> All these drivers are in-kernel, and the root device is passed via
>> command line. There is no userland at that point, that needs to wait.
>
> ok fair; but that does not change that the kernel does not know if a
> device is coming.
> Yes that sucks; sadly USB is just this way, you don't know when no new
> devices will come from a certain bus.

Perhaps -- but I can say that kernels <= 2.6.27 booted with 100%
reliability.

Now, Kernels >= 2.6.28 always fail.

The ONLY variable is the kernel.

Jeff

2009-04-15 03:08:21

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Tue, 14 Apr 2009 22:48:50 -0400
Jeff Garzik <[email protected]> wrote:

> Arjan van de Ven wrote:
> > On Tue, 14 Apr 2009 22:30:28 -0400
> > Jeff Garzik <[email protected]> wrote:
> >
> >> Arjan van de Ven wrote:
> >>> This change just made it go faster enough for you to be out of
> >>> luck; fundamentally your userland needs to wait if the device it
> >>> wants is not there.
> >> All these drivers are in-kernel, and the root device is passed via
> >> command line. There is no userland at that point, that needs to
> >> wait.
> >
> > ok fair; but that does not change that the kernel does not know if a
> > device is coming.
> > Yes that sucks; sadly USB is just this way, you don't know when no
> > new devices will come from a certain bus.
>
> Perhaps -- but I can say that kernels <= 2.6.27 booted with 100%
> reliability.
>
> Now, Kernels >= 2.6.28 always fail.
> d
> The ONLY variable is the kernel.

yes.
and you can get the old behavior back by just sticking an
msleep(100 * <number of USB ports you have minus one>);
back in near the end of the boot.

that really is not the right answer.

by your argument if anything else gets faster and your system is then
booting too fast again that has to get reverted as well.

If there was some reasonable way to wait on USB scanning being done
we'd do that. Not a nanosecond doubt about that.

But since there isn't, this "race who's faster" is an unsolvable
problem other than by you saying "wait for root to be there please".

rootwait [KNL] Wait (indefinitely) for root device to show up.
Useful for devices that are detected asynchronously
(e.g. USB and MMC devices).

that option has been there for a really long time, and has technically be required
for your case. You got lucky so far by pure timing... until recently.
It's not nice to not boot suddenly, I sympathize with that. But there's nothing
we can realistically do other than using the option that is there for exactly your case.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 05:19:36

by Greg KH

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Tue, Apr 14, 2009 at 10:35:59PM -0400, Jeff Garzik wrote:
> > Like Arjan said, this is because we are initializing faster now, and
> > things are a bit more asynchronous. Use the root_delay boot option,
> > that's what I use for my USB-based systems, and have not had a problem
> > with that at all.
>
> Is that solution really scalable to every user with a regression severe
> enough it prevents them from booting?
>
> When did regressions become an acceptable tradeoff for speed?

So, we aren't allowed to go faster?

What happens when you buy a new box with more USB host controllers and a
faster processor? Same problem.

> This system boots just fine under kernel 2.6.27, 2.6.26, 2.6.25, and so
> on. Switch the kernel to 2.6.28, and it no longer boots. A regression
> cannot get more clear than that.
>
> Maybe this commit should have been accompanied by one that checks "root=" ?

How would that be accomplished?

The issue is that you were just lucky that your machine worked properly
previously. My boxes with the same type of setup didn't, so I quickly
realized what the root delay boot option was for. You need to just do
the same thing here, there's nothing else we can do.

thanks,

greg k-h

2009-04-15 08:41:43

by Alan

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

> If this change added an implicit initramfs requirement, that is a pretty
> major regression.

USB root has always needed an initramfs. On some boxes it might happen to
work most of the time because of other implicit delays. You were just
lucky in the past.

If you are so initramfs averse you could try hacking the kernel to try
remounting the root every few seconds for a bit if it fails rather than
giving up.

Alan

2009-04-15 13:47:00

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Greg KH wrote:
> On Tue, Apr 14, 2009 at 10:35:59PM -0400, Jeff Garzik wrote:
>>> Like Arjan said, this is because we are initializing faster now, and
>>> things are a bit more asynchronous. Use the root_delay boot option,
>>> that's what I use for my USB-based systems, and have not had a problem
>>> with that at all.
>> Is that solution really scalable to every user with a regression severe
>> enough it prevents them from booting?
>>
>> When did regressions become an acceptable tradeoff for speed?
>
> So, we aren't allowed to go faster?

Well, when the result fails to boot, you are only going faster to a point :)

Oh well...

Jeff

2009-04-15 14:25:23

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Greg KH wrote:
> ..
> The issue is that you were just lucky that your machine worked properly
> previously. My boxes with the same type of setup didn't, so I quickly
> realized what the root delay boot option was for. You need to just do
> the same thing here, there's nothing else we can do.
..

Bad excuse.

SATA drives also take variable amounts of time to "show up" at boot.
Perhaps Jeff should customize libata for your and Arjan's exact setups,
just to help with understanding the point here. :)

The speed ups are fine (and welcome), but we really now need
Arjan to follow-up with a patch to have the kernel *by default*
wait a little longer for the rootfs to show up.

Not forever, just a few seconds to compensate for the regression.

2009-04-15 14:28:30

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, 15 Apr 2009 10:25:05 -0400
Mark Lord <[email protected]> wrote:

> Greg KH wrote:
> > ..
> > The issue is that you were just lucky that your machine worked
> > properly previously. My boxes with the same type of setup didn't,
> > so I quickly realized what the root delay boot option was for. You
> > need to just do the same thing here, there's nothing else we can do.
> ..
>
> Bad excuse.
>
> SATA drives also take variable amounts of time to "show up" at boot.
> Perhaps Jeff should customize libata for your and Arjan's exact
> setups, just to help with understanding the point here. :)

the difference is that with sata you know when you are done and have all
possible drives. No so much much with USB. So with SATA we can, and do,
wait for the scan to complete at the right point in the boot.

>
> The speed ups are fine (and welcome), but we really now need
> Arjan to follow-up with a patch to have the kernel *by default*
> wait a little longer for the rootfs to show up.
>
> Not forever, just a few seconds to compensate for the regression.

seconds!!!!!
The whole kernel boots in half a second!

for this case, which is unfortunately not detectable by the kernel,
there is the rootwait option.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 15:03:21

by Alan

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

> SATA drives also take variable amounts of time to "show up" at boot.
> Perhaps Jeff should customize libata for your and Arjan's exact setups,
> just to help with understanding the point here. :)

Actually I would argue the reverse. The sooner we can push this so that
libata isn't blocking mounting the rootfs the better.

> The speed ups are fine (and welcome), but we really now need
> Arjan to follow-up with a patch to have the kernel *by default*
> wait a little longer for the rootfs to show up.
>
> Not forever, just a few seconds to compensate for the regression.

Why should every user suffer a slower boot and a poorer resume time ?

Instead make the root fs mounting look like this

while(my_rootfs_hasnt_appeared_and_i_am_sad()) {
wait_on(&new_disk_discovery);
}

and poke the queue whenever we add a relevant device.

That way if you are booting off an initrd you can finish the SATA probe
in parallel to getting userspace ticking over.

On what is nowdays essentially a hot plug system it all needs turning
this way up - eg RAID volumes should assemble and come online as the
drives are discovered not at some fixed point later in userspace.

Alan

2009-04-15 15:37:57

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Arjan van de Ven wrote:
> On Wed, 15 Apr 2009 10:25:05 -0400
> Mark Lord <[email protected]> wrote:
>
>> Greg KH wrote:
>>> ..
>>> The issue is that you were just lucky that your machine worked
>>> properly previously. My boxes with the same type of setup didn't,
>>> so I quickly realized what the root delay boot option was for. You
>>> need to just do the same thing here, there's nothing else we can do.
>> ..
>>
>> Bad excuse.
>>
>> SATA drives also take variable amounts of time to "show up" at boot.
>> Perhaps Jeff should customize libata for your and Arjan's exact
>> setups, just to help with understanding the point here. :)
>
> the difference is that with sata you know when you are done and have all
> possible drives. No so much much with USB. So with SATA we can, and do,
> wait for the scan to complete at the right point in the boot.
>
>> The speed ups are fine (and welcome), but we really now need
>> Arjan to follow-up with a patch to have the kernel *by default*
>> wait a little longer for the rootfs to show up.
>>
>> Not forever, just a few seconds to compensate for the regression.
>
> seconds!!!!!
> The whole kernel boots in half a second!
..

Oh, absolutely I agree.

That's why I'm not suggesting a DELAY,
but rather a TIMEOUT (where it keeps trying up until the timeout).

For desktop, it should really just wait forever,
but I can understand situations (server room)
where that would be a Really Bad Idea.

So just have it sit there and retry the rootfs for a few seconds,
to compensate for this regression and also for others that have
yet to be discovered / reported.

Everyone's life will be easier that way.

Cheers

2009-04-15 15:47:49

by Alan Stern

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, 15 Apr 2009, Alan Cox wrote:

> Why should every user suffer a slower boot and a poorer resume time ?
>
> Instead make the root fs mounting look like this
>
>
> while(my_rootfs_hasnt_appeared_and_i_am_sad()) {
> wait_on(&new_disk_discovery);
> }
>
> and poke the queue whenever we add a relevant device.
>
> That way if you are booting off an initrd you can finish the SATA probe
> in parallel to getting userspace ticking over.
>
> On what is nowdays essentially a hot plug system it all needs turning
> this way up - eg RAID volumes should assemble and come online as the
> drives are discovered not at some fixed point later in userspace.

Indeed, something like this should also be used for
resume-from-hibernation, to wait for the swap device.

Alan Stern

2009-04-15 15:49:59

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Alan Stern wrote:
> On Wed, 15 Apr 2009, Alan Cox wrote:
>
>> Why should every user suffer a slower boot and a poorer resume time ?
>>
>> Instead make the root fs mounting look like this
>>
>>
>> while(my_rootfs_hasnt_appeared_and_i_am_sad()) {
>> wait_on(&new_disk_discovery);
>> }
>>
>> and poke the queue whenever we add a relevant device.
>>
>> That way if you are booting off an initrd you can finish the SATA probe
>> in parallel to getting userspace ticking over.
>>
>> On what is nowdays essentially a hot plug system it all needs turning
>> this way up - eg RAID volumes should assemble and come online as the
>> drives are discovered not at some fixed point later in userspace.
>
> Indeed, something like this should also be used for
> resume-from-hibernation, to wait for the swap device.
..

It just needs a way to set a finite timeout, so that server room
equipment can auto-panic-reboot and try again if a device has died.

-ml

2009-04-15 17:06:30

by David VomLehn

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, Apr 15, 2009 at 11:49:45AM -0400, Mark Lord wrote:
> Alan Stern wrote:
>> On Wed, 15 Apr 2009, Alan Cox wrote:
>>
>>> Why should every user suffer a slower boot and a poorer resume time ?
>>>
>>> Instead make the root fs mounting look like this
>>>
>>>
>>> while(my_rootfs_hasnt_appeared_and_i_am_sad()) {
>>> wait_on(&new_disk_discovery);
>>> }
>>>
>>> and poke the queue whenever we add a relevant device.
>>>
>>> That way if you are booting off an initrd you can finish the SATA probe
>>> in parallel to getting userspace ticking over.
>>>
>>> On what is nowdays essentially a hot plug system it all needs turning
>>> this way up - eg RAID volumes should assemble and come online as the
>>> drives are discovered not at some fixed point later in userspace.
>>
>> Indeed, something like this should also be used for
>> resume-from-hibernation, to wait for the swap device.
> ..
>
> It just needs a way to set a finite timeout, so that server room
> equipment can auto-panic-reboot and try again if a device has died.

The problem with USB root devices is the same one I brought up a couple of
weeks ago--faster booting means that USB boot devices fail. We now have
problems with three different classes of devices:

o Disks
o Network devices
o Serial consoles

Saying that we were "lucky" that things worked before is no help and
you should be aware that it ticks people off. I agree that this is not a
USB problem, but there is a *very* real problem:

The work to decrease boot time has exposed race conditions that
always existed, but are now making the kernel less usable.

So, instead of spending time denying that there is a USB problem, let's
focus on solve the boot device synchronization problem. I have already
posted a (probably incomplete, possibly wrong) patch to synchronize console
initialization. We need to do the same for other boot devices, too.

David VomLehn

2009-04-15 17:34:27

by Alan

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

> Saying that we were "lucky" that things worked before is no help and
> you should be aware that it ticks people off.

No doubt those who wrote the code and never expected that timing to
reliably work will take affront at those who blame them for it not
working. Those who expected it to work and never knew it was not
guaranteed will feel likewise in reverse.

The real questions are should it be fixed and is dumping the problem on
the initrd the right answer. The concensus appears to yes and no.

The console one is less clear but if we are doing one we should probably
do both.

Alan

2009-04-15 19:57:19

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, 15 Apr 2009 11:37:44 -0400
Mark Lord <[email protected]> wrote:

> Arjan van de Ven wrote:
> > On Wed, 15 Apr 2009 10:25:05 -0400
> > Mark Lord <[email protected]> wrote:
> >
> >> Greg KH wrote:
> >>> ..
> >>> The issue is that you were just lucky that your machine worked
> >>> properly previously. My boxes with the same type of setup didn't,
> >>> so I quickly realized what the root delay boot option was for.
> >>> You need to just do the same thing here, there's nothing else we
> >>> can do.
> >> ..
> >>
> >> Bad excuse.
> >>
> >> SATA drives also take variable amounts of time to "show up" at
> >> boot. Perhaps Jeff should customize libata for your and Arjan's
> >> exact setups, just to help with understanding the point here. :)
> >
> > the difference is that with sata you know when you are done and
> > have all possible drives. No so much much with USB. So with SATA we
> > can, and do, wait for the scan to complete at the right point in
> > the boot.
> >
> >> The speed ups are fine (and welcome), but we really now need
> >> Arjan to follow-up with a patch to have the kernel *by default*
> >> wait a little longer for the rootfs to show up.
> >>
> >> Not forever, just a few seconds to compensate for the regression.
> >
> > seconds!!!!!
> > The whole kernel boots in half a second!
> ..
>
> Oh, absolutely I agree.
>
> That's why I'm not suggesting a DELAY
> but rather a TIMEOUT (where it keeps trying up until the timeout).

This exists today. It's just not something Jeff chose to use ;)
(because he didn't need to)

>
> For desktop, it should really just wait forever,
> but I can understand situations (server room)
> where that would be a Really Bad Idea.

it's called rootwait and such :)

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 20:04:43

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, 15 Apr 2009 18:32:52 +0100
Alan Cox <[email protected]> wrote:

> > Saying that we were "lucky" that things worked before is no help and
> > you should be aware that it ticks people off.
>
> No doubt those who wrote the code and never expected that timing to
> reliably work will take affront at those who blame them for it not
> working.

It's even more than that; it's OTHER code that gets faster that
can/will break this.

> Those who expected it to work and never knew it was not
> guaranteed will feel likewise in reverse.

For storage at least the solution already existed (root_wait);
it just hasn't been used consistently.

For other pieces it's hard. Non-enumeratable busses just suck;
at some point all you can do is just wait (which we have already
available today for anyone to do). I realize people don't want to
just wait 4 seconds (the people who first objected to boot time
improvements then suddenly care about boot time ;-)...

For root fs there's some options, and I have patches to basically retry
on fail. (The patches have a bug and I don't have time to solve it this
week, so I'm not submitting them)
For other devices it is hard. Realistically we need hotplug to work
well enough so that when a device shows up, we can just hook it up when
it does.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-15 20:21:20

by David VomLehn

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, Apr 15, 2009 at 01:06:14PM -0700, Arjan van de Ven wrote:
> For other pieces it's hard. Non-enumeratable busses just suck;
> at some point all you can do is just wait (which we have already
> available today for anyone to do). I realize people don't want to
> just wait 4 seconds (the people who first objected to boot time
> improvements then suddenly care about boot time ;-)...

We can do better than just waiting. What we want to do is to wait
until a suitable device becomes available. For USB console, the patch
I submitted yesterday waits for the first console to be registered.
That might not be quite right, but it's a lot better than no console
at all. For network devices, we have to wait for a specific device and
I haven't had a chance to look at this. For both types of devices, we
can timeout after a reasonable interval, for some definition of reasonable.

Waiting for a suitable device preserves fast boot times, timing out allows
the system to come up enough to try to diagnose the problem.

David VomLehn

2009-04-15 21:55:32

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Arjan van de Ven wrote:
> On Wed, 15 Apr 2009 11:37:44 -0400
> Mark Lord <[email protected]> wrote:
>
>> Arjan van de Ven wrote:
>>> On Wed, 15 Apr 2009 10:25:05 -0400
>>> Mark Lord <[email protected]> wrote:
..
>>>> Not forever, just a few seconds to compensate for the regression.
>>> seconds!!!!!
>>> The whole kernel boots in half a second!
>> ..
>>
>> Oh, absolutely I agree.
>>
>> That's why I'm not suggesting a DELAY
>> but rather a TIMEOUT (where it keeps trying up until the timeout).
>
> This exists today. It's just not something Jeff chose to use ;)
> (because he didn't need to)
>
>> For desktop, it should really just wait forever,
>> but I can understand situations (server room)
>> where that would be a Really Bad Idea.
>
> it's called rootwait and such :)
..

No, that's not the same thing.
rootwait has no timeout -- it waits *forever*,
which will break auto-recovery on servers.

We just need it to wait/retry up to a timeout (parameter?).

Thanks

2009-04-16 01:30:41

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, 15 Apr 2009 17:55:08 -0400
Mark Lord <[email protected]> wrote:

> Arjan van de Ven wrote:
> > On Wed, 15 Apr 2009 11:37:44 -0400
> > Mark Lord <[email protected]> wrote:
> >
> >> Arjan van de Ven wrote:
> >>> On Wed, 15 Apr 2009 10:25:05 -0400
> >>> Mark Lord <[email protected]> wrote:
> ..
> >>>> Not forever, just a few seconds to compensate for the regression.
> >>> seconds!!!!!
> >>> The whole kernel boots in half a second!
> >> ..
> >>
> >> Oh, absolutely I agree.
> >>
> >> That's why I'm not suggesting a DELAY
> >> but rather a TIMEOUT (where it keeps trying up until the timeout).
> >
> > This exists today. It's just not something Jeff chose to use ;)
> > (because he didn't need to)
> >
> >> For desktop, it should really just wait forever,
> >> but I can understand situations (server room)
> >> where that would be a Really Bad Idea.
> >
> > it's called rootwait and such :)
> ..
>
> No, that's not the same thing.
> rootwait has no timeout -- it waits *forever*,
> which will break auto-recovery on servers.

btw while I don't disagree that something like this is nice...

.... who would boot their server from a USB stick for production use?

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-16 02:14:54

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Arjan van de Ven wrote:
> On Wed, 15 Apr 2009 17:55:08 -0400
..
>> No, that's not the same thing.
>> rootwait has no timeout -- it waits *forever*,
>> which will break auto-recovery on servers.
>
> btw while I don't disagree that something like this is nice...
>
> .... who would boot their server from a USB stick for production use?
..

At least one MAJOR storage vendor that I know of.
Probably more.

Though it's an onboard USB SSD, not a pluggable stick.

Cheers

2009-04-16 02:43:57

by Hal Murray

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

> .... who would boot their server from a USB stick for production use?

I've worked on a project that was doing that. Think of embedded computing
rather than a typical box in a server farm.

If you want to do something slightly more complicated than monitor
temperature, it's sometimes simpler/cheaper to trim down a real OS than it is
to build up an environment that has everything you need, especially if you
want to do any networking. With enough RAM, LInux runs fine without a disk,
and if you don't need a GUI, "enough RAM" isn't much by modern standards.

There are several distributions targeted at running out of RAM after booting
from an USB thumb drive or Compact Flash or CD or whatever.

Google for >Linux on a Stick< or >Puppy Linux< for a few starters.

--
These are my opinions, not necessarily my employer's. I hate spam.

2009-04-16 03:07:26

by Greg KH

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Wed, Apr 15, 2009 at 10:14:38PM -0400, Mark Lord wrote:
> Arjan van de Ven wrote:
> > On Wed, 15 Apr 2009 17:55:08 -0400
> ..
> >> No, that's not the same thing.
> >> rootwait has no timeout -- it waits *forever*,
> >> which will break auto-recovery on servers.
> >
> > btw while I don't disagree that something like this is nice...
> >
> > .... who would boot their server from a USB stick for production use?
> ..
>
> At least one MAJOR storage vendor that I know of.
> Probably more.
>
> Though it's an onboard USB SSD, not a pluggable stick.

Wow, that's insane. Leave it to a hardware designer to use a bus that
you never know if you have discovered all the devices to be the primary
device to boot from.

greg k-h

2009-04-16 03:43:17

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

Arjan van de Ven wrote:
> .... who would boot their server from a USB stick for production use?

Whole distros have chosen to do so. the real world doesn't care about
USB enumeration details, they just want to boot off their $9 Wal-Mart
USB sticks... :)

Jeff

2009-04-16 10:52:58

by Alan

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

> > Though it's an onboard USB SSD, not a pluggable stick.
>
> Wow, that's insane. Leave it to a hardware designer to use a bus that
> you never know if you have discovered all the devices to be the primary
> device to boot from.

Let me see:

- Standardised small component
- Low pin count and low wire count bus
- Cheap

In an embedded environment where you know the device is wired in at
software level that strikes me not as insane but very sensible.

2009-04-16 13:32:55

[permalink] [raw]

Subject: Re: USB storage no-boot regression (bisected)

On Thu, 16 Apr 2009 11:51:53 +0100
Alan Cox <[email protected]> wrote:

> > > Though it's an onboard USB SSD, not a pluggable stick.
> >
> > Wow, that's insane. Leave it to a hardware designer to use a bus
> > that you never know if you have discovered all the devices to be
> > the primary device to boot from.
>
> Let me see:
>
> - Standardised small component
> - Low pin count and low wire count bus
> - Cheap
>
> In an embedded environment where you know the device is wired in at
> software level that strikes me not as insane but very sensible.

I'm not surprised usb is used for storage. What I am surprised at is
that USB is used in such high end environments that rootwait is not
sufficient and that the panic-rebooter is needed for the reliability.

--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org

2009-04-16 13:49:40