Both of you are missing an important difference between Suspend-to-RAM and
Suspend-to-Disk.
Suspend-to-RAM is a true suspend operation, in that the hardware's state
is maintained _in the hardware_. External buses like USB will retain
suspend power, for instance (assuming the motherboard supports it; some
don't).
Suspend-to-Disk, by contrast, is _not_ a true suspend. It can more
accurately be described as checkpoint-and-turn-off. Hardware state is not
maintained. (Some systems may support a special ACPI state that does
maintain suspend power to external buses during shutdown, I forget what
it's called. And I down't know whether swsusp uses this state.)
So for example, let's say you have a filesystem mounted on a USB flash or
disk drive. With Suspend-to-RAM, there's a very good chance that the
connection and filesystem will still be intact when you resume. With
Suspend-to-Disk, the USB connection will terminate when the computer shuts
down. When you resume, the device will be gone and your filesystem will
be screwed.
Alan Stern
Alan Stern wrote:
> Both of you are missing an important difference between Suspend-to-RAM and
> Suspend-to-Disk.
>
> Suspend-to-RAM is a true suspend operation, in that the hardware's state
> is maintained _in the hardware_. External buses like USB will retain
> suspend power, for instance (assuming the motherboard supports it; some
> don't).
>
> Suspend-to-Disk, by contrast, is _not_ a true suspend. It can more
> accurately be described as checkpoint-and-turn-off. Hardware state is not
> maintained. (Some systems may support a special ACPI state that does
> maintain suspend power to external buses during shutdown, I forget what
> it's called. And I down't know whether swsusp uses this state.)
>
I would disagree. The only difference between the two is WHERE the
state is maintained - ram vs. disk. I won't really argue it though,
because it's just semantics -- call it whatever you want.
> So for example, let's say you have a filesystem mounted on a USB flash or
> disk drive. With Suspend-to-RAM, there's a very good chance that the
> connection and filesystem will still be intact when you resume. With
> Suspend-to-Disk, the USB connection will terminate when the computer shuts
> down. When you resume, the device will be gone and your filesystem will
> be screwed.
>
This is not true. The USB bus is shut down either way, and provided
that you have not unplugged the disk, nothing will be screwed when you
resume from disk or ram.
On Sun, 12 Feb 2006, Phillip Susi wrote:
> Alan Stern wrote:
> > Both of you are missing an important difference between Suspend-to-RAM and
> > Suspend-to-Disk.
> >
> > Suspend-to-RAM is a true suspend operation, in that the hardware's state
> > is maintained _in the hardware_. External buses like USB will retain
> > suspend power, for instance (assuming the motherboard supports it; some
> > don't).
> >
> > Suspend-to-Disk, by contrast, is _not_ a true suspend. It can more
> > accurately be described as checkpoint-and-turn-off. Hardware state is not
> > maintained. (Some systems may support a special ACPI state that does
> > maintain suspend power to external buses during shutdown, I forget what
> > it's called. And I down't know whether swsusp uses this state.)
> >
>
> I would disagree. The only difference between the two is WHERE the
> state is maintained - ram vs. disk. I won't really argue it though,
> because it's just semantics -- call it whatever you want.
It's not just semantics. There's a real difference between maintaining
state in the hardware and maintaining it somewhere else. The biggest
difference is that if the hardware retains suspend power, it is able to
detect disconnections. When the system resumes, it _knows_ whether a
device was attached the entire time, as opposed to being unplugged and
replugged (or possibly a different device plugged in!) while the system
was asleep. If the hardware is down completely, there is no way of
telling for certain whether a device attached to some port is the same one
that was there when the system got suspended.
Another difference is the possibility of remote wakeup. Clearly it can't
happen when there's no power available.
> > So for example, let's say you have a filesystem mounted on a USB flash or
> > disk drive. With Suspend-to-RAM, there's a very good chance that the
> > connection and filesystem will still be intact when you resume. With
> > Suspend-to-Disk, the USB connection will terminate when the computer shuts
> > down. When you resume, the device will be gone and your filesystem will
> > be screwed.
> >
>
> This is not true. The USB bus is shut down either way, and provided
> that you have not unplugged the disk, nothing will be screwed when you
> resume from disk or ram.
Have you actually tried it? I have.
In any case, it is undeniably true that if the bus is shut down then all
the USB connections are lost. When you resume it will be the same as if
you had unplugged all the USB devices and then replugged them. Not a good
thing to do when they contain mounted filesystems; all the memory mappings
are invalidated.
(Bear in mind that whether a USB bus gets shut down depends on the
motherboard; some supply suspend power and some don't. It depends on the
USB controller too; some support low-power states other than "completely
off" and others don't.)
Alan Stern
On Feb 12, 2006, at 19:51, Phillip Susi wrote:
> Alan Stern wrote:
>> Both of you are missing an important difference between Suspend-to-
>> RAM and Suspend-to-Disk. Suspend-to-RAM is a true suspend
>> operation, in that the hardware's state is maintained _in the
>> hardware_. External buses like USB will retain suspend power, for
>> instance (assuming the motherboard supports it; some don't).
>> Suspend-to-Disk, by contrast, is _not_ a true suspend. It can
>> more accurately be described as checkpoint-and-turn-off. Hardware
>> state is not maintained. (Some systems may support a special ACPI
>> state that does maintain suspend power to external buses during
>> shutdown, I forget what it's called. And I down't know whether
>> swsusp uses this state.)
>
> I would disagree. The only difference between the two is WHERE the
> state is maintained - ram vs. disk. I won't really argue it
> though, because it's just semantics -- call it whatever you want.
From the simple perspective, yes, that's the only difference. On
the other hand, from an efficiency standpoint, they are _completely_
different, to a degree that the OS needs to treat them as such or
performance will suck. Software suspend (to disk, network, file,
etc) _requires_ a copy; it's completely mandatory because RAM is
guaranteed to go away. It freezes everything, checkpoints the kernel
with a lot of really complex and oft-buggy code, unfreezes things,
stores data, and shuts off. Hardware suspend (to RAM only)
implicitly needs no copy. The CPU and memory architecture itself
supports the low-power state, so you don't have to do much of
anything special about kernel or userspace threads, you just have to
suspend the device tree once (already exists for per-device power
management) and then tell the firmware to finish up. The former is O
(memory), the latter is O(1).
For a significant majority of people, hardware suspend is
significantly better. I never understood why people need graphical
splash screens during suspend/resume... until I tried software-
suspend. With hardware-suspend, I close it, 2 seconds later the
little white light on the front comes on and it's sleeping, I open
it, 2 seconds later it's connected to wireless again and ready to
use. With software-suspend, it just _doesn't_ work that fast because
of the inherent way it does things; hard disks on laptops are _slow_.
>> So for example, let's say you have a filesystem mounted on a USB
>> flash or disk drive. With Suspend-to-RAM, there's a very good
>> chance that the connection and filesystem will still be intact
>> when you resume. With Suspend-to-Disk, the USB connection will
>> terminate when the computer shuts down. When you resume, the
>> device will be gone and your filesystem will be screwed.
>
> This is not true. The USB bus is shut down either way, and
> provided that you have not unplugged the disk, nothing will be
> screwed when you resume from disk or ram.
It depends on the hardware. For the disk case, yes, this is true.
On the other hand, for hardware-suspend a number of devices like
keyboards and mice may still be in a low-power suspend mode, allowing
you to wake the computer by pushing keys or mouse buttons.
Cheers,
Kyle Moffett
--
There are two ways of constructing a software design. One way is to
make it so simple that there are obviously no deficiencies. And the
other way is to make it so complicated that there are no obvious
deficiencies. The first method is far more difficult.
-- C.A.R. Hoare
Alan Stern wrote:
>
> It's not just semantics. There's a real difference between maintaining
> state in the hardware and maintaining it somewhere else. The biggest
> difference is that if the hardware retains suspend power, it is able to
> detect disconnections. When the system resumes, it _knows_ whether a
> device was attached the entire time, as opposed to being unplugged and
> replugged (or possibly a different device plugged in!) while the system
> was asleep. If the hardware is down completely, there is no way of
> telling for certain whether a device attached to some port is the same one
> that was there when the system got suspended.
>
During suspend the hardware is usually completely powered off, and in
either case, there is nothing running on the CPU to monitor device
insertion/removal. When the system is resumed the kernel decides if the
hardware has changed the same way for either system: it probes the
hardware to see if it is still there. There isn't anything special that
monitors device insertion/removal while suspended to ram.
>> This is not true. The USB bus is shut down either way, and provided
>> that you have not unplugged the disk, nothing will be screwed when you
>> resume from disk or ram.
>
> Have you actually tried it? I have.
If it doesn't work then you have found a bug and should file a report.
The state of the kernel after resuming from either suspend to disk, or
suspend to ram is the same. The filesystem is still mounted, and any
dirty pages in ram will be flushed just like normal. Whether the disk
is connected via SCSI, SATA, USB, or whatever does not matter.
>
> In any case, it is undeniably true that if the bus is shut down then all
> the USB connections are lost. When you resume it will be the same as if
> you had unplugged all the USB devices and then replugged them. Not a good
> thing to do when they contain mounted filesystems; all the memory mappings
> are invalidated.
>
The kernel knows that the same device is still there on the bus, and so
it picks up right where it left off. If it thinks the device has been
unplugged and reconnected, that is a bug.
> (Bear in mind that whether a USB bus gets shut down depends on the
> motherboard; some supply suspend power and some don't. It depends on the
> USB controller too; some support low-power states other than "completely
> off" and others don't.)
>
> Alan Stern
>
On Feb 12, 2006, at 22:52, Phillip Susi wrote:
> Alan Stern wrote:
>> It's not just semantics. There's a real difference between
>> maintaining state in the hardware and maintaining it somewhere
>> else. The biggest difference is that if the hardware retains
>> suspend power, it is able to detect disconnections. When the
>> system resumes, it _knows_ whether a device was attached the
>> entire time, as opposed to being unplugged and replugged (or
>> possibly a different device plugged in!) while the system was
>> asleep. If the hardware is down completely, there is no way of
>> telling for certain whether a device attached to some port is the
>> same one that was there when the system got suspended.
>
> During suspend the hardware is usually completely powered off,
This is true for software suspend, but not for hardware suspend (see
the differences now?) This is why the two are independent and should
not be mashed together into one "Generic Suspend". Let me bring up
the example of my PowerBook again. It's RAM is fully powered right
now, running from battery, and it has another couple days of sleep-
charge left before I have to worry about plugging it in again. When
I open it, the firmware automatically powers up the CPU and other
hardware and returns control to the OS. I can _also_ trigger it to
wake by leaving it closed and connecting an external VGA and USB (it
wakes every time I connect a USB, but my suspend script puts it to
sleep again if it's closed and has no external VGA).
> and in either case, there is nothing running on the CPU to monitor
> device insertion/removal.
You don't need the CPU, just a good USB controller and hubs with low-
power modes and such. The fact that plugging in a USB keyboard/mouse
and a VGA monitor is enough to wake the system when properly
configured should be proof enough.
> When the system is resumed the kernel decides if the hardware has
> changed the same way for either system: it probes the hardware to
> see if it is still there. There isn't anything special that
> monitors device insertion/removal while suspended to ram.
Sometimes not, but again, it depends on the hardware.
Cheers,
Kyle Moffett
--
I have yet to see any problem, however complicated, which, when you
looked at it in the right way, did not become still more complicated.
-- Poul Anderson
On Sun, 12 Feb 2006, Phillip Susi wrote:
> During suspend the hardware is usually completely powered off, and in
> either case, there is nothing running on the CPU to monitor device
> insertion/removal.
Like Kyle said, this depends to some extent on the system. However,
during Suspend-to-RAM it is definitely true that the RAM at least is
powered on. Other components may be powered as well. Otherwise you
wouldn't be able to awaken the system by pressing a button/opening the
case/whatever.
> When the system is resumed the kernel decides if the
> hardware has changed the same way for either system: it probes the
> hardware to see if it is still there. There isn't anything special that
> monitors device insertion/removal while suspended to ram.
Sorry, but you're wrong. First of all, testing if hardware is there is
different from testing whether it has changed -- it could have changed
while the system was asleep, with the result that hardware is indeed there
but it's not the _same_ hardware.
Second, with USB at any rate, in addition to checking that hardware is
still there, the kernel queries the USB controller to see if a disconnect
occurred while the system was asleep. (If the controller wasn't powered
during that time then it will report that every USB device was
disconnected.)
Third, there is indeed something special that monitors USB device
insertion/removal while suspended to RAM -- the USB host controller does
so if it has suspend power.
> >> This is not true. The USB bus is shut down either way, and provided
> >> that you have not unplugged the disk, nothing will be screwed when you
> >> resume from disk or ram.
> >
> > Have you actually tried it? I have.
>
> If it doesn't work then you have found a bug and should file a report.
No. It does work exactly as designed and it's not buggy. You just don't
understand it.
> The state of the kernel after resuming from either suspend to disk, or
> suspend to ram is the same. The filesystem is still mounted, and any
> dirty pages in ram will be flushed just like normal. Whether the disk
> is connected via SCSI, SATA, USB, or whatever does not matter.
Don't be silly. Dirty pages can't be flushed to disks that are no longer
attached! And if a USB disk was unplugged while the system was asleep,
the kernel will know that it is no longer attached. I don't know which
other bus drivers check for this sort of thing.
> The kernel knows that the same device is still there on the bus, and so
> it picks up right where it left off. If it thinks the device has been
> unplugged and reconnected, that is a bug.
It's not a bug if the device _has_ been unplugged and reconnected. When
that happens, there's no way for the kernel to tell whether the device
there now is the same as the device that used to be there.
Alan Stern
Kyle Moffett wrote:
> This is true for software suspend, but not for hardware suspend (see
> the differences now?) This is why the two are independent and should
> not be
No, that is not necessarily correct. Sometimes the ACPI bios can leave
certain devices in a standby mode so they can wake the machine, but it
does not have to, and often does not. Thus when suspended to ram,
typically your usb hard drive and almost allways your ide/sata/scsi
drive will be completely powered off.
> mashed together into one "Generic Suspend". Let me bring up the
> example of my PowerBook again. It's RAM is fully powered right now,
> running from battery, and it has another couple days of sleep-charge
> left before I have to worry about plugging it in again. When I open
> it, the firmware automatically powers up the CPU and other hardware
> and returns control to the OS. I can _also_ trigger it to wake by
> leaving it closed and connecting an external VGA and USB (it wakes
> every time I connect a USB, but my suspend script puts it to sleep
> again if it's closed and has no external VGA).
>
Then your motherboard keeps the bus in a lower power state such that it
is capable of causing a wake event when state changes. I'm also fairly
sure that when such a wake event happens, there is no indication to the
kernel about why it was woken up. Because of that, and the fact that
not all systems even support such wake modes, the kernel must reprobe
all hardware when it wakes up, and hopefully finds the same devices that
were there when it went to sleep. It does this for both types of suspend.
>> and in either case, there is nothing running on the CPU to monitor
>> device insertion/removal.
>
> You don't need the CPU, just a good USB controller and hubs with
> low-power modes and such. The fact that plugging in a USB
> keyboard/mouse and a VGA monitor is enough to wake the system when
> properly configured should be proof enough.
>
That is not proof of anything other than the bus controller has the
capability of generating a wake event. As I said before, once woken up,
the kernel must probe all hardware and if there is a new device, it will
find it. The hardware only detected that something changed and thus,
generated a wake even, it did not notice exactly what that change was
and inform the kernel.
>> When the system is resumed the kernel decides if the hardware has
>> changed the same way for either system: it probes the hardware to see
>> if it is still there. There isn't anything special that monitors
>> device insertion/removal while suspended to ram.
>
> Sometimes not, but again, it depends on the hardware.
Again, always not since the hardware doesn't actually tell the kernel
what happend, it just wakes it up.
>
> Cheers,
> Kyle Moffett
>
> --
> I have yet to see any problem, however complicated, which, when you
> looked at it in the right way, did not become still more complicated.
> -- Poul Anderson
>
>
Alan Stern wrote:
> Sorry, but you're wrong. First of all, testing if hardware is there is
> different from testing whether it has changed -- it could have changed
> while the system was asleep, with the result that hardware is indeed there
> but it's not the _same_ hardware.
>
>
If you believe I am wrong, then please offer proof. Specifically, refer
to the kernel code that handles verifying that the hardware is still
there after a resume. I very well may be wrong, but you can not simply
surmise this to be so without any proof. It is currently my belief that
the kernel probes the hardware the same way after a cold boot, resume
from s-t-r, and resume from s-t-d. Specifically it looks to see if a
device is located in the same bus location with the same serial number,
etc, and if so, access to it continues exactly where it left off prior
to the suspend.
This is why you can suspend to disk, and when you resume, your open
files are still valid; the disk is found to be still there, so it
continues to be accessible. Nothing gets clobbered as a result of the
disk seeming to be disconnected and reconnected. If that does happen,
it is a bug.
> Second, with USB at any rate, in addition to checking that hardware is
> still there, the kernel queries the USB controller to see if a disconnect
> occurred while the system was asleep. (If the controller wasn't powered
> during that time then it will report that every USB device was
> disconnected.)
>
AFAIK, there is no interface by which the kernel can query that
information from the controller, maybe you could show me? If that is
the case however, then I consider that to be a bug with the USB bus and
the kernel's handling of it. The kernel needs to be able to assume that
nothing was disconnected while it was shutdown, provided that the same
devices are there now as when it went to sleep. This is how it behaves
for say, SCSI disks in desktops/servers, where the controller certainly
is completely powered off. It should work the same for USB.
> Third, there is indeed something special that monitors USB device
> insertion/removal while suspended to RAM -- the USB host controller does
> so if it has suspend power.
>
>
Could you site references to that? AFAIK, the host controller is only
capable of generating a wake event when the bus state changes; it does
not have a means of informing the OS what has changed, aside from the OS
enumerating the devices on the bus again.
> No. It does work exactly as designed and it's not buggy. You just don't
> understand it.
>
>
If the mounted filesystem becomes corrupted over a hibernation because
the kernel thinks the drive was unplugged, then plugged back in, that
clearly is a bug. It does not do this for disks connected via other bus
types, and it clearly is undesirable to corrupt data in this way.
>> The state of the kernel after resuming from either suspend to disk, or
>> suspend to ram is the same. The filesystem is still mounted, and any
>> dirty pages in ram will be flushed just like normal. Whether the disk
>> is connected via SCSI, SATA, USB, or whatever does not matter.
>>
>
> Don't be silly. Dirty pages can't be flushed to disks that are no longer
> attached! And if a USB disk was unplugged while the system was asleep,
> the kernel will know that it is no longer attached. I don't know which
> other bus drivers check for this sort of thing.
>
>
The disk is still attached of course. The scenario we are talking about
does not actually involve disconnecting the drive. Since the drive is
not actually disconnected, if the kernel believes it has been, it's a bug.
>> The kernel knows that the same device is still there on the bus, and so
>> it picks up right where it left off. If it thinks the device has been
>> unplugged and reconnected, that is a bug.
>>
>
> It's not a bug if the device _has_ been unplugged and reconnected. When
> that happens, there's no way for the kernel to tell whether the device
> there now is the same as the device that used to be there.
>
>
Again, we're not talking about it actually being unplugged, though since
the kernel has no way of knowing it, you can unplug a scsi disk while
hibernated, then plug it back in before you resume, and it will work
just fine since the same type of hardware with the same serial number et
al is found in the same place on the bus.
On Mon, 13 Feb 2006, Phillip Susi wrote:
> If you believe I am wrong, then please offer proof. Specifically, refer
> to the kernel code that handles verifying that the hardware is still
> there after a resume. I very well may be wrong, but you can not simply
> surmise this to be so without any proof. It is currently my belief that
> the kernel probes the hardware the same way after a cold boot, resume
> from s-t-r, and resume from s-t-d. Specifically it looks to see if a
> device is located in the same bus location with the same serial number,
> etc, and if so, access to it continues exactly where it left off prior
> to the suspend.
Okay. Take a look at drivers/usb/core/hub.c. The usb_resume_device()
routine is called when resuming either from STR or STD. If
CONFIG_USB_SUSPEND has been set, it calls hub_port_resume(), which in turn
calls finish_device_resume(). Inside finish_device_resume() is a call to
usb_get_status(), which will fail if the device has not been connected and
powered-up throughout the entire suspend. That failure will cause
hub_port_resume() to call hub_port_logical_disconnect(), which has the
effect of doing a logical disconnect on the device.
There are other, redundant code paths that perform this check even when
CONFIG_USB_SUSPEND isn't set, but they are more difficult to describe.
For example, look at uhci_check_and_reset_hc() in
drivers/usb/host/pci-quirks.c.
You'll find that nowhere in the resume pathway does the kernel check
serial numbers or anything else of that nature. If the power session has
not been interrupted, that's sufficient proof that the device hasn't been
unplugged.
> This is why you can suspend to disk, and when you resume, your open
> files are still valid; the disk is found to be still there, so it
> continues to be accessible. Nothing gets clobbered as a result of the
> disk seeming to be disconnected and reconnected. If that does happen,
> it is a bug.
It may work that way with SCSI disks or IDE disks, which are not
hotpluggable. But it does happen with USB disks, and it's not a bug;
it's by design.
> > Second, with USB at any rate, in addition to checking that hardware is
> > still there, the kernel queries the USB controller to see if a disconnect
> > occurred while the system was asleep. (If the controller wasn't powered
> > during that time then it will report that every USB device was
> > disconnected.)
> >
>
> AFAIK, there is no interface by which the kernel can query that
> information from the controller, maybe you could show me?
Again, in hub.c look at hub_events(). It's a rather long routine, but at
some point you can see where it checks (portchange &
USB_PORT_STAT_C_CONNECTION). The constant stands for "Port Status Changed
Connection", meaning there has been a plug/unplug event. If the test
succeeds then connect_change is set to 1, causing
hub_port_connect_change() to be called. One of the first things that
routine does is call usb_disconnect() on the port's child device.
> If that is
> the case however, then I consider that to be a bug with the USB bus and
> the kernel's handling of it. The kernel needs to be able to assume that
> nothing was disconnected while it was shutdown, provided that the same
> devices are there now as when it went to sleep.
You've got it exactly backwards. The kernel doesn't need to make that
assumption because the hardware will _tell_ it whether anything was
disconnected. Rather, the kernel needs to _avoid_ making the assumption
that the device there now is the same as the device that was there before,
merely because a serial number (or something equivalent) happens to match.
Note: many USB mass storage devices don't have serial numbers.
> This is how it behaves
> for say, SCSI disks in desktops/servers, where the controller certainly
> is completely powered off. It should work the same for USB.
No it shouldn't. USB is a different kind of bus from SCSI; it has
different specifications and standards, and it should behave differently.
> > Third, there is indeed something special that monitors USB device
> > insertion/removal while suspended to RAM -- the USB host controller does
> > so if it has suspend power.
> >
> >
>
> Could you site references to that? AFAIK, the host controller is only
> capable of generating a wake event when the bus state changes; it does
> not have a means of informing the OS what has changed, aside from the OS
> enumerating the devices on the bus again.
Take a look, for example, at the UHCI specification (available from
<http://developer.intel.com/technology/usb/uhci11d.htm>). Section 2.1.7
describes the Port Status and Control register, which indicates (for each
port) whether a connect-change event has taken place, as well as many
other things.
> If the mounted filesystem becomes corrupted over a hibernation because
> the kernel thinks the drive was unplugged, then plugged back in, that
> clearly is a bug. It does not do this for disks connected via other bus
> types, and it clearly is undesirable to corrupt data in this way.
I agree that it's annoying and undesirable, but it's not a bug. Other
buses would work this way too if they were hotpluggable, like USB.
> > It's not a bug if the device _has_ been unplugged and reconnected. When
> > that happens, there's no way for the kernel to tell whether the device
> > there now is the same as the device that used to be there.
> >
> >
>
> Again, we're not talking about it actually being unplugged, though since
> the kernel has no way of knowing it, you can unplug a scsi disk while
> hibernated, then plug it back in before you resume, and it will work
> just fine since the same type of hardware with the same serial number et
> al is found in the same place on the bus.
As I said above, SCSI isn't the same as USB.
By the way, usb-storage in 2.4 used to work (still does, in fact) more
along the lines you're describing. You could unplug a drive and the
kernel's disk data structures would be kept intact. Later on, when you
replugged the drive it would be re-associated with those data structures,
using some not-go-great heuristics for trying to find a match, and you
could pick up where you left off.
Then at some point during the 2.5 development sequence, Linus put his foot
down. He said that when a device goes away, it's _gone_! That was the
end of it. Ever since, unplugging a USB drive (or any other kind of USB
device) causes all its device structures to be released.
Alan Stern
Alan Stern wrote:
> Okay. Take a look at drivers/usb/core/hub.c. The usb_resume_device()
> routine is called when resuming either from STR or STD. If
> CONFIG_USB_SUSPEND has been set, it calls hub_port_resume(), which in turn
> calls finish_device_resume(). Inside finish_device_resume() is a call to
> usb_get_status(), which will fail if the device has not been connected and
> powered-up throughout the entire suspend. That failure will cause
> hub_port_resume() to call hub_port_logical_disconnect(), which has the
> effect of doing a logical disconnect on the device.
>
Interesting. How does usb_get_status() decide if the device has been
connected or not the entire time? And do you not agree that if it
indicates that the device was disconnected during hibernation, when it
in fact, was not, that is a bug?
> There are other, redundant code paths that perform this check even when
> CONFIG_USB_SUSPEND isn't set, but they are more difficult to describe.
> For example, look at uhci_check_and_reset_hc() in
> drivers/usb/host/pci-quirks.c.
>
> You'll find that nowhere in the resume pathway does the kernel check
> serial numbers or anything else of that nature. If the power session has
> not been interrupted, that's sufficient proof that the device hasn't been
> unplugged.
>
>
And what if the entire bus was completely powered off, which some
systems do? I know my SCSI bus is completely powered off during
hibernation, yet the kernel has no problem figuring out that the same
devices are still connected after resume, so it doesn't generate a
disconnect event.
> It may work that way with SCSI disks or IDE disks, which are not
> hotpluggable. But it does happen with USB disks, and it's not a bug;
> it's by design.
>
SCSI and IDE very well can be hot pluggable. I have hot plugged
external SCSI devices numerous times, and even internal IDE drives a
time or two. Clearly if the kernel thinks you disconnected your drive
and causes data loss, when this is not true, it is a bug. If it is a
flaw by design, that is still a class of bug.
> Again, in hub.c look at hub_events(). It's a rather long routine, but at
> some point you can see where it checks (portchange &
> USB_PORT_STAT_C_CONNECTION). The constant stands for "Port Status Changed
> Connection", meaning there has been a plug/unplug event. If the test
> succeeds then connect_change is set to 1, causing
> hub_port_connect_change() to be called. One of the first things that
> routine does is call usb_disconnect() on the port's child device.
>
>
Interesting again. This bit exists for each node on the bus and is
tracked by the hardware? If that is the case, and the hardware is
informing the kernel that all devices were disconnected during
hibernation when this is not the case, then this clearly is a bug in the
hardware, and the kernel possibly should work around it knowing that the
hardware lies.
> You've got it exactly backwards. The kernel doesn't need to make that
> assumption because the hardware will _tell_ it whether anything was
>
Even if some hardware does, there is a lot of hardware that does not.
If hardware that is capable of delivering this information does so in an
unreliable manner ( i.e. it lies and says everything is disconnected
when it isn't ) then the kernel should ignore that information.
> disconnected. Rather, the kernel needs to _avoid_ making the assumption
> that the device there now is the same as the device that was there before,
> merely because a serial number (or something equivalent) happens to match.
> Note: many USB mass storage devices don't have serial numbers.
>
>
If the hardware is capable of accurately and reliably informing the
kernel about this information, then that would be useful, but if it is
not, then seeing a device that appears to be the same as the one you
expect to be there is good enough to decide to continue using it rather
than force data loss. It works fine like that for SCSI and IDE because
the user expects the system to work properly after they suspend/resume
when they don't mess with the hardware. If you actually disconnect a
device and replace it with another one that otherwise looks the same,
but isn't really, then all bets are off.
>> This is how it behaves
>> for say, SCSI disks in desktops/servers, where the controller certainly
>> is completely powered off. It should work the same for USB.
>>
>
> No it shouldn't. USB is a different kind of bus from SCSI; it has
> different specifications and standards, and it should behave differently.
>
>
Not if "differently" means "causes data loss when hibernating".
> Take a look, for example, at the UHCI specification (available from
> <http://developer.intel.com/technology/usb/uhci11d.htm>). Section 2.1.7
> describes the Port Status and Control register, which indicates (for each
> port) whether a connect-change event has taken place, as well as many
> other things.
>
>
I'll have to read up on that. If that is the case, then it seems the
hardware is broken because it incorrectly indicates disconnects that did
not actually happen. If this is a known problem, then the kernel should
work around it to avoid data loss.
>
> I agree that it's annoying and undesirable, but it's not a bug. Other
> buses would work this way too if they were hotpluggable, like USB.
>
Other busses can be hot plugged in this way without causing incorrect
disconnect events and data loss over hibernation. The fact that USB
causes data loss in the face of such a benegin event as hibernating and
resuming ( with no actual hardware change ) is a bug. Claiming that
data loss is an acceptable price to pay for being able to hot plug is
silly.
> As I said above, SCSI isn't the same as USB.
>
Right... which doesn't make this any less of a bug in the USB stack.
> By the way, usb-storage in 2.4 used to work (still does, in fact) more
> along the lines you're describing. You could unplug a drive and the
> kernel's disk data structures would be kept intact. Later on, when you
> replugged the drive it would be re-associated with those data structures,
> using some not-go-great heuristics for trying to find a match, and you
> could pick up where you left off.
>
> Then at some point during the 2.5 development sequence, Linus put his foot
> down. He said that when a device goes away, it's _gone_! That was the
> end of it. Ever since, unplugging a USB drive (or any other kind of USB
> device) causes all its device structures to be released.
>
If you ACTUALLY unplug the drive, that's fine... if you don't though,
and the kernel thinks you did, that is a bug. If the kernel can
reasonably decide that the drive has not actually been unplugged even
though the busted hardware indicates it has, then it should not generate
a disconnect.
On Mon, 13 Feb 2006, Phillip Susi wrote:
> Alan Stern wrote:
> > Okay. Take a look at drivers/usb/core/hub.c. The usb_resume_device()
> > routine is called when resuming either from STR or STD. If
> > CONFIG_USB_SUSPEND has been set, it calls hub_port_resume(), which in turn
> > calls finish_device_resume(). Inside finish_device_resume() is a call to
> > usb_get_status(), which will fail if the device has not been connected and
> > powered-up throughout the entire suspend. That failure will cause
> > hub_port_resume() to call hub_port_logical_disconnect(), which has the
> > effect of doing a logical disconnect on the device.
> >
>
> Interesting. How does usb_get_status() decide if the device has been
> connected or not the entire time?
It doesn't decide that -- the device itself does. If the device was
connected the entire time then it will respond properly. If it was
disconnected then it will reset itself, losing its address. Hence it will
not reply to further requests at the old address. usb_get_status() simply
indicates whether or not a response was received.
> And do you not agree that if it
> indicates that the device was disconnected during hibernation, when it
> in fact, was not, that is a bug?
Note: By "disconnected", I mean that the power session was interrupted.
So even if the cable remained plugged in, if the bus suspend power wasn't
present then the device was disconnected. Note also that it is impossible
to tell whether the cable has been unplugged -- the hardware is capable of
detecting only whether or not the power session was interrupted.
Given those caveats, yes, I agree that the routine should not indicate the
device was disconnected if in fact it wasn't.
> And what if the entire bus was completely powered off, which some
> systems do? I know my SCSI bus is completely powered off during
> hibernation, yet the kernel has no problem figuring out that the same
> devices are still connected after resume, so it doesn't generate a
> disconnect event.
Does the kernel have any problem figuring out when a _different_ device
of the same type is connected at the old address after resume?
With USB, if the entire bus is powered off then every device on it is
automatically disconnected. By definition.
> > It may work that way with SCSI disks or IDE disks, which are not
> > hotpluggable. But it does happen with USB disks, and it's not a bug;
> > it's by design.
> >
> SCSI and IDE very well can be hot pluggable. I have hot plugged
> external SCSI devices numerous times, and even internal IDE drives a
> time or two.
Have you tried unplugging a SCSI or IDE drive while it was mounted and the
system was suspended, and then plugging in a different drive in its place?
> Clearly if the kernel thinks you disconnected your drive
> and causes data loss, when this is not true, it is a bug. If it is a
> flaw by design, that is still a class of bug.
No. A bug is unintentional whereas a design flaw is intentional.
You are ignoring the question of how the kernel can tell whether two
devices are in fact the same. There is no safe way to do this, other than
having the hardware verify that the device was connected the whole time.
> > Again, in hub.c look at hub_events(). It's a rather long routine, but at
> > some point you can see where it checks (portchange &
> > USB_PORT_STAT_C_CONNECTION). The constant stands for "Port Status Changed
> > Connection", meaning there has been a plug/unplug event. If the test
> > succeeds then connect_change is set to 1, causing
> > hub_port_connect_change() to be called. One of the first things that
> > routine does is call usb_disconnect() on the port's child device.
> >
> >
> Interesting again. This bit exists for each node on the bus and is
> tracked by the hardware?
Yes. The registers in the host controller only keep track of devices
plugged directly into the computer. Similar registers in external hubs
keep track of the devices plugged into them.
> If that is the case, and the hardware is
> informing the kernel that all devices were disconnected during
> hibernation when this is not the case, then this clearly is a bug in the
> hardware, and the kernel possibly should work around it knowing that the
> hardware lies.
It's a matter of definition. By definition, "disconnected" means
essentially the same thing as "power interrupted". If you use the wrong
definition, of course you will think that the hardware lies.
> > You've got it exactly backwards. The kernel doesn't need to make that
> > assumption because the hardware will _tell_ it whether anything was
> >
> Even if some hardware does, there is a lot of hardware that does not.
> If hardware that is capable of delivering this information does so in an
> unreliable manner ( i.e. it lies and says everything is disconnected
> when it isn't ) then the kernel should ignore that information.
But it is very reliable. And all USB hardware does it; it's part of the
specification.
> If the hardware is capable of accurately and reliably informing the
> kernel about this information, then that would be useful, but if it is
> not, then seeing a device that appears to be the same as the one you
> expect to be there is good enough to decide to continue using it rather
> than force data loss. It works fine like that for SCSI and IDE because
> the user expects the system to work properly after they suspend/resume
> when they don't mess with the hardware. If you actually disconnect a
> device and replace it with another one that otherwise looks the same,
> but isn't really, then all bets are off.
Indeed. With USB, many devices look the same.
> > No it shouldn't. USB is a different kind of bus from SCSI; it has
> > different specifications and standards, and it should behave differently.
> >
> >
> Not if "differently" means "causes data loss when hibernating".
It works both ways. What about "causes data loss when a different device
is plugged in"?
> Other busses can be hot plugged in this way without causing incorrect
> disconnect events and data loss over hibernation. The fact that USB
> causes data loss in the face of such a benegin event as hibernating and
> resuming ( with no actual hardware change ) is a bug. Claiming that
> data loss is an acceptable price to pay for being able to hot plug is
> silly.
> If you ACTUALLY unplug the drive, that's fine... if you don't though,
> and the kernel thinks you did, that is a bug. If the kernel can
> reasonably decide that the drive has not actually been unplugged even
> though the busted hardware indicates it has, then it should not generate
> a disconnect.
You are complaining because you don't like the way USB was designed.
That's fine, but it leaves you advocating a non-standardized position.
Can you suggest a _reliable_ way to tell if the USB device present at a
port after resuming is the same device as was there before suspending?
Alan Stern
Hi,
On Monday 13 February 2006 22:24, Alan Stern wrote:
> On Mon, 13 Feb 2006, Phillip Susi wrote:
}-- snip --{
> You are complaining because you don't like the way USB was designed.
> That's fine, but it leaves you advocating a non-standardized position.
>
> Can you suggest a _reliable_ way to tell if the USB device present at a
> port after resuming is the same device as was there before suspending?
It seems to follow from your discussion that if I have a mounted filesystem
on a USB device and I suspend to disk, I can lose data unless the filesystem
has been mounted with "sync".
If this is the case, there should be a big fat warning in the swsusp
documentation, but there's nothing like that in there (at lease I can't find
it easily).
[If this is not the case, I've missed something and sorry for the noise.]
Greetings,
Rafael
On Mon, Feb 13, 2006 at 04:24:30PM -0500, Alan Stern wrote:
> Can you suggest a _reliable_ way to tell if the USB device present at a
> port after resuming is the same device as was there before suspending?
That's not really enough, is it? What if you suspend, unplug your usb
flash drive, plug it into your camera, take a few pictures, plug it back
into the camera, then resume?
If you don't throw away the old mount, that sounds equivalent to
allowing writes to /dev/hda1 while it has a local filesystem mounted on
it. There may be limits to what you can do to prevent that in this
case, but it seems best to err on the side of caution.
--b.
Alan Stern wrote:
> It doesn't decide that -- the device itself does. If the device was
> connected the entire time then it will respond properly. If it was
> disconnected then it will reset itself, losing its address. Hence it will
> not reply to further requests at the old address. usb_get_status() simply
> indicates whether or not a response was received.
>
I see. Then this information is unreliable and should not be trusted (
when resuming from a suspend ), as it leads to incorrect behavior.
> Note: By "disconnected", I mean that the power session was interrupted.
> So even if the cable remained plugged in, if the bus suspend power wasn't
> present then the device was disconnected. Note also that it is impossible
> to tell whether the cable has been unplugged -- the hardware is capable of
> detecting only whether or not the power session was interrupted.
>
> Given those caveats, yes, I agree that the routine should not indicate the
> device was disconnected if in fact it wasn't.
>
>
Exactly. Yes, there is no good way to determine _for certain_ that the
user did no do something stupid, such as replace the drive with another
one just like it, or change the contents in another machine, but that is
no reason to assume that the user DID do something like that, and break
the mount, when they in fact, did nothing of the sort.
> Does the kernel have any problem figuring out when a _different_ device
> of the same type is connected at the old address after resume?
>
> With USB, if the entire bus is powered off then every device on it is
> automatically disconnected. By definition.
>
>
Then the kernel needs to ignore those disconnect events ( provided that
the device appears to still actually be there ) because they are false
and lead to data loss.
> Have you tried unplugging a SCSI or IDE drive while it was mounted and the
> system was suspended, and then plugging in a different drive in its place?
>
No, because that would be a foolish user error.
> No. A bug is unintentional whereas a design flaw is intentional.
>
It doesn't matter if it is broken by design, or broken by
implementation; it's still broken.
> You are ignoring the question of how the kernel can tell whether two
> devices are in fact the same. There is no safe way to do this, other than
> having the hardware verify that the device was connected the whole time.
>
>
If there is no better way to tell for sure that the device that is now
there is not the same as the one that was there, then the kernel must
assume the user did not do something stupid and continue to use the
device as if it was not disconnected ( because odds are, it in fact, was
not ). In other words, which is more safe? ALLWAYS loosing data
because you can't be absolutely sure that the device is the same, or
only loosing data if the user does something as foolish as swapping
drives while suspended, and you can't tell they did?
> It's a matter of definition. By definition, "disconnected" means
> essentially the same thing as "power interrupted". If you use the wrong
> definition, of course you will think that the hardware lies.
>
>
And that appears to be exactly what the kernel is doing; interpreting
the hardware "power interrupted" flag as "disconnected", which leads to
broken behavior and data loss.
>
> But it is very reliable. And all USB hardware does it; it's part of the
> specification.
>
Except when the bus is powered down, in which case, using that flag as
an indicator that the device has in fact changed, is not reliable.
> It works both ways. What about "causes data loss when a different device
> is plugged in"?
>
>
Again, that's user error, not normal operation. You shouldn't go
changing devices around while suspended because it _might_ confuse the
kernel. The way things are now, the kernel _will_ get confused when the
user does nothing wrong.
> You are complaining because you don't like the way USB was designed.
> That's fine, but it leaves you advocating a non-standardized position.
>
No, I am complaining because the kernel interprets the notice that the
bus gives that the device _may_ have changed as a notice that the device
in fact, _has_ changed, and causes data loss when nothing is actually
wrong, and this could easily be avoided.
> Can you suggest a _reliable_ way to tell if the USB device present at a
> port after resuming is the same device as was there before suspending?
>
> Alan Stern
>
Again, if it appears to be the same as best you can tell, there is no
reason to require absolute certainty that it actually is; you can assume
that the user did not do foolish things while suspended. This is a much
safer assumption than always assuming that they DID because that always
leads to data loss, whereas assuming they didn't ( which they won't most
of the time ) doesn't lead to data loss when they didn't.
On Feb 13, 2006, at 18:47:41, Phillip Susi wrote:
> Alan Stern wrote:
>> It doesn't decide that -- the device itself does. If the device
>> was connected the entire time then it will respond properly. If
>> it was disconnected then it will reset itself, losing its
>> address. Hence it will not reply to further requests at the old
>> address. usb_get_status() simply indicates whether or not a
>> response was received.
>
> I see. Then this information is unreliable and should not be
> trusted ( when resuming from a suspend ), as it leads to incorrect
> behavior.
No, that information is the most reliable that can be obtained. It
tells us that we can no longer make any guarantees about the device
or its state. The USB spec is quite clear on this point.
>> Note: By "disconnected", I mean that the power session was
>> interrupted. So even if the cable remained plugged in, if the bus
>> suspend power wasn't present then the device was disconnected.
>> Note also that it is impossible to tell whether the cable has been
>> unplugged -- the hardware is capable of detecting only whether or
>> not the power session was interrupted.
>>
>> Given those caveats, yes, I agree that the routine should not
>> indicate the device was disconnected if in fact it wasn't.
>
> Exactly. Yes, there is no good way to determine _for certain_ that
> the user did no do something stupid, such as replace the drive with
> another one just like it, or change the contents in another
> machine, but that is no reason to assume that the user DID do
> something like that, and break the mount, when they in fact, did
> nothing of the sort.
Except we can't reliably decide that. Say I plug my USB camera in,
mount it, and download some pictures. I then suspend the computer,
unplug the camera after suspending, take more pictures, plug it back
in and resume. That's a fairly reasonable situation and the computer
considering the camera's state to be unchanged would be a serious bug
and probably result in data loss. By contrast, just considering the
camera to be spontaneously unplugged would cause no more data loss
than actually spontaneously unplugging the flash drive.
>> Does the kernel have any problem figuring out when a _different_
>> device of the same type is connected at the old address after resume?
>>
>> With USB, if the entire bus is powered off then every device on it
>> is automatically disconnected. By definition.
>
> Then the kernel needs to ignore those disconnect events ( provided
> that the device appears to still actually be there ) because they
> are false and lead to data loss.
This is why hardware suspend is a good thing. When I suspend and
resume my laptop, there are _no_ USB disconnects. The controller
puts all the hubs into low-power mode, but it never disconnects them
or causes problems.
>> You are complaining because you don't like the way USB was
>> designed. That's fine, but it leaves you advocating a non-
>> standardized position.
>
> No, I am complaining because the kernel interprets the notice that
> the bus gives that the device _may_ have changed as a notice that
> the device in fact, _has_ changed,
No, you have this wrong. The USB spec indicates that the device
_HAS_ changed and all old state should be thrown away (even the
address). There is no way around that issue. USB was designed to
support hardware suspend; you can put all the hardware in low-power
mode and still be able to detect changes.
In fact, I would argue that turning off all the busses completely
when you want to maintain a connection to a device is broken. If you
want to maintain the connection, you should keep the busses powered.
Otherwise, according to the USB spec, it's the _kernel_ that is
terminating the connection, and assuming that it exists after
explicitly terminating it is wrong.
Cheers,
Kyle Moffett
Kyle Moffett wrote:
>
> No, that information is the most reliable that can be obtained. It
> tells us that we can no longer make any guarantees about the device or
> its state. The USB spec is quite clear on this point.
>
That is what it says but the kernel is interpreting it as "this device
HAS been removed" rather than "this device MAY have been removed". That
is wrong and should be fixed.
>
> Except we can't reliably decide that. Say I plug my USB camera in,
> mount it, and download some pictures. I then suspend the computer,
> unplug the camera after suspending, take more pictures, plug it back in
> and resume. That's a fairly reasonable situation and the computer
> considering the camera's state to be unchanged would be a serious bug
> and probably result in data loss. By contrast, just considering the
> camera to be spontaneously unplugged would cause no more data loss than
> actually spontaneously unplugging the flash drive.
>
But again, that would be user error and thus, the data loss can not be
avoided and is their fault. Having data loss result from user error is
far more acceptable than having data loss ALLWAYS result from a
perfectly acceptable user action, namely hibernating the machine and
resuming it some time later without altering anything. You already
teach users not to unplug the drive without ejecting it from the desktop
first, why should you also force them to eject before hibernating?
>
> This is why hardware suspend is a good thing. When I suspend and resume
> my laptop, there are _no_ USB disconnects. The controller puts all the
> hubs into low-power mode, but it never disconnects them or causes problems.
>
That's fantastic for your system, but not all systems will maintain
standby power to USB, and users expect to be able to suspend to disk and
not loose data, just like they do with non USB drives.
>
> No, you have this wrong. The USB spec indicates that the device _HAS_
> changed and all old state should be thrown away (even the address).
> There is no way around that issue. USB was designed to support hardware
> suspend; you can put all the hardware in low-power mode and still be
> able to detect changes.
That's great, except that feature is not always used so you must be able
to live without it. The fact that the hardware flag is set is no
indication that the hardware HAS changed, you said so yourself; all it
knows is that the bus/device lost power. The use case we are talking
about is one in which power loss happens, but the device is still the
same, and so access to it should not be interrupted.
>
> In fact, I would argue that turning off all the busses completely when
> you want to maintain a connection to a device is broken. If you want to
> maintain the connection, you should keep the busses powered. Otherwise,
> according to the USB spec, it's the _kernel_ that is terminating the
> connection, and assuming that it exists after explicitly terminating it
> is wrong.
>
Yes, assuming that it exists is wrong. Probing the hardware and seeing
that it exists and is _probably_ the same device is entirely different.
In that case it is preferable to assume the probable case rather than
the improbable one because it will lead to less data loss.
On Feb 13, 2006, at 21:09, Phillip Susi wrote:
> Kyle Moffett wrote:
>> No, that information is the most reliable that can be obtained.
>> It tells us that we can no longer make any guarantees about the
>> device or its state. The USB spec is quite clear on this point.
>
> That is what it says but the kernel is interpreting it as "this
> device HAS been removed" rather than "this device MAY have been
> removed". That is wrong and should be fixed.
No, that's _exactly_ what the spec says (well, not verbatim but close
enough). When you disconnect, both the master and slave devices are
perfectly free to assume that the connection is completely broken and
no state is maintained. Anything that breaks that assumption is
against the spec and likely to break in odd scenarios.
> [multiple data-loss arguments]
Which causes worse data-loss, writing out cached pages and filesystem
metadata to a filesystem that has changed in the mean-time (possibly
allocating those for metadata, etc) or forcibly unmounting it as
though the user pulled the cable? Most filesystems are designed to
handle the latter (it's the same as a hard-shutdown), whereas _none_
are designed to handle the former.
A good set of suspend scripts should handle the hardware-suspend with
no extra work because hardware supporting hardware-suspend basically
inevitably supports USB low-power-mode, and handle software-suspend
by either forcibly syncing and unmounting USB filesystems or by
failing the suspend and asking the user to. You also could patch the
kernel to fail a powerdown software suspend if some USB device is
mounted or otherwise unremovably in-use.
>> In fact, I would argue that turning off all the busses completely
>> when you want to maintain a connection to a device is broken. If
>> you want to maintain the connection, you should keep the busses
>> powered. Otherwise, according to the USB spec, it's the _kernel_
>> that is terminating the connection, and assuming that it exists
>> after explicitly terminating it is wrong.
>
> Yes, assuming that it exists is wrong. Probing the hardware and
> seeing that it exists and is _probably_ the same device is entirely
> different. In that case it is preferable to assume the probable
> case rather than the improbable one because it will lead to less
> data loss.
Except that would make Linux broken with respect to the USB spec. It
is fallacious to assume that a USB device that the kernel has told to
disconnect will still have the same state when the kernel tries to
reconnect, even _if_ you could reliably identify it (which you can't
because there is no serial number of any sort on a lot of devices.
Cheers,
Kyle Moffett
--
I lost interest in "blade servers" when I found they didn't throw
knives at people who weren't supposed to be in your machine room.
-- Anthony de Boer
On Mon, 13 Feb 2006, Kyle Moffett wrote:
> Which causes worse data-loss, writing out cached pages and filesystem
> metadata to a filesystem that has changed in the mean-time (possibly
> allocating those for metadata, etc) or forcibly unmounting it as
> though the user pulled the cable? Most filesystems are designed to
> handle the latter (it's the same as a hard-shutdown), whereas _none_
> are designed to handle the former.
That's a good point. Furthermore, any decent suspend script will flush
all dirty buffers to disk before suspending anything.
> A good set of suspend scripts should handle the hardware-suspend with
> no extra work because hardware supporting hardware-suspend basically
> inevitably supports USB low-power-mode,
Unfortunately a lot of hardware doesn't support USB low-power mode. I
guess you'd say therefore it doesn't really support hardware-suspend.
This may be so, but it's small comfort to the owners of those systems.
I have to admit, although technically Phillip's argument is wrong, from a
useability standpoint it is right. Windows allows users to disconnect and
reconnect USB storage devices while the system is hibernating, with no
apparent ill effects -- although I've never tried to unplug one device and
then plug in a different one on the same port while the computer was
asleep. I don't know to what extent Windows checks descriptors/serial
numbers/disk labels/whatever when it wakes up.
Alan Stern
On Feb 13, 2006, at 23:28, Alan Stern wrote:
> On Mon, 13 Feb 2006, Kyle Moffett wrote:
>> A good set of suspend scripts should handle the hardware-suspend
>> with no extra work because hardware supporting hardware-suspend
>> basically inevitably supports USB low-power-mode,
>
> Unfortunately a lot of hardware doesn't support USB low-power
> mode. I guess you'd say therefore it doesn't really support
> hardware-suspend. This may be so, but it's small comfort to the
> owners of those systems.
>
> I have to admit, although technically Phillip's argument is wrong,
> from a usability standpoint it is right. Windows allows users to
> disconnect and reconnect USB storage devices while the system is
> hibernating, with no apparent ill effects -- although I've never
> tried to unplug one device and then plug in a different one on the
> same port while the computer was asleep. I don't know to what
> extent Windows checks descriptors/serial numbers/disk labels/
> whatever when it wakes up.
For the software-suspend/no-low-power-mode case, I see a couple of
practical and spec-conforming options:
1) The kernel should notice that it has a filesystem mounted from a
hotpluggable block device and abort the suspend process. This isn't
terribly user friendly, but is guaranteed to prevent data loss, and a
good set of suspend scripts could notice the reason for failure and
report it to the user (optionally unmounting the filesystems
automatically and retrying).
2) The kernel should notice that it has a filesystem mounted from a
hotpluggable block device and forcibly unmount said filesystem. This
is also not user-friendly, and has the disadvantage of not being
easily userspace-controllable.
3) The kernel should notice that it has a filesystem mounted from a
hotpluggable block device and forcibly disconnect the mount.
Beforehand, uswsusp would have saved information about all mounted
blockdevs into the suspend file/disk. When resuming, early userspace
would reread that information and attempt to relocate the block
devices from userspace, using any tools available to it at the time
(including a bunch of fs-probing tools and such). After it's scanned
devices and found any that it could reliably get, it would pass that
information to the kernel being resumed which would use it to
reattach filesystems to disks. This is a lot more complicated, but
more user friendly. It has the downside of making the kernel do a
lot of extra unreliable work looking up paths and files again, but it
might work in a good percentage of the cases. I doubt the advantages
of this one over (1) or (2) are worth the added complexity, though.
Cheers,
Kyle Moffett
--
Simple things should be simple and complex things should be possible
-- Alan Kay
Kyle Moffett wrote:
> No, that's _exactly_ what the spec says (well, not verbatim but close
> enough). When you disconnect, both the master and slave devices are
> perfectly free to assume that the connection is completely broken and no
> state is maintained. Anything that breaks that assumption is against
> the spec and likely to break in odd scenarios.
>
Perfectly free to != required to. When you know that the connection is
"broken" due to suspend and there is no way to differentiate between
that and the device really being disconnected, you have two choices:
1) Assume the user is a masochistic idiot and replaced the device with
one that looks just like it, or connected the device somewhere else and
modified it before replacing it, or
2) Assume the user is sane and did no such thing.
Assumption #2 will be correct most of the time and is a valid use case,
the first is not. If either assumption is wrong, it will lead to data
loss. All else being equal, which case do you optimize for? The common
and correct case, or the uncommon error case?
>
> Which causes worse data-loss, writing out cached pages and filesystem
> metadata to a filesystem that has changed in the mean-time (possibly
> allocating those for metadata, etc) or forcibly unmounting it as though
> the user pulled the cable? Most filesystems are designed to handle the
> latter (it's the same as a hard-shutdown), whereas _none_ are designed
> to handle the former.
>
So you condemn the common correct use case to always suffer data loss to
give _slightly_ better protection to the uncommon and incorrect use
case? I think most users prefer a system that works right when you use
it right to one that doesn't break quite as badly when you do something
stupid.
Also why is this argument more valid for USB than SCSI? I am just as
free to unplug a scsi disk and replace it with a different one while
hibernated, yet I don't suffer data loss when I don't do such
foolishness. By your argument, because the kernel can not know for sure
that I didn't do that, it must disconnect and break all the drives on
the bus.
> A good set of suspend scripts should handle the hardware-suspend with no
> extra work because hardware supporting hardware-suspend basically
> inevitably supports USB low-power-mode, and handle software-suspend by
> either forcibly syncing and unmounting USB filesystems or by failing the
> suspend and asking the user to. You also could patch the kernel to fail
> a powerdown software suspend if some USB device is mounted or otherwise
> unremovably in-use.
>
That would prevent data loss, so it would be better than the current
method of silently loosing data because you assume the user is out to
get you. There are other downsides to it though. What if someone wants
to boot from a usb drive and suspend to disk?
>
> Except that would make Linux broken with respect to the USB spec. It is
Please point out where in the USB spec it states that the OS _MUST_
assume that the device is no longer the same after a bus shutdown and
restart, even though it otherwise appears to be.
> fallacious to assume that a USB device that the kernel has told to
> disconnect will still have the same state when the kernel tries to
> reconnect, even _if_ you could reliably identify it (which you can't
> because there is no serial number of any sort on a lot of devices.
>
It is worse to assume that it HAS changed, when that is guaranteed to
lead to data loss even though it most likely has not changed. This is
no different than assuming that the user knows what they are doing when
they fdisk a drive. You don't refuse to fdisk a drive just because you
can't be 100% certain that they aren't doing something stupid that _may_
result in data loss, so you assume they know what they are doing and do
as they say.
On Sunday 12 February 2006 20:19, Alan Stern wrote:
> On Sun, 12 Feb 2006, Phillip Susi wrote:
> > Alan Stern wrote:
> > > Both of you are missing an important difference between Suspend-to-RAM
> > > and Suspend-to-Disk.
> > >
> > > Suspend-to-RAM is a true suspend operation, in that the hardware's
> > > state is maintained _in the hardware_. External buses like USB will
> > > retain suspend power, for instance (assuming the motherboard supports
> > > it; some don't).
> > >
> > > Suspend-to-Disk, by contrast, is _not_ a true suspend. It can more
> > > accurately be described as checkpoint-and-turn-off. Hardware state is
> > > not maintained. (Some systems may support a special ACPI state that
> > > does maintain suspend power to external buses during shutdown, I forget
> > > what it's called. And I down't know whether swsusp uses this state.)
> >
> > I would disagree. The only difference between the two is WHERE the
> > state is maintained - ram vs. disk. I won't really argue it though,
> > because it's just semantics -- call it whatever you want.
>
> It's not just semantics. There's a real difference between maintaining
> state in the hardware and maintaining it somewhere else. The biggest
> difference is that if the hardware retains suspend power, it is able to
> detect disconnections. When the system resumes, it _knows_ whether a
> device was attached the entire time, as opposed to being unplugged and
> replugged (or possibly a different device plugged in!) while the system
> was asleep. If the hardware is down completely, there is no way of
> telling for certain whether a device attached to some port is the same one
> that was there when the system got suspended.
>
> Another difference is the possibility of remote wakeup. Clearly it can't
> happen when there's no power available.
Well... sort of. I know that my notebook can be brought up over LAN when it is
off - suspended to disk or otherwise.
>
> > > So for example, let's say you have a filesystem mounted on a USB flash
> > > or disk drive. With Suspend-to-RAM, there's a very good chance that
> > > the connection and filesystem will still be intact when you resume.
> > > With Suspend-to-Disk, the USB connection will terminate when the
> > > computer shuts down. When you resume, the device will be gone and your
> > > filesystem will be screwed.
> >
> > This is not true. The USB bus is shut down either way, and provided
> > that you have not unplugged the disk, nothing will be screwed when you
> > resume from disk or ram.
>
> Have you actually tried it? I have.
>
> In any case, it is undeniably true that if the bus is shut down then all
> the USB connections are lost. When you resume it will be the same as if
> you had unplugged all the USB devices and then replugged them. Not a good
> thing to do when they contain mounted filesystems; all the memory mappings
> are invalidated.
It all depends on the machine I guess. Mine keeps even the CD drive running
when it's off!
>
> (Bear in mind that whether a USB bus gets shut down depends on the
> motherboard; some supply suspend power and some don't. It depends on the
> USB controller too; some support low-power states other than "completely
> off" and others don't.)
>
> Alan Stern
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
--hackmiester
If you can read this, you don't need glasses.
On Tue, 14 Feb 2006, Kyle Moffett wrote:
> For the software-suspend/no-low-power-mode case, I see a couple of
> practical and spec-conforming options:
>
> 1) The kernel should notice that it has a filesystem mounted from a
> hotpluggable block device and abort the suspend process. This isn't
> terribly user friendly, but is guaranteed to prevent data loss, and a
> good set of suspend scripts could notice the reason for failure and
> report it to the user (optionally unmounting the filesystems
> automatically and retrying).
...
There are some difficulties connected with these suggestions. They're
maybe not impossible, but it would be tricky.
For one thing, how does one go about telling at the USB level whether one
of the devices contains a mounted filesystem? Or any open file references
at all, for that matter? No doubt there's a way to do it, but I don't
know what it is.
For another, hardware behavior is very idiosyncratic. It's sometimes
possible to tell that a particular system doesn't supply suspend power to
a USB controller (because, for instance, the controller doesn't support
PCI PM or doesn't support D3hot), but this is far from reliable -- I've
seen mistakes both ways. It's not as simple as "power available during
STR, not available during STD".
Alan Stern
On Feb 14, 2006, at 01:27, Phillip Susi wrote:
> Kyle Moffett wrote:
>> No, that's _exactly_ what the spec says (well, not verbatim but
>> close enough). When you disconnect, both the master and slave
>> devices are perfectly free to assume that the connection is
>> completely broken and no state is maintained. Anything that
>> breaks that assumption is against the spec and likely to break in
>> odd scenarios.
>
> Perfectly free to != required to.
In this case they _are_ equivalent due to symmetry. If the other
device _may_ assume that the connection is broken, then you _must_
assume that the connection is broken. Since either device _may_
assume that, both devices therefore _must_ according to spec.
>> Which causes worse data-loss, writing out cached pages and
>> filesystem metadata to a filesystem that has changed in the mean-
>> time (possibly allocating those for metadata, etc) or forcibly
>> unmounting it as though the user pulled the cable? Most
>> filesystems are designed to handle the latter (it's the same as a
>> hard-shutdown), whereas _none_ are designed to handle the former.
>
> So you condemn the common correct use case to always suffer data
> loss to give _slightly_ better protection to the uncommon and
> incorrect use case?
No, as I said before, a good set of USB suspend scripts can solve
this problem. All of the ones I am aware of *now* already sync all
data, which is good enough to prevent data-loss in _every_ case where
the device is spontaneously unplugged. On the other hand, this is
_never_ good enough if the device is accidentally switched underneath
us while suspended (and that's not so terribly uncommon, I know a lot
of people who would do that accidentally, myself included).
> I think most users prefer a system that works right when you use it
> right to one that doesn't break quite as badly when you do
> something stupid.
I think you just proved my point. Running the "sync" command a
couple times then unplugging the USB stick basically never results in
data loss even if the filesystem is mounted. Spontaneously switching
block devices under a mounted filesystem is guaranteed to either
panic the machine or cause massive data corruption or both.
> Also why is this argument more valid for USB than SCSI? I am just
> as free to unplug a scsi disk and replace it with a different one
> while hibernated, yet I don't suffer data loss when I don't do such
> foolishness.
SCSI != USB. Users generally don't expect to hotplug SCSI devices
while booted and running (with the exception of some _really_
expensive hotplug-bays where we expect the admin to know what the
hell they're doing). On the other hand, users _do_ expect to hotplug
random USB devices whenever they feel like it.
Cheers,
Kyle Moffett
--
Q: Why do programmers confuse Halloween and Christmas?
A: Because OCT 31 == DEC 25.
Kyle Moffett wrote:
> In this case they _are_ equivalent due to symmetry. If the other
> device _may_ assume that the connection is broken, then you _must_
> assume that the connection is broken. Since either device _may_
> assume that, both devices therefore _must_ according to spec.
>
Only insofar as the kernel can not assume the device is still
connected. If the kernel sees that it is immediately reconnected and
has every reason to believe that the 'disconnect' was only a result of
momentary power loss, and it can probably continue to access the device
with no consequences, then it should do so.
>
> No, as I said before, a good set of USB suspend scripts can solve this
> problem. All of the ones I am aware of *now* already sync all data,
> which is good enough to prevent data-loss in _every_ case where the
> device is spontaneously unplugged. On the other hand, this is _never_
> good enough if the device is accidentally switched underneath us while
> suspended (and that's not so terribly uncommon, I know a lot of people
> who would do that accidentally, myself included).
>
I suppose this is true. Syncing before suspend will (mostly) keep the
kernel from shooting the user in the foot.
>> I think most users prefer a system that works right when you use it
>> right to one that doesn't break quite as badly when you do something
>> stupid.
>
> I think you just proved my point. Running the "sync" command a couple
> times then unplugging the USB stick basically never results in data
> loss even if the filesystem is mounted. Spontaneously switching block
> devices under a mounted filesystem is guaranteed to either panic the
> machine or cause massive data corruption or both.
>
But who cares? There are plenty of really stupid things users can do to
hose their system, it isn't right to prevent them from doing something
perfectly reasonable just because it reduces the damage done when they
do something completely unreasonable.
>> Also why is this argument more valid for USB than SCSI? I am just as
>> free to unplug a scsi disk and replace it with a different one while
>> hibernated, yet I don't suffer data loss when I don't do such
>> foolishness.
>
> SCSI != USB. Users generally don't expect to hotplug SCSI devices
> while booted and running (with the exception of some _really_
> expensive hotplug-bays where we expect the admin to know what the hell
> they're doing). On the other hand, users _do_ expect to hotplug
> random USB devices whenever they feel like it.
>
So because SCSI is more expensive than USB, it is ok to assume it will
only be used by people who know what they are doing? And since USB will
be used by people who don't know what they are doing, we must assume
they will always do silly things ( swap or modify the drive while
suspended ), at the expense of those who don't?
On Tue, Feb 14, 2006 at 11:23:53AM -0500, Kyle Moffett wrote:
> SCSI != USB. Users generally don't expect to hotplug SCSI devices
> while booted and running (with the exception of some _really_
> expensive hotplug-bays where we expect the admin to know what the
> hell they're doing). On the other hand, users _do_ expect to hotplug
> random USB devices whenever they feel like it.
I do expect to be able to move non-mounted disks around while
suspended to disk, whatever their kind is (ide, sata, scsi, whatever).
That's one of the main reasons you want a reliable suspend-to-disk on
servers, another one being riding predicted powerloss (moving boxes
around can be called a powerloss).
OG.
On Mon, 13 Feb 2006, Rafael J. Wysocki wrote:
> Hi,
>
> On Monday 13 February 2006 22:24, Alan Stern wrote:
> > On Mon, 13 Feb 2006, Phillip Susi wrote:
> }-- snip --{
> > You are complaining because you don't like the way USB was designed.
> > That's fine, but it leaves you advocating a non-standardized position.
> >
> > Can you suggest a _reliable_ way to tell if the USB device present at a
> > port after resuming is the same device as was there before suspending?
>
> It seems to follow from your discussion that if I have a mounted filesystem
> on a USB device and I suspend to disk, I can lose data unless the filesystem
> has been mounted with "sync".
That's right. It depends on your hardware, and it could be true even for
suspend-to-RAM. In fact, even with "-o sync" you can lose data if your
programs have information in buffers they haven't written out to disk.
If you're lucky, your hardware will support low-power modes for USB
controllers while the system is asleep. Lots of hardware doesn't,
however. Shutting off the power to a USB controller is equivalent to
unplugging all the attached devices.
Remember that it's always a bad idea to unplug a disk drive containing a
mounted filesystem. With USB that's true even when your system is asleep!
The safest thing is to unmount all USB-based filesystems before suspending
and remount them after resuming.
> If this is the case, there should be a big fat warning in the swsusp
> documentation, but there's nothing like that in there (at lease I can't find
> it easily).
>
> [If this is not the case, I've missed something and sorry for the noise.]
I'm not aware of any warnings about this in the documentation. If you
would like to add something, please go ahead.
Alan Stern
Olivier Galibert wrote:
>
> I do expect to be able to move non-mounted disks around while
> suspended to disk, whatever their kind is (ide, sata, scsi, whatever).
> That's one of the main reasons you want a reliable suspend-to-disk on
> servers, another one being riding predicted powerloss (moving boxes
> around can be called a powerloss).
You also expect to have mounted disks survive the suspend. Think about
the root filesystem. You can't unmount that and it would be really bad
if that went offline ( kernel panic anyone? ) after a suspend. And yes,
the root fs can be on a USB drive.
On Feb 14, 2006, at 13:39, Phillip Susi wrote:
>>> I think most users prefer a system that works right when you use
>>> it right to one that doesn't break quite as badly when you do
>>> something stupid.
>>
>> I think you just proved my point. Running the "sync" command a
>> couple times then unplugging the USB stick basically never results
>> in data loss even if the filesystem is mounted. Spontaneously
>> switching block devices under a mounted filesystem is guaranteed
>> to either panic the machine or cause massive data corruption or both.
>
> But who cares? There are plenty of really stupid things users can
> do to hose their system, it isn't right to prevent them from doing
> something perfectly reasonable just because it reduces the damage
> done when they do something completely unreasonable.
How is swapping USB devices while suspended unreasonable? Come to
think of it, I did it inadvertently about 15 minutes ago with my
PowerBook (while booted into Mac OS). I had a USB key that I was
copying a file to for someone. After shutting the lid, I unplugged
mouse, USB key, and power block. About 30 minutes later I had
somebody else with a key who wanted to give me a file. I had the key
plugged in before the laptop was finished waking up from sleep. Now,
I don't know for certain that neither key had a serial number, but
the two I have here in my hand certainly don't, and I could _easily_
see somebody swapping USB keys not knowing that they're "not supposed
to do that" and getting massive data corruption when the filesystem
reads and writes pages from a completely different block device.
>> SCSI != USB. Users generally don't expect to hotplug SCSI devices
>> while booted and running (with the exception of some _really_
>> expensive hotplug-bays where we expect the admin to know what the
>> hell they're doing). On the other hand, users _do_ expect to
>> hotplug random USB devices whenever they feel like it.
>
> So because SCSI is more expensive than USB, it is ok to assume it
> will only be used by people who know what they are doing? And
> since USB will be used by people who don't know what they are
> doing, we must assume they will always do silly things ( swap or
> modify the drive while suspended ), at the expense of those who don't?
Did you read what I wrote? People don't generally expect to randomly
plug and unplug SCSI drives whenever they feel like it. They _do_
expect to randomly plug and unplug USB drives, mice, keyboards,
tablets, network adapters, etc, because _everything_ supports such
random plugging.
Creating an extremely odd and hard to predict failure mode (when you
reconnect USB devices while suspended on hardware that doesn't
support proper USB suspend) with a high probability of causing data
corruption or crashes is wrong. Especially since you could easily
teach users that "You need to eject USB things before you sleep the
computer _or_ just fix the kernel to do it for you. That's probably
something we should be doing for all network filesystems anyways.
Cheers,
Kyle Moffett
--
I didn't say it would work as a defense, just that they can spin that
out for years in court if it came to it.
-- Rob Landley
Hi,
On Tuesday 14 February 2006 20:26, Alan Stern wrote:
> On Mon, 13 Feb 2006, Rafael J. Wysocki wrote:
> > On Monday 13 February 2006 22:24, Alan Stern wrote:
> > > On Mon, 13 Feb 2006, Phillip Susi wrote:
> > }-- snip --{
> > > You are complaining because you don't like the way USB was designed.
> > > That's fine, but it leaves you advocating a non-standardized position.
> > >
> > > Can you suggest a _reliable_ way to tell if the USB device present at a
> > > port after resuming is the same device as was there before suspending?
> >
> > It seems to follow from your discussion that if I have a mounted filesystem
> > on a USB device and I suspend to disk, I can lose data unless the filesystem
> > has been mounted with "sync".
>
> That's right. It depends on your hardware, and it could be true even for
> suspend-to-RAM. In fact, even with "-o sync" you can lose data if your
> programs have information in buffers they haven't written out to disk.
>
> If you're lucky, your hardware will support low-power modes for USB
> controllers while the system is asleep. Lots of hardware doesn't,
> however. Shutting off the power to a USB controller is equivalent to
> unplugging all the attached devices.
>
> Remember that it's always a bad idea to unplug a disk drive containing a
> mounted filesystem. With USB that's true even when your system is asleep!
> The safest thing is to unmount all USB-based filesystems before suspending
> and remount them after resuming.
Still, this may be impossible if the box is suspending because of its
battery running critical.
I'm afraid this behavior will cause support problems to appear in the long
run. [BTW, I wonder if it's USB-only, or some other subsystems behave
in a similar way, like ieee1394 or external SATA. And how about
NFS/CIFS/whatever network filesystems mounted on the suspending box?]
I think one solution to consider could be to add an abstract fs layer
on top of the removable filesystem, like subfs in SuSE, with the ability
to retain the user information accross device disconnects and to update
the fs state from the actual device when it reappears in the system and
to resolve possible conflicts (to a reasonable extent).
> > If this is the case, there should be a big fat warning in the swsusp
> > documentation, but there's nothing like that in there (at lease I can't find
> > it easily).
> >
> > [If this is not the case, I've missed something and sorry for the noise.]
>
> I'm not aware of any warnings about this in the documentation. If you
> would like to add something, please go ahead.
I'm going to do this. Can I use your explanation above?
Rafael
On Tue, 2006-02-14 at 21:41 +0100, Rafael J. Wysocki wrote:
> [BTW, I wonder if it's USB-only, or some other subsystems behave
> in a similar way, like ieee1394 or external SATA. And how about
> NFS/CIFS/whatever network filesystems mounted on the suspending box?]
>
NFS is stateless, it does not care about stuff like this. There's no
concept of an open file. write() does not return until the data is
committed to stable storage.
Lee
Kyle Moffett wrote:
> How is swapping USB devices while suspended unreasonable? Come to
> think of it, I did it inadvertently about 15 minutes ago with my PowerBook
Because you can not go yanking devices out from under the kernel without
it's knowledge or consent. This is no more acceptable than ejecting a
floppy without first unmounting it; the only difference is that the
floppy drive doesn't erroneously inform the kernel that you have done
this simply because you suspend.
> (while booted into Mac OS). I had a USB key that I was copying a file
> to for someone. After shutting the lid, I unplugged mouse, USB key,
> and power block. About 30 minutes later I had somebody else with a
> key who wanted to give me a file. I had the key plugged in before the
> laptop was finished waking up from sleep. Now, I don't know for
> certain that neither key had a serial number, but the two I have here
> in my hand certainly don't, and I could _easily_ see somebody swapping
> USB keys not knowing that they're "not supposed to do that" and
> getting massive data corruption when the filesystem reads and writes
> pages from a completely different block device.
>
Then they shot themselves in the foot. That is no different than
switching mounted floppies while suspended, or removing a mounted IDE
hard drive while suspended, so they get what they deserve.
> Did you read what I wrote? People don't generally expect to randomly
> plug and unplug SCSI drives whenever they feel like it. They _do_
> expect to randomly plug and unplug USB drives, mice, keyboards,
> tablets, network adapters, etc, because _everything_ supports such
> random plugging.
>
No, they don't. Users do not expect ( or should not and are told not to
by admins ) to be able to yank out their usb memory stick while it is
mounted. They are told to always unmount first. If they fail to do so,
then they get what they asked for. It doesn't matter if the disk is
SCSI or USB; you don't go yanking it out without unmounting it first, or
you will loose data.
> Creating an extremely odd and hard to predict failure mode (when you
> reconnect USB devices while suspended on hardware that doesn't support
> proper USB suspend) with a high probability of causing data corruption
> or crashes is wrong. Especially since you could easily teach users
> that "You need to eject USB things before you sleep the computer _or_
> just fix the kernel to do it for you. That's probably something we
> should be doing for all network filesystems anyways.
>
Users are already told to eject/unmount the media before removing it.
If they fail to do that, it doesn't matter if the system is suspended or
not; they broke the drive when they yanked it out while mounted. As for
fixing the kernel to unmount it for you, that is not always possible;
take the root on usb case.
Maybe it does make sense to have hibernation scripts unmount removable
media for the common silly user who can't remember to unmount disks
before ejecting/unplugging them, but you should be able to suspend a
system with its root on a usb disk and not have the kernel panic for no
good reason when it resumes.
On Feb 14, 2006, at 16:13, Phillip Susi wrote:
> Because you can not go yanking devices out from under the kernel
> without it's knowledge or consent. This is no more acceptable than
> ejecting a floppy without first unmounting it; the only difference
> is that the floppy drive doesn't erroneously inform the kernel that
> you have done this simply because you suspend.
Yes, but causing it to overwrite data over random blockdevs when the
unexpected happens is _not_ OK.
>> Did you read what I wrote? People don't generally expect to
>> randomly plug and unplug SCSI drives whenever they feel like it.
>> They _do_ expect to randomly plug and unplug USB drives, mice,
>> keyboards, tablets, network adapters, etc, because _everything_
>> supports such random plugging.
>
> No, they don't. Users do not expect ( or should not and are told
> not to by admins ) to be able to yank out their usb memory stick
> while it is mounted. They are told to always unmount first. If
> they fail to do so, then they get what they asked for. It doesn't
> matter if the disk is SCSI or USB; you don't go yanking it out
> without unmounting it first, or you will loose data.
You've obviously never administrated any kind of large scale linux
lab. We had so many people just saving files on their USB sticks and
pulling them that we would daily get people reporting that they
couldn't _mount_ their USB stick because the last user hadn't
unmounted it first. At one point we had to write a Perl script run
from cron that ran every 10 minutes and verified if USB-stick mounts
were still good, and if not it forcefully unmounted them. Now
obviously this is the kind of behavior that we want to avoid, but end-
users are bound to do stupid stuff until it comes back and bites them
at least twice (except on Linux with -o sync mounts it usually
doesn't). Think about how most linux distros automatically turn on -
o sync on USB keys and the like; they do it for precisely this reason.
> Maybe it does make sense to have hibernation scripts unmount
> removable media for the common silly user who can't remember to
> unmount disks before ejecting/unplugging them, but you should be
> able to suspend a system with its root on a usb disk and not have
> the kernel panic for no good reason when it resumes.
I described a workable method to handle root-on-USB (and I'm not
sure, but I doubt it works now). You would have early userspace find
your USB filesystems again and pass device information to the
resuming kernel, which would then restore RAM and then use the passed
information to attach its filesystems again.
One other alternative that just occurs to me now is to have a special
stackable filesystem that can suspend all IO and unmount the
filesystem underneath it then have the filesystem be easily
reattachable later on (the stackable layer would look up paths again
on remount. You would have an mlocked program start during suspend
that would create a new namespace with a basic tmpfs, procfs, and
sysfs. It would be able to rescan all removable devices on resume
and reattach them to the stackable mounts in the primary namespace
through /proc/<pid>/root. This seems to me to be the most race-free
and most flexible solution. It will even handle network suspend and
resume without too much extra effort.
Cheers,
Kyle Moffett
--
Simple things should be simple and complex things should be possible
-- Alan Kay
Kyle Moffett wrote:
> On Feb 14, 2006, at 16:13, Phillip Susi wrote:
>> Because you can not go yanking devices out from under the kernel
>> without it's knowledge or consent. This is no more acceptable than
>> ejecting a floppy without first unmounting it; the only difference is
>> that the floppy drive doesn't erroneously inform the kernel that you
>> have done this simply because you suspend.
>
> Yes, but causing it to overwrite data over random blockdevs when the
> unexpected happens is _not_ OK.
>
Why not? That is exactly what happens if you suspend and swap floppies.
Just don't do that and you're fine. It is nice if you can handle the
swapped case more gracefully, but not at the cost of data loss in the
_normal_ case.
> You've obviously never administrated any kind of large scale linux lab.
> We had so many people just saving files on their USB sticks and pulling
> them that we would daily get people reporting that they couldn't _mount_
> their USB stick because the last user hadn't unmounted it first. At one
> point we had to write a Perl script run from cron that ran every 10
> minutes and verified if USB-stick mounts were still good, and if not it
This is a very good argument for _correctly_ detecting device removal.
I am not arguing against that. The problem is with _incorrectly_
detecting device removal.
> forcefully unmounted them. Now obviously this is the kind of behavior
> that we want to avoid, but end-users are bound to do stupid stuff until
> it comes back and bites them at least twice (except on Linux with -o
> sync mounts it usually doesn't). Think about how most linux distros
> automatically turn on -o sync on USB keys and the like; they do it for
> precisely this reason.
>
Some distros do. Some do not. I side with those who do not because
doing so promotes stupid users ( they won't learn until it bites them
twice ) and because it burns out the flash memory 10 times faster and
slows down access.
>
> I described a workable method to handle root-on-USB (and I'm not sure,
> but I doubt it works now). You would have early userspace find your USB
> filesystems again and pass device information to the resuming kernel,
> which would then restore RAM and then use the passed information to
> attach its filesystems again.
>
There is currently no such method of reconnecting a filesystem to a
device once the device has been broken by an eject, is there? Also that
amounts to much the same thing as I have been saying all along: the
system should check to see if the device that is there now is the same
as the one that was there before, and if so, resume using it.
> One other alternative that just occurs to me now is to have a special
> stackable filesystem that can suspend all IO and unmount the filesystem
> underneath it then have the filesystem be easily reattachable later on
> (the stackable layer would look up paths again on remount. You would
> have an mlocked program start during suspend that would create a new
> namespace with a basic tmpfs, procfs, and sysfs. It would be able to
> rescan all removable devices on resume and reattach them to the
> stackable mounts in the primary namespace through /proc/<pid>/root.
> This seems to me to be the most race-free and most flexible solution.
> It will even handle network suspend and resume without too much extra
> effort.
>
That seems rather complex with little benefit. The only advantage that
would have over not breaking the mount in the first place is in the
pedantic case where the user screws with the media during suspend.
Seems like a high price to pay to cleanly deal with a pedantic error
condition.
On Tue, 14 Feb 2006, Rafael J. Wysocki wrote:
> > Remember that it's always a bad idea to unplug a disk drive containing a
> > mounted filesystem. With USB that's true even when your system is asleep!
> > The safest thing is to unmount all USB-based filesystems before suspending
> > and remount them after resuming.
>
> Still, this may be impossible if the box is suspending because of its
> battery running critical.
Yes. When there's nothing you can do... do nothing. :-)
> I'm afraid this behavior will cause support problems to appear in the long
> run. [BTW, I wonder if it's USB-only, or some other subsystems behave
> in a similar way, like ieee1394 or external SATA. And how about
> NFS/CIFS/whatever network filesystems mounted on the suspending box?]
Some networked filesystems may have this problem, for instance, if they
rely on some sort of keep-alive to maintain a connection. I don't know
anything about 1394 or SATA.
> I think one solution to consider could be to add an abstract fs layer
> on top of the removable filesystem, like subfs in SuSE, with the ability
> to retain the user information accross device disconnects and to update
> the fs state from the actual device when it reappears in the system and
> to resolve possible conflicts (to a reasonable extent).
Maybe. To me this seems an unavoidable policy decision that should be
settled by one of the big Linux panjandrums. Maybe someone can convince
Andrew (or even Linus) to offer a ruling...
> > I'm not aware of any warnings about this in the documentation. If you
> > would like to add something, please go ahead.
>
> I'm going to do this. Can I use your explanation above?
Feel free to use it.
Alan Stern
Hi!
> >You are ignoring the question of how the kernel can tell whether two
> >devices are in fact the same. There is no safe way to do this,
> >other than having the hardware verify that the device was connected
> >the whole time.
> >
> >
> If there is no better way to tell for sure that the device that is
> now there is not the same as the one that was there, then the kernel
> must assume the user did not do something stupid and continue to use
Must?! Are you Linus or what?
> the device as if it was not disconnected ( because odds are, it in
> fact, was not ). In other words, which is more safe? ALLWAYS
> loosing data because you can't be absolutely sure that the device is
> the same, or only loosing data if the user does something as foolish
> as swapping drives while suspended, and you can't tell they did?
You are missing this. In 1st case, no data is actually lost, because of
sync in suspend code;
while second case is "goodbye, filesystem".
Pavel
--
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms
Hi!
> > I would disagree. The only difference between the two is WHERE the
> > state is maintained - ram vs. disk. I won't really argue it though,
> > because it's just semantics -- call it whatever you want.
>
> It's not just semantics. There's a real difference between maintaining
> state in the hardware and maintaining it somewhere else. The biggest
> difference is that if the hardware retains suspend power, it is able to
> detect disconnections. When the system resumes, it _knows_ whether a
> device was attached the entire time, as opposed to being unplugged and
> replugged (or possibly a different device plugged in!) while the system
> was asleep. If the hardware is down completely, there is no way of
> telling for certain whether a device attached to some port is the same one
> that was there when the system got suspended.
I have strange system here (intel dual core prototype) that supplies
usb power even while it is "off". I'll need to find a way to fix that.
(Not that it is important to this discussion, just a reminder that strange hw exists)
--
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms
Pavel Machek wrote:
>
> Must?! Are you Linus or what?
>
Non sequitur.
>
> You are missing this. In 1st case, no data is actually lost, because of
> sync in suspend code;
> while second case is "goodbye, filesystem".
Provided that you sync before suspending, and there are no open files on
the filesystem, then yes, no data will be lost. If there are open files
on the fs, such as a half saved document, or a running binary, or say,
the whole root fs, then you're going to loose data and even panic the
kernel, sync or no sync. From the user perspective, this is unacceptable.
Why should the user give up such functionality just because the
connection to the drive thy are using is USB? Every other type of drive
and interface does not suffer from this problem.
Maybe Linux should take a page from windows' playbook here. I believe
windows handles this scenario with a USB drive the same way it does when
you eject a floppy and reinsert it. The driver detects that the
media/drive _may_ have changed and so it fails requests from the
filesystem with an error code indicating this. The filesystem then sets
an override flag so it can send down some reads to verify the media.
Generally the FS reads the super block and compares it with the in
memory one to make sure it appears to be the same media, and if so,
continues normal access without data loss.
On So 18-02-06 11:34:17, Phillip Susi wrote:
> Pavel Machek wrote:
> >You are missing this. In 1st case, no data is actually lost, because of
> >sync in suspend code;
> >while second case is "goodbye, filesystem".
>
> Provided that you sync before suspending, and there are no open files on
> the filesystem, then yes, no data will be lost. If there are open files
> on the fs, such as a half saved document, or a running binary, or say,
> the whole root fs, then you're going to loose data and even panic the
> kernel, sync or no sync. From the user perspective, this is
>unacceptable.
While with your solution, you do not loose one open file, you loose
whole filesystem. Which is unacceptable.
> Why should the user give up such functionality just because the
> connection to the drive thy are using is USB? Every other type of drive
> and interface does not suffer from this problem.
Because it is okay to unplug usb disk on runtime, while it is not okay
to unplug ATA disk on runtime. And because alternatives suck even more.
> Maybe Linux should take a page from windows' playbook here. I believe
> windows handles this scenario with a USB drive the same way it does when
> you eject a floppy and reinsert it. The driver detects that the
> media/drive _may_ have changed and so it fails requests from the
> filesystem with an error code indicating this. The filesystem then sets
> an override flag so it can send down some reads to verify the media.
> Generally the FS reads the super block and compares it with the in
> memory one to make sure it appears to be the same media, and if so,
> continues normal access without data loss.
Feel free to implement that.
Pavel
--
Web maintainer for suspend.sf.net (http://www.sf.net/projects/suspend) wanted...
On Fri, 17 Feb 2006, Pavel Machek wrote:
> > If there is no better way to tell for sure that the device that is
> > now there is not the same as the one that was there, then the kernel
> > must assume the user did not do something stupid and continue to use
>
> Must?! Are you Linus or what?
Technical issues aside, the question about recogizing formerly-attached
devices when waking up from a system sleep is really about
user-friendliness. The choice between what Phillip is advocating and what
the rest of us have been saying is a policy choice, one that the kernel
can't easily avoid making.
Given that this is a policy decision, it might not be such a bad idea to
ask Andrew or Linus for their opinion. In fact, I'll do that right now.
To summarize the background:
When a hotpluggable bus (USB for sure, possibly others as well) loses
power, the hardware interprets this as a disconnection of all attached
devices. When power is restored, it appears to the kernel as though a
completely new set of devices has now been plugged in.
This is unfriendly for people whose motherboard/USB-controller hardware
doesn't supply minimal power during hardware- or software-suspend. Any
mounted filesystems on a USB storage device get blown away when the system
resumes, since the kernel thinks the device was unplugged.
In principle the kernel is capable of detecting that a device present on a
port during resume is very similar to the device that had been present
during the suspend. Through a combination of checks (descriptors, serial
numbers, superblocks, maybe others) it could try to verify this and then
keep the device data structures intact, as though the device had been
connected all along.
The advantage is that people wouldn't lose half-stored data on their
removable drives, they wouldn't have unmount before suspending and remount
after resuming, and they would be able to suspend with their root
filesystem on a USB drive without causing a panic upon resume.
The disadvantage is that sometimes people do switch removable drives or
removable media while the system is asleep. If the kernel gets fooled
into thinking the new device is the same as the old one, it would proceed
to destroy the data on the new device by overwriting it with data from the
old one. (Not to mention that this kind of approach is contrary to the
USB specification.)
On the other hand, it's what Windows does. We don't want people to say
that Linux is incapable of implementing a feature that Windows has had for
a long time. :-)
So the question is: What should the kernel do? Assume the device has
changed (as it does now) or make some checks to see if it might still be
the same (dangerous though that may be)?
If you really want to waffle, you can recommend that the capability to do
this be present but normally disabled, controlled by a flag somewhere in
/sys/kernel -- thus pushing the decision off to userspace.
Alan Stern
Alan Stern <[email protected]> wrote:
>
> On Fri, 17 Feb 2006, Pavel Machek wrote:
>
> > > If there is no better way to tell for sure that the device that is
> > > now there is not the same as the one that was there, then the kernel
> > > must assume the user did not do something stupid and continue to use
> >
> > Must?! Are you Linus or what?
>
> Technical issues aside, the question about recogizing formerly-attached
> devices when waking up from a system sleep is really about
> user-friendliness. The choice between what Phillip is advocating and what
> the rest of us have been saying is a policy choice, one that the kernel
> can't easily avoid making.
>
> Given that this is a policy decision, it might not be such a bad idea to
> ask Andrew or Linus for their opinion. In fact, I'll do that right now.
>
>
> To summarize the background:
>
> When a hotpluggable bus (USB for sure, possibly others as well) loses
> power, the hardware interprets this as a disconnection of all attached
> devices. When power is restored, it appears to the kernel as though a
> completely new set of devices has now been plugged in.
>
> This is unfriendly for people whose motherboard/USB-controller hardware
> doesn't supply minimal power during hardware- or software-suspend. Any
> mounted filesystems on a USB storage device get blown away when the system
> resumes, since the kernel thinks the device was unplugged.
I'm all for being friendly to people.
> In principle the kernel is capable of detecting that a device present on a
> port during resume is very similar to the device that had been present
> during the suspend. Through a combination of checks (descriptors, serial
> numbers, superblocks, maybe others) it could try to verify this and then
> keep the device data structures intact, as though the device had been
> connected all along.
>
> The advantage is that people wouldn't lose half-stored data on their
> removable drives, they wouldn't have unmount before suspending and remount
> after resuming, and they would be able to suspend with their root
> filesystem on a USB drive without causing a panic upon resume.
>
> The disadvantage is that sometimes people do switch removable drives or
> removable media while the system is asleep. If the kernel gets fooled
> into thinking the new device is the same as the old one, it would proceed
> to destroy the data on the new device by overwriting it with data from the
> old one. (Not to mention that this kind of approach is contrary to the
> USB specification.)
>
> On the other hand, it's what Windows does. We don't want people to say
> that Linux is incapable of implementing a feature that Windows has had for
> a long time. :-)
>
>
> So the question is: What should the kernel do? Assume the device has
> changed (as it does now) or make some checks to see if it might still be
> the same (dangerous though that may be)?
>
> If you really want to waffle, you can recommend that the capability to do
> this be present but normally disabled, controlled by a flag somewhere in
> /sys/kernel -- thus pushing the decision off to userspace.
>
Correct me if I'm wrong, but this really only affects storage devices, yes?
That narrows the scope of the problem quite some.
I suspect we could do a very good job of working out whether we're still
talking to the same fs if filesystem drivers were to help in that.
But I suspect we could do an even better job if we did that in userspace.
The logic to determine whether the new device is the same as the old device
can be arbitrarily complex, with increasing levels of success. Various
heuristics can be applied, some of which will involve knowledge of
filesystem layout, etc.
So would it not be possible to optionally punt the device naming decision
up to the hotplug scripts? So code up there can go do direct-IO reads of
the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
superblocks, disk labels, partition tables, inode numbering, etc? Go look
up a database, work out what that filesystem was doing last time we saw it,
etc?
We could of course add things to the filesystems to help this process, but
it'd be good if all the state tracking and magic didn't have to be locked
up in the kernel.
Pavel Machek wrote:
>> Provided that you sync before suspending, and there are no open files on
>> the filesystem, then yes, no data will be lost. If there are open files
>> on the fs, such as a half saved document, or a running binary, or say,
>> the whole root fs, then you're going to loose data and even panic the
>> kernel, sync or no sync. From the user perspective, this is
>> unacceptable.
>
> While with your solution, you do not loose one open file, you loose
> whole filesystem. Which is unacceptable.
>
Only if the user is foolish enough to modify the media somehow while the
system is suspended and replace it, which is exactly how non USB disks
currently behave.
>> Why should the user give up such functionality just because the
>> connection to the drive thy are using is USB? Every other type of drive
>> and interface does not suffer from this problem.
>
> Because it is okay to unplug usb disk on runtime, while it is not okay
> to unplug ATA disk on runtime. And because alternatives suck even more.
>
Actually, no, it is not okay to unplug a usb disk at runtime while it is
mounted. It never has been and it never will be. Also we aren't
talking about runtime, we're talking about while the system is
suspended, when there is no way for the kernel to know whether or not
the device was unplugged, since it _allways_ appears to have been
unplugged. The alternative in the uncommon case ( where the user
modifies the media while suspended ) does not suck any worse than it
currently does on non usb media, and the common case ( where the user
doesn't ) sucks worse currently with usb than others.
>> Maybe Linux should take a page from windows' playbook here. I believe
>> windows handles this scenario with a USB drive the same way it does when
>> you eject a floppy and reinsert it. The driver detects that the
>> media/drive _may_ have changed and so it fails requests from the
>> filesystem with an error code indicating this. The filesystem then sets
>> an override flag so it can send down some reads to verify the media.
>> Generally the FS reads the super block and compares it with the in
>> memory one to make sure it appears to be the same media, and if so,
>> continues normal access without data loss.
>
> Feel free to implement that.
> Pavel
>
Andrew Morton wrote:
> Correct me if I'm wrong, but this really only affects storage devices, yes?
> That narrows the scope of the problem quite some.
>
> I suspect we could do a very good job of working out whether we're still
> talking to the same fs if filesystem drivers were to help in that.
>
Exactly, which is why windows involves the filesystem driver.
> But I suspect we could do an even better job if we did that in userspace.
>
> The logic to determine whether the new device is the same as the old device
> can be arbitrarily complex, with increasing levels of success. Various
> heuristics can be applied, some of which will involve knowledge of
> filesystem layout, etc.
>
> So would it not be possible to optionally punt the device naming decision
> up to the hotplug scripts? So code up there can go do direct-IO reads of
> the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
> superblocks, disk labels, partition tables, inode numbering, etc? Go look
> up a database, work out what that filesystem was doing last time we saw it,
> etc?
>
> We could of course add things to the filesystems to help this process, but
> it'd be good if all the state tracking and magic didn't have to be locked
> up in the kernel.
Hrm... interesting but sounds like that could be sticky. For instance,
what if the user script that does the verifying happens to be ON the
volume to be verified?
I think this kind of thing is really going to have to be in kernel, like
a remount.
Phillip Susi <[email protected]> wrote:
>
> > But I suspect we could do an even better job if we did that in userspace.
> >
> > The logic to determine whether the new device is the same as the old device
> > can be arbitrarily complex, with increasing levels of success. Various
> > heuristics can be applied, some of which will involve knowledge of
> > filesystem layout, etc.
> >
> > So would it not be possible to optionally punt the device naming decision
> > up to the hotplug scripts? So code up there can go do direct-IO reads of
> > the newly-present blockdev, use filesytem layout knowledge, peek at UUIDs,
> > superblocks, disk labels, partition tables, inode numbering, etc? Go look
> > up a database, work out what that filesystem was doing last time we saw it,
> > etc?
> >
> > We could of course add things to the filesystems to help this process, but
> > it'd be good if all the state tracking and magic didn't have to be locked
> > up in the kernel.
>
>
> Hrm... interesting but sounds like that could be sticky. For instance,
> what if the user script that does the verifying happens to be ON the
> volume to be verified?
Well that would be a bug. Solutions would be a) don't put the scripts on a
removable/power-downable device or b) use tmpfs.
On Ne 19-02-06 00:52:45, Phillip Susi wrote:
> Pavel Machek wrote:
> >>Provided that you sync before suspending, and there are no open files on
> >>the filesystem, then yes, no data will be lost. If there are open files
> >>on the fs, such as a half saved document, or a running binary, or say,
> >>the whole root fs, then you're going to loose data and even panic the
> >>kernel, sync or no sync. From the user perspective, this is
> >>unacceptable.
> >
> >While with your solution, you do not loose one open file, you loose
> >whole filesystem. Which is unacceptable.
>
> Only if the user is foolish enough to modify the media somehow while the
> system is suspended and replace it, which is exactly how non USB disks
> currently behave.
"Foolish enough"? Multiple users told you that they consider that use
case okay for hotpluggable drives.
> >>connection to the drive thy are using is USB? Every other type of drive
> >>and interface does not suffer from this problem.
> >
> >Because it is okay to unplug usb disk on runtime, while it is not okay
> >to unplug ATA disk on runtime. And because alternatives suck even more.
> >
>
> Actually, no, it is not okay to unplug a usb disk at runtime while it is
> mounted. It never has been and it never will be. Also we aren't
Ever heard about "journalling"?
> talking about runtime, we're talking about while the system is
> suspended, when there is no way for the kernel to know whether or
> not
If it is okay during runtime, it should be okay while suspended. Don't
expect users to know about power on USB buses. You may call any system
that does not support standby power on USB broken if you wish...
> the device was unplugged, since it _allways_ appears to have been
> unplugged. The alternative in the uncommon case ( where the user
> modifies the media while suspended ) does not suck any worse than it
> currently does on non usb media, and the common case ( where the user
> doesn't ) sucks worse currently with usb than others.
"Does not such any worse than non-usb" does not cut it here. USB disks
are too easy to unplug/replug.
Anyway, your mail came without a patch, again. That's useless; if you
implement layer above floppies/usb sticks that can recognize same
disk, maybe we can talk about that.
Pavel
--
Web maintainer for suspend.sf.net (http://www.sf.net/projects/suspend) wanted...
Pavel Machek wrote:
>
> "Foolish enough"? Multiple users told you that they consider that use
> case okay for hotpluggable drives.
>
They are confused. It has never been, is not, and never will be okay to
disconnect a drive while it is mounted. It is simply an easier mistake
to make with USB. Easy or not, it is still a mistake.
This user error may be rather common, but as long as you sacrifice other
functionality to prevent this user error from being particularly
irritating to users, they will not learn and will continue to make this
mistake. Let them be bitten by it once or twice, and they will learn
they need to properly unmount before yanking.
>
> Ever heard about "journalling"?
>
Yes, it makes it faster to recover the filesystem to a consistent state
after a crash. It does NOT make it okay to disconnect the drive
willy-nilly, and does not prevent data loss.
>
> If it is okay during runtime, it should be okay while suspended. Don't
> expect users to know about power on USB buses. You may call any system
> that does not support standby power on USB broken if you wish...
>
It is only okay to disconnect the drive at any time, running or
suspended, after you unmount it. Fail to do that and you're asking for
trouble.
>
> "Does not such any worse than non-usb" does not cut it here. USB disks
> are too easy to unplug/replug.
>
> Anyway, your mail came without a patch, again. That's useless; if you
> implement layer above floppies/usb sticks that can recognize same
> disk, maybe we can talk about that.
That's exactly what I've been talking about. You can add some logic
that can hopefully notice if the user has modified the volume, but those
tests need not be 100% foolproof in order to assume that the user did
not actually modify the volume while suspended.
Andrew Morton wrote:
>>
>> Hrm... interesting but sounds like that could be sticky. For instance,
>> what if the user script that does the verifying happens to be ON the
>> volume to be verified?
>
> Well that would be a bug. Solutions would be a) don't put the scripts on a
> removable/power-downable device or b) use tmpfs.
>
I don't see how it could be a bug, just think of the root on usb case.
Keeping the program locked in ram would sidestep that issue, but tmpfs
is pagable right? Swap on usb?
Also, this user space program isn't going to be able to keep fully up to
date on what the disk looks like. Imagine a filesystem that keeps a
generation counter in the super block and increments it every time it
writes to the disk. The user space daemon might read that, then the fs
changes it, you suspend, and when you wake up, the daemon thinks the
media changed because it wasn't fully up to date.
On Sun, 19 Feb 2006, Phillip Susi wrote:
> > If it is okay during runtime, it should be okay while suspended. Don't
> > expect users to know about power on USB buses. You may call any system
> > that does not support standby power on USB broken if you wish...
> >
>
> It is only okay to disconnect the drive at any time, running or
> suspended, after you unmount it. Fail to do that and you're asking for
> trouble.
I'm confused. Weren't you arguing, only a few days ago, that it _should_
be okay to unplug a USB drive while the system is suspended? After all,
since there's no difference (as far as the kernel can see) between power
loss on the bus and an actual unplug, you can hardly say that one should
be allowed and the other not.
Alan Stern
On Sun, 19 Feb 2006, Phillip Susi wrote:
> Andrew Morton wrote:
> >>
> >> Hrm... interesting but sounds like that could be sticky. For instance,
> >> what if the user script that does the verifying happens to be ON the
> >> volume to be verified?
> >
> > Well that would be a bug. Solutions would be a) don't put the scripts on a
> > removable/power-downable device or b) use tmpfs.
> >
>
> I don't see how it could be a bug, just think of the root on usb case.
> Keeping the program locked in ram would sidestep that issue, but tmpfs
> is pagable right? Swap on usb?
>
> Also, this user space program isn't going to be able to keep fully up to
> date on what the disk looks like. Imagine a filesystem that keeps a
> generation counter in the super block and increments it every time it
> writes to the disk. The user space daemon might read that, then the fs
> changes it, you suspend, and when you wake up, the daemon thinks the
> media changed because it wasn't fully up to date.
There are other, harder problems associated with doing this is userspace.
Really, the device needs to be considered separately at each level of
driver processing. At the USB level it may still appear to be the same,
but at the SCSI level it may appear different. Or at the SCSI level it
may be the same, but at the gendisk level it may be different (different
media, partition table changed). Or at the gendisk level it may be the
same, but at the filesystem level one or more of the partitions may be
different (superblock changed).
Each level would need its own way of checking for changes and recreating
the appropriate data structures. And while making the determinations, we
would need to temporarily set the device to read-only. Yes, userspace
could do _some_ of the work, but it would need a lot of help from the
kernel.
Alan Stern
Alan Stern wrote:
>
> I'm confused. Weren't you arguing, only a few days ago, that it _should_
> be okay to unplug a USB drive while the system is suspended? After all,
No, I wasn't arguing that it should be okay to unplug a usb drive while
the system is suspended, I was saying that it is better for the kernel
to assume you did not when it can't really tell, since you aren't
supposed to do that anyhow.
> since there's no difference (as far as the kernel can see) between power
> loss on the bus and an actual unplug, you can hardly say that one should
> be allowed and the other not.
But there _IS_ a difference between power loss and actual unplug, so
I very well can say one is allowed and the other is not. The fact that
the kernel can not tell the difference is simply a limitation that must
be dealt with.
> >since there's no difference (as far as the kernel can see) between power
> >loss on the bus and an actual unplug, you can hardly say that one should
> >be allowed and the other not.
>
> But there _IS_ a difference between power loss and actual unplug, so
> I very well can say one is allowed and the other is not. The fact that
> the kernel can not tell the difference is simply a limitation that must
> be dealt with.
Kernel can not tell the diference, and just because you don't like the
behaviour does not mean we have to work around hardware limitation.
You deal with it. Post a patch.
Pavel
--
Web maintainer for suspend.sf.net (http://www.sf.net/projects/suspend) wanted...
Alan Stern <[email protected]> wrote:
>
> On Sun, 19 Feb 2006, Phillip Susi wrote:
>
> > Andrew Morton wrote:
> > >>
> > >> Hrm... interesting but sounds like that could be sticky. For instance,
> > >> what if the user script that does the verifying happens to be ON the
> > >> volume to be verified?
> > >
> > > Well that would be a bug. Solutions would be a) don't put the scripts on a
> > > removable/power-downable device or b) use tmpfs.
> > >
> >
> > I don't see how it could be a bug, just think of the root on usb case.
> > Keeping the program locked in ram would sidestep that issue, but tmpfs
> > is pagable right? Swap on usb?
> >
> > Also, this user space program isn't going to be able to keep fully up to
> > date on what the disk looks like. Imagine a filesystem that keeps a
> > generation counter in the super block and increments it every time it
> > writes to the disk. The user space daemon might read that, then the fs
> > changes it, you suspend, and when you wake up, the daemon thinks the
> > media changed because it wasn't fully up to date.
>
> There are other, harder problems associated with doing this is userspace.
>
> Really, the device needs to be considered separately at each level of
> driver processing. At the USB level it may still appear to be the same,
> but at the SCSI level it may appear different. Or at the SCSI level it
> may be the same, but at the gendisk level it may be different (different
> media, partition table changed). Or at the gendisk level it may be the
> same, but at the filesystem level one or more of the partitions may be
> different (superblock changed).
>
> Each level would need its own way of checking for changes and recreating
> the appropriate data structures. And while making the determinations, we
> would need to temporarily set the device to read-only. Yes, userspace
> could do _some_ of the work, but it would need a lot of help from the
> kernel.
>
There are two things here: a) identifying the device and b) putting it into
the correct place in the various data structures. I suspect these can (and
should) be separated out.
For a), the current kernel behaviour is what we want - make the thing
appear at a new place in the namespace and in the hierarchy. Then
userspace can do whatever needs to be done to identify the device, and
apply some sort of policy decision to the result.
It's what comes next that we're missing: the ability for userspace to tell
the krnrel what the decice's naming and positioning _should_ be. The
kernel will then atomically tear down what it currently has and will then
set everything up as desired.
Phillip Susi <[email protected]> wrote:
> No, I wasn't arguing that it should be okay to unplug a usb drive while
> the system is suspended, I was saying that it is better for the kernel
> to assume you did not when it can't really tell, since you aren't
> supposed to do that anyhow.
I wonder if this problem can be solved with a generic multi-path layer.
Every time a new Device shows up, it is tested for known volumnes and
re-attached. That way the USB layer can still generate new devices, and the
filesystem layer will not lose the connection to the devices.
(with the addtional benefit that it also works cleaner for real multipath
server environments)
Gruss
Bernd
Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> For a), the current kernel behaviour is what we want - make the thing
> appear at a new place in the namespace and in the hierarchy. ?Then
> userspace can do whatever needs to be done to identify the device, and
> apply some sort of policy decision to the result.
How? If you have a running user space the connection to the open files
is already severed, as any access in that time window must fail.
For the rest we have udev.
Oliver
Oliver Neukum <[email protected]> wrote:
>
> Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > For a), the current kernel behaviour is what we want - make the thing
> > appear at a new place in the namespace and in the hierarchy. ?Then
> > userspace can do whatever needs to be done to identify the device, and
> > apply some sort of policy decision to the result.
>
> How? If you have a running user space the connection to the open files
> is already severed, as any access in that time window must fail.
That's a separate issue, which we haven't discussed yet. We have a device
which has gone away and which might come back later on. Presently we will
return an I/O error if I/O is attempted in that window. Obviously we'll
need to do something different, such as block reads and block or defer writes.
So an overall picture would be something like:
- When device is not present, reads block and writes are deferred or block
- When device appears (at a new point in the namespace) the hotplug
scripts will go off and will identify it and will apply some policy based
upon that identification. That policy could be one of:
a) accept device at its new address
b) accept device at its new address, abandon the old address (all
those blocked reads and writes suddenly return -EIO).
c) remove device from its new address, splice it back to where it
used to be, permit those blocked reads and writes to proceed.
- For devices which are absent-and-blocked, provide some ability for both
a timeout and manual cancellation, to unblock all those readers and
writers, make them get -EIO.
The number of potential races is, of course, huge.
On Sun, Feb 19, 2006 at 08:43:44PM +0100, Pavel Machek wrote:
> Kernel can not tell the diference, and just because you don't like the
> behaviour does not mean we have to work around hardware limitation.
>
> You deal with it. Post a patch.
It would take more than one patch, but it could be done:
Suspend time:
1- Freeze all processes enough that no filesystem activity happens
anymore.
2- Do a map of the currently-opened files. That is, for each opened
file, find a workable path to it. If you have the dentry, you have
it. Otherwise, on a filesystem that supports inodes, just
sillyrename them to something in /.suspend. For the non-inode
filesystems, require not dropping the dentries for reliable suspend.
They probably need to keep them in some form anyway for handling
multiple opens of the same file.
3- Sync the filesystems to the point that they're in clean state.
4- Add the path mapping from (2) to the suspend image.
5- Allow userspace to pick up any information it finds relevant
about the filesystems/devices and put that info in the image too.
Resume time:
6- Resume userspace checks the existing devices, find a mapping (or
a lack of) using the information from (5) and the paths from (4)
7- Rebuild the userspace from the image.
8- Re-open/remap the files from the paths, sillyrenamed entries are
unlinked once reopened. Paths not found and filesystems not found
are send to a virtual file on a virtual filesystem that just -EIOs
or SEGVs mmaps.
Pros:
- You never lose filesystems or files, ever. Only thing you can lose
is processes.
- You can touch files in the suspended filesystems (device plugging,
restart on the wrong kernel) without damage.
- You can remount the filesystems ro after 3, which allows access to
whatever files the chrome may like.
- You only have to save the anonymous pages and the metadata.
Cons:
- Suspend may be slower since the write patterns are more dispersed
than just swap. OTOH, it will be no slower than sync(1).
There are probably things I haven't thought of, but I think it's the
kind of approach you need to be sure to be reliable no matter what.
OG.
On Po 20-02-06 01:56:17, Olivier Galibert wrote:
> On Sun, Feb 19, 2006 at 08:43:44PM +0100, Pavel Machek wrote:
> > Kernel can not tell the diference, and just because you don't like the
> > behaviour does not mean we have to work around hardware limitation.
> >
> > You deal with it. Post a patch.
>
> It would take more than one patch, but it could be done:
Actually, if you really want to do this, it would probably make sense
to do at blockdevice level -- with device mapper magic or something.
That way you could prompt user "return that flash driver, I still want
to write to it" after surprise unplug, etc. And suspend is special
case of surprise unplug, then replug.
Pavel
--
Web maintainer for suspend.sf.net (http://www.sf.net/projects/suspend) wanted...
On Mon, Feb 20, 2006 at 02:01:02AM +0100, Pavel Machek wrote:
> Actually, if you really want to do this, it would probably make sense
> to do at blockdevice level -- with device mapper magic or something.
>
> That way you could prompt user "return that flash driver, I still want
> to write to it" after surprise unplug, etc. And suspend is special
> case of surprise unplug, then replug.
I'm not sure. Suspend is not a surprise, so you can do things so that
you don't lose anything (what I described is pretty much unmounting
while keeping file references). Surprise unplug, there is no way you
can make the filesystem clean if it wasn't already.
I also think that USB flash and this kind of things should go back to
clean state as soon as possible even whe mountd, but that's a
different issue.
OG.
On Mon, 20 Feb 2006, Olivier Galibert wrote:
> On Mon, Feb 20, 2006 at 02:01:02AM +0100, Pavel Machek wrote:
> > Actually, if you really want to do this, it would probably make sense
> > to do at blockdevice level -- with device mapper magic or something.
> >
> > That way you could prompt user "return that flash driver, I still want
> > to write to it" after surprise unplug, etc. And suspend is special
> > case of surprise unplug, then replug.
>
> I'm not sure. Suspend is not a surprise, so you can do things so that
> you don't lose anything (what I described is pretty much unmounting
> while keeping file references). Surprise unplug, there is no way you
> can make the filesystem clean if it wasn't already.
I like Pavel's suggestion. It's simpler at the device driver level
(unless you're working on the device-mapper driver!) and it can apply to
situations other than unplug-during-suspend.
As for making the filesystem clean following a surprise unplug... You can
if the device is plugged back in later and you recognize it as the same
device.
> I also think that USB flash and this kind of things should go back to
> clean state as soon as possible even whe mountd, but that's a
> different issue.
A new mount option, like "-o clean"? Something that would flush data
writes immediately (like "sync") and would flush metadata writes whenever
a file is closed and other similar occasions? People have been asking for
something like this. Mounting with "sync" is very bad for flash memory
devices, and mounting without it messes up progress-bar displays in user
programs. Something in between would be appreciated. Especially for the
VFAT driver.
Alan Stern
Am Sonntag, 19. Februar 2006 22:02 schrieb Andrew Morton:
> Oliver Neukum <[email protected]> wrote:
> >
> > Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > > For a), the current kernel behaviour is what we want - make the thing
> > > appear at a new place in the namespace and in the hierarchy. ?Then
> > > userspace can do whatever needs to be done to identify the device, and
> > > apply some sort of policy decision to the result.
> >
> > How? If you have a running user space the connection to the open files
> > is already severed, as any access in that time window must fail.
>
> That's a separate issue, which we haven't discussed yet. ?We have a device
> which has gone away and which might come back later on. ?Presently we will
> return an I/O error if I/O is attempted in that window. ?Obviously we'll
> need to do something different, such as block reads and block or defer writes.
But how do you handle memory management?
If you simply block writes, the system will stall random tasks laundering
pages, including those needed to make progress. Even syncing before
suspend won't help you, as a running user space may dirty pages.
And what about the rootfs?
Oliver
Oliver Neukum <[email protected]> wrote:
>
> Am Sonntag, 19. Februar 2006 22:02 schrieb Andrew Morton:
> > Oliver Neukum <[email protected]> wrote:
> > >
> > > Am Sonntag, 19. Februar 2006 21:02 schrieb Andrew Morton:
> > > > For a), the current kernel behaviour is what we want - make the thing
> > > > appear at a new place in the namespace and in the hierarchy. ?Then
> > > > userspace can do whatever needs to be done to identify the device, and
> > > > apply some sort of policy decision to the result.
> > >
> > > How? If you have a running user space the connection to the open files
> > > is already severed, as any access in that time window must fail.
> >
> > That's a separate issue, which we haven't discussed yet. ?We have a device
> > which has gone away and which might come back later on. ?Presently we will
> > return an I/O error if I/O is attempted in that window. ?Obviously we'll
> > need to do something different, such as block reads and block or defer writes.
>
> But how do you handle memory management?
> If you simply block writes, the system will stall random tasks laundering
> pages, including those needed to make progress. Even syncing before
> suspend won't help you, as a running user space may dirty pages.
Well of _course_ that will happen. If we have dirty pages we need to
either keep them in memory or lose them. If someone wants to run a massive
memory stresstest, unplug the disks in the middle of it and have the
machine serenely sail along as if nothing happened then they're being
unreasonable.
> And what about the rootfs?
If you disconnect that then everything stops until you reconnect it, provided
all the tools needed to handle the reconnect are in the correct place - if
the system providers got that wrong then the machine is obviously toast.
Your questions boil down to "what if the user is crazy or the implementation
is buggy?". Let's assume neither is true, OK?
Andrew Morton <[email protected]> wrote:
>
> > If you simply block writes, the system will stall random tasks laundering
> > pages, including those needed to make progress. Even syncing before
> > suspend won't help you, as a running user space may dirty pages.
>
> Well of _course_ that will happen.
Actually, it won't happen. There's already logic in there to help pdflush,
kswapd and memory-allocating tasks avoid blocking on congested queues.
It's trivial to extend that to avoidance of hotunplugged queues.
Things like sync(), fsync(), O_SYNC and reads will necessarily block.
We may or may not decide to block on page-dirtyings. Again, that's trivial
to do in balance_dirty_pages().
Race conditions are pretty much unavoidable - if someone goes and disables
a device when we're partway through and committed to I/O submission then
things will get very sticky. But we can have a pretty successful solution
to all of this without a ton of effort.
But this is all the easy part.