(I did subscribe to fs-devel, but it seems not to like @sf alias and unsubscribed me after a while - and did that circle a few times; please CC).
Have a rather broken hard disk at the moment - the bulk of it is ext3 on top of lvm2 - the default for fedora almost exactly 4 years ago. So been doing a lot of 'e2fsck -fv -y ...' lately.
Am a little surprised about what "Entry <file> in <dir> (<num>) has deleted/unused inode <num> Clear? yes" does. it deletes the file.
I wonder should it be re-tried, or relocated to /lost+found, or anything else?
I also want to make a suggestion: when such a decision is made, I'd like to keep a record of what files are lost, etc. Can an option be added to e2fsck to log its decisions somewhere?
Thanks for all the hard-work to make fsck even possible - the hard disk has been well-used :-).
For those people who might come across this in the future - here is a tips, and may be another question/discussion about sane behavior on error:
it appears that some combination of SATA->USB enclosure plus usb-storage plus usb hci driver doesn't like read errors, and the kernel resets the usb device and reconnects whenever that happens. That has the unfortunate side effect of renaming /dev/sdb into /dev/sdc, etc, underneath lvm2, and causes lvm2 to get stuck. I suppose that's the sane behavior since one wants to stop I/O and further damage, but maybe the kernel should only reset usb storage devices on write-errors, rather than read-errors? And somehow limit the device to read-only access after a read-error?
In any case, I somehow found that I could persuade the kernel not to reset the usb device, if I disable read-ahead (i.e. hdparm -a0/-A0). I also tried disabling read-ahead on the lvm2 device itself (-A0 /dev/dm-X, where X is the logical volume), and "echo 128 > /sys/block/sdc/device/max_sectors" (or smaller number).
So it seems that slowing it down make it more reliable for an unreliable disk. Maybe this can be made automatic inside the kernel?
On Feb 3, 2012, at 9:51 PM, Hin-Tak Leung wrote:
> (I did subscribe to fs-devel, but it seems not to like @sf alias and unsubscribed me after a while - and did that circle a few times; please CC).
Most of this message isn't appropriate for fs-devel. The ext4 bits are
germane to the ext4 list, and the USB read errors should be taken to
the usb folks. I've removed fs-devel from the reply-to header, and
I'll ask people who are replying to further edit the cc list if appropriate.
> Have a rather broken hard disk at the moment - the bulk of it is ext3 on top of lvm2 - the default for fedora almost exactly 4 years ago. So been doing a lot of 'e2fsck -fv -y ...' lately.
>
> Am a little surprised about what "Entry <file> in <dir> (<num>) has deleted/unused inode <num> Clear? yes" does. it deletes the file.
>
> I wonder should it be re-tried, or relocated to /lost+found, or anything else?
This error means the inode entry (and perhaps the entire inode table
block in your case) has been zapped. So all that's left is the directory
entry, containing the file name and inode number. There's nothing
really left to save.
> I also want to make a suggestion: when such a decision is made, I'd like to keep a record of what files are lost, etc. Can an option be added to e2fsck to log its decisions somewhere?
I suggest that you look at using the "script" program if you are running
e2fsck manually, or if you are interested in saving the output of fsck
during the boot sequence, if your distro isn't already using logsave,
consider using that binary.
> it appears that some combination of SATA->USB enclosure plus usb-storage plus usb hci driver doesn't like read errors, and the kernel resets the usb device and reconnects whenever that happens. That has the unfortunate side effect of renaming /dev/sdb into /dev/sdc, etc, underneath lvm2, and causes lvm2 to get stuck. I suppose that's the sane behavior since one wants to stop I/O and further damage, but maybe the kernel should only reset usb storage devices on write-errors, rather than read-errors? And somehow limit the device to read-only access after a read-error?
I'd suggest that you send your dmesg output to the USB folks.
That might give them more of an idea which portion of the
stack is causing the stack is causing the reset. It may very
well be the SATA->USB enclosure is reporting some error
much worse than a read error as a result of the error.
Regards,
-- Ted
--- On Sat, 4/2/12, Theodore Tso <[email protected]> wrote:
<snipped>
> Most of this message isn't appropriate for
> fs-devel.???The ext4 bits are
> germane to the ext4 list, and the USB read errors should be
> taken to
> the usb folks.? ? I've removed fs-devel from the
> reply-to header, and
> I'll ask people who are replying to further edit the cc list
> if appropriate.
Fair enough - I just thought the interaction between usb-storage and lvm2 would warrant some wider discussion.
> This error means the inode entry (and perhaps the entire
> inode table
> block in your case) has been zapped.? So all that's
> left is the directory
> entry, containing the file name and inode
> number.???There's nothing
> really left to save.
>
> > I also want to make a suggestion: when such a decision
> is made, I'd like to keep a record of what files are lost,
> etc. Can an option be added to e2fsck to log its decisions
> somewhere?
>
> I suggest that you look at using the "script" program if you
> are running
> e2fsck manually, or if you are interested in saving the
> output of fsck
> during the boot sequence, if your distro isn't already using
> logsave,
> consider using that binary.
That files are lost forever I understand, but from a user's point of view, the most important thing is preserving data, or at least, when lost is inevitable, know that they are lost and hope to recover that somewhere. Is it possible to offer that - i.e. prompt for a log file - when such a decision is needed, when run interactively?
> > it appears that some combination of SATA->USB
> enclosure plus usb-storage plus usb hci driver doesn't like
> read errors, and the kernel resets the usb device and
> reconnects whenever that happens. That has the unfortunate
> side effect of renaming /dev/sdb into /dev/sdc, etc,
> underneath lvm2, and causes lvm2 to get stuck. I suppose
> that's the sane behavior since one wants to stop I/O and
> further damage, but maybe the kernel should only reset usb
> storage devices on write-errors, rather than read-errors?
> And somehow limit the device to read-only access after a
> read-error?
>
> I'd suggest that you send your dmesg output to the USB
> folks.
> That might give them more of an idea which portion of the
> stack is causing the stack is causing the
> reset.???It may very
> well be the SATA->USB enclosure is reporting some error
> much worse than a read error as a result of the error.
Thanks. The curious thing is that running hdparm -a0/-A0 can get around the reset, so it seems to be a performance related issue.
On Sat, Feb 04, 2012 at 05:53:10PM +0000, Hin-Tak Leung wrote:
>
> Fair enough - I just thought the interaction between usb-storage and
> lvm2 would warrant some wider discussion.
If device driver thinks the device has disappeared, and then it
reappears so that it gets a new identity (i.e. sdb -> sdc), the
problem is below LVM2. It's either a problem with the hardware
actually disconnecting from the USB bus, or the with the device
driver. But the people who have expertise with the USB device driver
and the USB hardware aren't going to be on fs-devel.
> That files are lost forever I understand, but from a user's point of
> view, the most important thing is preserving data, or at least, when
> lost is inevitable, know that they are lost and hope to recover that
> somewhere. Is it possible to offer that - i.e. prompt for a log file
> - when such a decision is needed, when run interactively?
If you know that fsck has failed with the automatic (preen) pass, such
that you are running fsck interactively by definition you know there
will be some decision that will be required. So why not always use
the script command when you're running e2fsck interactively?
We could replicate the functionality of script in e2fsck, but given
that script is a perfectly good implementation, it's not clear it's
worth it.
> > It may very well be the SATA->USB enclosure is reporting some
> > error much worse than a read error as a result of the error.
>
> Thanks. The curious thing is that running hdparm -a0/-A0 can get
> around the reset, so it seems to be a performance related issue.
Well, we don't know that. This is why you really need to get the USB
storage folks involved, and they don't hang out on fs-devel.
- Ted