Hello Linux FAT developers and developers of related tools,
there seem to be different interpretations around how to read a FAT
directory that contains an entry starting with a NUL byte before further
entries *NOT* starting with NUL bytes. I uploaded an example image of
this type to [1]. It contains three zero-byte files "WINDOWS", "IS",
"GREAT", an empty directory slot and a file with contents named "NOT".
Listing a floppy with this directory contents in Windows gives the
output "WINDOWS", "IS" and "GREAT". Mounting the image in Linux results
in four visible files. Running dosfsck on that image outputs four files,
everything OK, while Windows CHKDSK will found one lost cluster in one
chain. mdir only lists the three files Windows will find too, but when
you copy a file that needs a long name to the image using mcopy, the
file will be appended where there are enough free slots in succession,
i.e. it will not fill the single empty slot. That means mcopy allocates
space for the file, writes a directory entry for it, but mdir does not
see it afterwards. On the other hand, as soon as you add a file that
does not need a long name entry that fills the hole, mdir shows all the
files it previously didn't show.
I'm afraid this is not a pure academic post, but I write it because I
spent hours on trying to find out why I had problem to make some USB
flash drive bootable with syslinux (which follows the MS model on
loading files while booting) after copying files with the linux kernel
(which follows the Linux model, obviously), additionally dosfsck on that
stick showed quite strange results (it dived into a directory that does
no longer exist). Another incarnation is mentioned in
https://wiki.physik.fu-berlin.de/linux-minidisc/doku.php?id=himddiskformat
in the section "Filesystem Issues with some Models on Linux". Some of
the Sony devices clear only the first half of the cluster containing
that subdirectory to zeroes, which works fine if software aborts on the
first entry being zero, but obviously makes problem if the second half
of the cluster is accessed.
I would really appreciate it if all FOSS working on FAT would agree to
one behaviour, and I think we have no choice except being MS compatible
(i.e. implement the behaviour as it has always been in DOS and Windows)
Please comment.
Regards,
Michael Karcher
Am Freitag, den 22.04.2011, 20:48 +0200 schrieb Michael Karcher:
> Hello Linux FAT developers and developers of related tools,
>
> there seem to be different interpretations around how to read a FAT
> directory that contains an entry starting with a NUL byte before further
> entries *NOT* starting with NUL bytes. I uploaded an example image of
> this type to [1]. It contains three zero-byte files "WINDOWS", "IS",
> "GREAT", an empty directory slot and a file with contents named "NOT".
[1]: http://users.physik.fu-berlin.de/~mkarcher/fatimg.gz
Michael Karcher <[email protected]> writes:
> I'm afraid this is not a pure academic post, but I write it because I
> spent hours on trying to find out why I had problem to make some USB
> flash drive bootable with syslinux (which follows the MS model on
> loading files while booting) after copying files with the linux kernel
> (which follows the Linux model, obviously), additionally dosfsck on that
> stick showed quite strange results (it dived into a directory that does
> no longer exist). Another incarnation is mentioned in
> https://wiki.physik.fu-berlin.de/linux-minidisc/doku.php?id=himddiskformat
> in the section "Filesystem Issues with some Models on Linux". Some of
> the Sony devices clear only the first half of the cluster containing
> that subdirectory to zeroes, which works fine if software aborts on the
> first entry being zero, but obviously makes problem if the second half
> of the cluster is accessed.
>
> I would really appreciate it if all FOSS working on FAT would agree to
> one behaviour, and I think we have no choice except being MS compatible
> (i.e. implement the behaviour as it has always been in DOS and Windows)
It is simply the bug of Sony devices, and can't work on it for both of
Windows and Linux (it have to be fixed by fsck). Yeah, I know Linux can
stop on first zeroed directory entry (I will not complain about it), but
it doesn't fix the problem at all.
Assume 0-3 directory entires fine, 4 is zeroed entry, and 5- is crappy data.
1| ENT1 valid
2| ENT2 valid
3| ENT3 valid
4| zeroed
5| djafdjoi crappy data
...
Windows will stop at 4, but if windows write new entry as 4, 5- will
show those up as crap like linux. I.e. never adds new zeroed entry as
end of directory. In short, in spec, directory must be initialized by
zero (it is same assumption on linux and windows).
Thanks.
--
OGAWA Hirofumi <[email protected]>
Am Samstag, den 23.04.2011, 05:40 +0900 schrieb OGAWA Hirofumi:
> It is simply the bug of Sony devices, and can't work on it for both of
> Windows and Linux (it have to be fixed by fsck).
You are right. Although as the sony device *does* clear out the first 8K
of the 16K cluster, that makes 256 correctly working directory entries.
This directory is only used for storing a dozen of files with no long
names, the existing problem you correctly describe is not relevant in
practice.
> Assume 0-3 directory entires fine, 4 is zeroed entry, and 5- is crappy data.
>
> 1| ENT1 valid
> 2| ENT2 valid
> 3| ENT3 valid
> 4| zeroed
> 5| djafdjoi crappy data
> ...
>
> Windows will stop at 4, but if windows write new entry as 4, 5- will
> show those up as crap like linux.
You are right. Thanks for your input. This means Windows and mtools
behave exactly the same way (and thus mtools can not really be
considered buggy). I expected Windows to fix up the end, but it does
not.
I still would prefer greatly if Windows and Linux at least read media
the same way (i.e. stop on the first zeroed entry), I even have a patch
for that somewhere around I can dig out.
I furthermore think it is quite important for dosfsck and the linux
kernel to behave consistent (think of valid entries after the zeroed
entry linking to "lost clusters"). If I prepare patches for the Linux
kernel to
- terminate reading at the first zeroed entry
- making sure that if a zeroed entry gets populated, the next entry
gets zeroed unless we hit the end of the directory (deviating from
Windows behaviour, intentionally, to keep the filesystem consistent)
And for dosfsck to
- terminate reading at the first zeroed entry
- offer filling the remainder of a directory cluster with zero bytes.
Would this set of patches be acceptable?
> In short, in spec, directory must be initialized by
> zero (it is same assumption on linux and windows).
Turns out you are right. Sorry for making the implicit assumption that
windows handles the case better.
Regards,
Michael Karcher
Michael Karcher <[email protected]> writes:
>> Windows will stop at 4, but if windows write new entry as 4, 5- will
>> show those up as crap like linux.
> You are right. Thanks for your input. This means Windows and mtools
> behave exactly the same way (and thus mtools can not really be
> considered buggy). I expected Windows to fix up the end, but it does
> not.
>
> I still would prefer greatly if Windows and Linux at least read media
> the same way (i.e. stop on the first zeroed entry), I even have a patch
> for that somewhere around I can dig out.
> I furthermore think it is quite important for dosfsck and the linux
> kernel to behave consistent (think of valid entries after the zeroed
> entry linking to "lost clusters"). If I prepare patches for the Linux
> kernel to
> - terminate reading at the first zeroed entry
This is acceptable (stop at zero is an simply optimization), and I'll
just care about stability and cleanness about it.
> - making sure that if a zeroed entry gets populated, the next entry
> gets zeroed unless we hit the end of the directory (deviating from
> Windows behaviour, intentionally, to keep the filesystem consistent)
This would be unacceptable. There are several FAT implementations like
this crap, I can't honor an one of those until to decide to try to
support all of craps.
I.e. This is the out of specs anymore. Why can you say those are invalid
entries even if Windows honors those entries? And the above is simulate
Windows implementation over specs, why isn't here? For overwriting the
zeroed entry, why we have to check all entries until see next zeroed
(yeah, now, we allowed the crappy data after zeroed)?
My state is simple - the current implement should be fine.
I.e. uninitialized directory is the simply bug. (However, I can honor to
stop at zero, because it is mentioned in spec as optimization.)
> And for dosfsck to
> - terminate reading at the first zeroed entry
> - offer filling the remainder of a directory cluster with zero bytes.
> Would this set of patches be acceptable?
Sorry. I can't say about dosfsck, I'm not maintainer of it. Ideas,
dosfstools maintainer?
--
OGAWA Hirofumi <[email protected]>
Am Samstag, den 23.04.2011, 09:06 +0900 schrieb OGAWA Hirofumi:
> > - terminate reading at the first zeroed entry
> This is acceptable (stop at zero is an simply optimization), and I'll
> just care about stability and cleanness about it.
OK, then I will dig out my patch for that.
> > - making sure that if a zeroed entry gets populated, the next entry
> > gets zeroed unless we hit the end of the directory (deviating from
> > Windows behaviour, intentionally, to keep the filesystem consistent)>
> I.e. This is the out of specs anymore. Why can you say those are invalid
> entries even if Windows honors those entries?
My point is that creating a file on a consistent file system (i.e. one
where all cluster chains are referenced by exactly one visible directory
entry) should still result in a consistent file system. And by
"consistent" I mean any state windows chkdsk accepts.
> And the above is simulate Windows implementation over specs, why isn't
> here?
Wow, there actually *is* a formal specification for FAT? I am positively
surprised by that. I used to think that the "spec" for FAT is "do as DOS
does". On a quick google, I do find the MS EFI FAT32 specification,
which is obviuosly non-free. Is there anything else?
> For overwriting the
> zeroed entry, why we have to check all entries until see next zeroed
> (yeah, now, we allowed the crappy data after zeroed)?
My point was not about checking all entries until the next zeroed entry,
which is overcomplicating stuff. I meant that if we hit the end (i.e.
the first zeroed entry) when searching for free slots, in that case we
just clear the one entry directly following the inserted entries. This
makes sure that no files magically appear. I *do* understand that they
also appear on Windows, though. My point about "modifying a FAT system
that passes MS chkdsk with Linux should result in a FAT system that
passes MS chkdsk" is in my eyes more important than "Windows damages a
file system it considers fine in this situation, so we should do the
same."
In the end, there are two valid interpretations on the problem at hand:
- DOS/Windows is buggy in exposing the crap when filling the "hole",
which is proven by chkdsk not complaining about the garbage in the
directory, and we should not follow the bug in DOS/Windows
- chkdsk is buggy in not complaining about trailing garbage in
directories, which are proven to be invalid by DOS/Windows messing up on
them, and we shouldn't complicate our FAT implementation by trying to
work around a bug in chkdsk.
Obviously, I tend to the first interpretation while you prefer the
second interpretation. As you are the maintainer, it looks like I have
to get away with that.
> My state is simple - the current implement should be fine.
> I.e. uninitialized directory is the simply bug.
For the uninitialized entries on Hi-MD media, I agree with you that
these are caused by a firmware bug of older Sony devices. And to support
these devices "good enough", the stop-at-zero would be enough.
Regretfully I don't know the history of the USB flash memory I recently
got where the root directory was ended by a single entry starting with a
NUL byte and containing valid entries past that. The experience with the
inconsistent behaviour of different tools on that flash memory made me
write the mail about the current state of the FAT implementation.
> > And for dosfsck to
> > - terminate reading at the first zeroed entry
> > - offer filling the remainder of a directory cluster with zero bytes.
> > Would this set of patches be acceptable?
> Sorry. I can't say about dosfsck, I'm not maintainer of it. Ideas,
> dosfstools maintainer?
I think the *most* important fact is to have at least dosfsck and linux
consistent about directories with trailing garbage. So I won't even try
to submit a patch to the Linux kernel that stops reading at the first
unused directory entry if dosfsck will not accept to stop reading at the
first unused directory entry.
Regards and thanks for discussing,
Michael Karcher
Michael Karcher <[email protected]> writes:
>> And the above is simulate Windows implementation over specs, why isn't
>> here?
> Wow, there actually *is* a formal specification for FAT? I am positively
> surprised by that. I used to think that the "spec" for FAT is "do as DOS
> does". On a quick google, I do find the MS EFI FAT32 specification,
> which is obviuosly non-free. Is there anything else?
I guess it is called fatgen.doc in past, but I'm not sure. I can't help
about what MS does.
>> For overwriting the
>> zeroed entry, why we have to check all entries until see next zeroed
>> (yeah, now, we allowed the crappy data after zeroed)?
> My point was not about checking all entries until the next zeroed entry,
> which is overcomplicating stuff. I meant that if we hit the end (i.e.
> the first zeroed entry) when searching for free slots, in that case we
> just clear the one entry directly following the inserted entries. This
> makes sure that no files magically appear. I *do* understand that they
> also appear on Windows, though. My point about "modifying a FAT system
> that passes MS chkdsk with Linux should result in a FAT system that
> passes MS chkdsk" is in my eyes more important than "Windows damages a
> file system it considers fine in this situation, so we should do the
> same."
I'll call it - first step of in-kernel fsck.
> In the end, there are two valid interpretations on the problem at hand:
> - DOS/Windows is buggy in exposing the crap when filling the "hole",
> which is proven by chkdsk not complaining about the garbage in the
> directory, and we should not follow the bug in DOS/Windows
First of all, the bug is non-initialized directory. That's all.
Yes, chkdsk should fix it. But, unfortunately chkdsk is not perfect, I
remember another breakage of chkdsk can't fix.
> - chkdsk is buggy in not complaining about trailing garbage in
> directories, which are proven to be invalid by DOS/Windows messing up on
> them, and we shouldn't complicate our FAT implementation by trying to
> work around a bug in chkdsk.
>
> Obviously, I tend to the first interpretation while you prefer the
> second interpretation. As you are the maintainer, it looks like I have
> to get away with that.
Because fsck is the job of fsck. Furthermore, without interactivity,
fsck can't perfectly decide how to fix it actually (yes, it can guess,
but that's guess).
>> My state is simple - the current implement should be fine.
>> I.e. uninitialized directory is the simply bug.
> For the uninitialized entries on Hi-MD media, I agree with you that
> these are caused by a firmware bug of older Sony devices. And to support
> these devices "good enough", the stop-at-zero would be enough.
> Regretfully I don't know the history of the USB flash memory I recently
> got where the root directory was ended by a single entry starting with a
> NUL byte and containing valid entries past that. The experience with the
> inconsistent behaviour of different tools on that flash memory made me
> write the mail about the current state of the FAT implementation.
Yes, zero is the end of directory. It's right and said in the above
fatgen. Guys might misread - fatgen says the rest of entries must be
zero (i.e. it means initialized by zero). Or some spec (I know there is
the FS spec in SD specs, I can't recall the detail though) may not
mention about latter part, I don't know.
> I think the *most* important fact is to have at least dosfsck and linux
> consistent about directories with trailing garbage. So I won't even try
> to submit a patch to the Linux kernel that stops reading at the first
> unused directory entry if dosfsck will not accept to stop reading at the
> first unused directory entry.
OK. I think "stop at zero" is not effected by dosfsck though. With "stop
at zero", linux will do like windows does, but crap data is already
showed in current implementation.
Whether dosfsck can fix it or not would be another story. Of course, I
hope it does though.
Thanks.
--
OGAWA Hirofumi <[email protected]>
Hi!
> > For overwriting the
> > zeroed entry, why we have to check all entries until see next zeroed
> > (yeah, now, we allowed the crappy data after zeroed)?
> My point was not about checking all entries until the next zeroed entry,
> which is overcomplicating stuff. I meant that if we hit the end (i.e.
> the first zeroed entry) when searching for free slots, in that case we
> just clear the one entry directly following the inserted entries. This
> makes sure that no files magically appear. I *do* understand that
This sounds like a good idea, and costs nothing. I hope we can
convince Ogawa...
> In the end, there are two valid interpretations on the problem at hand:
> - DOS/Windows is buggy in exposing the crap when filling the "hole",
> which is proven by chkdsk not complaining about the garbage in the
> directory, and we should not follow the bug in DOS/Windows
> - chkdsk is buggy in not complaining about trailing garbage in
> directories, which are proven to be invalid by DOS/Windows messing up on
> them, and we shouldn't complicate our FAT implementation by trying to
> work around a bug in chkdsk.
I believe 2) is the case; but we should still work around that chkdsk
bug.
> I think the *most* important fact is to have at least dosfsck and linux
> consistent about directories with trailing garbage. So I won't even try
> to submit a patch to the Linux kernel that stops reading at the first
> unused directory entry if dosfsck will not accept to stop reading at the
> first unused directory entry.
They should *not* be consistent.
Kernel should stop at zero invalid entry.
fsck should consider any garbage past zero entry as an error, and zero
it out. (Complaining about duplicate blocks is unhelpful but better
than nothing. It should really zero the garbage out.)
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Am Donnerstag, den 28.04.2011, 15:25 +0200 schrieb Pavel Machek:
> > > For overwriting the
> > > zeroed entry, why we have to check all entries until see next zeroed
> > > (yeah, now, we allowed the crappy data after zeroed)?
> > My point was not about checking all entries until the next zeroed entry,
> > which is overcomplicating stuff. I meant that if we hit the end (i.e.
> > the first zeroed entry) when searching for free slots, in that case we
> > just clear the one entry directly following the inserted entries. This
> > makes sure that no files magically appear. I *do* understand that
> This sounds like a good idea, and costs nothing. I hope we can
> convince Ogawa...
I am going to implement that and publish a patch. If the patch is clean
enough, maybe we can convince him. But the freedom of open source can't
prevent me from using a patch like that, of course. And maybe it also
helps when the patch gets some testing.
> > I think the *most* important fact is to have at least dosfsck and linux
> > consistent about directories with trailing garbage. So I won't even try
> > to submit a patch to the Linux kernel that stops reading at the first
> > unused directory entry if dosfsck will not accept to stop reading at the
> > first unused directory entry.
> They should *not* be consistent.
>
> Kernel should stop at zero invalid entry.
>
> fsck should consider any garbage past zero entry as an error, and zero
> it out. (Complaining about duplicate blocks is unhelpful but better
> than nothing. It should really zero the garbage out.)
In fact, what you describe is something I would call "consistent". I was
not implying that kernel and dosfsck should do exactly the same thing,
but I was implying that the kernel and dosfsck should either both or
none consider entries past the gap as existing file. Current state is:
both consider past-gap entries valid. Your described state will be: none
consider past-gap entries valid. The behaviour you describe is exactly
what I would imagine as best way to go. Maybe fsck should ask for "shift
post-gap entries/create a dummy deleted entry" vs. "clear post-gap
entries".
Regards,
Michael Karcher
Michael Karcher <[email protected]> writes:
> Am Donnerstag, den 28.04.2011, 15:25 +0200 schrieb Pavel Machek:
>> > > For overwriting the
>> > > zeroed entry, why we have to check all entries until see next zeroed
>> > > (yeah, now, we allowed the crappy data after zeroed)?
>> > My point was not about checking all entries until the next zeroed entry,
>> > which is overcomplicating stuff. I meant that if we hit the end (i.e.
>> > the first zeroed entry) when searching for free slots, in that case we
>> > just clear the one entry directly following the inserted entries. This
>> > makes sure that no files magically appear. I *do* understand that
>> This sounds like a good idea, and costs nothing. I hope we can
>> convince Ogawa...
>
> I am going to implement that and publish a patch. If the patch is clean
> enough, maybe we can convince him. But the freedom of open source can't
> prevent me from using a patch like that, of course. And maybe it also
> helps when the patch gets some testing.
Of course, it's good to publish. However if you can't detect buggy or
corrupted directory, I wouldn't ack to help buggy stuff.
So it can be the mount option, but I can't see at all the point it is
in-kernel as FS.
>> Kernel should stop at zero invalid entry.
I'm not complaining to stop at zero. Rather I acked already if it didn't
overwrite after zeroed entry.
>> fsck should consider any garbage past zero entry as an error, and zero
>> it out. (Complaining about duplicate blocks is unhelpful but better
>> than nothing. It should really zero the garbage out.)
>
> In fact, what you describe is something I would call "consistent". I was
> not implying that kernel and dosfsck should do exactly the same thing,
> but I was implying that the kernel and dosfsck should either both or
> none consider entries past the gap as existing file.
> Current state is: both consider past-gap entries valid. Your described
> state will be: none consider past-gap entries valid. The behaviour you
> describe is exactly what I would imagine as best way to go. Maybe fsck
> should ask for "shift post-gap entries/create a dummy deleted entry"
> vs. "clear post-gap entries".
Yes. Also my opinion is the fsck should get ack from user to overwrite
data or salvage (has possibility to salvage) if interactive mode, then
fixes it.
Thanks.
--
OGAWA Hirofumi <[email protected]>
Hi!
> > Kernel should stop at zero invalid entry.
> >
> > fsck should consider any garbage past zero entry as an error, and zero
> > it out. (Complaining about duplicate blocks is unhelpful but better
> > than nothing. It should really zero the garbage out.)
>
> In fact, what you describe is something I would call "consistent". I was
> not implying that kernel and dosfsck should do exactly the same thing,
> but I was implying that the kernel and dosfsck should either both or
Aha, violent agreement, then. Good :-).
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hello,
I've just released mtools 4.0.17
It fixes the issue by:
1. Considering all directory entry after an "endmark" (first byte of
filename zero) to be free, even if the first byte is not zero (this
avoids the problem of not finding any free space because of phantom
entries after endmark)
2. Writing an endmark after having created a filename that's beyond the
endmark (this avoids the problem of accidentally transforming phantom
entries into real entries by "filling the gap")
http://www.gnu.org/software/mtools/download.html
Thanks for drawing my attention to this issue.
Regards,
Alain
On 2011-04-28 17:44, OGAWA Hirofumi wrote:
> Michael Karcher <[email protected]> writes:
>
>> Am Donnerstag, den 28.04.2011, 15:25 +0200 schrieb Pavel Machek:
>>>>> For overwriting the
>>>>> zeroed entry, why we have to check all entries until see next zeroed
>>>>> (yeah, now, we allowed the crappy data after zeroed)?
>>>> My point was not about checking all entries until the next zeroed entry,
>>>> which is overcomplicating stuff. I meant that if we hit the end (i.e.
>>>> the first zeroed entry) when searching for free slots, in that case we
>>>> just clear the one entry directly following the inserted entries. This
>>>> makes sure that no files magically appear. I *do* understand that
>>> This sounds like a good idea, and costs nothing. I hope we can
>>> convince Ogawa...
>>
>> I am going to implement that and publish a patch. If the patch is clean
>> enough, maybe we can convince him. But the freedom of open source can't
>> prevent me from using a patch like that, of course. And maybe it also
>> helps when the patch gets some testing.
>
> Of course, it's good to publish. However if you can't detect buggy or
> corrupted directory, I wouldn't ack to help buggy stuff.
>
> So it can be the mount option, but I can't see at all the point it is
> in-kernel as FS.
>
>>> Kernel should stop at zero invalid entry.
>
> I'm not complaining to stop at zero. Rather I acked already if it didn't
> overwrite after zeroed entry.
>
>>> fsck should consider any garbage past zero entry as an error, and zero
>>> it out. (Complaining about duplicate blocks is unhelpful but better
>>> than nothing. It should really zero the garbage out.)
>>
>> In fact, what you describe is something I would call "consistent". I was
>> not implying that kernel and dosfsck should do exactly the same thing,
>> but I was implying that the kernel and dosfsck should either both or
>> none consider entries past the gap as existing file.
>> Current state is: both consider past-gap entries valid. Your described
>> state will be: none consider past-gap entries valid. The behaviour you
>> describe is exactly what I would imagine as best way to go. Maybe fsck
>> should ask for "shift post-gap entries/create a dummy deleted entry"
>> vs. "clear post-gap entries".
>
> Yes. Also my opinion is the fsck should get ack from user to overwrite
> data or salvage (has possibility to salvage) if interactive mode, then
> fixes it.
>
> Thanks.