Alan,
This is regarding a change I had to make to the kernel 2.2.17-14 to support
really large drives. In our project we deal with huge drives, sometimes
drives larger than 128GB. The file system is FAT32 and only one partition
is create on any drive. During our testing we realized that linux fat
implementation doesn't support partitions larger than 128GB(actually 64GB
because of a bug in fat implementation).
This limitation is imposed by the data structures used by the linux fat
implementation to read/write directory entries. A 'long' data type is used
to access directory entries on the disk, of which only 28 bits are used to
identify the sector that contains the directory entry and the rest 4 bits
are used to specify offset of the directory entry inside the sector. Using
28 bits to identify a sector means we cannot access sectors beyond 128GB
(2^28*512), thus limiting us from creating partitions larger than 128GB on
large disk drives.
I have made changes to fat, vfat and msdos file system implementations in
the kernel to use larger data types, thus allowing us to create larger
partitions. As per the GPL I would like to make the patch available to
everyone and also in case somebody has run into the same problem(who cares
about fat in the linux world). The patch has been fairly well tested only
on our systems(p3, 700MHz with FC). I truly appreciate if you & anybody in
the kernel mailing list have any feedback about the changes.
The patch is applicable to only 2.2.17-14 kernel and may require changes to
use with other kernel versions. As far as I know none of the kernel
versions provide any fix for this problem.
Thanks,
- Vijay
On Jan 03, 2002 15:42 -0800, Vijay Kumar wrote:
> This limitation is imposed by the data structures used by the linux fat
> implementation to read/write directory entries. A 'long' data type is used
> to access directory entries on the disk, of which only 28 bits are used to
> identify the sector that contains the directory entry and the rest 4 bits
> are used to specify offset of the directory entry inside the sector. Using
> 28 bits to identify a sector means we cannot access sectors beyond 128GB
> (2^28*512), thus limiting us from creating partitions larger than 128GB on
> large disk drives.
Some minor notes on your patch:
1) It appears you are running an editor with 4-space tabs, and as a result
it has broken the indentation of some of your changes. This is easily
seen when looking at the patch.
2) It is almost certainly wrong to use "loff_t" for an inode number. Maybe
you could use "u64" instead? I also think that using "long long"
explicitly is frowned upon.
> I have made changes to fat, vfat and msdos file system implementations in
> the kernel to use larger data types, thus allowing us to create larger
> partitions. As per the GPL I would like to make the patch available to
> everyone and also in case somebody has run into the same problem(who cares
> about fat in the linux world). The patch has been fairly well tested only
> on our systems(p3, 700MHz with FC). I truly appreciate if you & anybody in
> the kernel mailing list have any feedback about the changes.
Does this change the on-disk format for FAT at all, or is it merely a
kernel filesystem code issue? I think only the latter, but best to check.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
At 05:24 PM 1/3/2002 -0700, Andreas Dilger wrote:
>On Jan 03, 2002 15:42 -0800, Vijay Kumar wrote:
> > This limitation is imposed by the data structures used by the linux fat
> > implementation to read/write directory entries. A 'long' data type is used
> > to access directory entries on the disk, of which only 28 bits are used to
> > identify the sector that contains the directory entry and the rest 4 bits
> > are used to specify offset of the directory entry inside the sector. Using
> > 28 bits to identify a sector means we cannot access sectors beyond 128GB
> > (2^28*512), thus limiting us from creating partitions larger than 128GB on
> > large disk drives.
>
>Some minor notes on your patch:
>1) It appears you are running an editor with 4-space tabs, and as a result
> it has broken the indentation of some of your changes. This is easily
> seen when looking at the patch.
Well, I could fix it with little effort.
>2) It is almost certainly wrong to use "loff_t" for an inode number. Maybe
> you could use "u64" instead? I also think that using "long long"
> explicitly is frowned upon.
I guess its using u64 verses using loff_t. I have chosen the later because
its actually an offset. For fat implementations inode number is nothing but
offset of the directory entry on the disk. So I thought it makes more sense
to use loff_t.
> > I have made changes to fat, vfat and msdos file system implementations in
> > the kernel to use larger data types, thus allowing us to create larger
> > partitions. As per the GPL I would like to make the patch available to
> > everyone and also in case somebody has run into the same problem(who cares
> > about fat in the linux world). The patch has been fairly well tested only
> > on our systems(p3, 700MHz with FC). I truly appreciate if you & anybody in
> > the kernel mailing list have any feedback about the changes.
>
>Does this change the on-disk format for FAT at all, or is it merely a
>kernel filesystem code issue? I think only the latter, but best to check.
Doesn't change on disk format, only fixes implementation.
Thanks,
- Vijay
>Cheers, Andreas
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
FAT32 stores 28 bit cluster numbers. You need to increase cluster size
instead of the 28 bits to stay FAT32. The spec is fatgen102.pdf:, I'll
mail it to you.
Hardware White Paper Hardware White Paper
FAT: General Overview of On-Disk Format
Version 1.02, May 5, 1999
Microsoft Corporation
Vijay Kumar wrote:
> Alan,
>
> This is regarding a change I had to make to the kernel 2.2.17-14 to
> support really large drives. In our project we deal with huge drives,
> sometimes drives larger than 128GB. The file system is FAT32 and only
> one partition is create on any drive. During our testing we realized
> that linux fat implementation doesn't support partitions larger than
> 128GB(actually 64GB because of a bug in fat implementation).
>
> This limitation is imposed by the data structures used by the linux
> fat implementation to read/write directory entries. A 'long' data type
> is used to access directory entries on the disk, of which only 28 bits
> are used to identify the sector that contains the directory entry and
> the rest 4 bits are used to specify offset of the directory entry
> inside the sector. Using 28 bits to identify a sector means we cannot
> access sectors beyond 128GB (2^28*512), thus limiting us from creating
> partitions larger than 128GB on large disk drives.
>
> I have made changes to fat, vfat and msdos file system implementations
> in the kernel to use larger data types, thus allowing us to create
> larger partitions. As per the GPL I would like to make the patch
> available to everyone and also in case somebody has run into the same
> problem(who cares about fat in the linux world). The patch has been
> fairly well tested only on our systems(p3, 700MHz with FC). I truly
> appreciate if you & anybody in the kernel mailing list have any
> feedback about the changes.
>
> The patch is applicable to only 2.2.17-14 kernel and may require
> changes to use with other kernel versions. As far as I know none of
> the kernel versions provide any fix for this problem.
>
> Thanks,
> - Vijay
It is now up to Version 1.03
http://www.microsoft.com/hwdev/hardware/fatgen.asp
Phil
---------------------------------------------
Phil Randal
Network Engineer
Herefordshire Council
Hereford, UK
> -----Original Message-----
> From: CJ [mailto:[email protected]]
> Sent: 04 January 2002 02:49
> To: Vijay Kumar
> Cc: [email protected]; [email protected]
> Subject: Re: kernel patch support large fat partitions
>
>
> FAT32 stores 28 bit cluster numbers. You need to increase
> cluster size
> instead of the 28 bits to stay FAT32. The spec is
> fatgen102.pdf:, I'll
> mail it to you.
>
> Hardware White Paper Hardware White Paper
> FAT: General Overview of On-Disk Format
> Version 1.02, May 5, 1999
> Microsoft Corporation
>
> Vijay Kumar wrote:
>
> > Alan,
> >
> > This is regarding a change I had to make to the kernel 2.2.17-14 to
> > support really large drives. In our project we deal with
> huge drives,
> > sometimes drives larger than 128GB. The file system is
> FAT32 and only
> > one partition is create on any drive. During our testing we
> realized
> > that linux fat implementation doesn't support partitions
> larger than
> > 128GB(actually 64GB because of a bug in fat implementation).
> >
> > This limitation is imposed by the data structures used by the linux
> > fat implementation to read/write directory entries. A
> 'long' data type
> > is used to access directory entries on the disk, of which
> only 28 bits
> > are used to identify the sector that contains the directory
> entry and
> > the rest 4 bits are used to specify offset of the directory entry
> > inside the sector. Using 28 bits to identify a sector means
> we cannot
> > access sectors beyond 128GB (2^28*512), thus limiting us
> from creating
> > partitions larger than 128GB on large disk drives.
> >
> > I have made changes to fat, vfat and msdos file system
> implementations
> > in the kernel to use larger data types, thus allowing us to create
> > larger partitions. As per the GPL I would like to make the patch
> > available to everyone and also in case somebody has run
> into the same
> > problem(who cares about fat in the linux world). The patch has been
> > fairly well tested only on our systems(p3, 700MHz with FC). I truly
> > appreciate if you & anybody in the kernel mailing list have any
> > feedback about the changes.
> >
> > The patch is applicable to only 2.2.17-14 kernel and may require
> > changes to use with other kernel versions. As far as I know none of
> > the kernel versions provide any fix for this problem.
> >
> > Thanks,
> > - Vijay
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Using 28bit cluster numbers and 65536 cluster size I could go upto 16TB
which is lot more than what I wanted. So right now I have no problem with
the on-disk format of a fat partition. Nevertheless with hard drives prices
going down dramatically it may not be too long before we hit the limit.
Regards,
- Vijay
At 06:48 PM 1/3/2002 -0800, CJ wrote:
>FAT32 stores 28 bit cluster numbers. You need to increase cluster size
>instead of the 28 bits to stay FAT32. The spec is fatgen102.pdf:, I'll
>mail it to you.
>
>Hardware White Paper Hardware White Paper
>FAT: General Overview of On-Disk Format
>Version 1.02, May 5, 1999
>Microsoft Corporation
>
>Vijay Kumar wrote:
>
>>Alan,
>>
>>This is regarding a change I had to make to the kernel 2.2.17-14 to
>>support really large drives. In our project we deal with huge drives,
>>sometimes drives larger than 128GB. The file system is FAT32 and only one
>>partition is create on any drive. During our testing we realized that
>>linux fat implementation doesn't support partitions larger than
>>128GB(actually 64GB because of a bug in fat implementation).
>>
>>This limitation is imposed by the data structures used by the linux fat
>>implementation to read/write directory entries. A 'long' data type is
>>used to access directory entries on the disk, of which only 28 bits are
>>used to identify the sector that contains the directory entry and the
>>rest 4 bits are used to specify offset of the directory entry inside the
>>sector. Using 28 bits to identify a sector means we cannot access sectors
>>beyond 128GB (2^28*512), thus limiting us from creating partitions larger
>>than 128GB on large disk drives.
>>
>>I have made changes to fat, vfat and msdos file system implementations in
>>the kernel to use larger data types, thus allowing us to create larger
>>partitions. As per the GPL I would like to make the patch available to
>>everyone and also in case somebody has run into the same problem(who
>>cares about fat in the linux world). The patch has been fairly well
>>tested only on our systems(p3, 700MHz with FC). I truly appreciate if you
>>& anybody in the kernel mailing list have any feedback about the changes.
>>
>>The patch is applicable to only 2.2.17-14 kernel and may require changes
>>to use with other kernel versions. As far as I know none of the kernel
>>versions provide any fix for this problem.
>>
>>Thanks,
>>- Vijay
>
>
>
You may thinking that this will change the on-disk format of fat
partitions. That is not true. This change only fixes the linux fat
implementation to allow working with larger partitions. Those 28bits I
mentioned here have nothing to do with FAT table entries that you are
talking about.
At 06:48 PM 1/3/2002 -0800, CJ wrote:
>FAT32 stores 28 bit cluster numbers. You need to increase cluster size
>instead of the 28 bits to stay FAT32. The spec is fatgen102.pdf:, I'll
>mail it to you.
>
>Hardware White Paper Hardware White Paper
>FAT: General Overview of On-Disk Format
>Version 1.02, May 5, 1999
>Microsoft Corporation
>
>Vijay Kumar wrote:
>
>>Alan,
>>
>>This is regarding a change I had to make to the kernel 2.2.17-14 to
>>support really large drives. In our project we deal with huge drives,
>>sometimes drives larger than 128GB. The file system is FAT32 and only one
>>partition is create on any drive. During our testing we realized that
>>linux fat implementation doesn't support partitions larger than
>>128GB(actually 64GB because of a bug in fat implementation).
>>
>>This limitation is imposed by the data structures used by the linux fat
>>implementation to read/write directory entries. A 'long' data type is
>>used to access directory entries on the disk, of which only 28 bits are
>>used to identify the sector that contains the directory entry and the
>>rest 4 bits are used to specify offset of the directory entry inside the
>>sector. Using 28 bits to identify a sector means we cannot access sectors
>>beyond 128GB (2^28*512), thus limiting us from creating partitions larger
>>than 128GB on large disk drives.
>>
>>I have made changes to fat, vfat and msdos file system implementations in
>>the kernel to use larger data types, thus allowing us to create larger
>>partitions. As per the GPL I would like to make the patch available to
>>everyone and also in case somebody has run into the same problem(who
>>cares about fat in the linux world). The patch has been fairly well
>>tested only on our systems(p3, 700MHz with FC). I truly appreciate if you
>>& anybody in the kernel mailing list have any feedback about the changes.
>>
>>The patch is applicable to only 2.2.17-14 kernel and may require changes
>>to use with other kernel versions. As far as I know none of the kernel
>>versions provide any fix for this problem.
>>
>>Thanks,
>>- Vijay
>
>
>
Followup to: <[email protected]>
By author: Vijay Kumar <[email protected]>
In newsgroup: linux.dev.kernel
>
> Using 28bit cluster numbers and 65536 cluster size I could go upto 16TB
> which is lot more than what I wanted. So right now I have no problem with
> the on-disk format of a fat partition. Nevertheless with hard drives prices
> going down dramatically it may not be too long before we hit the limit.
>
> Regards,
> - Vijay
>
That's Microsoft's problem -- that's a fundamental limit of the format
they defined. The fact that they defined it in the first place is
part of the problem (the only way you can make a FAT filesystem work
*well* is by loading the entire FAT into memory ahead of time, and
"FAT32" breaks that), instead of creating a more sensible
replacement.
(FWIW, the reason they used to justify FAT32 was "it would be too much
work to make DOS handle NTFS", as if those were the only options...)
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>