Hi,
I'm pleased to announce another release of Squashfs. This is the 22nd
release in just over five years. Squashfs 3.3 has lots of nice improvements,
both to the filesystem itself (bigger blocks and sparse files), but
also to the Squashfs-tools Mksquashfs and Unsquashfs.
The next stage after this release is to fix the one remaining blocking issue
(filesystem endianness), and then try to get Squashfs mainlined into the
Linux kernel again.
The list of changes from the change-log are as follows:
1. Filesystem improvements:
1.1. Maximum block size has been increased to 1Mbyte, and the
default block size has been increased to 128 Kbytes.
This improves compression.
1.2. Sparse files are now supported. Sparse files are files
which have large areas of unallocated data commonly called
holes. These files are now detected by Squashfs and stored
more efficiently. This improves compression and read
performance for sparse files.
2. Mksquashfs improvements:
2.1. Exclude files have been extended to use wildcard pattern
matching and regular expressions. Support has also been
added for non-anchored excludes, which means it is
now possible to specify excludes which match anywhere
in the filesystem (i.e. leaf files), rather than always
having to specify exclude files starting from the root
directory (anchored excludes).
2.2. Recovery files are now created when appending to existing
Squashfs filesystems. This allows the original filesystem
to be recovered if Mksquashfs aborts unexpectedly
(i.e. power failure).
3. Unsquashfs improvements:
3.1. Multiple extract files can now be specified on the
command line, and the files/directories to be extracted can
now also be given in a file.
3.2. Extract files have been extended to use wildcard pattern
matching and regular expressions.
3.3. Filename printing has been enhanced and Unquashfs can
now display filenames with file attributes
('ls -l' style output).
3.4. A -stat option has been added which displays the filesystem
superblock information.
3.5. Unsquashfs now supports 1.x filesystems.
4. Miscellaneous improvements/bug fixes:
4.1. Squashfs kernel code improved to use SetPageError in
squashfs_readpage() if I/O error occurs.
4.2. Fixed Squashfs kernel code bug preventing file
seeking beyond 2GB.
4.3. Mksquashfs now detects file size changes between
first phase directory scan and second phase filesystem create.
Regards
Phillip Lougher
On Mon, Nov 05, 2007 at 11:13:14AM +0000, Phillip Lougher wrote:
> I'm pleased to announce another release of Squashfs. This is the 22nd
> release in just over five years. Squashfs 3.3 has lots of nice
> improvements,
> both to the filesystem itself (bigger blocks and sparse files), but
> also to the Squashfs-tools Mksquashfs and Unsquashfs.
>
> The next stage after this release is to fix the one remaining blocking issue
> (filesystem endianness), and then try to get Squashfs mainlined into the
> Linux kernel again.
>
that would be very cool!
with my hat as debian kernel maintainer i'd be very relieved to see it
mainlined. i don't know of any major distro that doesn't ship it.
thanks
--
maks
Phillip Lougher wrote:
> Hi,
>
> I'm pleased to announce another release of Squashfs. This is the 22nd
> release in just over five years.
Thanks Phillip.
A tiny bug[fix] I always forgot to send... In fs/squashfs/inode.c,
constants TASK_UNINTERRUPTIBLE and TASK_INTERRUPTIBLE are used, but
they aren't sometimes defined (declared in linux/sched.h):
CC [M] fs/squashfs/inode.o
fs/squashfs/inode.c: In function 'squashfs_get_cached_block':
fs/squashfs/inode.c:367: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c:367: error: (Each undeclared identifier is reported only once
fs/squashfs/inode.c:367: error: for each function it appears in.)
fs/squashfs/inode.c:367: warning: implicit declaration of function 'schedule'
fs/squashfs/inode.c:404: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c: In function 'release_cached_fragment':
fs/squashfs/inode.c:499: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c:499: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c: In function 'get_cached_fragment':
fs/squashfs/inode.c:522: error: 'TASK_UNINTERRUPTIBLE' undeclared (first use in this function)
fs/squashfs/inode.c:559: error: 'TASK_INTERRUPTIBLE' undeclared (first use in this function)
I'm not exactly sure which config option is "at blame"
(this is an i486-based UP generic-hardware config), but
I'm not interested to know, either. The following
trivial patch fixes it once for all.
--- linux-2.6.22.orig/fs/squashfs/inode.c 2007-07-12 14:57:22.000000000 +0400
+++ linux-2.6.22/fs/squashfs/inode.c 2007-07-12 14:57:53.000000000 +0400
@@ -31,6 +31,7 @@
#include <linux/vfs.h>
#include <linux/vmalloc.h>
+#include <linux/sched.h>
#include <linux/smp_lock.h>
#include "squashfs.h"
It was needed for v3.2 too.
Thanks.
/mjt
maximilian attems wrote:
> On Mon, Nov 05, 2007 at 11:13:14AM +0000, Phillip Lougher wrote:
>
>> The next stage after this release is to fix the one remaining blocking issue
>> (filesystem endianness), and then try to get Squashfs mainlined into the
>> Linux kernel again.
>>
>
> that would be very cool!
Yes, it would be cool :) Five years is a long time to maintain
something out of tree, especially recently when there's been
so many minor changes to the VFS interface between kernel releases.
> with my hat as debian kernel maintainer i'd be very relieved to see it
> mainlined. i don't know of any major distro that doesn't ship it.
>
I don't know of any major distro that doesn't ship Squashfs either
(except arguably Slackware). Putting my other hat on (one of the
Ubuntu kernel maintainers) I don't think Squashfs has caused
distros that many problems because it is an easy patch to apply
(it doesn't touch that many kernel files), but it is always good
to minimise the differences from the stock kernel.org kernel.
Phillip
On Wed, Nov 07, 2007 at 04:32:00PM +0000, Phillip Lougher wrote:
> maximilian attems wrote:
> > On Mon, Nov 05, 2007 at 11:13:14AM +0000, Phillip Lougher wrote:
> >
> >> The next stage after this release is to fix the one remaining blocking issue
> >> (filesystem endianness), and then try to get Squashfs mainlined into the
> >> Linux kernel again.
> >>
> >
> > that would be very cool!
>
> Yes, it would be cool :) Five years is a long time to maintain
> something out of tree, especially recently when there's been
> so many minor changes to the VFS interface between kernel releases.
>
> > with my hat as debian kernel maintainer i'd be very relieved to see it
> > mainlined. i don't know of any major distro that doesn't ship it.
> >
>
> I don't know of any major distro that doesn't ship Squashfs either
> (except arguably Slackware). Putting my other hat on (one of the
> Ubuntu kernel maintainers) I don't think Squashfs has caused
> distros that many problems because it is an easy patch to apply
> (it doesn't touch that many kernel files), but it is always good
> to minimise the differences from the stock kernel.org kernel.
The biggest problem we've seen with it (asides from having to rediff
it every time we rebase when there isn't a newer upstream) is complaints
along the lines of "my Fedora 7 kernel can't unpack squashfs images
from Fedora 5"
(s/Fedora 5/other random older distros/ )
If the format is now stable however, it would be great to get it upstream.
Dave
--
http://www.codemonkey.org.uk
Dave Jones wrote:
>
> The biggest problem we've seen with it (asides from having to rediff
> it every time we rebase when there isn't a newer upstream)
Yes, this is mainly my fault. There was a gap of 10 months between the
3.2 release in January this year, and the latest in November. With the
rate of new kernel releases this wasn't really acceptable because the
January release was stuck with a patch for kernels no newer than 2.6.20.
I received numerous complaints about it.
Some of you may be aware that I started work at Canonical, and this left
almost no spare-time to work on Squashfs for 9 months.
>is complaints
> along the lines of "my Fedora 7 kernel can't unpack squashfs images
> from Fedora 5"
> (s/Fedora 5/other random older distros/ )
>
Squashfs has backwards compatibility with older versions, and it should
mount all older versions back to 2.0 (released May 2004). Unfortunately
RedHat grabbed a CVS version of Squashfs just before the 3.0 release.
This was development code, and release testing showed it
had a bug where it couldn't mount older versions. It was fixed for
release.
> If the format is now stable however, it would be great to get it upstream.
>
The move from the 2.0 format to the later 3.0 format was mainly forced
by the demands of the Linux kernel mailing list when I first submitted
it early 2005. There was no other way to incorporate demands for larger
than 4GB filesystems, and provide support for "." and ".." in readdir
without modifying the filesystem format.
Unfortunately the move to fixed little endian filesystem will involve
another filesystem layout change. The current filesystem layout still
uses packed bitfield structures, and it is impossible to swap these
using the standard kernel swap macros. Removal of my routines that can
properly swap packed bitfield structures is another change demanded by
the Linux kernel mailing list.
Once the little endian work has been done, and hopefully once it is in
the kernel, I don't anticipate any further layout changes.
Phillip
On Wed, Nov 21, 2007 at 02:02:43PM +0000, Phillip Lougher wrote:
> Unfortunately the move to fixed little endian filesystem will involve
> another filesystem layout change. The current filesystem layout still
> uses packed bitfield structures, and it is impossible to swap these
> using the standard kernel swap macros. Removal of my routines that can
> properly swap packed bitfield structures is another change demanded by
> the Linux kernel mailing list.
The normal way to do it is to use shift and mask after doing the endian
conversion. But the problem with bitfields is that they can have different
kinds of layouts depending on the compiler or abi which is another reason
to avoid them in ondisk/wire formats.
Christoph Hellwig wrote:
> On Wed, Nov 21, 2007 at 02:02:43PM +0000, Phillip Lougher wrote:
>> Unfortunately the move to fixed little endian filesystem will involve
>> another filesystem layout change. The current filesystem layout still
>> uses packed bitfield structures, and it is impossible to swap these
>> using the standard kernel swap macros. Removal of my routines that can
>> properly swap packed bitfield structures is another change demanded by
>> the Linux kernel mailing list.
>
> The normal way to do it is to use shift and mask after doing the endian
> conversion. But the problem with bitfields is that they can have different
> kinds of layouts depending on the compiler or abi which is another reason
> to avoid them in ondisk/wire formats.
>
Yes, the bitfields are packed differently on little and big endian
architectures which mean they appear in different places in the
structure. I want to move away from that mess when I move to little
endian only.
Phillip