Hi,
Following the recent posting of patches by Lasse Collin to add XZ (LZMA2)
support to the kernel (http://thread.gmane.org/gmane.linux.kernel/1071297),
I have added support for this to Squashfs.
Advantages of Squashfs XZ over the Squashfs LZMA implementation:
- Significantly better decompressor API supporting multi-call decoding,
which requires less buffer overhead.
- Greater data robustness due to XZ's CRC32 check.
- BCJ filters which can produce smaller Squashfs images.
The following two patches add Squashfs kernel support. A git tree with
these patches including Lasse Collin's patches is available here:
http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-xz.git;a=summary
XZ support has (obviously) also been added to the squashfs tools (Mksquashfs
& Unsquashfs). These changes are available from the Squashfs CVS repository
(http://sourceforge.net/projects/squashfs/develop).
To build the Squashfs tools, edit the Makefile to enable XZ support
(by default it is disabled).
XZ compression can be specified by using the -comp option, e.g.
% mksquashfs xxx img.sqsh -comp xz
XZ BCJ filters (which can improve the compression of executable code
on certain architectures) are supported by using the -Xbcj option, e.g.
% mksquashfs xxx img.sqsh -comp xz -Xbcj x86
will compress blocks using XZ with no filter, and then XZ with the x86
filter in turn, and choose the best compression.
Multiple filters can be specified which is useful in cases where the source
file system has executable code from a mixture of different architectures,
and again each filter will be tried for each block and the best compression
used, e.g.
% mksquashfs xxx img.sqsh -comp xz -Xbcj x86,arm
will try both the x86 and arm BCJ filters.
Phillip
On Thu, Dec 09, 2010 at 06:05:55 +0000,
Phillip Lougher <[email protected]> wrote:
>
> XZ support has (obviously) also been added to the squashfs tools (Mksquashfs
> & Unsquashfs). These changes are available from the Squashfs CVS repository
> (http://sourceforge.net/projects/squashfs/develop).
Do you have an estimate for when you will make a release with these changes
and what will likely be the version number? I want to see if it makes sense
to start tracking the development version for F15 or F16 and what to use
for the version.
Bruno Wolff III wrote:
> On Thu, Dec 09, 2010 at 06:05:55 +0000,
> Phillip Lougher <[email protected]> wrote:
>> XZ support has (obviously) also been added to the squashfs tools (Mksquashfs
>> & Unsquashfs). These changes are available from the Squashfs CVS repository
>> (http://sourceforge.net/projects/squashfs/develop).
>
> Do you have an estimate for when you will make a release with these changes
> and what will likely be the version number? I want to see if it makes sense
> to start tracking the development version for F15 or F16 and what to use
> for the version.
>
Hmm, I don't know when F15 or F16 is destined to appear! (I guess roughly
3 to 6 months minimum time but correct me if I'm wrong).
Lasse collin's comment re: dictionary sizes threw a spanner in the works
so to speak as I realised my current patches don't allow users to specify
the dictionary size, which appears rather useful judging by Lasse's
comments that a dictionary size of 50% of block size only impacts
compression by 1%. This makes it rather obvious that allowing users to
specify dictionary size in addition to block size is extremely desirable.
The rather unfortunate issue is that the current Squashfs file format
doesn't allow compressor specific options to be stored, for example it
allows compression type (i.e. gzip, lzma, lzo etc) to be stored, but there
is nowhere to store a dictionary size.
So, this has led to a week of thinking about how to add compressor
specific options to the Squashfs file format which is 100% compatible with
pre-existing 4.0 file systems and file system code. I have settled on an
extension which is 100% compatible and which fits in well with the file
format (i.e. it is not a hack).
What does this mean in relation to your questions? Well basically it means
in a week's time or before I'll be in a position to respin the kernel
patches to include compressor specific options support (i.e. dictionary size),
and there will be the corresponding CVS changes for the Squashfs tools.
As far as I'm concerned this makes a good point to make a new Squashfs tools
release (very few people trust the CVS versions in my experience and always
go for the last released version). I'll probably wait for a couple of weeks
just in case CVS downloaders find any bugs (in practice I hardly ever get any
bug reports from the CVS version). This means a Squashfs tools 4.2
release mid January. How does this fit in with the release schedules for
F15 and F16?
Phillip
On Mon, Dec 20, 2010 at 06:17:30 +0000,
Phillip Lougher <[email protected]> wrote:
>
> As far as I'm concerned this makes a good point to make a new Squashfs tools
> release (very few people trust the CVS versions in my experience and always
> go for the last released version). I'll probably wait for a couple of weeks
> just in case CVS downloaders find any bugs (in practice I hardly ever get any
> bug reports from the CVS version). This means a Squashfs tools 4.2
> release mid January. How does this fit in with the release schedules for
> F15 and F16?
For the kernel stuff, I need to wait until it lands. This looks good for
making 2.6.38 which would make F15. If it doesn't then I'll continue to
wait. (Fedora is fairly strict on using upstream for kernel stuff except
where they have dedicated people as for video drivers.) (I may build
some personal kernels with the patches so I can test live images more
completely, but I can get by with testing squashfs-tools independently
for now.)
For squashfs-tools, a mid January release sounds plenty fine. It could slip
a lot and not cause a problem. I have some time off from work coming up next
week and will get a 4.2 prerelease (fromCVS) into rawhide next week so we can
get some testing there.
Thanks!