From: Geert Uytterhoeven <[email protected]>
Add a (de)compression module for the "zlib" format using the crypto API.
While both the "zlib" and "deflate" crypto modules are implemented on top of
the zlib library, they differ in the following aspects:
- The "deflate" crypto module (used by IPSec and UBIFS) does not support
partial decompression, i.e. all compressed data has to be passed at once.
The "zlib" crypto module does support partial decompression;
zlib_decompress() will return -EAGAIN if not all compressed data has been
passed.
- The deflate crypto module uses the raw deflate data format (zlib is
initialized with a windowBits parameter of -DEFLATE_DEF_WINBITS = -11),
while e.g. squashfs and axfs use the zlib data format, with the default
windowBits parameter DEF_WBITS = 15.
Both parameters are incompatible with each other due to the different data
formats, as indicated by the sign of the windowbits parameter.
The absolute value of this parameter is the base two logarithm of the
maximum window size, and larger values are backwards compatible with
smaller values (as far as decompression is concerned).
TODO:
- As the crypto wrapper around the zlib library supports both compression and
decompression (and always allocates temporary space for both during
initialization; lazy allocation was removed in 2004 because IPComp calls
the crypto routines in non-process context), more memory is needed:
* decompression (inflate) needs only ca. 41 KiB
* compression (deflate) needs ca. 262 KiB!
It would be nice to allow having support for decompression only.
Signed-off-by: Geert Uytterhoeven <[email protected]>
---
crypto/Kconfig | 8 ++
crypto/Makefile | 1
crypto/zlib.c | 216 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 225 insertions(+)
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -666,6 +666,14 @@ config CRYPTO_LZO
help
This is the LZO algorithm.
+config CRYPTO_ZLIB
+ tristate "Zlib compression algorithm"
+ select CRYPTO_ALGAPI
+ select ZLIB_INFLATE
+ select ZLIB_DEFLATE
+ help
+ This is the Zlib algorithm.
+
source "drivers/crypto/Kconfig"
endif # if CRYPTO
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -69,6 +69,7 @@ obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += mich
obj-$(CONFIG_CRYPTO_CRC32C) += crc32c.o
obj-$(CONFIG_CRYPTO_AUTHENC) += authenc.o
obj-$(CONFIG_CRYPTO_LZO) += lzo.o
+obj-$(CONFIG_CRYPTO_ZLIB) += zlib.o
obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o
--- /dev/null
+++ b/crypto/zlib.c
@@ -0,0 +1,216 @@
+/*
+ * Cryptographic API.
+ *
+ * "zlib" crypto module, based on the "deflate" crypto module
+ *
+ * Copyright (c) 2003 James Morris <[email protected]>
+ * Copyright 2008 Sony Corp.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * FIXME: deflate transforms will require up to a total of about 436k of kernel
+ * memory on i386 (390k for compression, the rest for decompression), as the
+ * current zlib kernel code uses a worst case pre-allocation system by default.
+ * This needs to be fixed so that the amount of memory required is properly
+ * related to the winbits and memlevel parameters.
+ */
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/crypto.h>
+#include <linux/zlib.h>
+#include <linux/vmalloc.h>
+#include <linux/interrupt.h>
+#include <linux/mm.h>
+#include <linux/net.h>
+#include <linux/slab.h>
+
+#define ZLIB_DEF_LEVEL Z_DEFAULT_COMPRESSION
+
+struct zlib_ctx {
+ struct z_stream_s comp_stream;
+ struct z_stream_s decomp_stream;
+ bool decomp_needs_reset;
+};
+
+static int zlib_comp_init(struct zlib_ctx *ctx)
+{
+ int ret = 0;
+ struct z_stream_s *stream = &ctx->comp_stream;
+
+ stream->workspace = vmalloc(zlib_deflate_workspacesize());
+ if (!stream->workspace ) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ memset(stream->workspace, 0, zlib_deflate_workspacesize());
+ ret = zlib_deflateInit(stream, ZLIB_DEF_LEVEL);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+out:
+ return ret;
+out_free:
+ vfree(stream->workspace);
+ goto out;
+}
+
+static int zlib_decomp_init(struct zlib_ctx *ctx)
+{
+ int ret = 0;
+ struct z_stream_s *stream = &ctx->decomp_stream;
+
+ stream->workspace = kzalloc(zlib_inflate_workspacesize(), GFP_KERNEL);
+ if (!stream->workspace ) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ret = zlib_inflateInit(stream);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+ ctx->decomp_needs_reset = true;
+out:
+ return ret;
+out_free:
+ kfree(stream->workspace);
+ goto out;
+}
+
+static void zlib_comp_exit(struct zlib_ctx *ctx)
+{
+ zlib_deflateEnd(&ctx->comp_stream);
+ vfree(ctx->comp_stream.workspace);
+}
+
+static void zlib_decomp_exit(struct zlib_ctx *ctx)
+{
+ zlib_inflateEnd(&ctx->decomp_stream);
+ kfree(ctx->decomp_stream.workspace);
+}
+
+static int zlib_init(struct crypto_tfm *tfm)
+{
+ struct zlib_ctx *ctx = crypto_tfm_ctx(tfm);
+ int ret;
+
+ ret = zlib_comp_init(ctx);
+ if (ret)
+ goto out;
+ ret = zlib_decomp_init(ctx);
+ if (ret)
+ zlib_comp_exit(ctx);
+out:
+ return ret;
+}
+
+static void zlib_exit(struct crypto_tfm *tfm)
+{
+ struct zlib_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ zlib_comp_exit(ctx);
+ zlib_decomp_exit(ctx);
+}
+
+static int zlib_compress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ int ret = 0;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->comp_stream;
+
+ ret = zlib_deflateReset(stream);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ stream->next_in = (u8 *)src;
+ stream->avail_in = slen;
+ stream->next_out = (u8 *)dst;
+ stream->avail_out = *dlen;
+
+ ret = zlib_deflate(stream, Z_FINISH);
+ if (ret != Z_STREAM_END) {
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = 0;
+ *dlen = stream->total_out;
+out:
+ return ret;
+}
+
+static int zlib_decompress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ int ret = 0;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->decomp_stream;
+ unsigned long old_total_out;
+
+ if (dctx->decomp_needs_reset) {
+ ret = zlib_inflateReset(stream);
+ if (ret != Z_OK)
+ return -EINVAL;
+ dctx->decomp_needs_reset = false;
+ }
+
+ stream->next_in = (u8 *)src;
+ stream->avail_in = slen;
+ stream->next_out = (u8 *)dst;
+ stream->avail_out = *dlen;
+ old_total_out = stream->total_out;
+
+ ret = zlib_inflate(stream, Z_SYNC_FLUSH);
+ switch (ret) {
+ case Z_STREAM_END:
+ dctx->decomp_needs_reset = true;
+ ret = 0;
+ break;
+
+ case Z_OK:
+ ret = -EAGAIN;
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ *dlen = stream->total_out - old_total_out;
+ return ret;
+}
+
+static struct crypto_alg alg = {
+ .cra_name = "zlib",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zlib_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_list = LIST_HEAD_INIT(alg.cra_list),
+ .cra_init = zlib_init,
+ .cra_exit = zlib_exit,
+ .cra_u = { .compress = {
+ .coa_compress = zlib_compress,
+ .coa_decompress = zlib_decompress } }
+};
+
+static int __init zlib_mod_init(void)
+{
+ return crypto_register_alg(&alg);
+}
+
+static void __exit zlib_mod_fini(void)
+{
+ crypto_unregister_alg(&alg);
+}
+
+module_init(zlib_mod_init);
+module_exit(zlib_mod_fini);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Zlib Compression Algorithm");
+
--
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
On Fri, Aug 29, 2008 at 01:41:59PM +0000, Geert Uytterhoeven wrote:
> From: Geert Uytterhoeven <[email protected]>
>
> Add a (de)compression module for the "zlib" format using the crypto API.
I think we can safely conclude that our current compression
interface sucks for what you're trying to achieve :)
> While both the "zlib" and "deflate" crypto modules are implemented on top of
> the zlib library, they differ in the following aspects:
> - The "deflate" crypto module (used by IPSec and UBIFS) does not support
> partial decompression, i.e. all compressed data has to be passed at once.
> The "zlib" crypto module does support partial decompression;
> zlib_decompress() will return -EAGAIN if not all compressed data has been
> passed.
> - The deflate crypto module uses the raw deflate data format (zlib is
> initialized with a windowBits parameter of -DEFLATE_DEF_WINBITS = -11),
> while e.g. squashfs and axfs use the zlib data format, with the default
> windowBits parameter DEF_WBITS = 15.
> Both parameters are incompatible with each other due to the different data
> formats, as indicated by the sign of the windowbits parameter.
> The absolute value of this parameter is the base two logarithm of the
> maximum window size, and larger values are backwards compatible with
> smaller values (as far as decompression is concerned).
Therefore I suggest that we change the interface rather than
mutilate the implementations.
So here are a few things we should add:
1) Separate alloc functions for comp/decomp to avoid the memory
wastage you've identified.
2) Provide parameters to these alloc functions through an opaque
pointer. The format of the paramters will be determined by the
name of the algorithm, i.e., if you want to change the parameters
then you should change the name as well (e.g., deflate => deflate2).
This removes the need to dupliate the implementation just because
you want 15 instead of -11.
3) Provide init/update/final (one set each for comp and decomp)
functions similar to crypto_hash, in addition to the current
comp/decomp functions.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Sat, 2008-08-30 at 16:23 +1000, Herbert Xu wrote:
> On Fri, Aug 29, 2008 at 01:41:59PM +0000, Geert Uytterhoeven wrote:
> > From: Geert Uytterhoeven <[email protected]>
> >
> > Add a (de)compression module for the "zlib" format using the crypto API.
>
> I think we can safely conclude that our current compression
> interface sucks for what you're trying to achieve :)
The main thing that's missing for JFFS2 is
"Compress as much of this as you can into X bytes"
--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation
On Sat, 30 Aug 2008, Herbert Xu wrote:
> On Fri, Aug 29, 2008 at 01:41:59PM +0000, Geert Uytterhoeven wrote:
> > From: Geert Uytterhoeven <[email protected]>
> >
> > Add a (de)compression module for the "zlib" format using the crypto API.
>
> I think we can safely conclude that our current compression
> interface sucks for what you're trying to achieve :)
>
> > While both the "zlib" and "deflate" crypto modules are implemented on top of
> > the zlib library, they differ in the following aspects:
> > - The "deflate" crypto module (used by IPSec and UBIFS) does not support
> > partial decompression, i.e. all compressed data has to be passed at once.
> > The "zlib" crypto module does support partial decompression;
> > zlib_decompress() will return -EAGAIN if not all compressed data has been
> > passed.
> > - The deflate crypto module uses the raw deflate data format (zlib is
> > initialized with a windowBits parameter of -DEFLATE_DEF_WINBITS = -11),
> > while e.g. squashfs and axfs use the zlib data format, with the default
> > windowBits parameter DEF_WBITS = 15.
> > Both parameters are incompatible with each other due to the different data
> > formats, as indicated by the sign of the windowbits parameter.
> > The absolute value of this parameter is the base two logarithm of the
> > maximum window size, and larger values are backwards compatible with
> > smaller values (as far as decompression is concerned).
>
> Therefore I suggest that we change the interface rather than
> mutilate the implementations.
>
> So here are a few things we should add:
>
> 1) Separate alloc functions for comp/decomp to avoid the memory
> wastage you've identified.
>
> 2) Provide parameters to these alloc functions through an opaque
> pointer. The format of the paramters will be determined by the
> name of the algorithm, i.e., if you want to change the parameters
> then you should change the name as well (e.g., deflate => deflate2).
>
> This removes the need to dupliate the implementation just because
> you want 15 instead of -11.
>
> 3) Provide init/update/final (one set each for comp and decomp)
> functions similar to crypto_hash, in addition to the current
> comp/decomp functions.
Thanks for your suggestions!
I started tackling #3 (init/update/final), as that looks like the most
complicated one.
As a proof-of-concept, I'm sending the following 4 patches, which gives a
working version of SquashFS using the CRYPTO API for decompression:
- [1/4] crypto: Add partial decompression support to the CRYPTO API.
- [2/4] crypto: Add a "zlib" crypto module
- [3/4] crypto: Add a test for the "zlib" crypto module
- [4/4] squashfs: Make SquashFS 4 use the new "zlib" crypto module
As pointed out in patch [1/4], some underlying (de)compression implementations
do not support partial (de)compression. This is the case for e.g. LZO, which
is already supported by a crypto module. While one-shot (de)compression can
easily be implemented on top of a partial (de)compression API, the inverse is
not true.
Hence there should be a way to ask for a "one-shot" vs. "partial" decompression
module at crypto_alloc_comp() time.
Do you have any suggestions how to handle this?
Perhaps by using "oneshot(%s)" and "partial(%s)", cfr. what is done in other
parts of the CRYPTO API where orthogonal features are combined?
For #1 (Separate alloc functions for comp/decomp), I'm wondering if a similar
scheme be used? "comp(%s)", "decomp(%s)", "both(%s)" (any better name?)?
Or crypto_alloc_comp_only(), crypto_alloc_decomp_only(), and
crypto_alloc_comp()?
Or would it be better to use the CRYPTO_ALG_TYPE_* flags in
crypto_alloc_comp()? I'm still a bit puzzled how exactly everything fits
together in the different layers of the CRYPTO API.
Thanks for your comments!
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
The current CRYPTO API supports one-shot (de)compression only, i.e. the whole
data buffer to be (de)compressed must be passed at once, and the whole
(de)compressed data buffer will be received at once.
In several use-cases (e.g. compressed file systems that store files in big
compressed blocks), this workflow is not suitable. Hence the CRYPTO API cannot
be used.
To solve this, add the following operations that support partial
(de)compression:
- crypto_compress_{init,update,final}() for compression,
- crypto_decompress_{init,update,final}() for decompression.
All these methods take a struct comp_request, which was mimicked after the
z_stream object in zlib, and contains buffer pointer and length pairs for input
and output.
Issues:
- Some underlying (de)compression implementations do not support partial
(de)compression. This is the case for e.g. LZO, which is already supported
by a crypto module. While one-shot (de)compression can easily be
implemented on top of a partial (de)compression API, the inverse is not
true.
Hence there should be a way to ask for a "one-shot" vs. "partial"
decompression module at crypto_alloc_comp() time.
Signed-off-by: Geert Uytterhoeven <[email protected]>
---
crypto/compress.c | 45 +++++++++++++++++++++++++++
include/linux/crypto.h | 79 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 123 insertions(+), 1 deletion(-)
diff --git a/crypto/compress.c b/crypto/compress.c
index 1ee3570..9dbe6f6 100644
--- a/crypto/compress.c
+++ b/crypto/compress.c
@@ -33,13 +33,56 @@ static int crypto_decompress(struct crypto_tfm *tfm,
dlen);
}
+static int crypto_compress_init(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ return tfm->__crt_alg->cra_compress.coa_compress_init(tfm, req);
+}
+
+static int crypto_compress_update(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ return tfm->__crt_alg->cra_compress.coa_compress_update(tfm, req);
+}
+
+static int crypto_compress_final(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ return tfm->__crt_alg->cra_compress.coa_compress_final(tfm, req);
+}
+
+static int crypto_decompress_init(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ return tfm->__crt_alg->cra_compress.coa_decompress_init(tfm, req);
+}
+
+static int crypto_decompress_update(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ return tfm->__crt_alg->cra_compress.coa_decompress_update(tfm, req);
+}
+
+static int crypto_decompress_final(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ return tfm->__crt_alg->cra_compress.coa_decompress_final(tfm, req);
+}
+
int crypto_init_compress_ops(struct crypto_tfm *tfm)
{
struct compress_tfm *ops = &tfm->crt_compress;
ops->cot_compress = crypto_compress;
ops->cot_decompress = crypto_decompress;
-
+
+ ops->cot_compress_init = crypto_compress_init;
+ ops->cot_compress_update = crypto_compress_update;
+ ops->cot_compress_final = crypto_compress_final;
+ ops->cot_decompress_init = crypto_decompress_init;
+ ops->cot_decompress_update = crypto_decompress_update;
+ ops->cot_decompress_final = crypto_decompress_final;
+
return 0;
}
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 3d2317e..cef556e 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -180,6 +180,13 @@ struct aead_request {
void *__ctx[] CRYPTO_MINALIGN_ATTR;
};
+struct comp_request {
+ const void *next_in; /* next input byte */
+ void *next_out; /* next output byte */
+ unsigned int avail_in; /* bytes available at next_in */
+ unsigned int avail_out; /* bytes available at next_out */
+};
+
struct blkcipher_desc {
struct crypto_blkcipher *tfm;
void *info;
@@ -294,10 +301,24 @@ struct hash_alg {
};
struct compress_alg {
+ /* one shot */
int (*coa_compress)(struct crypto_tfm *tfm, const u8 *src,
unsigned int slen, u8 *dst, unsigned int *dlen);
int (*coa_decompress)(struct crypto_tfm *tfm, const u8 *src,
unsigned int slen, u8 *dst, unsigned int *dlen);
+ /* partial */
+ int (*coa_compress_init)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*coa_compress_update)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*coa_compress_final)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*coa_decompress_init)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*coa_decompress_update)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*coa_decompress_final)(struct crypto_tfm *tfm,
+ struct comp_request *req);
};
struct rng_alg {
@@ -443,12 +464,26 @@ struct ahash_tfm {
};
struct compress_tfm {
+ /* one shot */
int (*cot_compress)(struct crypto_tfm *tfm,
const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen);
int (*cot_decompress)(struct crypto_tfm *tfm,
const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen);
+ /* partial */
+ int (*cot_compress_init)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*cot_compress_update)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*cot_compress_final)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*cot_decompress_init)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*cot_decompress_update)(struct crypto_tfm *tfm,
+ struct comp_request *req);
+ int (*cot_decompress_final)(struct crypto_tfm *tfm,
+ struct comp_request *req);
};
struct rng_tfm {
@@ -1323,6 +1358,7 @@ static inline struct compress_tfm *crypto_comp_crt(struct crypto_comp *tfm)
return &crypto_comp_tfm(tfm)->crt_compress;
}
+/* one shot */
static inline int crypto_comp_compress(struct crypto_comp *tfm,
const u8 *src, unsigned int slen,
u8 *dst, unsigned int *dlen)
@@ -1339,5 +1375,48 @@ static inline int crypto_comp_decompress(struct crypto_comp *tfm,
src, slen, dst, dlen);
}
+/* partial */
+static inline int crypto_comp_compress_init(struct crypto_comp *tfm,
+ struct comp_request *req)
+{
+ return crypto_comp_crt(tfm)->cot_compress_init(crypto_comp_tfm(tfm),
+ req);
+}
+
+static inline int crypto_comp_compress_update(struct crypto_comp *tfm,
+ struct comp_request *req)
+{
+ return crypto_comp_crt(tfm)->cot_compress_update(crypto_comp_tfm(tfm),
+ req);
+}
+
+static inline int crypto_comp_compress_final(struct crypto_comp *tfm,
+ struct comp_request *req)
+{
+ return crypto_comp_crt(tfm)->cot_compress_final(crypto_comp_tfm(tfm),
+ req);
+}
+
+static inline int crypto_comp_decompress_init(struct crypto_comp *tfm,
+ struct comp_request *req)
+{
+ return crypto_comp_crt(tfm)->cot_decompress_init(crypto_comp_tfm(tfm),
+ req);
+}
+
+static inline int crypto_comp_decompress_update(struct crypto_comp *tfm,
+ struct comp_request *req)
+{
+ return crypto_comp_crt(tfm)->cot_decompress_update(crypto_comp_tfm(tfm),
+ req);
+}
+
+static inline int crypto_comp_decompress_final(struct crypto_comp *tfm,
+ struct comp_request *req)
+{
+ return crypto_comp_crt(tfm)->cot_decompress_final(crypto_comp_tfm(tfm),
+ req);
+}
+
#endif /* _LINUX_CRYPTO_H */
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
Add a "zlib" crypto module, which differs from the existing "deflate" crypto
module in the following aspects:
- Support for the new partial (de)compression API has been added,
- The (de)compression parameters have been changed to accomodate SquashFS
("zlib" format with 15 window bits instead of "raw deflate" format with 11
windows bits).
Todo:
- Make the (de)compression parameters configurable, so it can be merged
back into the "deflate" crypto module
Caveats:
- Compression hasn't been tested yet
Signed-off-by: Geert Uytterhoeven <[email protected]>
---
Kconfig | 8 +
Makefile | 1
zlib.c | 389 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 398 insertions(+)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 39dbd8e..51bf6fa 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -695,6 +695,14 @@ config CRYPTO_LZO
help
This is the LZO algorithm.
+config CRYPTO_ZLIB
+ tristate "Zlib compression algorithm"
+ select CRYPTO_ALGAPI
+ select ZLIB_INFLATE
+ select ZLIB_DEFLATE
+ help
+ This is the Zlib algorithm.
+
comment "Random Number Generation"
config CRYPTO_ANSI_CPRNG
diff --git a/crypto/Makefile b/crypto/Makefile
index 5862b80..8d4cdb7 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -76,6 +76,7 @@ obj-$(CONFIG_CRYPTO_LZO) += lzo.o
obj-$(CONFIG_CRYPTO_RNG) += rng.o
obj-$(CONFIG_CRYPTO_RNG) += krng.o
obj-$(CONFIG_CRYPTO_ANSI_CPRNG) += ansi_cprng.o
+obj-$(CONFIG_CRYPTO_ZLIB) += zlib.o
obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o
#
diff --git a/crypto/zlib.c b/crypto/zlib.c
new file mode 100644
index 0000000..a95340d
--- /dev/null
+++ b/crypto/zlib.c
@@ -0,0 +1,389 @@
+#define DEBUG
+/*
+ * Cryptographic API.
+ *
+ * "zlib" crypto module, based on the "deflate" crypto module
+ *
+ * Copyright (c) 2003 James Morris <[email protected]>
+ * Copyright 2008 Sony Corp.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * FIXME: deflate transforms will require up to a total of about 436k of kernel
+ * memory on i386 (390k for compression, the rest for decompression), as the
+ * current zlib kernel code uses a worst case pre-allocation system by default.
+ * This needs to be fixed so that the amount of memory required is properly
+ * related to the winbits and memlevel parameters.
+ */
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/crypto.h>
+#include <linux/zlib.h>
+#include <linux/vmalloc.h>
+#include <linux/interrupt.h>
+#include <linux/mm.h>
+#include <linux/net.h>
+#include <linux/slab.h>
+
+#define ZLIB_DEF_LEVEL Z_DEFAULT_COMPRESSION
+
+struct zlib_ctx {
+ struct z_stream_s comp_stream;
+ struct z_stream_s decomp_stream;
+};
+
+static int zlib_comp_init(struct zlib_ctx *ctx)
+{
+ int ret = 0;
+ struct z_stream_s *stream = &ctx->comp_stream;
+
+ stream->workspace = vmalloc(zlib_deflate_workspacesize());
+ if (!stream->workspace) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ memset(stream->workspace, 0, zlib_deflate_workspacesize());
+ ret = zlib_deflateInit(stream, ZLIB_DEF_LEVEL);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+out:
+ return ret;
+out_free:
+ vfree(stream->workspace);
+ goto out;
+}
+
+static int zlib_decomp_init(struct zlib_ctx *ctx)
+{
+ int ret = 0;
+ struct z_stream_s *stream = &ctx->decomp_stream;
+
+ stream->workspace = kzalloc(zlib_inflate_workspacesize(), GFP_KERNEL);
+ if (!stream->workspace) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ret = zlib_inflateInit(stream);
+ if (ret != Z_OK) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+out:
+ return ret;
+out_free:
+ kfree(stream->workspace);
+ goto out;
+}
+
+static void zlib_comp_exit(struct zlib_ctx *ctx)
+{
+ zlib_deflateEnd(&ctx->comp_stream);
+ vfree(ctx->comp_stream.workspace);
+}
+
+static void zlib_decomp_exit(struct zlib_ctx *ctx)
+{
+ zlib_inflateEnd(&ctx->decomp_stream);
+ kfree(ctx->decomp_stream.workspace);
+}
+
+static int zlib_init(struct crypto_tfm *tfm)
+{
+ struct zlib_ctx *ctx = crypto_tfm_ctx(tfm);
+ int ret;
+
+ ret = zlib_comp_init(ctx);
+ if (ret)
+ goto out;
+ ret = zlib_decomp_init(ctx);
+ if (ret)
+ zlib_comp_exit(ctx);
+out:
+ return ret;
+}
+
+static void zlib_exit(struct crypto_tfm *tfm)
+{
+ struct zlib_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ zlib_comp_exit(ctx);
+ zlib_decomp_exit(ctx);
+}
+
+static int zlib_compress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->comp_stream;
+
+ pr_debug("%s: slen %u, dlen %u\n", __func__, slen, *dlen);
+ ret = zlib_deflateReset(stream);
+ if (ret != Z_OK) {
+ pr_err("%s: zlib_deflateReset failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ stream->next_in = (u8 *)src;
+ stream->avail_in = slen;
+ stream->next_out = (u8 *)dst;
+ stream->avail_out = *dlen;
+
+ ret = zlib_deflate(stream, Z_FINISH);
+ if (ret != Z_STREAM_END) {
+ pr_err("%s: zlib_deflate failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ *dlen = stream->total_out;
+ pr_debug("%s: dlen_out %u\n", __func__, *dlen);
+ return 0;
+}
+
+static int zlib_decompress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->decomp_stream;
+
+ pr_debug("%s: slen %u, dlen %u\n", __func__, slen, *dlen);
+ ret = zlib_inflateReset(stream);
+ if (ret != Z_OK) {
+ pr_err("%s: zlib_inflateReset failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ stream->next_in = (u8 *)src;
+ stream->avail_in = slen;
+ stream->next_out = (u8 *)dst;
+ stream->avail_out = *dlen;
+
+ ret = zlib_inflate(stream, Z_FINISH);
+ if (ret != Z_STREAM_END) {
+ pr_err("%s: zlib_inflate failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ *dlen = stream->total_out;
+ pr_debug("%s: dlen_out %u\n", __func__, *dlen);
+ return 0;
+}
+
+static int zlib_compress_update(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->comp_stream;
+
+ pr_debug("%s: avail_in %u, avail_out %u\n", __func__, req->avail_in,
+ req->avail_out);
+ stream->next_in = req->next_in;
+ stream->avail_in = req->avail_in;
+ stream->next_out = req->next_out;
+ stream->avail_out = req->avail_out;
+
+ ret = zlib_deflate(stream, Z_NO_FLUSH);
+ switch (ret) {
+ case Z_OK:
+ break;
+
+ case Z_BUF_ERROR:
+ pr_err("%s: zlib_deflate could not make progress\n", __func__);
+ return -EAGAIN;
+
+ default:
+ pr_err("%s: zlib_deflate failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ pr_debug("%s: avail_in %u, avail_out %u (consumed %u, produced %u)\n",
+ __func__, stream->avail_in, stream->avail_out,
+ req->avail_in - stream->avail_in,
+ req->avail_out - stream->avail_out);
+ req->next_in = stream->next_in;
+ req->avail_in = stream->avail_in;
+ req->next_out = stream->next_out;
+ req->avail_out = stream->avail_out;
+ return 0;
+}
+
+static int zlib_compress_init(struct crypto_tfm *tfm, struct comp_request *req)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->comp_stream;
+
+ pr_debug("%s: avail_in %u, avail_out %u\n", __func__, req->avail_in,
+ req->avail_out);
+ ret = zlib_deflateReset(stream);
+ if (ret != Z_OK)
+ return -EINVAL;
+
+ return zlib_compress_update(tfm, req);
+}
+
+static int zlib_compress_final(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->comp_stream;
+
+ pr_debug("%s: avail_in %u, avail_out %u\n", __func__, req->avail_in,
+ req->avail_out);
+ stream->next_in = req->next_in;
+ stream->avail_in = req->avail_in;
+ stream->next_out = req->next_out;
+ stream->avail_out = req->avail_out;
+
+ ret = zlib_deflate(stream, Z_FINISH);
+ if (ret != Z_OK) {
+ pr_err("%s: zlib_deflate failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ pr_debug("%s: avail_in %u, avail_out %u (consumed %u, produced %u)\n",
+ __func__, stream->avail_in, stream->avail_out,
+ req->avail_in - stream->avail_in,
+ req->avail_out - stream->avail_out);
+ req->next_in = stream->next_in;
+ req->avail_in = stream->avail_in;
+ req->next_out = stream->next_out;
+ req->avail_out = stream->avail_out;
+ return 0;
+}
+
+static int zlib_decompress_update(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->decomp_stream;
+
+ pr_debug("%s: avail_in %u, avail_out %u\n", __func__, req->avail_in,
+ req->avail_out);
+ stream->next_in = req->next_in;
+ stream->avail_in = req->avail_in;
+ stream->next_out = req->next_out;
+ stream->avail_out = req->avail_out;
+
+ ret = zlib_inflate(stream, Z_SYNC_FLUSH);
+ switch (ret) {
+ case Z_OK:
+ case Z_STREAM_END:
+ break;
+
+ case Z_BUF_ERROR:
+ pr_err("%s: zlib_inflate could not make progress\n", __func__);
+ return -EAGAIN;
+
+ default:
+ pr_err("%s: zlib_inflate failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ pr_debug("%s: avail_in %u, avail_out %u (consumed %u, produced %u)\n",
+ __func__, stream->avail_in, stream->avail_out,
+ req->avail_in - stream->avail_in,
+ req->avail_out - stream->avail_out);
+ req->next_in = stream->next_in;
+ req->avail_in = stream->avail_in;
+ req->next_out = stream->next_out;
+ req->avail_out = stream->avail_out;
+ return 0;
+}
+
+static int zlib_decompress_init(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->decomp_stream;
+
+ pr_debug("%s: avail_in %u, avail_out %u\n", __func__, req->avail_in,
+ req->avail_out);
+ ret = zlib_inflateReset(stream);
+ if (ret != Z_OK)
+ return -EINVAL;
+
+ return zlib_decompress_update(tfm, req);
+}
+
+static int zlib_decompress_final(struct crypto_tfm *tfm,
+ struct comp_request *req)
+{
+ int ret;
+ struct zlib_ctx *dctx = crypto_tfm_ctx(tfm);
+ struct z_stream_s *stream = &dctx->decomp_stream;
+
+ pr_debug("%s: avail_in %u, avail_out %u\n", __func__, req->avail_in,
+ req->avail_out);
+ stream->next_in = req->next_in;
+ stream->avail_in = req->avail_in;
+ stream->next_out = req->next_out;
+ stream->avail_out = req->avail_out;
+
+ ret = zlib_inflate(stream, Z_FINISH);
+ if (ret != Z_STREAM_END) {
+ pr_err("%s: zlib_inflate failed %d\n", __func__, ret);
+ return -EINVAL;
+ }
+
+ pr_debug("%s: avail_in %u, avail_out %u (consumed %u, produced %u)\n",
+ __func__, stream->avail_in, stream->avail_out,
+ req->avail_in - stream->avail_in,
+ req->avail_out - stream->avail_out);
+ req->next_in = stream->next_in;
+ req->avail_in = stream->avail_in;
+ req->next_out = stream->next_out;
+ req->avail_out = stream->avail_out;
+ return 0;
+}
+
+static struct crypto_alg alg = {
+ .cra_name = "zlib",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zlib_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_list = LIST_HEAD_INIT(alg.cra_list),
+ .cra_init = zlib_init,
+ .cra_exit = zlib_exit,
+ .cra_u = {
+ .compress = {
+ /* one shot */
+ .coa_compress = zlib_compress,
+ .coa_decompress = zlib_decompress,
+ /* partial */
+ .coa_compress_init = zlib_compress_init,
+ .coa_compress_update = zlib_compress_update,
+ .coa_compress_final = zlib_compress_final,
+ .coa_decompress_init = zlib_decompress_init,
+ .coa_decompress_update = zlib_decompress_update,
+ .coa_decompress_final = zlib_decompress_final
+ }
+ }
+};
+
+static int __init zlib_mod_init(void)
+{
+ return crypto_register_alg(&alg);
+}
+
+static void __exit zlib_mod_fini(void)
+{
+ crypto_unregister_alg(&alg);
+}
+
+module_init(zlib_mod_init);
+module_exit(zlib_mod_fini);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Zlib Compression Algorithm");
+
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
Signed-off-by: Geert Uytterhoeven <[email protected]>
---
tcrypt.c | 6 +++-
testmgr.c | 15 +++++++++++
testmgr.h | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 101 insertions(+), 1 deletion(-)
diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c
index 28a45a1..c3c9124 100644
--- a/crypto/tcrypt.c
+++ b/crypto/tcrypt.c
@@ -53,7 +53,7 @@ static char *check[] = {
"cast6", "arc4", "michael_mic", "deflate", "crc32c", "tea", "xtea",
"khazad", "wp512", "wp384", "wp256", "tnepres", "xeta", "fcrypt",
"camellia", "seed", "salsa20", "rmd128", "rmd160", "rmd256", "rmd320",
- "lzo", "cts", NULL
+ "lzo", "cts", "zlib", NULL
};
static int test_cipher_jiffies(struct blkcipher_desc *desc, int enc,
@@ -661,6 +661,10 @@ static void do_test(int m)
tcrypt_test("ecb(seed)");
break;
+ case 44:
+ tcrypt_test("zlib");
+ break;
+
case 100:
tcrypt_test("hmac(md5)");
break;
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index b828c6c..9320bb8 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -1770,6 +1770,21 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}
}
+ }, {
+ .alg = "zlib",
+ .test = alg_test_comp,
+ .suite = {
+ .comp = {
+ .comp = {
+ .vecs = zlib_comp_tv_template,
+ .count = ZLIB_COMP_TEST_VECTORS
+ },
+ .decomp = {
+ .vecs = zlib_decomp_tv_template,
+ .count = ZLIB_DECOMP_TEST_VECTORS
+ }
+ }
+ }
}
};
diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index dee94d9..6307b42 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -8509,6 +8509,87 @@ static struct comp_testvec lzo_decomp_tv_template[] = {
};
/*
+ * Zlib test vectors (null-terminated strings).
+ * Params: winbits=DEF_WBITS, Z_DEFAULT_COMPRESSION, MAX_MEM_LEVEL.
+ */
+#define ZLIB_COMP_TEST_VECTORS 2
+#define ZLIB_DECOMP_TEST_VECTORS 2
+
+static struct comp_testvec zlib_comp_tv_template[] = {
+ {
+ .inlen = 70,
+ .outlen = 44,
+ .input = "Join us now and share the software "
+ "Join us now and share the software ",
+ .output = "\x78\x9c\xf3\xca\xcf\xcc\x53\x28"
+ "\x2d\x56\xc8\xcb\x2f\x57\x48\xcc"
+ "\x4b\x51\x28\xce\x48\x2c\x4a\x55"
+ "\x28\xc9\x48\x55\x28\xce\x4f\x2b"
+ "\x29\x07\x71\xbc\x08\x2b\x01\x00"
+ "\x7c\x65\x19\x3d",
+ }, {
+ .inlen = 191,
+ .outlen = 128,
+ .input = "This document describes a compression method based on the DEFLATE"
+ "compression algorithm. This document defines the application of "
+ "the DEFLATE algorithm to the IP Payload Compression Protocol.",
+ .output = "\x78\x9c\x5d\x8d\x31\x0e\xc2\x30"
+ "\x10\x04\xbf\xb2\x2f\xc8\x1f\x10"
+ "\x04\x09\x89\xc2\x85\x3f\x70\xb1"
+ "\x2f\xf8\x24\xdb\x67\xd9\x47\xc1"
+ "\xef\x49\x68\x12\x51\xae\x76\x67"
+ "\xd6\x27\x19\x88\x1a\xde\x85\xab"
+ "\x21\xf2\x08\x5d\x16\x1e\x20\x04"
+ "\x2d\xad\xf3\x18\xa2\x15\x85\x2d"
+ "\x69\xc4\x42\x83\x23\xb6\x6c\x89"
+ "\x71\x9b\xef\xcf\x8b\x9f\xcf\x33"
+ "\xca\x2f\xed\x62\xa9\x4c\x80\xff"
+ "\x13\xaf\x52\x37\xed\x0e\x52\x6b"
+ "\x59\x02\xd9\x4e\xe8\x7a\x76\x1d"
+ "\x02\x98\xfe\x8a\x87\x83\xa3\x4f"
+ "\x56\x8a\xb8\x9e\x8e\x5c\x57\xd3"
+ "\xa0\x79\xfa\x02\x2e\x32\x45\x4e",
+ },
+};
+
+static struct comp_testvec zlib_decomp_tv_template[] = {
+ {
+ .inlen = 128,
+ .outlen = 191,
+ .input = "\x78\x9c\x5d\x8d\x31\x0e\xc2\x30"
+ "\x10\x04\xbf\xb2\x2f\xc8\x1f\x10"
+ "\x04\x09\x89\xc2\x85\x3f\x70\xb1"
+ "\x2f\xf8\x24\xdb\x67\xd9\x47\xc1"
+ "\xef\x49\x68\x12\x51\xae\x76\x67"
+ "\xd6\x27\x19\x88\x1a\xde\x85\xab"
+ "\x21\xf2\x08\x5d\x16\x1e\x20\x04"
+ "\x2d\xad\xf3\x18\xa2\x15\x85\x2d"
+ "\x69\xc4\x42\x83\x23\xb6\x6c\x89"
+ "\x71\x9b\xef\xcf\x8b\x9f\xcf\x33"
+ "\xca\x2f\xed\x62\xa9\x4c\x80\xff"
+ "\x13\xaf\x52\x37\xed\x0e\x52\x6b"
+ "\x59\x02\xd9\x4e\xe8\x7a\x76\x1d"
+ "\x02\x98\xfe\x8a\x87\x83\xa3\x4f"
+ "\x56\x8a\xb8\x9e\x8e\x5c\x57\xd3"
+ "\xa0\x79\xfa\x02\x2e\x32\x45\x4e",
+ .output = "This document describes a compression method based on the DEFLATE"
+ "compression algorithm. This document defines the application of "
+ "the DEFLATE algorithm to the IP Payload Compression Protocol.",
+ }, {
+ .inlen = 44,
+ .outlen = 70,
+ .input = "\x78\x9c\xf3\xca\xcf\xcc\x53\x28"
+ "\x2d\x56\xc8\xcb\x2f\x57\x48\xcc"
+ "\x4b\x51\x28\xce\x48\x2c\x4a\x55"
+ "\x28\xc9\x48\x55\x28\xce\x4f\x2b"
+ "\x29\x07\x71\xbc\x08\x2b\x01\x00"
+ "\x7c\x65\x19\x3d",
+ .output = "Join us now and share the software "
+ "Join us now and share the software ",
+ },
+};
+
+/*
* Michael MIC test vectors from IEEE 802.11i
*/
#define MICHAEL_MIC_TEST_VECTORS 6
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
Modify SquashFS 4 to use the new "zlib" crypto module as a proof-of-concept for
the partial decompression CRYPTO API enhancements.
Note: Patch against squashfs-2.6.git, as SquashFS is not yet in mainline
Signed-off-by: Geert Uytterhoeven <[email protected]>
---
Kconfig | 3 +-
squashfs/block.c | 67 +++++++++++++++++++++++++++-------------------
squashfs/squashfs_fs_sb.h | 4 ++
squashfs/super.c | 20 ++++++++-----
4 files changed, 58 insertions(+), 36 deletions(-)
diff --git a/fs/Kconfig b/fs/Kconfig
index 1d7a5f9..8eebe53 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -934,7 +934,8 @@ config CRAMFS
config SQUASHFS
tristate "SquashFS 4.0 - Squashed file system support"
depends on BLOCK
- select ZLIB_INFLATE
+ select CRYPTO
+ select CRYPTO_ZLIB
help
Saying Y here includes support for SquashFS 4.0 (a Compressed
Read-Only File System). Squashfs is a highly compressed read-only
diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c
index da5f88b..9a681cb 100644
--- a/fs/squashfs/block.c
+++ b/fs/squashfs/block.c
@@ -32,7 +32,7 @@
#include <linux/mutex.h>
#include <linux/string.h>
#include <linux/buffer_head.h>
-#include <linux/zlib.h>
+#include <linux/crypto.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
@@ -150,7 +150,8 @@ int squashfs_read_data(struct super_block *sb, void *buffer,
}
if (compressed) {
- int zlib_err = 0;
+ int error = 0;
+ struct comp_request req;
/*
* Uncompress block.
@@ -158,8 +159,8 @@ int squashfs_read_data(struct super_block *sb, void *buffer,
mutex_lock(&msblk->read_data_mutex);
- msblk->stream.next_out = buffer;
- msblk->stream.avail_out = srclength;
+ req.next_out = buffer;
+ req.avail_out = srclength;
for (bytes = 0; k < b; k++) {
avail = min(c_byte - bytes, msblk->devblksize - offset);
@@ -168,16 +169,19 @@ int squashfs_read_data(struct super_block *sb, void *buffer,
if (!buffer_uptodate(bh[k]))
goto release_mutex;
- msblk->stream.next_in = bh[k]->b_data + offset;
- msblk->stream.avail_in = avail;
+ req.next_in = bh[k]->b_data + offset;
+ req.avail_in = avail;
if (k == 0) {
- zlib_err = zlib_inflateInit(&msblk->stream);
- if (zlib_err != Z_OK) {
- ERROR("zlib_inflateInit returned"
- " unexpected result 0x%x,"
- " srclength %d\n", zlib_err,
- srclength);
+ error = crypto_comp_decompress_init(msblk->tfm,
+ &req);
+ if (error && error != -EAGAIN) {
+ ERROR("crypto_comp_decompress_init "
+ "returned unexpected result %d, "
+ "srclength %d, avail_in %u, "
+ "avail_out %u\n",
+ error, srclength, req.avail_in,
+ req.avail_out);
goto release_mutex;
}
@@ -186,15 +190,25 @@ int squashfs_read_data(struct super_block *sb, void *buffer,
brelse(bh[k]);
continue;
}
+
+ } else {
+ error = crypto_comp_decompress_update(msblk->tfm,
+ &req);
+ if (error) {
+ ERROR("crypto_comp_decompress_update "
+ "returned unexpected result %d, "
+ "srclength %d, avail_in %u, "
+ "avail_out %u\n",
+ error, srclength, req.avail_in,
+ req.avail_out);
+ goto release_mutex;
+ }
}
- zlib_err = zlib_inflate(&msblk->stream, Z_NO_FLUSH);
- if (zlib_err != Z_OK && zlib_err != Z_STREAM_END) {
- ERROR("zlib_inflate returned unexpected result"
- " 0x%x, srclength %d, avail_in %d,"
- " avail_out %d\n", zlib_err, srclength,
- msblk->stream.avail_in,
- msblk->stream.avail_out);
+ if (req.avail_in) {
+ ERROR("crypto_comp_decompress_* did not "
+ "consume all input data (%u left)\n",
+ req.avail_in);
goto release_mutex;
}
@@ -203,16 +217,15 @@ int squashfs_read_data(struct super_block *sb, void *buffer,
brelse(bh[k]);
}
- if (zlib_err != Z_STREAM_END)
- goto release_mutex;
-
- zlib_err = zlib_inflateEnd(&msblk->stream);
- if (zlib_err != Z_OK) {
- ERROR("zlib_inflateEnd returned unexpected result 0x%x,"
- " srclength %d\n", zlib_err, srclength);
+ error = crypto_comp_decompress_final(msblk->tfm, &req);
+ if (error) {
+ ERROR("crypto_comp_decompress_final returned "
+ "unexpected result %d, srclength %d, avail_in "
+ "%u, avail_out %u\n",
+ error, srclength, req.avail_in, req.avail_out);
goto release_mutex;
}
- bytes = msblk->stream.total_out;
+ bytes = req.next_out - buffer;
mutex_unlock(&msblk->read_data_mutex);
} else {
/*
diff --git a/fs/squashfs/squashfs_fs_sb.h b/fs/squashfs/squashfs_fs_sb.h
index 18e5af4..769bbef 100644
--- a/fs/squashfs/squashfs_fs_sb.h
+++ b/fs/squashfs/squashfs_fs_sb.h
@@ -23,6 +23,8 @@
* squashfs_fs_sb.h
*/
+#include <linux/crypto.h>
+
#include "squashfs_fs.h"
struct squashfs_cache_entry {
@@ -63,7 +65,7 @@ struct squashfs_sb_info {
struct mutex read_data_mutex;
struct mutex meta_index_mutex;
struct meta_index *meta_index;
- z_stream stream;
+ struct crypto_comp *tfm;
__le64 *inode_lookup_table;
long long inode_table;
long long directory_table;
diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
index 44f94aa..38e0535 100644
--- a/fs/squashfs/super.c
+++ b/fs/squashfs/super.c
@@ -35,13 +35,17 @@
#include <linux/pagemap.h>
#include <linux/init.h>
#include <linux/module.h>
-#include <linux/zlib.h>
+#include <linux/crypto.h>
#include "squashfs_fs.h"
#include "squashfs_fs_sb.h"
#include "squashfs_fs_i.h"
#include "squashfs.h"
+
+#define SQUASHFS_CRYPTO_ALG "zlib"
+
+
static struct file_system_type squashfs_fs_type;
static struct super_operations squashfs_super_ops;
@@ -86,9 +90,11 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
}
msblk = sb->s_fs_info;
- msblk->stream.workspace = vmalloc(zlib_inflate_workspacesize());
- if (msblk->stream.workspace == NULL) {
- ERROR("Failed to allocate zlib workspace\n");
+ msblk->tfm = crypto_alloc_comp(SQUASHFS_CRYPTO_ALG, 0,
+ CRYPTO_ALG_ASYNC);
+ if (IS_ERR(msblk->tfm)) {
+ ERROR("Failed to load %s crypto module\n", SQUASHFS_CRYPTO_ALG);
+ msblk->tfm = NULL;
goto failure;
}
@@ -284,14 +290,14 @@ failed_mount:
kfree(msblk->inode_lookup_table);
kfree(msblk->fragment_index);
kfree(msblk->id_table);
- vfree(msblk->stream.workspace);
+ crypto_free_comp(msblk->tfm);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
kfree(sblk);
return err;
failure:
- vfree(msblk->stream.workspace);
+ crypto_free_comp(msblk->tfm);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
return -ENOMEM;
@@ -333,7 +339,7 @@ static void squashfs_put_super(struct super_block *sb)
kfree(sbi->id_table);
kfree(sbi->fragment_index);
kfree(sbi->meta_index);
- vfree(sbi->stream.workspace);
+ crypto_free_comp(sbi->tfm);
kfree(sb->s_fs_info);
sb->s_fs_info = NULL;
}
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010
Thanks for the patches!
On Mon, Nov 24, 2008 at 05:11:42PM +0100, Geert Uytterhoeven wrote:
>
> As pointed out in patch [1/4], some underlying (de)compression implementations
> do not support partial (de)compression. This is the case for e.g. LZO, which
> is already supported by a crypto module. While one-shot (de)compression can
> easily be implemented on top of a partial (de)compression API, the inverse is
> not true.
But that's just because lib/lzo didn't implement partial operations,
which means that we can improve it to perform partial compression and
decompression, right?
> Hence there should be a way to ask for a "one-shot" vs. "partial" decompression
> module at crypto_alloc_comp() time.
If we do find such an algorithm (which I'm sceptical), then we can
easily add such a flag which can be used at alloc time to choose
the right type.
In fact, even if such algorithm did exist (i.e., one that requires
all the input to determine the first byte of the output), we can
always handle it by only ever writing output in the final function.
> Do you have any suggestions how to handle this?
> Perhaps by using "oneshot(%s)" and "partial(%s)", cfr. what is done in other
> parts of the CRYPTO API where orthogonal features are combined?
The standard way is to add a CRYPTO_ALG_FOO flag and use it at
alloc time.
But I really can't imagine how such an algorithm could possibly
work beyond a fixed size input.
Also, I'm not saying that you need to convert lzo yourself since
you can always add a new compress type alongside the existing
crypto_comp type. When someone does finally convert lzo we can
then kill the old type. See the shash/digest code for an example
of such a cohabitation.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Mon, Nov 24, 2008 at 05:12:37PM +0100, Geert Uytterhoeven wrote:
> To solve this, add the following operations that support partial
> (de)compression:
> - crypto_compress_{init,update,final}() for compression,
> - crypto_decompress_{init,update,final}() for decompression.
Probably best to put this in a new type instead of crypto_comp.
That way you can still get a bisectable tree without having to
lump everything into one patch.
Doing that also means that you don't have to convert all the algorithms
at once (but please do convert them eventually so I don't have to :)
To make the conversion happen gradually you'd make all algorithms
of the new type available through the existing type (crypto_comp).
See how crypto_shash handles this with regards to the existing
crypto_hash type.
Once all algorithms are converted then you can convert the existing
users (new users can begin using the new type immediately). Finally
you'd kill the crypto_comp type.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Tue, 25 Nov 2008, Herbert Xu wrote:
> On Mon, Nov 24, 2008 at 05:11:42PM +0100, Geert Uytterhoeven wrote:
> > As pointed out in patch [1/4], some underlying (de)compression implementations
> > do not support partial (de)compression. This is the case for e.g. LZO, which
> > is already supported by a crypto module. While one-shot (de)compression can
> > easily be implemented on top of a partial (de)compression API, the inverse is
> > not true.
>
> But that's just because lib/lzo didn't implement partial operations,
> which means that we can improve it to perform partial compression and
> decompression, right?
That's right. If the actual algorithm supports it, lib/lzo can be enhanced.
> > Hence there should be a way to ask for a "one-shot" vs. "partial" decompression
> > module at crypto_alloc_comp() time.
>
> If we do find such an algorithm (which I'm sceptical), then we can
> easily add such a flag which can be used at alloc time to choose
> the right type.
>
> In fact, even if such algorithm did exist (i.e., one that requires
> all the input to determine the first byte of the output), we can
> always handle it by only ever writing output in the final function.
Indeed. Whether the users can handle it is a different question. Current
Squashfs assumes the decompressor always consumes all input bytes on each and
every call.
> > Do you have any suggestions how to handle this?
> > Perhaps by using "oneshot(%s)" and "partial(%s)", cfr. what is done in other
> > parts of the CRYPTO API where orthogonal features are combined?
>
> The standard way is to add a CRYPTO_ALG_FOO flag and use it at
> alloc time.
>
> But I really can't imagine how such an algorithm could possibly
> work beyond a fixed size input.
>
> Also, I'm not saying that you need to convert lzo yourself since
> you can always add a new compress type alongside the existing
> crypto_comp type. When someone does finally convert lzo we can
> then kill the old type. See the shash/digest code for an example
> of such a cohabitation.
OK.
On Tue, 25 Nov 2008, Herbert Xu wrote:
> On Mon, Nov 24, 2008 at 05:12:37PM +0100, Geert Uytterhoeven wrote:
> > To solve this, add the following operations that support partial
> > (de)compression:
> > - crypto_compress_{init,update,final}() for compression,
> > - crypto_decompress_{init,update,final}() for decompression.
>
> Probably best to put this in a new type instead of crypto_comp.
> That way you can still get a bisectable tree without having to
> lump everything into one patch.
OK, I'll create a new crypto_* type.
> Doing that also means that you don't have to convert all the algorithms
> at once (but please do convert them eventually so I don't have to :)
Currently the only (de)compression algorithms supported by crypto/ are
deflate/zlib and lzo, so there's not that much to convert/migrate.
Obviously I'll take care of deflate/zlib.
> To make the conversion happen gradually you'd make all algorithms
> of the new type available through the existing type (crypto_comp).
> See how crypto_shash handles this with regards to the existing
> crypto_hash type.
OK, I'll look into that.
> Once all algorithms are converted then you can convert the existing
> users (new users can begin using the new type immediately). Finally
> you'd kill the crypto_comp type.
OK.
Thx!
With kind regards,
Geert Uytterhoeven
Software Architect
Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium
Phone: +32 (0)2 700 8453
Fax: +32 (0)2 700 8622
E-mail: [email protected]
Internet: http://www.sony-europe.com/
A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010