Hi all.
(Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
A while back now, I stopped supplying the LZF compression algorithm with
TuxOnIce and made LZO the default algorithm. Around the same time, we
started getting occasional errors when reading images; decompression
failures.
I've finally managed to find the time to properly look at this, and have
managed to find a data page that LZO compresses, but seems to be unable
to decompress back to the original contents. I'm wondering whether this
is because I'm doing something wrong, or because there really is some
data the LZO (or the kernel implementation) can't do reversible
compression on.
I've turned the test case and the way TuxOnIce invokes the compression
and decompression code into a standalone kernel module (attached). On my
64 bit machine, insmoding the module results in the error path being
triggered (skipping the dump of the original page):
[52205.333463] Output 716 bytes. Result 0.
[52205.333468] Compressed to: 06 f8 8f 09 00 00 ea ff ff 00 a1 00 40 7a
00 ff
[52205.333473] Compressed to: ff 27 3c 00 78 01 2b 0c 00 01 30 90 09 00
7c 05
[52205.333478] Compressed to: fc 00 d0 03 3f dd 00 68 dc 06 fc 00 20 07
dd 00
[52205.333483] Compressed to: a0 dc 06 fc 00 20 07 dd 00 d8 dc 06 fc 00
20 07
[52205.333488] Compressed to: de 00 10 91 bc 1b fc 00 20 07 dd 00 48 dc
06 fc
[52205.333493] Compressed to: 00 20 07 dd 00 80 dc 06 fc 00 20 07 dd 00
b8 dc
[52205.333498] Compressed to: 06 fc 00 20 07 dd 00 f0 dc 06 fc 00 20 07
de 00
[52205.333503] Compressed to: 28 92 bc 22 fc 00 20 07 dd 00 60 dc 06 fc
00 20
[52205.333508] Compressed to: 07 dd 00 98 dc 06 fc 00 20 07 dd 00 d0 dc
06 fc
[52205.333512] Compressed to: 00 20 07 de 00 08 93 bc 1b fc 00 20 07 dd
00 40
[52205.333517] Compressed to: dc 06 fc 00 20 07 dd 00 78 dc 06 fc 00 20
07 dd
[52205.333522] Compressed to: 00 b0 dc 06 fc 00 20 07 dd 00 e8 dc 06 fc
00 20
[52205.333527] Compressed to: 07 de 00 20 94 bc 22 fc 00 20 07 dd 00 58
dc 06
[52205.333532] Compressed to: fc 00 20 07 dd 00 90 dc 06 fc 00 20 07 dd
00 c8
[52205.333537] Compressed to: dc 06 fc 00 20 07 de 00 00 95 bc 1b 27 1c
00 a0
[52205.333542] Compressed to: 07 3f 3d 13 38 dc 06 fc 00 b9 06 00 3f dd
00 70
[52205.333546] Compressed to: dc 06 fc 00 20 07 dd 00 a8 dc 06 fc 00 20
07 dd
[52205.333551] Compressed to: 00 e0 dc 06 fc 00 20 07 de 00 18 96 bc 22
fc 00
[52205.333556] Compressed to: 20 07 dd 00 50 dc 06 fc 00 20 07 dd 00 88
dc 06
[52205.333561] Compressed to: fc 00 20 07 dd 00 c0 dc 06 fc 00 20 07 dd
00 f8
[52205.333566] Compressed to: dc 06 fc 00 20 07 de 00 30 97 bc 22 fc 00
20 07
[52205.333571] Compressed to: dd 00 68 dc 06 fc 00 20 07 dd 00 a0 dc 06
fc 00
[52205.333576] Compressed to: 20 07 dd 00 d8 dc 06 fc 00 20 07 de 00 10
98 bc
[52205.333581] Compressed to: 1b fc 00 20 07 dd 00 48 dc 06 fc 00 20 07
dd 00
[52205.333586] Compressed to: 80 dc 06 fc 00 20 07 dd 00 b8 dc 06 fc 00
20 07
[52205.333591] Compressed to: dd 00 f0 dc 06 fc 00 20 07 de 00 28 99 bc
22 fc
[52205.333596] Compressed to: 00 20 07 dd 00 60 dc 06 fc 00 20 07 dd 00
98 dc
[52205.333601] Compressed to: 06 fc 00 20 07 dd 00 d0 dc 06 fc 00 20 07
de 00
[52205.333606] Compressed to: 08 9a bc 1b fc 00 20 07 dd 00 40 dc 06 fc
00 20
[52205.333611] Compressed to: 07 dd 00 78 dc 06 fc 00 20 07 dd 00 b0 dc
06 fc
[52205.333616] Compressed to: 00 20 07 dd 00 e8 dc 06 fc 00 20 07 de 00
20 9b
[52205.333621] Compressed to: bc 22 fc 00 20 07 dd 00 58 dc 06 fc 00 20
07 dd
[52205.333625] Compressed to: 00 90 dc 06 fc 00 20 07 dd 00 c8 dc 06 fc
00 20
[52205.333630] Compressed to: 07 de 00 00 9c bc 1b 27 1c 00 a0 07 3f 1d
1b 38
[52205.333635] Compressed to: dc 06 fc 00 b8 06 20 02 fc 1b dd 06 70 dc
00 20
[52205.333640] Compressed to: 07 dd 00 a8 dc 05 fc 00 20 07 dd 00 e0 dc
06 fc
[52205.333645] Compressed to: 00 20 07 de 00 18 9d bc 22 fc 00 20 07 dd
00 50
[52205.333650] Compressed to: dc 06 fc 00 20 07 dd 00 88 dc 06 fc 00 20
07 dd
[52205.333655] Compressed to: 00 c0 dc 06 fc 00 20 07 dd 00 f8 dc 06 fc
00 20
[52205.333660] Compressed to: 07 de 00 30 9e bc 22 fc 00 20 07 dd 00 68
dc 06
[52205.333665] Compressed to: fc 00 20 07 dd 00 a0 dc 06 fc 00 20 07 dd
00 d8
[52205.333670] Compressed to: dc 06 fc 00 20 07 de 00 10 9f bc 1b fc 00
20 07
[52205.333675] Compressed to: dd 00 48 dc 06 fc 00 20 07 dd 00 80 dc 06
fc 00
[52205.333679] Compressed to: 20 07 dd 00 b8 dc 06 fc 00 20 07 dd 00 f0
dc 06
[52205.333684] Compressed to: 05 f0 9f 09 00 00 ea ff ff 11 00 00
[52205.333689]
[52205.333691] Restored to 0 bytes, result code -22.
Would someone be willing and able to tell me what (if anything) I'm
doing wrong, or whether there is something wrong with the algo or its
implementation?
Thanks!
Nigel
On Wednesday 03 February 2010, Nigel Cunningham wrote:
> Hi all.
>
> (Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
>
> A while back now, I stopped supplying the LZF compression algorithm with
> TuxOnIce and made LZO the default algorithm. Around the same time, we
> started getting occasional errors when reading images; decompression
> failures.
>
> I've finally managed to find the time to properly look at this, and have
> managed to find a data page that LZO compresses, but seems to be unable
> to decompress back to the original contents. I'm wondering whether this
> is because I'm doing something wrong, or because there really is some
> data the LZO (or the kernel implementation) can't do reversible
> compression on.
Well, FWIW, we have never had any problems with the userland LZO in s2disk,
so if anything is wrong with LZO here, I guess it's the kernel code.
Rafael
Hi Rafael.
Rafael J. Wysocki wrote:
> On Wednesday 03 February 2010, Nigel Cunningham wrote:
>> Hi all.
>>
>> (Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
>>
>> A while back now, I stopped supplying the LZF compression algorithm with
>> TuxOnIce and made LZO the default algorithm. Around the same time, we
>> started getting occasional errors when reading images; decompression
>> failures.
>>
>> I've finally managed to find the time to properly look at this, and have
>> managed to find a data page that LZO compresses, but seems to be unable
>> to decompress back to the original contents. I'm wondering whether this
>> is because I'm doing something wrong, or because there really is some
>> data the LZO (or the kernel implementation) can't do reversible
>> compression on.
>
> Well, FWIW, we have never had any problems with the userland LZO in s2disk,
> so if anything is wrong with LZO here, I guess it's the kernel code.
Okay. Guess I have to start shipping LZF again and make it the default
again then.
Regards,
Nigel
Nigel Cunningham wrote:
> Hi Rafael.
>
> Rafael J. Wysocki wrote:
>> On Wednesday 03 February 2010, Nigel Cunningham wrote:
>>> Hi all.
>>>
>>> (Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
>>>
>>> A while back now, I stopped supplying the LZF compression algorithm with
>>> TuxOnIce and made LZO the default algorithm. Around the same time, we
>>> started getting occasional errors when reading images; decompression
>>> failures.
>>>
>>> I've finally managed to find the time to properly look at this, and have
>>> managed to find a data page that LZO compresses, but seems to be unable
>>> to decompress back to the original contents. I'm wondering whether this
>>> is because I'm doing something wrong, or because there really is some
>>> data the LZO (or the kernel implementation) can't do reversible
>>> compression on.
>> Well, FWIW, we have never had any problems with the userland LZO in s2disk,
>> so if anything is wrong with LZO here, I guess it's the kernel code.
>
> Okay. Guess I have to start shipping LZF again and make it the default
> again then.
>
I would hope someone will look at the real problem, though, that LZO isn't
working properly. I have to assume that either the kernel decompress is broken
or that the page you have given is invalid, and the error lies in the compression.
It doesn't look as if you are doing something wrong, it looks broken.
--
Bill Davidsen <[email protected]>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
Hi Bill.
Bill Davidsen wrote:
> I would hope someone will look at the real problem, though, that LZO
> isn't working properly. I have to assume that either the kernel
> decompress is broken or that the page you have given is invalid, and the
> error lies in the compression.
>
> It doesn't look as if you are doing something wrong, it looks broken.
I did get hold of Richard Purdie and Nitin Gupta, who were the guys in
the know. We discovered that LZO is expecting decomp_size to be
initialised to the amount of available space when the decompression code
is called, so there was a bug in my testing code. Nitin was talking
about sending a patch to the documentation to make this requirement clearer.
That said, the actual code that TuxOnIce uses does already initialise
the variable to PAGE_SIZE, so it seems that I might just have to run
with the checking code enabled (with this fix) for a while, until the
issue is found.
Regards,
Nigel
On 8 Feb 2010, Nigel Cunningham verbalised:
> That said, the actual code that TuxOnIce uses does already initialise
> the variable to PAGE_SIZE, so it seems that I might just have to run
> with the checking code enabled (with this fix) for a while, until the
> issue is found.
If you push the fix somewhere I'll run with it as well and see if I can
make this problem happen again. (So far I've only seen it with both
compression and max_workers > 1, but I haven't done a systematic check
for this so it could be an illusion.)
Hi.
Nix wrote:
> On 8 Feb 2010, Nigel Cunningham verbalised:
>> That said, the actual code that TuxOnIce uses does already initialise
>> the variable to PAGE_SIZE, so it seems that I might just have to run
>> with the checking code enabled (with this fix) for a while, until the
>> issue is found.
>
> If you push the fix somewhere I'll run with it as well and see if I can
> make this problem happen again. (So far I've only seen it with both
> compression and max_workers > 1, but I haven't done a systematic check
> for this so it could be an illusion.)
I will do - I've just been very slow to because I'm in the middle of
switching website hosting at the moment (as well as the week-to-week
parts of my real job!). Too much to do and not enough time :)
Nigel