Konstantin,
I did my testing with data from /dev/urandom (which I now realize
wasn't the best choice of data source), but if I use /dev/zero (which
actually causes data compression to occur), the decompressor fails. I
don't know the internal workings of the lz4hc compressor or the lz4
decompressor. I couldn't find any examples of any code in the kernel
actually using the compressor. I've cc'ed the maintainers of the
lz4hc_compress.c to see if they my have some more insight to the
issue.
-- Brent
On Tue, Oct 22, 2013 at 5:10 AM, Konstantin Tokarev <[email protected]> wrote:
>
>
> 22.10.2013, 07:43, "Brent Taylor" <[email protected]>:
>> On Mon, Oct 21, 2013 at 10:59 AM, Konstantin Tokarev <[email protected]> wrote:
>>
>>> 04.10.2013, 07:09, "Brent Taylor" <[email protected]>:
>>>> Here is a patch based on linux-3.12-rc3. I haven't performed any
>>>> performance testing UBIFS using lz4hc, but I can mount UBIFS volumes
>>>> and haven't seen any problems yet. The only think I know that isn't
>>>> correct about the patch is the description for the Kconfig element for
>>>> select lz4hc as a compression option. I only copied the description
>>>> from the lzo description.
>>> Hi Brent,
>>>
>>> I'm testing your patch on my SH4 device. When I create new partition
>>> with lz4hc compressor, it works fine: I can copy file into it, and
>>> md5sums of original and copy match. However, after reboot I cannot
>>> read the file anymore:
>>>
>>> UBIFS error (pid 1101): ubifs_decompress: cannot decompress 934 bytes, compressor lz4hc, error -22
>>> UBIFS error (pid 1101): read_block: bad data node (block 1, inode 65)
>>> UBIFS error (pid 1101): do_readpage: cannot read page 1 of inode 65, error -22
>>>
>>> The same error appears if I use lz4hc-compressed ubifs image to flash rootfs
>>> (using patched mkfs.ubifs).
>>>
>>> Decompression error occurs in lz4_uncompress() function (lib/lz4/lz4_decompress.c),
>>> on the line 101:
>>>
>>> /* Error: offset create reference outside destination buffer */
>>> if (unlikely(ref < (BYTE *const) dest))
>>> goto _output_error;
>>>
>>> Brent: are you able to read data from lz4hc volume on your device?
>>> Anyone: any ideas what may happen here?
>>>
>>> --
>>> Regards,
>>> Konstantin
>>
>> Konstantin,
>> I haven't seen anything like that on my at91sam9m10g45-ek
>> development board. I haven't used a flash image from mkfs.ubifs yet.
>> Is it possible the file system was not umounted cleanly before the
>> reboot and UBIFS went through a recovery procedure? Maybe something
>> breaks with lz4hc when UBIFS does a recovery? That's just a guess.
>
> Could you save attached file on lz4hc volume, umount it and mount again?
> I get aforementioned error when doing `cat set11.cfg`
>
> --
> Regards,
> Konstantin
23.10.2013, 09:26, "Brent Taylor" <[email protected]>:
> Konstantin,
> ???I did my testing with data from /dev/urandom (which I now realize
> wasn't the best choice of data source), but if I use /dev/zero (which
> actually causes data compression to occur), the decompressor fails. ?I
> don't know the internal workings of the lz4hc compressor or the lz4
> decompressor. ?I couldn't find any examples of any code in the kernel
> actually using the compressor. ?I've cc'ed the maintainers of the
> lz4hc_compress.c to see if they my have some more insight to the
> issue.
Does decompressor fail for you with the same error messages?
Have you tried to copy my file to the volume? It looks like minimal test case
for my board, if I remove any line decompressor works fine.
--
Regards,
Konstantin
On Wed, Oct 23, 2013 at 2:40 AM, Konstantin Tokarev <[email protected]> wrote:
>
>
> 23.10.2013, 09:26, "Brent Taylor" <[email protected]>:
>> Konstantin,
>> I did my testing with data from /dev/urandom (which I now realize
>> wasn't the best choice of data source), but if I use /dev/zero (which
>> actually causes data compression to occur), the decompressor fails. I
>> don't know the internal workings of the lz4hc compressor or the lz4
>> decompressor. I couldn't find any examples of any code in the kernel
>> actually using the compressor. I've cc'ed the maintainers of the
>> lz4hc_compress.c to see if they my have some more insight to the
>> issue.
>
> Does decompressor fail for you with the same error messages?
>
> Have you tried to copy my file to the volume? It looks like minimal test case
> for my board, if I remove any line decompressor works fine.
>
> --
> Regards,
> Konstantin
Yes, I get the same error, here's a dump from UBIFS when I cat a file
filled with data from /dev/zero:
UBIFS error (pid 4288): ubifs_decompress: cannot decompress 12 bytes,
compressor lz4hc, error -22
UBIFS error (pid 4288): read_block: bad data node (block 0, inode 71)
magic 0x6101831
crc 0xff61a078
node_type 1 (data node)
group_type 0 (no node group)
sqnum 2700
len 60
key (71, data, 0)
size 512
compr_typ 3
data size 12
data:
00000000: 1f 00 01 00 ff e8 50 00 00 00 00 00
UBIFS error (pid 4288): do_readpage: cannot read page 0 of inode 71, error -22
cat: /opt/data/zero.bin: Input/output error
Steps to reproduce are:
1. Create a file with all zeros: dd if=/dev/zero bs=512 count=1
of=/opt/data/zero.bin
2. Unmount ubifs and detach ubi partition
3. attach ubi partition and mount ubifs
4. cat /opt/data/zero.bin
23.10.2013, 16:49, "Brent Taylor" <[email protected]>:
> On Wed, Oct 23, 2013 at 2:40 AM, Konstantin Tokarev <[email protected]> wrote:
>
>> ?23.10.2013, 09:26, "Brent Taylor" <[email protected]>:
>>> ?Konstantin,
>>> ????I did my testing with data from /dev/urandom (which I now realize
>>> ?wasn't the best choice of data source), but if I use /dev/zero (which
>>> ?actually causes data compression to occur), the decompressor fails. ?I
>>> ?don't know the internal workings of the lz4hc compressor or the lz4
>>> ?decompressor. ?I couldn't find any examples of any code in the kernel
>>> ?actually using the compressor. ?I've cc'ed the maintainers of the
>>> ?lz4hc_compress.c to see if they my have some more insight to the
>>> ?issue.
>> ?Does decompressor fail for you with the same error messages?
>>
>> ?Have you tried to copy my file to the volume? It looks like minimal test case
>> ?for my board, if I remove any line decompressor works fine.
>>
>> ?--
>> ?Regards,
>> ?Konstantin
>
> Yes, I get the same error, here's a dump from UBIFS when I cat a file
> filled with data from /dev/zero:
>
> UBIFS error (pid 4288): ubifs_decompress: cannot decompress 12 bytes,
> compressor lz4hc, error -22
> UBIFS error (pid 4288): read_block: bad data node (block 0, inode 71)
> ????????magic ?????????0x6101831
> ????????crc ???????????0xff61a078
> ????????node_type ?????1 (data node)
> ????????group_type ????0 (no node group)
> ????????sqnum ?????????2700
> ????????len ???????????60
> ????????key ???????????(71, data, 0)
> ????????size ??????????512
> ????????compr_typ ?????3
> ????????data size ?????12
> ????????data:
> ????????00000000: 1f 00 01 00 ff e8 50 00 00 00 00 00
> UBIFS error (pid 4288): do_readpage: cannot read page 0 of inode 71, error -22
> cat: /opt/data/zero.bin: Input/output error
>
> Steps to reproduce are:
> 1. ?Create a file with all zeros: dd if=/dev/zero bs=512 count=1
> of=/opt/data/zero.bin
> 2. ?Unmount ubifs and detach ubi partition
> 3. ?attach ubi partition and mount ubifs
> 4. cat /opt/data/zero.bin
Reproduced here.
--
Regards,
Konstantin