Received: by 10.213.65.68 with SMTP id h4csp342139imn; Fri, 23 Mar 2018 05:58:35 -0700 (PDT) X-Google-Smtp-Source: AG47ELvDJuwLcY59wLaIcrP9DLupCX2AtlGnPRFpkZ1CuFOQ8YyQ5yGLNhzlnH/32Hc9OWshWeWd X-Received: by 10.99.105.7 with SMTP id e7mr20774955pgc.193.1521809915872; Fri, 23 Mar 2018 05:58:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1521809915; cv=none; d=google.com; s=arc-20160816; b=r7dynhRg+/YUVlLayIy3Dui2gNT8zMiPma0f/Cp4umnjoA/79srIW2nPxgByup7f7M fJpl5WTNVBlcZrzyV54jXJdOXD0YvKxny/xXFgJDnnhOXLAeGwutW7oAjh5LOR+SR18+ XleFoC4TfrH73dHrHO7WWwZ6UESGVLZUMhmOiix1i7tjvA6peNyOf7/Q06DQ0ixQhd7C ttkSGrmsHA9UkTgPPX375qbXdQDvAS5X0EApIpWOaqfvqVsaecNMM8sXXKFcnaxR9TQH toU1BObLsrDvWO89/p8d43bzuItr3gF2d5+h1LeLmCxI1ztBbERWEaDvFxCe4RDN34d2 pZVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:to:cc:date:message-id:subject :mime-version:content-transfer-encoding:from :arc-authentication-results; bh=00Gf1MNSmmnbAcGRCWoU6zGN4U2/8FTv69v9IsNxFUc=; b=EOrdUSA0KuvK9OGoI6J1VuyMGC44XX9cjEBA0rSI/4dspSGyMRuuTVQcWC+7QV9Lch +e1cqaht2jvk5yjRg7nnmIVl73GWEjne3np/UocLwu9fw4aopnMtHcSEKCAFTqwDLP6P g8qz/vXgRUMMEJlJVgRRwnXJyHhttMEogpFPI6uo3RK7PL7q3kY1lUUbJrExz86Ebvfz JY8KOYlNeQnVd3dMNIpd1/4cREcoCWw1nZtoka48nQn99qdKa3cWQTNhZYVte8/e7XLP KUueWfd/QHmd7mYSSJlFiHeDJglNhvYqLAXfuhLDM3z8os9GSy58C1CKX1Q0C8bsisdT R06A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l8si6578269pff.237.2018.03.23.05.58.21; Fri, 23 Mar 2018 05:58:35 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752792AbeCWM5S convert rfc822-to-8bit (ORCPT + 99 others); Fri, 23 Mar 2018 08:57:18 -0400 Received: from mailhost.informatik.uni-hamburg.de ([134.100.9.70]:32934 "EHLO mailhost.informatik.uni-hamburg.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752593AbeCWM5P (ORCPT ); Fri, 23 Mar 2018 08:57:15 -0400 Received: from localhost (localhost [127.0.0.1]) by mailhost.informatik.uni-hamburg.de (Postfix) with ESMTP id E0A4DC5A; Fri, 23 Mar 2018 13:57:11 +0100 (CET) X-Virus-Scanned: amavisd-new at informatik.uni-hamburg.de Received: from mailhost.informatik.uni-hamburg.de ([127.0.0.1]) by localhost (mailhost.informatik.uni-hamburg.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id koWqaYd+WE7x; Fri, 23 Mar 2018 13:56:47 +0100 (CET) Received: from benjamins-mbp.fritz.box (port-42392.pppoe.wtnet.de [134.101.168.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: 4bwarnke@informatik.uni-hamburg.de) by mailhost.informatik.uni-hamburg.de (Postfix) with ESMTPSA id 8D307C43; Fri, 23 Mar 2018 13:56:43 +0100 (CET) From: Benjamin Warnke <4bwarnke@informatik.uni-hamburg.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8BIT Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: [RFC PATCH v5 0/5] add compression algorithm zBeWalgo Message-Id: <23372483-85EF-45F8-B068-9FE2045B6BBA@informatik.uni-hamburg.de> Date: Fri, 23 Mar 2018 13:56:42 +0100 Cc: Linux Crypto Mailing List , LKML , Herbert Xu , "David S. Miller" , minchan@kernel.org, Sergey Senozhatsky , ngupta@vflare.org, Philippe Ombredanne , Eric Biggers , Stephan Mueller To: Benjamin Warnke <4bwarnke@informatik.uni-hamburg.de> X-Mailer: Apple Mail (2.3445.5.20) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch series adds a new compression algorithm to the kernel and to the crypto api. Changes since v4: - Fix mismatching function-prototypes - Fix mismatching license errors - Add static to variables and functions where possible - Add ULL to long constants Changes since v3: - Split patch into patchset - Add Zstd = Zstandard to the list of benchmarked algorithms - Added configurable compression levels to crypto-api - Added multiple compression levels to the benchmarks below - Added unsafe decompressor functions to crypto-api - Added flag to mark unstable algorithms to crypto-api - Test the code using afl-fuzz -> and fix the code - Added 2 new Benchmark datasets - checkpatch.pl fixes Changes since v2: - added linux-kernel Mailinglist Changes since v1: - improved documentation - improved code style - replaced numerous casts with get_unaligned* - added tests in crypto/testmgr.h/c - added zBeWalgo to the list of algorithms shown by /sys/block/zram0/comp_algorithm Currently ZRAM uses compression-algorithms from the crypto-api. ZRAM compresses each page individually. As a result the compression algorithm is forced to use a very small sliding window. None of the available compression algorithms is designed to achieve high compression ratios with small inputs. This patch-set adds a new compression algorithm 'zBeWalgo' to the crypto api. This algorithm focusses on increasing the capacity of the compressed block-device created by ZRAM. The choice of compression algorithms is always a tradeoff between speed and compression ratio. If faster algorithms like 'lz4' are chosen the compression ratio is often lower than the ratio of zBeWalgo as shown in the following benchmarks. Due to the lower compression ratio, ZRAM needs to fall back to backing_devices mode often. If backing_devices are required, the effective speed of ZRAM is a weighted average of de/compression time and writing/reading from the backing_device. This should be considered when comparing the speeds in the benchmarks. There are different kinds of backing_devices, each with its own drawbacks. 1. HDDs: This kind of backing device is very slow. If the compression ratio of an algorithm is much lower than the ratio of zBeWalgo, it might be faster to use zBewalgo instead. 2. SSDs: I tested a swap partition on my NVME-SSD. The speed is even higher than zram with lz4, but after about 5 Minutes the SSD is blocking all read/write requests due to overheating. This is definitly not an option. Benchmarks: To obtain reproducable benchmarks, the datasets were first loaded into a userspace-program. Than the data is written directly to a clean zram-partition without any filesystem. Between writing and reading 'sync' and 'echo 3 > /proc/sys/vm/drop_caches' is called. All time measurements are wall clock times, and the benchmarks are using only one cpu-core at a time. The new algorithm is compared to all available compression algorithms from the crypto-api. Before loading the datasets to user-space deduplication is applied, since none Algorithm has deduplication. Duplicated pages are removed to prevent an algorithm to obtain high/low ratios, just because a single page can be compressed very well - or not. All Algorithms marked with '*' are using unsafe decompression. All Read and Write Speed Measurements are given in MBit/s zbewalgo' uses per dataset specialized different combinations. These can be specified at runtime via /sys/kernel/zbewalgo/combinations. - '/dev/zero' This dataset is used to measure the speed limitations for ZRAM. ZRAM filters zero-data internally and does not even call the specified compression algorithm. Algorithm write read --zram-- 2724.08 2828.87 - 'ecoham' This dataset is one of the input files for the scientific application ECOHAM which runs an ocean simulation. This dataset contains a lot of zeros - even after deduplication. Where the data is not zero there are arrays of floating point values, adjacent float values are likely to be similar to each other, allowing for high compression ratios. zbewalgo reaches very high compression ratios and is a lot faster than other algorithms with similar compression ratios. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lz4*_10 6.73 1303.12 1547.17 lz4_10 6.73 1303.12 1574.51 lzo 6.88 1205.98 1468.09 lz4*_05 7.00 1291.81 1642.41 lz4_05 7.00 1291.81 1682.81 lz4_07 7.13 1250.29 1593.89 lz4*_07 7.13 1250.29 1677.08 lz4_06 7.16 1307.62 1666.66 lz4*_06 7.16 1307.62 1669.42 lz4_03 7.21 1250.87 1449.48 lz4*_03 7.21 1250.87 1621.97 lz4*_04 7.23 1281.62 1645.56 lz4_04 7.23 1281.62 1666.81 lz4_02 7.33 1267.54 1523.11 lz4*_02 7.33 1267.54 1576.54 lz4_09 7.36 1140.55 1510.01 lz4*_09 7.36 1140.55 1692.38 lz4*_01 7.36 1215.40 1575.38 lz4_01 7.36 1215.40 1676.65 lz4_08 7.36 1242.73 1544.07 lz4*_08 7.36 1242.73 1692.92 lz4hc_01 7.51 235.85 1545.61 lz4hc*_01 7.51 235.85 1678.00 lz4hc_02 7.62 226.30 1697.42 lz4hc*_02 7.62 226.30 1738.79 lz4hc*_03 7.71 194.64 1711.58 lz4hc_03 7.71 194.64 1713.59 lz4hc*_04 7.76 177.17 1642.39 lz4hc_04 7.76 177.17 1698.36 deflate_1 7.80 84.71 584.89 lz4hc*_05 7.81 149.11 1558.43 lz4hc_05 7.81 149.11 1686.71 deflate_2 7.82 82.83 599.38 deflate_3 7.86 84.27 616.05 lz4hc_06 7.88 106.61 1680.52 lz4hc*_06 7.88 106.61 1739.78 zstd_07 7.92 230.34 1016.91 zstd_05 7.92 252.71 1070.46 zstd_06 7.93 237.84 1062.11 lz4hc*_07 7.94 75.22 1751.91 lz4hc_07 7.94 75.22 1768.98 zstd_04 7.94 403.21 1080.62 zstd_03 7.94 411.91 1077.26 zstd_01 7.94 455.89 1082.54 zstd_09 7.94 456.81 1079.22 zstd_08 7.94 459.54 1082.07 zstd_02 7.94 465.82 1056.67 zstd_11 7.95 150.15 1070.31 zstd_10 7.95 169.95 1107.86 lz4hc_08 7.98 49.53 1611.61 lz4hc*_08 7.98 49.53 1793.68 lz4hc_09 7.98 49.62 1629.63 lz4hc*_09 7.98 49.62 1639.83 lz4hc*_10 7.99 37.96 1742.65 lz4hc_10 7.99 37.96 1790.08 zbewalgo 8.02 38.58 237.92 zbewalgo* 8.02 38.58 239.10 842 8.05 169.90 597.01 zstd_13 8.06 129.78 1131.66 zstd_12 8.06 135.50 1126.59 deflate_4 8.16 71.14 546.52 deflate_5 8.17 70.86 537.05 zstd_17 8.19 61.46 1061.45 zstd_14 8.20 124.43 1133.68 zstd_18 8.21 56.82 1151.25 zstd_19 8.22 51.51 1161.83 zstd_20 8.24 44.26 1108.36 zstd_16 8.25 76.26 1042.82 zstd_15 8.25 86.65 1181.98 deflate_6 8.28 66.45 619.62 deflate_7 8.30 63.83 631.13 zstd_21 8.41 6.73 1177.38 zstd_22 8.46 2.23 1188.39 deflate_9 8.47 44.16 678.43 deflate_8 8.47 48.00 677.50 zbewalgo' 8.80 634.68 1247.56 zbewalgo'* 8.80 634.68 1429.42 - 'source-code' This dataset is a tarball of the source-code from a linux-kernel. zBeWalgo is very bad in compressing text based datasets. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lz4_10 1.49 584.41 1200.01 lz4*_10 1.49 584.41 1251.79 lz4*_07 1.64 559.05 1160.75 lz4_07 1.64 559.05 1160.97 842 1.65 63.66 158.53 lz4_06 1.71 513.03 1068.18 lz4*_06 1.71 513.03 1162.68 lz4_05 1.78 526.31 1136.51 lz4*_05 1.78 526.31 1144.81 lz4*_04 1.87 506.63 1106.31 lz4_04 1.87 506.63 1132.96 zbewalgo 1.89 27.56 35.04 zbewalgo* 1.89 27.56 36.20 zbewalgo' 1.89 46.62 34.75 zbewalgo'* 1.89 46.62 36.34 lz4_03 1.98 485.91 984.92 lz4*_03 1.98 485.91 1125.68 lz4_02 2.07 454.96 1061.05 lz4*_02 2.07 454.96 1133.42 lz4_01 2.17 441.11 1141.52 lz4*_01 2.17 441.11 1146.26 lz4*_08 2.17 446.45 1103.61 lz4_08 2.17 446.45 1163.91 lz4*_09 2.17 453.21 1071.91 lz4_09 2.17 453.21 1155.43 lzo 2.27 430.27 871.87 lz4hc*_01 2.35 137.71 1089.94 lz4hc_01 2.35 137.71 1200.45 lz4hc_02 2.38 139.18 1117.44 lz4hc*_02 2.38 139.18 1210.58 lz4hc_03 2.39 127.09 1097.90 lz4hc*_03 2.39 127.09 1214.22 lz4hc_10 2.40 96.26 1203.89 lz4hc*_10 2.40 96.26 1221.94 lz4hc*_08 2.40 98.80 1191.79 lz4hc_08 2.40 98.80 1226.59 lz4hc*_09 2.40 102.36 1213.34 lz4hc_09 2.40 102.36 1225.45 lz4hc*_07 2.40 113.81 1217.63 lz4hc_07 2.40 113.81 1218.49 lz4hc*_06 2.40 117.32 1214.13 lz4hc_06 2.40 117.32 1224.51 lz4hc_05 2.40 122.12 1108.34 lz4hc*_05 2.40 122.12 1214.97 lz4hc*_04 2.40 124.91 1093.58 lz4hc_04 2.40 124.91 1222.05 zstd_01 2.93 200.01 401.15 zstd_08 2.93 200.01 414.52 zstd_09 2.93 200.26 394.83 zstd_02 3.00 201.12 405.73 deflate_1 3.01 53.83 240.64 deflate_2 3.05 52.58 243.31 deflate_3 3.08 52.07 244.84 zstd_04 3.10 158.80 365.06 zstd_03 3.10 169.56 405.92 zstd_05 3.18 125.00 410.23 zstd_06 3.20 106.50 404.81 zstd_07 3.21 99.02 404.23 zstd_15 3.22 24.95 376.58 zstd_16 3.22 26.88 416.44 deflate_4 3.22 45.26 225.56 zstd_13 3.22 62.53 388.33 zstd_14 3.22 64.15 391.81 zstd_12 3.22 66.24 417.67 zstd_11 3.22 66.44 404.31 zstd_10 3.22 73.13 401.98 zstd_17 3.24 14.66 412.00 zstd_18 3.25 13.37 408.46 deflate_5 3.26 43.54 252.18 deflate_7 3.27 39.37 245.63 deflate_6 3.27 42.51 251.33 deflate_9 3.28 40.02 253.99 deflate_8 3.28 40.10 253.98 zstd_19 3.34 10.36 399.85 zstd_22 3.35 4.88 353.63 zstd_21 3.35 6.02 323.33 zstd_20 3.35 8.34 339.81 - 'hpcg' This dataset is a (partial) memory-snapshot of the running hpcg-benchmark. At the time of the snapshot, that application performed a sparse matrix - vector multiplication. The compression ratio of zBeWalgo on this dataset is nearly 3 times higher than the ratio of any other algorithm regardless of the compression-level specified. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lz4*_10 1.00 1130.73 2131.82 lz4_10 1.00 1130.73 2181.60 lz4_06 1.34 625.48 1145.74 lz4*_06 1.34 625.48 1145.90 lz4_07 1.57 515.39 895.42 lz4*_07 1.57 515.39 1062.53 lz4*_05 1.72 539.40 1030.76 lz4_05 1.72 539.40 1038.86 lzo 1.76 475.20 805.41 lz4_08 1.76 480.35 939.16 lz4*_08 1.76 480.35 1015.04 lz4*_03 1.76 488.05 893.13 lz4_03 1.76 488.05 1013.65 lz4*_09 1.76 501.49 1032.69 lz4_09 1.76 501.49 1105.47 lz4*_01 1.76 501.54 1040.72 lz4_01 1.76 501.54 1102.22 lz4*_02 1.76 510.79 1014.78 lz4_02 1.76 510.79 1080.69 lz4_04 1.76 516.18 1047.06 lz4*_04 1.76 516.18 1049.55 842 2.35 109.68 192.50 lz4hc_07 2.36 152.57 1265.77 lz4hc*_07 2.36 152.57 1331.01 lz4hc*_06 2.36 155.78 1313.85 lz4hc_06 2.36 155.78 1346.52 lz4hc*_08 2.36 158.80 1297.16 lz4hc_08 2.36 158.80 1382.54 lz4hc*_10 2.36 159.84 1317.81 lz4hc_10 2.36 159.84 1346.85 lz4hc*_03 2.36 160.01 1162.91 lz4hc_03 2.36 160.01 1377.09 lz4hc*_09 2.36 161.02 1320.87 lz4hc_09 2.36 161.02 1374.39 lz4hc*_05 2.36 164.67 1324.40 lz4hc_05 2.36 164.67 1341.64 lz4hc*_04 2.36 168.11 1323.19 lz4hc_04 2.36 168.11 1377.56 lz4hc_01 2.36 168.40 1231.55 lz4hc*_01 2.36 168.40 1329.72 lz4hc*_02 2.36 170.74 1316.54 lz4hc_02 2.36 170.74 1337.42 deflate_3 3.52 46.51 336.67 deflate_2 3.52 62.05 343.03 deflate_1 3.52 65.68 359.96 deflate_4 4.01 61.01 432.66 deflate_8 4.61 41.51 408.29 deflate_5 4.61 44.09 434.79 deflate_9 4.61 45.14 417.18 deflate_7 4.61 45.22 440.27 deflate_6 4.61 46.01 440.39 zstd_09 5.95 277.11 542.93 zstd_08 5.95 277.40 541.27 zstd_01 5.95 277.41 540.61 zstd_16 5.97 32.05 465.03 zstd_15 5.97 39.12 515.07 zstd_13 5.97 70.90 511.94 zstd_14 5.97 72.20 522.68 zstd_11 5.97 74.14 512.18 zstd_12 5.97 74.27 497.95 zstd_10 5.97 86.98 519.78 zstd_07 5.97 135.16 504.07 zstd_06 5.97 145.49 505.10 zstd_05 6.02 177.86 510.08 zstd_04 6.02 205.13 516.29 zstd_03 6.02 217.82 515.50 zstd_02 6.02 260.97 484.64 zstd_18 6.27 12.10 490.72 zstd_17 6.27 12.33 462.65 zstd_21 6.70 9.25 391.16 zstd_20 6.70 9.50 395.38 zstd_22 6.70 9.74 390.99 zstd_19 6.70 9.99 450.42 zbewalgo 16.33 47.17 430.06 zbewalgo* 16.33 47.17 436.92 zbewalgo' 16.33 188.86 427.78 zbewalgo'* 16.33 188.86 437.43 - 'partdiff' (8 GiB) Array of double values. Adjacent doubles are similar, but not equal. This array is produced by a partial differential equation solver using a Jakobi-implementation. zBewalgo gains higher compression ratios than all other algorithms. Some algorithms are even slower than a hdd without any compression at all. Algorithm ratio write read zstd_18 1.00 13.77 2080.06 zstd_17 1.00 13.80 2075.23 zstd_16 1.00 28.04 2138.99 zstd_15 1.00 45.04 2143.32 zstd_13 1.00 55.72 2128.27 zstd_14 1.00 56.09 2123.54 zstd_11 1.00 57.31 2095.04 zstd_12 1.00 57.53 2134.61 842 1.00 61.61 2267.89 zstd_10 1.00 80.40 2081.35 zstd_07 1.00 120.66 2119.09 zstd_06 1.00 128.80 2134.02 zstd_05 1.00 131.25 2133.01 --hdd-- 1.00 134.70 156.62 lz4hc*_03 1.00 152.82 1982.94 lz4hc_03 1.00 152.82 2261.55 lz4hc*_07 1.00 159.43 1990.03 lz4hc_07 1.00 159.43 2269.05 lz4hc_10 1.00 166.33 2243.78 lz4hc*_10 1.00 166.33 2260.63 lz4hc_09 1.00 167.03 2244.20 lz4hc*_09 1.00 167.03 2264.72 lz4hc*_06 1.00 167.17 2245.15 lz4hc_06 1.00 167.17 2271.88 lz4hc_08 1.00 167.49 2237.79 lz4hc*_08 1.00 167.49 2283.98 lz4hc_02 1.00 167.51 2275.36 lz4hc*_02 1.00 167.51 2279.72 lz4hc*_05 1.00 167.52 2248.92 lz4hc_05 1.00 167.52 2273.99 lz4hc*_04 1.00 167.71 2268.23 lz4hc_04 1.00 167.71 2268.78 lz4hc*_01 1.00 167.91 2268.76 lz4hc_01 1.00 167.91 2269.16 zstd_04 1.00 175.84 2241.60 zstd_03 1.00 176.35 2285.13 zstd_02 1.00 195.41 2269.51 zstd_09 1.00 199.47 2271.91 zstd_01 1.00 199.74 2287.15 zstd_08 1.00 199.87 2286.27 lz4_01 1.00 1160.95 2257.78 lz4*_01 1.00 1160.95 2275.42 lz4_08 1.00 1164.37 2280.06 lz4*_08 1.00 1164.37 2280.43 lz4*_09 1.00 1166.30 2263.05 lz4_09 1.00 1166.30 2280.54 lz4*_03 1.00 1174.00 2074.96 lz4_03 1.00 1174.00 2257.37 lz4_02 1.00 1212.18 2273.60 lz4*_02 1.00 1212.18 2285.66 lz4*_04 1.00 1253.55 2259.60 lz4_04 1.00 1253.55 2287.15 lz4_05 1.00 1279.88 2282.47 lz4*_05 1.00 1279.88 2287.05 lz4_06 1.00 1292.22 2277.95 lz4*_06 1.00 1292.22 2284.84 lz4*_07 1.00 1303.58 2276.10 lz4_07 1.00 1303.58 2276.99 lz4*_10 1.00 1304.80 2183.30 lz4_10 1.00 1304.80 2285.25 lzo 1.00 1360.88 2281.19 deflate_7 1.07 33.51 463.73 deflate_2 1.07 33.99 473.07 deflate_9 1.07 34.05 473.57 deflate_6 1.07 34.06 473.69 deflate_8 1.07 34.12 472.86 deflate_5 1.07 34.22 468.03 deflate_4 1.07 34.32 447.33 deflate_1 1.07 35.45 431.95 deflate_3 1.07 35.63 472.56 zstd_22 1.11 9.81 668.64 zstd_21 1.11 10.71 734.52 zstd_20 1.11 10.78 714.86 zstd_19 1.11 12.02 790.71 zbewalgo 1.29 25.93 225.07 zbewalgo* 1.29 25.93 226.72 zbewalgo'* 1.31 23.54 84.29 zbewalgo' 1.31 23.54 86.08 - 'Isabella CLOUDf01' This dataset is an array of floating point values between 0.00000 and 0.00332. Detailed Information about this dataset is online available at http://www.vets.ucar.edu/vg/isabeldata/readme.html All algorithms obtain similar compression ratios. The compression ratio of zBeWalgo is slightly higher, and the speed is higher too. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lzo 2.06 1022.09 916.22 lz4*_10 2.09 1126.03 1533.35 lz4_10 2.09 1126.03 1569.06 lz4*_07 2.09 1135.89 1444.21 lz4_07 2.09 1135.89 1581.96 lz4*_01 2.10 972.22 1405.21 lz4_01 2.10 972.22 1579.78 lz4*_09 2.10 982.39 1429.17 lz4_09 2.10 982.39 1490.27 lz4_08 2.10 1006.56 1491.14 lz4*_08 2.10 1006.56 1558.66 lz4_02 2.10 1019.82 1366.16 lz4*_02 2.10 1019.82 1578.79 lz4_03 2.10 1129.74 1417.33 lz4*_03 2.10 1129.74 1456.68 lz4_04 2.10 1131.28 1478.27 lz4*_04 2.10 1131.28 1517.84 lz4_06 2.10 1147.78 1424.90 lz4*_06 2.10 1147.78 1462.47 lz4*_05 2.10 1172.44 1434.86 lz4_05 2.10 1172.44 1578.80 lz4hc*_10 2.11 29.01 1498.01 lz4hc_10 2.11 29.01 1580.23 lz4hc*_09 2.11 56.30 1510.26 lz4hc_09 2.11 56.30 1583.11 lz4hc_08 2.11 56.39 1426.43 lz4hc*_08 2.11 56.39 1565.12 lz4hc_07 2.11 129.27 1540.38 lz4hc*_07 2.11 129.27 1578.35 lz4hc*_06 2.11 162.72 1456.27 lz4hc_06 2.11 162.72 1581.69 lz4hc*_05 2.11 183.78 1487.71 lz4hc_05 2.11 183.78 1589.10 lz4hc*_04 2.11 187.41 1431.35 lz4hc_04 2.11 187.41 1566.24 lz4hc*_03 2.11 190.21 1531.98 lz4hc_03 2.11 190.21 1580.81 lz4hc*_02 2.11 199.69 1432.00 lz4hc_02 2.11 199.69 1565.10 lz4hc_01 2.11 205.87 1540.33 lz4hc*_01 2.11 205.87 1567.68 842 2.15 89.89 414.49 deflate_1 2.29 48.84 352.09 deflate_2 2.29 49.47 353.77 deflate_3 2.30 50.00 345.88 zstd_22 2.31 5.59 658.59 zstd_21 2.31 14.34 664.02 zstd_20 2.31 21.22 665.77 zstd_19 2.31 24.26 587.99 zstd_17 2.31 26.24 670.14 zstd_18 2.31 26.47 668.64 deflate_9 2.31 33.79 345.81 deflate_8 2.31 34.67 347.96 deflate_4 2.31 41.46 326.50 deflate_7 2.31 42.56 346.99 deflate_6 2.31 43.51 343.56 deflate_5 2.31 45.83 343.86 zstd_05 2.31 126.01 571.70 zstd_04 2.31 178.39 597.26 zstd_03 2.31 192.04 644.24 zstd_01 2.31 206.31 563.68 zstd_08 2.31 207.39 669.05 zstd_02 2.31 216.98 600.77 zstd_09 2.31 236.92 667.64 zstd_16 2.32 41.47 660.06 zstd_15 2.32 60.37 584.45 zstd_14 2.32 74.60 673.10 zstd_12 2.32 75.16 661.96 zstd_13 2.32 75.22 676.12 zstd_11 2.32 75.58 636.75 zstd_10 2.32 95.05 645.07 zstd_07 2.32 139.52 672.88 zstd_06 2.32 145.40 670.45 zbewalgo'* 2.37 337.07 463.32 zbewalgo' 2.37 337.07 468.96 zbewalgo* 2.60 101.17 578.35 zbewalgo 2.60 101.17 586.88 - 'Isabella TCf01' This dataset is an array of floating point values between -83.00402 and 31.51576. Detailed Information about this dataset is online available at http://www.vets.ucar.edu/vg/isabeldata/readme.html zBeWalgo is the only algorithm which can compress this dataset with a noticeable compressionratio. Algorithm ratio write read 842 1.00 60.09 1956.26 --hdd-- 1.00 134.70 156.62 lz4hc_01 1.00 154.81 1839.37 lz4hc*_01 1.00 154.81 2105.53 lz4hc_10 1.00 157.33 2078.69 lz4hc*_10 1.00 157.33 2113.14 lz4hc_09 1.00 158.50 2018.51 lz4hc*_09 1.00 158.50 2093.65 lz4hc*_02 1.00 159.54 2104.91 lz4hc_02 1.00 159.54 2117.34 lz4hc_03 1.00 161.26 2070.76 lz4hc*_03 1.00 161.26 2107.27 lz4hc*_08 1.00 161.34 2100.74 lz4hc_08 1.00 161.34 2105.26 lz4hc*_04 1.00 161.95 2080.96 lz4hc_04 1.00 161.95 2104.00 lz4hc_05 1.00 162.17 2044.43 lz4hc*_05 1.00 162.17 2101.74 lz4hc*_06 1.00 163.61 2087.19 lz4hc_06 1.00 163.61 2104.61 lz4hc_07 1.00 164.51 2094.78 lz4hc*_07 1.00 164.51 2105.53 lz4_01 1.00 1134.89 2109.70 lz4*_01 1.00 1134.89 2118.71 lz4*_08 1.00 1141.96 2104.87 lz4_08 1.00 1141.96 2118.97 lz4_09 1.00 1145.55 2087.76 lz4*_09 1.00 1145.55 2118.85 lz4_02 1.00 1157.28 2094.33 lz4*_02 1.00 1157.28 2124.67 lz4*_03 1.00 1194.18 2106.36 lz4_03 1.00 1194.18 2119.89 lz4_04 1.00 1195.09 2117.03 lz4*_04 1.00 1195.09 2120.23 lz4*_05 1.00 1225.56 2109.04 lz4_05 1.00 1225.56 2120.52 lz4*_06 1.00 1261.67 2109.14 lz4_06 1.00 1261.67 2121.13 lz4*_07 1.00 1270.86 1844.63 lz4_07 1.00 1270.86 2041.08 lz4_10 1.00 1305.36 2109.22 lz4*_10 1.00 1305.36 2120.65 lzo 1.00 1338.61 2109.66 zstd_17 1.03 13.93 1138.94 zstd_18 1.03 14.01 1170.78 zstd_16 1.03 27.12 1073.75 zstd_15 1.03 43.52 1061.97 zstd_14 1.03 49.60 1082.98 zstd_12 1.03 55.03 1042.43 zstd_13 1.03 55.14 1173.50 zstd_11 1.03 55.24 1178.05 zstd_10 1.03 70.01 1173.05 zstd_07 1.03 118.10 1041.92 zstd_06 1.03 123.00 1171.59 zstd_05 1.03 124.61 1165.74 zstd_01 1.03 166.80 1005.29 zstd_04 1.03 170.25 1127.75 zstd_03 1.03 171.40 1172.34 zstd_02 1.03 174.08 1017.34 zstd_09 1.03 195.30 1176.82 zstd_08 1.03 195.98 1175.09 deflate_9 1.05 30.15 483.55 deflate_8 1.05 30.45 466.67 deflate_5 1.05 31.25 480.92 deflate_4 1.05 31.84 472.81 deflate_7 1.05 31.84 484.18 deflate_6 1.05 31.94 481.37 deflate_2 1.05 33.07 484.09 deflate_3 1.05 33.11 463.57 deflate_1 1.05 33.19 469.71 zstd_22 1.06 8.89 647.75 zstd_21 1.06 10.70 700.11 zstd_20 1.06 10.80 723.42 zstd_19 1.06 12.41 764.24 zbewalgo* 1.51 146.45 581.43 zbewalgo 1.51 146.45 592.86 zbewalgo'* 1.54 38.14 120.96 zbewalgo' 1.54 38.14 125.81 Signed-off-by: Benjamin Warnke <4bwarnke@informatik.uni-hamburg.de> --- -- 2.14.1