Received: by 10.213.65.68 with SMTP id h4csp447615imn; Tue, 20 Mar 2018 07:09:03 -0700 (PDT) X-Google-Smtp-Source: AG47ELtJNU/u2ngEy2ymucv56eJKAfEGEC12U7+7AGD/yAr8bZDb7ADu89MyK3id6tq6eoTzcfZf X-Received: by 10.98.210.7 with SMTP id c7mr10288857pfg.92.1521554943656; Tue, 20 Mar 2018 07:09:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1521554943; cv=none; d=google.com; s=arc-20160816; b=SBesCmEHNXGiWdaRKsUA+It6VsDUIgr/muS0cKYEM3qRKMfyLVta3a2HD9+uDV5mzu XCGD2QFSnhDrdzxr2cC6bqtsZl/PENyxu1QgjTkhXYv1GQ1Gx0n/DoryrmQoSqL3xAI+ 6Ekm9AHOrv6cpqbe3myrM7oCJG7Jm04GVJ/+l9T5wCFovf07VaMplxtG6nLCfJQjME3d pR+76t4q3cOgEGjvUzCyhv9tGBeLYN66Q7xdlKbrnDCs/XIrqsF0jBF9T+/1bEAM4yGs 91Y24POcnOdxd63iRCEKgWbGHYFuVVSgmGA+1NxiLT756H4kaOosAKw/5pzf+khpGOym PC+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:to:cc:date:message-id:subject :mime-version:content-transfer-encoding:from :arc-authentication-results; bh=yjaZg1hTB9r1NJye0LNZtVjjFIghCiFzU9oRBFsCWwk=; b=Jnz6644a8bZAcNk5ram+aQbM5RNcxS2Hbm2Wm+F57jSUwp413uJ4Boe3ULcX4y7ZSw flXK1gb/4NXm59758hPJRjszAG14jKgLcGEdLDTPZ6uqqFQtcGK/lDt617o1soTgdPM4 FqmBSHtMR7T9ZOVtLTJYkTHnyCNQ1fbbZXvQcs8S6ZAsJ5A5EALgBvc9vwNZBM2Syj3K Dd2pUvBJ2isH/SFHTZYJAmw9dCKswnVGjzRHFcfiyCf3XROk9WCm7taQhyFfZVRkdWxJ ArK0x018pP4ktXudl18kwv1auSJMVR72/PFKMILZ7/R3t/e9+LZ1DDkouXauVsWU9br5 Ji6A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p3si1225272pgc.776.2018.03.20.07.08.44; Tue, 20 Mar 2018 07:09:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753958AbeCTOE5 convert rfc822-to-8bit (ORCPT + 99 others); Tue, 20 Mar 2018 10:04:57 -0400 Received: from mailhost.informatik.uni-hamburg.de ([134.100.9.70]:38570 "EHLO mailhost.informatik.uni-hamburg.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753944AbeCTOEy (ORCPT ); Tue, 20 Mar 2018 10:04:54 -0400 Received: from localhost (localhost [127.0.0.1]) by mailhost.informatik.uni-hamburg.de (Postfix) with ESMTP id CCBFA90D; Tue, 20 Mar 2018 15:04:52 +0100 (CET) X-Virus-Scanned: amavisd-new at informatik.uni-hamburg.de Received: from mailhost.informatik.uni-hamburg.de ([127.0.0.1]) by localhost (mailhost.informatik.uni-hamburg.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id CJBCSHXjBod4; Tue, 20 Mar 2018 15:04:27 +0100 (CET) Received: from benjamins-mbp.fritz.box (port-8343.pppoe.wtnet.de [84.46.32.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: 4bwarnke@informatik.uni-hamburg.de) by mailhost.informatik.uni-hamburg.de (Postfix) with ESMTPSA id 9E6388DF; Tue, 20 Mar 2018 15:04:24 +0100 (CET) From: Benjamin Warnke <4bwarnke@informatik.uni-hamburg.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8BIT Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: [PATCH 0/5 v4] add compression algorithm zBeWalgo Message-Id: Date: Tue, 20 Mar 2018 15:04:23 +0100 Cc: Linux Crypto Mailing List , linux-kernel@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, minchan@kernel.org, ngupta@vflare.org, Sergey Senozhatsky To: Benjamin Warnke <4bwarnke@informatik.uni-hamburg.de> X-Mailer: Apple Mail (2.3445.5.20) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch series adds a new compression algorithm to the kernel and to the crypto api. Changes since v3: - Split patch into patchset - Add Zstd = Zstandard to the list of benchmarked algorithms - Added configurable compression levels to crypto-api - Added multiple compression levels to the benchmarks below - Added unsafe decompressor functions to crypto-api - Added flag to mark unstable algorithms to crypto-api - Test the code using afl-fuzz -> and fix the code - Added 2 new Benchmark datasets - checkpatch.pl fixes Changes since v2: - added linux-kernel Mailinglist Changes since v1: - improved documentation - improved code style - replaced numerous casts with get_unaligned* - added tests in crypto/testmgr.h/c - added zBeWalgo to the list of algorithms shown by /sys/block/zram0/comp_algorithm Currently ZRAM uses compression-algorithms from the crypto-api. ZRAM compresses each page individually. As a result the compression algorithm is forced to use a very small sliding window. None of the available compression algorithms is designed to achieve high compression ratios with small inputs. This patch-set adds a new compression algorithm 'zBeWalgo' to the crypto api. This algorithm focusses on increasing the capacity of the compressed block-device created by ZRAM. The choice of compression algorithms is always a tradeoff between speed and compression ratio. If faster algorithms like 'lz4' are chosen the compression ratio is often lower than the ratio of zBeWalgo as shown in the following benchmarks. Due to the lower compression ratio, ZRAM needs to fall back to backing_devices mode often. If backing_devices are required, the effective speed of ZRAM is a weighted average of de/compression time and writing/reading from the backing_device. This should be considered when comparing the speeds in the benchmarks. There are different kinds of backing_devices, each with its own drawbacks. 1. HDDs: This kind of backing device is very slow. If the compression ratio of an algorithm is much lower than the ratio of zBeWalgo, it might be faster to use zBewalgo instead. 2. SSDs: I tested a swap partition on my NVME-SSD. The speed is even higher than zram with lz4, but after about 5 Minutes the SSD is blocking all read/write requests due to overheating. This is definitly not an option. Benchmarks: To obtain reproducable benchmarks, the datasets were first loaded into a userspace-program. Than the data is written directly to a clean zram-partition without any filesystem. Between writing and reading 'sync' and 'echo 3 > /proc/sys/vm/drop_caches' is called. All time measurements are wall clock times, and the benchmarks are using only one cpu-core at a time. The new algorithm is compared to all available compression algorithms from the crypto-api. Before loading the datasets to user-space deduplication is applied, since none Algorithm has deduplication. Duplicated pages are removed to prevent an algorithm to obtain high/low ratios, just because a single page can be compressed very well - or not. All Algorithms marked with '*' are using unsafe decompression. All Read and Write Speed Measurements are given in MBit/s zbewalgo' uses per dataset specialized different combinations. These can be specified at runtime via /sys/kernel/zbewalgo/combinations. - '/dev/zero' This dataset is used to measure the speed limitations for ZRAM. ZRAM filters zero-data internally and does not even call the specified compression algorithm. Algorithm write read --zram-- 2724.08 2828.87 - 'ecoham' This dataset is one of the input files for the scientific application ECOHAM which runs an ocean simulation. This dataset contains a lot of zeros - even after deduplication. Where the data is not zero there are arrays of floating point values, adjacent float values are likely to be similar to each other, allowing for high compression ratios. zbewalgo reaches very high compression ratios and is a lot faster than other algorithms with similar compression ratios. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lz4*_10 6.73 1303.12 1547.17 lz4_10 6.73 1303.12 1574.51 lzo 6.88 1205.98 1468.09 lz4*_05 7.00 1291.81 1642.41 lz4_05 7.00 1291.81 1682.81 lz4_07 7.13 1250.29 1593.89 lz4*_07 7.13 1250.29 1677.08 lz4_06 7.16 1307.62 1666.66 lz4*_06 7.16 1307.62 1669.42 lz4_03 7.21 1250.87 1449.48 lz4*_03 7.21 1250.87 1621.97 lz4*_04 7.23 1281.62 1645.56 lz4_04 7.23 1281.62 1666.81 lz4_02 7.33 1267.54 1523.11 lz4*_02 7.33 1267.54 1576.54 lz4_09 7.36 1140.55 1510.01 lz4*_09 7.36 1140.55 1692.38 lz4*_01 7.36 1215.40 1575.38 lz4_01 7.36 1215.40 1676.65 lz4_08 7.36 1242.73 1544.07 lz4*_08 7.36 1242.73 1692.92 lz4hc_01 7.51 235.85 1545.61 lz4hc*_01 7.51 235.85 1678.00 lz4hc_02 7.62 226.30 1697.42 lz4hc*_02 7.62 226.30 1738.79 lz4hc*_03 7.71 194.64 1711.58 lz4hc_03 7.71 194.64 1713.59 lz4hc*_04 7.76 177.17 1642.39 lz4hc_04 7.76 177.17 1698.36 deflate_1 7.80 84.71 584.89 lz4hc*_05 7.81 149.11 1558.43 lz4hc_05 7.81 149.11 1686.71 deflate_2 7.82 82.83 599.38 deflate_3 7.86 84.27 616.05 lz4hc_06 7.88 106.61 1680.52 lz4hc*_06 7.88 106.61 1739.78 zstd_07 7.92 230.34 1016.91 zstd_05 7.92 252.71 1070.46 zstd_06 7.93 237.84 1062.11 lz4hc*_07 7.94 75.22 1751.91 lz4hc_07 7.94 75.22 1768.98 zstd_04 7.94 403.21 1080.62 zstd_03 7.94 411.91 1077.26 zstd_01 7.94 455.89 1082.54 zstd_09 7.94 456.81 1079.22 zstd_08 7.94 459.54 1082.07 zstd_02 7.94 465.82 1056.67 zstd_11 7.95 150.15 1070.31 zstd_10 7.95 169.95 1107.86 lz4hc_08 7.98 49.53 1611.61 lz4hc*_08 7.98 49.53 1793.68 lz4hc_09 7.98 49.62 1629.63 lz4hc*_09 7.98 49.62 1639.83 lz4hc*_10 7.99 37.96 1742.65 lz4hc_10 7.99 37.96 1790.08 zbewalgo 8.02 38.58 237.92 zbewalgo* 8.02 38.58 239.10 842 8.05 169.90 597.01 zstd_13 8.06 129.78 1131.66 zstd_12 8.06 135.50 1126.59 deflate_4 8.16 71.14 546.52 deflate_5 8.17 70.86 537.05 zstd_17 8.19 61.46 1061.45 zstd_14 8.20 124.43 1133.68 zstd_18 8.21 56.82 1151.25 zstd_19 8.22 51.51 1161.83 zstd_20 8.24 44.26 1108.36 zstd_16 8.25 76.26 1042.82 zstd_15 8.25 86.65 1181.98 deflate_6 8.28 66.45 619.62 deflate_7 8.30 63.83 631.13 zstd_21 8.41 6.73 1177.38 zstd_22 8.46 2.23 1188.39 deflate_9 8.47 44.16 678.43 deflate_8 8.47 48.00 677.50 zbewalgo' 8.80 634.68 1247.56 zbewalgo'* 8.80 634.68 1429.42 - 'source-code' This dataset is a tarball of the source-code from a linux-kernel. zBeWalgo is very bad in compressing text based datasets. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lz4_10 1.49 584.41 1200.01 lz4*_10 1.49 584.41 1251.79 lz4*_07 1.64 559.05 1160.75 lz4_07 1.64 559.05 1160.97 842 1.65 63.66 158.53 lz4_06 1.71 513.03 1068.18 lz4*_06 1.71 513.03 1162.68 lz4_05 1.78 526.31 1136.51 lz4*_05 1.78 526.31 1144.81 lz4*_04 1.87 506.63 1106.31 lz4_04 1.87 506.63 1132.96 zbewalgo 1.89 27.56 35.04 zbewalgo* 1.89 27.56 36.20 zbewalgo' 1.89 46.62 34.75 zbewalgo'* 1.89 46.62 36.34 lz4_03 1.98 485.91 984.92 lz4*_03 1.98 485.91 1125.68 lz4_02 2.07 454.96 1061.05 lz4*_02 2.07 454.96 1133.42 lz4_01 2.17 441.11 1141.52 lz4*_01 2.17 441.11 1146.26 lz4*_08 2.17 446.45 1103.61 lz4_08 2.17 446.45 1163.91 lz4*_09 2.17 453.21 1071.91 lz4_09 2.17 453.21 1155.43 lzo 2.27 430.27 871.87 lz4hc*_01 2.35 137.71 1089.94 lz4hc_01 2.35 137.71 1200.45 lz4hc_02 2.38 139.18 1117.44 lz4hc*_02 2.38 139.18 1210.58 lz4hc_03 2.39 127.09 1097.90 lz4hc*_03 2.39 127.09 1214.22 lz4hc_10 2.40 96.26 1203.89 lz4hc*_10 2.40 96.26 1221.94 lz4hc*_08 2.40 98.80 1191.79 lz4hc_08 2.40 98.80 1226.59 lz4hc*_09 2.40 102.36 1213.34 lz4hc_09 2.40 102.36 1225.45 lz4hc*_07 2.40 113.81 1217.63 lz4hc_07 2.40 113.81 1218.49 lz4hc*_06 2.40 117.32 1214.13 lz4hc_06 2.40 117.32 1224.51 lz4hc_05 2.40 122.12 1108.34 lz4hc*_05 2.40 122.12 1214.97 lz4hc*_04 2.40 124.91 1093.58 lz4hc_04 2.40 124.91 1222.05 zstd_01 2.93 200.01 401.15 zstd_08 2.93 200.01 414.52 zstd_09 2.93 200.26 394.83 zstd_02 3.00 201.12 405.73 deflate_1 3.01 53.83 240.64 deflate_2 3.05 52.58 243.31 deflate_3 3.08 52.07 244.84 zstd_04 3.10 158.80 365.06 zstd_03 3.10 169.56 405.92 zstd_05 3.18 125.00 410.23 zstd_06 3.20 106.50 404.81 zstd_07 3.21 99.02 404.23 zstd_15 3.22 24.95 376.58 zstd_16 3.22 26.88 416.44 deflate_4 3.22 45.26 225.56 zstd_13 3.22 62.53 388.33 zstd_14 3.22 64.15 391.81 zstd_12 3.22 66.24 417.67 zstd_11 3.22 66.44 404.31 zstd_10 3.22 73.13 401.98 zstd_17 3.24 14.66 412.00 zstd_18 3.25 13.37 408.46 deflate_5 3.26 43.54 252.18 deflate_7 3.27 39.37 245.63 deflate_6 3.27 42.51 251.33 deflate_9 3.28 40.02 253.99 deflate_8 3.28 40.10 253.98 zstd_19 3.34 10.36 399.85 zstd_22 3.35 4.88 353.63 zstd_21 3.35 6.02 323.33 zstd_20 3.35 8.34 339.81 - 'hpcg' This dataset is a (partial) memory-snapshot of the running hpcg-benchmark. At the time of the snapshot, that application performed a sparse matrix - vector multiplication. The compression ratio of zBeWalgo on this dataset is nearly 3 times higher than the ratio of any other algorithm regardless of the compression-level specified. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lz4*_10 1.00 1130.73 2131.82 lz4_10 1.00 1130.73 2181.60 lz4_06 1.34 625.48 1145.74 lz4*_06 1.34 625.48 1145.90 lz4_07 1.57 515.39 895.42 lz4*_07 1.57 515.39 1062.53 lz4*_05 1.72 539.40 1030.76 lz4_05 1.72 539.40 1038.86 lzo 1.76 475.20 805.41 lz4_08 1.76 480.35 939.16 lz4*_08 1.76 480.35 1015.04 lz4*_03 1.76 488.05 893.13 lz4_03 1.76 488.05 1013.65 lz4*_09 1.76 501.49 1032.69 lz4_09 1.76 501.49 1105.47 lz4*_01 1.76 501.54 1040.72 lz4_01 1.76 501.54 1102.22 lz4*_02 1.76 510.79 1014.78 lz4_02 1.76 510.79 1080.69 lz4_04 1.76 516.18 1047.06 lz4*_04 1.76 516.18 1049.55 842 2.35 109.68 192.50 lz4hc_07 2.36 152.57 1265.77 lz4hc*_07 2.36 152.57 1331.01 lz4hc*_06 2.36 155.78 1313.85 lz4hc_06 2.36 155.78 1346.52 lz4hc*_08 2.36 158.80 1297.16 lz4hc_08 2.36 158.80 1382.54 lz4hc*_10 2.36 159.84 1317.81 lz4hc_10 2.36 159.84 1346.85 lz4hc*_03 2.36 160.01 1162.91 lz4hc_03 2.36 160.01 1377.09 lz4hc*_09 2.36 161.02 1320.87 lz4hc_09 2.36 161.02 1374.39 lz4hc*_05 2.36 164.67 1324.40 lz4hc_05 2.36 164.67 1341.64 lz4hc*_04 2.36 168.11 1323.19 lz4hc_04 2.36 168.11 1377.56 lz4hc_01 2.36 168.40 1231.55 lz4hc*_01 2.36 168.40 1329.72 lz4hc*_02 2.36 170.74 1316.54 lz4hc_02 2.36 170.74 1337.42 deflate_3 3.52 46.51 336.67 deflate_2 3.52 62.05 343.03 deflate_1 3.52 65.68 359.96 deflate_4 4.01 61.01 432.66 deflate_8 4.61 41.51 408.29 deflate_5 4.61 44.09 434.79 deflate_9 4.61 45.14 417.18 deflate_7 4.61 45.22 440.27 deflate_6 4.61 46.01 440.39 zstd_09 5.95 277.11 542.93 zstd_08 5.95 277.40 541.27 zstd_01 5.95 277.41 540.61 zstd_16 5.97 32.05 465.03 zstd_15 5.97 39.12 515.07 zstd_13 5.97 70.90 511.94 zstd_14 5.97 72.20 522.68 zstd_11 5.97 74.14 512.18 zstd_12 5.97 74.27 497.95 zstd_10 5.97 86.98 519.78 zstd_07 5.97 135.16 504.07 zstd_06 5.97 145.49 505.10 zstd_05 6.02 177.86 510.08 zstd_04 6.02 205.13 516.29 zstd_03 6.02 217.82 515.50 zstd_02 6.02 260.97 484.64 zstd_18 6.27 12.10 490.72 zstd_17 6.27 12.33 462.65 zstd_21 6.70 9.25 391.16 zstd_20 6.70 9.50 395.38 zstd_22 6.70 9.74 390.99 zstd_19 6.70 9.99 450.42 zbewalgo 16.33 47.17 430.06 zbewalgo* 16.33 47.17 436.92 zbewalgo' 16.33 188.86 427.78 zbewalgo'* 16.33 188.86 437.43 - 'partdiff' (8 GiB) Array of double values. Adjacent doubles are similar, but not equal. This array is produced by a partial differential equation solver using a Jakobi-implementation. zBewalgo gains higher compression ratios than all other algorithms. Some algorithms are even slower than a hdd without any compression at all. Algorithm ratio write read zstd_18 1.00 13.77 2080.06 zstd_17 1.00 13.80 2075.23 zstd_16 1.00 28.04 2138.99 zstd_15 1.00 45.04 2143.32 zstd_13 1.00 55.72 2128.27 zstd_14 1.00 56.09 2123.54 zstd_11 1.00 57.31 2095.04 zstd_12 1.00 57.53 2134.61 842 1.00 61.61 2267.89 zstd_10 1.00 80.40 2081.35 zstd_07 1.00 120.66 2119.09 zstd_06 1.00 128.80 2134.02 zstd_05 1.00 131.25 2133.01 --hdd-- 1.00 134.70 156.62 lz4hc*_03 1.00 152.82 1982.94 lz4hc_03 1.00 152.82 2261.55 lz4hc*_07 1.00 159.43 1990.03 lz4hc_07 1.00 159.43 2269.05 lz4hc_10 1.00 166.33 2243.78 lz4hc*_10 1.00 166.33 2260.63 lz4hc_09 1.00 167.03 2244.20 lz4hc*_09 1.00 167.03 2264.72 lz4hc*_06 1.00 167.17 2245.15 lz4hc_06 1.00 167.17 2271.88 lz4hc_08 1.00 167.49 2237.79 lz4hc*_08 1.00 167.49 2283.98 lz4hc_02 1.00 167.51 2275.36 lz4hc*_02 1.00 167.51 2279.72 lz4hc*_05 1.00 167.52 2248.92 lz4hc_05 1.00 167.52 2273.99 lz4hc*_04 1.00 167.71 2268.23 lz4hc_04 1.00 167.71 2268.78 lz4hc*_01 1.00 167.91 2268.76 lz4hc_01 1.00 167.91 2269.16 zstd_04 1.00 175.84 2241.60 zstd_03 1.00 176.35 2285.13 zstd_02 1.00 195.41 2269.51 zstd_09 1.00 199.47 2271.91 zstd_01 1.00 199.74 2287.15 zstd_08 1.00 199.87 2286.27 lz4_01 1.00 1160.95 2257.78 lz4*_01 1.00 1160.95 2275.42 lz4_08 1.00 1164.37 2280.06 lz4*_08 1.00 1164.37 2280.43 lz4*_09 1.00 1166.30 2263.05 lz4_09 1.00 1166.30 2280.54 lz4*_03 1.00 1174.00 2074.96 lz4_03 1.00 1174.00 2257.37 lz4_02 1.00 1212.18 2273.60 lz4*_02 1.00 1212.18 2285.66 lz4*_04 1.00 1253.55 2259.60 lz4_04 1.00 1253.55 2287.15 lz4_05 1.00 1279.88 2282.47 lz4*_05 1.00 1279.88 2287.05 lz4_06 1.00 1292.22 2277.95 lz4*_06 1.00 1292.22 2284.84 lz4*_07 1.00 1303.58 2276.10 lz4_07 1.00 1303.58 2276.99 lz4*_10 1.00 1304.80 2183.30 lz4_10 1.00 1304.80 2285.25 lzo 1.00 1360.88 2281.19 deflate_7 1.07 33.51 463.73 deflate_2 1.07 33.99 473.07 deflate_9 1.07 34.05 473.57 deflate_6 1.07 34.06 473.69 deflate_8 1.07 34.12 472.86 deflate_5 1.07 34.22 468.03 deflate_4 1.07 34.32 447.33 deflate_1 1.07 35.45 431.95 deflate_3 1.07 35.63 472.56 zstd_22 1.11 9.81 668.64 zstd_21 1.11 10.71 734.52 zstd_20 1.11 10.78 714.86 zstd_19 1.11 12.02 790.71 zbewalgo 1.29 25.93 225.07 zbewalgo* 1.29 25.93 226.72 zbewalgo'* 1.31 23.54 84.29 zbewalgo' 1.31 23.54 86.08 - 'Isabella CLOUDf01' This dataset is an array of floating point values between 0.00000 and 0.00332. Detailed Information about this dataset is online available at http://www.vets.ucar.edu/vg/isabeldata/readme.html All algorithms obtain similar compression ratios. The compression ratio of zBeWalgo is slightly higher, and the speed is higher too. Algorithm ratio write read --hdd-- 1.00 134.70 156.62 lzo 2.06 1022.09 916.22 lz4*_10 2.09 1126.03 1533.35 lz4_10 2.09 1126.03 1569.06 lz4*_07 2.09 1135.89 1444.21 lz4_07 2.09 1135.89 1581.96 lz4*_01 2.10 972.22 1405.21 lz4_01 2.10 972.22 1579.78 lz4*_09 2.10 982.39 1429.17 lz4_09 2.10 982.39 1490.27 lz4_08 2.10 1006.56 1491.14 lz4*_08 2.10 1006.56 1558.66 lz4_02 2.10 1019.82 1366.16 lz4*_02 2.10 1019.82 1578.79 lz4_03 2.10 1129.74 1417.33 lz4*_03 2.10 1129.74 1456.68 lz4_04 2.10 1131.28 1478.27 lz4*_04 2.10 1131.28 1517.84 lz4_06 2.10 1147.78 1424.90 lz4*_06 2.10 1147.78 1462.47 lz4*_05 2.10 1172.44 1434.86 lz4_05 2.10 1172.44 1578.80 lz4hc*_10 2.11 29.01 1498.01 lz4hc_10 2.11 29.01 1580.23 lz4hc*_09 2.11 56.30 1510.26 lz4hc_09 2.11 56.30 1583.11 lz4hc_08 2.11 56.39 1426.43 lz4hc*_08 2.11 56.39 1565.12 lz4hc_07 2.11 129.27 1540.38 lz4hc*_07 2.11 129.27 1578.35 lz4hc*_06 2.11 162.72 1456.27 lz4hc_06 2.11 162.72 1581.69 lz4hc*_05 2.11 183.78 1487.71 lz4hc_05 2.11 183.78 1589.10 lz4hc*_04 2.11 187.41 1431.35 lz4hc_04 2.11 187.41 1566.24 lz4hc*_03 2.11 190.21 1531.98 lz4hc_03 2.11 190.21 1580.81 lz4hc*_02 2.11 199.69 1432.00 lz4hc_02 2.11 199.69 1565.10 lz4hc_01 2.11 205.87 1540.33 lz4hc*_01 2.11 205.87 1567.68 842 2.15 89.89 414.49 deflate_1 2.29 48.84 352.09 deflate_2 2.29 49.47 353.77 deflate_3 2.30 50.00 345.88 zstd_22 2.31 5.59 658.59 zstd_21 2.31 14.34 664.02 zstd_20 2.31 21.22 665.77 zstd_19 2.31 24.26 587.99 zstd_17 2.31 26.24 670.14 zstd_18 2.31 26.47 668.64 deflate_9 2.31 33.79 345.81 deflate_8 2.31 34.67 347.96 deflate_4 2.31 41.46 326.50 deflate_7 2.31 42.56 346.99 deflate_6 2.31 43.51 343.56 deflate_5 2.31 45.83 343.86 zstd_05 2.31 126.01 571.70 zstd_04 2.31 178.39 597.26 zstd_03 2.31 192.04 644.24 zstd_01 2.31 206.31 563.68 zstd_08 2.31 207.39 669.05 zstd_02 2.31 216.98 600.77 zstd_09 2.31 236.92 667.64 zstd_16 2.32 41.47 660.06 zstd_15 2.32 60.37 584.45 zstd_14 2.32 74.60 673.10 zstd_12 2.32 75.16 661.96 zstd_13 2.32 75.22 676.12 zstd_11 2.32 75.58 636.75 zstd_10 2.32 95.05 645.07 zstd_07 2.32 139.52 672.88 zstd_06 2.32 145.40 670.45 zbewalgo'* 2.37 337.07 463.32 zbewalgo' 2.37 337.07 468.96 zbewalgo* 2.60 101.17 578.35 zbewalgo 2.60 101.17 586.88 - 'Isabella TCf01' This dataset is an array of floating point values between -83.00402 and 31.51576. Detailed Information about this dataset is online available at http://www.vets.ucar.edu/vg/isabeldata/readme.html zBeWalgo is the only algorithm which can compress this dataset with a noticeable compressionratio. Algorithm ratio write read 842 1.00 60.09 1956.26 --hdd-- 1.00 134.70 156.62 lz4hc_01 1.00 154.81 1839.37 lz4hc*_01 1.00 154.81 2105.53 lz4hc_10 1.00 157.33 2078.69 lz4hc*_10 1.00 157.33 2113.14 lz4hc_09 1.00 158.50 2018.51 lz4hc*_09 1.00 158.50 2093.65 lz4hc*_02 1.00 159.54 2104.91 lz4hc_02 1.00 159.54 2117.34 lz4hc_03 1.00 161.26 2070.76 lz4hc*_03 1.00 161.26 2107.27 lz4hc*_08 1.00 161.34 2100.74 lz4hc_08 1.00 161.34 2105.26 lz4hc*_04 1.00 161.95 2080.96 lz4hc_04 1.00 161.95 2104.00 lz4hc_05 1.00 162.17 2044.43 lz4hc*_05 1.00 162.17 2101.74 lz4hc*_06 1.00 163.61 2087.19 lz4hc_06 1.00 163.61 2104.61 lz4hc_07 1.00 164.51 2094.78 lz4hc*_07 1.00 164.51 2105.53 lz4_01 1.00 1134.89 2109.70 lz4*_01 1.00 1134.89 2118.71 lz4*_08 1.00 1141.96 2104.87 lz4_08 1.00 1141.96 2118.97 lz4_09 1.00 1145.55 2087.76 lz4*_09 1.00 1145.55 2118.85 lz4_02 1.00 1157.28 2094.33 lz4*_02 1.00 1157.28 2124.67 lz4*_03 1.00 1194.18 2106.36 lz4_03 1.00 1194.18 2119.89 lz4_04 1.00 1195.09 2117.03 lz4*_04 1.00 1195.09 2120.23 lz4*_05 1.00 1225.56 2109.04 lz4_05 1.00 1225.56 2120.52 lz4*_06 1.00 1261.67 2109.14 lz4_06 1.00 1261.67 2121.13 lz4*_07 1.00 1270.86 1844.63 lz4_07 1.00 1270.86 2041.08 lz4_10 1.00 1305.36 2109.22 lz4*_10 1.00 1305.36 2120.65 lzo 1.00 1338.61 2109.66 zstd_17 1.03 13.93 1138.94 zstd_18 1.03 14.01 1170.78 zstd_16 1.03 27.12 1073.75 zstd_15 1.03 43.52 1061.97 zstd_14 1.03 49.60 1082.98 zstd_12 1.03 55.03 1042.43 zstd_13 1.03 55.14 1173.50 zstd_11 1.03 55.24 1178.05 zstd_10 1.03 70.01 1173.05 zstd_07 1.03 118.10 1041.92 zstd_06 1.03 123.00 1171.59 zstd_05 1.03 124.61 1165.74 zstd_01 1.03 166.80 1005.29 zstd_04 1.03 170.25 1127.75 zstd_03 1.03 171.40 1172.34 zstd_02 1.03 174.08 1017.34 zstd_09 1.03 195.30 1176.82 zstd_08 1.03 195.98 1175.09 deflate_9 1.05 30.15 483.55 deflate_8 1.05 30.45 466.67 deflate_5 1.05 31.25 480.92 deflate_4 1.05 31.84 472.81 deflate_7 1.05 31.84 484.18 deflate_6 1.05 31.94 481.37 deflate_2 1.05 33.07 484.09 deflate_3 1.05 33.11 463.57 deflate_1 1.05 33.19 469.71 zstd_22 1.06 8.89 647.75 zstd_21 1.06 10.70 700.11 zstd_20 1.06 10.80 723.42 zstd_19 1.06 12.41 764.24 zbewalgo* 1.51 146.45 581.43 zbewalgo 1.51 146.45 592.86 zbewalgo'* 1.54 38.14 120.96 zbewalgo' 1.54 38.14 125.81 Signed-off-by: Benjamin Warnke <4bwarnke@informatik.uni-hamburg.de> --- -- 2.14.1