HCTR2 is a length-preserving encryption mode that is efficient on
processors with instructions to accelerate AES and carryless
multiplication, e.g. x86 processors with AES-NI and CLMUL, and ARM
processors with the ARMv8 Crypto Extensions.
HCTR2 is specified in https://ia.cr/2021/1441 "Length-preserving
encryption with HCTR2" which shows that if AES is secure and HCTR2 is
instantiated with AES, then HCTR2 is secure. Reference code and test
vectors are at https://github.com/google/hctr2.
As a length-preserving encryption mode, HCTR2 is suitable for applications
such as storage encryption where ciphertext expansion is not possible, and
thus authenticated encryption cannot be used. Currently, such
applications usually use XTS, or in some cases Adiantum. XTS has the
disadvantage that it is a narrow-block mode: a bitflip will only change 16
bytes in the resulting ciphertext or plaintext. This reveals more
information to an attacker than necessary.
HCTR2 is a wide-block mode, so it provides a stronger security property: a
bitflip will change the entire message. HCTR2 is somewhat similar to
Adiantum, which is also a wide-block mode. However, HCTR2 is designed to
take advantage of existing crypto instructions, while Adiantum targets
devices without such hardware support. Adiantum is also designed with
longer messages in mind, while HCTR2 is designed to be efficient even on
short messages.
The first intended use of this mode in the kernel is for the encryption of
filenames, where for efficiency reasons encryption must be fully
deterministic (only one ciphertext for each plaintext) and the existing
CBC solution leaks more information than necessary for filenames with
common prefixes.
HCTR2 uses two passes of an ε-almost-∆-universal hash function called
POLYVAL and one pass of a block cipher mode called XCTR. POLYVAL is a
polynomial hash designed for efficiency on modern processors and was
originally specified for use in AES-GCM-SIV (RFC 8452). XCTR mode is a
variant of CTR mode that is more efficient on little-endian machines.
This patchset adds HCTR2 to Linux's crypto API, including generic
implementations of XCTR and POLYVAL, hardware accelerated implementations of
XCTR and POLYVAL for both x86-64 and ARM64, a templated implementation of HCTR2,
and an fscrypt policy for using HCTR2 for filename encryption.
Changes in v3:
* Improve testvec coverage for XCTR, POLYVAL and HCTR2
* Fix endianness bug in xctr.c
* Fix alignment issues in polyval-generic.c
* Optimize hctr2.c by exporting/importing hash states
* Fix blockcipher name derivation in hctr2.c
* Move x86-64 XCTR implementation into aes_ctrby8_avx-x86_64.S
* Reuse ARM64 CTR mode tail handling in ARM64 XCTR
* Fix x86-64 POLYVAL comments
* Fix x86-64 POLYVAL key_powers type to match asm
* Fix ARM64 POLYVAL comments
* Fix ARM64 POLYVAL key_powers type to match asm
* Add XTS + HCTR2 policy to fscrypt
Nathan Huckleberry (8):
crypto: xctr - Add XCTR support
crypto: polyval - Add POLYVAL support
crypto: hctr2 - Add HCTR2 support
crypto: x86/aesni-xctr: Add accelerated implementation of XCTR
crypto: arm64/aes-xctr: Add accelerated implementation of XCTR
crypto: x86/polyval: Add PCLMULQDQ accelerated implementation of
POLYVAL
crypto: arm64/polyval: Add PMULL accelerated implementation of POLYVAL
fscrypt: Add HCTR2 support for filename encryption
Documentation/filesystems/fscrypt.rst | 19 +-
arch/arm64/crypto/Kconfig | 11 +-
arch/arm64/crypto/Makefile | 3 +
arch/arm64/crypto/aes-glue.c | 65 +-
arch/arm64/crypto/aes-modes.S | 134 ++
arch/arm64/crypto/polyval-ce-core.S | 372 ++++++
arch/arm64/crypto/polyval-ce-glue.c | 363 ++++++
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 233 ++--
arch/x86/crypto/aesni-intel_asm.S | 70 ++
arch/x86/crypto/aesni-intel_glue.c | 89 ++
arch/x86/crypto/polyval-clmulni_asm.S | 376 ++++++
arch/x86/crypto/polyval-clmulni_glue.c | 361 ++++++
crypto/Kconfig | 40 +-
crypto/Makefile | 3 +
crypto/hctr2.c | 580 +++++++++
crypto/polyval-generic.c | 205 +++
crypto/tcrypt.c | 10 +
crypto/testmgr.c | 20 +
crypto/testmgr.h | 1536 +++++++++++++++++++++++
crypto/xctr.c | 193 +++
fs/crypto/fscrypt_private.h | 2 +-
fs/crypto/keysetup.c | 7 +
fs/crypto/policy.c | 4 +
include/crypto/polyval.h | 17 +
include/uapi/linux/fscrypt.h | 3 +-
tools/include/uapi/linux/fscrypt.h | 3 +-
27 files changed, 4633 insertions(+), 89 deletions(-)
create mode 100644 arch/arm64/crypto/polyval-ce-core.S
create mode 100644 arch/arm64/crypto/polyval-ce-glue.c
create mode 100644 arch/x86/crypto/polyval-clmulni_asm.S
create mode 100644 arch/x86/crypto/polyval-clmulni_glue.c
create mode 100644 crypto/hctr2.c
create mode 100644 crypto/polyval-generic.c
create mode 100644 crypto/xctr.c
create mode 100644 include/crypto/polyval.h
--
2.35.1.723.g4982287a31-goog
Add a generic implementation of XCTR mode as a template. XCTR is a
blockcipher mode similar to CTR mode. XCTR uses XORs and little-endian
addition rather than big-endian arithmetic which has two advantages: It
is slightly faster on little-endian CPUs and it is less likely to be
implemented incorrect since integer overflows are not possible on
practical input sizes. XCTR is used as a component to implement HCTR2.
More information on XCTR mode can be found in the HCTR2 paper:
https://eprint.iacr.org/2021/1441.pdf
Signed-off-by: Nathan Huckleberry <[email protected]>
---
crypto/Kconfig | 9 +
crypto/Makefile | 1 +
crypto/tcrypt.c | 1 +
crypto/testmgr.c | 6 +
crypto/testmgr.h | 693 +++++++++++++++++++++++++++++++++++++++++++++++
crypto/xctr.c | 193 +++++++++++++
6 files changed, 903 insertions(+)
create mode 100644 crypto/xctr.c
diff --git a/crypto/Kconfig b/crypto/Kconfig
index d6d7e84bb7f8..47752aaa16ff 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -460,6 +460,15 @@ config CRYPTO_PCBC
PCBC: Propagating Cipher Block Chaining mode
This block cipher algorithm is required for RxRPC.
+config CRYPTO_XCTR
+ tristate
+ select CRYPTO_SKCIPHER
+ select CRYPTO_MANAGER
+ help
+ XCTR: XOR Counter mode. This blockcipher mode is a variant of CTR mode
+ using XORs and little-endian addition rather than big-endian arithmetic.
+ XCTR mode is used to implement HCTR2.
+
config CRYPTO_XTS
tristate "XTS support"
select CRYPTO_SKCIPHER
diff --git a/crypto/Makefile b/crypto/Makefile
index d76bff8d0ffd..6b3fe3df1489 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -93,6 +93,7 @@ obj-$(CONFIG_CRYPTO_CTS) += cts.o
obj-$(CONFIG_CRYPTO_LRW) += lrw.o
obj-$(CONFIG_CRYPTO_XTS) += xts.o
obj-$(CONFIG_CRYPTO_CTR) += ctr.o
+obj-$(CONFIG_CRYPTO_XCTR) += xctr.o
obj-$(CONFIG_CRYPTO_KEYWRAP) += keywrap.o
obj-$(CONFIG_CRYPTO_ADIANTUM) += adiantum.o
obj-$(CONFIG_CRYPTO_NHPOLY1305) += nhpoly1305.o
diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c
index 2bacf8384f59..fd671d0e2012 100644
--- a/crypto/tcrypt.c
+++ b/crypto/tcrypt.c
@@ -1556,6 +1556,7 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
ret += tcrypt_test("rfc3686(ctr(aes))");
ret += tcrypt_test("ofb(aes)");
ret += tcrypt_test("cfb(aes)");
+ ret += tcrypt_test("xctr(aes)");
break;
case 11:
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 2d632a285869..fbb12d7d78af 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -5490,6 +5490,12 @@ static const struct alg_test_desc alg_test_descs[] = {
.suite = {
.cipher = __VECS(xchacha20_tv_template)
},
+ }, {
+ .alg = "xctr(aes)",
+ .test = alg_test_skcipher,
+ .suite = {
+ .cipher = __VECS(aes_xctr_tv_template)
+ }
}, {
.alg = "xts(aes)",
.generic_driver = "xts(ecb(aes-generic))",
diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index d1aa90993bbd..bf4ff97eeb37 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -34236,4 +34236,697 @@ static const struct hash_testvec blakes2s_256_tv_template[] = {{
0xd5, 0x06, 0xb5, 0x3a, 0x7c, 0x7a, 0x65, 0x1d, },
}};
+/*
+ * Test vectors generated using https://github.com/google/hctr2
+ */
+static const struct cipher_testvec aes_xctr_tv_template[] = {
+ {
+ .key = "\x9c\x8d\xc4\xbd\x71\x36\xdc\x82"
+ "\x7c\xa1\xca\xa3\x23\x5a\xdb\xa4",
+ .iv = "\x8d\xe7\xa5\x6a\x95\x86\x42\xde"
+ "\xba\xea\x6e\x69\x03\x33\x86\x0f",
+ .ptext = "\xbd",
+ .ctext = "\xb9",
+ .klen = 16,
+ .len = 1,
+ },
+ {
+ .key = "\xbc\x1b\x12\x0c\x3f\x18\xcc\x1f"
+ "\x5a\x1d\xab\x81\xa8\x68\x7c\x63",
+ .iv = "\x22\xc1\xdd\x25\x0b\x18\xcb\xa5"
+ "\x4a\xda\x15\x07\x73\xd9\x88\x10",
+ .ptext = "\x24\x6e\x64\xc6\x15\x26\x9c\xda"
+ "\x2a\x4b\x57\x12\xff\x7c\xd6\xb5",
+ .ctext = "\xd6\x47\x8d\x58\x92\xb2\x84\xf9"
+ "\xb7\xee\x0d\x98\xa1\x39\x4d\x8f",
+ .klen = 16,
+ .len = 16,
+ },
+ {
+ .key = "\x44\x03\xbf\x4c\x30\xf0\xa7\xd6"
+ "\xbd\x54\xbb\x66\x8e\xa6\x0e\x8a",
+ .iv = "\xe6\xf7\x26\xdf\x8c\x3c\xaa\x88"
+ "\xce\xc1\xbd\x43\x3b\x09\x62\xad",
+ .ptext = "\x3c\xe3\x46\xb9\x8f\x9d\x3f\x8d"
+ "\xef\xf2\x53\xab\x24\xe2\x29\x08"
+ "\xf8\x7e\x1d\xa6\x6d\x86\x7d\x60"
+ "\x97\x63\x93\x29\x71\x94\xb4",
+ .ctext = "\xd4\xa3\xc6\xb8\xc1\x6f\x70\x1a"
+ "\x52\x0c\xed\x4c\xaf\x51\x56\x23"
+ "\x48\x45\x07\x10\x34\xc5\xba\x71"
+ "\xe5\xf8\x1e\xd8\xcb\xa6\xe7",
+ .klen = 16,
+ .len = 31,
+ },
+ {
+ .key = "\x5b\x17\x30\x94\x19\x31\xa1\xae"
+ "\x24\x8e\x42\x1e\x82\xe6\xec\xb8",
+ .iv = "\xd1\x2e\xb9\xb8\xf8\x49\xeb\x68"
+ "\x06\xeb\x65\x33\x34\xa2\xeb\xf0",
+ .ptext = "\x19\x75\xec\x59\x60\x1b\x7a\x3e"
+ "\x62\x46\x87\xf0\xde\xab\x81\x36"
+ "\x63\x53\x11\xa0\x1f\xce\x25\x85"
+ "\x49\x6b\x28\xfa\x1c\x92\xe5\x18"
+ "\x38\x14\x00\x79\xf2\x9e\xeb\xfc"
+ "\x36\xa7\x6b\xe1\xe5\xcf\x04\x48"
+ "\x44\x6d\xbd\x64\xb3\xcb\x78\x05"
+ "\x8d\x7f\x9a\xaf\x3c\xcf\x6c\x45"
+ "\x6c\x7c\x46\x4c\xa8\xc0\x1e\xe4"
+ "\x33\xa5\x7b\xbb\x26\xd9\xc0\x32"
+ "\x9d\x8a\xb3\xf3\x3d\x52\xe6\x48"
+ "\x4c\x9b\x4c\x6e\xa4\xa3\xad\x66"
+ "\x56\x48\xd5\x98\x3a\x93\xc4\x85"
+ "\xe9\x89\xca\xa6\xc1\xc8\xe7\xf8"
+ "\xc3\xe9\xef\xbe\x77\xe6\xd1\x3a"
+ "\xa6\x99\xc8\x2d\xdf\x40\x0f\x44",
+ .ctext = "\xc6\x1a\x01\x1a\x00\xba\x04\xff"
+ "\x10\xd1\x7e\x5d\xad\x91\xde\x8c"
+ "\x08\x55\x95\xae\xd7\x22\x77\x40"
+ "\xf0\x33\x1b\x51\xef\xfe\x3d\x67"
+ "\xdf\xc4\x9f\x39\x47\x67\x93\xab"
+ "\xaa\x37\x55\xfe\x41\xe0\xba\xcd"
+ "\x25\x02\x7c\x61\x51\xa1\xcc\x72"
+ "\x7a\x20\x26\xb9\x06\x68\xbd\x19"
+ "\xc5\x2e\x1b\x75\x4a\x40\xb2\xd2"
+ "\xc4\xee\xd8\x5b\xa4\x55\x7d\x25"
+ "\xfc\x01\x4d\x6f\x0a\xfd\x37\x5d"
+ "\x3e\x67\xc0\x35\x72\x53\x7b\xe2"
+ "\xd6\x19\x5b\x92\x6c\x3a\x8c\x2a"
+ "\xe2\xc2\xa2\x4f\x2a\xf2\xb5\x15"
+ "\x65\xc5\x8d\x97\xf9\xbf\x8c\x98"
+ "\xe4\x50\x1a\xf2\x76\x55\x07\x49",
+ .klen = 16,
+ .len = 128,
+ },
+ {
+ .key = "\x17\xa6\x01\x3d\x5d\xd6\xef\x2d"
+ "\x69\x8f\x4c\x54\x5b\xae\x43\xf0",
+ .iv = "\xa9\x1b\x47\x60\x26\x82\xf7\x1c"
+ "\x80\xf8\x88\xdd\xfb\x44\xd9\xda",
+ .ptext = "\xf7\x67\xcd\xa6\x04\x65\x53\x99"
+ "\x90\x5c\xa2\x56\x74\xd7\x9d\xf2"
+ "\x0b\x03\x7f\x4e\xa7\x84\x72\x2b"
+ "\xf0\xa5\xbf\xe6\x9a\x62\x3a\xfe"
+ "\x69\x5c\x93\x79\x23\x86\x64\x85"
+ "\xeb\x13\xb1\x5a\xd5\x48\x39\xa0"
+ "\x70\xfb\x06\x9a\xd7\x12\x5a\xb9"
+ "\xbe\xed\x2c\x81\x64\xf7\xcf\x80"
+ "\xee\xe6\x28\x32\x2d\x37\x4c\x32"
+ "\xf4\x1f\x23\x21\xe9\xc8\xc9\xbf"
+ "\x54\xbc\xcf\xb4\xc2\x65\x39\xdf"
+ "\xa5\xfb\x14\x11\xed\x62\x38\xcf"
+ "\x9b\x58\x11\xdd\xe9\xbd\x37\x57"
+ "\x75\x4c\x9e\xd5\x67\x0a\x48\xc6"
+ "\x0d\x05\x4e\xb1\x06\xd7\xec\x2e"
+ "\x9e\x59\xde\x4f\xab\x38\xbb\xe5"
+ "\x87\x04\x5a\x2c\x2a\xa2\x8f\x3c"
+ "\xe7\xe1\x46\xa9\x49\x9f\x24\xad"
+ "\x2d\xb0\x55\x40\x64\xd5\xda\x7e"
+ "\x1e\x77\xb8\x29\x72\x73\xc3\x84"
+ "\xcd\xf3\x94\x90\x58\x76\xc9\x2c"
+ "\x2a\xad\x56\xde\x33\x18\xb6\x3b"
+ "\x10\xe9\xe9\x8d\xf0\xa9\x7f\x05"
+ "\xf7\xb5\x8c\x13\x7e\x11\x3d\x1e"
+ "\x02\xbb\x5b\xea\x69\xff\x85\xcf"
+ "\x6a\x18\x97\x45\xe3\x96\xba\x4d"
+ "\x2d\x7a\x70\x78\x15\x2c\xe9\xdc"
+ "\x4e\x09\x92\x57\x04\xd8\x0b\xa6"
+ "\x20\x71\x76\x47\x76\x96\x89\xa0"
+ "\xd9\x29\xa2\x5a\x06\xdb\x56\x39"
+ "\x60\x33\x59\x04\x95\x89\xf6\x18"
+ "\x1d\x70\x75\x85\x3a\xb7\x6e",
+ .ctext = "\xe1\xe7\x3f\xd3\x6a\xb9\x2f\x64"
+ "\x37\xc5\xa4\xe9\xca\x0a\xa1\xd6"
+ "\xea\x7d\x39\xe5\xe6\xcc\x80\x54"
+ "\x74\x31\x2a\x04\x33\x79\x8c\x8e"
+ "\x4d\x47\x84\x28\x27\x9b\x3c\x58"
+ "\x54\x58\x20\x4f\x70\x01\x52\x5b"
+ "\xac\x95\x61\x49\x5f\xef\xba\xce"
+ "\xd7\x74\x56\xe7\xbb\xe0\x3c\xd0"
+ "\x7f\xa9\x23\x57\x33\x2a\xf6\xcb"
+ "\xbe\x42\x14\x95\xa8\xf9\x7a\x7e"
+ "\x12\x53\x3a\xe2\x13\xfe\x2d\x89"
+ "\xeb\xac\xd7\xa8\xa5\xf8\x27\xf3"
+ "\x74\x9a\x65\x63\xd1\x98\x3a\x7e"
+ "\x27\x7b\xc0\x20\x00\x4d\xf4\xe5"
+ "\x7b\x69\xa6\xa8\x06\x50\x85\xb6"
+ "\x7f\xac\x7f\xda\x1f\xf5\x37\x56"
+ "\x9b\x2f\xd3\x86\x6b\x70\xbd\x0e"
+ "\x55\x9a\x9d\x4b\x08\xb5\x5b\x7b"
+ "\xd4\x7c\xb4\x71\x49\x92\x4a\x1e"
+ "\xed\x6d\x11\x09\x47\x72\x32\x6a"
+ "\x97\x53\x36\xaf\xf3\x06\x06\x2c"
+ "\x69\xf1\x59\x00\x36\x95\x28\x2a"
+ "\xb6\xcd\x10\x21\x84\x73\x5c\x96"
+ "\x86\x14\x2c\x3d\x02\xdb\x53\x9a"
+ "\x61\xde\xea\x99\x84\x7a\x27\xf6"
+ "\xf7\xc8\x49\x73\x4b\xb8\xeb\xd3"
+ "\x41\x33\xdd\x09\x68\xe2\x64\xb8"
+ "\x5f\x75\x74\x97\x91\x54\xda\xc2"
+ "\x73\x2c\x1e\x5a\x84\x48\x01\x1a"
+ "\x0d\x8b\x0a\xdf\x07\x2e\xee\x77"
+ "\x1d\x17\x41\x7a\xc9\x33\x63\xfa"
+ "\x9f\xc3\x74\x57\x5f\x03\x4c",
+ .klen = 16,
+ .len = 255,
+ },
+ {
+ .key = "\xe5\xf1\x48\x2e\x88\xdb\xc7\x28"
+ "\xa2\x55\x5d\x2f\x90\x02\xdc\xd3"
+ "\xf5\xd3\x9e\x87\xd5\x58\x30\x4a",
+ .iv = "\xa6\x40\x39\xf9\x63\x6c\x2d\xd4"
+ "\x1b\x71\x05\xa4\x88\x86\x11\xd3",
+ .ptext = "\xb6\x06\xae\x15\x11\x96\xc1\x44"
+ "\x44\xc2\x98\xf9\xa8\x0a\x0b",
+ .ctext = "\x27\x3b\x68\x40\xa9\x5e\x74\x6b"
+ "\x74\x67\x18\xf9\x37\xed\xed",
+ .klen = 24,
+ .len = 15,
+ },
+ {
+ .key = "\xc8\xa0\x27\x67\x04\x3f\xed\xa5"
+ "\xb4\x0c\x51\x91\x2d\x27\x77\x33"
+ "\xa5\xfc\x2a\x9f\x78\xd8\x1c\x68",
+ .iv = "\x83\x99\x1a\xe2\x84\xca\xa9\x16"
+ "\x8d\xc4\x2d\x1b\x67\xc8\x86\x21",
+ .ptext = "\xd6\x22\x85\xb8\x5d\x7e\x26\x2e"
+ "\xbe\x04\x9d\x0c\x03\x91\x45\x4a"
+ "\x36",
+ .ctext = "\x0f\x44\xa9\x62\x72\xec\x12\x26"
+ "\x3a\xc6\x83\x26\x62\x5e\xb7\x13"
+ "\x05",
+ .klen = 24,
+ .len = 17,
+ },
+ {
+ .key = "\xc5\x87\x18\x09\x0a\x4e\x66\x3e"
+ "\x50\x90\x19\x93\xc0\x33\xcf\x80"
+ "\x3a\x36\x6b\x6c\x43\xd7\xe4\x93",
+ .iv = "\xdd\x0b\x75\x1f\xee\x2f\xb4\x52"
+ "\x10\x82\x1f\x79\x8a\xa4\x9b\x87",
+ .ptext = "\x56\xf9\x13\xce\x9f\x30\x10\x11"
+ "\x1b\x59\xfd\x39\x5a\x29\xa3\x44"
+ "\x78\x97\x8c\xf6\x99\x6d\x26\xf1"
+ "\x32\x60\x6a\xeb\x04\x47\x29\x4c"
+ "\x7e\x14\xef\x4d\x55\x29\xfe\x36"
+ "\x37\xcf\x0b\x6e\xf3\xce\x15\xd2",
+ .ctext = "\x8f\x98\xe1\x5a\x7f\xfe\xc7\x05"
+ "\x76\xb0\xd5\xde\x90\x52\x2b\xa8"
+ "\xf3\x6e\x3c\x77\xa5\x33\x63\xdd"
+ "\x6f\x62\x12\xb0\x80\x10\xc1\x28"
+ "\x58\xe5\xd6\x24\x44\x04\x55\xf3"
+ "\x6d\x94\xcb\x2c\x7e\x7a\x85\x79",
+ .klen = 24,
+ .len = 48,
+ },
+ {
+ .key = "\x84\x9b\xe8\x10\x4c\xb3\xd1\x7a"
+ "\xb3\xab\x4e\x6f\x90\x12\x07\xf8"
+ "\xef\xde\x42\x09\xbf\x34\x95\xb2",
+ .iv = "\x66\x62\xf9\x48\x9d\x17\xf7\xdf"
+ "\x06\x67\xf4\x6d\xf2\xbc\xa2\xe5",
+ .ptext = "\x2f\xd6\x16\x6b\xf9\x4b\x44\x14"
+ "\x90\x93\xe5\xfd\x05\xaa\x00\x26"
+ "\xbd\xab\x11\xb8\xf0\xcb\x11\x72"
+ "\xdd\xc5\x15\x4f\x4e\x1b\xf8\xc9"
+ "\x8f\x4a\xd5\x69\xf8\x9e\xfb\x05"
+ "\x8a\x37\x46\xfe\xfa\x58\x9b\x0e"
+ "\x72\x90\x9a\x06\xa5\x42\xf4\x7c"
+ "\x35\xd5\x64\x70\x72\x67\xfc\x8b"
+ "\xab\x5a\x2f\x64\x9b\xa1\xec\xe7"
+ "\xe6\x92\x69\xdb\x62\xa4\xe7\x44"
+ "\x88\x28\xd4\x52\x64\x19\xa9\xd7"
+ "\x0c\x00\xe6\xe7\xc1\x28\xc1\xf5"
+ "\x72\xc5\xfa\x09\x22\x2e\xf4\x82"
+ "\xa3\xdc\xc1\x68\xf9\x29\x55\x8d"
+ "\x04\x67\x13\xa6\x52\x04\x3c\x0c"
+ "\x14\xf2\x87\x23\x61\xab\x82\xcb"
+ "\x49\x5b\x6b\xd4\x4f\x0d\xd4\x95"
+ "\x82\xcd\xe3\x69\x47\x1b\x31\x73"
+ "\x73\x77\xc1\x53\x7d\x43\x5e\x4a"
+ "\x80\x3a\xca\x9c\xc7\x04\x1a\x31"
+ "\x8e\xe6\x76\x7f\xe1\xb3\xd0\x57"
+ "\xa2\xb2\xf6\x09\x51\xc9\x6d\xbc"
+ "\x79\xed\x57\x50\x36\xd2\x93\xa4"
+ "\x40\x5d\xac\x3a\x3b\xb6\x2d\x89"
+ "\x78\xa2\xbd\x23\xec\x35\x06\xf0"
+ "\xa8\xc8\xc9\xb0\xe3\x28\x2b\xba"
+ "\x70\xa0\xfe\xed\x13\xc4\xd7\x90"
+ "\xb1\x6a\xe0\xe1\x30\x71\x15\xd0"
+ "\xe2\xb3\xa6\x4e\xb0\x01\xf9\xe7"
+ "\x59\xc6\x1e\xed\x46\x2b\xe3\xa8"
+ "\x22\xeb\x7f\x1c\xd9\xcd\xe0\xa6"
+ "\x72\x42\x2c\x06\x75\xbb\xb7\x6b"
+ "\xca\x49\x5e\xa1\x47\x8d\x9e\xfe"
+ "\x60\xcc\x34\x95\x8e\xfa\x1e\x3e"
+ "\x85\x4b\x03\x54\xea\x34\x1c\x41"
+ "\x90\x45\xa6\xbe\xcf\x58\x4f\xca"
+ "\x2c\x79\xc0\x3e\x8f\xd7\x3b\xd4"
+ "\x55\x74\xa8\xe1\x57\x09\xbf\xab"
+ "\x2c\xf9\xe4\xdd\x17\x99\x57\x60"
+ "\x4b\x88\x2a\x7f\x43\x86\xb9\x9a"
+ "\x60\xbf\x4c\xcf\x9b\x41\xb8\x99"
+ "\x69\x15\x4f\x91\x4d\xeb\xdf\x6f"
+ "\xcc\x4c\xf9\x6f\xf2\x33\x23\xe7"
+ "\x02\x44\xaa\xa2\xfa\xb1\x39\xa5"
+ "\xff\x88\xf5\x37\x02\x33\x24\xfc"
+ "\x79\x11\x4c\x94\xc2\x31\x87\x9c"
+ "\x53\x19\x99\x32\xe4\xde\x18\xf4"
+ "\x8f\xe2\xe8\xa3\xfb\x0b\xaa\x7c"
+ "\xdb\x83\x0f\xf6\xc0\x8a\x9b\xcd"
+ "\x7b\x16\x05\x5b\xe4\xb4\x34\x03"
+ "\xe3\x8f\xc9\x4b\x56\x84\x2a\x4c"
+ "\x36\x72\x3c\x84\x4f\xba\xa2\x7f"
+ "\xf7\x1b\xba\x4d\x8a\xb8\x5d\x51"
+ "\x36\xfb\xef\x23\x18\x6f\x33\x2d"
+ "\xbb\x06\x24\x8e\x33\x98\x6e\xcd"
+ "\x63\x11\x18\x6b\xcc\x1b\x66\xb9"
+ "\x38\x8d\x06\x8d\x98\x1a\xef\xaa"
+ "\x35\x4a\x90\xfa\xb1\xd3\xcc\x11"
+ "\x50\x4c\x54\x18\x60\x5d\xe4\x11"
+ "\xfc\x19\xe1\x53\x20\x5c\xe7\xef"
+ "\x8a\x2b\xa8\x82\x51\x5f\x5d\x43"
+ "\x34\xe5\xcf\x7b\x1b\x6f\x81\x19"
+ "\xb7\xdf\xa8\x9e\x81\x89\x5f\x33"
+ "\x69\xaf\xde\x89\x68\x88\xf0\x71",
+ .ctext = "\xab\x15\x46\x5b\xed\x4f\xa8\xac"
+ "\xbf\x31\x30\x84\x55\xa4\xb8\x98"
+ "\x79\xba\xa0\x15\xa4\x55\x20\xec"
+ "\xf9\x94\x71\xe6\x6a\x6f\xee\x87"
+ "\x2e\x3a\xa2\x95\xae\x6e\x56\x09"
+ "\xe9\xc0\x0f\xe2\xc6\xb7\x30\xa9"
+ "\x73\x8e\x59\x7c\xfd\xe3\x71\xf7"
+ "\xae\x8b\x91\xab\x5e\x36\xe9\xa8"
+ "\xff\x17\xfa\xa2\x94\x93\x11\x42"
+ "\x67\x96\x99\xc5\xf0\xad\x2a\x57"
+ "\xf9\xa6\x70\x4a\xdf\x71\xff\xc0"
+ "\xe2\xaf\x9a\xae\x57\x58\x13\x3b"
+ "\x2d\xf1\xc7\x8f\xdb\x8a\xcc\xce"
+ "\x53\x1a\x69\x55\x39\xc8\xbe\xc3"
+ "\x2d\xb1\x03\xd9\xa3\x99\xf4\x8d"
+ "\xd9\x2d\x27\xae\xa5\xe7\x77\x7f"
+ "\xbb\x88\x84\xea\xfa\x19\x3f\x44"
+ "\x61\x21\x8a\x1f\xbe\xac\x60\xb4"
+ "\xaf\xe9\x00\xab\xef\x3c\x53\x56"
+ "\xcd\x4b\x53\xd8\x9b\xfe\x88\x23"
+ "\x5b\x85\x76\x08\xec\xd1\x6e\x4a"
+ "\x87\xa4\x7d\x29\x4e\x4f\x3f\xc9"
+ "\xa4\xab\x63\xea\xdd\xef\x9f\x79"
+ "\x38\x18\x7d\x90\x90\xf9\x12\x57"
+ "\x1d\x89\xea\xfe\xd4\x47\x45\x32"
+ "\x6a\xf6\xe7\xde\x22\x7e\xee\xc1"
+ "\xbc\x2d\xc3\xbb\xe5\xd4\x13\xac"
+ "\x63\xff\x5b\xb1\x05\x96\xd5\xf3"
+ "\x07\x9a\x62\xb6\x30\xea\x7d\x1e"
+ "\xee\x75\x0a\x1b\xcc\x6e\x4d\xa7"
+ "\xf7\x4d\x74\xd8\x60\x32\x5e\xd0"
+ "\x93\xd7\x19\x90\x4e\x26\xdb\xe4"
+ "\x5e\xd4\xa8\xb9\x76\xba\x56\x91"
+ "\xc4\x75\x04\x1e\xc2\x77\x24\x6f"
+ "\xf9\xe8\x4a\xec\x7f\x86\x95\xb3"
+ "\x5c\x2c\x97\xab\xf0\xf7\x74\x5b"
+ "\x0b\xc2\xda\x42\x40\x34\x16\xed"
+ "\x06\xc1\x25\x53\x17\x0d\x81\x4e"
+ "\xe6\xf2\x0f\x6d\x94\x3c\x90\x7a"
+ "\xae\x20\xe9\x3f\xf8\x18\x67\x6a"
+ "\x49\x1e\x41\xb6\x46\xab\xc8\xa7"
+ "\xcb\x19\x96\xf5\x99\xc0\x66\x3e"
+ "\x77\xcf\x73\x52\x83\x2a\xe2\x48"
+ "\x27\x6c\xeb\xe7\xe7\xc4\xd5\x6a"
+ "\x40\x67\xbc\xbf\x6b\x3c\xf3\xbb"
+ "\x51\x5e\x31\xac\x03\x81\xab\x61"
+ "\xfa\xa5\xa6\x7d\x8b\xc3\x8a\x75"
+ "\x28\x7a\x71\x9c\xac\x8f\x76\xfc"
+ "\xf9\x6c\x5d\x9b\xd7\xf6\x36\x2d"
+ "\x61\xd5\x61\xaa\xdd\x01\xfc\x57"
+ "\x91\x10\xcd\xcd\x6d\x27\x63\x24"
+ "\x67\x46\x7a\xbb\x61\x56\x39\xb1"
+ "\xd6\x79\xfe\x77\xca\xd6\x73\x59"
+ "\x6e\x58\x11\x90\x03\x26\x74\x2a"
+ "\xfa\x52\x12\x47\xfb\x12\xeb\x3e"
+ "\x88\xf0\x52\x6c\xc0\x54\x7a\x88"
+ "\x8c\xe5\xde\x9e\xba\xb9\xf2\xe1"
+ "\x97\x2e\x5c\xbd\xf4\x13\x7e\xf3"
+ "\xc4\xe1\x87\xa5\x35\xfa\x7c\x71"
+ "\x1a\xc9\xf4\xa8\x57\xe2\x5a\x6b"
+ "\x14\xe0\x73\xaf\x56\x6b\xa0\x00"
+ "\x9e\x5f\x64\xac\x00\xfb\xc4\x92"
+ "\xe5\xe2\x8a\xb2\x9e\x75\x49\x85"
+ "\x25\x66\xa5\x1a\xf9\x7d\x1d\x60",
+ .klen = 24,
+ .len = 512,
+ },
+ {
+ .key = "\x05\x60\x3a\x7e\x60\x90\x46\x18"
+ "\x6c\x60\xba\xeb\x12\xd7\xbe\xd1"
+ "\xd3\xf6\x10\x46\x9d\xf1\x0c\xb4"
+ "\x73\xe3\x93\x27\xa8\x2c\x13\xaa",
+ .iv = "\xf5\x96\xd1\xb6\xcb\x44\xd8\xd0"
+ "\x3e\xdb\x92\x80\x08\x94\xcd\xd3",
+ .ptext = "\x78",
+ .ctext = "\xc5",
+ .klen = 32,
+ .len = 1,
+ },
+ {
+ .key = "\x35\xca\x38\xf3\xd9\xd6\x34\xef"
+ "\xcd\xee\xa3\x26\x86\xba\xfb\x45"
+ "\x01\xfa\x52\x67\xff\xc5\x9d\xaa"
+ "\x64\x9a\x05\xbb\x85\x20\xa7\xf2",
+ .iv = "\xe3\xda\xf5\xff\x42\x59\x87\x86"
+ "\xee\x7b\xd6\xb4\x6a\x25\x44\xff",
+ .ptext = "\x44\x67\x1e\x04\x53\xd2\x4b\xd9"
+ "\x96\x33\x07\x54\xe4\x8e\x20",
+ .ctext = "\xcc\x55\x40\x79\x47\x5c\x8b\xa6"
+ "\xca\x7b\x9f\x50\xe3\x21\xea",
+ .klen = 32,
+ .len = 15,
+ },
+ {
+ .key = "\xaf\xd9\x14\x14\xd5\xdb\xc9\xce"
+ "\x76\x5c\x5a\xbf\x43\x05\x29\x24"
+ "\xc4\x13\x68\xcc\xe8\x37\xbd\xb9"
+ "\x41\x20\xf5\x53\x48\xd0\xa2\xd6",
+ .iv = "\xa7\xb4\x00\x08\x79\x10\xae\xf5"
+ "\x02\xbf\x85\xb2\x69\x4c\xc6\x04",
+ .ptext = "\xac\x6a\xa8\x0c\xb0\x84\xbf\x4c"
+ "\xae\x94\x20\x58\x7e\x00\x93\x89",
+ .ctext = "\xd5\xaa\xe2\xe9\x86\x4c\x95\x4e"
+ "\xde\xb6\x15\xcb\xdc\x1f\x13\x38",
+ .klen = 32,
+ .len = 16,
+ },
+ {
+ .key = "\xed\xe3\x8b\xe7\x1c\x17\xbf\x4a"
+ "\x02\xe2\xfc\x76\xac\xf5\x3c\x00"
+ "\x5d\xdc\xfc\x83\xeb\x45\xb4\xcb"
+ "\x59\x62\x60\xec\x69\x9c\x16\x45",
+ .iv = "\xe4\x0e\x2b\x90\xd2\xfa\x94\x2e"
+ "\x10\xe5\x64\x2b\x97\x28\x15\xc7",
+ .ptext = "\xe6\x53\xff\x60\x0e\xc4\x51\xe4"
+ "\x93\x4d\xe5\x55\xc5\xd9\xad\x48"
+ "\x52",
+ .ctext = "\xba\x25\x28\xf5\xcf\x31\x91\x80"
+ "\xda\x2b\x95\x5f\x20\xcb\xfb\x9f"
+ "\xc6",
+ .klen = 32,
+ .len = 17,
+ },
+ {
+ .key = "\x77\x5c\xc0\x73\x9a\x64\x97\x91"
+ "\x2f\xee\xe0\x20\xc2\x04\x59\x2e"
+ "\x97\xd2\xa7\x70\xb3\xb0\x21\x6b"
+ "\x8f\xbf\xb8\x51\xa8\xea\x0f\x62",
+ .iv = "\x31\x8e\x1f\xcd\xfd\x23\xeb\x7f"
+ "\x8a\x1f\x1b\x23\x53\x27\x44\xe5",
+ .ptext = "\xcd\xff\x8c\x9b\x94\x5a\x51\x3f"
+ "\x40\x93\x56\x93\x66\x39\x63\x1f"
+ "\xbf\xe6\xa4\xfa\xbe\x79\x93\x03"
+ "\xf5\x66\x74\x16\xfc\xe4\xce",
+ .ctext = "\x8b\xd3\xc3\xce\x66\xf8\x66\x4c"
+ "\xad\xd6\xf5\x0f\xd8\x99\x5a\x75"
+ "\xa1\x3c\xab\x0b\x21\x36\x57\x72"
+ "\x88\x29\xe9\xea\x4a\x8d\xe9",
+ .klen = 32,
+ .len = 31,
+ },
+ {
+ .key = "\xa1\x2f\x4d\xde\xfe\xa1\xff\xa8"
+ "\x73\xdd\xe3\xe2\x95\xfc\xea\x9c"
+ "\xd0\x80\x42\x0c\xb8\x43\x3e\x99"
+ "\x39\x38\x0a\x8c\xe8\x45\x3a\x7b",
+ .iv = "\x32\xc4\x6f\xb1\x14\x43\xd1\x87"
+ "\xe2\x6f\x5a\x58\x02\x36\x7e\x2a",
+ .ptext = "\x9e\x5c\x1e\xf1\xd6\x7d\x09\x57"
+ "\x18\x48\x55\xda\x7d\x44\xf9\x6d"
+ "\xac\xcd\x59\xbb\x10\xa2\x94\x67"
+ "\xd1\x6f\xfe\x6b\x4a\x11\xe8\x04"
+ "\x09\x26\x4f\x8d\x5d\xa1\x7b\x42"
+ "\xf9\x4b\x66\x76\x38\x12\xfe\xfe",
+ .ctext = "\x42\xbc\xa7\x64\x15\x9a\x04\x71"
+ "\x2c\x5f\x94\xba\x89\x3a\xad\xbc"
+ "\x87\xb3\xf4\x09\x4f\x57\x06\x18"
+ "\xdc\x84\x20\xf7\x64\x85\xca\x3b"
+ "\xab\xe6\x33\x56\x34\x60\x5d\x4b"
+ "\x2e\x16\x13\xd4\x77\xde\x2d\x2b",
+ .klen = 32,
+ .len = 48,
+ },
+ {
+ .key = "\xfb\xf5\xb7\x3d\xa6\x95\x42\xbf"
+ "\xd2\x94\x6c\x74\x0f\xbc\x5a\x28"
+ "\x35\x3c\x51\x58\x84\xfb\x7d\x11"
+ "\x16\x1e\x00\x97\x37\x08\xb7\x16",
+ .iv = "\x9b\x53\x57\x40\xe6\xd9\xa7\x27"
+ "\x78\xd4\x9b\xd2\x29\x1d\x24\xa9",
+ .ptext = "\x8b\x02\x60\x0a\x3e\xb7\x10\x59"
+ "\xc3\xac\xd5\x2a\x75\x81\xf2\xdb"
+ "\x55\xca\x65\x86\x44\xfb\xfe\x91"
+ "\x26\xbb\x45\xb2\x46\x22\x3e\x08"
+ "\xa2\xbf\x46\xcb\x68\x7d\x45\x7b"
+ "\xa1\x6a\x3c\x6e\x25\xeb\xed\x31"
+ "\x7a\x8b\x47\xf9\xde\xec\x3d\x87"
+ "\x09\x20\x2e\xfa\xba\x8b\x9b\xc5"
+ "\x6c\x25\x9c\x9d\x2a\xe8\xab\x90"
+ "\x3f\x86\xee\x61\x13\x21\xd4\xde"
+ "\xe1\x0c\x95\xfc\x5c\x8a\x6e\x0a"
+ "\x73\xcf\x08\x69\x44\x4e\xde\x25"
+ "\xaf\xaa\x56\x04\xc4\xb3\x60\x44"
+ "\x3b\x8b\x3d\xee\xae\x42\x4b\xd2"
+ "\x9a\x6c\xa0\x8e\x52\x06\xb2\xd1"
+ "\x5d\x38\x30\x6d\x27\x9b\x1a\xd8",
+ .ctext = "\xa3\x78\x33\x78\x95\x95\x97\x07"
+ "\x53\xa3\xa1\x5b\x18\x32\x27\xf7"
+ "\x09\x12\x53\x70\x83\xb5\x6a\x9f"
+ "\x26\x6d\x10\x0d\xe0\x1c\xe6\x2b"
+ "\x70\x00\xdc\xa1\x60\xef\x1b\xee"
+ "\xc5\xa5\x51\x17\xae\xcc\xf2\xed"
+ "\xc4\x60\x07\xdf\xd5\x7a\xe9\x90"
+ "\x3c\x9f\x96\x5d\x72\x65\x5d\xef"
+ "\xd0\x94\x32\xc4\x85\x90\x78\xa1"
+ "\x2e\x64\xf6\xee\x8e\x74\x3f\x20"
+ "\x2f\x12\x3b\x3d\xd5\x39\x8e\x5a"
+ "\xf9\x8f\xce\x94\x5d\x82\x18\x66"
+ "\x14\xaf\x4c\xfe\xe0\x91\xc3\x4a"
+ "\x85\xcf\xe7\xe8\xf7\xcb\xf0\x31"
+ "\x88\x7d\xc9\x5b\x71\x9d\x5f\xd2"
+ "\xfa\xed\xa6\x24\xda\xbb\xb1\x84",
+ .klen = 32,
+ .len = 128,
+ },
+ {
+ .key = "\x32\x37\x2b\x8f\x7b\xb1\x23\x79"
+ "\x05\x52\xde\x05\xf1\x68\x3f\x6c"
+ "\xa4\xae\xbc\x21\xc2\xc6\xf0\xbd"
+ "\x0f\x20\xb7\xa4\xc5\x05\x7b\x64",
+ .iv = "\xff\x26\x4e\x67\x48\xdd\xcf\xfe"
+ "\x42\x09\x04\x98\x5f\x1e\xfa\x80",
+ .ptext = "\x99\xdc\x3b\x19\x41\xf9\xff\x6e"
+ "\x76\xb5\x03\xfa\x61\xed\xf8\x44"
+ "\x70\xb9\xf0\x83\x80\x6e\x31\x77"
+ "\x77\xe4\xc7\xb4\x77\x02\xab\x91"
+ "\x82\xc6\xf8\x7c\x46\x61\x03\x69"
+ "\x09\xa0\xf7\x12\xb7\x81\x6c\xa9"
+ "\x10\x5c\xbb\x55\xb3\x44\xed\xb5"
+ "\xa2\x52\x48\x71\x90\x5d\xda\x40"
+ "\x0b\x7f\x4a\x11\x6d\xa7\x3d\x8e"
+ "\x1b\xcd\x9d\x4e\x75\x8b\x7d\x87"
+ "\xe5\x39\x34\x32\x1e\xe6\x8d\x51"
+ "\xd4\x1f\xe3\x1d\x50\xa0\x22\x37"
+ "\x7c\xb0\xd9\xfb\xb6\xb2\x16\xf6"
+ "\x6d\x26\xa0\x4e\x8c\x6a\xe6\xb6"
+ "\xbe\x4c\x7c\xe3\x88\x10\x18\x90"
+ "\x11\x50\x19\x90\xe7\x19\x3f\xd0"
+ "\x31\x15\x0f\x06\x96\xfe\xa7\x7b"
+ "\xc3\x32\x88\x69\xa4\x12\xe3\x64"
+ "\x02\x30\x17\x74\x6c\x88\x7c\x9b"
+ "\xd6\x6d\x75\xdf\x11\x86\x70\x79"
+ "\x48\x7d\x34\x3e\x33\x58\x07\x8b"
+ "\xd2\x50\xac\x35\x15\x45\x05\xb4"
+ "\x4d\x31\x97\x19\x87\x23\x4b\x87"
+ "\x53\xdc\xa9\x19\x78\xf1\xbf\x35"
+ "\x30\x04\x14\xd4\xcf\xb2\x8c\x87"
+ "\x7d\xdb\x69\xc9\xcd\xfe\x40\x3e"
+ "\x8d\x66\x5b\x61\xe5\xf0\x2d\x87"
+ "\x93\x3a\x0c\x2b\x04\x98\x05\xc2"
+ "\x56\x4d\xc4\x6c\xcd\x7a\x98\x7e"
+ "\xe2\x2d\x79\x07\x91\x9f\xdf\x2f"
+ "\x72\xc9\x8f\xcb\x0b\x87\x1b\xb7"
+ "\x04\x86\xcb\x47\xfa\x5d\x03",
+ .ctext = "\x0b\x00\xf7\xf2\xc8\x6a\xba\x9a"
+ "\x0a\x97\x18\x7a\x00\xa0\xdb\xf4"
+ "\x5e\x8e\x4a\xb7\xe0\x51\xf1\x75"
+ "\x17\x8b\xb4\xf1\x56\x11\x05\x9f"
+ "\x2f\x2e\xba\x67\x04\xe1\xb4\xa5"
+ "\xfc\x7c\x8c\xad\xc6\xb9\xd1\x64"
+ "\xca\xbd\x5d\xaf\xdb\x65\x48\x4f"
+ "\x1b\xb3\x94\x5c\x0b\xd0\xee\xcd"
+ "\xb5\x7f\x43\x8a\xd8\x8b\x66\xde"
+ "\xd2\x9c\x13\x65\xa4\x47\xa7\x03"
+ "\xc5\xa1\x46\x8f\x2f\x84\xbc\xef"
+ "\x48\x9d\x9d\xb5\xbd\x43\xff\xd2"
+ "\xd2\x7a\x5a\x13\xbf\xb4\xf6\x05"
+ "\x17\xcd\x01\x12\xf0\x35\x27\x96"
+ "\xf4\xc1\x65\xf7\x69\xef\x64\x1b"
+ "\x6e\x4a\xe8\x77\xce\x83\x01\xb7"
+ "\x60\xe6\x45\x2a\xcd\x41\x4a\xb5"
+ "\x8e\xcc\x45\x93\xf1\xd6\x64\x5f"
+ "\x32\x60\xe4\x29\x4a\x82\x6c\x86"
+ "\x16\xe4\xcc\xdb\x5f\xc8\x11\xa6"
+ "\xfe\x88\xd6\xc3\xe5\x5c\xbb\x67"
+ "\xec\xa5\x7b\xf5\xa8\x4f\x77\x25"
+ "\x5d\x0c\x2a\x99\xf9\xb9\xd1\xae"
+ "\x3c\x83\x2a\x93\x9b\x66\xec\x68"
+ "\x2c\x93\x02\x8a\x8a\x1e\x2f\x50"
+ "\x09\x37\x19\x5c\x2a\x3a\xc2\xcb"
+ "\xcb\x89\x82\x81\xb7\xbb\xef\x73"
+ "\x8b\xc9\xae\x42\x96\xef\x70\xc0"
+ "\x89\xc7\x3e\x6a\x26\xc3\xe4\x39"
+ "\x53\xa9\xcf\x63\x7d\x05\xf3\xff"
+ "\x52\x04\xf6\x7f\x23\x96\xe9\xf7"
+ "\xff\xd6\x50\xa3\x0e\x20\x71",
+ .klen = 32,
+ .len = 255,
+ },
+ {
+ .key = "\x39\x5f\xf4\x9c\x90\x3a\x9a\x25"
+ "\x15\x11\x79\x39\xed\x26\x5e\xf6"
+ "\xda\xcf\x33\x4f\x82\x97\xab\x10"
+ "\xc1\x55\x48\x82\x80\xa8\x02\xb2",
+ .iv = "\x82\x60\xd9\x06\xeb\x40\x99\x76"
+ "\x08\xc5\xa4\x83\x45\xb8\x38\x5a",
+ .ptext = "\xa1\xa8\xac\xac\x08\xaf\x8f\x84"
+ "\xbf\xcc\x79\x31\x5e\x61\x01\xd1"
+ "\x4d\x5f\x9b\xcd\x91\x92\x9a\xa1"
+ "\x99\x0d\x49\xb2\xd7\xfd\x25\x93"
+ "\x51\x96\xbd\x91\x8b\x08\xf1\xc6"
+ "\x0d\x17\xf6\xef\xfd\xd2\x78\x16"
+ "\xc8\x08\x27\x7b\xca\x98\xc6\x12"
+ "\x86\x11\xdb\xd5\x08\x3d\x5a\x2c"
+ "\xcf\x15\x0e\x9b\x42\x78\xeb\x1f"
+ "\x52\xbc\xd7\x5a\x8a\x33\x6c\x14"
+ "\xfc\x61\xad\x2e\x1e\x03\x66\xea"
+ "\x79\x0e\x88\x88\xde\x93\xe3\x81"
+ "\xb5\xc4\x1c\xe6\x9c\x08\x18\x8e"
+ "\xa0\x87\xda\xe6\xf8\xcb\x30\x44"
+ "\x2d\x4e\xc0\xa3\x60\xf9\x62\x7b"
+ "\x4b\xd5\x61\x6d\xe2\x67\x95\x54"
+ "\x10\xd1\xca\x22\xe8\xb6\xb1\x3a"
+ "\x2d\xd7\x35\x5b\x22\x88\x55\x67"
+ "\x3d\x83\x8f\x07\x98\xa8\xf2\xcf"
+ "\x04\xb7\x9e\x52\xca\xe0\x98\x72"
+ "\x5c\xc1\x00\xd4\x1f\x2c\x61\xf3"
+ "\xe8\x40\xaf\x4a\xee\x66\x41\xa0"
+ "\x02\x77\x29\x30\x65\x59\x4b\x20"
+ "\x7b\x0d\x80\x97\x27\x7f\xd5\x90"
+ "\xbb\x9d\x76\x90\xe5\x43\x43\x72"
+ "\xd0\xd4\x14\x75\x66\xb3\xb6\xaf"
+ "\x09\xe4\x23\xb0\x62\xad\x17\x28"
+ "\x39\x26\xab\xf5\xf7\x5c\xb6\x33"
+ "\xbd\x27\x09\x5b\x29\xe4\x40\x0b"
+ "\xc1\x26\x32\xdb\x9a\xdf\xf9\x5a"
+ "\xae\x03\x2c\xa4\x40\x84\x9a\xb7"
+ "\x4e\x47\xa8\x0f\x23\xc7\xbb\xcf"
+ "\x2b\xf2\x32\x6c\x35\x6a\x91\xba"
+ "\x0e\xea\xa2\x8b\x2f\xbd\xb5\xea"
+ "\x6e\xbc\xb5\x4b\x03\xb3\x86\xe0"
+ "\x86\xcf\xba\xcb\x38\x2c\x32\xa6"
+ "\x6d\xe5\x28\xa6\xad\xd2\x7f\x73"
+ "\x43\x14\xf8\xb1\x99\x12\x2d\x2b"
+ "\xdf\xcd\xf2\x81\x43\x94\xdf\xb1"
+ "\x17\xc9\x33\xa6\x3d\xef\x96\xb8"
+ "\xd6\x0d\x00\xec\x49\x66\x85\x5d"
+ "\x44\x62\x12\x04\x55\x5c\x48\xd3"
+ "\xbd\x73\xac\x54\x8f\xbf\x97\x8e"
+ "\x85\xfd\xc2\xa1\x25\x32\x38\x6a"
+ "\x1f\xac\x57\x3c\x4f\x56\x73\xf2"
+ "\x1d\xb6\x48\x68\xc7\x0c\xe7\x60"
+ "\xd2\x8e\x4d\xfb\xc7\x20\x7b\xb7"
+ "\x45\x28\x12\xc6\x26\xae\xea\x7c"
+ "\x5d\xe2\x46\xb5\xae\xe1\xc3\x98"
+ "\x6f\x72\xd5\xa2\xfd\xed\x40\xfd"
+ "\xf9\xdf\x61\xec\x45\x2c\x15\xe0"
+ "\x1e\xbb\xde\x71\x37\x5f\x73\xc2"
+ "\x11\xcc\x6e\x6d\xe1\xb5\x1b\xd2"
+ "\x2a\xdd\x19\x8a\xc2\xe1\xa0\xa4"
+ "\x26\xeb\xb2\x2c\x4f\x77\x52\xf1"
+ "\x42\x72\x6c\xad\xd7\x78\x5d\x72"
+ "\xc9\x16\x26\x25\x1b\x4c\xe6\x58"
+ "\x79\x57\xb5\x06\x15\x4f\xe5\xba"
+ "\xa2\x7f\x2d\x5b\x87\x8a\x44\x70"
+ "\xec\xc7\xef\x84\xae\x60\xa2\x61"
+ "\x86\xe9\x18\xcd\x28\xc4\xa4\xf5"
+ "\xbc\x84\xb8\x86\xa0\xba\xf1\xf1"
+ "\x08\x3b\x32\x75\x35\x22\x7a\x65"
+ "\xca\x48\xe8\xef\x6e\xe2\x8e\x00",
+ .ctext = "\x2f\xae\xd8\x67\xeb\x15\xde\x75"
+ "\x53\xa3\x0e\x5a\xcf\x1c\xbe\xea"
+ "\xde\xf9\xcf\xc2\x9f\xfd\x0f\x44"
+ "\xc0\xe0\x7a\x76\x1d\xcb\x4a\xf8"
+ "\x35\xd6\xe3\x95\x98\x6b\x3f\x89"
+ "\xc4\xe6\xb6\x6f\xe1\x8b\x39\x4b"
+ "\x1c\x6c\x77\xe4\xe1\x8a\xbc\x61"
+ "\x00\x6a\xb1\x37\x2f\x45\xe6\x04"
+ "\x52\x0b\xfc\x1e\x32\xc1\xd8\x9d"
+ "\xfa\xdd\x67\x5c\xe0\x75\x83\xd0"
+ "\x21\x9e\x02\xea\xc0\x7f\xc0\x29"
+ "\xb3\x6c\xa5\x97\xb3\x29\x82\x1a"
+ "\x94\xa5\xb4\xb6\x49\xe5\xa5\xad"
+ "\x95\x40\x52\x7c\x84\x88\xa4\xa8"
+ "\x26\xe4\xd9\x5d\x41\xf2\x93\x7b"
+ "\xa4\x48\x1b\x66\x91\xb9\x7c\xc2"
+ "\x99\x29\xdf\xd8\x30\xac\xd4\x47"
+ "\x42\xa0\x14\x87\x67\xb8\xfd\x0b"
+ "\x1e\xcb\x5e\x5c\x9a\xc2\x04\x8b"
+ "\x17\x29\x9d\x99\x7f\x86\x4c\xe2"
+ "\x5c\x96\xa6\x0f\xb6\x47\x33\x5c"
+ "\xe4\x50\x49\xd5\x4f\x92\x0b\x9a"
+ "\xbc\x52\x4c\x41\xf5\xc9\x3e\x76"
+ "\x55\x55\xd4\xdc\x71\x14\x23\xfc"
+ "\x5f\xd5\x08\xde\xa0\xf7\x28\xc0"
+ "\xe1\x61\xac\x64\x66\xf6\xd1\x31"
+ "\xe4\xa4\xa9\xed\xbc\xad\x4f\x3b"
+ "\x59\xb9\x48\x1b\xe7\xb1\x6f\xc6"
+ "\xba\x40\x1c\x0b\xe7\x2f\x31\x65"
+ "\x85\xf5\xe9\x14\x0a\x31\xf5\xf3"
+ "\xc0\x1c\x20\x35\x73\x38\x0f\x8e"
+ "\x39\xf0\x68\xae\x08\x9c\x87\x4b"
+ "\x42\xfc\x22\x17\xee\x96\x51\x2a"
+ "\xd8\x57\x5a\x35\xea\x72\x74\xfc"
+ "\xb3\x0e\x69\x9a\xe1\x4f\x24\x90"
+ "\xc5\x4b\xe5\xd7\xe3\x82\x2f\xc5"
+ "\x62\x46\x3e\xab\x72\x4e\xe0\xf3"
+ "\x90\x09\x4c\xb2\xe1\xe8\xa0\xf5"
+ "\x46\x40\x2b\x47\x85\x3c\x21\x90"
+ "\x3d\xad\x25\x5a\x36\xdf\xe5\xbc"
+ "\x7e\x80\x4d\x53\x77\xf1\x79\xa6"
+ "\xec\x22\x80\x88\x68\xd6\x2d\x8b"
+ "\x3e\xf7\x52\xc7\x2a\x20\x42\x5c"
+ "\xed\x99\x4f\x32\x80\x00\x7e\x73"
+ "\xd7\x6d\x7f\x7d\x42\x54\x4a\xfe"
+ "\xff\x6f\x61\xca\x2a\xbb\x4f\xeb"
+ "\x4f\xe4\x4e\xaf\x2c\x4f\x82\xcd"
+ "\xa1\xa7\x11\xb3\x34\x33\xcf\x32"
+ "\x63\x0e\x24\x3a\x35\xbe\x06\xd5"
+ "\x17\xcb\x02\x30\x33\x6e\x8c\x49"
+ "\x40\x6e\x34\x8c\x07\xd4\x3e\xe6"
+ "\xaf\x78\x6d\x8c\x10\x5f\x21\x58"
+ "\x49\x26\xc5\xaf\x0d\x7d\xd4\xaf"
+ "\xcd\x5b\xa1\xe3\xf6\x39\x1c\x9b"
+ "\x8e\x00\xa1\xa7\x9e\x17\x4a\xc0"
+ "\x54\x56\x9e\xcf\xcf\x88\x79\x8d"
+ "\x50\xf7\x56\x8e\x0a\x73\x46\x6b"
+ "\xc3\xb9\x9b\x6c\x7d\xc4\xc8\xb6"
+ "\x03\x5f\x30\x62\x7d\xe6\xdb\x15"
+ "\xe1\x39\x02\x8c\xff\xda\xc8\x43"
+ "\xf2\xa9\xbf\x00\xe7\x3a\x61\x89"
+ "\xdf\xb0\xca\x7d\x8c\x8a\x6a\x9f"
+ "\x18\x89\x3d\x39\xac\x36\x6f\x05"
+ "\x1f\xb5\xda\x00\xea\xe1\x51\x21",
+ .klen = 32,
+ .len = 512,
+ },
+
+};
+
#endif /* _CRYPTO_TESTMGR_H */
diff --git a/crypto/xctr.c b/crypto/xctr.c
new file mode 100644
index 000000000000..3d3e9b2d6a3a
--- /dev/null
+++ b/crypto/xctr.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * XCTR: XOR Counter mode - Adapted from ctr.c
+ *
+ * (C) Copyright IBM Corp. 2007 - Joy Latten <[email protected]>
+ * Copyright 2021 Google LLC
+ */
+
+/*
+ * XCTR mode is a blockcipher mode of operation used to implement HCTR2. XCTR is
+ * closely related to the CTR mode of operation; the main difference is that CTR
+ * generates the keystream using E(CTR + IV) whereas XCTR generates the
+ * keystream using E(CTR ^ IV). This allows implementations to avoid dealing
+ * with multi-limb integers (as is required in CTR mode). XCTR is also specified
+ * using little-endian arithmetic which makes it slightly faster on LE machines.
+ *
+ * See the HCTR2 paper for more details:
+ * Length-preserving encryption with HCTR2
+ * (https://eprint.iacr.org/2021/1441.pdf)
+ */
+
+#include <crypto/algapi.h>
+#include <crypto/internal/cipher.h>
+#include <crypto/internal/skcipher.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+// Limited to 16-byte blocks for simplicity
+#define XCTR_BLOCKSIZE 16
+
+static void crypto_xctr_crypt_final(struct skcipher_walk *walk,
+ struct crypto_cipher *tfm, u32 byte_ctr)
+{
+ u8 keystream[XCTR_BLOCKSIZE];
+ u8 *src = walk->src.virt.addr;
+ u8 *dst = walk->dst.virt.addr;
+ unsigned int nbytes = walk->nbytes;
+ __le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
+
+ crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+ crypto_cipher_encrypt_one(tfm, keystream, walk->iv);
+ crypto_xor_cpy(dst, keystream, src, nbytes);
+ crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+}
+
+static int crypto_xctr_crypt_segment(struct skcipher_walk *walk,
+ struct crypto_cipher *tfm, u32 byte_ctr)
+{
+ void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
+ crypto_cipher_alg(tfm)->cia_encrypt;
+ u8 *src = walk->src.virt.addr;
+ u8 *dst = walk->dst.virt.addr;
+ unsigned int nbytes = walk->nbytes;
+ __le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
+
+ do {
+ /* create keystream */
+ crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+ fn(crypto_cipher_tfm(tfm), dst, walk->iv);
+ crypto_xor(dst, src, XCTR_BLOCKSIZE);
+ crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+
+ ctr32 = cpu_to_le32(le32_to_cpu(ctr32) + 1);
+
+ src += XCTR_BLOCKSIZE;
+ dst += XCTR_BLOCKSIZE;
+ } while ((nbytes -= XCTR_BLOCKSIZE) >= XCTR_BLOCKSIZE);
+
+ return nbytes;
+}
+
+static int crypto_xctr_crypt_inplace(struct skcipher_walk *walk,
+ struct crypto_cipher *tfm, u32 byte_ctr)
+{
+ void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
+ crypto_cipher_alg(tfm)->cia_encrypt;
+ unsigned long alignmask = crypto_cipher_alignmask(tfm);
+ unsigned int nbytes = walk->nbytes;
+ u8 *src = walk->src.virt.addr;
+ u8 tmp[XCTR_BLOCKSIZE + MAX_CIPHER_ALIGNMASK];
+ u8 *keystream = PTR_ALIGN(tmp + 0, alignmask + 1);
+ __le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
+
+ do {
+ /* create keystream */
+ crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+ fn(crypto_cipher_tfm(tfm), keystream, walk->iv);
+ crypto_xor(src, keystream, XCTR_BLOCKSIZE);
+ crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
+
+ ctr32 = cpu_to_le32(le32_to_cpu(ctr32) + 1);
+
+ src += XCTR_BLOCKSIZE;
+ } while ((nbytes -= XCTR_BLOCKSIZE) >= XCTR_BLOCKSIZE);
+
+ return nbytes;
+}
+
+static int crypto_xctr_crypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
+ struct skcipher_walk walk;
+ unsigned int nbytes;
+ int err;
+ u32 byte_ctr = 0;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ while (walk.nbytes >= XCTR_BLOCKSIZE) {
+ if (walk.src.virt.addr == walk.dst.virt.addr)
+ nbytes = crypto_xctr_crypt_inplace(&walk, cipher,
+ byte_ctr);
+ else
+ nbytes = crypto_xctr_crypt_segment(&walk, cipher,
+ byte_ctr);
+
+ byte_ctr += walk.nbytes - nbytes;
+ err = skcipher_walk_done(&walk, nbytes);
+ }
+
+ if (walk.nbytes) {
+ crypto_xctr_crypt_final(&walk, cipher, byte_ctr);
+ err = skcipher_walk_done(&walk, 0);
+ }
+
+ return err;
+}
+
+static int crypto_xctr_create(struct crypto_template *tmpl, struct rtattr **tb)
+{
+ struct skcipher_instance *inst;
+ struct crypto_alg *alg;
+ int err;
+
+ inst = skcipher_alloc_instance_simple(tmpl, tb);
+ if (IS_ERR(inst))
+ return PTR_ERR(inst);
+
+ alg = skcipher_ialg_simple(inst);
+
+ /* Block size must be 16 bytes. */
+ err = -EINVAL;
+ if (alg->cra_blocksize != XCTR_BLOCKSIZE)
+ goto out_free_inst;
+
+ /* XCTR mode is a stream cipher. */
+ inst->alg.base.cra_blocksize = 1;
+
+ /*
+ * To simplify the implementation, configure the skcipher walk to only
+ * give a partial block at the very end, never earlier.
+ */
+ inst->alg.chunksize = alg->cra_blocksize;
+
+ inst->alg.encrypt = crypto_xctr_crypt;
+ inst->alg.decrypt = crypto_xctr_crypt;
+
+ err = skcipher_register_instance(tmpl, inst);
+ if (err) {
+out_free_inst:
+ inst->free(inst);
+ }
+
+ return err;
+}
+
+static struct crypto_template crypto_xctr_tmpl = {
+ .name = "xctr",
+ .create = crypto_xctr_create,
+ .module = THIS_MODULE,
+};
+
+static int __init crypto_xctr_module_init(void)
+{
+ return crypto_register_template(&crypto_xctr_tmpl);
+}
+
+static void __exit crypto_xctr_module_exit(void)
+{
+ crypto_unregister_template(&crypto_xctr_tmpl);
+}
+
+subsys_initcall(crypto_xctr_module_init);
+module_exit(crypto_xctr_module_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("XCTR block cipher mode of operation");
+MODULE_ALIAS_CRYPTO("xctr");
+MODULE_IMPORT_NS(CRYPTO_INTERNAL);
--
2.35.1.723.g4982287a31-goog
Add hardware accelerated versions of XCTR for x86-64 CPUs with AESNI
support. These implementations are modified versions of the CTR
implementations found in aesni-intel_asm.S and aes_ctrby8_avx-x86_64.S.
More information on XCTR can be found in the HCTR2 paper:
Length-preserving encryption with HCTR2:
https://enterprint.iacr.org/2021/1441.pdf
Signed-off-by: Nathan Huckleberry <[email protected]>
---
arch/x86/crypto/aes_ctrby8_avx-x86_64.S | 233 ++++++++++++++++--------
arch/x86/crypto/aesni-intel_asm.S | 70 +++++++
arch/x86/crypto/aesni-intel_glue.c | 89 +++++++++
crypto/Kconfig | 2 +-
4 files changed, 317 insertions(+), 77 deletions(-)
diff --git a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
index 43852ba6e19c..9e20d7d3d6da 100644
--- a/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
+++ b/arch/x86/crypto/aes_ctrby8_avx-x86_64.S
@@ -53,6 +53,10 @@
#define KEY_192 2
#define KEY_256 3
+// XCTR mode only
+#define counter %r9
+#define xiv %xmm8
+
.section .rodata
.align 16
@@ -102,38 +106,67 @@ ddq_add_8:
* do_aes num_in_par load_keys key_len
* This increments p_in, but not p_out
*/
-.macro do_aes b, k, key_len
+.macro do_aes b, k, key_len, xctr
.set by, \b
.set load_keys, \k
.set klen, \key_len
+ .if (\xctr == 1)
+ .set i, 0
+ .rept (by)
+ club XDATA, i
+ movq counter, var_xdata
+ .set i, (i +1)
+ .endr
+ .endif
+
.if (load_keys)
vmovdqa 0*16(p_keys), xkey0
.endif
- vpshufb xbyteswap, xcounter, xdata0
-
- .set i, 1
- .rept (by - 1)
- club XDATA, i
- vpaddq (ddq_add_1 + 16 * (i - 1))(%rip), xcounter, var_xdata
- vptest ddq_low_msk(%rip), var_xdata
- jnz 1f
- vpaddq ddq_high_add_1(%rip), var_xdata, var_xdata
- vpaddq ddq_high_add_1(%rip), xcounter, xcounter
- 1:
- vpshufb xbyteswap, var_xdata, var_xdata
- .set i, (i +1)
- .endr
+ .if (\xctr == 0)
+ vpshufb xbyteswap, xcounter, xdata0
+ .set i, 1
+ .rept (by - 1)
+ club XDATA, i
+ vpaddq (ddq_add_1 + 16 * (i - 1))(%rip), xcounter, var_xdata
+ vptest ddq_low_msk(%rip), var_xdata
+ jnz 1f
+ vpaddq ddq_high_add_1(%rip), var_xdata, var_xdata
+ vpaddq ddq_high_add_1(%rip), xcounter, xcounter
+ 1:
+ vpshufb xbyteswap, var_xdata, var_xdata
+ .set i, (i +1)
+ .endr
+ .endif
+ .if (\xctr == 1)
+ .set i, 0
+ .rept (by)
+ club XDATA, i
+ vpaddq (ddq_add_1 + 16 * i)(%rip), var_xdata, var_xdata
+ .set i, (i +1)
+ .endr
+ .set i, 0
+ .rept (by)
+ club XDATA, i
+ vpxor xiv, var_xdata, var_xdata
+ .set i, (i +1)
+ .endr
+ .endif
vmovdqa 1*16(p_keys), xkeyA
vpxor xkey0, xdata0, xdata0
- vpaddq (ddq_add_1 + 16 * (by - 1))(%rip), xcounter, xcounter
- vptest ddq_low_msk(%rip), xcounter
- jnz 1f
- vpaddq ddq_high_add_1(%rip), xcounter, xcounter
- 1:
+ .if (\xctr == 0)
+ vpaddq (ddq_add_1 + 16 * (by - 1))(%rip), xcounter, xcounter
+ vptest ddq_low_msk(%rip), xcounter
+ jnz 1f
+ vpaddq ddq_high_add_1(%rip), xcounter, xcounter
+ 1:
+ .endif
+ .if (\xctr == 1)
+ add $by, counter
+ .endif
.set i, 1
.rept (by - 1)
@@ -371,94 +404,101 @@ ddq_add_8:
.endr
.endm
-.macro do_aes_load val, key_len
- do_aes \val, 1, \key_len
+.macro do_aes_load val, key_len, xctr
+ do_aes \val, 1, \key_len, \xctr
.endm
-.macro do_aes_noload val, key_len
- do_aes \val, 0, \key_len
+.macro do_aes_noload val, key_len, xctr
+ do_aes \val, 0, \key_len, \xctr
.endm
/* main body of aes ctr load */
-.macro do_aes_ctrmain key_len
+.macro do_aes_ctrmain key_len, xctr
cmp $16, num_bytes
- jb .Ldo_return2\key_len
+ jb .Ldo_return2\xctr\key_len
vmovdqa byteswap_const(%rip), xbyteswap
- vmovdqu (p_iv), xcounter
- vpshufb xbyteswap, xcounter, xcounter
+ .if (\xctr == 0)
+ vmovdqu (p_iv), xcounter
+ vpshufb xbyteswap, xcounter, xcounter
+ .endif
+ .if (\xctr == 1)
+ andq $(~0xf), num_bytes
+ shr $4, counter
+ vmovdqu (p_iv), xiv
+ .endif
mov num_bytes, tmp
and $(7*16), tmp
- jz .Lmult_of_8_blks\key_len
+ jz .Lmult_of_8_blks\xctr\key_len
/* 1 <= tmp <= 7 */
cmp $(4*16), tmp
- jg .Lgt4\key_len
- je .Leq4\key_len
+ jg .Lgt4\xctr\key_len
+ je .Leq4\xctr\key_len
-.Llt4\key_len:
+.Llt4\xctr\key_len:
cmp $(2*16), tmp
- jg .Leq3\key_len
- je .Leq2\key_len
+ jg .Leq3\xctr\key_len
+ je .Leq2\xctr\key_len
-.Leq1\key_len:
- do_aes_load 1, \key_len
+.Leq1\xctr\key_len:
+ do_aes_load 1, \key_len, \xctr
add $(1*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Leq2\key_len:
- do_aes_load 2, \key_len
+.Leq2\xctr\key_len:
+ do_aes_load 2, \key_len, \xctr
add $(2*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Leq3\key_len:
- do_aes_load 3, \key_len
+.Leq3\xctr\key_len:
+ do_aes_load 3, \key_len, \xctr
add $(3*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Leq4\key_len:
- do_aes_load 4, \key_len
+.Leq4\xctr\key_len:
+ do_aes_load 4, \key_len, \xctr
add $(4*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Lgt4\key_len:
+.Lgt4\xctr\key_len:
cmp $(6*16), tmp
- jg .Leq7\key_len
- je .Leq6\key_len
+ jg .Leq7\xctr\key_len
+ je .Leq6\xctr\key_len
-.Leq5\key_len:
- do_aes_load 5, \key_len
+.Leq5\xctr\key_len:
+ do_aes_load 5, \key_len, \xctr
add $(5*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Leq6\key_len:
- do_aes_load 6, \key_len
+.Leq6\xctr\key_len:
+ do_aes_load 6, \key_len, \xctr
add $(6*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Leq7\key_len:
- do_aes_load 7, \key_len
+.Leq7\xctr\key_len:
+ do_aes_load 7, \key_len, \xctr
add $(7*16), p_out
and $(~7*16), num_bytes
- jz .Ldo_return2\key_len
- jmp .Lmain_loop2\key_len
+ jz .Ldo_return2\xctr\key_len
+ jmp .Lmain_loop2\xctr\key_len
-.Lmult_of_8_blks\key_len:
+.Lmult_of_8_blks\xctr\key_len:
.if (\key_len != KEY_128)
vmovdqa 0*16(p_keys), xkey0
vmovdqa 4*16(p_keys), xkey4
@@ -471,17 +511,19 @@ ddq_add_8:
vmovdqa 9*16(p_keys), xkey12
.endif
.align 16
-.Lmain_loop2\key_len:
+.Lmain_loop2\xctr\key_len:
/* num_bytes is a multiple of 8 and >0 */
- do_aes_noload 8, \key_len
+ do_aes_noload 8, \key_len, \xctr
add $(8*16), p_out
sub $(8*16), num_bytes
- jne .Lmain_loop2\key_len
+ jne .Lmain_loop2\xctr\key_len
-.Ldo_return2\key_len:
- /* return updated IV */
- vpshufb xbyteswap, xcounter, xcounter
- vmovdqu xcounter, (p_iv)
+.Ldo_return2\xctr\key_len:
+ .if (\xctr == 0)
+ /* return updated IV */
+ vpshufb xbyteswap, xcounter, xcounter
+ vmovdqu xcounter, (p_iv)
+ .endif
RET
.endm
@@ -494,7 +536,7 @@ ddq_add_8:
*/
SYM_FUNC_START(aes_ctr_enc_128_avx_by8)
/* call the aes main loop */
- do_aes_ctrmain KEY_128
+ do_aes_ctrmain KEY_128 0
SYM_FUNC_END(aes_ctr_enc_128_avx_by8)
@@ -507,7 +549,7 @@ SYM_FUNC_END(aes_ctr_enc_128_avx_by8)
*/
SYM_FUNC_START(aes_ctr_enc_192_avx_by8)
/* call the aes main loop */
- do_aes_ctrmain KEY_192
+ do_aes_ctrmain KEY_192 0
SYM_FUNC_END(aes_ctr_enc_192_avx_by8)
@@ -520,6 +562,45 @@ SYM_FUNC_END(aes_ctr_enc_192_avx_by8)
*/
SYM_FUNC_START(aes_ctr_enc_256_avx_by8)
/* call the aes main loop */
- do_aes_ctrmain KEY_256
+ do_aes_ctrmain KEY_256 0
SYM_FUNC_END(aes_ctr_enc_256_avx_by8)
+
+/*
+ * routine to do AES128 XCTR enc/decrypt "by8"
+ * XMM registers are clobbered.
+ * Saving/restoring must be done at a higher level
+ * aes_xctr_enc_128_avx_by8(const u8 *in, const u8 *iv, const aes_ctx *keys, u8
+ * *out, unsigned int num_bytes, unsigned int byte_ctr)
+ */
+SYM_FUNC_START(aes_xctr_enc_128_avx_by8)
+ /* call the aes main loop */
+ do_aes_ctrmain KEY_128 1
+
+SYM_FUNC_END(aes_xctr_enc_128_avx_by8)
+
+/*
+ * routine to do AES192 XCTR enc/decrypt "by8"
+ * XMM registers are clobbered.
+ * Saving/restoring must be done at a higher level
+ * aes_xctr_enc_192_avx_by8(const u8 *in, const u8 *iv, const aes_ctx *keys, u8
+ * *out, unsigned int num_bytes, unsigned int byte_ctr)
+ */
+SYM_FUNC_START(aes_xctr_enc_192_avx_by8)
+ /* call the aes main loop */
+ do_aes_ctrmain KEY_192 1
+
+SYM_FUNC_END(aes_xctr_enc_192_avx_by8)
+
+/*
+ * routine to do AES256 XCTR enc/decrypt "by8"
+ * XMM registers are clobbered.
+ * Saving/restoring must be done at a higher level
+ * aes_xctr_enc_256_avx_by8(const u8 *in, const u8 *iv, const aes_ctx *keys, u8
+ * *out, unsigned int num_bytes, unsigned int byte_ctr)
+ */
+SYM_FUNC_START(aes_xctr_enc_256_avx_by8)
+ /* call the aes main loop */
+ do_aes_ctrmain KEY_256 1
+
+SYM_FUNC_END(aes_xctr_enc_256_avx_by8)
diff --git a/arch/x86/crypto/aesni-intel_asm.S b/arch/x86/crypto/aesni-intel_asm.S
index 363699dd7220..ce17fe630150 100644
--- a/arch/x86/crypto/aesni-intel_asm.S
+++ b/arch/x86/crypto/aesni-intel_asm.S
@@ -2821,6 +2821,76 @@ SYM_FUNC_END(aesni_ctr_enc)
#endif
+#ifdef __x86_64__
+/*
+ * void aesni_xctr_enc(struct crypto_aes_ctx *ctx, const u8 *dst, u8 *src,
+ * size_t len, u8 *iv, int byte_ctr)
+ */
+SYM_FUNC_START(aesni_xctr_enc)
+ FRAME_BEGIN
+ cmp $16, LEN
+ jb .Lxctr_ret
+ shr $4, %arg6
+ movq %arg6, CTR
+ mov 480(KEYP), KLEN
+ movups (IVP), IV
+ cmp $64, LEN
+ jb .Lxctr_enc_loop1
+.align 4
+.Lxctr_enc_loop4:
+ movaps IV, STATE1
+ vpaddq ONE(%rip), CTR, CTR
+ vpxor CTR, STATE1, STATE1
+ movups (INP), IN1
+ movaps IV, STATE2
+ vpaddq ONE(%rip), CTR, CTR
+ vpxor CTR, STATE2, STATE2
+ movups 0x10(INP), IN2
+ movaps IV, STATE3
+ vpaddq ONE(%rip), CTR, CTR
+ vpxor CTR, STATE3, STATE3
+ movups 0x20(INP), IN3
+ movaps IV, STATE4
+ vpaddq ONE(%rip), CTR, CTR
+ vpxor CTR, STATE4, STATE4
+ movups 0x30(INP), IN4
+ call _aesni_enc4
+ pxor IN1, STATE1
+ movups STATE1, (OUTP)
+ pxor IN2, STATE2
+ movups STATE2, 0x10(OUTP)
+ pxor IN3, STATE3
+ movups STATE3, 0x20(OUTP)
+ pxor IN4, STATE4
+ movups STATE4, 0x30(OUTP)
+ sub $64, LEN
+ add $64, INP
+ add $64, OUTP
+ cmp $64, LEN
+ jge .Lxctr_enc_loop4
+ cmp $16, LEN
+ jb .Lxctr_ret
+.align 4
+.Lxctr_enc_loop1:
+ movaps IV, STATE
+ vpaddq ONE(%rip), CTR, CTR
+ vpxor CTR, STATE1, STATE1
+ movups (INP), IN
+ call _aesni_enc1
+ pxor IN, STATE
+ movups STATE, (OUTP)
+ sub $16, LEN
+ add $16, INP
+ add $16, OUTP
+ cmp $16, LEN
+ jge .Lxctr_enc_loop1
+.Lxctr_ret:
+ FRAME_END
+ RET
+SYM_FUNC_END(aesni_xctr_enc)
+
+#endif
+
.section .rodata.cst16.gf128mul_x_ble_mask, "aM", @progbits, 16
.align 16
.Lgf128mul_x_ble_mask:
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index 41901ba9d3a2..74021bd524b6 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -112,6 +112,11 @@ asmlinkage void aesni_ctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
const u8 *in, unsigned int len, u8 *iv);
DEFINE_STATIC_CALL(aesni_ctr_enc_tfm, aesni_ctr_enc);
+asmlinkage void aesni_xctr_enc(struct crypto_aes_ctx *ctx, u8 *out,
+ const u8 *in, unsigned int len, u8 *iv,
+ unsigned int byte_ctr);
+DEFINE_STATIC_CALL(aesni_xctr_enc_tfm, aesni_xctr_enc);
+
/* Scatter / Gather routines, with args similar to above */
asmlinkage void aesni_gcm_init(void *ctx,
struct gcm_context_data *gdata,
@@ -135,6 +140,16 @@ asmlinkage void aes_ctr_enc_192_avx_by8(const u8 *in, u8 *iv,
void *keys, u8 *out, unsigned int num_bytes);
asmlinkage void aes_ctr_enc_256_avx_by8(const u8 *in, u8 *iv,
void *keys, u8 *out, unsigned int num_bytes);
+
+asmlinkage void aes_xctr_enc_128_avx_by8(const u8 *in, u8 *iv, void *keys, u8
+ *out, unsigned int num_bytes, unsigned int byte_ctr);
+
+asmlinkage void aes_xctr_enc_192_avx_by8(const u8 *in, u8 *iv, void *keys, u8
+ *out, unsigned int num_bytes, unsigned int byte_ctr);
+
+asmlinkage void aes_xctr_enc_256_avx_by8(const u8 *in, u8 *iv, void *keys, u8
+ *out, unsigned int num_bytes, unsigned int byte_ctr);
+
/*
* asmlinkage void aesni_gcm_init_avx_gen2()
* gcm_data *my_ctx_data, context data
@@ -527,6 +542,61 @@ static int ctr_crypt(struct skcipher_request *req)
return err;
}
+static void aesni_xctr_enc_avx_tfm(struct crypto_aes_ctx *ctx, u8 *out, const u8
+ *in, unsigned int len, u8 *iv, unsigned int
+ byte_ctr)
+{
+ if (ctx->key_length == AES_KEYSIZE_128)
+ aes_xctr_enc_128_avx_by8(in, iv, (void *)ctx, out, len,
+ byte_ctr);
+ else if (ctx->key_length == AES_KEYSIZE_192)
+ aes_xctr_enc_192_avx_by8(in, iv, (void *)ctx, out, len,
+ byte_ctr);
+ else
+ aes_xctr_enc_256_avx_by8(in, iv, (void *)ctx, out, len,
+ byte_ctr);
+}
+
+static int xctr_crypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct crypto_aes_ctx *ctx = aes_ctx(crypto_skcipher_ctx(tfm));
+ u8 keystream[AES_BLOCK_SIZE];
+ u8 ctr[AES_BLOCK_SIZE];
+ struct skcipher_walk walk;
+ unsigned int nbytes;
+ unsigned int byte_ctr = 0;
+ int err;
+ __le32 ctr32;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ while ((nbytes = walk.nbytes) > 0) {
+ kernel_fpu_begin();
+ if (nbytes & AES_BLOCK_MASK)
+ static_call(aesni_xctr_enc_tfm)(ctx, walk.dst.virt.addr,
+ walk.src.virt.addr, nbytes & AES_BLOCK_MASK,
+ walk.iv, byte_ctr);
+ nbytes &= ~AES_BLOCK_MASK;
+ byte_ctr += walk.nbytes - nbytes;
+
+ if (walk.nbytes == walk.total && nbytes > 0) {
+ ctr32 = cpu_to_le32(byte_ctr / AES_BLOCK_SIZE + 1);
+ memcpy(ctr, walk.iv, AES_BLOCK_SIZE);
+ crypto_xor(ctr, (u8 *)&ctr32, sizeof(ctr32));
+ aesni_enc(ctx, keystream, ctr);
+ crypto_xor_cpy(walk.dst.virt.addr + walk.nbytes -
+ nbytes, walk.src.virt.addr + walk.nbytes
+ - nbytes, keystream, nbytes);
+ byte_ctr += nbytes;
+ nbytes = 0;
+ }
+ kernel_fpu_end();
+ err = skcipher_walk_done(&walk, nbytes);
+ }
+ return err;
+}
+
static int
rfc4106_set_hash_subkey(u8 *hash_subkey, const u8 *key, unsigned int key_len)
{
@@ -1026,6 +1096,23 @@ static struct skcipher_alg aesni_skciphers[] = {
.setkey = aesni_skcipher_setkey,
.encrypt = ctr_crypt,
.decrypt = ctr_crypt,
+ }, {
+ .base = {
+ .cra_name = "__xctr(aes)",
+ .cra_driver_name = "__xctr-aes-aesni",
+ .cra_priority = 400,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = 1,
+ .cra_ctxsize = CRYPTO_AES_CTX_SIZE,
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ .setkey = aesni_skcipher_setkey,
+ .encrypt = xctr_crypt,
+ .decrypt = xctr_crypt,
#endif
}, {
.base = {
@@ -1162,6 +1249,8 @@ static int __init aesni_init(void)
/* optimize performance of ctr mode encryption transform */
static_call_update(aesni_ctr_enc_tfm, aesni_ctr_enc_avx_tfm);
pr_info("AES CTR mode by8 optimization enabled\n");
+ static_call_update(aesni_xctr_enc_tfm, aesni_xctr_enc_avx_tfm);
+ pr_info("AES XCTR mode by8 optimization enabled\n");
}
#endif
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 0dedba74db4a..aa06af0e0ebe 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1161,7 +1161,7 @@ config CRYPTO_AES_NI_INTEL
In addition to AES cipher algorithm support, the acceleration
for some popular block cipher mode is supported too, including
ECB, CBC, LRW, XTS. The 64 bit version has additional
- acceleration for CTR.
+ acceleration for CTR and XCTR.
config CRYPTO_AES_SPARC64
tristate "AES cipher algorithms (SPARC64)"
--
2.35.1.723.g4982287a31-goog
Add hardware accelerated version of POLYVAL for x86-64 CPUs with
PCLMULQDQ support.
This implementation is accelerated using PCLMULQDQ instructions to
perform the finite field computations. For added efficiency, 8 blocks
of the message are processed simultaneously by precomputing the first
8 powers of the key.
Schoolbook multiplication is used instead of Karatsuba multiplication
because it was found to be slightly faster on x86-64 machines.
Montgomery reduction must be used instead of Barrett reduction due to
the difference in modulus between POLYVAL's field and other finite
fields.
More information on POLYVAL can be found in the HCTR2 paper:
Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf
Signed-off-by: Nathan Huckleberry <[email protected]>
---
arch/x86/crypto/Makefile | 3 +
arch/x86/crypto/polyval-clmulni_asm.S | 376 +++++++++++++++++++++++++
arch/x86/crypto/polyval-clmulni_glue.c | 361 ++++++++++++++++++++++++
crypto/Kconfig | 10 +
4 files changed, 750 insertions(+)
create mode 100644 arch/x86/crypto/polyval-clmulni_asm.S
create mode 100644 arch/x86/crypto/polyval-clmulni_glue.c
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 2831685adf6f..b9847152acd8 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -69,6 +69,9 @@ libblake2s-x86_64-y := blake2s-core.o blake2s-glue.o
obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
+obj-$(CONFIG_CRYPTO_POLYVAL_CLMUL_NI) += polyval-clmulni.o
+polyval-clmulni-y := polyval-clmulni_asm.o polyval-clmulni_glue.o
+
obj-$(CONFIG_CRYPTO_CRC32C_INTEL) += crc32c-intel.o
crc32c-intel-y := crc32c-intel_glue.o
crc32c-intel-$(CONFIG_64BIT) += crc32c-pcl-intel-asm_64.o
diff --git a/arch/x86/crypto/polyval-clmulni_asm.S b/arch/x86/crypto/polyval-clmulni_asm.S
new file mode 100644
index 000000000000..ad7126d9f0ff
--- /dev/null
+++ b/arch/x86/crypto/polyval-clmulni_asm.S
@@ -0,0 +1,376 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2021 Google LLC
+ */
+/*
+ * This is an efficient implementation of POLYVAL using intel PCLMULQDQ-NI
+ * instructions. It works on 8 blocks at a time, by precomputing the first 8
+ * keys powers h^8, ..., h^1 in the POLYVAL finite field. This precomputation
+ * allows us to split finite field multiplication into two steps.
+ *
+ * In the first step, we consider h^i, m_i as normal polynomials of degree less
+ * than 128. We then compute p(x) = h^8m_0 + ... + h^1m_7 where multiplication
+ * is simply polynomial multiplication.
+ *
+ * In the second step, we compute the reduction of p(x) modulo the finite field
+ * modulus g(x) = x^128 + x^127 + x^126 + x^121 + 1.
+ *
+ * This two step process is equivalent to computing h^8m_0 + ... + h^1m_7 where
+ * multiplication is finite field multiplication. The advantage is that the
+ * two-step process only requires 1 finite field reduction for every 8
+ * polynomial multiplications. Further parallelism is gained by interleaving the
+ * multiplications and polynomial reductions.
+ */
+
+#include <linux/linkage.h>
+#include <asm/frame.h>
+
+#define NUM_PRECOMPUTE_POWERS 8
+
+#define GSTAR %xmm7
+#define PL %xmm8
+#define PH %xmm9
+#define T %xmm10
+#define V %xmm11
+#define LO %xmm12
+#define HI %xmm13
+#define MI %xmm14
+#define SUM %xmm15
+
+#define BLOCKS_LEFT %rdx
+#define MSG %rdi
+#define KEY_POWERS %r10
+#define IDX %r11
+#define TMP %rax
+
+.section .rodata.cst16.gstar, "aM", @progbits, 16
+.align 16
+
+.Lgstar:
+ .quad 0xc200000000000000, 0xc200000000000000
+
+.text
+
+/*
+ * Performs schoolbook1_iteration on two lists of 128-bit polynomials of length
+ * b pointed to by MSG and KEY_POWERS.
+ */
+.macro schoolbook1 count
+ .set i, 0
+ .rept (\count)
+ schoolbook1_iteration i 0
+ .set i, (i +1)
+ .endr
+.endm
+
+/*
+ * Computes the product of two 128-bit polynomials at the memory locations
+ * specified by (MSG + 16*i) and (KEY_POWERS + 16*i) and XORs the components of the
+ * 256-bit product into LO, MI, HI.
+ *
+ * The multiplication produces four parts:
+ * LOW: The polynomial given by performing carryless multiplication of the
+ * bottom 64-bits of each polynomial
+ * MID1: The polynomial given by performing carryless multiplication of the
+ * bottom 64-bits of the first polynomial and the top 64-bits of the second
+ * MID2: The polynomial given by performing carryless multiplication of the
+ * bottom 64-bits of the second polynomial and the top 64-bits of the first
+ * HIGH: The polynomial given by performing carryless multiplication of the
+ * top 64-bits of each polynomial
+ *
+ * We compute:
+ * LO ^= LOW
+ * MI ^= MID1 ^ MID2
+ * HI ^= HIGH
+ *
+ * Later, the 256-bit result can be extracted as:
+ * [HI_H : HI_L ^ MI_H : LO_H ^ MI_L : LO_L]
+ * This step is done when computing the polynomial reduction for efficiency
+ * reasons.
+ *
+ * If xor_sum == 1, then also XOR the value of SUM into m_0. This avoids an
+ * extra multiplication of SUM and h^N.
+ */
+.macro schoolbook1_iteration i xor_sum
+ movups (16*\i)(MSG), %xmm0
+ .if (\i == 0 && \xor_sum == 1)
+ pxor SUM, %xmm0
+ .endif
+ vpclmulqdq $0x00, (16*\i)(KEY_POWERS), %xmm0, %xmm2
+ vpclmulqdq $0x01, (16*\i)(KEY_POWERS), %xmm0, %xmm1
+ vpclmulqdq $0x11, (16*\i)(KEY_POWERS), %xmm0, %xmm3
+ vpclmulqdq $0x10, (16*\i)(KEY_POWERS), %xmm0, %xmm4
+ vpxor %xmm2, LO, LO
+ vpxor %xmm1, MI, MI
+ vpxor %xmm4, MI, MI
+ vpxor %xmm3, HI, HI
+.endm
+
+/*
+ * Performs the same computation as schoolbook1_iteration, except we expect the
+ * arguments to already be loaded into xmm0 and xmm1.
+ */
+.macro schoolbook1_noload
+ vpclmulqdq $0x01, %xmm0, %xmm1, %xmm2
+ vpclmulqdq $0x00, %xmm0, %xmm1, %xmm3
+ vpclmulqdq $0x11, %xmm0, %xmm1, %xmm4
+ vpclmulqdq $0x10, %xmm0, %xmm1, %xmm5
+ vpxor %xmm2, MI, MI
+ vpxor %xmm3, LO, LO
+ vpxor %xmm5, MI, MI
+ vpxor %xmm4, HI, HI
+.endm
+
+/*
+ * Computes the 256-bit polynomial represented by LO, HI, MI. Stores
+ * the result in PL, PH.
+ * [PH :: PL] = [HI_H : HI_L ^ MI_H :: LO_H ^ MI_L : LO_L]
+ */
+.macro schoolbook2
+ vpslldq $8, MI, PL
+ vpsrldq $8, MI, PH
+ pxor LO, PL
+ pxor HI, PH
+.endm
+
+/*
+ * Computes the 128-bit reduction of PL, PH. Stores the result in PH.
+ *
+ * This macro computes p(x) mod g(x) where p(x) is in montgomery form and g(x) =
+ * x^128 + x^127 + x^126 + x^121 + 1.
+ *
+ * We have a 256-bit polynomial P_3 : P_2 : P_1 : P_0 that is the product of
+ * two 128-bit polynomials in Montgomery form. We need to reduce it mod g(x).
+ * Also, since polynomials in Montgomery form have an "extra" factor of x^128,
+ * this product has two extra factors of x^128. To get it back into Montgomery
+ * form, we need to remove one of these factors by dividing by x^128.
+ *
+ * To accomplish both of these goals, we add multiples of g(x) that cancel out
+ * the low 128 bits P_1 : P_0, leaving just the high 128 bits. Since the low
+ * bits are zero, the polynomial division by x^128 can be done by right shifting.
+ *
+ * Since the only nonzero term in the low 64 bits of g(x) is the constant term,
+ * the multiple of g(x) needed to cancel out P_0 is P_0 * g(x). The CPU can
+ * only do 64x64 bit multiplications, so split P_0 * g(x) into x^128 * P_0 +
+ * x^64 g*(x) * P_0 + P_0, where g*(x) is bits 64-127 of g(x). Adding this to
+ * the original polynomial gives P_3 : P_2 + P_0 + T_1 : P_1 + T_0 : 0, where T
+ * = T_1 : T_0 = g*(x) * P0. Thus, bits 0-63 got "folded" into bits 64-191.
+ *
+ * Repeating this same process on the next 64 bits "folds" bits 64-127 into bits
+ * 128-255, giving the answer in bits 128-255. This time, we need to cancel P_1
+ * + T_0 in bits 64-127. The multiple of g(x) required is (P_1 + T_0) * g(x) *
+ * x^64. Adding this to our previous computation gives P_3 + P_1 + T_0 + V_1 :
+ * P_2 + P_0 + T_1 + V_0 : 0 : 0, where V = V_1 : V_0 = g*(x) * (P_1 + T_0).
+ *
+ * So our final computation is:
+ * T = T_1 : T_0 = g*(x) * P_0
+ * V = V_1 : V_0 = g*(x) * (T_0 ^ P_1)
+ * p(x) / x^{128} mod g(x) = P_3 ^ P_1 ^ V_1 ^ T_0 : P_2 ^ P_0 ^ V_0 ^ T_1
+ *
+ * The implementation below saves a XOR instruction by computing P_1 ^ T_0 : P_0
+ * ^ T_1 and XORing into PH, rather than directly XORing P_1 : P_0, T_0 : T1
+ * into PH. This allows us to reuse P_1 ^ T_0 when computing V.
+ */
+.macro montgomery_reduction
+ movdqa PL, T
+ pclmulqdq $0x00, GSTAR, T # T = [P_0 * g*(x)]
+ pshufd $0b01001110, T, V # V = [T_0 : T_1]
+ pxor V, PL # PL = [P_1 ^ T_0 : P_0 ^ T_1]
+ pxor PL, PH # PH = [P_1 ^ T_0 ^ P_3 : P_0 ^ T_1 ^ P_2]
+ pclmulqdq $0x11, GSTAR, PL # PL = [(P_1 ^ T_0) * g*(x)]
+ pxor PL, PH
+.endm
+
+/*
+ * Compute schoolbook multiplication for 8 blocks
+ * m_0h^8 + ... + m_7h^1
+ *
+ * If reduce is set, also computes the montgomery reduction of the
+ * previous full_stride call and XORs with the first message block.
+ * (m_0 + REDUCE(PL, PH))h^8 + ... + m_7h^1.
+ * I.e., the first multiplication uses m_0 + REDUCE(PL, PH) instead of m_0.
+ *
+ * Sets PL, PH
+ * Clobbers LO, HI, MI
+ *
+ */
+.macro full_stride reduce
+ mov %rsi, KEY_POWERS
+ pxor LO, LO
+ pxor HI, HI
+ pxor MI, MI
+
+ schoolbook1_iteration 7 0
+ .if (\reduce)
+ movdqa PL, T
+ .endif
+
+ schoolbook1_iteration 6 0
+ .if (\reduce)
+ pclmulqdq $0x00, GSTAR, T # T = [X0 * g*(x)]
+ .endif
+
+ schoolbook1_iteration 5 0
+ .if (\reduce)
+ pshufd $0b01001110, T, V # V = [T0 : T1]
+ .endif
+
+ schoolbook1_iteration 4 0
+ .if (\reduce)
+ pxor V, PL # PL = [X1 ^ T0 : X0 ^ T1]
+ .endif
+
+ schoolbook1_iteration 3 0
+ .if (\reduce)
+ pxor PL, PH # PH = [X1 ^ T0 ^ X3 : X0 ^ T1 ^ X2]
+ .endif
+
+ schoolbook1_iteration 2 0
+ .if (\reduce)
+ pclmulqdq $0x11, GSTAR, PL # PL = [X1 ^ T0 * g*(x)]
+ .endif
+
+ schoolbook1_iteration 1 0
+ .if (\reduce)
+ pxor PL, PH
+ movdqa PH, SUM
+ .endif
+
+ schoolbook1_iteration 0 1
+
+ addq $(8*16), MSG
+ addq $(8*16), KEY_POWERS
+ schoolbook2
+.endm
+
+/*
+ * Compute poly on window size of %rdx blocks
+ * 0 < %rdx < NUM_PRECOMPUTE_POWERS
+ */
+.macro partial_stride
+ pxor LO, LO
+ pxor HI, HI
+ pxor MI, MI
+ mov BLOCKS_LEFT, TMP
+ shlq $4, TMP
+ mov %rsi, KEY_POWERS
+ addq $(16*NUM_PRECOMPUTE_POWERS), KEY_POWERS
+ subq TMP, KEY_POWERS
+ # Multiply sum by h^N
+ movups (KEY_POWERS), %xmm0
+ movdqa SUM, %xmm1
+ schoolbook1_noload
+ schoolbook2
+ montgomery_reduction
+ movdqa PH, SUM
+ pxor LO, LO
+ pxor HI, HI
+ pxor MI, MI
+ xor IDX, IDX
+.LloopPartial:
+ cmpq BLOCKS_LEFT, IDX # IDX < rdx
+ jae .LloopExitPartial
+
+ movq BLOCKS_LEFT, TMP
+ subq IDX, TMP # TMP = rdx - IDX
+
+ cmp $4, TMP # TMP < 4 ?
+ jl .Llt4Partial
+ schoolbook1 4
+ addq $4, IDX
+ addq $(4*16), MSG
+ addq $(4*16), KEY_POWERS
+ jmp .LoutPartial
+.Llt4Partial:
+ cmp $3, TMP # TMP < 3 ?
+ jl .Llt3Partial
+ schoolbook1 3
+ addq $3, IDX
+ addq $(3*16), MSG
+ addq $(3*16), KEY_POWERS
+ jmp .LoutPartial
+.Llt3Partial:
+ cmp $2, TMP # TMP < 2 ?
+ jl .Llt2Partial
+ schoolbook1 2
+ addq $2, IDX
+ addq $(2*16), MSG
+ addq $(2*16), KEY_POWERS
+ jmp .LoutPartial
+.Llt2Partial:
+ schoolbook1 1 # TMP < 1 ?
+ addq $1, IDX
+ addq $(1*16), MSG
+ addq $(1*16), KEY_POWERS
+.LoutPartial:
+ jmp .LloopPartial
+.LloopExitPartial:
+ schoolbook2
+ montgomery_reduction
+ pxor PH, SUM
+.endm
+
+/*
+ * Perform montgomery multiplication in GF(2^128) and store result in op1.
+ *
+ * Computes op1*op2*x^{-128} mod x^128 + x^127 + x^126 + x^121 + 1
+ * If op1, op2 are in montgomery form, this computes the montgomery
+ * form of op1*op2.
+ *
+ * void clmul_polyval_mul(u8 *op1, const u8 *op2);
+ */
+SYM_FUNC_START(clmul_polyval_mul)
+ FRAME_BEGIN
+ vmovdqa .Lgstar(%rip), GSTAR
+ pxor LO, LO
+ pxor HI, HI
+ pxor MI, MI
+ movups (%rdi), %xmm0
+ movups (%rsi), %xmm1
+ schoolbook1_noload
+ schoolbook2
+ montgomery_reduction
+ movups PH, (%rdi)
+ FRAME_END
+ ret
+SYM_FUNC_END(clmul_polyval_mul)
+
+/*
+ * Perform polynomial evaluation as specified by POLYVAL. This computes:
+ * h^n * accumulator + h^n * m_0 + ... + h^1 * m_{n-1}
+ * where n=nblocks, h is the hash key, and m_i are the message blocks.
+ *
+ * rdi - pointer to message blocks
+ * rsi - pointer to precomputed key powers h^8 ... h^1
+ * rdx - number of blocks to hash
+ * rcx - pointer to the accumulator
+ *
+ * void clmul_polyval_update(const u8 *in, const struct polyval_ctx *ctx,
+ * size_t nblocks, u8 *accumulator);
+ */
+SYM_FUNC_START(clmul_polyval_update)
+ FRAME_BEGIN
+ vmovdqa .Lgstar(%rip), GSTAR
+ movups (%rcx), SUM
+ cmpq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
+ jb .LstrideLoopExit
+ full_stride 0
+ subq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
+.LstrideLoop:
+ cmpq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
+ jb .LstrideLoopExitReduce
+ full_stride 1
+ subq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
+ jmp .LstrideLoop
+.LstrideLoopExitReduce:
+ montgomery_reduction
+ movdqa PH, SUM
+.LstrideLoopExit:
+ test BLOCKS_LEFT, BLOCKS_LEFT
+ je .LskipPartial
+ partial_stride
+.LskipPartial:
+ movups SUM, (%rcx)
+ FRAME_END
+ ret
+SYM_FUNC_END(clmul_polyval_update)
diff --git a/arch/x86/crypto/polyval-clmulni_glue.c b/arch/x86/crypto/polyval-clmulni_glue.c
new file mode 100644
index 000000000000..ae73750ba059
--- /dev/null
+++ b/arch/x86/crypto/polyval-clmulni_glue.c
@@ -0,0 +1,361 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Accelerated POLYVAL implementation with Intel PCLMULQDQ-NI
+ * instructions. This file contains glue code.
+ *
+ * Copyright (c) 2007 Nokia Siemens Networks - Mikko Herranen <[email protected]>
+ * Copyright (c) 2009 Intel Corp.
+ * Author: Huang Ying <[email protected]>
+ * Copyright 2021 Google LLC
+ */
+/*
+ * Glue code based on ghash-clmulni-intel_glue.c.
+ *
+ * This implementation of POLYVAL uses montgomery multiplication
+ * accelerated by PCLMULQDQ-NI to implement the finite field
+ * operations.
+ *
+ */
+
+#include <crypto/algapi.h>
+#include <crypto/cryptd.h>
+#include <crypto/gf128mul.h>
+#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
+#include <crypto/polyval.h>
+#include <linux/crypto.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <asm/cpu_device_id.h>
+#include <asm/simd.h>
+
+#define NUM_PRECOMPUTE_POWERS 8
+
+struct polyval_async_ctx {
+ struct cryptd_ahash *cryptd_tfm;
+};
+
+struct polyval_ctx {
+ /*
+ * These powers must be in the order h^8, ..., h^1.
+ */
+ u8 key_powers[NUM_PRECOMPUTE_POWERS][POLYVAL_BLOCK_SIZE];
+};
+
+struct polyval_desc_ctx {
+ u8 buffer[POLYVAL_BLOCK_SIZE];
+ u32 bytes;
+};
+
+asmlinkage void clmul_polyval_update(const u8 *in, struct polyval_ctx *keys,
+ size_t nblocks, u8 *accumulator);
+asmlinkage void clmul_polyval_mul(u8 *op1, const u8 *op2);
+
+static int polyval_init(struct shash_desc *desc)
+{
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+
+ memset(dctx, 0, sizeof(*dctx));
+
+ return 0;
+}
+
+static int polyval_setkey(struct crypto_shash *tfm,
+ const u8 *key, unsigned int keylen)
+{
+ struct polyval_ctx *ctx = crypto_shash_ctx(tfm);
+ int i;
+
+ if (keylen != POLYVAL_BLOCK_SIZE)
+ return -EINVAL;
+
+ memcpy(ctx->key_powers[NUM_PRECOMPUTE_POWERS-1], key,
+ POLYVAL_BLOCK_SIZE);
+
+ kernel_fpu_begin();
+ for (i = NUM_PRECOMPUTE_POWERS-2; i >= 0; i--) {
+ memcpy(ctx->key_powers[i], key, POLYVAL_BLOCK_SIZE);
+ clmul_polyval_mul(ctx->key_powers[i], ctx->key_powers[i+1]);
+ }
+ kernel_fpu_end();
+
+ return 0;
+}
+
+static int polyval_update(struct shash_desc *desc,
+ const u8 *src, unsigned int srclen)
+{
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+ struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm);
+ u8 *pos;
+ unsigned int nblocks;
+ int n;
+
+ kernel_fpu_begin();
+ if (dctx->bytes) {
+ n = min(srclen, dctx->bytes);
+ pos = dctx->buffer + POLYVAL_BLOCK_SIZE - dctx->bytes;
+
+ dctx->bytes -= n;
+ srclen -= n;
+
+ while (n--)
+ *pos++ ^= *src++;
+
+ if (!dctx->bytes)
+ clmul_polyval_mul(dctx->buffer,
+ ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]);
+ }
+
+ nblocks = srclen/POLYVAL_BLOCK_SIZE;
+ clmul_polyval_update(src, ctx, nblocks, dctx->buffer);
+ srclen -= nblocks*POLYVAL_BLOCK_SIZE;
+ kernel_fpu_end();
+
+ if (srclen) {
+ dctx->bytes = POLYVAL_BLOCK_SIZE - srclen;
+ src += nblocks*POLYVAL_BLOCK_SIZE;
+ pos = dctx->buffer;
+ while (srclen--)
+ *pos++ ^= *src++;
+ }
+
+ return 0;
+}
+
+static int polyval_final(struct shash_desc *desc, u8 *dst)
+{
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+ struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm);
+
+ if (dctx->bytes) {
+ kernel_fpu_begin();
+ clmul_polyval_mul(dctx->buffer,
+ ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]);
+ kernel_fpu_end();
+ }
+
+ dctx->bytes = 0;
+ memcpy(dst, dctx->buffer, POLYVAL_BLOCK_SIZE);
+
+ return 0;
+}
+
+static struct shash_alg polyval_alg = {
+ .digestsize = POLYVAL_DIGEST_SIZE,
+ .init = polyval_init,
+ .update = polyval_update,
+ .final = polyval_final,
+ .setkey = polyval_setkey,
+ .descsize = sizeof(struct polyval_desc_ctx),
+ .base = {
+ .cra_name = "__polyval",
+ .cra_driver_name = "__polyval-clmulni",
+ .cra_priority = 0,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = POLYVAL_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct polyval_ctx),
+ .cra_module = THIS_MODULE,
+ },
+};
+
+static int polyval_async_init(struct ahash_request *req)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
+
+ desc->tfm = child;
+ return crypto_shash_init(desc);
+}
+
+static int polyval_async_update(struct ahash_request *req)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc;
+
+ if (!crypto_simd_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
+ memcpy(cryptd_req, req, sizeof(*req));
+ ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
+ return crypto_ahash_update(cryptd_req);
+ }
+ desc = cryptd_shash_desc(cryptd_req);
+
+ return shash_ahash_update(req, desc);
+}
+
+static int polyval_async_final(struct ahash_request *req)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc;
+
+ if (!crypto_simd_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
+ memcpy(cryptd_req, req, sizeof(*req));
+ ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
+ return crypto_ahash_final(cryptd_req);
+ }
+ desc = cryptd_shash_desc(cryptd_req);
+
+ return crypto_shash_final(desc, req->result);
+}
+
+static int polyval_async_import(struct ahash_request *req, const void *in)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+
+ polyval_async_init(req);
+ memcpy(dctx, in, sizeof(*dctx));
+ return 0;
+
+}
+
+static int polyval_async_export(struct ahash_request *req, void *out)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+
+ memcpy(out, dctx, sizeof(*dctx));
+ return 0;
+
+}
+
+static int polyval_async_digest(struct ahash_request *req)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc;
+ struct crypto_shash *child;
+
+ if (!crypto_simd_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
+ memcpy(cryptd_req, req, sizeof(*req));
+ ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
+ return crypto_ahash_digest(cryptd_req);
+ }
+ desc = cryptd_shash_desc(cryptd_req);
+ child = cryptd_ahash_child(cryptd_tfm);
+
+ desc->tfm = child;
+ return shash_ahash_digest(req, desc);
+}
+
+static int polyval_async_setkey(struct crypto_ahash *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct crypto_ahash *child = &ctx->cryptd_tfm->base;
+
+ crypto_ahash_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+ crypto_ahash_set_flags(child, crypto_ahash_get_flags(tfm)
+ & CRYPTO_TFM_REQ_MASK);
+ return crypto_ahash_setkey(child, key, keylen);
+}
+
+static int polyval_async_init_tfm(struct crypto_tfm *tfm)
+{
+ struct cryptd_ahash *cryptd_tfm;
+ struct polyval_async_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ cryptd_tfm = cryptd_alloc_ahash("__polyval-clmulni",
+ CRYPTO_ALG_INTERNAL,
+ CRYPTO_ALG_INTERNAL);
+ if (IS_ERR(cryptd_tfm))
+ return PTR_ERR(cryptd_tfm);
+ ctx->cryptd_tfm = cryptd_tfm;
+ crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
+ sizeof(struct ahash_request) +
+ crypto_ahash_reqsize(&cryptd_tfm->base));
+
+ return 0;
+}
+
+static void polyval_async_exit_tfm(struct crypto_tfm *tfm)
+{
+ struct polyval_async_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ cryptd_free_ahash(ctx->cryptd_tfm);
+}
+
+static struct ahash_alg polyval_async_alg = {
+ .init = polyval_async_init,
+ .update = polyval_async_update,
+ .final = polyval_async_final,
+ .setkey = polyval_async_setkey,
+ .digest = polyval_async_digest,
+ .export = polyval_async_export,
+ .import = polyval_async_import,
+ .halg = {
+ .digestsize = POLYVAL_DIGEST_SIZE,
+ .statesize = sizeof(struct polyval_desc_ctx),
+ .base = {
+ .cra_name = "polyval",
+ .cra_driver_name = "polyval-clmulni",
+ .cra_priority = 200,
+ .cra_ctxsize = sizeof(struct polyval_async_ctx),
+ .cra_flags = CRYPTO_ALG_ASYNC,
+ .cra_blocksize = POLYVAL_BLOCK_SIZE,
+ .cra_module = THIS_MODULE,
+ .cra_init = polyval_async_init_tfm,
+ .cra_exit = polyval_async_exit_tfm,
+ },
+ },
+};
+
+static const struct x86_cpu_id pcmul_cpu_id[] = {
+ X86_MATCH_FEATURE(X86_FEATURE_PCLMULQDQ, NULL), /* Pickle-Mickle-Duck */
+ {}
+};
+MODULE_DEVICE_TABLE(x86cpu, pcmul_cpu_id);
+
+static int __init polyval_clmulni_mod_init(void)
+{
+ int err;
+
+ if (!x86_match_cpu(pcmul_cpu_id))
+ return -ENODEV;
+
+ err = crypto_register_shash(&polyval_alg);
+ if (err)
+ goto err_out;
+ err = crypto_register_ahash(&polyval_async_alg);
+ if (err)
+ goto err_shash;
+
+ return 0;
+
+err_shash:
+ crypto_unregister_shash(&polyval_alg);
+err_out:
+ return err;
+}
+
+static void __exit polyval_clmulni_mod_exit(void)
+{
+ crypto_unregister_ahash(&polyval_async_alg);
+ crypto_unregister_shash(&polyval_alg);
+}
+
+module_init(polyval_clmulni_mod_init);
+module_exit(polyval_clmulni_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("POLYVAL hash function accelerated by PCLMULQDQ-NI");
+MODULE_ALIAS_CRYPTO("polyval");
+MODULE_ALIAS_CRYPTO("polyval-clmulni");
diff --git a/crypto/Kconfig b/crypto/Kconfig
index aa06af0e0ebe..c6aec88213b1 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -787,6 +787,16 @@ config CRYPTO_POLYVAL
POLYVAL is the hash function used in HCTR2. It is not a general-purpose
cryptographic hash function.
+config CRYPTO_POLYVAL_CLMUL_NI
+ tristate "POLYVAL hash function (CLMUL-NI accelerated)"
+ depends on X86 && 64BIT
+ select CRYPTO_CRYPTD
+ select CRYPTO_POLYVAL
+ help
+ This is the x86_64 CLMUL-NI accelerated implementation of POLYVAL. It is
+ used to efficiently implement HCTR2 on x86-64 processors that support
+ carry-less multiplication instructions.
+
config CRYPTO_POLY1305
tristate "Poly1305 authenticator algorithm"
select CRYPTO_HASH
--
2.35.1.723.g4982287a31-goog
Add hardware accelerated version of XCTR for ARM64 CPUs with ARMv8
Crypto Extension support. This XCTR implementation is based on the CTR
implementation in aes-modes.S.
More information on XCTR can be found in
the HCTR2 paper: Length-preserving encryption with HCTR2:
https://eprint.iacr.org/2021/1441.pdf
Signed-off-by: Nathan Huckleberry <[email protected]>
---
arch/arm64/crypto/Kconfig | 4 +-
arch/arm64/crypto/aes-glue.c | 65 ++++++++++++++++-
arch/arm64/crypto/aes-modes.S | 134 ++++++++++++++++++++++++++++++++++
3 files changed, 199 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig
index 2a965aa0188d..897f9a4b5b67 100644
--- a/arch/arm64/crypto/Kconfig
+++ b/arch/arm64/crypto/Kconfig
@@ -84,13 +84,13 @@ config CRYPTO_AES_ARM64_CE_CCM
select CRYPTO_LIB_AES
config CRYPTO_AES_ARM64_CE_BLK
- tristate "AES in ECB/CBC/CTR/XTS modes using ARMv8 Crypto Extensions"
+ tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using ARMv8 Crypto Extensions"
depends on KERNEL_MODE_NEON
select CRYPTO_SKCIPHER
select CRYPTO_AES_ARM64_CE
config CRYPTO_AES_ARM64_NEON_BLK
- tristate "AES in ECB/CBC/CTR/XTS modes using NEON instructions"
+ tristate "AES in ECB/CBC/CTR/XTS/XCTR modes using NEON instructions"
depends on KERNEL_MODE_NEON
select CRYPTO_SKCIPHER
select CRYPTO_LIB_AES
diff --git a/arch/arm64/crypto/aes-glue.c b/arch/arm64/crypto/aes-glue.c
index 561dd2332571..06ebd466cf7c 100644
--- a/arch/arm64/crypto/aes-glue.c
+++ b/arch/arm64/crypto/aes-glue.c
@@ -34,10 +34,11 @@
#define aes_essiv_cbc_encrypt ce_aes_essiv_cbc_encrypt
#define aes_essiv_cbc_decrypt ce_aes_essiv_cbc_decrypt
#define aes_ctr_encrypt ce_aes_ctr_encrypt
+#define aes_xctr_encrypt ce_aes_xctr_encrypt
#define aes_xts_encrypt ce_aes_xts_encrypt
#define aes_xts_decrypt ce_aes_xts_decrypt
#define aes_mac_update ce_aes_mac_update
-MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions");
+MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 Crypto Extensions");
#else
#define MODE "neon"
#define PRIO 200
@@ -50,16 +51,18 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 Crypto Extensions");
#define aes_essiv_cbc_encrypt neon_aes_essiv_cbc_encrypt
#define aes_essiv_cbc_decrypt neon_aes_essiv_cbc_decrypt
#define aes_ctr_encrypt neon_aes_ctr_encrypt
+#define aes_xctr_encrypt neon_aes_xctr_encrypt
#define aes_xts_encrypt neon_aes_xts_encrypt
#define aes_xts_decrypt neon_aes_xts_decrypt
#define aes_mac_update neon_aes_mac_update
-MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS using ARMv8 NEON");
+MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 NEON");
#endif
#if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS)
MODULE_ALIAS_CRYPTO("ecb(aes)");
MODULE_ALIAS_CRYPTO("cbc(aes)");
MODULE_ALIAS_CRYPTO("ctr(aes)");
MODULE_ALIAS_CRYPTO("xts(aes)");
+MODULE_ALIAS_CRYPTO("xctr(aes)");
#endif
MODULE_ALIAS_CRYPTO("cts(cbc(aes))");
MODULE_ALIAS_CRYPTO("essiv(cbc(aes),sha256)");
@@ -89,6 +92,10 @@ asmlinkage void aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[],
asmlinkage void aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[]);
+asmlinkage void aes_xctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
+ int rounds, int bytes, u8 ctr[], u8 finalbuf[],
+ int byte_ctr);
+
asmlinkage void aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[], u8 iv[],
int first);
@@ -442,6 +449,44 @@ static int __maybe_unused essiv_cbc_decrypt(struct skcipher_request *req)
return err ?: cbc_decrypt_walk(req, &walk);
}
+static int __maybe_unused xctr_encrypt(struct skcipher_request *req)
+{
+ struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
+ struct crypto_aes_ctx *ctx = crypto_skcipher_ctx(tfm);
+ int err, rounds = 6 + ctx->key_length / 4;
+ struct skcipher_walk walk;
+ unsigned int byte_ctr = 0;
+
+ err = skcipher_walk_virt(&walk, req, false);
+
+ while (walk.nbytes > 0) {
+ const u8 *src = walk.src.virt.addr;
+ unsigned int nbytes = walk.nbytes;
+ u8 *dst = walk.dst.virt.addr;
+ u8 buf[AES_BLOCK_SIZE];
+
+ if (unlikely(nbytes < AES_BLOCK_SIZE))
+ src = dst = memcpy(buf + sizeof(buf) - nbytes,
+ src, nbytes);
+ else if (nbytes < walk.total)
+ nbytes &= ~(AES_BLOCK_SIZE - 1);
+
+ kernel_neon_begin();
+ aes_xctr_encrypt(dst, src, ctx->key_enc, rounds, nbytes,
+ walk.iv, buf, byte_ctr);
+ kernel_neon_end();
+
+ if (unlikely(nbytes < AES_BLOCK_SIZE))
+ memcpy(walk.dst.virt.addr,
+ buf + sizeof(buf) - nbytes, nbytes);
+ byte_ctr += nbytes;
+
+ err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
+ }
+
+ return err;
+}
+
static int __maybe_unused ctr_encrypt(struct skcipher_request *req)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
@@ -669,6 +714,22 @@ static struct skcipher_alg aes_algs[] = { {
.setkey = skcipher_aes_setkey,
.encrypt = ctr_encrypt,
.decrypt = ctr_encrypt,
+}, {
+ .base = {
+ .cra_name = "xctr(aes)",
+ .cra_driver_name = "xctr-aes-" MODE,
+ .cra_priority = PRIO,
+ .cra_blocksize = 1,
+ .cra_ctxsize = sizeof(struct crypto_aes_ctx),
+ .cra_module = THIS_MODULE,
+ },
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ .setkey = skcipher_aes_setkey,
+ .encrypt = xctr_encrypt,
+ .decrypt = xctr_encrypt,
}, {
.base = {
.cra_name = "xts(aes)",
diff --git a/arch/arm64/crypto/aes-modes.S b/arch/arm64/crypto/aes-modes.S
index dc35eb0245c5..ac37e2f7ca84 100644
--- a/arch/arm64/crypto/aes-modes.S
+++ b/arch/arm64/crypto/aes-modes.S
@@ -479,6 +479,140 @@ ST5( mov v3.16b, v4.16b )
b .Lctrout
AES_FUNC_END(aes_ctr_encrypt)
+ /*
+ * aes_xctr_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
+ * int bytes, u8 const ctr[], u8 finalbuf[], int
+ * byte_ctr)
+ */
+
+AES_FUNC_START(aes_xctr_encrypt)
+ stp x29, x30, [sp, #-16]!
+ mov x29, sp
+
+ enc_prepare w3, x2, x12
+ ld1 {vctr.16b}, [x5]
+
+ umov x12, vctr.d[0] /* keep ctr in reg */
+ lsr x7, x7, #4
+ add x11, x7, #1
+
+.LxctrloopNx:
+ add w7, w4, #15
+ sub w4, w4, #MAX_STRIDE << 4
+ lsr w7, w7, #4
+ mov w8, #MAX_STRIDE
+ cmp w7, w8
+ csel w7, w7, w8, lt
+ add x11, x11, x7
+
+ mov v0.16b, vctr.16b
+ mov v1.16b, vctr.16b
+ mov v2.16b, vctr.16b
+ mov v3.16b, vctr.16b
+ST5( mov v4.16b, vctr.16b )
+
+ sub x7, x11, #MAX_STRIDE
+ eor x7, x12, x7
+ ins v0.d[0], x7
+ sub x7, x11, #MAX_STRIDE - 1
+ sub x8, x11, #MAX_STRIDE - 2
+ eor x7, x7, x12
+ sub x9, x11, #MAX_STRIDE - 3
+ mov v1.d[0], x7
+ eor x8, x8, x12
+ eor x9, x9, x12
+ST5( sub x10, x11, #MAX_STRIDE - 4)
+ mov v2.d[0], x8
+ eor x10, x10, x12
+ mov v3.d[0], x9
+ST5( mov v4.d[0], x10 )
+ tbnz w4, #31, .Lxctrtail
+ ld1 {v5.16b-v7.16b}, [x1], #48
+ST4( bl aes_encrypt_block4x )
+ST5( bl aes_encrypt_block5x )
+ eor v0.16b, v5.16b, v0.16b
+ST4( ld1 {v5.16b}, [x1], #16 )
+ eor v1.16b, v6.16b, v1.16b
+ST5( ld1 {v5.16b-v6.16b}, [x1], #32 )
+ eor v2.16b, v7.16b, v2.16b
+ eor v3.16b, v5.16b, v3.16b
+ST5( eor v4.16b, v6.16b, v4.16b )
+ st1 {v0.16b-v3.16b}, [x0], #64
+ST5( st1 {v4.16b}, [x0], #16 )
+ cbz w4, .Lxctrout
+ b .LxctrloopNx
+
+.Lxctrout:
+ ldp x29, x30, [sp], #16
+ ret
+
+.Lxctrtail:
+ /* XOR up to MAX_STRIDE * 16 - 1 bytes of in/output with v0 ... v3/v4 */
+ mov x16, #16
+ ands x6, x4, #0xf
+ csel x13, x6, x16, ne
+
+ST5( cmp w4, #64 - (MAX_STRIDE << 4) )
+ST5( csel x14, x16, xzr, gt )
+ cmp w4, #48 - (MAX_STRIDE << 4)
+ csel x15, x16, xzr, gt
+ cmp w4, #32 - (MAX_STRIDE << 4)
+ csel x16, x16, xzr, gt
+ cmp w4, #16 - (MAX_STRIDE << 4)
+
+ adr_l x12, .Lcts_permute_table
+ add x12, x12, x13
+ ble .Lctrtail1x
+
+ST5( ld1 {v5.16b}, [x1], x14 )
+ ld1 {v6.16b}, [x1], x15
+ ld1 {v7.16b}, [x1], x16
+
+ST4( bl aes_encrypt_block4x )
+ST5( bl aes_encrypt_block5x )
+
+ ld1 {v8.16b}, [x1], x13
+ ld1 {v9.16b}, [x1]
+ ld1 {v10.16b}, [x12]
+
+ST4( eor v6.16b, v6.16b, v0.16b )
+ST4( eor v7.16b, v7.16b, v1.16b )
+ST4( tbl v3.16b, {v3.16b}, v10.16b )
+ST4( eor v8.16b, v8.16b, v2.16b )
+ST4( eor v9.16b, v9.16b, v3.16b )
+
+ST5( eor v5.16b, v5.16b, v0.16b )
+ST5( eor v6.16b, v6.16b, v1.16b )
+ST5( tbl v4.16b, {v4.16b}, v10.16b )
+ST5( eor v7.16b, v7.16b, v2.16b )
+ST5( eor v8.16b, v8.16b, v3.16b )
+ST5( eor v9.16b, v9.16b, v4.16b )
+
+ST5( st1 {v5.16b}, [x0], x14 )
+ st1 {v6.16b}, [x0], x15
+ st1 {v7.16b}, [x0], x16
+ add x13, x13, x0
+ st1 {v9.16b}, [x13] // overlapping stores
+ st1 {v8.16b}, [x0]
+ b .Lctrout
+.Lxctrtail1x:
+ sub x7, x6, #16
+ csel x6, x6, x7, eq
+ add x1, x1, x6
+ add x0, x0, x6
+ ld1 {v5.16b}, [x1]
+ ld1 {v6.16b}, [x0]
+ST5( mov v3.16b, v4.16b )
+ encrypt_block v3, w3, x2, x8, w7
+ ld1 {v10.16b-v11.16b}, [x12]
+ tbl v3.16b, {v3.16b}, v10.16b
+ sshr v11.16b, v11.16b, #7
+ eor v5.16b, v5.16b, v3.16b
+ bif v5.16b, v6.16b, v11.16b
+ st1 {v5.16b}, [x0]
+ b .Lctrout
+AES_FUNC_END(aes_xctr_encrypt)
+
/*
* aes_xts_encrypt(u8 out[], u8 const in[], u8 const rk1[], int rounds,
--
2.35.1.723.g4982287a31-goog
HCTR2 is a tweakable, length-preserving encryption mode. It has the
same security guarantees as Adiantum, but is intended for use on CPUs
with dedicated crypto instructions. It fixes a known weakness with
filename encryption: when two filenames in the same directory share a
prefix of >= 16 bytes, with CTS-CBC their encrypted filenames share a
common substring, leaking information. HCTR2 does not have this
problem.
More information on HCTR2 can be found here: Length-preserving
encryption with HCTR2: https://eprint.iacr.org/2021/1441.pdf
Signed-off-by: Nathan Huckleberry <[email protected]>
---
Documentation/filesystems/fscrypt.rst | 19 ++++++++++++++-----
fs/crypto/fscrypt_private.h | 2 +-
fs/crypto/keysetup.c | 7 +++++++
fs/crypto/policy.c | 4 ++++
include/uapi/linux/fscrypt.h | 3 ++-
tools/include/uapi/linux/fscrypt.h | 3 ++-
6 files changed, 30 insertions(+), 8 deletions(-)
diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst
index 4d5d50dca65c..09915086abd8 100644
--- a/Documentation/filesystems/fscrypt.rst
+++ b/Documentation/filesystems/fscrypt.rst
@@ -337,6 +337,7 @@ Currently, the following pairs of encryption modes are supported:
- AES-256-XTS for contents and AES-256-CTS-CBC for filenames
- AES-128-CBC for contents and AES-128-CTS-CBC for filenames
- Adiantum for both contents and filenames
+- AES-256-XTS for contents and AES-256-HCTR2 for filenames
If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
@@ -357,6 +358,14 @@ To use Adiantum, CONFIG_CRYPTO_ADIANTUM must be enabled. Also, fast
implementations of ChaCha and NHPoly1305 should be enabled, e.g.
CONFIG_CRYPTO_CHACHA20_NEON and CONFIG_CRYPTO_NHPOLY1305_NEON for ARM.
+AES-256-HCTR2 is another true wide-block encryption mode. It has the same
+security guarantees as Adiantum, but is intended for use on CPUs with dedicated
+crypto instructions. See the paper "Length-preserving encryption with HCTR2"
+(https://eprint.iacr.org/2021/1441.pdf) for more details. To use HCTR2,
+CONFIG_CRYPTO_HCTR2 must be enabled. Also, fast implementations of XCTR and
+POLYVAL should be enabled, e.g. CRYPTO_POLYVAL_ARM64_CE and
+CRYPTO_AES_ARM64_CE_BLK for ARM64.
+
New encryption modes can be added relatively easily, without changes
to individual filesystems. However, authenticated encryption (AE)
modes are not currently supported because of the difficulty of dealing
@@ -404,11 +413,11 @@ alternatively has the file's nonce (for `DIRECT_KEY policies`_) or
inode number (for `IV_INO_LBLK_64 policies`_) included in the IVs.
Thus, IV reuse is limited to within a single directory.
-With CTS-CBC, the IV reuse means that when the plaintext filenames
-share a common prefix at least as long as the cipher block size (16
-bytes for AES), the corresponding encrypted filenames will also share
-a common prefix. This is undesirable. Adiantum does not have this
-weakness, as it is a wide-block encryption mode.
+With CTS-CBC, the IV reuse means that when the plaintext filenames share a
+common prefix at least as long as the cipher block size (16 bytes for AES), the
+corresponding encrypted filenames will also share a common prefix. This is
+undesirable. Adiantum and HCTR2 do not have this weakness, as they are
+wide-block encryption modes.
All supported filenames encryption modes accept any plaintext length
>= 16 bytes; cipher block alignment is not required. However,
diff --git a/fs/crypto/fscrypt_private.h b/fs/crypto/fscrypt_private.h
index 5b0a9e6478b5..d8617d01f7bd 100644
--- a/fs/crypto/fscrypt_private.h
+++ b/fs/crypto/fscrypt_private.h
@@ -31,7 +31,7 @@
#define FSCRYPT_CONTEXT_V2 2
/* Keep this in sync with include/uapi/linux/fscrypt.h */
-#define FSCRYPT_MODE_MAX FSCRYPT_MODE_ADIANTUM
+#define FSCRYPT_MODE_MAX FSCRYPT_MODE_AES_256_HCTR2
struct fscrypt_context_v1 {
u8 version; /* FSCRYPT_CONTEXT_V1 */
diff --git a/fs/crypto/keysetup.c b/fs/crypto/keysetup.c
index eede186b04ce..ae24b581d3d7 100644
--- a/fs/crypto/keysetup.c
+++ b/fs/crypto/keysetup.c
@@ -53,6 +53,13 @@ struct fscrypt_mode fscrypt_modes[] = {
.ivsize = 32,
.blk_crypto_mode = BLK_ENCRYPTION_MODE_ADIANTUM,
},
+ [FSCRYPT_MODE_AES_256_HCTR2] = {
+ .friendly_name = "HCTR2",
+ .cipher_str = "hctr2(aes)",
+ .keysize = 32,
+ .security_strength = 32,
+ .ivsize = 32,
+ },
};
static DEFINE_MUTEX(fscrypt_mode_key_setup_mutex);
diff --git a/fs/crypto/policy.c b/fs/crypto/policy.c
index ed3d623724cd..fa8bdb8c76b7 100644
--- a/fs/crypto/policy.c
+++ b/fs/crypto/policy.c
@@ -54,6 +54,10 @@ static bool fscrypt_valid_enc_modes(u32 contents_mode, u32 filenames_mode)
filenames_mode == FSCRYPT_MODE_ADIANTUM)
return true;
+ if (contents_mode == FSCRYPT_MODE_AES_256_XTS &&
+ filenames_mode == FSCRYPT_MODE_AES_256_HCTR2)
+ return true;
+
return false;
}
diff --git a/include/uapi/linux/fscrypt.h b/include/uapi/linux/fscrypt.h
index 9f4428be3e36..a756b29afcc2 100644
--- a/include/uapi/linux/fscrypt.h
+++ b/include/uapi/linux/fscrypt.h
@@ -27,7 +27,8 @@
#define FSCRYPT_MODE_AES_128_CBC 5
#define FSCRYPT_MODE_AES_128_CTS 6
#define FSCRYPT_MODE_ADIANTUM 9
-/* If adding a mode number > 9, update FSCRYPT_MODE_MAX in fscrypt_private.h */
+#define FSCRYPT_MODE_AES_256_HCTR2 10
+/* If adding a mode number > 10, update FSCRYPT_MODE_MAX in fscrypt_private.h */
/*
* Legacy policy version; ad-hoc KDF and no key verification.
diff --git a/tools/include/uapi/linux/fscrypt.h b/tools/include/uapi/linux/fscrypt.h
index 9f4428be3e36..a756b29afcc2 100644
--- a/tools/include/uapi/linux/fscrypt.h
+++ b/tools/include/uapi/linux/fscrypt.h
@@ -27,7 +27,8 @@
#define FSCRYPT_MODE_AES_128_CBC 5
#define FSCRYPT_MODE_AES_128_CTS 6
#define FSCRYPT_MODE_ADIANTUM 9
-/* If adding a mode number > 9, update FSCRYPT_MODE_MAX in fscrypt_private.h */
+#define FSCRYPT_MODE_AES_256_HCTR2 10
+/* If adding a mode number > 10, update FSCRYPT_MODE_MAX in fscrypt_private.h */
/*
* Legacy policy version; ad-hoc KDF and no key verification.
--
2.35.1.723.g4982287a31-goog
Add hardware accelerated version of POLYVAL for ARM64 CPUs with
Crypto Extension support.
This implementation is accelerated using PMULL instructions to perform
the finite field computations. For added efficiency, 8 blocks of the
message are processed simultaneously by precomputing the first 8
powers of the key.
Karatsuba multiplication is used instead of Schoolbook multiplication
because it was found to be slightly faster on ARM64 CPUs. Montgomery
reduction must be used instead of Barrett reduction due to the
difference in modulus between POLYVAL's field and other finite fields.
Signed-off-by: Nathan Huckleberry <[email protected]>
---
arch/arm64/crypto/Kconfig | 7 +
arch/arm64/crypto/Makefile | 3 +
arch/arm64/crypto/polyval-ce-core.S | 372 ++++++++++++++++++++++++++++
arch/arm64/crypto/polyval-ce-glue.c | 363 +++++++++++++++++++++++++++
4 files changed, 745 insertions(+)
create mode 100644 arch/arm64/crypto/polyval-ce-core.S
create mode 100644 arch/arm64/crypto/polyval-ce-glue.c
diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig
index 897f9a4b5b67..f7fbe8637e5c 100644
--- a/arch/arm64/crypto/Kconfig
+++ b/arch/arm64/crypto/Kconfig
@@ -60,6 +60,13 @@ config CRYPTO_GHASH_ARM64_CE
select CRYPTO_GF128MUL
select CRYPTO_LIB_AES
+config CRYPTO_POLYVAL_ARM64_CE
+ tristate "POLYVAL using ARMv8 Crypto Extensions (for HCTR2)"
+ depends on KERNEL_MODE_NEON
+ select CRYPTO_CRYPTD
+ select CRYPTO_HASH
+ select CRYPTO_POLYVAL
+
config CRYPTO_CRCT10DIF_ARM64_CE
tristate "CRCT10DIF digest algorithm using PMULL instructions"
depends on KERNEL_MODE_NEON && CRC_T10DIF
diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile
index 09a805cc32d7..53f9af962b86 100644
--- a/arch/arm64/crypto/Makefile
+++ b/arch/arm64/crypto/Makefile
@@ -26,6 +26,9 @@ sm4-ce-y := sm4-ce-glue.o sm4-ce-core.o
obj-$(CONFIG_CRYPTO_GHASH_ARM64_CE) += ghash-ce.o
ghash-ce-y := ghash-ce-glue.o ghash-ce-core.o
+obj-$(CONFIG_CRYPTO_POLYVAL_ARM64_CE) += polyval-ce.o
+polyval-ce-y := polyval-ce-glue.o polyval-ce-core.o
+
obj-$(CONFIG_CRYPTO_CRCT10DIF_ARM64_CE) += crct10dif-ce.o
crct10dif-ce-y := crct10dif-ce-core.o crct10dif-ce-glue.o
diff --git a/arch/arm64/crypto/polyval-ce-core.S b/arch/arm64/crypto/polyval-ce-core.S
new file mode 100644
index 000000000000..9c0fba11716c
--- /dev/null
+++ b/arch/arm64/crypto/polyval-ce-core.S
@@ -0,0 +1,372 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Implementation of POLYVAL using ARMv8 Crypto Extensions.
+ *
+ * Copyright 2021 Google LLC
+ */
+/*
+ * This is an efficient implementation of POLYVAL using ARMv8 Crypto Extensions
+ * It works on 8 blocks at a time, by precomputing the first 8 keys powers h^8,
+ * ..., h^1 in the POLYVAL finite field. This precomputation allows us to split
+ * finite field multiplication into two steps.
+ *
+ * In the first step, we consider h^i, m_i as normal polynomials of degree less
+ * than 128. We then compute p(x) = h^8m_0 + ... + h^1m_7 where multiplication
+ * is simply polynomial multiplication.
+ *
+ * In the second step, we compute the reduction of p(x) modulo the finite field
+ * modulus g(x) = x^128 + x^127 + x^126 + x^121 + 1.
+ *
+ * This two step process is equivalent to computing h^8m_0 + ... + h^1m_7 where
+ * multiplication is finite field multiplication. The advantage is that the
+ * two-step process only requires 1 finite field reduction for every 8
+ * polynomial multiplications. Further parallelism is gained by interleaving the
+ * multiplications and polynomial reductions.
+ */
+
+#include <linux/linkage.h>
+#define NUM_PRECOMPUTE_POWERS 8
+
+BLOCKS_LEFT .req x2
+KEY_START .req x10
+EXTRA_BYTES .req x11
+IND .req x12
+TMP .req x13
+PARTIAL_LEFT .req x14
+
+M0 .req v0
+M1 .req v1
+M2 .req v2
+M3 .req v3
+M4 .req v4
+M5 .req v5
+M6 .req v6
+M7 .req v7
+KEY8 .req v8
+KEY7 .req v9
+KEY6 .req v10
+KEY5 .req v11
+KEY4 .req v12
+KEY3 .req v13
+KEY2 .req v14
+KEY1 .req v15
+PL .req v16
+PH .req v17
+T .req v18
+V .req v19
+LO .req v20
+MI .req v21
+HI .req v22
+SUM .req v23
+GSTAR .req v24
+
+ .text
+ .align 4
+
+ .arch armv8-a+crypto
+ .align 4
+
+.Lgstar:
+ .quad 0xc200000000000000, 0xc200000000000000
+
+/*
+ * Computes the product of two 128-bit polynomials in X and Y and XORs the
+ * components of the 256-bit product into LO, MI, HI.
+ *
+ * The multiplication produces four parts:
+ * LOW: The polynomial given by performing carryless multiplication of X_L and
+ * Y_L
+ * MID: The polynomial given by performing carryless multiplication of (X_L ^
+ * X_H) and (Y_L ^ Y_H)
+ * HIGH: The polynomial given by performing carryless multiplication of X_H
+ * and Y_H
+ *
+ * We compute:
+ * LO ^= LOW
+ * MI ^= MID
+ * HI ^= HIGH
+ *
+ * Later, the 256-bit result can be extracted as:
+ * [HI_H : HI_L ^ HI_H ^ MI_H ^ LO_H :: LO_H ^ HI_L ^ MI_L ^ LO_L : LO_L]
+ * This step is done when computing the polynomial reduction for efficiency
+ * reasons.
+ */
+.macro karatsuba1 X Y
+ X .req \X
+ Y .req \Y
+ ext v25.16b, X.16b, Y.16b, #8
+ ext v26.16b, Y.16b, Y.16b, #8
+ eor v25.16b, v25.16b, X.16b
+ eor v26.16b, v26.16b, Y.16b
+ pmull v27.1q, v25.1d, v26.1d
+ pmull2 v28.1q, X.2d, Y.2d
+ pmull v29.1q, X.1d, Y.1d
+ eor HI.16b, HI.16b, v27.16b
+ eor LO.16b, LO.16b, v28.16b
+ eor MI.16b, MI.16b, v29.16b
+ .unreq X
+ .unreq Y
+.endm
+
+/*
+ * Computes the 256-bit polynomial represented by LO, HI, MI. Stores
+ * the result in PL, PH.
+ * [PH :: PL] = [HI_H : HI_L ^ HI_H ^ MI_H ^ LO_H :: LO_H ^ HI_L ^ MI_L ^ LO_L
+ * : LO_L]
+ */
+.macro karatsuba2
+ ext v4.16b, MI.16b, LO.16b, #8
+ eor HI.16b, HI.16b, v4.16b //[HI1 ^ LO0 : HI0 ^ MI1]
+ eor v4.16b, LO.16b, MI.16b //[LO1 ^ MI1 : LO0 ^ MI0]
+ //[LO0 ^ LO1 ^ MI1 ^ HI1 : MI1 ^ LO0 ^ MI0 ^ HI0]
+ eor v4.16b, HI.16b, v4.16b
+ ext LO.16b, LO.16b, LO.16b, #8 // [LO0 : LO1]
+ ext MI.16b, MI.16b, MI.16b, #8 // [MI0 : MI1]
+ ext PH.16b, v4.16b, LO.16b, #8 //[LO1 : LO1 ^ MI1 ^ HI1 ^ LO0]
+ ext PL.16b, MI.16b, v4.16b, #8 //[MI1 ^ LO0 ^ MI0 ^ HI0 : MI0]
+.endm
+
+/*
+ * Computes the 128-bit reduction of PL, PH. Stores the result in PH.
+ *
+ * This macro computes p(x) mod g(x) where p(x) is in montgomery form and g(x) =
+ * x^128 + x^127 + x^126 + x^121 + 1.
+ *
+ * We have a 256-bit polynomial P_3 : P_2 : P_1 : P_0 that is the product of
+ * two 128-bit polynomials in Montgomery form. We need to reduce it mod g(x).
+ * Also, since polynomials in Montgomery form have an "extra" factor of x^128,
+ * this product has two extra factors of x^128. To get it back into Montgomery
+ * form, we need to remove one of these factors by dividing by x^128.
+ *
+ * To accomplish both of these goals, we add multiples of g(x) that cancel out
+ * the low 128 bits P_1 : P_0, leaving just the high 128 bits. Since the low
+ * bits are zero, the polynomial division by x^128 can be done by right shifting.
+ *
+ * Since the only nonzero term in the low 64 bits of g(x) is the constant term,
+ * the multiple of g(x) needed to cancel out P_0 is P_0 * g(x). The CPU can
+ * only do 64x64 bit multiplications, so split P_0 * g(x) into x^128 * P_0 +
+ * x^64 g*(x) * P_0 + P_0, where g*(x) is bits 64-127 of g(x). Adding this to
+ * the original polynomial gives P_3 : P_2 + P_0 + T_1 : P_1 + T_0 : 0, where T
+ * = T_1 : T_0 = g*(x) * P0. Thus, bits 0-63 got "folded" into bits 64-191.
+ *
+ * Repeating this same process on the next 64 bits "folds" bits 64-127 into bits
+ * 128-255, giving the answer in bits 128-255. This time, we need to cancel P_1
+ * + T_0 in bits 64-127. The multiple of g(x) required is (P_1 + T_0) * g(x) *
+ * x^64. Adding this to our previous computation gives P_3 + P_1 + T_0 + V_1 :
+ * P_2 + P_0 + T_1 + V_0 : 0 : 0, where V = V_1 : V_0 = g*(x) * (P_1 + T_0).
+ *
+ * So our final computation is:
+ * T = T_1 : T_0 = g*(x) * P_0
+ * V = V_1 : V_0 = g*(x) * (T_0 ^ P_1)
+ * p(x) / x^{128} mod g(x) = P_3 ^ P_1 ^ V_1 ^ T_0 : P_2 ^ P_0 ^ V_0 ^ T_1
+ *
+ * The implementation below saves a XOR instruction by computing P_1 ^ T_0 : P_0
+ * ^ T_1 and XORing it into V, rather than directly XORing P_1 : P_0, T_0 : T1
+ * into PH. This allows us to reuse P_1 ^ T_0 when computing V.
+ */
+.macro montgomery_reduction
+ pmull T.1q, GSTAR.1d, PL.1d
+ ext T.16b, T.16b, T.16b, #8
+ eor PL.16b, PL.16b, T.16b
+ pmull2 V.1q, GSTAR.2d, PL.2d
+ eor V.16b, PL.16b, V.16b
+ eor PH.16b, PH.16b, V.16b
+.endm
+
+/*
+ * Compute Polyval on 8 blocks.
+ *
+ * If reduce is set, also computes the montgomery reduction of the
+ * previous full_stride call and XORs with the first message block.
+ * (m_0 + REDUCE(PL, PH))h^8 + ... + m_7h^1.
+ * I.e., the first multiplication uses m_0 + REDUCE(PL, PH) instead of m_0.
+ *
+ * Sets PL, PH.
+ */
+.macro full_stride reduce
+ eor LO.16b, LO.16b, LO.16b
+ eor MI.16b, MI.16b, MI.16b
+ eor HI.16b, HI.16b, HI.16b
+
+ ld1 {M0.16b, M1.16b, M2.16b, M3.16b}, [x0], #64
+ ld1 {M4.16b, M5.16b, M6.16b, M7.16b}, [x0], #64
+
+ karatsuba1 M7 KEY1
+ .if (\reduce)
+ pmull T.1q, GSTAR.1d, PL.1d
+ .endif
+
+ karatsuba1 M6 KEY2
+ .if (\reduce)
+ ext T.16b, T.16b, T.16b, #8
+ .endif
+
+ karatsuba1 M5 KEY3
+ .if (\reduce)
+ eor PL.16b, PL.16b, T.16b
+ .endif
+
+ karatsuba1 M4 KEY4
+ .if (\reduce)
+ pmull2 V.1q, GSTAR.2d, PL.2d
+ .endif
+
+ karatsuba1 M3 KEY5
+ .if (\reduce)
+ eor V.16b, PL.16b, V.16b
+ .endif
+
+ karatsuba1 M2 KEY6
+ .if (\reduce)
+ eor PH.16b, PH.16b, V.16b
+ .endif
+
+ karatsuba1 M1 KEY7
+ .if (\reduce)
+ mov SUM.16b, PH.16b
+ .endif
+ eor M0.16b, M0.16b, SUM.16b
+
+ karatsuba1 M0 KEY8
+
+ karatsuba2
+.endm
+
+/*
+ * Handle any extra blocks before
+ * full_stride loop.
+ */
+.macro partial_stride
+ eor LO.16b, LO.16b, LO.16b
+ eor MI.16b, MI.16b, MI.16b
+ eor HI.16b, HI.16b, HI.16b
+ add KEY_START, x1, #(NUM_PRECOMPUTE_POWERS << 4)
+ sub KEY_START, KEY_START, PARTIAL_LEFT, lsl #4
+ ld1 {v0.16b}, [KEY_START]
+ mov v1.16b, SUM.16b
+ karatsuba1 v0 v1
+ karatsuba2
+ montgomery_reduction
+ mov SUM.16b, PH.16b
+ eor LO.16b, LO.16b, LO.16b
+ eor MI.16b, MI.16b, MI.16b
+ eor HI.16b, HI.16b, HI.16b
+ mov IND, XZR
+.LloopPartial:
+ cmp IND, PARTIAL_LEFT
+ bge .LloopExitPartial
+
+ sub TMP, IND, PARTIAL_LEFT
+
+ cmp TMP, #-4
+ bgt .Lgt4Partial
+ ld1 {M0.16b, M1.16b, M2.16b, M3.16b}, [x0], #64
+ // Clobber key registers
+ ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [KEY_START], #64
+ karatsuba1 M0 KEY8
+ karatsuba1 M1 KEY7
+ karatsuba1 M2 KEY6
+ karatsuba1 M3 KEY5
+ add IND, IND, #4
+ b .LoutPartial
+
+.Lgt4Partial:
+ cmp TMP, #-3
+ bgt .Lgt3Partial
+ ld1 {M0.16b, M1.16b, M2.16b}, [x0], #48
+ // Clobber key registers
+ ld1 {KEY8.16b, KEY7.16b, KEY6.16b}, [KEY_START], #48
+ karatsuba1 M0 KEY8
+ karatsuba1 M1 KEY7
+ karatsuba1 M2 KEY6
+ add IND, IND, #3
+ b .LoutPartial
+
+.Lgt3Partial:
+ cmp TMP, #-2
+ bgt .Lgt2Partial
+ ld1 {M0.16b, M1.16b}, [x0], #32
+ // Clobber key registers
+ ld1 {KEY8.16b, KEY7.16b}, [KEY_START], #32
+ karatsuba1 M0 KEY8
+ karatsuba1 M1 KEY7
+ add IND, IND, #2
+ b .LoutPartial
+
+.Lgt2Partial:
+ ld1 {M0.16b}, [x0], #16
+ // Clobber key registers
+ ld1 {KEY8.16b}, [KEY_START], #16
+ karatsuba1 M0 KEY8
+ add IND, IND, #1
+.LoutPartial:
+ b .LloopPartial
+.LloopExitPartial:
+ karatsuba2
+ montgomery_reduction
+ eor SUM.16b, SUM.16b, PH.16b
+.endm
+
+/*
+ * Perform montgomery multiplication in GF(2^128) and store result in op1.
+ *
+ * Computes op1*op2*x^{-128} mod x^128 + x^127 + x^126 + x^121 + 1
+ * If op1, op2 are in montgomery form, this computes the montgomery
+ * form of op1*op2.
+ *
+ * void pmull_polyval_mul(u8 *op1, const u8 *op2);
+ */
+SYM_FUNC_START(pmull_polyval_mul)
+ adr TMP, .Lgstar
+ ld1 {GSTAR.2d}, [TMP]
+ eor LO.16b, LO.16b, LO.16b
+ eor MI.16b, MI.16b, MI.16b
+ eor HI.16b, HI.16b, HI.16b
+ ld1 {v0.16b}, [x0]
+ ld1 {v1.16b}, [x1]
+ karatsuba1 v0 v1
+ karatsuba2
+ montgomery_reduction
+ st1 {PH.16b}, [x0]
+ ret
+SYM_FUNC_END(pmull_polyval_mul)
+
+/*
+ * Perform polynomial evaluation as specified by POLYVAL. This computes:
+ * h^n * accumulator + h^n * m_0 + ... + h^1 * m_{n-1}
+ * where n=nblocks, h is the hash key, and m_i are the message blocks.
+ *
+ * x0 - pointer to message blocks
+ * x1 - pointer to precomputed key powers h^8 ... h^1
+ * x2 - number of blocks to hash
+ * x3 - pointer to accumulator
+ *
+ * void pmull_polyval_update(const u8 *in, const struct polyval_ctx *ctx,
+ * size_t nblocks, u8 *accumulator);
+ */
+SYM_FUNC_START(pmull_polyval_update)
+ adr TMP, .Lgstar
+ ld1 {GSTAR.2d}, [TMP]
+ ld1 {SUM.16b}, [x3]
+ ands PARTIAL_LEFT, BLOCKS_LEFT, #7
+ beq .LskipPartial
+ partial_stride
+.LskipPartial:
+ subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
+ blt .LstrideLoopExit
+ ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [x1], #64
+ ld1 {KEY4.16b, KEY3.16b, KEY2.16b, KEY1.16b}, [x1], #64
+ full_stride 0
+ subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
+ blt .LstrideLoopExitReduce
+.LstrideLoop:
+ full_stride 1
+ subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
+ bge .LstrideLoop
+.LstrideLoopExitReduce:
+ montgomery_reduction
+ mov SUM.16b, PH.16b
+.LstrideLoopExit:
+ st1 {SUM.16b}, [x3]
+ ret
+SYM_FUNC_END(pmull_polyval_update)
diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyval-ce-glue.c
new file mode 100644
index 000000000000..52cf87c36043
--- /dev/null
+++ b/arch/arm64/crypto/polyval-ce-glue.c
@@ -0,0 +1,363 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Accelerated POLYVAL implementation with ARMv8 Crypto Extension
+ * instructions. This file contains glue code.
+ *
+ * Copyright (c) 2007 Nokia Siemens Networks - Mikko Herranen <[email protected]>
+ * Copyright (c) 2009 Intel Corp.
+ * Author: Huang Ying <[email protected]>
+ * Copyright 2021 Google LLC
+ */
+/*
+ * Glue code based on ghash-clmulni-intel_glue.c.
+ *
+ * This implementation of POLYVAL uses montgomery multiplication accelerated by
+ * ARMv8 Crypto Extension instructions to implement the finite field operations.
+ *
+ */
+
+#include <crypto/algapi.h>
+#include <crypto/cryptd.h>
+#include <crypto/gf128mul.h>
+#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
+#include <crypto/polyval.h>
+#include <linux/crypto.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/cpufeature.h>
+#include <asm/neon.h>
+#include <asm/simd.h>
+#include <asm/unaligned.h>
+
+#define NUM_PRECOMPUTE_POWERS 8
+
+struct polyval_async_ctx {
+ struct cryptd_ahash *cryptd_tfm;
+};
+
+struct polyval_ctx {
+ u8 key_powers[NUM_PRECOMPUTE_POWERS][POLYVAL_BLOCK_SIZE];
+};
+
+struct polyval_desc_ctx {
+ u8 buffer[POLYVAL_BLOCK_SIZE];
+ u32 bytes;
+};
+
+asmlinkage void pmull_polyval_update(const u8 *in, const struct polyval_ctx
+ *ctx, size_t nblocks, u8 *accumulator);
+asmlinkage void pmull_polyval_mul(u8 *op1, const u8 *op2);
+
+static int polyval_init(struct shash_desc *desc)
+{
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+
+ memset(dctx, 0, sizeof(*dctx));
+
+ return 0;
+}
+
+static int polyval_setkey(struct crypto_shash *tfm,
+ const u8 *key, unsigned int keylen)
+{
+ struct polyval_ctx *ctx = crypto_shash_ctx(tfm);
+ int i;
+
+ if (keylen != POLYVAL_BLOCK_SIZE)
+ return -EINVAL;
+
+ BUILD_BUG_ON(sizeof(u128) != POLYVAL_BLOCK_SIZE);
+
+ memcpy(ctx->key_powers[NUM_PRECOMPUTE_POWERS-1], key,
+ POLYVAL_BLOCK_SIZE);
+
+ kernel_neon_begin();
+ for (i = NUM_PRECOMPUTE_POWERS-2; i >= 0; i--) {
+ memcpy(ctx->key_powers[i], key, POLYVAL_BLOCK_SIZE);
+ pmull_polyval_mul(ctx->key_powers[i], ctx->key_powers[i+1]);
+ }
+ kernel_neon_end();
+
+ return 0;
+}
+
+static int polyval_update(struct shash_desc *desc,
+ const u8 *src, unsigned int srclen)
+{
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+ struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm);
+ u8 *pos;
+ unsigned int nblocks;
+ unsigned int n;
+
+ kernel_neon_begin();
+ if (dctx->bytes) {
+ n = min(srclen, dctx->bytes);
+ pos = dctx->buffer + POLYVAL_BLOCK_SIZE - dctx->bytes;
+
+ dctx->bytes -= n;
+ srclen -= n;
+
+ while (n--)
+ *pos++ ^= *src++;
+
+ if (!dctx->bytes)
+ pmull_polyval_mul(dctx->buffer,
+ ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]);
+ }
+
+ nblocks = srclen/POLYVAL_BLOCK_SIZE;
+ pmull_polyval_update(src, ctx, nblocks, dctx->buffer);
+ srclen -= nblocks*POLYVAL_BLOCK_SIZE;
+ kernel_neon_end();
+
+ if (srclen) {
+ dctx->bytes = POLYVAL_BLOCK_SIZE - srclen;
+ src += nblocks*POLYVAL_BLOCK_SIZE;
+ pos = dctx->buffer;
+ while (srclen--)
+ *pos++ ^= *src++;
+ }
+
+ return 0;
+}
+
+static int polyval_final(struct shash_desc *desc, u8 *dst)
+{
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+ struct polyval_ctx *ctx = crypto_shash_ctx(desc->tfm);
+
+ if (dctx->bytes) {
+ kernel_neon_begin();
+ pmull_polyval_mul(dctx->buffer,
+ ctx->key_powers[NUM_PRECOMPUTE_POWERS-1]);
+ kernel_neon_end();
+ }
+
+ dctx->bytes = 0;
+ memcpy(dst, dctx->buffer, POLYVAL_BLOCK_SIZE);
+
+ return 0;
+}
+
+static struct shash_alg polyval_alg = {
+ .digestsize = POLYVAL_DIGEST_SIZE,
+ .init = polyval_init,
+ .update = polyval_update,
+ .final = polyval_final,
+ .setkey = polyval_setkey,
+ .descsize = sizeof(struct polyval_desc_ctx),
+ .base = {
+ .cra_name = "__polyval",
+ .cra_driver_name = "__polyval-ce",
+ .cra_priority = 0,
+ .cra_flags = CRYPTO_ALG_INTERNAL,
+ .cra_blocksize = POLYVAL_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct polyval_ctx),
+ .cra_module = THIS_MODULE,
+ },
+};
+
+static int polyval_async_init(struct ahash_request *req)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct crypto_shash *child = cryptd_ahash_child(cryptd_tfm);
+
+ desc->tfm = child;
+ return crypto_shash_init(desc);
+}
+
+static int polyval_async_update(struct ahash_request *req)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc;
+
+ if (!crypto_simd_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
+ memcpy(cryptd_req, req, sizeof(*req));
+ ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
+ return crypto_ahash_update(cryptd_req);
+ }
+ desc = cryptd_shash_desc(cryptd_req);
+
+ return shash_ahash_update(req, desc);
+}
+
+static int polyval_async_final(struct ahash_request *req)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc;
+
+ if (!crypto_simd_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
+ memcpy(cryptd_req, req, sizeof(*req));
+ ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
+ return crypto_ahash_final(cryptd_req);
+ }
+ desc = cryptd_shash_desc(cryptd_req);
+
+ return crypto_shash_final(desc, req->result);
+}
+
+static int polyval_async_import(struct ahash_request *req, const void *in)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+
+ polyval_async_init(req);
+ memcpy(dctx, in, sizeof(*dctx));
+ return 0;
+
+}
+
+static int polyval_async_export(struct ahash_request *req, void *out)
+{
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct shash_desc *desc = cryptd_shash_desc(cryptd_req);
+ struct polyval_desc_ctx *dctx = shash_desc_ctx(desc);
+
+ memcpy(out, dctx, sizeof(*dctx));
+ return 0;
+
+}
+
+static int polyval_async_digest(struct ahash_request *req)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct ahash_request *cryptd_req = ahash_request_ctx(req);
+ struct cryptd_ahash *cryptd_tfm = ctx->cryptd_tfm;
+ struct shash_desc *desc;
+ struct crypto_shash *child;
+
+ if (!crypto_simd_usable() ||
+ (in_atomic() && cryptd_ahash_queued(cryptd_tfm))) {
+ memcpy(cryptd_req, req, sizeof(*req));
+ ahash_request_set_tfm(cryptd_req, &cryptd_tfm->base);
+ return crypto_ahash_digest(cryptd_req);
+ }
+ desc = cryptd_shash_desc(cryptd_req);
+ child = cryptd_ahash_child(cryptd_tfm);
+
+ desc->tfm = child;
+ return shash_ahash_digest(req, desc);
+}
+
+static int polyval_async_setkey(struct crypto_ahash *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct polyval_async_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct crypto_ahash *child = &ctx->cryptd_tfm->base;
+
+ crypto_ahash_clear_flags(child, CRYPTO_TFM_REQ_MASK);
+ crypto_ahash_set_flags(child, crypto_ahash_get_flags(tfm)
+ & CRYPTO_TFM_REQ_MASK);
+ return crypto_ahash_setkey(child, key, keylen);
+}
+
+static int polyval_async_init_tfm(struct crypto_tfm *tfm)
+{
+ struct cryptd_ahash *cryptd_tfm;
+ struct polyval_async_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ cryptd_tfm = cryptd_alloc_ahash("__polyval-ce",
+ CRYPTO_ALG_INTERNAL,
+ CRYPTO_ALG_INTERNAL);
+ if (IS_ERR(cryptd_tfm))
+ return PTR_ERR(cryptd_tfm);
+ ctx->cryptd_tfm = cryptd_tfm;
+ crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
+ sizeof(struct ahash_request) +
+ crypto_ahash_reqsize(&cryptd_tfm->base));
+
+ return 0;
+}
+
+static void polyval_async_exit_tfm(struct crypto_tfm *tfm)
+{
+ struct polyval_async_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ cryptd_free_ahash(ctx->cryptd_tfm);
+}
+
+static struct ahash_alg polyval_async_alg = {
+ .init = polyval_async_init,
+ .update = polyval_async_update,
+ .final = polyval_async_final,
+ .setkey = polyval_async_setkey,
+ .digest = polyval_async_digest,
+ .export = polyval_async_export,
+ .import = polyval_async_import,
+ .halg = {
+ .digestsize = POLYVAL_DIGEST_SIZE,
+ .statesize = sizeof(struct polyval_desc_ctx),
+ .base = {
+ .cra_name = "polyval",
+ .cra_driver_name = "polyval-ce",
+ .cra_priority = 200,
+ .cra_ctxsize = sizeof(struct polyval_async_ctx),
+ .cra_flags = CRYPTO_ALG_ASYNC,
+ .cra_blocksize = POLYVAL_BLOCK_SIZE,
+ .cra_module = THIS_MODULE,
+ .cra_init = polyval_async_init_tfm,
+ .cra_exit = polyval_async_exit_tfm,
+ },
+ },
+};
+
+static int __init polyval_ce_mod_init(void)
+{
+ int err;
+
+ if (!cpu_have_named_feature(ASIMD))
+ return -ENODEV;
+
+ if (!cpu_have_named_feature(PMULL))
+ return -ENODEV;
+
+ err = crypto_register_shash(&polyval_alg);
+ if (err)
+ goto err_out;
+ err = crypto_register_ahash(&polyval_async_alg);
+ if (err)
+ goto err_shash;
+
+ return 0;
+
+err_shash:
+ crypto_unregister_shash(&polyval_alg);
+err_out:
+ return err;
+}
+
+static void __exit polyval_ce_mod_exit(void)
+{
+ crypto_unregister_ahash(&polyval_async_alg);
+ crypto_unregister_shash(&polyval_alg);
+}
+
+static const struct cpu_feature polyval_cpu_feature[] = {
+ { cpu_feature(PMULL) }, { }
+};
+MODULE_DEVICE_TABLE(cpu, polyval_cpu_feature);
+
+module_init(polyval_ce_mod_init);
+module_exit(polyval_ce_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("POLYVAL hash function accelerated by ARMv8 Crypto Extension");
+MODULE_ALIAS_CRYPTO("polyval");
+MODULE_ALIAS_CRYPTO("polyval-ce");
--
2.35.1.723.g4982287a31-goog
On Tue, Mar 15, 2022 at 11:00:28PM +0000, Nathan Huckleberry wrote:
> Add a generic implementation of XCTR mode as a template. XCTR is a
> blockcipher mode similar to CTR mode. XCTR uses XORs and little-endian
> addition rather than big-endian arithmetic which has two advantages: It
> is slightly faster on little-endian CPUs and it is less likely to be
> implemented incorrect since integer overflows are not possible on
> practical input sizes. XCTR is used as a component to implement HCTR2.
>
> More information on XCTR mode can be found in the HCTR2 paper:
> https://eprint.iacr.org/2021/1441.pdf
>
> Signed-off-by: Nathan Huckleberry <[email protected]>
Looks good, feel free to add:
Reviewed-by: Eric Biggers <[email protected]>
A few minor nits below:
> +// Limited to 16-byte blocks for simplicity
> +#define XCTR_BLOCKSIZE 16
> +
> +static void crypto_xctr_crypt_final(struct skcipher_walk *walk,
> + struct crypto_cipher *tfm, u32 byte_ctr)
> +{
> + u8 keystream[XCTR_BLOCKSIZE];
> + u8 *src = walk->src.virt.addr;
Use 'const u8 *src'
> +static int crypto_xctr_crypt_segment(struct skcipher_walk *walk,
> + struct crypto_cipher *tfm, u32 byte_ctr)
> +{
> + void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
> + crypto_cipher_alg(tfm)->cia_encrypt;
> + u8 *src = walk->src.virt.addr;
Likewise, 'const u8 *src'
> + u8 *dst = walk->dst.virt.addr;
> + unsigned int nbytes = walk->nbytes;
> + __le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
> +
> + do {
> + /* create keystream */
> + crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
> + fn(crypto_cipher_tfm(tfm), dst, walk->iv);
> + crypto_xor(dst, src, XCTR_BLOCKSIZE);
> + crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
The comment "/* create keystream /*" is a bit misleading, since the part of the
code that it describes isn't just creating the keystream, but also XOR'ing it
with the data. It would be better to just remove that comment.
> +
> + ctr32 = cpu_to_le32(le32_to_cpu(ctr32) + 1);
This could use le32_add_cpu().
> +
> + src += XCTR_BLOCKSIZE;
> + dst += XCTR_BLOCKSIZE;
> + } while ((nbytes -= XCTR_BLOCKSIZE) >= XCTR_BLOCKSIZE);
> +
> + return nbytes;
> +}
> +
> +static int crypto_xctr_crypt_inplace(struct skcipher_walk *walk,
> + struct crypto_cipher *tfm, u32 byte_ctr)
> +{
> + void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
> + crypto_cipher_alg(tfm)->cia_encrypt;
> + unsigned long alignmask = crypto_cipher_alignmask(tfm);
> + unsigned int nbytes = walk->nbytes;
> + u8 *src = walk->src.virt.addr;
Perhaps call this 'data' instead of 'src', since here it's both the source and
destination?
> + u8 tmp[XCTR_BLOCKSIZE + MAX_CIPHER_ALIGNMASK];
> + u8 *keystream = PTR_ALIGN(tmp + 0, alignmask + 1);
> + __le32 ctr32 = cpu_to_le32(byte_ctr / XCTR_BLOCKSIZE + 1);
> +
> + do {
> + /* create keystream */
Likewise, remove or clarify the '/* create keystream */' comment.
> + crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
> + fn(crypto_cipher_tfm(tfm), keystream, walk->iv);
> + crypto_xor(src, keystream, XCTR_BLOCKSIZE);
> + crypto_xor(walk->iv, (u8 *)&ctr32, sizeof(ctr32));
> +
> + ctr32 = cpu_to_le32(le32_to_cpu(ctr32) + 1);
Likewise, le32_add_cpu().
- Eric
On Tue, Mar 15, 2022 at 11:00:33PM +0000, Nathan Huckleberry wrote:
> diff --git a/arch/x86/crypto/polyval-clmulni_asm.S b/arch/x86/crypto/polyval-clmulni_asm.S
> new file mode 100644
> index 000000000000..ad7126d9f0ff
> --- /dev/null
> +++ b/arch/x86/crypto/polyval-clmulni_asm.S
> @@ -0,0 +1,376 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright 2021 Google LLC
> + */
> +/*
> + * This is an efficient implementation of POLYVAL using intel PCLMULQDQ-NI
> + * instructions. It works on 8 blocks at a time, by precomputing the first 8
> + * keys powers h^8, ..., h^1 in the POLYVAL finite field. This precomputation
> + * allows us to split finite field multiplication into two steps.
> + *
> + * In the first step, we consider h^i, m_i as normal polynomials of degree less
> + * than 128. We then compute p(x) = h^8m_0 + ... + h^1m_7 where multiplication
> + * is simply polynomial multiplication.
> + *
> + * In the second step, we compute the reduction of p(x) modulo the finite field
> + * modulus g(x) = x^128 + x^127 + x^126 + x^121 + 1.
> + *
> + * This two step process is equivalent to computing h^8m_0 + ... + h^1m_7 where
> + * multiplication is finite field multiplication. The advantage is that the
> + * two-step process only requires 1 finite field reduction for every 8
> + * polynomial multiplications. Further parallelism is gained by interleaving the
> + * multiplications and polynomial reductions.
> + */
> +
> +#include <linux/linkage.h>
> +#include <asm/frame.h>
> +
> +#define NUM_PRECOMPUTE_POWERS 8
STRIDE_BLOCKS might be a better name than NUM_PRECOMPUTE_POWERS. Shorter but
more descriptive, IMO.
> +/*
> + * Performs schoolbook1_iteration on two lists of 128-bit polynomials of length
> + * b pointed to by MSG and KEY_POWERS.
> + */
> +.macro schoolbook1 count
> + .set i, 0
> + .rept (\count)
> + schoolbook1_iteration i 0
> + .set i, (i +1)
> + .endr
> +.endm
'count', not 'b'.
> +/*
> + * Computes the product of two 128-bit polynomials at the memory locations
> + * specified by (MSG + 16*i) and (KEY_POWERS + 16*i) and XORs the components of the
> + * 256-bit product into LO, MI, HI.
> + *
> + * The multiplication produces four parts:
> + * LOW: The polynomial given by performing carryless multiplication of the
> + * bottom 64-bits of each polynomial
> + * MID1: The polynomial given by performing carryless multiplication of the
> + * bottom 64-bits of the first polynomial and the top 64-bits of the second
> + * MID2: The polynomial given by performing carryless multiplication of the
> + * bottom 64-bits of the second polynomial and the top 64-bits of the first
> + * HIGH: The polynomial given by performing carryless multiplication of the
> + * top 64-bits of each polynomial
> + *
> + * We compute:
> + * LO ^= LOW
> + * MI ^= MID1 ^ MID2
> + * HI ^= HIGH
> + *
> + * Later, the 256-bit result can be extracted as:
> + * [HI_H : HI_L ^ MI_H : LO_H ^ MI_L : LO_L]
> + * This step is done when computing the polynomial reduction for efficiency
> + * reasons.
> + *
> + * If xor_sum == 1, then also XOR the value of SUM into m_0. This avoids an
> + * extra multiplication of SUM and h^N.
> + */
h^8 instead of h^N? The above is one of only two places where "N" is mentioned,
and the other uses to mean something different from here.
> +/*
> + * Performs the same computation as schoolbook1_iteration, except we expect the
> + * arguments to already be loaded into xmm0 and xmm1.
> + */
> +.macro schoolbook1_noload
> + vpclmulqdq $0x01, %xmm0, %xmm1, %xmm2
> + vpclmulqdq $0x00, %xmm0, %xmm1, %xmm3
> + vpclmulqdq $0x11, %xmm0, %xmm1, %xmm4
> + vpclmulqdq $0x10, %xmm0, %xmm1, %xmm5
> + vpxor %xmm2, MI, MI
> + vpxor %xmm3, LO, LO
> + vpxor %xmm5, MI, MI
> + vpxor %xmm4, HI, HI
> +.endm
How about making this macro set LO, MI, HI and directly instead of XOR'ing into
them? That's actually what the two users of it want. I.e.:
/*
* Performs the same computation as schoolbook1_iteration, except we expect the
* arguments to already be loaded into xmm0 and xmm1, and we set the result
* registers LO, MI, and HI directly rather than XOR'ing into them.
*/
.macro schoolbook1_noload
vpclmulqdq $0x01, %xmm0, %xmm1, MI
vpclmulqdq $0x10, %xmm0, %xmm1, %xmm2
vpclmulqdq $0x00, %xmm0, %xmm1, LO
vpclmulqdq $0x11, %xmm0, %xmm1, HI
vpxor %xmm2, MI, MI
.endm
That would save some instructions.
> +/*
> + * Computes the 128-bit reduction of PL, PH. Stores the result in PH.
"PL, PH" => "PH : PL".
Also mention which register this clobbers.
> + *
> + * This macro computes p(x) mod g(x) where p(x) is in montgomery form and g(x) =
> + * x^128 + x^127 + x^126 + x^121 + 1.
> + *
> + * We have a 256-bit polynomial P_3 : P_2 : P_1 : P_0 that is the product of
"P_3 : P_2 : P_1 : P_0" => "PH : PL = P_3 : P_2 : P_1 : P_0", so that it's clear
how P_3 through P_0 relate to PH and PL.
> + * two 128-bit polynomials in Montgomery form. We need to reduce it mod g(x).
> + * Also, since polynomials in Montgomery form have an "extra" factor of x^128,
> + * this product has two extra factors of x^128. To get it back into Montgomery
> + * form, we need to remove one of these factors by dividing by x^128.
> + *
> + * To accomplish both of these goals, we add multiples of g(x) that cancel out
> + * the low 128 bits P_1 : P_0, leaving just the high 128 bits. Since the low
> + * bits are zero, the polynomial division by x^128 can be done by right shifting.
> + *
> + * Since the only nonzero term in the low 64 bits of g(x) is the constant term,
> + * the multiple of g(x) needed to cancel out P_0 is P_0 * g(x). The CPU can
> + * only do 64x64 bit multiplications, so split P_0 * g(x) into x^128 * P_0 +
> + * x^64 g*(x) * P_0 + P_0, where g*(x) is bits 64-127 of g(x). Adding this to
"x^64 g*(x)" => "x^64 * g*(x)"
> + * the original polynomial gives P_3 : P_2 + P_0 + T_1 : P_1 + T_0 : 0, where T
> + * = T_1 : T_0 = g*(x) * P0. Thus, bits 0-63 got "folded" into bits 64-191.
"P0" => "P_0"
> + *
> + * Repeating this same process on the next 64 bits "folds" bits 64-127 into bits
> + * 128-255, giving the answer in bits 128-255. This time, we need to cancel P_1
> + * + T_0 in bits 64-127. The multiple of g(x) required is (P_1 + T_0) * g(x) *
> + * x^64. Adding this to our previous computation gives P_3 + P_1 + T_0 + V_1 :
> + * P_2 + P_0 + T_1 + V_0 : 0 : 0, where V = V_1 : V_0 = g*(x) * (P_1 + T_0).
> + *
> + * So our final computation is:
> + * T = T_1 : T_0 = g*(x) * P_0
> + * V = V_1 : V_0 = g*(x) * (T_0 ^ P_1)
> + * p(x) / x^{128} mod g(x) = P_3 ^ P_1 ^ V_1 ^ T_0 : P_2 ^ P_0 ^ V_0 ^ T_1
The notation suddenly changes from + to ^. How about consistently using +?
Or ^, either one as long as it's consistent...
Also, for the final line, the order "P_3 + P_1 + T_0 + V_1 : P_2 + P_0 + T_1 +
V_0" would make more sense, as it would match the logic of the code.
> + *
> + * The implementation below saves a XOR instruction by computing P_1 ^ T_0 : P_0
> + * ^ T_1 and XORing into PH, rather than directly XORing P_1 : P_0, T_0 : T1
> + * into PH. This allows us to reuse P_1 ^ T_0 when computing V.
> + */
> +.macro montgomery_reduction
> + movdqa PL, T
> + pclmulqdq $0x00, GSTAR, T # T = [P_0 * g*(x)]
> + pshufd $0b01001110, T, V # V = [T_0 : T_1]
> + pxor V, PL # PL = [P_1 ^ T_0 : P_0 ^ T_1]
> + pxor PL, PH # PH = [P_1 ^ T_0 ^ P_3 : P_0 ^ T_1 ^ P_2]
> + pclmulqdq $0x11, GSTAR, PL # PL = [(P_1 ^ T_0) * g*(x)]
> + pxor PL, PH
> +.endm
Several comments here:
- Aligning the comments would make them much easier to read.
- Only one temporary register is needed, since T isn't used after it's used to
compute V.
- The thing called V isn't actually the same as the V described in the long
comment above. Maybe just call the temporary variable 'TMP_XMM' or something?
Or even just hard-code %xmm6, similar to %xmm0-%xmm5.
- It's not necessary to modify PL.
- Since this file is relying on AVX anyway, the three-operand instructions are
available, and can be used to avoid the 'movdqa' at the beginning.
- None of the users of this macro really want the result in register PH. How
about passing the destination register as an argument and using vpxor to put
it in the appropriate place?
So in summary, this is what I'd suggest:
.macro montgomery_reduction dest
vpclmulqdq $0x00, GSTAR, PL, TMP_XMM # TMP_XMM = T_1 : T_0 = P_0 * g*(x)
pshufd $0b01001110, TMP_XMM, TMP_XMM # TMP_XMM = T_0 : T_1
pxor PL, TMP_XMM # TMP_XMM = P_1 + T_0 : P_0 + T_1
pxor TMP_XMM, PH # PH = P_3 + P_1 + T_0 : P_2 + P_0 + T_1
pclmulqdq $0x11, GSTAR, TMP_XMM # TMP_XMM = V_1 : V_0 = V = [(P_1 + T_0) * g*(x)]
vpxor TMP_XMM, PH, \dest
.endm
> +
> +/*
> + * Compute schoolbook multiplication for 8 blocks
> + * m_0h^8 + ... + m_7h^1
> + *
> + * If reduce is set, also computes the montgomery reduction of the
> + * previous full_stride call and XORs with the first message block.
> + * (m_0 + REDUCE(PL, PH))h^8 + ... + m_7h^1.
> + * I.e., the first multiplication uses m_0 + REDUCE(PL, PH) instead of m_0.
> + *
> + * Sets PL, PH
> + * Clobbers LO, HI, MI
> + *
> + */
> +.macro full_stride reduce
> + mov %rsi, KEY_POWERS
I don't see why KEY_POWERS and %rsi are different registers. Why not just
define KEY_POWERS to %rsi? It stays the same during any full_strides, and then
will be incremented by partial_stride. That's fine.
[...]
> + addq $(8*16), KEY_POWERS
As per the above, there's no need to increment KEY_POWERS here.
> + schoolbook2
> +.endm
> +
> +/*
> + * Compute poly on window size of %rdx blocks
> + * 0 < %rdx < NUM_PRECOMPUTE_POWERS
> + */
The code doesn't actually use %rdx directly. It should be BLOCKS_LEFT.
> +.macro partial_stride
> + pxor LO, LO
> + pxor HI, HI
> + pxor MI, MI
> + mov BLOCKS_LEFT, TMP
> + shlq $4, TMP
> + mov %rsi, KEY_POWERS
> + addq $(16*NUM_PRECOMPUTE_POWERS), KEY_POWERS
> + subq TMP, KEY_POWERS
> + # Multiply sum by h^N
> + movups (KEY_POWERS), %xmm0
> + movdqa SUM, %xmm1
> + schoolbook1_noload
> + schoolbook2
> + montgomery_reduction
> + movdqa PH, SUM
> + pxor LO, LO
> + pxor HI, HI
> + pxor MI, MI
> + xor IDX, IDX
> +.LloopPartial:
> + cmpq BLOCKS_LEFT, IDX # IDX < rdx
> + jae .LloopExitPartial
> +
> + movq BLOCKS_LEFT, TMP
> + subq IDX, TMP # TMP = rdx - IDX
> +
> + cmp $4, TMP # TMP < 4 ?
> + jl .Llt4Partial
> + schoolbook1 4
> + addq $4, IDX
> + addq $(4*16), MSG
> + addq $(4*16), KEY_POWERS
> + jmp .LoutPartial
> +.Llt4Partial:
> + cmp $3, TMP # TMP < 3 ?
> + jl .Llt3Partial
> + schoolbook1 3
> + addq $3, IDX
> + addq $(3*16), MSG
> + addq $(3*16), KEY_POWERS
> + jmp .LoutPartial
> +.Llt3Partial:
> + cmp $2, TMP # TMP < 2 ?
> + jl .Llt2Partial
> + schoolbook1 2
> + addq $2, IDX
> + addq $(2*16), MSG
> + addq $(2*16), KEY_POWERS
> + jmp .LoutPartial
> +.Llt2Partial:
> + schoolbook1 1 # TMP < 1 ?
> + addq $1, IDX
> + addq $(1*16), MSG
> + addq $(1*16), KEY_POWERS
> +.LoutPartial:
> + jmp .LloopPartial
> +.LloopExitPartial:
> + schoolbook2
> + montgomery_reduction
> + pxor PH, SUM
> +.endm
This can be simplified and optimized quite a bit:
- The first schoolbook2 and montgomery_reduction are unnecessary.
- The IDX variable is unnecessary.
- There's no need for a loop if there are going to be separate cases for 4, 2,
and 1 blocks anyway. We can just always jump forward.
- There's no need to increment MSG and KEY_POWERS after the last block.
Can you consider the following?
/*
* Process BLOCKS_LEFT blocks, where 0 < BLOCKS_LEFT < STRIDE_BLOCKS
*/
.macro partial_stride
mov BLOCKS_LEFT, TMP
shlq $4, TMP
addq $(16*STRIDE_BLOCKS), KEY_POWERS
subq TMP, KEY_POWERS
movups (MSG), %xmm0
pxor SUM, %xmm0
movaps (KEY_POWERS), %xmm1
schoolbook1_noload
dec BLOCKS_LEFT
addq $16, MSG
addq $16, KEY_POWERS
test $4, BLOCKS_LEFT
jz .Lpartial4BlocksDone
schoolbook1 4
addq $(4*16), MSG
addq $(4*16), KEY_POWERS
.Lpartial4BlocksDone:
test $2, BLOCKS_LEFT
jz .Lpartial2BlocksDone
schoolbook1 2
addq $(2*16), MSG
addq $(2*16), KEY_POWERS
.Lpartial2BlocksDone:
test $1, BLOCKS_LEFT
jz .LpartialDone
schoolbook1 1
.LpartialDone:
schoolbook2
montgomery_reduction SUM
.endm
> + FRAME_END
> + ret
> +SYM_FUNC_END(clmul_polyval_mul)
It needs to be RET, not ret. See https://git.kernel.org/linus/f94909ceb1ed4bfd
> +
> +/*
> + * Perform polynomial evaluation as specified by POLYVAL. This computes:
> + * h^n * accumulator + h^n * m_0 + ... + h^1 * m_{n-1}
> + * where n=nblocks, h is the hash key, and m_i are the message blocks.
> + *
> + * rdi - pointer to message blocks
> + * rsi - pointer to precomputed key powers h^8 ... h^1
> + * rdx - number of blocks to hash
> + * rcx - pointer to the accumulator
> + *
> + * void clmul_polyval_update(const u8 *in, const struct polyval_ctx *ctx,
> + * size_t nblocks, u8 *accumulator);
> + */
> +SYM_FUNC_START(clmul_polyval_update)
> + FRAME_BEGIN
> + vmovdqa .Lgstar(%rip), GSTAR
> + movups (%rcx), SUM
> + cmpq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
> + jb .LstrideLoopExit
> + full_stride 0
> + subq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
> +.LstrideLoop:
> + cmpq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
> + jb .LstrideLoopExitReduce
> + full_stride 1
> + subq $NUM_PRECOMPUTE_POWERS, BLOCKS_LEFT
> + jmp .LstrideLoop
> +.LstrideLoopExitReduce:
> + montgomery_reduction
> + movdqa PH, SUM
> +.LstrideLoopExit:
> + test BLOCKS_LEFT, BLOCKS_LEFT
> + je .LskipPartial
> + partial_stride
> +.LskipPartial:
> + movups SUM, (%rcx)
> + FRAME_END
> + ret
> +SYM_FUNC_END(clmul_polyval_update)
There are several unneeded instructions above. Unconditional jumps can be
avoided, as can comparisons if they are already paired with subtractions using
the same amounts (since on x86, subtractions set the flags too).
Consider the following:
SYM_FUNC_START(clmul_polyval_update)
FRAME_BEGIN
vmovdqa .Lgstar(%rip), GSTAR
movups (%rcx), SUM
subq $STRIDE_BLOCKS, BLOCKS_LEFT
js .LstrideLoopExit
full_stride 0
subq $STRIDE_BLOCKS, BLOCKS_LEFT
js .LstrideLoopExitReduce
.LstrideLoop:
full_stride 1
subq $STRIDE_BLOCKS, BLOCKS_LEFT
jns .LstrideLoop
.LstrideLoopExitReduce:
montgomery_reduction SUM
.LstrideLoopExit:
add $STRIDE_BLOCKS, BLOCKS_LEFT
jz .LskipPartial
partial_stride
.LskipPartial:
movups SUM, (%rcx)
FRAME_END
RET
SYM_FUNC_END(clmul_polyval_update)
- Eric
On Tue, Mar 15, 2022 at 11:00:34PM +0000, Nathan Huckleberry wrote:
> Add hardware accelerated version of POLYVAL for ARM64 CPUs with
> Crypto Extension support.
Nit: It's "Crypto Extensions", not "Crypto Extension".
> +config CRYPTO_POLYVAL_ARM64_CE
> + tristate "POLYVAL using ARMv8 Crypto Extensions (for HCTR2)"
> + depends on KERNEL_MODE_NEON
> + select CRYPTO_CRYPTD
> + select CRYPTO_HASH
> + select CRYPTO_POLYVAL
CRYPTO_POLYVAL selects CRYPTO_HASH already, so there's no need to select it
here.
> +/*
> + * Perform polynomial evaluation as specified by POLYVAL. This computes:
> + * h^n * accumulator + h^n * m_0 + ... + h^1 * m_{n-1}
> + * where n=nblocks, h is the hash key, and m_i are the message blocks.
> + *
> + * x0 - pointer to message blocks
> + * x1 - pointer to precomputed key powers h^8 ... h^1
> + * x2 - number of blocks to hash
> + * x3 - pointer to accumulator
> + *
> + * void pmull_polyval_update(const u8 *in, const struct polyval_ctx *ctx,
> + * size_t nblocks, u8 *accumulator);
> + */
> +SYM_FUNC_START(pmull_polyval_update)
> + adr TMP, .Lgstar
> + ld1 {GSTAR.2d}, [TMP]
> + ld1 {SUM.16b}, [x3]
> + ands PARTIAL_LEFT, BLOCKS_LEFT, #7
> + beq .LskipPartial
> + partial_stride
> +.LskipPartial:
> + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
> + blt .LstrideLoopExit
> + ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [x1], #64
> + ld1 {KEY4.16b, KEY3.16b, KEY2.16b, KEY1.16b}, [x1], #64
> + full_stride 0
> + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
> + blt .LstrideLoopExitReduce
> +.LstrideLoop:
> + full_stride 1
> + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
> + bge .LstrideLoop
> +.LstrideLoopExitReduce:
> + montgomery_reduction
> + mov SUM.16b, PH.16b
> +.LstrideLoopExit:
> + st1 {SUM.16b}, [x3]
> + ret
> +SYM_FUNC_END(pmull_polyval_update)
Is there a reason why partial_stride is done first in the arm64 implementation,
but last in the x86 implementation? It would be nice if the implementations
worked the same way. Probably last would be better? What is the advantage of
doing it first?
Besides that, many of the comments I made on the x86 implementation apply to the
arm64 implementation too.
- Eric
On Wed, Mar 23, 2022 at 8:37 PM Eric Biggers <[email protected]> wrote:
>
> On Tue, Mar 15, 2022 at 11:00:34PM +0000, Nathan Huckleberry wrote:
> > Add hardware accelerated version of POLYVAL for ARM64 CPUs with
> > Crypto Extension support.
>
> Nit: It's "Crypto Extensions", not "Crypto Extension".
>
> > +config CRYPTO_POLYVAL_ARM64_CE
> > + tristate "POLYVAL using ARMv8 Crypto Extensions (for HCTR2)"
> > + depends on KERNEL_MODE_NEON
> > + select CRYPTO_CRYPTD
> > + select CRYPTO_HASH
> > + select CRYPTO_POLYVAL
>
> CRYPTO_POLYVAL selects CRYPTO_HASH already, so there's no need to select it
> here.
>
> > +/*
> > + * Perform polynomial evaluation as specified by POLYVAL. This computes:
> > + * h^n * accumulator + h^n * m_0 + ... + h^1 * m_{n-1}
> > + * where n=nblocks, h is the hash key, and m_i are the message blocks.
> > + *
> > + * x0 - pointer to message blocks
> > + * x1 - pointer to precomputed key powers h^8 ... h^1
> > + * x2 - number of blocks to hash
> > + * x3 - pointer to accumulator
> > + *
> > + * void pmull_polyval_update(const u8 *in, const struct polyval_ctx *ctx,
> > + * size_t nblocks, u8 *accumulator);
> > + */
> > +SYM_FUNC_START(pmull_polyval_update)
> > + adr TMP, .Lgstar
> > + ld1 {GSTAR.2d}, [TMP]
> > + ld1 {SUM.16b}, [x3]
> > + ands PARTIAL_LEFT, BLOCKS_LEFT, #7
> > + beq .LskipPartial
> > + partial_stride
> > +.LskipPartial:
> > + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
> > + blt .LstrideLoopExit
> > + ld1 {KEY8.16b, KEY7.16b, KEY6.16b, KEY5.16b}, [x1], #64
> > + ld1 {KEY4.16b, KEY3.16b, KEY2.16b, KEY1.16b}, [x1], #64
> > + full_stride 0
> > + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
> > + blt .LstrideLoopExitReduce
> > +.LstrideLoop:
> > + full_stride 1
> > + subs BLOCKS_LEFT, BLOCKS_LEFT, #NUM_PRECOMPUTE_POWERS
> > + bge .LstrideLoop
> > +.LstrideLoopExitReduce:
> > + montgomery_reduction
> > + mov SUM.16b, PH.16b
> > +.LstrideLoopExit:
> > + st1 {SUM.16b}, [x3]
> > + ret
> > +SYM_FUNC_END(pmull_polyval_update)
>
> Is there a reason why partial_stride is done first in the arm64 implementation,
> but last in the x86 implementation? It would be nice if the implementations
> worked the same way. Probably last would be better? What is the advantage of
> doing it first?
It was so I could return early without loading keys into registers,
since I only need them if there's
a full stride. I was able to rewrite it in the same way that the x86
implementation works.
>
> Besides that, many of the comments I made on the x86 implementation apply to the
> arm64 implementation too.
>
> - Eric