The generic AES implementation we rely on if no architecture specific
one is available relies on lookup tables that are relatively large with
respect to the typical L1 D-cache size, which not only affects
performance, it may also result in timing variances that correlate with
the encryption keys.
So we tend to avoid the generic code if we can, usually by using a
driver that makes use of special AES instructions which supplant most of
the logic of the table based implementation the AES algorithm.
The Zkn RISC-V extension provides another interesting take on this: it
defines instructions operating on scalar registers that implement the
table lookups without relying on tables in memory. Those tables carry
32-bit quantities, making them a natural fit for a 32-bit architecture.
And given the use of scalars, we don't have to rely in in-kernel SIMD,
which is a bonus.
So let's use the instructions to implement the core AES cipher for RV32.
Cc: Paul Walmsley <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Albert Ou <[email protected]>
Cc: Christoph Müllner <[email protected]>
Cc: Heiko Stuebner <[email protected]>
Signed-off-by: Ard Biesheuvel <[email protected]>
---
arch/riscv/crypto/Kconfig | 12 ++
arch/riscv/crypto/Makefile | 3 +
arch/riscv/crypto/aes-riscv32-glue.c | 75 ++++++++++++
arch/riscv/crypto/aes-riscv32-zkned.S | 119 ++++++++++++++++++++
4 files changed, 209 insertions(+)
diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig
index 7542330916079447..fa3917859c2bbbc3 100644
--- a/arch/riscv/crypto/Kconfig
+++ b/arch/riscv/crypto/Kconfig
@@ -2,6 +2,18 @@
menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
+config CRYPTO_AES_RISCV32
+ tristate "Scalar AES using the Zkn extension"
+ depends on !64BIT
+ select CRYPTO_LIB_AES
+ help
+ Implement scalar AES using the RV32 instructions specified by the Zkn
+ extension. These instructions replace the table lookups used by the
+ generic implementation, making this implementation time invariant
+ (provided that the instructions themselves are). It also reduces the
+ D-cache footprint substantially, as the AES lookup tables are quite
+ large.
+
config CRYPTO_AES_RISCV
tristate "Ciphers: AES (RISCV)"
depends on 64BIT && RISCV_ISA_V
diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile
index f7faba6c12c9d863..d073b18d2a0bbba3 100644
--- a/arch/riscv/crypto/Makefile
+++ b/arch/riscv/crypto/Makefile
@@ -3,6 +3,9 @@
# linux/arch/riscv/crypto/Makefile
#
+obj-$(CONFIG_CRYPTO_AES_RISCV32) += aes-riscv32.o
+aes-riscv32-y := aes-riscv32-glue.o aes-riscv32-zkned.o
+
obj-$(CONFIG_CRYPTO_AES_RISCV) += aes-riscv.o
aes-riscv-y := aes-riscv-glue.o aes-riscv64-zvkned.o
diff --git a/arch/riscv/crypto/aes-riscv32-glue.c b/arch/riscv/crypto/aes-riscv32-glue.c
new file mode 100644
index 0000000000000000..2055213304e21829
--- /dev/null
+++ b/arch/riscv/crypto/aes-riscv32-glue.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Scalar AES core transform
+ *
+ * Copyright (C) 2023 Google, LLC.
+ * Author: Ard Biesheuvel <[email protected]>
+ */
+
+#include <crypto/aes.h>
+#include <crypto/algapi.h>
+#include <linux/module.h>
+
+asmlinkage void __aes_riscv32_encrypt(u32 *rk, u8 *out, const u8 *in, int rounds);
+asmlinkage void __aes_riscv32_decrypt(u32 *rk, u8 *out, const u8 *in, int rounds);
+
+static int aes_riscv32_set_key(struct crypto_tfm *tfm, const u8 *in_key,
+ unsigned int key_len)
+{
+ struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return aes_expandkey(ctx, in_key, key_len);
+}
+
+static void aes_riscv32_encrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
+{
+ struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+ int rounds = 6 + ctx->key_length / 4;
+
+ __aes_riscv32_encrypt(ctx->key_enc, out, in, rounds);
+}
+
+static void aes_riscv32_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
+{
+ struct crypto_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+ int rounds = 6 + ctx->key_length / 4;
+
+ __aes_riscv32_decrypt(ctx->key_dec, out, in, rounds);
+}
+
+static struct crypto_alg aes_alg = {
+ .cra_name = "aes",
+ .cra_driver_name = "aes-riscv32",
+ .cra_priority = 200,
+ .cra_flags = CRYPTO_ALG_TYPE_CIPHER,
+ .cra_blocksize = AES_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct crypto_aes_ctx),
+ .cra_module = THIS_MODULE,
+
+ .cra_cipher.cia_min_keysize = AES_MIN_KEY_SIZE,
+ .cra_cipher.cia_max_keysize = AES_MAX_KEY_SIZE,
+ .cra_cipher.cia_setkey = aes_riscv32_set_key,
+ .cra_cipher.cia_encrypt = aes_riscv32_encrypt,
+ .cra_cipher.cia_decrypt = aes_riscv32_decrypt
+};
+
+static int __init riscv32_aes_init(void)
+{
+ if (!riscv_isa_extension_available(NULL, ZKNE) ||
+ !riscv_isa_extension_available(NULL, ZKND))
+ return -ENODEV;
+ return crypto_register_alg(&aes_alg);
+}
+
+static void __exit riscv32_aes_fini(void)
+{
+ crypto_unregister_alg(&aes_alg);
+}
+
+module_init(riscv32_aes_init);
+module_exit(riscv32_aes_fini);
+
+MODULE_DESCRIPTION("Scalar AES cipher for riscv32");
+MODULE_AUTHOR("Ard Biesheuvel <[email protected]>");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_CRYPTO("aes");
diff --git a/arch/riscv/crypto/aes-riscv32-zkned.S b/arch/riscv/crypto/aes-riscv32-zkned.S
new file mode 100644
index 0000000000000000..89276bdc57b0045f
--- /dev/null
+++ b/arch/riscv/crypto/aes-riscv32-zkned.S
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Scalar AES core transform
+ *
+ * Copyright (C) 2023 Google, LLC.
+ * Author: Ard Biesheuvel <[email protected]>
+ */
+
+#include <linux/linkage.h>
+
+ .irpc r, 23456789
+ .set .Ls\r, \r + 16
+ .endr
+
+ // Zkn RV32 opcodes
+ .macro aes32esmi, rd, rs1, rs2, bs
+ .long 0x26000033 | (\bs << 30) | (.L\rd << 7) | (.L\rs1 << 15) | (.L\rs2 << 20)
+ .endm
+
+ .macro aes32esi, rd, rs1, rs2, bs
+ .long 0x22000033 | (\bs << 30) | (.L\rd << 7) | (.L\rs1 << 15) | (.L\rs2 << 20)
+ .endm
+
+ .macro aes32dsmi, rd, rs1, rs2, bs
+ .long 0x2e000033 | (\bs << 30) | (.L\rd << 7) | (.L\rs1 << 15) | (.L\rs2 << 20)
+ .endm
+
+ .macro aes32dsi, rd, rs1, rs2, bs
+ .long 0x2a000033 | (\bs << 30) | (.L\rd << 7) | (.L\rs1 << 15) | (.L\rs2 << 20)
+ .endm
+
+ // AES quarter round
+ .macro qround, op, o0, o1, o2, o3, i0, i1, i2, i3, bs
+ aes32\op \o0, \o0, \i0, \bs
+ aes32\op \o1, \o1, \i1, \bs
+ aes32\op \o2, \o2, \i2, \bs
+ aes32\op \o3, \o3, \i3, \bs
+ .endm
+
+ // One AES round
+ .macro round, ed, op, o0, o1, o2, o3, i0, i1, i2, i3
+ .ifc \ed,e
+ qround e\op, \o0, \o1, \o2, \o3, \i0, \i1, \i2, \i3, 0
+ qround e\op, \o0, \o1, \o2, \o3, \i1, \i2, \i3, \i0, 1
+ qround e\op, \o0, \o1, \o2, \o3, \i2, \i3, \i0, \i1, 2
+ qround e\op, \o0, \o1, \o2, \o3, \i3, \i0, \i1, \i2, 3
+ .else
+ qround d\op, \o0, \o1, \o2, \o3, \i0, \i1, \i2, \i3, 0
+ qround d\op, \o0, \o1, \o2, \o3, \i3, \i0, \i1, \i2, 1
+ qround d\op, \o0, \o1, \o2, \o3, \i2, \i3, \i0, \i1, 2
+ qround d\op, \o0, \o1, \o2, \o3, \i1, \i2, \i3, \i0, 3
+ .endif
+ .endm
+
+ // Load next round key and advance round key pointer
+ .macro next_rk, rk, out0, out1, out2, out3
+ lw \out0, 0(\rk)
+ lw \out1, 4(\rk)
+ lw \out2, 8(\rk)
+ lw \out3, 12(\rk)
+ add \rk, \rk, 16
+ .endm
+
+ .macro crypt, ed
+ add sp, sp, -32
+ sw s2, 0(sp)
+ sw s3, 4(sp)
+ sw s4, 8(sp)
+ sw s5, 12(sp)
+ sw s6, 16(sp)
+ sw s7, 20(sp)
+ sw s8, 24(sp)
+ sw s9, 28(sp)
+
+ lw s2, 0(a2)
+ lw s3, 4(a2)
+ lw s4, 8(a2)
+ lw s5, 12(a2)
+
+ next_rk a0, s6, s7, s8, s9
+
+ xor s2, s2, s6
+ xor s3, s3, s7
+ xor s4, s4, s8
+ xor s5, s5, s9
+
+0: add a3, a3, -2
+ next_rk a0, s6, s7, s8, s9
+ round \ed, smi, s6, s7, s8, s9, s2, s3, s4, s5
+ next_rk a0, s2, s3, s4, s5
+ beqz a3, 1f
+ round \ed, smi, s2, s3, s4, s5, s6, s7, s8, s9
+ j 0b
+1: round \ed, si, s2, s3, s4, s5, s6, s7, s8, s9
+
+ sw s2, 0(a1)
+ sw s3, 4(a1)
+ sw s4, 8(a1)
+ sw s5, 12(a1)
+
+ lw s2, 0(sp)
+ lw s3, 4(sp)
+ lw s4, 8(sp)
+ lw s5, 12(sp)
+ lw s6, 16(sp)
+ lw s7, 20(sp)
+ lw s8, 24(sp)
+ lw s9, 28(sp)
+ add sp, sp, 32
+ ret
+ .endm
+
+SYM_FUNC_START(__aes_riscv32_encrypt)
+ crypt e
+SYM_FUNC_END(__aes_riscv32_encrypt)
+
+SYM_FUNC_START(__aes_riscv32_decrypt)
+ crypt d
+SYM_FUNC_END(__aes_riscv32_decrypt)
--
2.39.2
On Wed, Jul 26, 2023 at 07:29:58PM +0200, Ard Biesheuvel wrote:
> The generic AES implementation we rely on if no architecture specific
> one is available relies on lookup tables that are relatively large with
> respect to the typical L1 D-cache size, which not only affects
> performance, it may also result in timing variances that correlate with
> the encryption keys.
>
> So we tend to avoid the generic code if we can, usually by using a
> driver that makes use of special AES instructions which supplant most of
> the logic of the table based implementation the AES algorithm.
>
> The Zkn RISC-V extension provides another interesting take on this: it
> defines instructions operating on scalar registers that implement the
> table lookups without relying on tables in memory. Those tables carry
> 32-bit quantities, making them a natural fit for a 32-bit architecture.
> And given the use of scalars, we don't have to rely in in-kernel SIMD,
> which is a bonus.
>
> So let's use the instructions to implement the core AES cipher for RV32.
>
> Cc: Paul Walmsley <[email protected]>
> Cc: Palmer Dabbelt <[email protected]>
> Cc: Albert Ou <[email protected]>
> Cc: Christoph M?llner <[email protected]>
> Cc: Heiko Stuebner <[email protected]>
> Signed-off-by: Ard Biesheuvel <[email protected]>
> ---
> arch/riscv/crypto/Kconfig | 12 ++
> arch/riscv/crypto/Makefile | 3 +
> arch/riscv/crypto/aes-riscv32-glue.c | 75 ++++++++++++
> arch/riscv/crypto/aes-riscv32-zkned.S | 119 ++++++++++++++++++++
> 4 files changed, 209 insertions(+)
Hi Ard:
Any chance you could postpone this til after I've finished removing
crypto_cipher?
Thanks,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Fri, 4 Aug 2023 at 10:21, Herbert Xu <[email protected]> wrote:
>
> On Wed, Jul 26, 2023 at 07:29:58PM +0200, Ard Biesheuvel wrote:
> > The generic AES implementation we rely on if no architecture specific
> > one is available relies on lookup tables that are relatively large with
> > respect to the typical L1 D-cache size, which not only affects
> > performance, it may also result in timing variances that correlate with
> > the encryption keys.
> >
> > So we tend to avoid the generic code if we can, usually by using a
> > driver that makes use of special AES instructions which supplant most of
> > the logic of the table based implementation the AES algorithm.
> >
> > The Zkn RISC-V extension provides another interesting take on this: it
> > defines instructions operating on scalar registers that implement the
> > table lookups without relying on tables in memory. Those tables carry
> > 32-bit quantities, making them a natural fit for a 32-bit architecture.
> > And given the use of scalars, we don't have to rely in in-kernel SIMD,
> > which is a bonus.
> >
> > So let's use the instructions to implement the core AES cipher for RV32.
> >
> > Cc: Paul Walmsley <[email protected]>
> > Cc: Palmer Dabbelt <[email protected]>
> > Cc: Albert Ou <[email protected]>
> > Cc: Christoph Müllner <[email protected]>
> > Cc: Heiko Stuebner <[email protected]>
> > Signed-off-by: Ard Biesheuvel <[email protected]>
> > ---
> > arch/riscv/crypto/Kconfig | 12 ++
> > arch/riscv/crypto/Makefile | 3 +
> > arch/riscv/crypto/aes-riscv32-glue.c | 75 ++++++++++++
> > arch/riscv/crypto/aes-riscv32-zkned.S | 119 ++++++++++++++++++++
> > 4 files changed, 209 insertions(+)
>
> Hi Ard:
>
> Any chance you could postpone this til after I've finished removing
> crypto_cipher?
>
That's fine with me. Do you have an ETA on that? Need any help?
I have implemented the scalar 64-bit counterpart as well in the mean time
On Fri, 4 Aug 2023 at 10:31, Ard Biesheuvel <[email protected]> wrote:
>
> On Fri, 4 Aug 2023 at 10:21, Herbert Xu <[email protected]> wrote:
> >
...
> > Hi Ard:
> >
> > Any chance you could postpone this til after I've finished removing
> > crypto_cipher?
> >
>
> That's fine with me. Do you have an ETA on that? Need any help?
>
> I have implemented the scalar 64-bit counterpart as well in the mean time
Is this still happening?
On Thu, 31 Aug 2023 10:10:21 PDT (-0700), Ard Biesheuvel wrote:
> On Fri, 4 Aug 2023 at 10:31, Ard Biesheuvel <[email protected]> wrote:
>>
>> On Fri, 4 Aug 2023 at 10:21, Herbert Xu <[email protected]> wrote:
>> >
> ...
>
>> > Hi Ard:
>> >
>> > Any chance you could postpone this til after I've finished removing
>> > crypto_cipher?
>> >
>>
>> That's fine with me. Do you have an ETA on that? Need any help?
>>
>> I have implemented the scalar 64-bit counterpart as well in the mean time
>
> Is this still happening?
I don't really know much about the crypto stuff, but looks like there's
still a "struct crypto_cipher" in my trees. Am I still supposed to be
waiting on something?
Acked-by: Palmer Dabbelt <[email protected]>
In case it's eaiser to take this via some other tree.
On Fri, Oct 27, 2023 at 06:11:40AM -0700, Palmer Dabbelt wrote:
> On Thu, 31 Aug 2023 10:10:21 PDT (-0700), Ard Biesheuvel wrote:
> > On Fri, 4 Aug 2023 at 10:31, Ard Biesheuvel <[email protected]> wrote:
> > >
> > > On Fri, 4 Aug 2023 at 10:21, Herbert Xu <[email protected]> wrote:
> > > >
> > ...
> >
> > > > Hi Ard:
> > > >
> > > > Any chance you could postpone this til after I've finished removing
> > > > crypto_cipher?
> > > >
> > >
> > > That's fine with me. Do you have an ETA on that? Need any help?
> > >
> > > I have implemented the scalar 64-bit counterpart as well in the mean time
> >
> > Is this still happening?
>
> I don't really know much about the crypto stuff, but looks like there's
> still a "struct crypto_cipher" in my trees. Am I still supposed to be
> waiting on something?
Regardless of crypto_cipher structs, this needs whatever series that
actually implements Zkn detection from DT/ACPI to be merged first,
as otherwise the definitions that iscv_isa_extension_available() depends
on don't exist.
>
> Acked-by: Palmer Dabbelt <[email protected]>
>
> In case it's eaiser to take this via some other tree.
>
> _______________________________________________
> linux-riscv mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-riscv
On Fri, Oct 27, 2023 at 06:11:40AM -0700, Palmer Dabbelt wrote:
>
> I don't really know much about the crypto stuff, but looks like there's
> still a "struct crypto_cipher" in my trees. Am I still supposed to be
> waiting on something?
Yes lskcipher has now been added. Please don't add any new code
through the old crypto_cipher interface.
Thanks,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Fri, 27 Oct 2023 at 15:17, Conor Dooley <[email protected]> wrote:
>
> On Fri, Oct 27, 2023 at 06:11:40AM -0700, Palmer Dabbelt wrote:
> > On Thu, 31 Aug 2023 10:10:21 PDT (-0700), Ard Biesheuvel wrote:
> > > On Fri, 4 Aug 2023 at 10:31, Ard Biesheuvel <[email protected]> wrote:
> > > >
> > > > On Fri, 4 Aug 2023 at 10:21, Herbert Xu <[email protected]> wrote:
> > > > >
> > > ...
> > >
> > > > > Hi Ard:
> > > > >
> > > > > Any chance you could postpone this til after I've finished removing
> > > > > crypto_cipher?
> > > > >
> > > >
> > > > That's fine with me. Do you have an ETA on that? Need any help?
> > > >
> > > > I have implemented the scalar 64-bit counterpart as well in the mean time
> > >
> > > Is this still happening?
> >
> > I don't really know much about the crypto stuff, but looks like there's
> > still a "struct crypto_cipher" in my trees. Am I still supposed to be
> > waiting on something?
>
> Regardless of crypto_cipher structs, this needs whatever series that
> actually implements Zkn detection from DT/ACPI to be merged first,
> as otherwise the definitions that iscv_isa_extension_available() depends
> on don't exist.
>
Please disregard this patch. I have an updated version already, but
I'll need to rebase it once the prerequisites are in place.
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=riscv-scalar-aes