2012-06-23 00:44:56

by Kim Phillips

Subject: [PATCH 00/20] crypto: caam - driver updates

The following updates bring the upstream CAAM driver mostly up to date wrt
Freescale's SDK 1.2 release:

patch 1 is a cosmetic fix

patch 2 is a mapping size fix

patches 3-5 are protocol descriptor generation updates (not currently used
in the kernel driver, but these header files are shared with other projects in
the SDK; hope that's ok).

patches 6-9 refactor code in preparation for ahash and hwrng support; and
patch 7 removes the job ring registration interface in favour of user-managed
selection and sharing of rings (see caam_rng_init, which simply chooses
job ring 0 for all its operations).

patches 10-12 add ahash support (keyed and unkeyed)

patch 13 adds hwrng support

patches 14-15 add support for the SEC v5.x series, which most notably gets a
new RNG (tested on P5040 h/w)

patches 16-20 contain various performance improvements

There are two patches remaining in the SDK that I'm holding off for now,
as I believe they need more testing: use NAPI instead of tasklet, and
set descriptor sharing type to SERIAL.

Since the ahash support patch is quite large, and I have no way of telling
if it'll make the list, I've pushed the entire patchseries to a git tree based
on Herbert's cryptodev tree:

git://git.freescale.com/crypto/cryptodev.git master

So if a patch doesn't make it, please fetch it from the tree in order to
review (or suggest an alternative method of submitting these large changes).

Thanks to everyone that contributed to this patchseries,

Cheers,

Kim

Hemant Agrawal (1):
crypto: caam - add PDB (Protocol Descriptor Block) definitions

Kim Phillips (10):
crypto: caam - remove line continuations from
ablkcipher_append_src_dst
crypto: caam - fix input job ring element dma mapping size
crypto: caam - fix descriptor length adjustments for protocol
descriptors
crypto: caam - assign 40-bit masks on SEC v5.0 and above
crypto: caam - add support for SEC v5.x RNG4
crypto: caam - disable IRQ coalescing by default
crypto: caam - use non-irq versions of spinlocks for job rings
crypto: caam - only query h/w in job ring dequeue path
crypto: caam - consolidate memory barriers from job ring en/dequeue
crypto: caam - one tasklet per job ring

Yashpal Dutta (1):
crypto: caam - fix start index for Protocol shared descriptors

Yuan Kang (8):
crypto: caam - support external seq in/out lengths
crypto: caam - remove jr register/deregister
crypto: caam - refactor key_gen, sg
crypto: caam - link_tbl rename
crypto: caam - ahash hmac support
crypto: caam - unkeyed ahash support
crypto: caam - chaining support
crypto: caam - hwrng support

drivers/crypto/caam/Kconfig | 30 +-
drivers/crypto/caam/Makefile | 4 +-
drivers/crypto/caam/caamalg.c | 572 ++++-------
drivers/crypto/caam/caamhash.c | 1878 +++++++++++++++++++++++++++++++++++++
drivers/crypto/caam/caamrng.c | 309 ++++++
drivers/crypto/caam/compat.h | 2 +
drivers/crypto/caam/ctrl.c | 138 ++-
drivers/crypto/caam/desc.h | 31 +-
drivers/crypto/caam/desc_constr.h | 57 +-
drivers/crypto/caam/error.c | 44 +-
drivers/crypto/caam/intern.h | 6 +-
drivers/crypto/caam/jr.c | 80 +-
drivers/crypto/caam/key_gen.c | 122 +++
drivers/crypto/caam/key_gen.h | 17 +
drivers/crypto/caam/pdb.h | 401 ++++++++
drivers/crypto/caam/regs.h | 32 +-
drivers/crypto/caam/sg_sw_sec4.h | 156 +++
17 files changed, 3415 insertions(+), 464 deletions(-)
create mode 100644 drivers/crypto/caam/caamhash.c
create mode 100644 drivers/crypto/caam/caamrng.c
create mode 100644 drivers/crypto/caam/key_gen.c
create mode 100644 drivers/crypto/caam/key_gen.h
create mode 100644 drivers/crypto/caam/pdb.h
create mode 100644 drivers/crypto/caam/sg_sw_sec4.h

--
1.7.11.1

2012-06-23 00:45:02

by Kim Phillips

Subject: [PATCH 01/20] crypto: caam - remove line continuations from ablkcipher_append_src_dst

presumably leftovers from possible macro development.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/caamalg.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index 4eec389..5c10dc5 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -143,11 +143,11 @@ static inline void aead_append_ld_iv(u32 *desc, int ivsize)
*/
static inline void ablkcipher_append_src_dst(u32 *desc)
{
- append_math_add(desc, VARSEQOUTLEN, SEQINLEN, REG0, CAAM_CMD_SZ); \
- append_math_add(desc, VARSEQINLEN, SEQINLEN, REG0, CAAM_CMD_SZ); \
- append_seq_fifo_load(desc, 0, FIFOLD_CLASS_CLASS1 | \
- KEY_VLF | FIFOLD_TYPE_MSG | FIFOLD_TYPE_LAST1); \
- append_seq_fifo_store(desc, 0, FIFOST_TYPE_MESSAGE_DATA | KEY_VLF); \
+ append_math_add(desc, VARSEQOUTLEN, SEQINLEN, REG0, CAAM_CMD_SZ);
+ append_math_add(desc, VARSEQINLEN, SEQINLEN, REG0, CAAM_CMD_SZ);
+ append_seq_fifo_load(desc, 0, FIFOLD_CLASS_CLASS1 |
+ KEY_VLF | FIFOLD_TYPE_MSG | FIFOLD_TYPE_LAST1);
+ append_seq_fifo_store(desc, 0, FIFOST_TYPE_MESSAGE_DATA | KEY_VLF);
}

/*
--
1.7.11.1

2012-06-23 00:45:02

by Kim Phillips

Subject: [PATCH 02/20] crypto: caam - fix input job ring element dma mapping size

SEC4 h/w gets configured in 32- vs. 36-bit physical
addressing modes depending on the size of dma_addr_t,
which is not always equal to sizeof(u32 *).

Also fixed alignment of a dma_unmap call whilst in there.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/jr.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 340fa32..6ce4c41 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -376,7 +376,7 @@ static int caam_jr_init(struct device *dev)

/* Setup rings */
inpbusaddr = dma_map_single(dev, jrp->inpring,
- sizeof(u32 *) * JOBR_DEPTH,
+ sizeof(dma_addr_t) * JOBR_DEPTH,
DMA_BIDIRECTIONAL);
if (dma_mapping_error(dev, inpbusaddr)) {
dev_err(dev, "caam_jr_init(): can't map input ring\n");
@@ -391,9 +391,9 @@ static int caam_jr_init(struct device *dev)
DMA_BIDIRECTIONAL);
if (dma_mapping_error(dev, outbusaddr)) {
dev_err(dev, "caam_jr_init(): can't map output ring\n");
- dma_unmap_single(dev, inpbusaddr,
- sizeof(u32 *) * JOBR_DEPTH,
- DMA_BIDIRECTIONAL);
+ dma_unmap_single(dev, inpbusaddr,
+ sizeof(dma_addr_t) * JOBR_DEPTH,
+ DMA_BIDIRECTIONAL);
kfree(jrp->inpring);
kfree(jrp->outring);
kfree(jrp->entinfo);
@@ -447,7 +447,7 @@ int caam_jr_shutdown(struct device *dev)
dma_unmap_single(dev, outbusaddr,
sizeof(struct jr_outentry) * JOBR_DEPTH,
DMA_BIDIRECTIONAL);
- dma_unmap_single(dev, inpbusaddr, sizeof(u32 *) * JOBR_DEPTH,
+ dma_unmap_single(dev, inpbusaddr, sizeof(dma_addr_t) * JOBR_DEPTH,
DMA_BIDIRECTIONAL);
kfree(jrp->outring);
kfree(jrp->inpring);
--
1.7.11.1

2012-06-23 00:45:05

by Kim Phillips

Subject: [PATCH 03/20] crypto: caam - fix start index for Protocol shared descriptors

From: Yashpal Dutta <[email protected]>

In case of protocol acceleration descriptors, Shared descriptor header must
carry size of header length + PDB length in words which will be skipped by
DECO while processing descriptor to provide first command word offset

Signed-off-by: Yashpal Dutta <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/desc_constr.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/caam/desc_constr.h b/drivers/crypto/caam/desc_constr.h
index 348b882..0d31e27 100644
--- a/drivers/crypto/caam/desc_constr.h
+++ b/drivers/crypto/caam/desc_constr.h
@@ -1,7 +1,7 @@
/*
* caam descriptor construction helper functions
*
- * Copyright 2008-2011 Freescale Semiconductor, Inc.
+ * Copyright 2008-2012 Freescale Semiconductor, Inc.
*/

#include "desc.h"
@@ -64,7 +64,7 @@ static inline void init_sh_desc_pdb(u32 *desc, u32 options, size_t pdb_bytes)
{
u32 pdb_len = pdb_bytes / CAAM_CMD_SZ + 1;

- init_sh_desc(desc, ((pdb_len << HDR_START_IDX_SHIFT) + pdb_len) |
+ init_sh_desc(desc, (((pdb_len + 1) << HDR_START_IDX_SHIFT) + pdb_len) |
options);
}

--
1.7.11.1

2012-06-23 00:45:06

by Kim Phillips

Subject: [PATCH 04/20] crypto: caam - fix descriptor length adjustments for protocol descriptors

init_desc, by always ORing with 1 for the descriptor header inclusion
into the descriptor length, and init_sh_desc_pdb, by always specifying
the descriptor length modification for the PDB via options, would not
allow for odd length PDBs to be embedded in the constructed descriptor
length. Fix this by simply changing the OR to an addition.

also round-up pdb_bytes to the next SEC command unit size, to
allow for, e.g., optional packet header bytes that aren't a
multiple of CAAM_CMD_SZ.

Reported-by: Radu-Andrei BULIE <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
Cc: Yashpal Dutta <[email protected]>
---
drivers/crypto/caam/desc_constr.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/caam/desc_constr.h b/drivers/crypto/caam/desc_constr.h
index 0d31e27..8e1056f 100644
--- a/drivers/crypto/caam/desc_constr.h
+++ b/drivers/crypto/caam/desc_constr.h
@@ -51,7 +51,7 @@ static inline void *sh_desc_pdb(u32 *desc)

static inline void init_desc(u32 *desc, u32 options)
{
- *desc = options | HDR_ONE | 1;
+ *desc = (options | HDR_ONE) + 1;
}

static inline void init_sh_desc(u32 *desc, u32 options)
@@ -62,7 +62,7 @@ static inline void init_sh_desc(u32 *desc, u32 options)

static inline void init_sh_desc_pdb(u32 *desc, u32 options, size_t pdb_bytes)
{
- u32 pdb_len = pdb_bytes / CAAM_CMD_SZ + 1;
+ u32 pdb_len = (pdb_bytes + CAAM_CMD_SZ - 1) / CAAM_CMD_SZ;

init_sh_desc(desc, (((pdb_len + 1) << HDR_START_IDX_SHIFT) + pdb_len) |
options);
--
1.7.11.1

2012-06-23 00:45:07

by Kim Phillips

Subject: [PATCH 05/20] crypto: caam - add PDB (Protocol Descriptor Block) definitions

From: Hemant Agrawal <[email protected]>

Add a PDB header file to support building protocol descriptors.

Signed-off-by: Steve Cornelius <[email protected]>
Signed-off-by: Hemant Agrawal <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/desc.h | 16 --
drivers/crypto/caam/pdb.h | 401 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 401 insertions(+), 16 deletions(-)
create mode 100644 drivers/crypto/caam/pdb.h

diff --git a/drivers/crypto/caam/desc.h b/drivers/crypto/caam/desc.h
index a17c295..af25e76 100644
--- a/drivers/crypto/caam/desc.h
+++ b/drivers/crypto/caam/desc.h
@@ -1585,20 +1585,4 @@
#define NFIFOENTRY_PLEN_SHIFT 0
#define NFIFOENTRY_PLEN_MASK (0xFF << NFIFOENTRY_PLEN_SHIFT)

-/*
- * PDB internal definitions
- */
-
-/* IPSec ESP CBC Encap/Decap Options */
-#define PDBOPTS_ESPCBC_ARSNONE 0x00 /* no antireplay window */
-#define PDBOPTS_ESPCBC_ARS32 0x40 /* 32-entry antireplay window */
-#define PDBOPTS_ESPCBC_ARS64 0xc0 /* 64-entry antireplay window */
-#define PDBOPTS_ESPCBC_IVSRC 0x20 /* IV comes from internal random gen */
-#define PDBOPTS_ESPCBC_ESN 0x10 /* extended sequence included */
-#define PDBOPTS_ESPCBC_OUTFMT 0x08 /* output only decapsulation (decap) */
-#define PDBOPTS_ESPCBC_IPHDRSRC 0x08 /* IP header comes from PDB (encap) */
-#define PDBOPTS_ESPCBC_INCIPHDR 0x04 /* Prepend IP header to output frame */
-#define PDBOPTS_ESPCBC_IPVSN 0x02 /* process IPv6 header */
-#define PDBOPTS_ESPCBC_TUNNEL 0x01 /* tunnel mode next-header byte */
-
#endif /* DESC_H */
diff --git a/drivers/crypto/caam/pdb.h b/drivers/crypto/caam/pdb.h
new file mode 100644
index 0000000..62950d2
--- /dev/null
+++ b/drivers/crypto/caam/pdb.h
@@ -0,0 +1,401 @@
+/*
+ * CAAM Protocol Data Block (PDB) definition header file
+ *
+ * Copyright 2008-2012 Freescale Semiconductor, Inc.
+ *
+ */
+
+#ifndef CAAM_PDB_H
+#define CAAM_PDB_H
+
+/*
+ * PDB- IPSec ESP Header Modification Options
+ */
+#define PDBHMO_ESP_DECAP_SHIFT 12
+#define PDBHMO_ESP_ENCAP_SHIFT 4
+/*
+ * Encap and Decap - Decrement TTL (Hop Limit) - Based on the value of the
+ * Options Byte IP version (IPvsn) field:
+ * if IPv4, decrement the inner IP header TTL field (byte 8);
+ * if IPv6 decrement the inner IP header Hop Limit field (byte 7).
+*/
+#define PDBHMO_ESP_DECAP_DEC_TTL (0x02 << PDBHMO_ESP_DECAP_SHIFT)
+#define PDBHMO_ESP_ENCAP_DEC_TTL (0x02 << PDBHMO_ESP_ENCAP_SHIFT)
+/*
+ * Decap - DiffServ Copy - Copy the IPv4 TOS or IPv6 Traffic Class byte
+ * from the outer IP header to the inner IP header.
+ */
+#define PDBHMO_ESP_DIFFSERV (0x01 << PDBHMO_ESP_DECAP_SHIFT)
+/*
+ * Encap- Copy DF bit -if an IPv4 tunnel mode outer IP header is coming from
+ * the PDB, copy the DF bit from the inner IP header to the outer IP header.
+ */
+#define PDBHMO_ESP_DFBIT (0x04 << PDBHMO_ESP_ENCAP_SHIFT)
+
+/*
+ * PDB - IPSec ESP Encap/Decap Options
+ */
+#define PDBOPTS_ESP_ARSNONE 0x00 /* no antireplay window */
+#define PDBOPTS_ESP_ARS32 0x40 /* 32-entry antireplay window */
+#define PDBOPTS_ESP_ARS64 0xc0 /* 64-entry antireplay window */
+#define PDBOPTS_ESP_IVSRC 0x20 /* IV comes from internal random gen */
+#define PDBOPTS_ESP_ESN 0x10 /* extended sequence included */
+#define PDBOPTS_ESP_OUTFMT 0x08 /* output only decapsulation (decap) */
+#define PDBOPTS_ESP_IPHDRSRC 0x08 /* IP header comes from PDB (encap) */
+#define PDBOPTS_ESP_INCIPHDR 0x04 /* Prepend IP header to output frame */
+#define PDBOPTS_ESP_IPVSN 0x02 /* process IPv6 header */
+#define PDBOPTS_ESP_TUNNEL 0x01 /* tunnel mode next-header byte */
+#define PDBOPTS_ESP_IPV6 0x02 /* ip header version is V6 */
+#define PDBOPTS_ESP_DIFFSERV 0x40 /* copy TOS/TC from inner iphdr */
+#define PDBOPTS_ESP_UPDATE_CSUM 0x80 /* encap-update ip header checksum */
+#define PDBOPTS_ESP_VERIFY_CSUM 0x20 /* decap-validate ip header checksum */
+
+/*
+ * General IPSec encap/decap PDB definitions
+ */
+struct ipsec_encap_cbc {
+ u32 iv[4];
+};
+
+struct ipsec_encap_ctr {
+ u32 ctr_nonce;
+ u32 ctr_initial;
+ u32 iv[2];
+};
+
+struct ipsec_encap_ccm {
+ u32 salt; /* lower 24 bits */
+ u8 b0_flags;
+ u8 ctr_flags;
+ u16 ctr_initial;
+ u32 iv[2];
+};
+
+struct ipsec_encap_gcm {
+ u32 salt; /* lower 24 bits */
+ u32 rsvd1;
+ u32 iv[2];
+};
+
+struct ipsec_encap_pdb {
+ u8 hmo_rsvd;
+ u8 ip_nh;
+ u8 ip_nh_offset;
+ u8 options;
+ u32 seq_num_ext_hi;
+ u32 seq_num;
+ union {
+ struct ipsec_encap_cbc cbc;
+ struct ipsec_encap_ctr ctr;
+ struct ipsec_encap_ccm ccm;
+ struct ipsec_encap_gcm gcm;
+ };
+ u32 spi;
+ u16 rsvd1;
+ u16 ip_hdr_len;
+ u32 ip_hdr[0]; /* optional IP Header content */
+};
+
+struct ipsec_decap_cbc {
+ u32 rsvd[2];
+};
+
+struct ipsec_decap_ctr {
+ u32 salt;
+ u32 ctr_initial;
+};
+
+struct ipsec_decap_ccm {
+ u32 salt;
+ u8 iv_flags;
+ u8 ctr_flags;
+ u16 ctr_initial;
+};
+
+struct ipsec_decap_gcm {
+ u32 salt;
+ u32 resvd;
+};
+
+struct ipsec_decap_pdb {
+ u16 hmo_ip_hdr_len;
+ u8 ip_nh_offset;
+ u8 options;
+ union {
+ struct ipsec_decap_cbc cbc;
+ struct ipsec_decap_ctr ctr;
+ struct ipsec_decap_ccm ccm;
+ struct ipsec_decap_gcm gcm;
+ };
+ u32 seq_num_ext_hi;
+ u32 seq_num;
+ u32 anti_replay[2];
+ u32 end_index[0];
+};
+
+/*
+ * IPSec ESP Datapath Protocol Override Register (DPOVRD)
+ */
+struct ipsec_deco_dpovrd {
+#define IPSEC_ENCAP_DECO_DPOVRD_USE 0x80
+ u8 ovrd_ecn;
+ u8 ip_hdr_len;
+ u8 nh_offset;
+ u8 next_header; /* reserved if decap */
+};
+
+/*
+ * IEEE 802.11i WiFi Protocol Data Block
+ */
+#define WIFI_PDBOPTS_FCS 0x01
+#define WIFI_PDBOPTS_AR 0x40
+
+struct wifi_encap_pdb {
+ u16 mac_hdr_len;
+ u8 rsvd;
+ u8 options;
+ u8 iv_flags;
+ u8 pri;
+ u16 pn1;
+ u32 pn2;
+ u16 frm_ctrl_mask;
+ u16 seq_ctrl_mask;
+ u8 rsvd1[2];
+ u8 cnst;
+ u8 key_id;
+ u8 ctr_flags;
+ u8 rsvd2;
+ u16 ctr_init;
+};
+
+struct wifi_decap_pdb {
+ u16 mac_hdr_len;
+ u8 rsvd;
+ u8 options;
+ u8 iv_flags;
+ u8 pri;
+ u16 pn1;
+ u32 pn2;
+ u16 frm_ctrl_mask;
+ u16 seq_ctrl_mask;
+ u8 rsvd1[4];
+ u8 ctr_flags;
+ u8 rsvd2;
+ u16 ctr_init;
+};
+
+/*
+ * IEEE 802.16 WiMAX Protocol Data Block
+ */
+#define WIMAX_PDBOPTS_FCS 0x01
+#define WIMAX_PDBOPTS_AR 0x40 /* decap only */
+
+struct wimax_encap_pdb {
+ u8 rsvd[3];
+ u8 options;
+ u32 nonce;
+ u8 b0_flags;
+ u8 ctr_flags;
+ u16 ctr_init;
+ /* begin DECO writeback region */
+ u32 pn;
+ /* end DECO writeback region */
+};
+
+struct wimax_decap_pdb {
+ u8 rsvd[3];
+ u8 options;
+ u32 nonce;
+ u8 iv_flags;
+ u8 ctr_flags;
+ u16 ctr_init;
+ /* begin DECO writeback region */
+ u32 pn;
+ u8 rsvd1[2];
+ u16 antireplay_len;
+ u64 antireplay_scorecard;
+ /* end DECO writeback region */
+};
+
+/*
+ * IEEE 801.AE MacSEC Protocol Data Block
+ */
+#define MACSEC_PDBOPTS_FCS 0x01
+#define MACSEC_PDBOPTS_AR 0x40 /* used in decap only */
+
+struct macsec_encap_pdb {
+ u16 aad_len;
+ u8 rsvd;
+ u8 options;
+ u64 sci;
+ u16 ethertype;
+ u8 tci_an;
+ u8 rsvd1;
+ /* begin DECO writeback region */
+ u32 pn;
+ /* end DECO writeback region */
+};
+
+struct macsec_decap_pdb {
+ u16 aad_len;
+ u8 rsvd;
+ u8 options;
+ u64 sci;
+ u8 rsvd1[3];
+ /* begin DECO writeback region */
+ u8 antireplay_len;
+ u32 pn;
+ u64 antireplay_scorecard;
+ /* end DECO writeback region */
+};
+
+/*
+ * SSL/TLS/DTLS Protocol Data Blocks
+ */
+
+#define TLS_PDBOPTS_ARS32 0x40
+#define TLS_PDBOPTS_ARS64 0xc0
+#define TLS_PDBOPTS_OUTFMT 0x08
+#define TLS_PDBOPTS_IV_WRTBK 0x02 /* 1.1/1.2/DTLS only */
+#define TLS_PDBOPTS_EXP_RND_IV 0x01 /* 1.1/1.2/DTLS only */
+
+struct tls_block_encap_pdb {
+ u8 type;
+ u8 version[2];
+ u8 options;
+ u64 seq_num;
+ u32 iv[4];
+};
+
+struct tls_stream_encap_pdb {
+ u8 type;
+ u8 version[2];
+ u8 options;
+ u64 seq_num;
+ u8 i;
+ u8 j;
+ u8 rsvd1[2];
+};
+
+struct dtls_block_encap_pdb {
+ u8 type;
+ u8 version[2];
+ u8 options;
+ u16 epoch;
+ u16 seq_num[3];
+ u32 iv[4];
+};
+
+struct tls_block_decap_pdb {
+ u8 rsvd[3];
+ u8 options;
+ u64 seq_num;
+ u32 iv[4];
+};
+
+struct tls_stream_decap_pdb {
+ u8 rsvd[3];
+ u8 options;
+ u64 seq_num;
+ u8 i;
+ u8 j;
+ u8 rsvd1[2];
+};
+
+struct dtls_block_decap_pdb {
+ u8 rsvd[3];
+ u8 options;
+ u16 epoch;
+ u16 seq_num[3];
+ u32 iv[4];
+ u64 antireplay_scorecard;
+};
+
+/*
+ * SRTP Protocol Data Blocks
+ */
+#define SRTP_PDBOPTS_MKI 0x08
+#define SRTP_PDBOPTS_AR 0x40
+
+struct srtp_encap_pdb {
+ u8 x_len;
+ u8 mki_len;
+ u8 n_tag;
+ u8 options;
+ u32 cnst0;
+ u8 rsvd[2];
+ u16 cnst1;
+ u16 salt[7];
+ u16 cnst2;
+ u32 rsvd1;
+ u32 roc;
+ u32 opt_mki;
+};
+
+struct srtp_decap_pdb {
+ u8 x_len;
+ u8 mki_len;
+ u8 n_tag;
+ u8 options;
+ u32 cnst0;
+ u8 rsvd[2];
+ u16 cnst1;
+ u16 salt[7];
+ u16 cnst2;
+ u16 rsvd1;
+ u16 seq_num;
+ u32 roc;
+ u64 antireplay_scorecard;
+};
+
+/*
+ * DSA/ECDSA Protocol Data Blocks
+ * Two of these exist: DSA-SIGN, and DSA-VERIFY. They are similar
+ * except for the treatment of "w" for verify, "s" for sign,
+ * and the placement of "a,b".
+ */
+#define DSA_PDB_SGF_SHIFT 24
+#define DSA_PDB_SGF_MASK (0xff << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_Q (0x80 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_R (0x40 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_G (0x20 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_W (0x10 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_S (0x10 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_F (0x08 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_C (0x04 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_D (0x02 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_AB_SIGN (0x02 << DSA_PDB_SGF_SHIFT)
+#define DSA_PDB_SGF_AB_VERIFY (0x01 << DSA_PDB_SGF_SHIFT)
+
+#define DSA_PDB_L_SHIFT 7
+#define DSA_PDB_L_MASK (0x3ff << DSA_PDB_L_SHIFT)
+
+#define DSA_PDB_N_MASK 0x7f
+
+struct dsa_sign_pdb {
+ u32 sgf_ln; /* Use DSA_PDB_ defintions per above */
+ u8 *q;
+ u8 *r;
+ u8 *g; /* or Gx,y */
+ u8 *s;
+ u8 *f;
+ u8 *c;
+ u8 *d;
+ u8 *ab; /* ECC only */
+ u8 *u;
+};
+
+struct dsa_verify_pdb {
+ u32 sgf_ln;
+ u8 *q;
+ u8 *r;
+ u8 *g; /* or Gx,y */
+ u8 *w; /* or Wx,y */
+ u8 *f;
+ u8 *c;
+ u8 *d;
+ u8 *tmp; /* temporary data block */
+ u8 *ab; /* only used if ECC processing */
+};
+
+#endif
--
1.7.11.1

2012-06-23 00:50:57

by Kim Phillips

Subject: [PATCH 19/20] crypto: caam - consolidate memory barriers from job ring en/dequeue

Memory barriers are implied by the i/o register write implementation
(at least on Power). So we can remove the redundant wmb() in
caam_jr_enqueue, and, in dequeue(), hoist the h/w done notification
write up to before we need to increment the head of the ring, and
save an smp_mb.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/jr.c | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 7ae5e51..0adaad1 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -94,7 +94,8 @@ static void caam_jr_dequeue(unsigned long devarg)
userdesc = jrp->entinfo[sw_idx].desc_addr_virt;
userstatus = jrp->outring[hw_idx].jrstatus;

- smp_mb();
+ /* set done */
+ wr_reg32(&jrp->rregs->outring_rmvd, 1);

jrp->out_ring_read_index = (jrp->out_ring_read_index + 1) &
(JOBR_DEPTH - 1);
@@ -114,9 +115,6 @@ static void caam_jr_dequeue(unsigned long devarg)
jrp->tail = tail;
}

- /* set done */
- wr_reg32(&jrp->rregs->outring_rmvd, 1);
-
spin_unlock_bh(&jrp->outlock);

/* Finally, execute user's callback */
@@ -265,8 +263,6 @@ int caam_jr_enqueue(struct device *dev, u32 *desc,
(JOBR_DEPTH - 1);
jrp->head = (head + 1) & (JOBR_DEPTH - 1);

- wmb();
-
wr_reg32(&jrp->rregs->inpring_jobadd, 1);

spin_unlock(&jrp->inplock);
--
1.7.11.1

2012-06-23 00:50:56

by Kim Phillips

Subject: [PATCH 13/20] crypto: caam - hwrng support

From: Yuan Kang <[email protected]>

caam_read copies random bytes from two buffers into output.

caam rng can fill empty buffer 0xffff bytes at a time,
but the buffer sizes are rounded down to multiple of cacheline size.

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/Kconfig | 13 ++
drivers/crypto/caam/Makefile | 1 +
drivers/crypto/caam/caamrng.c | 309 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 323 insertions(+)
create mode 100644 drivers/crypto/caam/caamrng.c

diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index e61b7f5..502a7f3 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -82,3 +82,16 @@ config CRYPTO_DEV_FSL_CAAM_AHASH_API

To compile this as a module, choose M here: the module
will be called caamhash.
+
+config CRYPTO_DEV_FSL_CAAM_RNG_API
+ tristate "Register caam device for hwrng API"
+ depends on CRYPTO_DEV_FSL_CAAM
+ default y
+ select CRYPTO_RNG
+ select HW_RANDOM
+ help
+ Selecting this will register the SEC4 hardware rng to
+ the hw_random API for suppying the kernel entropy pool.
+
+ To compile this as a module, choose M here: the module
+ will be called caamrng.
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index 9ef1cb0..b1eb448 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -5,5 +5,6 @@
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o
+obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_RNG_API) += caamrng.o

caam-objs := ctrl.o jr.o error.o key_gen.o
diff --git a/drivers/crypto/caam/caamrng.c b/drivers/crypto/caam/caamrng.c
new file mode 100644
index 0000000..e2bfe16
--- /dev/null
+++ b/drivers/crypto/caam/caamrng.c
@@ -0,0 +1,309 @@
+/*
+ * caam - Freescale FSL CAAM support for hw_random
+ *
+ * Copyright 2011 Freescale Semiconductor, Inc.
+ *
+ * Based on caamalg.c crypto API driver.
+ *
+ * relationship between job descriptors to shared descriptors:
+ *
+ * --------------- --------------
+ * | JobDesc #0 |-------------------->| ShareDesc |
+ * | *(buffer 0) | |------------->| (generate) |
+ * --------------- | | (move) |
+ * | | (store) |
+ * --------------- | --------------
+ * | JobDesc #1 |------|
+ * | *(buffer 1) |
+ * ---------------
+ *
+ * A job desc looks like this:
+ *
+ * ---------------------
+ * | Header |
+ * | ShareDesc Pointer |
+ * | SEQ_OUT_PTR |
+ * | (output buffer) |
+ * ---------------------
+ *
+ * The SharedDesc never changes, and each job descriptor points to one of two
+ * buffers for each device, from which the data will be copied into the
+ * requested destination
+ */
+
+#include <linux/hw_random.h>
+#include <linux/completion.h>
+#include <linux/atomic.h>
+
+#include "compat.h"
+
+#include "regs.h"
+#include "intern.h"
+#include "desc_constr.h"
+#include "jr.h"
+#include "error.h"
+
+/*
+ * Maximum buffer size: maximum number of random, cache-aligned bytes that
+ * will be generated and moved to seq out ptr (extlen not allowed)
+ */
+#define RN_BUF_SIZE (0xffff / L1_CACHE_BYTES * \
+ L1_CACHE_BYTES)
+
+/* length of descriptors */
+#define DESC_JOB_O_LEN (CAAM_CMD_SZ * 2 + CAAM_PTR_SZ * 2)
+#define DESC_RNG_LEN (10 * CAAM_CMD_SZ)
+
+/* Buffer, its dma address and lock */
+struct buf_data {
+ u8 buf[RN_BUF_SIZE];
+ dma_addr_t addr;
+ struct completion filled;
+ u32 hw_desc[DESC_JOB_O_LEN];
+#define BUF_NOT_EMPTY 0
+#define BUF_EMPTY 1
+#define BUF_PENDING 2 /* Empty, but with job pending --don't submit another */
+ atomic_t empty;
+};
+
+/* rng per-device context */
+struct caam_rng_ctx {
+ struct device *jrdev;
+ dma_addr_t sh_desc_dma;
+ u32 sh_desc[DESC_RNG_LEN];
+ unsigned int cur_buf_idx;
+ int current_buf;
+ struct buf_data bufs[2];
+};
+
+static struct caam_rng_ctx rng_ctx;
+
+static inline void rng_unmap_buf(struct device *jrdev, struct buf_data *bd)
+{
+ if (bd->addr)
+ dma_unmap_single(jrdev, bd->addr, RN_BUF_SIZE,
+ DMA_FROM_DEVICE);
+}
+
+static inline void rng_unmap_ctx(struct caam_rng_ctx *ctx)
+{
+ struct device *jrdev = ctx->jrdev;
+
+ if (ctx->sh_desc_dma)
+ dma_unmap_single(jrdev, ctx->sh_desc_dma, DESC_RNG_LEN,
+ DMA_TO_DEVICE);
+ rng_unmap_buf(jrdev, &ctx->bufs[0]);
+ rng_unmap_buf(jrdev, &ctx->bufs[1]);
+}
+
+static void rng_done(struct device *jrdev, u32 *desc, u32 err, void *context)
+{
+ struct buf_data *bd;
+
+ bd = (struct buf_data *)((char *)desc -
+ offsetof(struct buf_data, hw_desc));
+
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(jrdev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ atomic_set(&bd->empty, BUF_NOT_EMPTY);
+ complete(&bd->filled);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "rng refreshed buf@: ",
+ DUMP_PREFIX_ADDRESS, 16, 4, bd->buf, RN_BUF_SIZE, 1);
+#endif
+}
+
+static inline int submit_job(struct caam_rng_ctx *ctx, int to_current)
+{
+ struct buf_data *bd = &ctx->bufs[!(to_current ^ ctx->current_buf)];
+ struct device *jrdev = ctx->jrdev;
+ u32 *desc = bd->hw_desc;
+ int err;
+
+ dev_dbg(jrdev, "submitting job %d\n", !(to_current ^ ctx->current_buf));
+ init_completion(&bd->filled);
+ err = caam_jr_enqueue(jrdev, desc, rng_done, ctx);
+ if (err)
+ complete(&bd->filled); /* don't wait on failed job*/
+ else
+ atomic_inc(&bd->empty); /* note if pending */
+
+ return err;
+}
+
+static int caam_read(struct hwrng *rng, void *data, size_t max, bool wait)
+{
+ struct caam_rng_ctx *ctx = &rng_ctx;
+ struct buf_data *bd = &ctx->bufs[ctx->current_buf];
+ int next_buf_idx, copied_idx;
+ int err;
+
+ if (atomic_read(&bd->empty)) {
+ /* try to submit job if there wasn't one */
+ if (atomic_read(&bd->empty) == BUF_EMPTY) {
+ err = submit_job(ctx, 1);
+ /* if can't submit job, can't even wait */
+ if (err)
+ return 0;
+ }
+ /* no immediate data, so exit if not waiting */
+ if (!wait)
+ return 0;
+
+ /* waiting for pending job */
+ if (atomic_read(&bd->empty))
+ wait_for_completion(&bd->filled);
+ }
+
+ next_buf_idx = ctx->cur_buf_idx + max;
+ dev_dbg(ctx->jrdev, "%s: start reading at buffer %d, idx %d\n",
+ __func__, ctx->current_buf, ctx->cur_buf_idx);
+
+ /* if enough data in current buffer */
+ if (next_buf_idx < RN_BUF_SIZE) {
+ memcpy(data, bd->buf + ctx->cur_buf_idx, max);
+ ctx->cur_buf_idx = next_buf_idx;
+ return max;
+ }
+
+ /* else, copy what's left... */
+ copied_idx = RN_BUF_SIZE - ctx->cur_buf_idx;
+ memcpy(data, bd->buf + ctx->cur_buf_idx, copied_idx);
+ ctx->cur_buf_idx = 0;
+ atomic_set(&bd->empty, BUF_EMPTY);
+
+ /* ...refill... */
+ submit_job(ctx, 1);
+
+ /* and use next buffer */
+ ctx->current_buf = !ctx->current_buf;
+ dev_dbg(ctx->jrdev, "switched to buffer %d\n", ctx->current_buf);
+
+ /* since there already is some data read, don't wait */
+ return copied_idx + caam_read(rng, data + copied_idx,
+ max - copied_idx, false);
+}
+
+static inline void rng_create_sh_desc(struct caam_rng_ctx *ctx)
+{
+ struct device *jrdev = ctx->jrdev;
+ u32 *desc = ctx->sh_desc;
+
+ init_sh_desc(desc, HDR_SHARE_WAIT);
+
+ /* Propagate errors from shared to job descriptor */
+ append_cmd(desc, SET_OK_NO_PROP_ERRORS | CMD_LOAD);
+
+ /* Generate random bytes */
+ append_operation(desc, OP_ALG_ALGSEL_RNG | OP_TYPE_CLASS1_ALG);
+
+ /* Store bytes */
+ append_seq_fifo_store(desc, RN_BUF_SIZE, FIFOST_TYPE_RNGSTORE);
+
+ ctx->sh_desc_dma = dma_map_single(jrdev, desc, desc_bytes(desc),
+ DMA_TO_DEVICE);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "rng shdesc@: ", DUMP_PREFIX_ADDRESS, 16, 4,
+ desc, desc_bytes(desc), 1);
+#endif
+}
+
+static inline void rng_create_job_desc(struct caam_rng_ctx *ctx, int buf_id)
+{
+ struct device *jrdev = ctx->jrdev;
+ struct buf_data *bd = &ctx->bufs[buf_id];
+ u32 *desc = bd->hw_desc;
+ int sh_len = desc_len(ctx->sh_desc);
+
+ init_job_desc_shared(desc, ctx->sh_desc_dma, sh_len, HDR_SHARE_DEFER |
+ HDR_REVERSE);
+
+ bd->addr = dma_map_single(jrdev, bd->buf, RN_BUF_SIZE, DMA_FROM_DEVICE);
+
+ append_seq_out_ptr_intlen(desc, bd->addr, RN_BUF_SIZE, 0);
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "rng job desc@: ", DUMP_PREFIX_ADDRESS, 16, 4,
+ desc, desc_bytes(desc), 1);
+#endif
+}
+
+static void caam_cleanup(struct hwrng *rng)
+{
+ int i;
+ struct buf_data *bd;
+
+ for (i = 0; i < 2; i++) {
+ bd = &rng_ctx.bufs[i];
+ if (atomic_read(&bd->empty) == BUF_PENDING)
+ wait_for_completion(&bd->filled);
+ }
+
+ rng_unmap_ctx(&rng_ctx);
+}
+
+static void caam_init_buf(struct caam_rng_ctx *ctx, int buf_id)
+{
+ struct buf_data *bd = &ctx->bufs[buf_id];
+
+ rng_create_job_desc(ctx, buf_id);
+ atomic_set(&bd->empty, BUF_EMPTY);
+ submit_job(ctx, buf_id == ctx->current_buf);
+ wait_for_completion(&bd->filled);
+}
+
+static void caam_init_rng(struct caam_rng_ctx *ctx, struct device *jrdev)
+{
+ ctx->jrdev = jrdev;
+ rng_create_sh_desc(ctx);
+ ctx->current_buf = 0;
+ ctx->cur_buf_idx = 0;
+ caam_init_buf(ctx, 0);
+ caam_init_buf(ctx, 1);
+}
+
+static struct hwrng caam_rng = {
+ .name = "rng-caam",
+ .cleanup = caam_cleanup,
+ .read = caam_read,
+};
+
+static void __exit caam_rng_exit(void)
+{
+ hwrng_unregister(&caam_rng);
+}
+
+static int __init caam_rng_init(void)
+{
+ struct device_node *dev_node;
+ struct platform_device *pdev;
+ struct device *ctrldev;
+ struct caam_drv_private *priv;
+
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
+ if (!dev_node)
+ return -ENODEV;
+
+ pdev = of_find_device_by_node(dev_node);
+ if (!pdev)
+ return -ENODEV;
+
+ ctrldev = &pdev->dev;
+ priv = dev_get_drvdata(ctrldev);
+ of_node_put(dev_node);
+
+ caam_init_rng(&rng_ctx, priv->jrdev[0]);
+
+ dev_info(priv->jrdev[0], "registering rng-caam\n");
+ return hwrng_register(&caam_rng);
+}
+
+module_init(caam_rng_init);
+module_exit(caam_rng_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("FSL CAAM support for hw_random API");
+MODULE_AUTHOR("Freescale Semiconductor - NMG");
--
1.7.11.1

2012-06-23 00:50:56

by Kim Phillips

Subject: [PATCH 14/20] crypto: caam - assign 40-bit masks on SEC v5.0 and above

SEC v4.x were only 36-bit, SEC v5+ are 40-bit capable.
Also set a DMA mask for any job ring devices created.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/ctrl.c | 9 +++++++--
drivers/crypto/caam/jr.c | 8 ++++++++
2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index 77557eb..9a2db9c 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -82,13 +82,18 @@ static int caam_probe(struct platform_device *pdev)

/*
* Enable DECO watchdogs and, if this is a PHYS_ADDR_T_64BIT kernel,
- * 36-bit pointers in master configuration register
+ * long pointers in master configuration register
*/
setbits32(&topregs->ctrl.mcr, MCFGR_WDENABLE |
(sizeof(dma_addr_t) == sizeof(u64) ? MCFGR_LONG_PTR : 0));

if (sizeof(dma_addr_t) == sizeof(u64))
- dma_set_mask(dev, DMA_BIT_MASK(36));
+ if (of_device_is_compatible(nprop, "fsl,sec-v5.0"))
+ dma_set_mask(dev, DMA_BIT_MASK(40));
+ else
+ dma_set_mask(dev, DMA_BIT_MASK(36));
+ else
+ dma_set_mask(dev, DMA_BIT_MASK(32));

/*
* Detect and enable JobRs
diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 6ce4c41..9f16b2c 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -503,6 +503,14 @@ int caam_jr_probe(struct platform_device *pdev, struct device_node *np,
dev_set_drvdata(jrdev, jrpriv);
ctrlpriv->jrdev[ring] = jrdev;

+ if (sizeof(dma_addr_t) == sizeof(u64))
+ if (of_device_is_compatible(np, "fsl,sec-v5.0-job-ring"))
+ dma_set_mask(jrdev, DMA_BIT_MASK(40));
+ else
+ dma_set_mask(jrdev, DMA_BIT_MASK(36));
+ else
+ dma_set_mask(jrdev, DMA_BIT_MASK(32));
+
/* Identify the interrupt */
jrpriv->irq = of_irq_to_resource(np, 0, NULL);

--
1.7.11.1

2012-06-23 00:50:57

by Kim Phillips

Subject: [PATCH 16/20] crypto: caam - disable IRQ coalescing by default

It has been observed that in zero-loss benchmarks, when a
slow traffic rate is being tested, the IRQ timer coalescing
parameter was set too high, and the ethernet controller
would start dropping packets because the job ring back half
wouldn't be executed in time before the ethernet controller
would fill its buffers, thereby significantly reducing the
zero-loss performance figures.

Empirical testing has shown that the best zero-loss performance
is achieved when IRQ coalescing is set to minimum values and/or
turned off, since apparently the job ring driver already implements
an adequately-performing general-purpose IRQ mitigation strategy
in software.

Whilst we could go with minimal count (2-8) and timing settings
(192-256), we prefer just turning h/w coalescing altogether off
to minimize setkey latency (due to split key generation), and
for consistent cross-SoC performance (the SEC vs. core clock
ratio changes).

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/Kconfig | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index 502a7f3..65c7668 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -32,10 +32,13 @@ config CRYPTO_DEV_FSL_CAAM_RINGSIZE
config CRYPTO_DEV_FSL_CAAM_INTC
bool "Job Ring interrupt coalescing"
depends on CRYPTO_DEV_FSL_CAAM
- default y
+ default n
help
Enable the Job Ring's interrupt coalescing feature.

+ Note: the driver already provides adequate
+ interrupt coalescing in software.
+
config CRYPTO_DEV_FSL_CAAM_INTC_COUNT_THLD
int "Job Ring interrupt coalescing count threshold"
depends on CRYPTO_DEV_FSL_CAAM_INTC
--
1.7.11.1

2012-06-23 00:50:56

by Kim Phillips

Subject: [PATCH 09/20] crypto: caam - link_tbl rename

From: Yuan Kang <[email protected]>

- rename scatterlist and link_tbl functions
- link_tbl changed to sec4_sg
- sg_to_link_tbl_one changed to dma_to_sec4_sg_one,
since no scatterlist is use

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/caamalg.c | 226 ++++++++++-----------
drivers/crypto/caam/desc.h | 6 +-
.../crypto/caam/{sg_link_tbl.h => sg_sw_sec4.h} | 42 ++--
3 files changed, 137 insertions(+), 137 deletions(-)
rename drivers/crypto/caam/{sg_link_tbl.h => sg_sw_sec4.h} (57%)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index ea0295d..5ab480a 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -51,7 +51,7 @@
#include "desc_constr.h"
#include "jr.h"
#include "error.h"
-#include "sg_link_tbl.h"
+#include "sg_sw_sec4.h"
#include "key_gen.h"

/*
@@ -658,8 +658,8 @@ static int ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
* @dst_nents: number of segments in output scatterlist
* @iv_dma: dma address of iv for checking continuity and link table
* @desc: h/w descriptor (variable length; must not exceed MAX_CAAM_DESCSIZE)
- * @link_tbl_bytes: length of dma mapped link_tbl space
- * @link_tbl_dma: bus physical mapped address of h/w link table
+ * @sec4_sg_bytes: length of dma mapped sec4_sg space
+ * @sec4_sg_dma: bus physical mapped address of h/w link table
* @hw_desc: the h/w job descriptor followed by any referenced link tables
*/
struct aead_edesc {
@@ -667,9 +667,9 @@ struct aead_edesc {
int src_nents;
int dst_nents;
dma_addr_t iv_dma;
- int link_tbl_bytes;
- dma_addr_t link_tbl_dma;
- struct link_tbl_entry *link_tbl;
+ int sec4_sg_bytes;
+ dma_addr_t sec4_sg_dma;
+ struct sec4_sg_entry *sec4_sg;
u32 hw_desc[0];
};

@@ -679,24 +679,24 @@ struct aead_edesc {
* @dst_nents: number of segments in output scatterlist
* @iv_dma: dma address of iv for checking continuity and link table
* @desc: h/w descriptor (variable length; must not exceed MAX_CAAM_DESCSIZE)
- * @link_tbl_bytes: length of dma mapped link_tbl space
- * @link_tbl_dma: bus physical mapped address of h/w link table
+ * @sec4_sg_bytes: length of dma mapped sec4_sg space
+ * @sec4_sg_dma: bus physical mapped address of h/w link table
* @hw_desc: the h/w job descriptor followed by any referenced link tables
*/
struct ablkcipher_edesc {
int src_nents;
int dst_nents;
dma_addr_t iv_dma;
- int link_tbl_bytes;
- dma_addr_t link_tbl_dma;
- struct link_tbl_entry *link_tbl;
+ int sec4_sg_bytes;
+ dma_addr_t sec4_sg_dma;
+ struct sec4_sg_entry *sec4_sg;
u32 hw_desc[0];
};

static void caam_unmap(struct device *dev, struct scatterlist *src,
struct scatterlist *dst, int src_nents, int dst_nents,
- dma_addr_t iv_dma, int ivsize, dma_addr_t link_tbl_dma,
- int link_tbl_bytes)
+ dma_addr_t iv_dma, int ivsize, dma_addr_t sec4_sg_dma,
+ int sec4_sg_bytes)
{
if (unlikely(dst != src)) {
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
@@ -707,8 +707,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,

if (iv_dma)
dma_unmap_single(dev, iv_dma, ivsize, DMA_TO_DEVICE);
- if (link_tbl_bytes)
- dma_unmap_single(dev, link_tbl_dma, link_tbl_bytes,
+ if (sec4_sg_bytes)
+ dma_unmap_single(dev, sec4_sg_dma, sec4_sg_bytes,
DMA_TO_DEVICE);
}

@@ -723,8 +723,8 @@ static void aead_unmap(struct device *dev,

caam_unmap(dev, req->src, req->dst,
edesc->src_nents, edesc->dst_nents,
- edesc->iv_dma, ivsize, edesc->link_tbl_dma,
- edesc->link_tbl_bytes);
+ edesc->iv_dma, ivsize, edesc->sec4_sg_dma,
+ edesc->sec4_sg_bytes);
}

static void ablkcipher_unmap(struct device *dev,
@@ -736,8 +736,8 @@ static void ablkcipher_unmap(struct device *dev,

caam_unmap(dev, req->src, req->dst,
edesc->src_nents, edesc->dst_nents,
- edesc->iv_dma, ivsize, edesc->link_tbl_dma,
- edesc->link_tbl_bytes);
+ edesc->iv_dma, ivsize, edesc->sec4_sg_dma,
+ edesc->sec4_sg_bytes);
}

static void aead_encrypt_done(struct device *jrdev, u32 *desc, u32 err,
@@ -828,7 +828,7 @@ static void aead_decrypt_done(struct device *jrdev, u32 *desc, u32 err,
sizeof(struct iphdr) + req->assoclen +
((req->cryptlen > 1500) ? 1500 : req->cryptlen) +
ctx->authsize + 36, 1);
- if (!err && edesc->link_tbl_bytes) {
+ if (!err && edesc->sec4_sg_bytes) {
struct scatterlist *sg = sg_last(req->src, edesc->src_nents);
print_hex_dump(KERN_ERR, "sglastout@"xstr(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, sg_virt(sg),
@@ -927,7 +927,7 @@ static void init_aead_job(u32 *sh_desc, dma_addr_t ptr,
u32 *desc = edesc->hw_desc;
u32 out_options = 0, in_options;
dma_addr_t dst_dma, src_dma;
- int len, link_tbl_index = 0;
+ int len, sec4_sg_index = 0;

#ifdef DEBUG
debug("assoclen %d cryptlen %d authsize %d\n",
@@ -953,9 +953,9 @@ static void init_aead_job(u32 *sh_desc, dma_addr_t ptr,
src_dma = sg_dma_address(req->assoc);
in_options = 0;
} else {
- src_dma = edesc->link_tbl_dma;
- link_tbl_index += (edesc->assoc_nents ? : 1) + 1 +
- (edesc->src_nents ? : 1);
+ src_dma = edesc->sec4_sg_dma;
+ sec4_sg_index += (edesc->assoc_nents ? : 1) + 1 +
+ (edesc->src_nents ? : 1);
in_options = LDST_SGF;
}
if (encrypt)
@@ -969,7 +969,7 @@ static void init_aead_job(u32 *sh_desc, dma_addr_t ptr,
if (all_contig) {
dst_dma = sg_dma_address(req->src);
} else {
- dst_dma = src_dma + sizeof(struct link_tbl_entry) *
+ dst_dma = src_dma + sizeof(struct sec4_sg_entry) *
((edesc->assoc_nents ? : 1) + 1);
out_options = LDST_SGF;
}
@@ -977,9 +977,9 @@ static void init_aead_job(u32 *sh_desc, dma_addr_t ptr,
if (!edesc->dst_nents) {
dst_dma = sg_dma_address(req->dst);
} else {
- dst_dma = edesc->link_tbl_dma +
- link_tbl_index *
- sizeof(struct link_tbl_entry);
+ dst_dma = edesc->sec4_sg_dma +
+ sec4_sg_index *
+ sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
}
@@ -1005,7 +1005,7 @@ static void init_aead_giv_job(u32 *sh_desc, dma_addr_t ptr,
u32 *desc = edesc->hw_desc;
u32 out_options = 0, in_options;
dma_addr_t dst_dma, src_dma;
- int len, link_tbl_index = 0;
+ int len, sec4_sg_index = 0;

#ifdef DEBUG
debug("assoclen %d cryptlen %d authsize %d\n",
@@ -1030,8 +1030,8 @@ static void init_aead_giv_job(u32 *sh_desc, dma_addr_t ptr,
src_dma = sg_dma_address(req->assoc);
in_options = 0;
} else {
- src_dma = edesc->link_tbl_dma;
- link_tbl_index += edesc->assoc_nents + 1 + edesc->src_nents;
+ src_dma = edesc->sec4_sg_dma;
+ sec4_sg_index += edesc->assoc_nents + 1 + edesc->src_nents;
in_options = LDST_SGF;
}
append_seq_in_ptr(desc, src_dma, req->assoclen + ivsize +
@@ -1041,13 +1041,13 @@ static void init_aead_giv_job(u32 *sh_desc, dma_addr_t ptr,
dst_dma = edesc->iv_dma;
} else {
if (likely(req->src == req->dst)) {
- dst_dma = src_dma + sizeof(struct link_tbl_entry) *
+ dst_dma = src_dma + sizeof(struct sec4_sg_entry) *
edesc->assoc_nents;
out_options = LDST_SGF;
} else {
- dst_dma = edesc->link_tbl_dma +
- link_tbl_index *
- sizeof(struct link_tbl_entry);
+ dst_dma = edesc->sec4_sg_dma +
+ sec4_sg_index *
+ sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
}
@@ -1068,7 +1068,7 @@ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr,
u32 *desc = edesc->hw_desc;
u32 out_options = 0, in_options;
dma_addr_t dst_dma, src_dma;
- int len, link_tbl_index = 0;
+ int len, sec4_sg_index = 0;

#ifdef DEBUG
print_hex_dump(KERN_ERR, "presciv@"xstr(__LINE__)": ",
@@ -1086,8 +1086,8 @@ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr,
src_dma = edesc->iv_dma;
in_options = 0;
} else {
- src_dma = edesc->link_tbl_dma;
- link_tbl_index += (iv_contig ? 0 : 1) + edesc->src_nents;
+ src_dma = edesc->sec4_sg_dma;
+ sec4_sg_index += (iv_contig ? 0 : 1) + edesc->src_nents;
in_options = LDST_SGF;
}
append_seq_in_ptr(desc, src_dma, req->nbytes + ivsize, in_options);
@@ -1096,16 +1096,16 @@ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr,
if (!edesc->src_nents && iv_contig) {
dst_dma = sg_dma_address(req->src);
} else {
- dst_dma = edesc->link_tbl_dma +
- sizeof(struct link_tbl_entry);
+ dst_dma = edesc->sec4_sg_dma +
+ sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
} else {
if (!edesc->dst_nents) {
dst_dma = sg_dma_address(req->dst);
} else {
- dst_dma = edesc->link_tbl_dma +
- link_tbl_index * sizeof(struct link_tbl_entry);
+ dst_dma = edesc->sec4_sg_dma +
+ sec4_sg_index * sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
}
@@ -1129,7 +1129,7 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
int sgc;
bool all_contig = true;
int ivsize = crypto_aead_ivsize(aead);
- int link_tbl_index, link_tbl_len = 0, link_tbl_bytes;
+ int sec4_sg_index, sec4_sg_len = 0, sec4_sg_bytes;

assoc_nents = sg_count(req->assoc, req->assoclen);
src_nents = sg_count(req->src, req->cryptlen);
@@ -1157,15 +1157,15 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
all_contig = false;
assoc_nents = assoc_nents ? : 1;
src_nents = src_nents ? : 1;
- link_tbl_len = assoc_nents + 1 + src_nents;
+ sec4_sg_len = assoc_nents + 1 + src_nents;
}
- link_tbl_len += dst_nents;
+ sec4_sg_len += dst_nents;

- link_tbl_bytes = link_tbl_len * sizeof(struct link_tbl_entry);
+ sec4_sg_bytes = sec4_sg_len * sizeof(struct sec4_sg_entry);

/* allocate space for base edesc and hw desc commands, link tables */
edesc = kmalloc(sizeof(struct aead_edesc) + desc_bytes +
- link_tbl_bytes, GFP_DMA | flags);
+ sec4_sg_bytes, GFP_DMA | flags);
if (!edesc) {
dev_err(jrdev, "could not allocate extended descriptor\n");
return ERR_PTR(-ENOMEM);
@@ -1175,32 +1175,32 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents;
edesc->iv_dma = iv_dma;
- edesc->link_tbl_bytes = link_tbl_bytes;
- edesc->link_tbl = (void *)edesc + sizeof(struct aead_edesc) +
- desc_bytes;
- edesc->link_tbl_dma = dma_map_single(jrdev, edesc->link_tbl,
- link_tbl_bytes, DMA_TO_DEVICE);
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) +
+ desc_bytes;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
*all_contig_ptr = all_contig;

- link_tbl_index = 0;
+ sec4_sg_index = 0;
if (!all_contig) {
- sg_to_link_tbl(req->assoc,
- (assoc_nents ? : 1),
- edesc->link_tbl +
- link_tbl_index, 0);
- link_tbl_index += assoc_nents ? : 1;
- sg_to_link_tbl_one(edesc->link_tbl + link_tbl_index,
+ sg_to_sec4_sg(req->assoc,
+ (assoc_nents ? : 1),
+ edesc->sec4_sg +
+ sec4_sg_index, 0);
+ sec4_sg_index += assoc_nents ? : 1;
+ dma_to_sec4_sg_one(edesc->sec4_sg + sec4_sg_index,
iv_dma, ivsize, 0);
- link_tbl_index += 1;
- sg_to_link_tbl_last(req->src,
- (src_nents ? : 1),
- edesc->link_tbl +
- link_tbl_index, 0);
- link_tbl_index += src_nents ? : 1;
+ sec4_sg_index += 1;
+ sg_to_sec4_sg_last(req->src,
+ (src_nents ? : 1),
+ edesc->sec4_sg +
+ sec4_sg_index, 0);
+ sec4_sg_index += src_nents ? : 1;
}
if (dst_nents) {
- sg_to_link_tbl_last(req->dst, dst_nents,
- edesc->link_tbl + link_tbl_index, 0);
+ sg_to_sec4_sg_last(req->dst, dst_nents,
+ edesc->sec4_sg + sec4_sg_index, 0);
}

return edesc;
@@ -1307,7 +1307,7 @@ static struct aead_edesc *aead_giv_edesc_alloc(struct aead_givcrypt_request
int sgc;
u32 contig = GIV_SRC_CONTIG | GIV_DST_CONTIG;
int ivsize = crypto_aead_ivsize(aead);
- int link_tbl_index, link_tbl_len = 0, link_tbl_bytes;
+ int sec4_sg_index, sec4_sg_len = 0, sec4_sg_bytes;

assoc_nents = sg_count(req->assoc, req->assoclen);
src_nents = sg_count(req->src, req->cryptlen);
@@ -1336,22 +1336,22 @@ static struct aead_edesc *aead_giv_edesc_alloc(struct aead_givcrypt_request
contig &= ~GIV_DST_CONTIG;
if (unlikely(req->src != req->dst)) {
dst_nents = dst_nents ? : 1;
- link_tbl_len += 1;
+ sec4_sg_len += 1;
}
if (!(contig & GIV_SRC_CONTIG)) {
assoc_nents = assoc_nents ? : 1;
src_nents = src_nents ? : 1;
- link_tbl_len += assoc_nents + 1 + src_nents;
+ sec4_sg_len += assoc_nents + 1 + src_nents;
if (likely(req->src == req->dst))
contig &= ~GIV_DST_CONTIG;
}
- link_tbl_len += dst_nents;
+ sec4_sg_len += dst_nents;

- link_tbl_bytes = link_tbl_len * sizeof(struct link_tbl_entry);
+ sec4_sg_bytes = sec4_sg_len * sizeof(struct sec4_sg_entry);

/* allocate space for base edesc and hw desc commands, link tables */
edesc = kmalloc(sizeof(struct aead_edesc) + desc_bytes +
- link_tbl_bytes, GFP_DMA | flags);
+ sec4_sg_bytes, GFP_DMA | flags);
if (!edesc) {
dev_err(jrdev, "could not allocate extended descriptor\n");
return ERR_PTR(-ENOMEM);
@@ -1361,33 +1361,33 @@ static struct aead_edesc *aead_giv_edesc_alloc(struct aead_givcrypt_request
edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents;
edesc->iv_dma = iv_dma;
- edesc->link_tbl_bytes = link_tbl_bytes;
- edesc->link_tbl = (void *)edesc + sizeof(struct aead_edesc) +
- desc_bytes;
- edesc->link_tbl_dma = dma_map_single(jrdev, edesc->link_tbl,
- link_tbl_bytes, DMA_TO_DEVICE);
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) +
+ desc_bytes;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
*contig_ptr = contig;

- link_tbl_index = 0;
+ sec4_sg_index = 0;
if (!(contig & GIV_SRC_CONTIG)) {
- sg_to_link_tbl(req->assoc, assoc_nents,
- edesc->link_tbl +
- link_tbl_index, 0);
- link_tbl_index += assoc_nents;
- sg_to_link_tbl_one(edesc->link_tbl + link_tbl_index,
+ sg_to_sec4_sg(req->assoc, assoc_nents,
+ edesc->sec4_sg +
+ sec4_sg_index, 0);
+ sec4_sg_index += assoc_nents;
+ dma_to_sec4_sg_one(edesc->sec4_sg + sec4_sg_index,
iv_dma, ivsize, 0);
- link_tbl_index += 1;
- sg_to_link_tbl_last(req->src, src_nents,
- edesc->link_tbl +
- link_tbl_index, 0);
- link_tbl_index += src_nents;
+ sec4_sg_index += 1;
+ sg_to_sec4_sg_last(req->src, src_nents,
+ edesc->sec4_sg +
+ sec4_sg_index, 0);
+ sec4_sg_index += src_nents;
}
if (unlikely(req->src != req->dst && !(contig & GIV_DST_CONTIG))) {
- sg_to_link_tbl_one(edesc->link_tbl + link_tbl_index,
+ dma_to_sec4_sg_one(edesc->sec4_sg + sec4_sg_index,
iv_dma, ivsize, 0);
- link_tbl_index += 1;
- sg_to_link_tbl_last(req->dst, dst_nents,
- edesc->link_tbl + link_tbl_index, 0);
+ sec4_sg_index += 1;
+ sg_to_sec4_sg_last(req->dst, dst_nents,
+ edesc->sec4_sg + sec4_sg_index, 0);
}

return edesc;
@@ -1453,13 +1453,13 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
CRYPTO_TFM_REQ_MAY_SLEEP)) ?
GFP_KERNEL : GFP_ATOMIC;
- int src_nents, dst_nents = 0, link_tbl_bytes;
+ int src_nents, dst_nents = 0, sec4_sg_bytes;
struct ablkcipher_edesc *edesc;
dma_addr_t iv_dma = 0;
bool iv_contig = false;
int sgc;
int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
- int link_tbl_index;
+ int sec4_sg_index;

src_nents = sg_count(req->src, req->nbytes);

@@ -1485,12 +1485,12 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
iv_contig = true;
else
src_nents = src_nents ? : 1;
- link_tbl_bytes = ((iv_contig ? 0 : 1) + src_nents + dst_nents) *
- sizeof(struct link_tbl_entry);
+ sec4_sg_bytes = ((iv_contig ? 0 : 1) + src_nents + dst_nents) *
+ sizeof(struct sec4_sg_entry);

/* allocate space for base edesc and hw desc commands, link tables */
edesc = kmalloc(sizeof(struct ablkcipher_edesc) + desc_bytes +
- link_tbl_bytes, GFP_DMA | flags);
+ sec4_sg_bytes, GFP_DMA | flags);
if (!edesc) {
dev_err(jrdev, "could not allocate extended descriptor\n");
return ERR_PTR(-ENOMEM);
@@ -1498,31 +1498,31 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request

edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents;
- edesc->link_tbl_bytes = link_tbl_bytes;
- edesc->link_tbl = (void *)edesc + sizeof(struct ablkcipher_edesc) +
- desc_bytes;
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ablkcipher_edesc) +
+ desc_bytes;

- link_tbl_index = 0;
+ sec4_sg_index = 0;
if (!iv_contig) {
- sg_to_link_tbl_one(edesc->link_tbl, iv_dma, ivsize, 0);
- sg_to_link_tbl_last(req->src, src_nents,
- edesc->link_tbl + 1, 0);
- link_tbl_index += 1 + src_nents;
+ dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0);
+ sg_to_sec4_sg_last(req->src, src_nents,
+ edesc->sec4_sg + 1, 0);
+ sec4_sg_index += 1 + src_nents;
}

if (unlikely(dst_nents)) {
- sg_to_link_tbl_last(req->dst, dst_nents,
- edesc->link_tbl + link_tbl_index, 0);
+ sg_to_sec4_sg_last(req->dst, dst_nents,
+ edesc->sec4_sg + sec4_sg_index, 0);
}

- edesc->link_tbl_dma = dma_map_single(jrdev, edesc->link_tbl,
- link_tbl_bytes, DMA_TO_DEVICE);
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
edesc->iv_dma = iv_dma;

#ifdef DEBUG
- print_hex_dump(KERN_ERR, "ablkcipher link_tbl@"xstr(__LINE__)": ",
- DUMP_PREFIX_ADDRESS, 16, 4, edesc->link_tbl,
- link_tbl_bytes, 1);
+ print_hex_dump(KERN_ERR, "ablkcipher sec4_sg@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, edesc->sec4_sg,
+ sec4_sg_bytes, 1);
#endif

*iv_contig_out = iv_contig;
diff --git a/drivers/crypto/caam/desc.h b/drivers/crypto/caam/desc.h
index 48c1927..3e68506 100644
--- a/drivers/crypto/caam/desc.h
+++ b/drivers/crypto/caam/desc.h
@@ -8,10 +8,10 @@
#ifndef DESC_H
#define DESC_H

-struct link_tbl_entry {
+struct sec4_sg_entry {
u64 ptr;
-#define LINK_TBL_LEN_FIN 0x40000000
-#define LINK_TBL_LEN_EXT 0x80000000
+#define SEC4_SG_LEN_FIN 0x40000000
+#define SEC4_SG_LEN_EXT 0x80000000
u32 len;
u8 reserved;
u8 buf_pool_id;
diff --git a/drivers/crypto/caam/sg_link_tbl.h b/drivers/crypto/caam/sg_sw_sec4.h
similarity index 57%
rename from drivers/crypto/caam/sg_link_tbl.h
rename to drivers/crypto/caam/sg_sw_sec4.h
index 6df4349..a6ad7a4 100644
--- a/drivers/crypto/caam/sg_link_tbl.h
+++ b/drivers/crypto/caam/sg_sw_sec4.h
@@ -5,23 +5,23 @@
*
*/

-struct link_tbl_entry;
+struct sec4_sg_entry;

/*
* convert single dma address to h/w link table format
*/
-static inline void sg_to_link_tbl_one(struct link_tbl_entry *link_tbl_ptr,
+static inline void dma_to_sec4_sg_one(struct sec4_sg_entry *sec4_sg_ptr,
dma_addr_t dma, u32 len, u32 offset)
{
- link_tbl_ptr->ptr = dma;
- link_tbl_ptr->len = len;
- link_tbl_ptr->reserved = 0;
- link_tbl_ptr->buf_pool_id = 0;
- link_tbl_ptr->offset = offset;
+ sec4_sg_ptr->ptr = dma;
+ sec4_sg_ptr->len = len;
+ sec4_sg_ptr->reserved = 0;
+ sec4_sg_ptr->buf_pool_id = 0;
+ sec4_sg_ptr->offset = offset;
#ifdef DEBUG
- print_hex_dump(KERN_ERR, "link_tbl_ptr@: ",
- DUMP_PREFIX_ADDRESS, 16, 4, link_tbl_ptr,
- sizeof(struct link_tbl_entry), 1);
+ print_hex_dump(KERN_ERR, "sec4_sg_ptr@: ",
+ DUMP_PREFIX_ADDRESS, 16, 4, sec4_sg_ptr,
+ sizeof(struct sec4_sg_entry), 1);
#endif
}

@@ -29,30 +29,30 @@ static inline void sg_to_link_tbl_one(struct link_tbl_entry *link_tbl_ptr,
* convert scatterlist to h/w link table format
* but does not have final bit; instead, returns last entry
*/
-static inline struct link_tbl_entry *
-sg_to_link_tbl(struct scatterlist *sg, int sg_count,
- struct link_tbl_entry *link_tbl_ptr, u32 offset)
+static inline struct sec4_sg_entry *
+sg_to_sec4_sg(struct scatterlist *sg, int sg_count,
+ struct sec4_sg_entry *sec4_sg_ptr, u32 offset)
{
while (sg_count) {
- sg_to_link_tbl_one(link_tbl_ptr, sg_dma_address(sg),
+ dma_to_sec4_sg_one(sec4_sg_ptr, sg_dma_address(sg),
sg_dma_len(sg), offset);
- link_tbl_ptr++;
+ sec4_sg_ptr++;
sg = sg_next(sg);
sg_count--;
}
- return link_tbl_ptr - 1;
+ return sec4_sg_ptr - 1;
}

/*
* convert scatterlist to h/w link table format
* scatterlist must have been previously dma mapped
*/
-static inline void sg_to_link_tbl_last(struct scatterlist *sg, int sg_count,
- struct link_tbl_entry *link_tbl_ptr,
- u32 offset)
+static inline void sg_to_sec4_sg_last(struct scatterlist *sg, int sg_count,
+ struct sec4_sg_entry *sec4_sg_ptr,
+ u32 offset)
{
- link_tbl_ptr = sg_to_link_tbl(sg, sg_count, link_tbl_ptr, offset);
- link_tbl_ptr->len |= LINK_TBL_LEN_FIN;
+ sec4_sg_ptr = sg_to_sec4_sg(sg, sg_count, sec4_sg_ptr, offset);
+ sec4_sg_ptr->len |= SEC4_SG_LEN_FIN;
}

/* count number of elements in scatterlist */
--
1.7.11.1

2012-06-23 00:50:57

by Kim Phillips

Subject: [PATCH 15/20] crypto: caam - add support for SEC v5.x RNG4

The SEC v4.x' RNGB h/w block self-initialized. RNG4, available
on SEC versions 5 and beyond, is based on a different standard
that requires manual initialization.

Also update any new errors From the SEC v5.2 reference manual:
The SEC v5.2's RNG4 unit reuses some error IDs, thus the addition
of rng_err_id_list over the CHA-independent err_id_list.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/ctrl.c | 129 +++++++++++++++++++++++++++++++++++++++++++-
drivers/crypto/caam/desc.h | 5 ++
drivers/crypto/caam/error.c | 44 +++++++++++----
drivers/crypto/caam/regs.h | 32 ++++++++++-
4 files changed, 196 insertions(+), 14 deletions(-)

diff --git a/drivers/crypto/caam/ctrl.c b/drivers/crypto/caam/ctrl.c
index 9a2db9c..ac6abb3 100644
--- a/drivers/crypto/caam/ctrl.c
+++ b/drivers/crypto/caam/ctrl.c
@@ -2,13 +2,15 @@
* CAAM control-plane driver backend
* Controller-level driver, kernel property detection, initialization
*
- * Copyright 2008-2011 Freescale Semiconductor, Inc.
+ * Copyright 2008-2012 Freescale Semiconductor, Inc.
*/

#include "compat.h"
#include "regs.h"
#include "intern.h"
#include "jr.h"
+#include "desc_constr.h"
+#include "error.h"

static int caam_remove(struct platform_device *pdev)
{
@@ -43,10 +45,120 @@ static int caam_remove(struct platform_device *pdev)
return ret;
}

+/*
+ * Descriptor to instantiate RNG State Handle 0 in normal mode and
+ * load the JDKEK, TDKEK and TDSK registers
+ */
+static void build_instantiation_desc(u32 *desc)
+{
+ u32 *jump_cmd;
+
+ init_job_desc(desc, 0);
+
+ /* INIT RNG in non-test mode */
+ append_operation(desc, OP_TYPE_CLASS1_ALG | OP_ALG_ALGSEL_RNG |
+ OP_ALG_AS_INIT);
+
+ /* wait for done */
+ jump_cmd = append_jump(desc, JUMP_CLASS_CLASS1);
+ set_jump_tgt_here(desc, jump_cmd);
+
+ /*
+ * load 1 to clear written reg:
+ * resets the done interrrupt and returns the RNG to idle.
+ */
+ append_load_imm_u32(desc, 1, LDST_SRCDST_WORD_CLRW);
+
+ /* generate secure keys (non-test) */
+ append_operation(desc, OP_TYPE_CLASS1_ALG | OP_ALG_ALGSEL_RNG |
+ OP_ALG_RNG4_SK);
+}
+
+struct instantiate_result {
+ struct completion completion;
+ int err;
+};
+
+static void rng4_init_done(struct device *dev, u32 *desc, u32 err,
+ void *context)
+{
+ struct instantiate_result *instantiation = context;
+
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(dev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ instantiation->err = err;
+ complete(&instantiation->completion);
+}
+
+static int instantiate_rng(struct device *jrdev)
+{
+ struct instantiate_result instantiation;
+
+ dma_addr_t desc_dma;
+ u32 *desc;
+ int ret;
+
+ desc = kmalloc(CAAM_CMD_SZ * 6, GFP_KERNEL | GFP_DMA);
+ if (!desc) {
+ dev_err(jrdev, "cannot allocate RNG init descriptor memory\n");
+ return -ENOMEM;
+ }
+
+ build_instantiation_desc(desc);
+ desc_dma = dma_map_single(jrdev, desc, desc_bytes(desc), DMA_TO_DEVICE);
+ init_completion(&instantiation.completion);
+ ret = caam_jr_enqueue(jrdev, desc, rng4_init_done, &instantiation);
+ if (!ret) {
+ wait_for_completion_interruptible(&instantiation.completion);
+ ret = instantiation.err;
+ if (ret)
+ dev_err(jrdev, "unable to instantiate RNG\n");
+ }
+
+ dma_unmap_single(jrdev, desc_dma, desc_bytes(desc), DMA_TO_DEVICE);
+
+ kfree(desc);
+
+ return ret;
+}
+
+/*
+ * By default, the TRNG runs for 200 clocks per sample;
+ * 800 clocks per sample generates better entropy.
+ */
+static void kick_trng(struct platform_device *pdev)
+{
+ struct device *ctrldev = &pdev->dev;
+ struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
+ struct caam_full __iomem *topregs;
+ struct rng4tst __iomem *r4tst;
+ u32 val;
+
+ topregs = (struct caam_full __iomem *)ctrlpriv->ctrl;
+ r4tst = &topregs->ctrl.r4tst[0];
+
+ /* put RNG4 into program mode */
+ setbits32(&r4tst->rtmctl, RTMCTL_PRGM);
+ /* 800 clocks per sample */
+ val = rd_reg32(&r4tst->rtsdctl);
+ val = (val & ~RTSDCTL_ENT_DLY_MASK) | (800 << RTSDCTL_ENT_DLY_SHIFT);
+ wr_reg32(&r4tst->rtsdctl, val);
+ /* min. freq. count */
+ wr_reg32(&r4tst->rtfrqmin, 400);
+ /* max. freq. count */
+ wr_reg32(&r4tst->rtfrqmax, 6400);
+ /* put RNG4 into run mode */
+ clrbits32(&r4tst->rtmctl, RTMCTL_PRGM);
+}
+
/* Probe routine for CAAM top (controller) level */
static int caam_probe(struct platform_device *pdev)
{
- int ring, rspec;
+ int ret, ring, rspec;
struct device *dev;
struct device_node *nprop, *np;
struct caam_ctrl __iomem *ctrl;
@@ -146,6 +258,19 @@ static int caam_probe(struct platform_device *pdev)
return -ENOMEM;
}

+ /*
+ * RNG4 based SECs (v5+) need special initialization prior
+ * to executing any descriptors
+ */
+ if (of_device_is_compatible(nprop, "fsl,sec-v5.0")) {
+ kick_trng(pdev);
+ ret = instantiate_rng(ctrlpriv->jrdev[0]);
+ if (ret) {
+ caam_remove(pdev);
+ return ret;
+ }
+ }
+
/* NOTE: RTIC detection ought to go here, around Si time */

/* Initialize queue allocator lock */
diff --git a/drivers/crypto/caam/desc.h b/drivers/crypto/caam/desc.h
index 3e68506..f7f833b 100644
--- a/drivers/crypto/caam/desc.h
+++ b/drivers/crypto/caam/desc.h
@@ -1172,6 +1172,11 @@ struct sec4_sg_entry {
#define OP_ALG_AAI_GSM (0x10 << OP_ALG_AAI_SHIFT)
#define OP_ALG_AAI_EDGE (0x20 << OP_ALG_AAI_SHIFT)

+/* RNG4 set */
+#define OP_ALG_RNG4_SHIFT 4
+#define OP_ALG_RNG4_MASK (0x1f3 << OP_ALG_RNG4_SHIFT)
+
+#define OP_ALG_RNG4_SK (0x100 << OP_ALG_RNG4_SHIFT)

#define OP_ALG_AS_SHIFT 2
#define OP_ALG_AS_MASK (0x3 << OP_ALG_AS_SHIFT)
diff --git a/drivers/crypto/caam/error.c b/drivers/crypto/caam/error.c
index 7e2d54b..9955ed9 100644
--- a/drivers/crypto/caam/error.c
+++ b/drivers/crypto/caam/error.c
@@ -39,18 +39,20 @@ static void report_ccb_status(u32 status, char *outstr)
char *cha_id_list[] = {
"",
"AES",
- "DES, 3DES",
+ "DES",
"ARC4",
- "MD5, SHA-1, SH-224, SHA-256, SHA-384, SHA-512",
+ "MDHA",
"RNG",
"SNOW f8",
- "Kasumi f8, f9",
- "All Public Key Algorithms",
- "CRC",
+ "Kasumi f8/9",
+ "PKHA",
+ "CRCA",
"SNOW f9",
+ "ZUCE",
+ "ZUCA",
};
char *err_id_list[] = {
- "None. No error.",
+ "No error.",
"Mode error.",
"Data size error.",
"Key size error.",
@@ -67,6 +69,20 @@ static void report_ccb_status(u32 status, char *outstr)
"Invalid CHA combination was selected",
"Invalid CHA selected.",
};
+ char *rng_err_id_list[] = {
+ "",
+ "",
+ "",
+ "Instantiate",
+ "Not instantiated",
+ "Test instantiate",
+ "Prediction resistance",
+ "",
+ "Prediction resistance and test request",
+ "Uninstantiate",
+ "",
+ "Secure key generation",
+ };
u8 cha_id = (status & JRSTA_CCBERR_CHAID_MASK) >>
JRSTA_CCBERR_CHAID_SHIFT;
u8 err_id = status & JRSTA_CCBERR_ERRID_MASK;
@@ -81,7 +97,13 @@ static void report_ccb_status(u32 status, char *outstr)
cha_id, sizeof("ff"));
}

- if (err_id < ARRAY_SIZE(err_id_list)) {
+ if ((cha_id << JRSTA_CCBERR_CHAID_SHIFT) == JRSTA_CCBERR_CHAID_RNG &&
+ err_id < ARRAY_SIZE(rng_err_id_list) &&
+ strlen(rng_err_id_list[err_id])) {
+ /* RNG-only error */
+ SPRINTFCAT(outstr, "%s", rng_err_id_list[err_id],
+ strlen(rng_err_id_list[err_id]));
+ } else if (err_id < ARRAY_SIZE(err_id_list)) {
SPRINTFCAT(outstr, "%s", err_id_list[err_id],
strlen(err_id_list[err_id]));
} else {
@@ -101,10 +123,10 @@ static void report_deco_status(u32 status, char *outstr)
u8 value;
char *error_text;
} desc_error_list[] = {
- { 0x00, "None. No error." },
+ { 0x00, "No error." },
{ 0x01, "SGT Length Error. The descriptor is trying to read "
"more data than is contained in the SGT table." },
- { 0x02, "Reserved." },
+ { 0x02, "SGT Null Entry Error." },
{ 0x03, "Job Ring Control Error. There is a bad value in the "
"Job Ring Control register." },
{ 0x04, "Invalid Descriptor Command. The Descriptor Command "
@@ -116,7 +138,7 @@ static void report_deco_status(u32 status, char *outstr)
{ 0x09, "Invalid OPERATION Command" },
{ 0x0A, "Invalid FIFO LOAD Command" },
{ 0x0B, "Invalid FIFO STORE Command" },
- { 0x0C, "Invalid MOVE Command" },
+ { 0x0C, "Invalid MOVE/MOVE_LEN Command" },
{ 0x0D, "Invalid JUMP Command. A nonlocal JUMP Command is "
"invalid because the target is not a Job Header "
"Command, or the jump is from a Trusted Descriptor to "
@@ -166,6 +188,8 @@ static void report_deco_status(u32 status, char *outstr)
"(input frame; block ciphers) and IPsec decap (output "
"frame, when doing the next header byte update) and "
"DCRC (output frame)." },
+ { 0x23, "Read Input Frame error" },
+ { 0x24, "JDKEK, TDKEK or TDSK not loaded error" },
{ 0x80, "DNR (do not run) error" },
{ 0x81, "undefined protocol command" },
{ 0x82, "invalid setting in PDB" },
diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h
index e9f7a70..6d9f1d9 100644
--- a/drivers/crypto/caam/regs.h
+++ b/drivers/crypto/caam/regs.h
@@ -167,7 +167,7 @@ struct partid {
u32 pidr; /* partition ID, DECO */
};

-/* RNG test mode (replicated twice in some configurations) */
+/* RNGB test mode (replicated twice in some configurations) */
/* Padded out to 0x100 */
struct rngtst {
u32 mode; /* RTSTMODEx - Test mode */
@@ -200,6 +200,31 @@ struct rngtst {
u32 rsvd14[15];
};

+/* RNG4 TRNG test registers */
+struct rng4tst {
+#define RTMCTL_PRGM 0x00010000 /* 1 -> program mode, 0 -> run mode */
+ u32 rtmctl; /* misc. control register */
+ u32 rtscmisc; /* statistical check misc. register */
+ u32 rtpkrrng; /* poker range register */
+ union {
+ u32 rtpkrmax; /* PRGM=1: poker max. limit register */
+ u32 rtpkrsq; /* PRGM=0: poker square calc. result register */
+ };
+#define RTSDCTL_ENT_DLY_SHIFT 16
+#define RTSDCTL_ENT_DLY_MASK (0xffff << RTSDCTL_ENT_DLY_SHIFT)
+ u32 rtsdctl; /* seed control register */
+ union {
+ u32 rtsblim; /* PRGM=1: sparse bit limit register */
+ u32 rttotsam; /* PRGM=0: total samples register */
+ };
+ u32 rtfrqmin; /* frequency count min. limit register */
+ union {
+ u32 rtfrqmax; /* PRGM=1: freq. count max. limit register */
+ u32 rtfrqcnt; /* PRGM=0: freq. count register */
+ };
+ u32 rsvd1[56];
+};
+
/*
* caam_ctrl - basic core configuration
* starts base + 0x0000 padded out to 0x1000
@@ -249,7 +274,10 @@ struct caam_ctrl {

/* RNG Test/Verification/Debug Access 600-7ff */
/* (Useful in Test/Debug modes only...) */
- struct rngtst rtst[2];
+ union {
+ struct rngtst rtst[2];
+ struct rng4tst r4tst[2];
+ };

u32 rsvd9[448];

--
1.7.11.1

2012-06-23 00:50:55

by Kim Phillips

Subject: [PATCH 07/20] crypto: caam - remove jr register/deregister

From: Yuan Kang <[email protected]>

remove caam_jr_register and caam_jr_deregister
to allow sharing of job rings.

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/caamalg.c | 30 ++----------------------------
drivers/crypto/caam/intern.h | 2 --
2 files changed, 2 insertions(+), 30 deletions(-)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index d0f8df1..a4e266f 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -2228,7 +2228,7 @@ static int caam_cra_init(struct crypto_tfm *tfm)
* distribute tfms across job rings to ensure in-order
* crypto request processing per tfm
*/
- ctx->jrdev = priv->algapi_jr[(tgt_jr / 2) % priv->num_jrs_for_algapi];
+ ctx->jrdev = priv->jrdev[(tgt_jr / 2) % priv->total_jobrs];

/* copy descriptor header template value */
ctx->class1_alg_type = OP_TYPE_CLASS1_ALG | caam_alg->class1_alg_type;
@@ -2265,7 +2265,6 @@ static void __exit caam_algapi_exit(void)
struct device *ctrldev;
struct caam_drv_private *priv;
struct caam_crypto_alg *t_alg, *n;
- int i, err;

dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
if (!dev_node) {
@@ -2290,13 +2289,6 @@ static void __exit caam_algapi_exit(void)
list_del(&t_alg->entry);
kfree(t_alg);
}
-
- for (i = 0; i < priv->total_jobrs; i++) {
- err = caam_jr_deregister(priv->algapi_jr[i]);
- if (err < 0)
- break;
- }
- kfree(priv->algapi_jr);
}

static struct caam_crypto_alg *caam_alg_alloc(struct device *ctrldev,
@@ -2349,7 +2341,7 @@ static int __init caam_algapi_init(void)
{
struct device_node *dev_node;
struct platform_device *pdev;
- struct device *ctrldev, **jrdev;
+ struct device *ctrldev;
struct caam_drv_private *priv;
int i = 0, err = 0;

@@ -2370,24 +2362,6 @@ static int __init caam_algapi_init(void)

INIT_LIST_HEAD(&priv->alg_list);

- jrdev = kmalloc(sizeof(*jrdev) * priv->total_jobrs, GFP_KERNEL);
- if (!jrdev)
- return -ENOMEM;
-
- for (i = 0; i < priv->total_jobrs; i++) {
- err = caam_jr_register(ctrldev, &jrdev[i]);
- if (err < 0)
- break;
- }
- if (err < 0 && i == 0) {
- dev_err(ctrldev, "algapi error in job ring registration: %d\n",
- err);
- kfree(jrdev);
- return err;
- }
-
- priv->num_jrs_for_algapi = i;
- priv->algapi_jr = jrdev;
atomic_set(&priv->tfm_count, -1);

/* register crypto algorithms the device supports */
diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h
index a34be01..462be99 100644
--- a/drivers/crypto/caam/intern.h
+++ b/drivers/crypto/caam/intern.h
@@ -86,8 +86,6 @@ struct caam_drv_private {

/* which jr allocated to scatterlist crypto */
atomic_t tfm_count ____cacheline_aligned;
- int num_jrs_for_algapi;
- struct device **algapi_jr;
/* list of registered crypto algorithms (mk generic context handle?) */
struct list_head alg_list;

--
1.7.11.1

2012-06-23 00:50:58

by Kim Phillips

Subject: [PATCH 10/20] crypto: caam - ahash hmac support

From: Yuan Kang <[email protected]>

caam supports ahash hmac with sha algorithms and md5.

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/Kconfig | 12 +
drivers/crypto/caam/Makefile | 1 +
drivers/crypto/caam/caamhash.c | 1814 ++++++++++++++++++++++++++++++++++++++
drivers/crypto/caam/compat.h | 2 +
drivers/crypto/caam/intern.h | 2 +
drivers/crypto/caam/sg_sw_sec4.h | 38 +
6 files changed, 1869 insertions(+)
create mode 100644 drivers/crypto/caam/caamhash.c

diff --git a/drivers/crypto/caam/Kconfig b/drivers/crypto/caam/Kconfig
index 2d876bb..e61b7f5 100644
--- a/drivers/crypto/caam/Kconfig
+++ b/drivers/crypto/caam/Kconfig
@@ -70,3 +70,15 @@ config CRYPTO_DEV_FSL_CAAM_CRYPTO_API

To compile this as a module, choose M here: the module
will be called caamalg.
+
+config CRYPTO_DEV_FSL_CAAM_AHASH_API
+ tristate "Register hash algorithm implementations with Crypto API"
+ depends on CRYPTO_DEV_FSL_CAAM
+ default y
+ select CRYPTO_AHASH
+ help
+ Selecting this will offload ahash for users of the
+ scatterlist crypto API to the SEC4 via job ring.
+
+ To compile this as a module, choose M here: the module
+ will be called caamhash.
diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index 4447e57..9ef1cb0 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -4,5 +4,6 @@

obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o
+obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_AHASH_API) += caamhash.o

caam-objs := ctrl.o jr.o error.o key_gen.o
diff --git a/drivers/crypto/caam/caamhash.c b/drivers/crypto/caam/caamhash.c
new file mode 100644
index 0000000..c0b6d86
--- /dev/null
+++ b/drivers/crypto/caam/caamhash.c
@@ -0,0 +1,1814 @@
+/*
+ * caam - Freescale FSL CAAM support for ahash functions of crypto API
+ *
+ * Copyright 2011 Freescale Semiconductor, Inc.
+ *
+ * Based on caamalg.c crypto API driver.
+ *
+ * relationship of digest job descriptor or first job descriptor after init to
+ * shared descriptors:
+ *
+ * --------------- ---------------
+ * | JobDesc #1 |-------------------->| ShareDesc |
+ * | *(packet 1) | | (hashKey) |
+ * --------------- | (operation) |
+ * ---------------
+ *
+ * relationship of subsequent job descriptors to shared descriptors:
+ *
+ * --------------- ---------------
+ * | JobDesc #2 |-------------------->| ShareDesc |
+ * | *(packet 2) | |------------->| (hashKey) |
+ * --------------- | |-------->| (operation) |
+ * . | | | (load ctx2) |
+ * . | | ---------------
+ * --------------- | |
+ * | JobDesc #3 |------| |
+ * | *(packet 3) | |
+ * --------------- |
+ * . |
+ * . |
+ * --------------- |
+ * | JobDesc #4 |------------
+ * | *(packet 4) |
+ * ---------------
+ *
+ * The SharedDesc never changes for a connection unless rekeyed, but
+ * each packet will likely be in a different place. So all we need
+ * to know to process the packet is where the input is, where the
+ * output goes, and what context we want to process with. Context is
+ * in the SharedDesc, packet references in the JobDesc.
+ *
+ * So, a job desc looks like:
+ *
+ * ---------------------
+ * | Header |
+ * | ShareDesc Pointer |
+ * | SEQ_OUT_PTR |
+ * | (output buffer) |
+ * | (output length) |
+ * | SEQ_IN_PTR |
+ * | (input buffer) |
+ * | (input length) |
+ * ---------------------
+ */
+
+#include "compat.h"
+
+#include "regs.h"
+#include "intern.h"
+#include "desc_constr.h"
+#include "jr.h"
+#include "error.h"
+#include "sg_sw_sec4.h"
+#include "key_gen.h"
+
+#define CAAM_CRA_PRIORITY 3000
+
+/* max hash key is max split key size */
+#define CAAM_MAX_HASH_KEY_SIZE (SHA512_DIGEST_SIZE * 2)
+
+#define CAAM_MAX_HASH_BLOCK_SIZE SHA512_BLOCK_SIZE
+#define CAAM_MAX_HASH_DIGEST_SIZE SHA512_DIGEST_SIZE
+
+/* length of descriptors text */
+#define DESC_JOB_IO_LEN (CAAM_CMD_SZ * 5 + CAAM_PTR_SZ * 3)
+
+#define DESC_AHASH_BASE (4 * CAAM_CMD_SZ)
+#define DESC_AHASH_UPDATE_LEN (6 * CAAM_CMD_SZ)
+#define DESC_AHASH_UPDATE_FIRST_LEN (DESC_AHASH_BASE + 4 * CAAM_CMD_SZ)
+#define DESC_AHASH_FINAL_LEN (DESC_AHASH_BASE + 5 * CAAM_CMD_SZ)
+#define DESC_AHASH_FINUP_LEN (DESC_AHASH_BASE + 5 * CAAM_CMD_SZ)
+#define DESC_AHASH_DIGEST_LEN (DESC_AHASH_BASE + 4 * CAAM_CMD_SZ)
+
+#define DESC_HASH_MAX_USED_BYTES (DESC_AHASH_FINAL_LEN + \
+ CAAM_MAX_HASH_KEY_SIZE)
+#define DESC_HASH_MAX_USED_LEN (DESC_HASH_MAX_USED_BYTES / CAAM_CMD_SZ)
+
+/* caam context sizes for hashes: running digest + 8 */
+#define HASH_MSG_LEN 8
+#define MAX_CTX_LEN (HASH_MSG_LEN + SHA512_DIGEST_SIZE)
+
+#ifdef DEBUG
+/* for print_hex_dumps with line references */
+#define xstr(s) str(s)
+#define str(s) #s
+#define debug(format, arg...) printk(format, arg)
+#else
+#define debug(format, arg...)
+#endif
+
+/* ahash per-session context */
+struct caam_hash_ctx {
+ struct device *jrdev;
+ u32 sh_desc_update[DESC_HASH_MAX_USED_LEN];
+ u32 sh_desc_update_first[DESC_HASH_MAX_USED_LEN];
+ u32 sh_desc_fin[DESC_HASH_MAX_USED_LEN];
+ u32 sh_desc_digest[DESC_HASH_MAX_USED_LEN];
+ u32 sh_desc_finup[DESC_HASH_MAX_USED_LEN];
+ dma_addr_t sh_desc_update_dma;
+ dma_addr_t sh_desc_update_first_dma;
+ dma_addr_t sh_desc_fin_dma;
+ dma_addr_t sh_desc_digest_dma;
+ dma_addr_t sh_desc_finup_dma;
+ u32 alg_type;
+ u32 alg_op;
+ u8 key[CAAM_MAX_HASH_KEY_SIZE];
+ dma_addr_t key_dma;
+ int ctx_len;
+ unsigned int split_key_len;
+ unsigned int split_key_pad_len;
+};
+
+/* ahash state */
+struct caam_hash_state {
+ dma_addr_t buf_dma;
+ dma_addr_t ctx_dma;
+ u8 buf_0[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned;
+ int buflen_0;
+ u8 buf_1[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned;
+ int buflen_1;
+ u8 caam_ctx[MAX_CTX_LEN];
+ int (*update)(struct ahash_request *req);
+ int (*final)(struct ahash_request *req);
+ int (*finup)(struct ahash_request *req);
+ int current_buf;
+};
+
+/* Common job descriptor seq in/out ptr routines */
+
+/* Map state->caam_ctx, and append seq_out_ptr command that points to it */
+static inline void map_seq_out_ptr_ctx(u32 *desc, struct device *jrdev,
+ struct caam_hash_state *state,
+ int ctx_len)
+{
+ state->ctx_dma = dma_map_single(jrdev, state->caam_ctx,
+ ctx_len, DMA_FROM_DEVICE);
+ append_seq_out_ptr(desc, state->ctx_dma, ctx_len, 0);
+}
+
+/* Map req->result, and append seq_out_ptr command that points to it */
+static inline dma_addr_t map_seq_out_ptr_result(u32 *desc, struct device *jrdev,
+ u8 *result, int digestsize)
+{
+ dma_addr_t dst_dma;
+
+ dst_dma = dma_map_single(jrdev, result, digestsize, DMA_FROM_DEVICE);
+ append_seq_out_ptr(desc, dst_dma, digestsize, 0);
+
+ return dst_dma;
+}
+
+/* Map current buffer in state and put it in link table */
+static inline dma_addr_t buf_map_to_sec4_sg(struct device *jrdev,
+ struct sec4_sg_entry *sec4_sg,
+ u8 *buf, int buflen)
+{
+ dma_addr_t buf_dma;
+
+ buf_dma = dma_map_single(jrdev, buf, buflen, DMA_TO_DEVICE);
+ dma_to_sec4_sg_one(sec4_sg, buf_dma, buflen, 0);
+
+ return buf_dma;
+}
+
+/* Map req->src and put it in link table */
+static inline void src_map_to_sec4_sg(struct device *jrdev,
+ struct scatterlist *src, int src_nents,
+ struct sec4_sg_entry *sec4_sg)
+{
+ dma_map_sg(jrdev, src, src_nents, DMA_TO_DEVICE);
+ sg_to_sec4_sg_last(src, src_nents, sec4_sg, 0);
+}
+
+/*
+ * Only put buffer in link table if it contains data, which is possible,
+ * since a buffer has previously been used, and needs to be unmapped,
+ */
+static inline dma_addr_t
+try_buf_map_to_sec4_sg(struct device *jrdev, struct sec4_sg_entry *sec4_sg,
+ u8 *buf, dma_addr_t buf_dma, int buflen,
+ int last_buflen)
+{
+ if (buf_dma && !dma_mapping_error(jrdev, buf_dma))
+ dma_unmap_single(jrdev, buf_dma, last_buflen, DMA_TO_DEVICE);
+ if (buflen)
+ buf_dma = buf_map_to_sec4_sg(jrdev, sec4_sg, buf, buflen);
+ else
+ buf_dma = 0;
+
+ return buf_dma;
+}
+
+/* Map state->caam_ctx, and add it to link table */
+static inline void ctx_map_to_sec4_sg(u32 *desc, struct device *jrdev,
+ struct caam_hash_state *state,
+ int ctx_len,
+ struct sec4_sg_entry *sec4_sg,
+ u32 flag)
+{
+ state->ctx_dma = dma_map_single(jrdev, state->caam_ctx, ctx_len, flag);
+ dma_to_sec4_sg_one(sec4_sg, state->ctx_dma, ctx_len, 0);
+}
+
+/* Common shared descriptor commands */
+static inline void append_key_ahash(u32 *desc, struct caam_hash_ctx *ctx)
+{
+ append_key_as_imm(desc, ctx->key, ctx->split_key_pad_len,
+ ctx->split_key_len, CLASS_2 |
+ KEY_DEST_MDHA_SPLIT | KEY_ENC);
+}
+
+/* Append key if it has been set */
+static inline void init_sh_desc_key_ahash(u32 *desc, struct caam_hash_ctx *ctx)
+{
+ u32 *key_jump_cmd;
+
+ init_sh_desc(desc, HDR_SHARE_WAIT);
+
+ if (ctx->split_key_len) {
+ /* Skip if already shared */
+ key_jump_cmd = append_jump(desc, JUMP_JSL | JUMP_TEST_ALL |
+ JUMP_COND_SHRD);
+
+ append_key_ahash(desc, ctx);
+
+ set_jump_tgt_here(desc, key_jump_cmd);
+ }
+
+ /* Propagate errors from shared to job descriptor */
+ append_cmd(desc, SET_OK_NO_PROP_ERRORS | CMD_LOAD);
+}
+
+/*
+ * For ahash read data from seqin following state->caam_ctx,
+ * and write resulting class2 context to seqout, which may be state->caam_ctx
+ * or req->result
+ */
+static inline void ahash_append_load_str(u32 *desc, int digestsize)
+{
+ /* Calculate remaining bytes to read */
+ append_math_add(desc, VARSEQINLEN, SEQINLEN, REG0, CAAM_CMD_SZ);
+
+ /* Read remaining bytes */
+ append_seq_fifo_load(desc, 0, FIFOLD_CLASS_CLASS2 | FIFOLD_TYPE_LAST2 |
+ FIFOLD_TYPE_MSG | KEY_VLF);
+
+ /* Store class2 context bytes */
+ append_seq_store(desc, digestsize, LDST_CLASS_2_CCB |
+ LDST_SRCDST_BYTE_CONTEXT);
+}
+
+/*
+ * For ahash update, final and finup, import context, read and write to seqout
+ */
+static inline void ahash_ctx_data_to_out(u32 *desc, u32 op, u32 state,
+ int digestsize,
+ struct caam_hash_ctx *ctx)
+{
+ init_sh_desc_key_ahash(desc, ctx);
+
+ /* Import context from software */
+ append_cmd(desc, CMD_SEQ_LOAD | LDST_SRCDST_BYTE_CONTEXT |
+ LDST_CLASS_2_CCB | ctx->ctx_len);
+
+ /* Class 2 operation */
+ append_operation(desc, op | state | OP_ALG_ENCRYPT);
+
+ /*
+ * Load from buf and/or src and write to req->result or state->context
+ */
+ ahash_append_load_str(desc, digestsize);
+}
+
+/* For ahash firsts and digest, read and write to seqout */
+static inline void ahash_data_to_out(u32 *desc, u32 op, u32 state,
+ int digestsize, struct caam_hash_ctx *ctx)
+{
+ init_sh_desc_key_ahash(desc, ctx);
+
+ /* Class 2 operation */
+ append_operation(desc, op | state | OP_ALG_ENCRYPT);
+
+ /*
+ * Load from buf and/or src and write to req->result or state->context
+ */
+ ahash_append_load_str(desc, digestsize);
+}
+
+static int ahash_set_sh_desc(struct crypto_ahash *ahash)
+{
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ int digestsize = crypto_ahash_digestsize(ahash);
+ struct device *jrdev = ctx->jrdev;
+ u32 have_key = 0;
+ u32 *desc;
+
+ if (ctx->split_key_len)
+ have_key = OP_ALG_AAI_HMAC_PRECOMP;
+
+ /* ahash_update shared descriptor */
+ desc = ctx->sh_desc_update;
+
+ init_sh_desc(desc, HDR_SHARE_WAIT);
+
+ /* Import context from software */
+ append_cmd(desc, CMD_SEQ_LOAD | LDST_SRCDST_BYTE_CONTEXT |
+ LDST_CLASS_2_CCB | ctx->ctx_len);
+
+ /* Class 2 operation */
+ append_operation(desc, ctx->alg_type | OP_ALG_AS_UPDATE |
+ OP_ALG_ENCRYPT);
+
+ /* Load data and write to result or context */
+ ahash_append_load_str(desc, ctx->ctx_len);
+
+ ctx->sh_desc_update_dma = dma_map_single(jrdev, desc, desc_bytes(desc),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, ctx->sh_desc_update_dma)) {
+ dev_err(jrdev, "unable to map shared descriptor\n");
+ return -ENOMEM;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ahash update shdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ /* ahash_update_first shared descriptor */
+ desc = ctx->sh_desc_update_first;
+
+ ahash_data_to_out(desc, have_key | ctx->alg_type, OP_ALG_AS_INIT,
+ ctx->ctx_len, ctx);
+
+ ctx->sh_desc_update_first_dma = dma_map_single(jrdev, desc,
+ desc_bytes(desc),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, ctx->sh_desc_update_first_dma)) {
+ dev_err(jrdev, "unable to map shared descriptor\n");
+ return -ENOMEM;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ahash update first shdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ /* ahash_final shared descriptor */
+ desc = ctx->sh_desc_fin;
+
+ ahash_ctx_data_to_out(desc, have_key | ctx->alg_type,
+ OP_ALG_AS_FINALIZE, digestsize, ctx);
+
+ ctx->sh_desc_fin_dma = dma_map_single(jrdev, desc, desc_bytes(desc),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, ctx->sh_desc_fin_dma)) {
+ dev_err(jrdev, "unable to map shared descriptor\n");
+ return -ENOMEM;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ahash final shdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc,
+ desc_bytes(desc), 1);
+#endif
+
+ /* ahash_finup shared descriptor */
+ desc = ctx->sh_desc_finup;
+
+ ahash_ctx_data_to_out(desc, have_key | ctx->alg_type,
+ OP_ALG_AS_FINALIZE, digestsize, ctx);
+
+ ctx->sh_desc_finup_dma = dma_map_single(jrdev, desc, desc_bytes(desc),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, ctx->sh_desc_finup_dma)) {
+ dev_err(jrdev, "unable to map shared descriptor\n");
+ return -ENOMEM;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ahash finup shdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc,
+ desc_bytes(desc), 1);
+#endif
+
+ /* ahash_digest shared descriptor */
+ desc = ctx->sh_desc_digest;
+
+ ahash_data_to_out(desc, have_key | ctx->alg_type, OP_ALG_AS_INITFINAL,
+ digestsize, ctx);
+
+ ctx->sh_desc_digest_dma = dma_map_single(jrdev, desc,
+ desc_bytes(desc),
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, ctx->sh_desc_digest_dma)) {
+ dev_err(jrdev, "unable to map shared descriptor\n");
+ return -ENOMEM;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ahash digest shdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc,
+ desc_bytes(desc), 1);
+#endif
+
+ return 0;
+}
+
+static u32 gen_split_hash_key(struct caam_hash_ctx *ctx, const u8 *key_in,
+ u32 keylen)
+{
+ return gen_split_key(ctx->jrdev, ctx->key, ctx->split_key_len,
+ ctx->split_key_pad_len, key_in, keylen,
+ ctx->alg_op);
+}
+
+/* Digest hash size if it is too large */
+static u32 hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
+ u32 *keylen, u8 *key_out, u32 digestsize)
+{
+ struct device *jrdev = ctx->jrdev;
+ u32 *desc;
+ struct split_key_result result;
+ dma_addr_t src_dma, dst_dma;
+ int ret = 0;
+
+ desc = kmalloc(CAAM_CMD_SZ * 6 + CAAM_PTR_SZ * 2, GFP_KERNEL | GFP_DMA);
+
+ init_job_desc(desc, 0);
+
+ src_dma = dma_map_single(jrdev, (void *)key_in, *keylen,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, src_dma)) {
+ dev_err(jrdev, "unable to map key input memory\n");
+ kfree(desc);
+ return -ENOMEM;
+ }
+ dst_dma = dma_map_single(jrdev, (void *)key_out, digestsize,
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(jrdev, dst_dma)) {
+ dev_err(jrdev, "unable to map key output memory\n");
+ dma_unmap_single(jrdev, src_dma, *keylen, DMA_TO_DEVICE);
+ kfree(desc);
+ return -ENOMEM;
+ }
+
+ /* Job descriptor to perform unkeyed hash on key_in */
+ append_operation(desc, ctx->alg_type | OP_ALG_ENCRYPT |
+ OP_ALG_AS_INITFINAL);
+ append_seq_in_ptr(desc, src_dma, *keylen, 0);
+ append_seq_fifo_load(desc, *keylen, FIFOLD_CLASS_CLASS2 |
+ FIFOLD_TYPE_LAST2 | FIFOLD_TYPE_MSG);
+ append_seq_out_ptr(desc, dst_dma, digestsize, 0);
+ append_seq_store(desc, digestsize, LDST_CLASS_2_CCB |
+ LDST_SRCDST_BYTE_CONTEXT);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "key_in@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key_in, *keylen, 1);
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ result.err = 0;
+ init_completion(&result.completion);
+
+ ret = caam_jr_enqueue(jrdev, desc, split_key_done, &result);
+ if (!ret) {
+ /* in progress */
+ wait_for_completion_interruptible(&result.completion);
+ ret = result.err;
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "digested key@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key_in,
+ digestsize, 1);
+#endif
+ }
+ *keylen = digestsize;
+
+ dma_unmap_single(jrdev, src_dma, *keylen, DMA_TO_DEVICE);
+ dma_unmap_single(jrdev, dst_dma, digestsize, DMA_FROM_DEVICE);
+
+ kfree(desc);
+
+ return ret;
+}
+
+static int ahash_setkey(struct crypto_ahash *ahash,
+ const u8 *key, unsigned int keylen)
+{
+ /* Sizes for MDHA pads (*not* keys): MD5, SHA1, 224, 256, 384, 512 */
+ static const u8 mdpadlen[] = { 16, 20, 32, 32, 64, 64 };
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct device *jrdev = ctx->jrdev;
+ int blocksize = crypto_tfm_alg_blocksize(&ahash->base);
+ int digestsize = crypto_ahash_digestsize(ahash);
+ int ret = 0;
+ u8 *hashed_key = NULL;
+
+#ifdef DEBUG
+ printk(KERN_ERR "keylen %d\n", keylen);
+#endif
+
+ if (keylen > blocksize) {
+ hashed_key = kmalloc(sizeof(u8) * digestsize, GFP_KERNEL |
+ GFP_DMA);
+ if (!hashed_key)
+ return -ENOMEM;
+ ret = hash_digest_key(ctx, key, &keylen, hashed_key,
+ digestsize);
+ if (ret)
+ goto badkey;
+ key = hashed_key;
+ }
+
+ /* Pick class 2 key length from algorithm submask */
+ ctx->split_key_len = mdpadlen[(ctx->alg_op & OP_ALG_ALGSEL_SUBMASK) >>
+ OP_ALG_ALGSEL_SHIFT] * 2;
+ ctx->split_key_pad_len = ALIGN(ctx->split_key_len, 16);
+
+#ifdef DEBUG
+ printk(KERN_ERR "split_key_len %d split_key_pad_len %d\n",
+ ctx->split_key_len, ctx->split_key_pad_len);
+ print_hex_dump(KERN_ERR, "key in @"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
+#endif
+
+ ret = gen_split_hash_key(ctx, key, keylen);
+ if (ret)
+ goto badkey;
+
+ ctx->key_dma = dma_map_single(jrdev, ctx->key, ctx->split_key_pad_len,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, ctx->key_dma)) {
+ dev_err(jrdev, "unable to map key i/o memory\n");
+ return -ENOMEM;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx.key@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, ctx->key,
+ ctx->split_key_pad_len, 1);
+#endif
+
+ ret = ahash_set_sh_desc(ahash);
+ if (ret) {
+ dma_unmap_single(jrdev, ctx->key_dma, ctx->split_key_pad_len,
+ DMA_TO_DEVICE);
+ }
+
+ kfree(hashed_key);
+ return ret;
+badkey:
+ kfree(hashed_key);
+ crypto_ahash_set_flags(ahash, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+}
+
+/*
+ * ahash_edesc - s/w-extended ahash descriptor
+ * @dst_dma: physical mapped address of req->result
+ * @sec4_sg_dma: physical mapped address of h/w link table
+ * @src_nents: number of segments in input scatterlist
+ * @sec4_sg_bytes: length of dma mapped sec4_sg space
+ * @sec4_sg: pointer to h/w link table
+ * @hw_desc: the h/w job descriptor followed by any referenced link tables
+ */
+struct ahash_edesc {
+ dma_addr_t dst_dma;
+ dma_addr_t sec4_sg_dma;
+ int src_nents;
+ int sec4_sg_bytes;
+ struct sec4_sg_entry *sec4_sg;
+ u32 hw_desc[0];
+};
+
+static inline void ahash_unmap(struct device *dev,
+ struct ahash_edesc *edesc,
+ struct ahash_request *req, int dst_len)
+{
+ if (edesc->src_nents)
+ dma_unmap_sg(dev, req->src, edesc->src_nents, DMA_TO_DEVICE);
+ if (edesc->dst_dma)
+ dma_unmap_single(dev, edesc->dst_dma, dst_len, DMA_FROM_DEVICE);
+
+ if (edesc->sec4_sg_bytes)
+ dma_unmap_single(dev, edesc->sec4_sg_dma,
+ edesc->sec4_sg_bytes, DMA_TO_DEVICE);
+}
+
+static inline void ahash_unmap_ctx(struct device *dev,
+ struct ahash_edesc *edesc,
+ struct ahash_request *req, int dst_len, u32 flag)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ if (state->ctx_dma)
+ dma_unmap_single(dev, state->ctx_dma, ctx->ctx_len, flag);
+ ahash_unmap(dev, edesc, req, dst_len);
+}
+
+static void ahash_done(struct device *jrdev, u32 *desc, u32 err,
+ void *context)
+{
+ struct ahash_request *req = context;
+ struct ahash_edesc *edesc;
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ int digestsize = crypto_ahash_digestsize(ahash);
+#ifdef DEBUG
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
+#endif
+
+ edesc = (struct ahash_edesc *)((char *)desc -
+ offsetof(struct ahash_edesc, hw_desc));
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(jrdev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ ahash_unmap(jrdev, edesc, req, digestsize);
+ kfree(edesc);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
+ ctx->ctx_len, 1);
+ if (req->result)
+ print_hex_dump(KERN_ERR, "result@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->result,
+ digestsize, 1);
+#endif
+
+ req->base.complete(&req->base, err);
+}
+
+static void ahash_done_bi(struct device *jrdev, u32 *desc, u32 err,
+ void *context)
+{
+ struct ahash_request *req = context;
+ struct ahash_edesc *edesc;
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+#ifdef DEBUG
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ int digestsize = crypto_ahash_digestsize(ahash);
+
+ dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
+#endif
+
+ edesc = (struct ahash_edesc *)((char *)desc -
+ offsetof(struct ahash_edesc, hw_desc));
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(jrdev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len, DMA_BIDIRECTIONAL);
+ kfree(edesc);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
+ ctx->ctx_len, 1);
+ if (req->result)
+ print_hex_dump(KERN_ERR, "result@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->result,
+ digestsize, 1);
+#endif
+
+ req->base.complete(&req->base, err);
+}
+
+static void ahash_done_ctx_src(struct device *jrdev, u32 *desc, u32 err,
+ void *context)
+{
+ struct ahash_request *req = context;
+ struct ahash_edesc *edesc;
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ int digestsize = crypto_ahash_digestsize(ahash);
+#ifdef DEBUG
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
+#endif
+
+ edesc = (struct ahash_edesc *)((char *)desc -
+ offsetof(struct ahash_edesc, hw_desc));
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(jrdev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
+ kfree(edesc);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
+ ctx->ctx_len, 1);
+ if (req->result)
+ print_hex_dump(KERN_ERR, "result@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->result,
+ digestsize, 1);
+#endif
+
+ req->base.complete(&req->base, err);
+}
+
+static void ahash_done_ctx_dst(struct device *jrdev, u32 *desc, u32 err,
+ void *context)
+{
+ struct ahash_request *req = context;
+ struct ahash_edesc *edesc;
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+#ifdef DEBUG
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ int digestsize = crypto_ahash_digestsize(ahash);
+
+ dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
+#endif
+
+ edesc = (struct ahash_edesc *)((char *)desc -
+ offsetof(struct ahash_edesc, hw_desc));
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(jrdev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len, DMA_TO_DEVICE);
+ kfree(edesc);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
+ ctx->ctx_len, 1);
+ if (req->result)
+ print_hex_dump(KERN_ERR, "result@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, req->result,
+ digestsize, 1);
+#endif
+
+ req->base.complete(&req->base, err);
+}
+
+/* submit update job descriptor */
+static int ahash_update_ctx(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *buf = state->current_buf ? state->buf_1 : state->buf_0;
+ int *buflen = state->current_buf ? &state->buflen_1 : &state->buflen_0;
+ u8 *next_buf = state->current_buf ? state->buf_0 : state->buf_1;
+ int *next_buflen = state->current_buf ? &state->buflen_0 :
+ &state->buflen_1, last_buflen;
+ int in_len = *buflen + req->nbytes, to_hash;
+ u32 *sh_desc = ctx->sh_desc_update, *desc;
+ dma_addr_t ptr = ctx->sh_desc_update_dma;
+ int src_nents, sec4_sg_bytes, sec4_sg_src_index;
+ struct ahash_edesc *edesc;
+ int ret = 0;
+ int sh_len;
+
+ last_buflen = *next_buflen;
+ *next_buflen = in_len & (crypto_tfm_alg_blocksize(&ahash->base) - 1);
+ to_hash = in_len - *next_buflen;
+
+ if (to_hash) {
+ src_nents = __sg_count(req->src, req->nbytes - (*next_buflen));
+ sec4_sg_src_index = 1 + (*buflen ? 1 : 0);
+ sec4_sg_bytes = (sec4_sg_src_index + src_nents) *
+ sizeof(struct sec4_sg_entry);
+
+ /*
+ * allocate space for base edesc and hw desc commands,
+ * link tables
+ */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN +
+ sec4_sg_bytes, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev,
+ "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes,
+ DMA_TO_DEVICE);
+
+ ctx_map_to_sec4_sg(desc, jrdev, state, ctx->ctx_len,
+ edesc->sec4_sg, DMA_BIDIRECTIONAL);
+
+ state->buf_dma = try_buf_map_to_sec4_sg(jrdev,
+ edesc->sec4_sg + 1,
+ buf, state->buf_dma,
+ *buflen, last_buflen);
+
+ if (src_nents) {
+ src_map_to_sec4_sg(jrdev, req->src, src_nents,
+ edesc->sec4_sg + sec4_sg_src_index);
+ if (*next_buflen) {
+ sg_copy_part(next_buf, req->src, to_hash -
+ *buflen, req->nbytes);
+ state->current_buf = !state->current_buf;
+ }
+ } else {
+ (edesc->sec4_sg + sec4_sg_src_index - 1)->len |=
+ SEC4_SG_LEN_FIN;
+ }
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER |
+ HDR_REVERSE);
+
+ append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len +
+ to_hash, LDST_SGF);
+
+ append_seq_out_ptr(desc, state->ctx_dma, ctx->ctx_len, 0);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc,
+ desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done_bi, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len,
+ DMA_BIDIRECTIONAL);
+ kfree(edesc);
+ }
+ } else if (*next_buflen) {
+ sg_copy(buf + *buflen, req->src, req->nbytes);
+ *buflen = *next_buflen;
+ *next_buflen = last_buflen;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "buf@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, buf, *buflen, 1);
+ print_hex_dump(KERN_ERR, "next buf@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, next_buf,
+ *next_buflen, 1);
+#endif
+
+ return ret;
+}
+
+static int ahash_final_ctx(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *buf = state->current_buf ? state->buf_1 : state->buf_0;
+ int buflen = state->current_buf ? state->buflen_1 : state->buflen_0;
+ int last_buflen = state->current_buf ? state->buflen_0 :
+ state->buflen_1;
+ u32 *sh_desc = ctx->sh_desc_fin, *desc;
+ dma_addr_t ptr = ctx->sh_desc_fin_dma;
+ int sec4_sg_bytes;
+ int digestsize = crypto_ahash_digestsize(ahash);
+ struct ahash_edesc *edesc;
+ int ret = 0;
+ int sh_len;
+
+ sec4_sg_bytes = (1 + (buflen ? 1 : 0)) * sizeof(struct sec4_sg_entry);
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN +
+ sec4_sg_bytes, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev, "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);
+
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
+ edesc->src_nents = 0;
+
+ ctx_map_to_sec4_sg(desc, jrdev, state, ctx->ctx_len, edesc->sec4_sg,
+ DMA_TO_DEVICE);
+
+ state->buf_dma = try_buf_map_to_sec4_sg(jrdev, edesc->sec4_sg + 1,
+ buf, state->buf_dma, buflen,
+ last_buflen);
+ (edesc->sec4_sg + sec4_sg_bytes - 1)->len |= SEC4_SG_LEN_FIN;
+
+ append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len + buflen,
+ LDST_SGF);
+
+ edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
+ digestsize);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done_ctx_src, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
+ kfree(edesc);
+ }
+
+ return ret;
+}
+
+static int ahash_finup_ctx(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *buf = state->current_buf ? state->buf_1 : state->buf_0;
+ int buflen = state->current_buf ? state->buflen_1 : state->buflen_0;
+ int last_buflen = state->current_buf ? state->buflen_0 :
+ state->buflen_1;
+ u32 *sh_desc = ctx->sh_desc_finup, *desc;
+ dma_addr_t ptr = ctx->sh_desc_finup_dma;
+ int sec4_sg_bytes, sec4_sg_src_index;
+ int src_nents;
+ int digestsize = crypto_ahash_digestsize(ahash);
+ struct ahash_edesc *edesc;
+ int ret = 0;
+ int sh_len;
+
+ src_nents = __sg_count(req->src, req->nbytes);
+ sec4_sg_src_index = 1 + (buflen ? 1 : 0);
+ sec4_sg_bytes = (sec4_sg_src_index + src_nents) *
+ sizeof(struct sec4_sg_entry);
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN +
+ sec4_sg_bytes, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev, "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);
+
+ edesc->src_nents = src_nents;
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
+
+ ctx_map_to_sec4_sg(desc, jrdev, state, ctx->ctx_len, edesc->sec4_sg,
+ DMA_TO_DEVICE);
+
+ state->buf_dma = try_buf_map_to_sec4_sg(jrdev, edesc->sec4_sg + 1,
+ buf, state->buf_dma, buflen,
+ last_buflen);
+
+ src_map_to_sec4_sg(jrdev, req->src, src_nents, edesc->sec4_sg +
+ sec4_sg_src_index);
+
+ append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len +
+ buflen + req->nbytes, LDST_SGF);
+
+ edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
+ digestsize);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done_ctx_src, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
+ kfree(edesc);
+ }
+
+ return ret;
+}
+
+static int ahash_digest(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u32 *sh_desc = ctx->sh_desc_digest, *desc;
+ dma_addr_t ptr = ctx->sh_desc_digest_dma;
+ int digestsize = crypto_ahash_digestsize(ahash);
+ int src_nents, sec4_sg_bytes;
+ dma_addr_t src_dma;
+ struct ahash_edesc *edesc;
+ int ret = 0;
+ u32 options;
+ int sh_len;
+
+ src_nents = sg_count(req->src, req->nbytes);
+ dma_map_sg(jrdev, req->src, src_nents ? : 1, DMA_TO_DEVICE);
+ sec4_sg_bytes = src_nents * sizeof(struct sec4_sg_entry);
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + sec4_sg_bytes +
+ DESC_JOB_IO_LEN, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev, "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
+ edesc->src_nents = src_nents;
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);
+
+ if (src_nents) {
+ sg_to_sec4_sg_last(req->src, src_nents, edesc->sec4_sg, 0);
+ src_dma = edesc->sec4_sg_dma;
+ options = LDST_SGF;
+ } else {
+ src_dma = sg_dma_address(req->src);
+ options = 0;
+ }
+ append_seq_in_ptr(desc, src_dma, req->nbytes, options);
+
+ edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
+ digestsize);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ahash_unmap(jrdev, edesc, req, digestsize);
+ kfree(edesc);
+ }
+
+ return ret;
+}
+
+/* submit ahash final if it the first job descriptor */
+static int ahash_final_no_ctx(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *buf = state->current_buf ? state->buf_1 : state->buf_0;
+ int buflen = state->current_buf ? state->buflen_1 : state->buflen_0;
+ u32 *sh_desc = ctx->sh_desc_digest, *desc;
+ dma_addr_t ptr = ctx->sh_desc_digest_dma;
+ int digestsize = crypto_ahash_digestsize(ahash);
+ struct ahash_edesc *edesc;
+ int ret = 0;
+ int sh_len;
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN,
+ GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev, "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);
+
+ state->buf_dma = dma_map_single(jrdev, buf, buflen, DMA_TO_DEVICE);
+
+ append_seq_in_ptr(desc, state->buf_dma, buflen, 0);
+
+ edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
+ digestsize);
+ edesc->src_nents = 0;
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ahash_unmap(jrdev, edesc, req, digestsize);
+ kfree(edesc);
+ }
+
+ return ret;
+}
+
+/* submit ahash update if it the first job descriptor after update */
+static int ahash_update_no_ctx(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *buf = state->current_buf ? state->buf_1 : state->buf_0;
+ int *buflen = state->current_buf ? &state->buflen_1 : &state->buflen_0;
+ u8 *next_buf = state->current_buf ? state->buf_0 : state->buf_1;
+ int *next_buflen = state->current_buf ? &state->buflen_0 :
+ &state->buflen_1;
+ int in_len = *buflen + req->nbytes, to_hash;
+ int sec4_sg_bytes, src_nents;
+ struct ahash_edesc *edesc;
+ u32 *desc, *sh_desc = ctx->sh_desc_update_first;
+ dma_addr_t ptr = ctx->sh_desc_update_first_dma;
+ int ret = 0;
+ int sh_len;
+
+ *next_buflen = in_len & (crypto_tfm_alg_blocksize(&ahash->base) - 1);
+ to_hash = in_len - *next_buflen;
+
+ if (to_hash) {
+ src_nents = __sg_count(req->src, req->nbytes - (*next_buflen));
+ sec4_sg_bytes = (1 + src_nents) *
+ sizeof(struct sec4_sg_entry);
+
+ /*
+ * allocate space for base edesc and hw desc commands,
+ * link tables
+ */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN +
+ sec4_sg_bytes, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev,
+ "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes,
+ DMA_TO_DEVICE);
+
+ state->buf_dma = buf_map_to_sec4_sg(jrdev, edesc->sec4_sg,
+ buf, *buflen);
+ src_map_to_sec4_sg(jrdev, req->src, src_nents,
+ edesc->sec4_sg + 1);
+ if (*next_buflen) {
+ sg_copy_part(next_buf, req->src, to_hash - *buflen,
+ req->nbytes);
+ state->current_buf = !state->current_buf;
+ }
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER |
+ HDR_REVERSE);
+
+ append_seq_in_ptr(desc, edesc->sec4_sg_dma, to_hash, LDST_SGF);
+
+ map_seq_out_ptr_ctx(desc, jrdev, state, ctx->ctx_len);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc,
+ desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done_ctx_dst, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ state->update = ahash_update_ctx;
+ state->finup = ahash_finup_ctx;
+ state->final = ahash_final_ctx;
+ } else {
+ ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len,
+ DMA_TO_DEVICE);
+ kfree(edesc);
+ }
+ } else if (*next_buflen) {
+ sg_copy(buf + *buflen, req->src, req->nbytes);
+ *buflen = *next_buflen;
+ *next_buflen = 0;
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "buf@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, buf, *buflen, 1);
+ print_hex_dump(KERN_ERR, "next buf@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, next_buf,
+ *next_buflen, 1);
+#endif
+
+ return ret;
+}
+
+/* submit ahash finup if it the first job descriptor after update */
+static int ahash_finup_no_ctx(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *buf = state->current_buf ? state->buf_1 : state->buf_0;
+ int buflen = state->current_buf ? state->buflen_1 : state->buflen_0;
+ int last_buflen = state->current_buf ? state->buflen_0 :
+ state->buflen_1;
+ u32 *sh_desc = ctx->sh_desc_digest, *desc;
+ dma_addr_t ptr = ctx->sh_desc_digest_dma;
+ int sec4_sg_bytes, sec4_sg_src_index, src_nents;
+ int digestsize = crypto_ahash_digestsize(ahash);
+ struct ahash_edesc *edesc;
+ int sh_len;
+ int ret = 0;
+
+ src_nents = __sg_count(req->src, req->nbytes);
+ sec4_sg_src_index = 2;
+ sec4_sg_bytes = (sec4_sg_src_index + src_nents) *
+ sizeof(struct sec4_sg_entry);
+
+ /* allocate space for base edesc and hw desc commands, link tables */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN +
+ sec4_sg_bytes, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev, "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);
+
+ edesc->src_nents = src_nents;
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes, DMA_TO_DEVICE);
+
+ state->buf_dma = try_buf_map_to_sec4_sg(jrdev, edesc->sec4_sg, buf,
+ state->buf_dma, buflen,
+ last_buflen);
+
+ src_map_to_sec4_sg(jrdev, req->src, src_nents, edesc->sec4_sg + 1);
+
+ append_seq_in_ptr(desc, edesc->sec4_sg_dma, buflen +
+ req->nbytes, LDST_SGF);
+
+ edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
+ digestsize);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done, req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ } else {
+ ahash_unmap(jrdev, edesc, req, digestsize);
+ kfree(edesc);
+ }
+
+ return ret;
+}
+
+/* submit first update job descriptor after init */
+static int ahash_update_first(struct ahash_request *req)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+ struct device *jrdev = ctx->jrdev;
+ gfp_t flags = (req->base.flags & (CRYPTO_TFM_REQ_MAY_BACKLOG |
+ CRYPTO_TFM_REQ_MAY_SLEEP)) ? GFP_KERNEL : GFP_ATOMIC;
+ u8 *next_buf = state->buf_0 + state->current_buf *
+ CAAM_MAX_HASH_BLOCK_SIZE;
+ int *next_buflen = &state->buflen_0 + state->current_buf;
+ int to_hash;
+ u32 *sh_desc = ctx->sh_desc_update_first, *desc;
+ dma_addr_t ptr = ctx->sh_desc_update_first_dma;
+ int sec4_sg_bytes, src_nents;
+ dma_addr_t src_dma;
+ u32 options;
+ struct ahash_edesc *edesc;
+ int ret = 0;
+ int sh_len;
+
+ *next_buflen = req->nbytes & (crypto_tfm_alg_blocksize(&ahash->base) -
+ 1);
+ to_hash = req->nbytes - *next_buflen;
+
+ if (to_hash) {
+ src_nents = sg_count(req->src, req->nbytes - (*next_buflen));
+ dma_map_sg(jrdev, req->src, src_nents ? : 1, DMA_TO_DEVICE);
+ sec4_sg_bytes = src_nents * sizeof(struct sec4_sg_entry);
+
+ /*
+ * allocate space for base edesc and hw desc commands,
+ * link tables
+ */
+ edesc = kmalloc(sizeof(struct ahash_edesc) + DESC_JOB_IO_LEN +
+ sec4_sg_bytes, GFP_DMA | flags);
+ if (!edesc) {
+ dev_err(jrdev,
+ "could not allocate extended descriptor\n");
+ return -ENOMEM;
+ }
+
+ edesc->src_nents = src_nents;
+ edesc->sec4_sg_bytes = sec4_sg_bytes;
+ edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
+ DESC_JOB_IO_LEN;
+ edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
+ sec4_sg_bytes,
+ DMA_TO_DEVICE);
+
+ if (src_nents) {
+ sg_to_sec4_sg_last(req->src, src_nents,
+ edesc->sec4_sg, 0);
+ src_dma = edesc->sec4_sg_dma;
+ options = LDST_SGF;
+ } else {
+ src_dma = sg_dma_address(req->src);
+ options = 0;
+ }
+
+ if (*next_buflen)
+ sg_copy_part(next_buf, req->src, to_hash, req->nbytes);
+
+ sh_len = desc_len(sh_desc);
+ desc = edesc->hw_desc;
+ init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER |
+ HDR_REVERSE);
+
+ append_seq_in_ptr(desc, src_dma, to_hash, options);
+
+ map_seq_out_ptr_ctx(desc, jrdev, state, ctx->ctx_len);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc,
+ desc_bytes(desc), 1);
+#endif
+
+ ret = caam_jr_enqueue(jrdev, desc, ahash_done_ctx_dst,
+ req);
+ if (!ret) {
+ ret = -EINPROGRESS;
+ state->update = ahash_update_ctx;
+ state->finup = ahash_finup_ctx;
+ state->final = ahash_final_ctx;
+ } else {
+ ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len,
+ DMA_TO_DEVICE);
+ kfree(edesc);
+ }
+ } else if (*next_buflen) {
+ state->update = ahash_update_no_ctx;
+ state->finup = ahash_finup_no_ctx;
+ state->final = ahash_final_no_ctx;
+ sg_copy(next_buf, req->src, req->nbytes);
+ }
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "next buf@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, next_buf,
+ *next_buflen, 1);
+#endif
+
+ return ret;
+}
+
+static int ahash_finup_first(struct ahash_request *req)
+{
+ return ahash_digest(req);
+}
+
+static int ahash_init(struct ahash_request *req)
+{
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ state->update = ahash_update_first;
+ state->finup = ahash_finup_first;
+ state->final = ahash_final_no_ctx;
+
+ state->current_buf = 0;
+
+ return 0;
+}
+
+static int ahash_update(struct ahash_request *req)
+{
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ return state->update(req);
+}
+
+static int ahash_finup(struct ahash_request *req)
+{
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ return state->finup(req);
+}
+
+static int ahash_final(struct ahash_request *req)
+{
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ return state->final(req);
+}
+
+static int ahash_export(struct ahash_request *req, void *out)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ memcpy(out, ctx, sizeof(struct caam_hash_ctx));
+ memcpy(out + sizeof(struct caam_hash_ctx), state,
+ sizeof(struct caam_hash_state));
+ return 0;
+}
+
+static int ahash_import(struct ahash_request *req, const void *in)
+{
+ struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
+ struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
+ struct caam_hash_state *state = ahash_request_ctx(req);
+
+ memcpy(ctx, in, sizeof(struct caam_hash_ctx));
+ memcpy(state, in + sizeof(struct caam_hash_ctx),
+ sizeof(struct caam_hash_state));
+ return 0;
+}
+
+struct caam_hash_template {
+ char name[CRYPTO_MAX_ALG_NAME];
+ char driver_name[CRYPTO_MAX_ALG_NAME];
+ unsigned int blocksize;
+ struct ahash_alg template_ahash;
+ u32 alg_type;
+ u32 alg_op;
+};
+
+/* ahash descriptors */
+static struct caam_hash_template driver_hash[] = {
+ {
+ .name = "hmac(sha1)",
+ .driver_name = "hmac-sha1-caam",
+ .blocksize = SHA1_BLOCK_SIZE,
+ .template_ahash = {
+ .init = ahash_init,
+ .update = ahash_update,
+ .final = ahash_final,
+ .finup = ahash_finup,
+ .digest = ahash_digest,
+ .export = ahash_export,
+ .import = ahash_import,
+ .setkey = ahash_setkey,
+ .halg = {
+ .digestsize = SHA1_DIGEST_SIZE,
+ },
+ },
+ .alg_type = OP_ALG_ALGSEL_SHA1,
+ .alg_op = OP_ALG_ALGSEL_SHA1 | OP_ALG_AAI_HMAC,
+ }, {
+ .name = "hmac(sha224)",
+ .driver_name = "hmac-sha224-caam",
+ .blocksize = SHA224_BLOCK_SIZE,
+ .template_ahash = {
+ .init = ahash_init,
+ .update = ahash_update,
+ .final = ahash_final,
+ .finup = ahash_finup,
+ .digest = ahash_digest,
+ .export = ahash_export,
+ .import = ahash_import,
+ .setkey = ahash_setkey,
+ .halg = {
+ .digestsize = SHA224_DIGEST_SIZE,
+ },
+ },
+ .alg_type = OP_ALG_ALGSEL_SHA224,
+ .alg_op = OP_ALG_ALGSEL_SHA224 | OP_ALG_AAI_HMAC,
+ }, {
+ .name = "hmac(sha256)",
+ .driver_name = "hmac-sha256-caam",
+ .blocksize = SHA256_BLOCK_SIZE,
+ .template_ahash = {
+ .init = ahash_init,
+ .update = ahash_update,
+ .final = ahash_final,
+ .finup = ahash_finup,
+ .digest = ahash_digest,
+ .export = ahash_export,
+ .import = ahash_import,
+ .setkey = ahash_setkey,
+ .halg = {
+ .digestsize = SHA256_DIGEST_SIZE,
+ },
+ },
+ .alg_type = OP_ALG_ALGSEL_SHA256,
+ .alg_op = OP_ALG_ALGSEL_SHA256 | OP_ALG_AAI_HMAC,
+ }, {
+ .name = "hmac(sha384)",
+ .driver_name = "hmac-sha384-caam",
+ .blocksize = SHA384_BLOCK_SIZE,
+ .template_ahash = {
+ .init = ahash_init,
+ .update = ahash_update,
+ .final = ahash_final,
+ .finup = ahash_finup,
+ .digest = ahash_digest,
+ .export = ahash_export,
+ .import = ahash_import,
+ .setkey = ahash_setkey,
+ .halg = {
+ .digestsize = SHA384_DIGEST_SIZE,
+ },
+ },
+ .alg_type = OP_ALG_ALGSEL_SHA384,
+ .alg_op = OP_ALG_ALGSEL_SHA384 | OP_ALG_AAI_HMAC,
+ }, {
+ .name = "hmac(sha512)",
+ .driver_name = "hmac-sha512-caam",
+ .blocksize = SHA512_BLOCK_SIZE,
+ .template_ahash = {
+ .init = ahash_init,
+ .update = ahash_update,
+ .final = ahash_final,
+ .finup = ahash_finup,
+ .digest = ahash_digest,
+ .export = ahash_export,
+ .import = ahash_import,
+ .setkey = ahash_setkey,
+ .halg = {
+ .digestsize = SHA512_DIGEST_SIZE,
+ },
+ },
+ .alg_type = OP_ALG_ALGSEL_SHA512,
+ .alg_op = OP_ALG_ALGSEL_SHA512 | OP_ALG_AAI_HMAC,
+ }, {
+ .name = "hmac(md5)",
+ .driver_name = "hmac-md5-caam",
+ .blocksize = MD5_BLOCK_WORDS * 4,
+ .template_ahash = {
+ .init = ahash_init,
+ .update = ahash_update,
+ .final = ahash_final,
+ .finup = ahash_finup,
+ .digest = ahash_digest,
+ .export = ahash_export,
+ .import = ahash_import,
+ .setkey = ahash_setkey,
+ .halg = {
+ .digestsize = MD5_DIGEST_SIZE,
+ },
+ },
+ .alg_type = OP_ALG_ALGSEL_MD5,
+ .alg_op = OP_ALG_ALGSEL_MD5 | OP_ALG_AAI_HMAC,
+ },
+};
+
+struct caam_hash_alg {
+ struct list_head entry;
+ struct device *ctrldev;
+ int alg_type;
+ int alg_op;
+ struct ahash_alg ahash_alg;
+};
+
+static int caam_hash_cra_init(struct crypto_tfm *tfm)
+{
+ struct crypto_ahash *ahash = __crypto_ahash_cast(tfm);
+ struct crypto_alg *base = tfm->__crt_alg;
+ struct hash_alg_common *halg =
+ container_of(base, struct hash_alg_common, base);
+ struct ahash_alg *alg =
+ container_of(halg, struct ahash_alg, halg);
+ struct caam_hash_alg *caam_hash =
+ container_of(alg, struct caam_hash_alg, ahash_alg);
+ struct caam_hash_ctx *ctx = crypto_tfm_ctx(tfm);
+ struct caam_drv_private *priv = dev_get_drvdata(caam_hash->ctrldev);
+ /* Sizes for MDHA running digests: MD5, SHA1, 224, 256, 384, 512 */
+ static const u8 runninglen[] = { HASH_MSG_LEN + MD5_DIGEST_SIZE,
+ HASH_MSG_LEN + SHA1_DIGEST_SIZE,
+ HASH_MSG_LEN + 32,
+ HASH_MSG_LEN + SHA256_DIGEST_SIZE,
+ HASH_MSG_LEN + 64,
+ HASH_MSG_LEN + SHA512_DIGEST_SIZE };
+ int tgt_jr = atomic_inc_return(&priv->tfm_count);
+ int ret = 0;
+
+ /*
+ * distribute tfms across job rings to ensure in-order
+ * crypto request processing per tfm
+ */
+ ctx->jrdev = priv->jrdev[tgt_jr % priv->total_jobrs];
+
+ /* copy descriptor header template value */
+ ctx->alg_type = OP_TYPE_CLASS2_ALG | caam_hash->alg_type;
+ ctx->alg_op = OP_TYPE_CLASS2_ALG | caam_hash->alg_op;
+
+ ctx->ctx_len = runninglen[(ctx->alg_op & OP_ALG_ALGSEL_SUBMASK) >>
+ OP_ALG_ALGSEL_SHIFT];
+
+ crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
+ sizeof(struct caam_hash_state));
+
+ ret = ahash_set_sh_desc(ahash);
+
+ return ret;
+}
+
+static void caam_hash_cra_exit(struct crypto_tfm *tfm)
+{
+ struct caam_hash_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ if (ctx->sh_desc_update_dma &&
+ !dma_mapping_error(ctx->jrdev, ctx->sh_desc_update_dma))
+ dma_unmap_single(ctx->jrdev, ctx->sh_desc_update_dma,
+ desc_bytes(ctx->sh_desc_update),
+ DMA_TO_DEVICE);
+ if (ctx->sh_desc_update_first_dma &&
+ !dma_mapping_error(ctx->jrdev, ctx->sh_desc_update_first_dma))
+ dma_unmap_single(ctx->jrdev, ctx->sh_desc_update_first_dma,
+ desc_bytes(ctx->sh_desc_update_first),
+ DMA_TO_DEVICE);
+ if (ctx->sh_desc_fin_dma &&
+ !dma_mapping_error(ctx->jrdev, ctx->sh_desc_fin_dma))
+ dma_unmap_single(ctx->jrdev, ctx->sh_desc_fin_dma,
+ desc_bytes(ctx->sh_desc_fin), DMA_TO_DEVICE);
+ if (ctx->sh_desc_digest_dma &&
+ !dma_mapping_error(ctx->jrdev, ctx->sh_desc_digest_dma))
+ dma_unmap_single(ctx->jrdev, ctx->sh_desc_digest_dma,
+ desc_bytes(ctx->sh_desc_digest),
+ DMA_TO_DEVICE);
+ if (ctx->sh_desc_finup_dma &&
+ !dma_mapping_error(ctx->jrdev, ctx->sh_desc_finup_dma))
+ dma_unmap_single(ctx->jrdev, ctx->sh_desc_finup_dma,
+ desc_bytes(ctx->sh_desc_finup), DMA_TO_DEVICE);
+}
+
+static void __exit caam_algapi_hash_exit(void)
+{
+ struct device_node *dev_node;
+ struct platform_device *pdev;
+ struct device *ctrldev;
+ struct caam_drv_private *priv;
+ struct caam_hash_alg *t_alg, *n;
+
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
+ if (!dev_node)
+ return;
+
+ pdev = of_find_device_by_node(dev_node);
+ if (!pdev)
+ return;
+
+ ctrldev = &pdev->dev;
+ of_node_put(dev_node);
+ priv = dev_get_drvdata(ctrldev);
+
+ if (!priv->hash_list.next)
+ return;
+
+ list_for_each_entry_safe(t_alg, n, &priv->hash_list, entry) {
+ crypto_unregister_ahash(&t_alg->ahash_alg);
+ list_del(&t_alg->entry);
+ kfree(t_alg);
+ }
+}
+
+static struct caam_hash_alg *
+caam_hash_alloc(struct device *ctrldev, struct caam_hash_template *template)
+{
+ struct caam_hash_alg *t_alg;
+ struct ahash_alg *halg;
+ struct crypto_alg *alg;
+
+ t_alg = kzalloc(sizeof(struct caam_hash_alg), GFP_KERNEL);
+ if (!t_alg) {
+ dev_err(ctrldev, "failed to allocate t_alg\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ t_alg->ahash_alg = template->template_ahash;
+ halg = &t_alg->ahash_alg;
+ alg = &halg->halg.base;
+
+ snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s", template->name);
+ snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->driver_name);
+ alg->cra_module = THIS_MODULE;
+ alg->cra_init = caam_hash_cra_init;
+ alg->cra_exit = caam_hash_cra_exit;
+ alg->cra_ctxsize = sizeof(struct caam_hash_ctx);
+ alg->cra_priority = CAAM_CRA_PRIORITY;
+ alg->cra_blocksize = template->blocksize;
+ alg->cra_alignmask = 0;
+ alg->cra_flags = CRYPTO_ALG_ASYNC | CRYPTO_ALG_TYPE_AHASH;
+ alg->cra_type = &crypto_ahash_type;
+
+ t_alg->alg_type = template->alg_type;
+ t_alg->alg_op = template->alg_op;
+ t_alg->ctrldev = ctrldev;
+
+ return t_alg;
+}
+
+static int __init caam_algapi_hash_init(void)
+{
+ struct device_node *dev_node;
+ struct platform_device *pdev;
+ struct device *ctrldev;
+ struct caam_drv_private *priv;
+ int i = 0, err = 0;
+
+ dev_node = of_find_compatible_node(NULL, NULL, "fsl,sec-v4.0");
+ if (!dev_node)
+ return -ENODEV;
+
+ pdev = of_find_device_by_node(dev_node);
+ if (!pdev)
+ return -ENODEV;
+
+ ctrldev = &pdev->dev;
+ priv = dev_get_drvdata(ctrldev);
+ of_node_put(dev_node);
+
+ INIT_LIST_HEAD(&priv->hash_list);
+
+ atomic_set(&priv->tfm_count, -1);
+
+ /* register crypto algorithms the device supports */
+ for (i = 0; i < ARRAY_SIZE(driver_hash); i++) {
+ /* TODO: check if h/w supports alg */
+ struct caam_hash_alg *t_alg;
+
+ t_alg = caam_hash_alloc(ctrldev, &driver_hash[i]);
+ if (IS_ERR(t_alg)) {
+ err = PTR_ERR(t_alg);
+ dev_warn(ctrldev, "%s alg allocation failed\n",
+ driver_hash[i].driver_name);
+ continue;
+ }
+
+ err = crypto_register_ahash(&t_alg->ahash_alg);
+ if (err) {
+ dev_warn(ctrldev, "%s alg registration failed\n",
+ t_alg->ahash_alg.halg.base.cra_driver_name);
+ kfree(t_alg);
+ } else
+ list_add_tail(&t_alg->entry, &priv->hash_list);
+ }
+
+ return err;
+}
+
+module_init(caam_algapi_hash_init);
+module_exit(caam_algapi_hash_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("FSL CAAM support for ahash functions of crypto API");
+MODULE_AUTHOR("Freescale Semiconductor - NMG");
diff --git a/drivers/crypto/caam/compat.h b/drivers/crypto/caam/compat.h
index a63bc65..762aeff 100644
--- a/drivers/crypto/caam/compat.h
+++ b/drivers/crypto/caam/compat.h
@@ -11,6 +11,7 @@
#include <linux/device.h>
#include <linux/interrupt.h>
#include <linux/crypto.h>
+#include <linux/hash.h>
#include <linux/hw_random.h>
#include <linux/of_platform.h>
#include <linux/dma-mapping.h>
@@ -33,5 +34,6 @@
#include <crypto/authenc.h>
#include <crypto/scatterwalk.h>
#include <crypto/internal/skcipher.h>
+#include <crypto/internal/hash.h>

#endif /* !defined(CAAM_COMPAT_H) */
diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h
index 462be99..ed2b739 100644
--- a/drivers/crypto/caam/intern.h
+++ b/drivers/crypto/caam/intern.h
@@ -88,6 +88,8 @@ struct caam_drv_private {
atomic_t tfm_count ____cacheline_aligned;
/* list of registered crypto algorithms (mk generic context handle?) */
struct list_head alg_list;
+ /* list of registered hash algorithms (mk generic context handle?) */
+ struct list_head hash_list;

/*
* debugfs entries for developer view into driver/device
diff --git a/drivers/crypto/caam/sg_sw_sec4.h b/drivers/crypto/caam/sg_sw_sec4.h
index a6ad7a4..2dda9e3 100644
--- a/drivers/crypto/caam/sg_sw_sec4.h
+++ b/drivers/crypto/caam/sg_sw_sec4.h
@@ -82,3 +82,41 @@ static inline int sg_count(struct scatterlist *sg_list, int nbytes)

return sg_nents;
}
+
+/* Copy from len bytes of sg to dest, starting from beginning */
+static inline void sg_copy(u8 *dest, struct scatterlist *sg, unsigned int len)
+{
+ struct scatterlist *current_sg = sg;
+ int cpy_index = 0, next_cpy_index = current_sg->length;
+
+ while (next_cpy_index < len) {
+ memcpy(dest + cpy_index, (u8 *) sg_virt(current_sg),
+ current_sg->length);
+ current_sg = scatterwalk_sg_next(current_sg);
+ cpy_index = next_cpy_index;
+ next_cpy_index += current_sg->length;
+ }
+ if (cpy_index < len)
+ memcpy(dest + cpy_index, (u8 *) sg_virt(current_sg),
+ len - cpy_index);
+}
+
+/* Copy sg data, from to_skip to end, to dest */
+static inline void sg_copy_part(u8 *dest, struct scatterlist *sg,
+ int to_skip, unsigned int end)
+{
+ struct scatterlist *current_sg = sg;
+ int sg_index, cpy_index;
+
+ sg_index = current_sg->length;
+ while (sg_index <= to_skip) {
+ current_sg = scatterwalk_sg_next(current_sg);
+ sg_index += current_sg->length;
+ }
+ cpy_index = sg_index - to_skip;
+ memcpy(dest, (u8 *) sg_virt(current_sg) +
+ current_sg->length - cpy_index, cpy_index);
+ current_sg = scatterwalk_sg_next(current_sg);
+ if (end - sg_index)
+ sg_copy(dest + cpy_index, current_sg, end - sg_index);
+}
--
1.7.11.1

2012-06-23 00:50:58

by Kim Phillips

Subject: [PATCH 17/20] crypto: caam - use non-irq versions of spinlocks for job rings

The enqueue lock isn't used in any interrupt context, and
the dequeue lock isn't used in the h/w interrupt context,
only in bh context.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/jr.c | 30 +++++++++++++-----------------
1 file changed, 13 insertions(+), 17 deletions(-)

diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 9f16b2c..11d93f2 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -2,7 +2,7 @@
* CAAM/SEC 4.x transport/backend driver
* JobR backend functionality
*
- * Copyright 2008-2011 Freescale Semiconductor, Inc.
+ * Copyright 2008-2012 Freescale Semiconductor, Inc.
*/

#include "compat.h"
@@ -58,9 +58,8 @@ static void caam_jr_dequeue(unsigned long devarg)
void (*usercall)(struct device *dev, u32 *desc, u32 status, void *arg);
u32 *userdesc, userstatus;
void *userarg;
- unsigned long flags;

- spin_lock_irqsave(&jrp->outlock, flags);
+ spin_lock_bh(&jrp->outlock);

head = ACCESS_ONCE(jrp->head);
sw_idx = tail = jrp->tail;
@@ -118,18 +117,18 @@ static void caam_jr_dequeue(unsigned long devarg)
/* set done */
wr_reg32(&jrp->rregs->outring_rmvd, 1);

- spin_unlock_irqrestore(&jrp->outlock, flags);
+ spin_unlock_bh(&jrp->outlock);

/* Finally, execute user's callback */
usercall(dev, userdesc, userstatus, userarg);

- spin_lock_irqsave(&jrp->outlock, flags);
+ spin_lock_bh(&jrp->outlock);

head = ACCESS_ONCE(jrp->head);
sw_idx = tail = jrp->tail;
}

- spin_unlock_irqrestore(&jrp->outlock, flags);
+ spin_unlock_bh(&jrp->outlock);

/* reenable / unmask IRQs */
clrbits32(&jrp->rregs->rconfig_lo, JRCFG_IMSK);
@@ -148,23 +147,22 @@ int caam_jr_register(struct device *ctrldev, struct device **rdev)
{
struct caam_drv_private *ctrlpriv = dev_get_drvdata(ctrldev);
struct caam_drv_private_jr *jrpriv = NULL;
- unsigned long flags;
int ring;

/* Lock, if free ring - assign, unlock */
- spin_lock_irqsave(&ctrlpriv->jr_alloc_lock, flags);
+ spin_lock(&ctrlpriv->jr_alloc_lock);
for (ring = 0; ring < ctrlpriv->total_jobrs; ring++) {
jrpriv = dev_get_drvdata(ctrlpriv->jrdev[ring]);
if (jrpriv->assign == JOBR_UNASSIGNED) {
jrpriv->assign = JOBR_ASSIGNED;
*rdev = ctrlpriv->jrdev[ring];
- spin_unlock_irqrestore(&ctrlpriv->jr_alloc_lock, flags);
+ spin_unlock(&ctrlpriv->jr_alloc_lock);
return ring;
}
}

/* If assigned, write dev where caller needs it */
- spin_unlock_irqrestore(&ctrlpriv->jr_alloc_lock, flags);
+ spin_unlock(&ctrlpriv->jr_alloc_lock);
*rdev = NULL;

return -ENODEV;
@@ -182,7 +180,6 @@ int caam_jr_deregister(struct device *rdev)
{
struct caam_drv_private_jr *jrpriv = dev_get_drvdata(rdev);
struct caam_drv_private *ctrlpriv;
- unsigned long flags;

/* Get the owning controller's private space */
ctrlpriv = dev_get_drvdata(jrpriv->parentdev);
@@ -195,9 +192,9 @@ int caam_jr_deregister(struct device *rdev)
return -EBUSY;

/* Release ring */
- spin_lock_irqsave(&ctrlpriv->jr_alloc_lock, flags);
+ spin_lock(&ctrlpriv->jr_alloc_lock);
jrpriv->assign = JOBR_UNASSIGNED;
- spin_unlock_irqrestore(&ctrlpriv->jr_alloc_lock, flags);
+ spin_unlock(&ctrlpriv->jr_alloc_lock);

return 0;
}
@@ -238,7 +235,6 @@ int caam_jr_enqueue(struct device *dev, u32 *desc,
{
struct caam_drv_private_jr *jrp = dev_get_drvdata(dev);
struct caam_jrentry_info *head_entry;
- unsigned long flags;
int head, tail, desc_size;
dma_addr_t desc_dma;

@@ -249,14 +245,14 @@ int caam_jr_enqueue(struct device *dev, u32 *desc,
return -EIO;
}

- spin_lock_irqsave(&jrp->inplock, flags);
+ spin_lock(&jrp->inplock);

head = jrp->head;
tail = ACCESS_ONCE(jrp->tail);

if (!rd_reg32(&jrp->rregs->inpring_avail) ||
CIRC_SPACE(head, tail, JOBR_DEPTH) <= 0) {
- spin_unlock_irqrestore(&jrp->inplock, flags);
+ spin_unlock(&jrp->inplock);
dma_unmap_single(dev, desc_dma, desc_size, DMA_TO_DEVICE);
return -EBUSY;
}
@@ -280,7 +276,7 @@ int caam_jr_enqueue(struct device *dev, u32 *desc,

wr_reg32(&jrp->rregs->inpring_jobadd, 1);

- spin_unlock_irqrestore(&jrp->inplock, flags);
+ spin_unlock(&jrp->inplock);

return 0;
}
--
1.7.11.1

2012-06-23 00:50:58

by Kim Phillips

Subject: [PATCH 12/20] crypto: caam - chaining support

From: Yuan Kang <[email protected]>

support chained scatterlists for aead, ablkcipher and ahash.

Signed-off-by: Yuan Kang <[email protected]>

- fix dma unmap leak
- un-unlikely src == dst, due to experience with AF_ALG

Signed-off-by: Kudupudi Ugendreshwar <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/caamalg.c | 114 ++++++++++++++++++++++++---------------
drivers/crypto/caam/caamhash.c | 53 ++++++++++++------
drivers/crypto/caam/sg_sw_sec4.h | 44 +++++++++++++--
3 files changed, 147 insertions(+), 64 deletions(-)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index 5ab480a..0c1ea84 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -654,8 +654,11 @@ static int ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
/*
* aead_edesc - s/w-extended aead descriptor
* @assoc_nents: number of segments in associated data (SPI+Seq) scatterlist
+ * @assoc_chained: if source is chained
* @src_nents: number of segments in input scatterlist
+ * @src_chained: if source is chained
* @dst_nents: number of segments in output scatterlist
+ * @dst_chained: if destination is chained
* @iv_dma: dma address of iv for checking continuity and link table
* @desc: h/w descriptor (variable length; must not exceed MAX_CAAM_DESCSIZE)
* @sec4_sg_bytes: length of dma mapped sec4_sg space
@@ -664,8 +667,11 @@ static int ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
*/
struct aead_edesc {
int assoc_nents;
+ bool assoc_chained;
int src_nents;
+ bool src_chained;
int dst_nents;
+ bool dst_chained;
dma_addr_t iv_dma;
int sec4_sg_bytes;
dma_addr_t sec4_sg_dma;
@@ -676,7 +682,9 @@ struct aead_edesc {
/*
* ablkcipher_edesc - s/w-extended ablkcipher descriptor
* @src_nents: number of segments in input scatterlist
+ * @src_chained: if source is chained
* @dst_nents: number of segments in output scatterlist
+ * @dst_chained: if destination is chained
* @iv_dma: dma address of iv for checking continuity and link table
* @desc: h/w descriptor (variable length; must not exceed MAX_CAAM_DESCSIZE)
* @sec4_sg_bytes: length of dma mapped sec4_sg space
@@ -685,7 +693,9 @@ struct aead_edesc {
*/
struct ablkcipher_edesc {
int src_nents;
+ bool src_chained;
int dst_nents;
+ bool dst_chained;
dma_addr_t iv_dma;
int sec4_sg_bytes;
dma_addr_t sec4_sg_dma;
@@ -694,15 +704,19 @@ struct ablkcipher_edesc {
};

static void caam_unmap(struct device *dev, struct scatterlist *src,
- struct scatterlist *dst, int src_nents, int dst_nents,
+ struct scatterlist *dst, int src_nents,
+ bool src_chained, int dst_nents, bool dst_chained,
dma_addr_t iv_dma, int ivsize, dma_addr_t sec4_sg_dma,
int sec4_sg_bytes)
{
- if (unlikely(dst != src)) {
- dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
- dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
+ if (dst != src) {
+ dma_unmap_sg_chained(dev, src, src_nents ? : 1, DMA_TO_DEVICE,
+ src_chained);
+ dma_unmap_sg_chained(dev, dst, dst_nents ? : 1, DMA_FROM_DEVICE,
+ dst_chained);
} else {
- dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
+ dma_unmap_sg_chained(dev, src, src_nents ? : 1,
+ DMA_BIDIRECTIONAL, src_chained);
}

if (iv_dma)
@@ -719,12 +733,13 @@ static void aead_unmap(struct device *dev,
struct crypto_aead *aead = crypto_aead_reqtfm(req);
int ivsize = crypto_aead_ivsize(aead);

- dma_unmap_sg(dev, req->assoc, edesc->assoc_nents, DMA_TO_DEVICE);
+ dma_unmap_sg_chained(dev, req->assoc, edesc->assoc_nents,
+ DMA_TO_DEVICE, edesc->assoc_chained);

caam_unmap(dev, req->src, req->dst,
- edesc->src_nents, edesc->dst_nents,
- edesc->iv_dma, ivsize, edesc->sec4_sg_dma,
- edesc->sec4_sg_bytes);
+ edesc->src_nents, edesc->src_chained, edesc->dst_nents,
+ edesc->dst_chained, edesc->iv_dma, ivsize,
+ edesc->sec4_sg_dma, edesc->sec4_sg_bytes);
}

static void ablkcipher_unmap(struct device *dev,
@@ -735,9 +750,9 @@ static void ablkcipher_unmap(struct device *dev,
int ivsize = crypto_ablkcipher_ivsize(ablkcipher);

caam_unmap(dev, req->src, req->dst,
- edesc->src_nents, edesc->dst_nents,
- edesc->iv_dma, ivsize, edesc->sec4_sg_dma,
- edesc->sec4_sg_bytes);
+ edesc->src_nents, edesc->src_chained, edesc->dst_nents,
+ edesc->dst_chained, edesc->iv_dma, ivsize,
+ edesc->sec4_sg_dma, edesc->sec4_sg_bytes);
}

static void aead_encrypt_done(struct device *jrdev, u32 *desc, u32 err,
@@ -1128,25 +1143,26 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
dma_addr_t iv_dma = 0;
int sgc;
bool all_contig = true;
+ bool assoc_chained = false, src_chained = false, dst_chained = false;
int ivsize = crypto_aead_ivsize(aead);
int sec4_sg_index, sec4_sg_len = 0, sec4_sg_bytes;

- assoc_nents = sg_count(req->assoc, req->assoclen);
- src_nents = sg_count(req->src, req->cryptlen);
+ assoc_nents = sg_count(req->assoc, req->assoclen, &assoc_chained);
+ src_nents = sg_count(req->src, req->cryptlen, &src_chained);

if (unlikely(req->dst != req->src))
- dst_nents = sg_count(req->dst, req->cryptlen);
+ dst_nents = sg_count(req->dst, req->cryptlen, &dst_chained);

- sgc = dma_map_sg(jrdev, req->assoc, assoc_nents ? : 1,
- DMA_BIDIRECTIONAL);
+ sgc = dma_map_sg_chained(jrdev, req->assoc, assoc_nents ? : 1,
+ DMA_BIDIRECTIONAL, assoc_chained);
if (likely(req->src == req->dst)) {
- sgc = dma_map_sg(jrdev, req->src, src_nents ? : 1,
- DMA_BIDIRECTIONAL);
+ sgc = dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_BIDIRECTIONAL, src_chained);
} else {
- sgc = dma_map_sg(jrdev, req->src, src_nents ? : 1,
- DMA_TO_DEVICE);
- sgc = dma_map_sg(jrdev, req->dst, dst_nents ? : 1,
- DMA_FROM_DEVICE);
+ sgc = dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_TO_DEVICE, src_chained);
+ sgc = dma_map_sg_chained(jrdev, req->dst, dst_nents ? : 1,
+ DMA_FROM_DEVICE, dst_chained);
}

/* Check if data are contiguous */
@@ -1172,8 +1188,11 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
}

edesc->assoc_nents = assoc_nents;
+ edesc->assoc_chained = assoc_chained;
edesc->src_nents = src_nents;
+ edesc->src_chained = src_chained;
edesc->dst_nents = dst_nents;
+ edesc->dst_chained = dst_chained;
edesc->iv_dma = iv_dma;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) +
@@ -1307,24 +1326,25 @@ static struct aead_edesc *aead_giv_edesc_alloc(struct aead_givcrypt_request
int sgc;
u32 contig = GIV_SRC_CONTIG | GIV_DST_CONTIG;
int ivsize = crypto_aead_ivsize(aead);
+ bool assoc_chained = false, src_chained = false, dst_chained = false;
int sec4_sg_index, sec4_sg_len = 0, sec4_sg_bytes;

- assoc_nents = sg_count(req->assoc, req->assoclen);
- src_nents = sg_count(req->src, req->cryptlen);
+ assoc_nents = sg_count(req->assoc, req->assoclen, &assoc_chained);
+ src_nents = sg_count(req->src, req->cryptlen, &src_chained);

if (unlikely(req->dst != req->src))
- dst_nents = sg_count(req->dst, req->cryptlen);
+ dst_nents = sg_count(req->dst, req->cryptlen, &dst_chained);

- sgc = dma_map_sg(jrdev, req->assoc, assoc_nents ? : 1,
- DMA_BIDIRECTIONAL);
+ sgc = dma_map_sg_chained(jrdev, req->assoc, assoc_nents ? : 1,
+ DMA_BIDIRECTIONAL, assoc_chained);
if (likely(req->src == req->dst)) {
- sgc = dma_map_sg(jrdev, req->src, src_nents ? : 1,
- DMA_BIDIRECTIONAL);
+ sgc = dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_BIDIRECTIONAL, src_chained);
} else {
- sgc = dma_map_sg(jrdev, req->src, src_nents ? : 1,
- DMA_TO_DEVICE);
- sgc = dma_map_sg(jrdev, req->dst, dst_nents ? : 1,
- DMA_FROM_DEVICE);
+ sgc = dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_TO_DEVICE, src_chained);
+ sgc = dma_map_sg_chained(jrdev, req->dst, dst_nents ? : 1,
+ DMA_FROM_DEVICE, dst_chained);
}

/* Check if data are contiguous */
@@ -1358,8 +1378,11 @@ static struct aead_edesc *aead_giv_edesc_alloc(struct aead_givcrypt_request
}

edesc->assoc_nents = assoc_nents;
+ edesc->assoc_chained = assoc_chained;
edesc->src_nents = src_nents;
+ edesc->src_chained = src_chained;
edesc->dst_nents = dst_nents;
+ edesc->dst_chained = dst_chained;
edesc->iv_dma = iv_dma;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) +
@@ -1459,21 +1482,22 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
bool iv_contig = false;
int sgc;
int ivsize = crypto_ablkcipher_ivsize(ablkcipher);
+ bool src_chained = false, dst_chained = false;
int sec4_sg_index;

- src_nents = sg_count(req->src, req->nbytes);
+ src_nents = sg_count(req->src, req->nbytes, &src_chained);

- if (unlikely(req->dst != req->src))
- dst_nents = sg_count(req->dst, req->nbytes);
+ if (req->dst != req->src)
+ dst_nents = sg_count(req->dst, req->nbytes, &dst_chained);

if (likely(req->src == req->dst)) {
- sgc = dma_map_sg(jrdev, req->src, src_nents ? : 1,
- DMA_BIDIRECTIONAL);
+ sgc = dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_BIDIRECTIONAL, src_chained);
} else {
- sgc = dma_map_sg(jrdev, req->src, src_nents ? : 1,
- DMA_TO_DEVICE);
- sgc = dma_map_sg(jrdev, req->dst, dst_nents ? : 1,
- DMA_FROM_DEVICE);
+ sgc = dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_TO_DEVICE, src_chained);
+ sgc = dma_map_sg_chained(jrdev, req->dst, dst_nents ? : 1,
+ DMA_FROM_DEVICE, dst_chained);
}

/*
@@ -1497,7 +1521,9 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
}

edesc->src_nents = src_nents;
+ edesc->src_chained = src_chained;
edesc->dst_nents = dst_nents;
+ edesc->dst_chained = dst_chained;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct ablkcipher_edesc) +
desc_bytes;
@@ -1510,7 +1536,7 @@ static struct ablkcipher_edesc *ablkcipher_edesc_alloc(struct ablkcipher_request
sec4_sg_index += 1 + src_nents;
}

- if (unlikely(dst_nents)) {
+ if (dst_nents) {
sg_to_sec4_sg_last(req->dst, dst_nents,
edesc->sec4_sg + sec4_sg_index, 0);
}
diff --git a/drivers/crypto/caam/caamhash.c b/drivers/crypto/caam/caamhash.c
index 7dcf28f..895aaf2 100644
--- a/drivers/crypto/caam/caamhash.c
+++ b/drivers/crypto/caam/caamhash.c
@@ -175,9 +175,10 @@ static inline dma_addr_t buf_map_to_sec4_sg(struct device *jrdev,
/* Map req->src and put it in link table */
static inline void src_map_to_sec4_sg(struct device *jrdev,
struct scatterlist *src, int src_nents,
- struct sec4_sg_entry *sec4_sg)
+ struct sec4_sg_entry *sec4_sg,
+ bool chained)
{
- dma_map_sg(jrdev, src, src_nents, DMA_TO_DEVICE);
+ dma_map_sg_chained(jrdev, src, src_nents, DMA_TO_DEVICE, chained);
sg_to_sec4_sg_last(src, src_nents, sec4_sg, 0);
}

@@ -563,6 +564,7 @@ badkey:
* ahash_edesc - s/w-extended ahash descriptor
* @dst_dma: physical mapped address of req->result
* @sec4_sg_dma: physical mapped address of h/w link table
+ * @chained: if source is chained
* @src_nents: number of segments in input scatterlist
* @sec4_sg_bytes: length of dma mapped sec4_sg space
* @sec4_sg: pointer to h/w link table
@@ -571,6 +573,7 @@ badkey:
struct ahash_edesc {
dma_addr_t dst_dma;
dma_addr_t sec4_sg_dma;
+ bool chained;
int src_nents;
int sec4_sg_bytes;
struct sec4_sg_entry *sec4_sg;
@@ -582,7 +585,8 @@ static inline void ahash_unmap(struct device *dev,
struct ahash_request *req, int dst_len)
{
if (edesc->src_nents)
- dma_unmap_sg(dev, req->src, edesc->src_nents, DMA_TO_DEVICE);
+ dma_unmap_sg_chained(dev, req->src, edesc->src_nents,
+ DMA_TO_DEVICE, edesc->chained);
if (edesc->dst_dma)
dma_unmap_single(dev, edesc->dst_dma, dst_len, DMA_FROM_DEVICE);

@@ -775,6 +779,7 @@ static int ahash_update_ctx(struct ahash_request *req)
dma_addr_t ptr = ctx->sh_desc_update_dma;
int src_nents, sec4_sg_bytes, sec4_sg_src_index;
struct ahash_edesc *edesc;
+ bool chained = false;
int ret = 0;
int sh_len;

@@ -783,7 +788,8 @@ static int ahash_update_ctx(struct ahash_request *req)
to_hash = in_len - *next_buflen;

if (to_hash) {
- src_nents = __sg_count(req->src, req->nbytes - (*next_buflen));
+ src_nents = __sg_count(req->src, req->nbytes - (*next_buflen),
+ &chained);
sec4_sg_src_index = 1 + (*buflen ? 1 : 0);
sec4_sg_bytes = (sec4_sg_src_index + src_nents) *
sizeof(struct sec4_sg_entry);
@@ -801,6 +807,7 @@ static int ahash_update_ctx(struct ahash_request *req)
}

edesc->src_nents = src_nents;
+ edesc->chained = chained;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
DESC_JOB_IO_LEN;
@@ -818,7 +825,8 @@ static int ahash_update_ctx(struct ahash_request *req)

if (src_nents) {
src_map_to_sec4_sg(jrdev, req->src, src_nents,
- edesc->sec4_sg + sec4_sg_src_index);
+ edesc->sec4_sg + sec4_sg_src_index,
+ chained);
if (*next_buflen) {
sg_copy_part(next_buf, req->src, to_hash -
*buflen, req->nbytes);
@@ -958,10 +966,11 @@ static int ahash_finup_ctx(struct ahash_request *req)
int src_nents;
int digestsize = crypto_ahash_digestsize(ahash);
struct ahash_edesc *edesc;
+ bool chained = false;
int ret = 0;
int sh_len;

- src_nents = __sg_count(req->src, req->nbytes);
+ src_nents = __sg_count(req->src, req->nbytes, &chained);
sec4_sg_src_index = 1 + (buflen ? 1 : 0);
sec4_sg_bytes = (sec4_sg_src_index + src_nents) *
sizeof(struct sec4_sg_entry);
@@ -979,6 +988,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);

edesc->src_nents = src_nents;
+ edesc->chained = chained;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
DESC_JOB_IO_LEN;
@@ -993,7 +1003,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
last_buflen);

src_map_to_sec4_sg(jrdev, req->src, src_nents, edesc->sec4_sg +
- sec4_sg_src_index);
+ sec4_sg_src_index, chained);

append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len +
buflen + req->nbytes, LDST_SGF);
@@ -1030,12 +1040,14 @@ static int ahash_digest(struct ahash_request *req)
int src_nents, sec4_sg_bytes;
dma_addr_t src_dma;
struct ahash_edesc *edesc;
+ bool chained = false;
int ret = 0;
u32 options;
int sh_len;

- src_nents = sg_count(req->src, req->nbytes);
- dma_map_sg(jrdev, req->src, src_nents ? : 1, DMA_TO_DEVICE);
+ src_nents = sg_count(req->src, req->nbytes, &chained);
+ dma_map_sg_chained(jrdev, req->src, src_nents ? : 1, DMA_TO_DEVICE,
+ chained);
sec4_sg_bytes = src_nents * sizeof(struct sec4_sg_entry);

/* allocate space for base edesc and hw desc commands, link tables */
@@ -1050,6 +1062,7 @@ static int ahash_digest(struct ahash_request *req)
edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
sec4_sg_bytes, DMA_TO_DEVICE);
edesc->src_nents = src_nents;
+ edesc->chained = chained;

sh_len = desc_len(sh_desc);
desc = edesc->hw_desc;
@@ -1157,6 +1170,7 @@ static int ahash_update_no_ctx(struct ahash_request *req)
struct ahash_edesc *edesc;
u32 *desc, *sh_desc = ctx->sh_desc_update_first;
dma_addr_t ptr = ctx->sh_desc_update_first_dma;
+ bool chained = false;
int ret = 0;
int sh_len;

@@ -1164,7 +1178,8 @@ static int ahash_update_no_ctx(struct ahash_request *req)
to_hash = in_len - *next_buflen;

if (to_hash) {
- src_nents = __sg_count(req->src, req->nbytes - (*next_buflen));
+ src_nents = __sg_count(req->src, req->nbytes - (*next_buflen),
+ &chained);
sec4_sg_bytes = (1 + src_nents) *
sizeof(struct sec4_sg_entry);

@@ -1181,6 +1196,7 @@ static int ahash_update_no_ctx(struct ahash_request *req)
}

edesc->src_nents = src_nents;
+ edesc->chained = chained;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
DESC_JOB_IO_LEN;
@@ -1191,7 +1207,7 @@ static int ahash_update_no_ctx(struct ahash_request *req)
state->buf_dma = buf_map_to_sec4_sg(jrdev, edesc->sec4_sg,
buf, *buflen);
src_map_to_sec4_sg(jrdev, req->src, src_nents,
- edesc->sec4_sg + 1);
+ edesc->sec4_sg + 1, chained);
if (*next_buflen) {
sg_copy_part(next_buf, req->src, to_hash - *buflen,
req->nbytes);
@@ -1258,10 +1274,11 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
int sec4_sg_bytes, sec4_sg_src_index, src_nents;
int digestsize = crypto_ahash_digestsize(ahash);
struct ahash_edesc *edesc;
+ bool chained = false;
int sh_len;
int ret = 0;

- src_nents = __sg_count(req->src, req->nbytes);
+ src_nents = __sg_count(req->src, req->nbytes, &chained);
sec4_sg_src_index = 2;
sec4_sg_bytes = (sec4_sg_src_index + src_nents) *
sizeof(struct sec4_sg_entry);
@@ -1279,6 +1296,7 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
init_job_desc_shared(desc, ptr, sh_len, HDR_SHARE_DEFER | HDR_REVERSE);

edesc->src_nents = src_nents;
+ edesc->chained = chained;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
DESC_JOB_IO_LEN;
@@ -1289,7 +1307,8 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
state->buf_dma, buflen,
last_buflen);

- src_map_to_sec4_sg(jrdev, req->src, src_nents, edesc->sec4_sg + 1);
+ src_map_to_sec4_sg(jrdev, req->src, src_nents, edesc->sec4_sg + 1,
+ chained);

append_seq_in_ptr(desc, edesc->sec4_sg_dma, buflen +
req->nbytes, LDST_SGF);
@@ -1332,6 +1351,7 @@ static int ahash_update_first(struct ahash_request *req)
dma_addr_t src_dma;
u32 options;
struct ahash_edesc *edesc;
+ bool chained = false;
int ret = 0;
int sh_len;

@@ -1340,8 +1360,10 @@ static int ahash_update_first(struct ahash_request *req)
to_hash = req->nbytes - *next_buflen;

if (to_hash) {
- src_nents = sg_count(req->src, req->nbytes - (*next_buflen));
- dma_map_sg(jrdev, req->src, src_nents ? : 1, DMA_TO_DEVICE);
+ src_nents = sg_count(req->src, req->nbytes - (*next_buflen),
+ &chained);
+ dma_map_sg_chained(jrdev, req->src, src_nents ? : 1,
+ DMA_TO_DEVICE, chained);
sec4_sg_bytes = src_nents * sizeof(struct sec4_sg_entry);

/*
@@ -1357,6 +1379,7 @@ static int ahash_update_first(struct ahash_request *req)
}

edesc->src_nents = src_nents;
+ edesc->chained = chained;
edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (void *)edesc + sizeof(struct ahash_edesc) +
DESC_JOB_IO_LEN;
diff --git a/drivers/crypto/caam/sg_sw_sec4.h b/drivers/crypto/caam/sg_sw_sec4.h
index 2dda9e3..e0037c8 100644
--- a/drivers/crypto/caam/sg_sw_sec4.h
+++ b/drivers/crypto/caam/sg_sw_sec4.h
@@ -37,7 +37,7 @@ sg_to_sec4_sg(struct scatterlist *sg, int sg_count,
dma_to_sec4_sg_one(sec4_sg_ptr, sg_dma_address(sg),
sg_dma_len(sg), offset);
sec4_sg_ptr++;
- sg = sg_next(sg);
+ sg = scatterwalk_sg_next(sg);
sg_count--;
}
return sec4_sg_ptr - 1;
@@ -56,7 +56,8 @@ static inline void sg_to_sec4_sg_last(struct scatterlist *sg, int sg_count,
}

/* count number of elements in scatterlist */
-static inline int __sg_count(struct scatterlist *sg_list, int nbytes)
+static inline int __sg_count(struct scatterlist *sg_list, int nbytes,
+ bool *chained)
{
struct scatterlist *sg = sg_list;
int sg_nents = 0;
@@ -65,7 +66,7 @@ static inline int __sg_count(struct scatterlist *sg_list, int nbytes)
sg_nents++;
nbytes -= sg->length;
if (!sg_is_last(sg) && (sg + 1)->length == 0)
- BUG(); /* Not support chaining */
+ *chained = true;
sg = scatterwalk_sg_next(sg);
}

@@ -73,9 +74,10 @@ static inline int __sg_count(struct scatterlist *sg_list, int nbytes)
}

/* derive number of elements in scatterlist, but return 0 for 1 */
-static inline int sg_count(struct scatterlist *sg_list, int nbytes)
+static inline int sg_count(struct scatterlist *sg_list, int nbytes,
+ bool *chained)
{
- int sg_nents = __sg_count(sg_list, nbytes);
+ int sg_nents = __sg_count(sg_list, nbytes, chained);

if (likely(sg_nents == 1))
return 0;
@@ -83,6 +85,38 @@ static inline int sg_count(struct scatterlist *sg_list, int nbytes)
return sg_nents;
}

+static int dma_map_sg_chained(struct device *dev, struct scatterlist *sg,
+ unsigned int nents, enum dma_data_direction dir,
+ bool chained)
+{
+ if (unlikely(chained)) {
+ int i;
+ for (i = 0; i < nents; i++) {
+ dma_map_sg(dev, sg, 1, dir);
+ sg = scatterwalk_sg_next(sg);
+ }
+ } else {
+ dma_map_sg(dev, sg, nents, dir);
+ }
+ return nents;
+}
+
+static int dma_unmap_sg_chained(struct device *dev, struct scatterlist *sg,
+ unsigned int nents, enum dma_data_direction dir,
+ bool chained)
+{
+ if (unlikely(chained)) {
+ int i;
+ for (i = 0; i < nents; i++) {
+ dma_unmap_sg(dev, sg, 1, dir);
+ sg = scatterwalk_sg_next(sg);
+ }
+ } else {
+ dma_unmap_sg(dev, sg, nents, dir);
+ }
+ return nents;
+}
+
/* Copy from len bytes of sg to dest, starting from beginning */
static inline void sg_copy(u8 *dest, struct scatterlist *sg, unsigned int len)
{
--
1.7.11.1

2012-06-23 00:52:03

by Kim Phillips

Subject: [PATCH 18/20] crypto: caam - only query h/w in job ring dequeue path

Code was needlessly checking the s/w job ring when there
would be nothing to process if the h/w's output completion
ring were empty anyway.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/jr.c | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)

diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 11d93f2..7ae5e51 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -59,15 +59,15 @@ static void caam_jr_dequeue(unsigned long devarg)
u32 *userdesc, userstatus;
void *userarg;

- spin_lock_bh(&jrp->outlock);
+ while (rd_reg32(&jrp->rregs->outring_used)) {

- head = ACCESS_ONCE(jrp->head);
- sw_idx = tail = jrp->tail;
+ head = ACCESS_ONCE(jrp->head);

- while (CIRC_CNT(head, tail, JOBR_DEPTH) >= 1 &&
- rd_reg32(&jrp->rregs->outring_used)) {
+ spin_lock_bh(&jrp->outlock);

+ sw_idx = tail = jrp->tail;
hw_idx = jrp->out_ring_read_index;
+
for (i = 0; CIRC_CNT(head, tail + i, JOBR_DEPTH) >= 1; i++) {
sw_idx = (tail + i) & (JOBR_DEPTH - 1);

@@ -121,15 +121,8 @@ static void caam_jr_dequeue(unsigned long devarg)

/* Finally, execute user's callback */
usercall(dev, userdesc, userstatus, userarg);
-
- spin_lock_bh(&jrp->outlock);
-
- head = ACCESS_ONCE(jrp->head);
- sw_idx = tail = jrp->tail;
}

- spin_unlock_bh(&jrp->outlock);
-
/* reenable / unmask IRQs */
clrbits32(&jrp->rregs->rconfig_lo, JRCFG_IMSK);
}
--
1.7.11.1

2012-06-23 00:52:08

by Kim Phillips

Subject: [PATCH 20/20] crypto: caam - one tasklet per job ring

there is no noticeable benefit for multiple cores to process one
job ring's output ring: in fact, we can benefit from cache effects
of having the back-half stay on the core that receives a particular
ring's interrupts, and further relax general contention and the
locking involved with reading outring_used, since tasklets run
atomically.

Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/intern.h | 2 +-
drivers/crypto/caam/jr.c | 13 +++++--------
2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/drivers/crypto/caam/intern.h b/drivers/crypto/caam/intern.h
index ed2b739..5cd4c1b 100644
--- a/drivers/crypto/caam/intern.h
+++ b/drivers/crypto/caam/intern.h
@@ -43,7 +43,7 @@ struct caam_drv_private_jr {
struct device *parentdev; /* points back to controller dev */
int ridx;
struct caam_job_ring __iomem *rregs; /* JobR's register space */
- struct tasklet_struct irqtask[NR_CPUS];
+ struct tasklet_struct irqtask;
int irq; /* One per queue */
int assign; /* busy/free */

diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 0adaad1..7074a1a 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -43,7 +43,7 @@ static irqreturn_t caam_jr_interrupt(int irq, void *st_dev)
wr_reg32(&jrp->rregs->jrintstatus, irqstate);

preempt_disable();
- tasklet_schedule(&jrp->irqtask[smp_processor_id()]);
+ tasklet_schedule(&jrp->irqtask);
preempt_enable();

return IRQ_HANDLED;
@@ -322,11 +322,9 @@ static int caam_jr_init(struct device *dev)

jrp = dev_get_drvdata(dev);

- /* Connect job ring interrupt handler. */
- for_each_possible_cpu(i)
- tasklet_init(&jrp->irqtask[i], caam_jr_dequeue,
- (unsigned long)dev);
+ tasklet_init(&jrp->irqtask, caam_jr_dequeue, (unsigned long)dev);

+ /* Connect job ring interrupt handler. */
error = request_irq(jrp->irq, caam_jr_interrupt, IRQF_SHARED,
"caam-jobr", dev);
if (error) {
@@ -416,12 +414,11 @@ int caam_jr_shutdown(struct device *dev)
{
struct caam_drv_private_jr *jrp = dev_get_drvdata(dev);
dma_addr_t inpbusaddr, outbusaddr;
- int ret, i;
+ int ret;

ret = caam_reset_hw_jr(dev);

- for_each_possible_cpu(i)
- tasklet_kill(&jrp->irqtask[i]);
+ tasklet_kill(&jrp->irqtask);

/* Release interrupt */
free_irq(jrp->irq, dev);
--
1.7.11.1

2012-06-23 00:52:09

by Kim Phillips

Subject: [PATCH 11/20] crypto: caam - unkeyed ahash support

From: Yuan Kang <[email protected]>

caam supports and registers unkeyed sha algorithms and md5.

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/caamhash.c | 75 ++++++++++++++++++++++++++++++++----------
1 file changed, 58 insertions(+), 17 deletions(-)

diff --git a/drivers/crypto/caam/caamhash.c b/drivers/crypto/caam/caamhash.c
index c0b6d86..7dcf28f 100644
--- a/drivers/crypto/caam/caamhash.c
+++ b/drivers/crypto/caam/caamhash.c
@@ -1485,6 +1485,8 @@ static int ahash_import(struct ahash_request *req, const void *in)
struct caam_hash_template {
char name[CRYPTO_MAX_ALG_NAME];
char driver_name[CRYPTO_MAX_ALG_NAME];
+ char hmac_name[CRYPTO_MAX_ALG_NAME];
+ char hmac_driver_name[CRYPTO_MAX_ALG_NAME];
unsigned int blocksize;
struct ahash_alg template_ahash;
u32 alg_type;
@@ -1494,8 +1496,10 @@ struct caam_hash_template {
/* ahash descriptors */
static struct caam_hash_template driver_hash[] = {
{
- .name = "hmac(sha1)",
- .driver_name = "hmac-sha1-caam",
+ .name = "sha1",
+ .driver_name = "sha1-caam",
+ .hmac_name = "hmac(sha1)",
+ .hmac_driver_name = "hmac-sha1-caam",
.blocksize = SHA1_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
@@ -1513,8 +1517,10 @@ static struct caam_hash_template driver_hash[] = {
.alg_type = OP_ALG_ALGSEL_SHA1,
.alg_op = OP_ALG_ALGSEL_SHA1 | OP_ALG_AAI_HMAC,
}, {
- .name = "hmac(sha224)",
- .driver_name = "hmac-sha224-caam",
+ .name = "sha224",
+ .driver_name = "sha224-caam",
+ .hmac_name = "hmac(sha224)",
+ .hmac_driver_name = "hmac-sha224-caam",
.blocksize = SHA224_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
@@ -1532,8 +1538,10 @@ static struct caam_hash_template driver_hash[] = {
.alg_type = OP_ALG_ALGSEL_SHA224,
.alg_op = OP_ALG_ALGSEL_SHA224 | OP_ALG_AAI_HMAC,
}, {
- .name = "hmac(sha256)",
- .driver_name = "hmac-sha256-caam",
+ .name = "sha256",
+ .driver_name = "sha256-caam",
+ .hmac_name = "hmac(sha256)",
+ .hmac_driver_name = "hmac-sha256-caam",
.blocksize = SHA256_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
@@ -1551,8 +1559,10 @@ static struct caam_hash_template driver_hash[] = {
.alg_type = OP_ALG_ALGSEL_SHA256,
.alg_op = OP_ALG_ALGSEL_SHA256 | OP_ALG_AAI_HMAC,
}, {
- .name = "hmac(sha384)",
- .driver_name = "hmac-sha384-caam",
+ .name = "sha384",
+ .driver_name = "sha384-caam",
+ .hmac_name = "hmac(sha384)",
+ .hmac_driver_name = "hmac-sha384-caam",
.blocksize = SHA384_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
@@ -1570,8 +1580,10 @@ static struct caam_hash_template driver_hash[] = {
.alg_type = OP_ALG_ALGSEL_SHA384,
.alg_op = OP_ALG_ALGSEL_SHA384 | OP_ALG_AAI_HMAC,
}, {
- .name = "hmac(sha512)",
- .driver_name = "hmac-sha512-caam",
+ .name = "sha512",
+ .driver_name = "sha512-caam",
+ .hmac_name = "hmac(sha512)",
+ .hmac_driver_name = "hmac-sha512-caam",
.blocksize = SHA512_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
@@ -1589,8 +1601,10 @@ static struct caam_hash_template driver_hash[] = {
.alg_type = OP_ALG_ALGSEL_SHA512,
.alg_op = OP_ALG_ALGSEL_SHA512 | OP_ALG_AAI_HMAC,
}, {
- .name = "hmac(md5)",
- .driver_name = "hmac-md5-caam",
+ .name = "md5",
+ .driver_name = "md5-caam",
+ .hmac_name = "hmac(md5)",
+ .hmac_driver_name = "hmac-md5-caam",
.blocksize = MD5_BLOCK_WORDS * 4,
.template_ahash = {
.init = ahash_init,
@@ -1721,7 +1735,8 @@ static void __exit caam_algapi_hash_exit(void)
}

static struct caam_hash_alg *
-caam_hash_alloc(struct device *ctrldev, struct caam_hash_template *template)
+caam_hash_alloc(struct device *ctrldev, struct caam_hash_template *template,
+ bool keyed)
{
struct caam_hash_alg *t_alg;
struct ahash_alg *halg;
@@ -1737,9 +1752,17 @@ caam_hash_alloc(struct device *ctrldev, struct caam_hash_template *template)
halg = &t_alg->ahash_alg;
alg = &halg->halg.base;

- snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s", template->name);
- snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
- template->driver_name);
+ if (keyed) {
+ snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->hmac_name);
+ snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->hmac_driver_name);
+ } else {
+ snprintf(alg->cra_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->name);
+ snprintf(alg->cra_driver_name, CRYPTO_MAX_ALG_NAME, "%s",
+ template->driver_name);
+ }
alg->cra_module = THIS_MODULE;
alg->cra_init = caam_hash_cra_init;
alg->cra_exit = caam_hash_cra_exit;
@@ -1786,7 +1809,25 @@ static int __init caam_algapi_hash_init(void)
/* TODO: check if h/w supports alg */
struct caam_hash_alg *t_alg;

- t_alg = caam_hash_alloc(ctrldev, &driver_hash[i]);
+ /* register hmac version */
+ t_alg = caam_hash_alloc(ctrldev, &driver_hash[i], true);
+ if (IS_ERR(t_alg)) {
+ err = PTR_ERR(t_alg);
+ dev_warn(ctrldev, "%s alg allocation failed\n",
+ driver_hash[i].driver_name);
+ continue;
+ }
+
+ err = crypto_register_ahash(&t_alg->ahash_alg);
+ if (err) {
+ dev_warn(ctrldev, "%s alg registration failed\n",
+ t_alg->ahash_alg.halg.base.cra_driver_name);
+ kfree(t_alg);
+ } else
+ list_add_tail(&t_alg->entry, &priv->hash_list);
+
+ /* register unkeyed version */
+ t_alg = caam_hash_alloc(ctrldev, &driver_hash[i], false);
if (IS_ERR(t_alg)) {
err = PTR_ERR(t_alg);
dev_warn(ctrldev, "%s alg allocation failed\n",
--
1.7.11.1

2012-06-23 00:52:03

by Kim Phillips

Subject: [PATCH 08/20] crypto: caam - refactor key_gen, sg

From: Yuan Kang <[email protected]>

create separate files for split key generation and scatterlist functions.

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/Makefile | 2 +-
drivers/crypto/caam/caamalg.c | 197 ++------------------------------------
drivers/crypto/caam/desc.h | 10 ++
drivers/crypto/caam/key_gen.c | 122 +++++++++++++++++++++++
drivers/crypto/caam/key_gen.h | 17 ++++
drivers/crypto/caam/sg_link_tbl.h | 84 ++++++++++++++++
6 files changed, 242 insertions(+), 190 deletions(-)
create mode 100644 drivers/crypto/caam/key_gen.c
create mode 100644 drivers/crypto/caam/key_gen.h
create mode 100644 drivers/crypto/caam/sg_link_tbl.h

diff --git a/drivers/crypto/caam/Makefile b/drivers/crypto/caam/Makefile
index ef39011..4447e57 100644
--- a/drivers/crypto/caam/Makefile
+++ b/drivers/crypto/caam/Makefile
@@ -5,4 +5,4 @@
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM) += caam.o
obj-$(CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API) += caamalg.o

-caam-objs := ctrl.o jr.o error.o
+caam-objs := ctrl.o jr.o error.o key_gen.o
diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index a4e266f..ea0295d 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -51,6 +51,8 @@
#include "desc_constr.h"
#include "jr.h"
#include "error.h"
+#include "sg_link_tbl.h"
+#include "key_gen.h"

/*
* crypto alg
@@ -453,121 +455,12 @@ static int aead_setauthsize(struct crypto_aead *authenc,
return 0;
}

-struct split_key_result {
- struct completion completion;
- int err;
-};
-
-static void split_key_done(struct device *dev, u32 *desc, u32 err,
- void *context)
+static u32 gen_split_aead_key(struct caam_ctx *ctx, const u8 *key_in,
+ u32 authkeylen)
{
- struct split_key_result *res = context;
-
-#ifdef DEBUG
- dev_err(dev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
-#endif
-
- if (err) {
- char tmp[CAAM_ERROR_STR_MAX];
-
- dev_err(dev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
- }
-
- res->err = err;
-
- complete(&res->completion);
-}
-
-/*
-get a split ipad/opad key
-
-Split key generation-----------------------------------------------
-
-[00] 0xb0810008 jobdesc: stidx=1 share=never len=8
-[01] 0x04000014 key: class2->keyreg len=20
- @0xffe01000
-[03] 0x84410014 operation: cls2-op sha1 hmac init dec
-[04] 0x24940000 fifold: class2 msgdata-last2 len=0 imm
-[05] 0xa4000001 jump: class2 local all ->1 [06]
-[06] 0x64260028 fifostr: class2 mdsplit-jdk len=40
- @0xffe04000
-*/
-static u32 gen_split_key(struct caam_ctx *ctx, const u8 *key_in, u32 authkeylen)
-{
- struct device *jrdev = ctx->jrdev;
- u32 *desc;
- struct split_key_result result;
- dma_addr_t dma_addr_in, dma_addr_out;
- int ret = 0;
-
- desc = kmalloc(CAAM_CMD_SZ * 6 + CAAM_PTR_SZ * 2, GFP_KERNEL | GFP_DMA);
-
- init_job_desc(desc, 0);
-
- dma_addr_in = dma_map_single(jrdev, (void *)key_in, authkeylen,
- DMA_TO_DEVICE);
- if (dma_mapping_error(jrdev, dma_addr_in)) {
- dev_err(jrdev, "unable to map key input memory\n");
- kfree(desc);
- return -ENOMEM;
- }
- append_key(desc, dma_addr_in, authkeylen, CLASS_2 |
- KEY_DEST_CLASS_REG);
-
- /* Sets MDHA up into an HMAC-INIT */
- append_operation(desc, ctx->alg_op | OP_ALG_DECRYPT |
- OP_ALG_AS_INIT);
-
- /*
- * do a FIFO_LOAD of zero, this will trigger the internal key expansion
- into both pads inside MDHA
- */
- append_fifo_load_as_imm(desc, NULL, 0, LDST_CLASS_2_CCB |
- FIFOLD_TYPE_MSG | FIFOLD_TYPE_LAST2);
-
- /*
- * FIFO_STORE with the explicit split-key content store
- * (0x26 output type)
- */
- dma_addr_out = dma_map_single(jrdev, ctx->key, ctx->split_key_pad_len,
- DMA_FROM_DEVICE);
- if (dma_mapping_error(jrdev, dma_addr_out)) {
- dev_err(jrdev, "unable to map key output memory\n");
- kfree(desc);
- return -ENOMEM;
- }
- append_fifo_store(desc, dma_addr_out, ctx->split_key_len,
- LDST_CLASS_2_CCB | FIFOST_TYPE_SPLIT_KEK);
-
-#ifdef DEBUG
- print_hex_dump(KERN_ERR, "ctx.key@"xstr(__LINE__)": ",
- DUMP_PREFIX_ADDRESS, 16, 4, key_in, authkeylen, 1);
- print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
- DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
-#endif
-
- result.err = 0;
- init_completion(&result.completion);
-
- ret = caam_jr_enqueue(jrdev, desc, split_key_done, &result);
- if (!ret) {
- /* in progress */
- wait_for_completion_interruptible(&result.completion);
- ret = result.err;
-#ifdef DEBUG
- print_hex_dump(KERN_ERR, "ctx.key@"xstr(__LINE__)": ",
- DUMP_PREFIX_ADDRESS, 16, 4, ctx->key,
- ctx->split_key_pad_len, 1);
-#endif
- }
-
- dma_unmap_single(jrdev, dma_addr_out, ctx->split_key_pad_len,
- DMA_FROM_DEVICE);
- dma_unmap_single(jrdev, dma_addr_in, authkeylen, DMA_TO_DEVICE);
-
- kfree(desc);
-
- return ret;
+ return gen_split_key(ctx->jrdev, ctx->key, ctx->split_key_len,
+ ctx->split_key_pad_len, key_in, authkeylen,
+ ctx->alg_op);
}

static int aead_setkey(struct crypto_aead *aead,
@@ -611,7 +504,7 @@ static int aead_setkey(struct crypto_aead *aead,
DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
#endif

- ret = gen_split_key(ctx, key, authkeylen);
+ ret = gen_split_aead_key(ctx, key, authkeylen);
if (ret) {
goto badkey;
}
@@ -758,14 +651,6 @@ static int ablkcipher_setkey(struct crypto_ablkcipher *ablkcipher,
return ret;
}

-struct link_tbl_entry {
- u64 ptr;
- u32 len;
- u8 reserved;
- u8 buf_pool_id;
- u16 offset;
-};
-
/*
* aead_edesc - s/w-extended aead descriptor
* @assoc_nents: number of segments in associated data (SPI+Seq) scatterlist
@@ -1027,50 +912,6 @@ static void ablkcipher_decrypt_done(struct device *jrdev, u32 *desc, u32 err,
ablkcipher_request_complete(req, err);
}

-static void sg_to_link_tbl_one(struct link_tbl_entry *link_tbl_ptr,
- dma_addr_t dma, u32 len, u32 offset)
-{
- link_tbl_ptr->ptr = dma;
- link_tbl_ptr->len = len;
- link_tbl_ptr->reserved = 0;
- link_tbl_ptr->buf_pool_id = 0;
- link_tbl_ptr->offset = offset;
-#ifdef DEBUG
- print_hex_dump(KERN_ERR, "link_tbl_ptr@"xstr(__LINE__)": ",
- DUMP_PREFIX_ADDRESS, 16, 4, link_tbl_ptr,
- sizeof(struct link_tbl_entry), 1);
-#endif
-}
-
-/*
- * convert scatterlist to h/w link table format
- * but does not have final bit; instead, returns last entry
- */
-static struct link_tbl_entry *sg_to_link_tbl(struct scatterlist *sg,
- int sg_count, struct link_tbl_entry
- *link_tbl_ptr, u32 offset)
-{
- while (sg_count) {
- sg_to_link_tbl_one(link_tbl_ptr, sg_dma_address(sg),
- sg_dma_len(sg), offset);
- link_tbl_ptr++;
- sg = sg_next(sg);
- sg_count--;
- }
- return link_tbl_ptr - 1;
-}
-
-/*
- * convert scatterlist to h/w link table format
- * scatterlist must have been previously dma mapped
- */
-static void sg_to_link_tbl_last(struct scatterlist *sg, int sg_count,
- struct link_tbl_entry *link_tbl_ptr, u32 offset)
-{
- link_tbl_ptr = sg_to_link_tbl(sg, sg_count, link_tbl_ptr, offset);
- link_tbl_ptr->len |= 0x40000000;
-}
-
/*
* Fill in aead job descriptor
*/
@@ -1272,28 +1113,6 @@ static void init_ablkcipher_job(u32 *sh_desc, dma_addr_t ptr,
}

/*
- * derive number of elements in scatterlist
- */
-static int sg_count(struct scatterlist *sg_list, int nbytes)
-{
- struct scatterlist *sg = sg_list;
- int sg_nents = 0;
-
- while (nbytes > 0) {
- sg_nents++;
- nbytes -= sg->length;
- if (!sg_is_last(sg) && (sg + 1)->length == 0)
- BUG(); /* Not support chaining */
- sg = scatterwalk_sg_next(sg);
- }
-
- if (likely(sg_nents == 1))
- return 0;
-
- return sg_nents;
-}
-
-/*
* allocate and map the aead extended descriptor
*/
static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
diff --git a/drivers/crypto/caam/desc.h b/drivers/crypto/caam/desc.h
index af25e76..48c1927 100644
--- a/drivers/crypto/caam/desc.h
+++ b/drivers/crypto/caam/desc.h
@@ -8,6 +8,16 @@
#ifndef DESC_H
#define DESC_H

+struct link_tbl_entry {
+ u64 ptr;
+#define LINK_TBL_LEN_FIN 0x40000000
+#define LINK_TBL_LEN_EXT 0x80000000
+ u32 len;
+ u8 reserved;
+ u8 buf_pool_id;
+ u16 offset;
+};
+
/* Max size of any CAAM descriptor in 32-bit words, inclusive of header */
#define MAX_CAAM_DESCSIZE 64

diff --git a/drivers/crypto/caam/key_gen.c b/drivers/crypto/caam/key_gen.c
new file mode 100644
index 0000000..0028881
--- /dev/null
+++ b/drivers/crypto/caam/key_gen.c
@@ -0,0 +1,122 @@
+/*
+ * CAAM/SEC 4.x functions for handling key-generation jobs
+ *
+ * Copyright 2008-2011 Freescale Semiconductor, Inc.
+ *
+ */
+#include "compat.h"
+#include "jr.h"
+#include "error.h"
+#include "desc_constr.h"
+#include "key_gen.h"
+
+void split_key_done(struct device *dev, u32 *desc, u32 err,
+ void *context)
+{
+ struct split_key_result *res = context;
+
+#ifdef DEBUG
+ dev_err(dev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
+#endif
+
+ if (err) {
+ char tmp[CAAM_ERROR_STR_MAX];
+
+ dev_err(dev, "%08x: %s\n", err, caam_jr_strstatus(tmp, err));
+ }
+
+ res->err = err;
+
+ complete(&res->completion);
+}
+EXPORT_SYMBOL(split_key_done);
+/*
+get a split ipad/opad key
+
+Split key generation-----------------------------------------------
+
+[00] 0xb0810008 jobdesc: stidx=1 share=never len=8
+[01] 0x04000014 key: class2->keyreg len=20
+ @0xffe01000
+[03] 0x84410014 operation: cls2-op sha1 hmac init dec
+[04] 0x24940000 fifold: class2 msgdata-last2 len=0 imm
+[05] 0xa4000001 jump: class2 local all ->1 [06]
+[06] 0x64260028 fifostr: class2 mdsplit-jdk len=40
+ @0xffe04000
+*/
+u32 gen_split_key(struct device *jrdev, u8 *key_out, int split_key_len,
+ int split_key_pad_len, const u8 *key_in, u32 keylen,
+ u32 alg_op)
+{
+ u32 *desc;
+ struct split_key_result result;
+ dma_addr_t dma_addr_in, dma_addr_out;
+ int ret = 0;
+
+ desc = kmalloc(CAAM_CMD_SZ * 6 + CAAM_PTR_SZ * 2, GFP_KERNEL | GFP_DMA);
+
+ init_job_desc(desc, 0);
+
+ dma_addr_in = dma_map_single(jrdev, (void *)key_in, keylen,
+ DMA_TO_DEVICE);
+ if (dma_mapping_error(jrdev, dma_addr_in)) {
+ dev_err(jrdev, "unable to map key input memory\n");
+ kfree(desc);
+ return -ENOMEM;
+ }
+ append_key(desc, dma_addr_in, keylen, CLASS_2 | KEY_DEST_CLASS_REG);
+
+ /* Sets MDHA up into an HMAC-INIT */
+ append_operation(desc, alg_op | OP_ALG_DECRYPT | OP_ALG_AS_INIT);
+
+ /*
+ * do a FIFO_LOAD of zero, this will trigger the internal key expansion
+ * into both pads inside MDHA
+ */
+ append_fifo_load_as_imm(desc, NULL, 0, LDST_CLASS_2_CCB |
+ FIFOLD_TYPE_MSG | FIFOLD_TYPE_LAST2);
+
+ /*
+ * FIFO_STORE with the explicit split-key content store
+ * (0x26 output type)
+ */
+ dma_addr_out = dma_map_single(jrdev, key_out, split_key_pad_len,
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(jrdev, dma_addr_out)) {
+ dev_err(jrdev, "unable to map key output memory\n");
+ kfree(desc);
+ return -ENOMEM;
+ }
+ append_fifo_store(desc, dma_addr_out, split_key_len,
+ LDST_CLASS_2_CCB | FIFOST_TYPE_SPLIT_KEK);
+
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx.key@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key_in, keylen, 1);
+ print_hex_dump(KERN_ERR, "jobdesc@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
+#endif
+
+ result.err = 0;
+ init_completion(&result.completion);
+
+ ret = caam_jr_enqueue(jrdev, desc, split_key_done, &result);
+ if (!ret) {
+ /* in progress */
+ wait_for_completion_interruptible(&result.completion);
+ ret = result.err;
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "ctx.key@"xstr(__LINE__)": ",
+ DUMP_PREFIX_ADDRESS, 16, 4, key_out,
+ split_key_pad_len, 1);
+#endif
+ }
+
+ dma_unmap_single(jrdev, dma_addr_out, split_key_pad_len,
+ DMA_FROM_DEVICE);
+ dma_unmap_single(jrdev, dma_addr_in, keylen, DMA_TO_DEVICE);
+
+ kfree(desc);
+
+ return ret;
+}
diff --git a/drivers/crypto/caam/key_gen.h b/drivers/crypto/caam/key_gen.h
new file mode 100644
index 0000000..d95d290
--- /dev/null
+++ b/drivers/crypto/caam/key_gen.h
@@ -0,0 +1,17 @@
+/*
+ * CAAM/SEC 4.x definitions for handling key-generation jobs
+ *
+ * Copyright 2008-2011 Freescale Semiconductor, Inc.
+ *
+ */
+
+struct split_key_result {
+ struct completion completion;
+ int err;
+};
+
+void split_key_done(struct device *dev, u32 *desc, u32 err, void *context);
+
+u32 gen_split_key(struct device *jrdev, u8 *key_out, int split_key_len,
+ int split_key_pad_len, const u8 *key_in, u32 keylen,
+ u32 alg_op);
diff --git a/drivers/crypto/caam/sg_link_tbl.h b/drivers/crypto/caam/sg_link_tbl.h
new file mode 100644
index 0000000..6df4349
--- /dev/null
+++ b/drivers/crypto/caam/sg_link_tbl.h
@@ -0,0 +1,84 @@
+/*
+ * CAAM/SEC 4.x functions for using scatterlists in caam driver
+ *
+ * Copyright 2008-2011 Freescale Semiconductor, Inc.
+ *
+ */
+
+struct link_tbl_entry;
+
+/*
+ * convert single dma address to h/w link table format
+ */
+static inline void sg_to_link_tbl_one(struct link_tbl_entry *link_tbl_ptr,
+ dma_addr_t dma, u32 len, u32 offset)
+{
+ link_tbl_ptr->ptr = dma;
+ link_tbl_ptr->len = len;
+ link_tbl_ptr->reserved = 0;
+ link_tbl_ptr->buf_pool_id = 0;
+ link_tbl_ptr->offset = offset;
+#ifdef DEBUG
+ print_hex_dump(KERN_ERR, "link_tbl_ptr@: ",
+ DUMP_PREFIX_ADDRESS, 16, 4, link_tbl_ptr,
+ sizeof(struct link_tbl_entry), 1);
+#endif
+}
+
+/*
+ * convert scatterlist to h/w link table format
+ * but does not have final bit; instead, returns last entry
+ */
+static inline struct link_tbl_entry *
+sg_to_link_tbl(struct scatterlist *sg, int sg_count,
+ struct link_tbl_entry *link_tbl_ptr, u32 offset)
+{
+ while (sg_count) {
+ sg_to_link_tbl_one(link_tbl_ptr, sg_dma_address(sg),
+ sg_dma_len(sg), offset);
+ link_tbl_ptr++;
+ sg = sg_next(sg);
+ sg_count--;
+ }
+ return link_tbl_ptr - 1;
+}
+
+/*
+ * convert scatterlist to h/w link table format
+ * scatterlist must have been previously dma mapped
+ */
+static inline void sg_to_link_tbl_last(struct scatterlist *sg, int sg_count,
+ struct link_tbl_entry *link_tbl_ptr,
+ u32 offset)
+{
+ link_tbl_ptr = sg_to_link_tbl(sg, sg_count, link_tbl_ptr, offset);
+ link_tbl_ptr->len |= LINK_TBL_LEN_FIN;
+}
+
+/* count number of elements in scatterlist */
+static inline int __sg_count(struct scatterlist *sg_list, int nbytes)
+{
+ struct scatterlist *sg = sg_list;
+ int sg_nents = 0;
+
+ while (nbytes > 0) {
+ sg_nents++;
+ nbytes -= sg->length;
+ if (!sg_is_last(sg) && (sg + 1)->length == 0)
+ BUG(); /* Not support chaining */
+ sg = scatterwalk_sg_next(sg);
+ }
+
+ return sg_nents;
+}
+
+/* derive number of elements in scatterlist, but return 0 for 1 */
+static inline int sg_count(struct scatterlist *sg_list, int nbytes)
+{
+ int sg_nents = __sg_count(sg_list, nbytes);
+
+ if (likely(sg_nents == 1))
+ return 0;
+
+ return sg_nents;
+}
--
1.7.11.1

2012-06-23 00:52:08

by Kim Phillips

Subject: [PATCH 06/20] crypto: caam - support external seq in/out lengths

From: Yuan Kang <[email protected]>

functions for external storage of seq in/out lengths,
i.e., for 32-bit lengths.

These type-dependent functions automatically determine whether to
store the length internally (embedded in the command header word) or
externally (after the address pointer), based on size of the type
given.

Signed-off-by: Yuan Kang <[email protected]>
Signed-off-by: Kim Phillips <[email protected]>
---
drivers/crypto/caam/caamalg.c | 5 ++--
drivers/crypto/caam/desc_constr.h | 49 +++++++++++++++++++++++++++++++++++++--
2 files changed, 50 insertions(+), 4 deletions(-)

diff --git a/drivers/crypto/caam/caamalg.c b/drivers/crypto/caam/caamalg.c
index 5c10dc5..d0f8df1 100644
--- a/drivers/crypto/caam/caamalg.c
+++ b/drivers/crypto/caam/caamalg.c
@@ -37,9 +37,10 @@
* | ShareDesc Pointer |
* | SEQ_OUT_PTR |
* | (output buffer) |
+ * | (output length) |
* | SEQ_IN_PTR |
* | (input buffer) |
- * | LOAD (to DECO) |
+ * | (input length) |
* ---------------------
*/

@@ -62,7 +63,7 @@
#define CAAM_MAX_IV_LENGTH 16

/* length of descriptors text */
-#define DESC_JOB_IO_LEN (CAAM_CMD_SZ * 3 + CAAM_PTR_SZ * 3)
+#define DESC_JOB_IO_LEN (CAAM_CMD_SZ * 5 + CAAM_PTR_SZ * 3)

#define DESC_AEAD_BASE (4 * CAAM_CMD_SZ)
#define DESC_AEAD_ENC_LEN (DESC_AEAD_BASE + 16 * CAAM_CMD_SZ)
diff --git a/drivers/crypto/caam/desc_constr.h b/drivers/crypto/caam/desc_constr.h
index 8e1056f..c85c1f0 100644
--- a/drivers/crypto/caam/desc_constr.h
+++ b/drivers/crypto/caam/desc_constr.h
@@ -117,6 +117,15 @@ static inline void append_cmd_ptr(u32 *desc, dma_addr_t ptr, int len,
append_ptr(desc, ptr);
}

+/* Write length after pointer, rather than inside command */
+static inline void append_cmd_ptr_extlen(u32 *desc, dma_addr_t ptr,
+ unsigned int len, u32 command)
+{
+ append_cmd(desc, command);
+ append_ptr(desc, ptr);
+ append_cmd(desc, len);
+}
+
static inline void append_cmd_data(u32 *desc, void *data, int len,
u32 command)
{
@@ -166,13 +175,22 @@ static inline void append_##cmd(u32 *desc, dma_addr_t ptr, unsigned int len, \
append_cmd_ptr(desc, ptr, len, CMD_##op | options); \
}
APPEND_CMD_PTR(key, KEY)
-APPEND_CMD_PTR(seq_in_ptr, SEQ_IN_PTR)
-APPEND_CMD_PTR(seq_out_ptr, SEQ_OUT_PTR)
APPEND_CMD_PTR(load, LOAD)
APPEND_CMD_PTR(store, STORE)
APPEND_CMD_PTR(fifo_load, FIFO_LOAD)
APPEND_CMD_PTR(fifo_store, FIFO_STORE)

+#define APPEND_SEQ_PTR_INTLEN(cmd, op) \
+static inline void append_seq_##cmd##_ptr_intlen(u32 *desc, dma_addr_t ptr, \
+ unsigned int len, \
+ u32 options) \
+{ \
+ PRINT_POS; \
+ append_cmd_ptr(desc, ptr, len, CMD_SEQ_##op##_PTR | options); \
+}
+APPEND_SEQ_PTR_INTLEN(in, IN)
+APPEND_SEQ_PTR_INTLEN(out, OUT)
+
#define APPEND_CMD_PTR_TO_IMM(cmd, op) \
static inline void append_##cmd##_as_imm(u32 *desc, void *data, \
unsigned int len, u32 options) \
@@ -183,6 +201,33 @@ static inline void append_##cmd##_as_imm(u32 *desc, void *data, \
APPEND_CMD_PTR_TO_IMM(load, LOAD);
APPEND_CMD_PTR_TO_IMM(fifo_load, FIFO_LOAD);

+#define APPEND_CMD_PTR_EXTLEN(cmd, op) \
+static inline void append_##cmd##_extlen(u32 *desc, dma_addr_t ptr, \
+ unsigned int len, u32 options) \
+{ \
+ PRINT_POS; \
+ append_cmd_ptr_extlen(desc, ptr, len, CMD_##op | SQIN_EXT | options); \
+}
+APPEND_CMD_PTR_EXTLEN(seq_in_ptr, SEQ_IN_PTR)
+APPEND_CMD_PTR_EXTLEN(seq_out_ptr, SEQ_OUT_PTR)
+
+/*
+ * Determine whether to store length internally or externally depending on
+ * the size of its type
+ */
+#define APPEND_CMD_PTR_LEN(cmd, op, type) \
+static inline void append_##cmd(u32 *desc, dma_addr_t ptr, \
+ type len, u32 options) \
+{ \
+ PRINT_POS; \
+ if (sizeof(type) > sizeof(u16)) \
+ append_##cmd##_extlen(desc, ptr, len, options); \
+ else \
+ append_##cmd##_intlen(desc, ptr, len, options); \
+}
+APPEND_CMD_PTR_LEN(seq_in_ptr, SEQ_IN_PTR, u32)
+APPEND_CMD_PTR_LEN(seq_out_ptr, SEQ_OUT_PTR, u32)
+
/*
* 2nd variant for commands whose specified immediate length differs
* from length of immediate data provided, e.g., split keys
--
1.7.11.1

2012-06-27 06:51:47

by Herbert Xu

Subject: Re: [PATCH 00/20] crypto: caam - driver updates

On Fri, Jun 22, 2012 at 07:42:34PM -0500, Kim Phillips wrote:
> The following updates bring the upstream CAAM driver mostly up to date wrt
> Freescale's SDK 1.2 release:
>
> patch 1 is a cosmetic fix
>
> patch 2 is a mapping size fix
>
> patches 3-5 are protocol descriptor generation updates (not currently used
> in the kernel driver, but these header files are shared with other projects in
> the SDK; hope that's ok).
>
> patches 6-9 refactor code in preparation for ahash and hwrng support; and
> patch 7 removes the job ring registration interface in favour of user-managed
> selection and sharing of rings (see caam_rng_init, which simply chooses
> job ring 0 for all its operations).
>
> patches 10-12 add ahash support (keyed and unkeyed)
>
> patch 13 adds hwrng support
>
> patches 14-15 add support for the SEC v5.x series, which most notably gets a
> new RNG (tested on P5040 h/w)
>
> patches 16-20 contain various performance improvements

All applied. Thanks Kim!

> There are two patches remaining in the SDK that I'm holding off for now,
> as I believe they need more testing: use NAPI instead of tasklet, and
> set descriptor sharing type to SERIAL.

While using the NAPI concept definitely makes sense, I'm still
unsure about using napi_struct directly in crypto code. Dave,
did you have a chance to look at their patch to use NAPI in their
crypto driver?

Cheers,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

2012-06-27 08:17:25

by David Miller

Subject: Re: [PATCH 00/20] crypto: caam - driver updates

From: Herbert Xu <[email protected]>
Date: Wed, 27 Jun 2012 14:51:44 +0800

> While using the NAPI concept definitely makes sense, I'm still
> unsure about using napi_struct directly in crypto code. Dave,
> did you have a chance to look at their patch to use NAPI in their
> crypto driver?

I'll take a look at it tomorrow, thanks.

2012-06-30 08:18:52

by David Miller

Subject: Re: [PATCH 00/20] crypto: caam - driver updates

From: Herbert Xu <[email protected]>
Date: Wed, 27 Jun 2012 14:51:44 +0800

> While using the NAPI concept definitely makes sense, I'm still
> unsure about using napi_struct directly in crypto code. Dave,
> did you have a chance to look at their patch to use NAPI in their
> crypto driver?

So I think using the existing napi context is overkill for a crypto
driver, there is so much stuff in that structure that will not be used
and just end up being wasted space.

Better to create a crypto_napi struct that caters more closely to
just what crypto driver need.

2012-06-30 13:54:46

by Herbert Xu

Subject: Re: [PATCH 00/20] crypto: caam - driver updates

On Sat, Jun 30, 2012 at 01:18:51AM -0700, David Miller wrote:
> From: Herbert Xu <[email protected]>
> Date: Wed, 27 Jun 2012 14:51:44 +0800
>
> > While using the NAPI concept definitely makes sense, I'm still
> > unsure about using napi_struct directly in crypto code. Dave,
> > did you have a chance to look at their patch to use NAPI in their
> > crypto driver?
>
> So I think using the existing napi context is overkill for a crypto
> driver, there is so much stuff in that structure that will not be used
> and just end up being wasted space.
>
> Better to create a crypto_napi struct that caters more closely to
> just what crypto driver need.

Yeah I agree completely.

In fact, now that I have had a bit more time to think this through
it seems that what they need isn't necessarily NAPI but better
management of the tasklet to achieve the same level of turnaround
performance as provided by NAPI + net_rx_action.

I'll try to spend some time on this next week.

Thanks,
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt