Hi,
The kernel crypto API requires the caller to set an IV in the request data
structure. That request data structure shall define one particular cipher
operation. During the cipher operation, the IV is read by the cipher
implementation and eventually the potentially updated IV (e.g. in case of CBC)
is written back to the memory location the request data structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in
the AF_ALG context that is unique to one particular AF_ALG socket. Note the
analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one
request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one
recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived
from one IOVEC) implies that one request data structure is created with the
data to be processed by the cipher implementation. The IV that was set with
the sendmsg call is registered with the request data structure before the
cipher operation.
In case of an AIO operation, the cipher operation invocation returns
immediately, queuing the request to the hardware. While the AIO request is
processed by the hardware, recvmsg processes the next IOVEC for which another
request is created. Again, the IV buffer from the AF_ALG socket context is
registered with the new request and the cipher operation is invoked.
You may now see that there is a potential race condition regarding the IV
handling, because there is *no* separate IV buffer for the different requests.
This is nicely demonstrated with libkcapi using the following command which
creates an AIO request with two IOCBs each encrypting one AES block in CBC
mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is processed,
the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input to
the 2nd block.
In case the first AIO request is not completed before the 2nd request
commences, the result is two identical AES blocks (i.e. both use the same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be a
memory corruption in the IV buffer if both AIO requests write to the IV buffer
at the same time.
This needs to be solved somehow. I see the following options which I would
like to have vetted by the community.
1. Require that the cipher implementations serialize any AIO requests that
have dependencies. I.e. for CBC, requests need to be serialized by the driver.
For, say, ECB or XTS no serialization is necessary.
2. Change AF_ALG to require a per-request IV. This could be implemented by
moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg
code path would obtain the IV.
I would tend to favor option 2 as this requires code change at only location.
If option 2 is considered, I would recommend to still allow setting the IV via
sendmsg CMSG (to keep the interface stable). If, however, the caller provides
an IV via recvmsg, this takes precedence.
If there are other options, please allow us to learn about them.
Ciao
Stephan
The kernel crypto API requires the caller to set an IV in the request
data structure. That request data structure shall define one particular
cipher operation. During the cipher operation, the IV is read by the
cipher implementation and eventually the potentially updated IV (e.g.
in case of CBC) is written back to the memory location the request data
structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is
stored in the AF_ALG context that is unique to one particular AF_ALG
socket. Note the analogy: an AF_ALG socket is like a TFM where one
recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
with one recvmsg call, multiple IOVECs can be specified. Each
individual IOCB (derived from one IOVEC) implies that one request data
structure is created with the data to be processed by the cipher
implementation. The IV that was set with the sendmsg call is registered
with the request data structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns
immediately, queuing the request to the hardware. While the AIO request
is processed by the hardware, recvmsg processes the next IOVEC for
which another request is created. Again, the IV buffer from the AF_ALG
socket context is registered with the new request and the cipher
operation is invoked.
You may now see that there is a potential race condition regarding the
IV handling, because there is *no* separate IV buffer for the different
requests. This is nicely demonstrated with libkcapi using the following
command which creates an AIO request with two IOCBs each encrypting one
AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is
processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input
to the 2nd block.
In case the first AIO request is not completed before the 2nd request
commences, the result is two identical AES blocks (i.e. both use the
same IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can
be a memory corruption in the IV buffer if both AIO requests write to
the IV buffer at the same time.
The solution is to allow providing the IV data supplied as part of the
plaintext/ciphertext. To do so, the AF_ALG interface treats the
ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
cipher operation together with the flag whether the operation should
enable support for inline IV handling.
If inline IV handling is enabled, the IV is expected to be the first
part of the input plaintext/ciphertext. This IV is only used for one
cipher operation and will not retained in the kernel for subsequent
cipher operations.
The AEAD support required a slight re-arragning of the code, because
obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
access in _aead_recvmsg must be moved below the IV gathering.
The AEAD code to find the first SG with data in the TX SGL is moved to a
common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is
allowed to provide an IV via sendmsg. It only extends the interface by
giving the user the choice to provide the IV either via sendmsg (the
current approach) or as part of the data (the additional approach).
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 79 ++++++++++++++++++++++++++++++++++++++++++++-
crypto/algif_aead.c | 54 ++++++++++++++-----------------
crypto/algif_skcipher.c | 14 +++++---
include/crypto/if_alg.h | 17 ++++++++++
include/uapi/linux/if_alg.h | 6 ++--
5 files changed, 133 insertions(+), 37 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 5231f421ad00..87394dc90ac0 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -14,6 +14,7 @@
#include <linux/atomic.h>
#include <crypto/if_alg.h>
+#include <crypto/scatterwalk.h>
#include <linux/crypto.h>
#include <linux/init.h>
#include <linux/kernel.h>
@@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
+ bool iiv = 0;
bool init = 0;
int err = 0;
@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
return err;
init = 1;
- switch (con.op) {
+ switch (con.op & ALG_OP_CIPHER_MASK) {
case ALG_OP_ENCRYPT:
enc = 1;
break;
@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
return -EINVAL;
}
+ if (con.op & ALG_OP_INLINE_IV)
+ iiv = 1;
+
if (con.iv && con.iv->ivlen != ivsize)
return -EINVAL;
}
@@ -866,6 +871,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
if (init) {
ctx->enc = enc;
+ ctx->iiv = iiv;
if (con.iv)
memcpy(ctx->iv, con.iv->iv, ivsize);
@@ -1031,6 +1037,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)
struct sock *sk = areq->sk;
af_alg_free_areq_sgls(areq);
+ if (areq->ivlen)
+ sock_kfree_s(sk, areq->iv, areq->ivlen);
sock_kfree_s(sk, areq, areq->areqlen);
}
EXPORT_SYMBOL_GPL(af_alg_free_resources);
@@ -1175,6 +1183,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
}
EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
+/**
+ * af_alg_get_tsgl_sg - get first SG entry with data from TX SGL
+ * @ctx [in] AF_ALG context with TX sgl
+ * @return pointer to SG with data or NULL if none found
+ */
+struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx)
+{
+ struct af_alg_tsgl *sgl, *tmp;
+ struct scatterlist *sg = NULL;
+ unsigned int i;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ for (i = 0; i < sgl->cur; i++) {
+ struct scatterlist *process_sg = sgl->sg + i;
+
+ if (!(process_sg->length) || !sg_page(process_sg))
+ continue;
+ sg = process_sg;
+ break;
+ }
+ if (sg)
+ break;
+ }
+
+ return sg;
+}
+EXPORT_SYMBOL_GPL(af_alg_get_tsgl_sg);
+
+/**
+ * af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX
+ *
+ * @sk [in] AF_ALG socket
+ * @areq [in/out] request buffer that will receive the IV
+ * to be used for the cipher
+ * @return 0 on success, < 0 on error
+ */
+int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+ struct scatterlist *sg;
+
+ /* No inline IV or cipher has no IV, use ctx IV buffer. */
+ if (!ctx->iiv || !ctx->ivlen) {
+ areq->iv = ctx->iv;
+ areq->ivlen = 0; // areq->iv will not be freed
+ return 0;
+ }
+
+ /* There must be the IV data present. */
+ if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
+ return -EINVAL;
+
+ areq->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
+ if (unlikely(!areq->iv))
+ return -ENOMEM;
+ areq->ivlen = ctx->ivlen; // areq->iv will be freed
+
+ /* Get ivlen data from TX SGL and copy it into areq->iv. */
+ sg = af_alg_get_tsgl_sg(ctx);
+ if (!sg)
+ return -EFAULT;
+ scatterwalk_map_and_copy(areq->iv, sg, 0, ctx->ivlen, 0);
+ af_alg_pull_tsgl(sk, ctx->ivlen, NULL, 0);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(af_alg_get_iv);
+
static int __init af_alg_init(void)
{
int err = proto_register(&alg_proto, 0);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 4b07edd5a9ff..7eb7cb132c09 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
struct aead_tfm *aeadc = pask->private;
struct crypto_aead *tfm = aeadc->aead;
struct crypto_skcipher *null_tfm = aeadc->null_tfm;
- unsigned int i, as = crypto_aead_authsize(tfm);
+ unsigned int as = crypto_aead_authsize(tfm);
struct af_alg_async_req *areq;
- struct af_alg_tsgl *tsgl, *tmp;
struct scatterlist *rsgl_src, *tsgl_src = NULL;
int err = 0;
size_t used = 0; /* [in] TX bufs to be en/decrypted */
@@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
return err;
}
- /*
- * Data length provided by caller via sendmsg/sendpage that has not
- * yet been processed.
- */
- used = ctx->used;
-
/*
* Make sure sufficient data is present -- note, the same check is
* is also present in sendmsg/sendpage. The checks in sendpage/sendmsg
@@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (!aead_sufficient_data(sk))
return -EINVAL;
+ /* Allocate cipher request for current operation. */
+ areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
+ crypto_aead_reqsize(tfm));
+ if (IS_ERR(areq))
+ return PTR_ERR(areq);
+
+ /* Set areq->iv. */
+ err = af_alg_get_iv(sk, areq);
+ if (err)
+ goto free;
+
+ /*
+ * Data length provided by caller via sendmsg/sendpage that has not
+ * yet been processed.
+ */
+ used = ctx->used;
+
/*
* Calculate the minimum output buffer size holding the result of the
* cipher operation. When encrypting data, the receiving buffer is
@@ -153,12 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
*/
used -= ctx->aead_assoclen;
- /* Allocate cipher request for current operation. */
- areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
- crypto_aead_reqsize(tfm));
- if (IS_ERR(areq))
- return PTR_ERR(areq);
-
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
if (err)
@@ -183,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
}
processed = used + ctx->aead_assoclen;
- list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) {
- for (i = 0; i < tsgl->cur; i++) {
- struct scatterlist *process_sg = tsgl->sg + i;
-
- if (!(process_sg->length) || !sg_page(process_sg))
- continue;
- tsgl_src = process_sg;
- break;
- }
- if (tsgl_src)
- break;
- }
+ tsgl_src = af_alg_get_tsgl_sg(ctx);
if (processed && !tsgl_src) {
err = -EFAULT;
goto free;
@@ -282,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
/* Initialize the crypto operation */
aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src,
- areq->first_rsgl.sgl.sg, used, ctx->iv);
+ areq->first_rsgl.sgl.sg, used, areq->iv);
aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen);
aead_request_set_tfm(&areq->cra_u.aead_req, tfm);
@@ -549,19 +542,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
struct aead_tfm *tfm = private;
struct crypto_aead *aead = tfm->aead;
unsigned int len = sizeof(*ctx);
- unsigned int ivlen = crypto_aead_ivsize(aead);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
return -ENOMEM;
memset(ctx, 0, len);
- ctx->iv = sock_kmalloc(sk, ivlen, GFP_KERNEL);
+ ctx->ivlen = crypto_aead_ivsize(aead);
+ ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
if (!ctx->iv) {
sock_kfree_s(sk, ctx, len);
return -ENOMEM;
}
- memset(ctx->iv, 0, ivlen);
+ memset(ctx->iv, 0, ctx->ivlen);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
@@ -570,6 +563,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->iiv = 0;
ctx->aead_assoclen = 0;
crypto_init_wait(&ctx->wait);
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index c88e5e4cd6a6..d40e1d6797d8 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -77,6 +77,11 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
+ /* Set areq->iv */
+ err = af_alg_get_iv(sk, areq);
+ if (err)
+ goto free;
+
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, -1, &len);
if (err)
@@ -112,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
/* Initialize the crypto operation */
skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm);
skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl,
- areq->first_rsgl.sgl.sg, len, ctx->iv);
+ areq->first_rsgl.sgl.sg, len, areq->iv);
if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
/* AIO operation */
@@ -345,14 +350,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
if (!ctx)
return -ENOMEM;
- ctx->iv = sock_kmalloc(sk, crypto_skcipher_ivsize(tfm),
- GFP_KERNEL);
+ ctx->ivlen = crypto_skcipher_ivsize(tfm);
+ ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
if (!ctx->iv) {
sock_kfree_s(sk, ctx, len);
return -ENOMEM;
}
- memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
+ memset(ctx->iv, 0, ctx->ivlen);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
@@ -361,6 +366,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->iiv = 0;
crypto_init_wait(&ctx->wait);
ask->private = ctx;
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index f38227a78eae..ebc651ceb54a 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -95,6 +95,14 @@ struct af_alg_rsgl {
* @tsgl_entries: Number of entries in priv. TX SGL
* @outlen: Number of output bytes generated by crypto op
* @areqlen: Length of this data structure
+ * @iv: Buffer holding the IV for the cipher operation (this
+ * is either ctx->iv where ivlen is set to 0 or a newly
+ * allocated buffer where ivlen represents the size of
+ * this buffer).
+ * @ivlen: IV size -- if the IV buffer is to be freed during
+ * releasing of this data structure, ivlen is non-zero.
+ * If ivlen is zero, but iv is non-NULL, the iv buffer is
+ * not freed.
* @cra_u: Cipher request
*/
struct af_alg_async_req {
@@ -111,6 +119,9 @@ struct af_alg_async_req {
unsigned int outlen;
unsigned int areqlen;
+ void *iv;
+ unsigned int ivlen;
+
union {
struct aead_request aead_req;
struct skcipher_request skcipher_req;
@@ -127,6 +138,7 @@ struct af_alg_async_req {
*
* @tsgl_list: Link to TX SGL
* @iv: IV for cipher operation
+ * @ivlen: IV size
* @aead_assoclen: Length of AAD for AEAD cipher operations
* @completion: Work queue for synchronous operation
* @used: TX bytes sent to kernel. This variable is used to
@@ -140,12 +152,14 @@ struct af_alg_async_req {
* SG?
* @enc: Cryptographic operation to be performed when
* recvmsg is invoked.
+ * @iiv: Use inline IV: first part of TX data is IV
* @len: Length of memory allocated for this data structure.
*/
struct af_alg_ctx {
struct list_head tsgl_list;
void *iv;
+ unsigned int ivlen;
size_t aead_assoclen;
struct crypto_wait wait;
@@ -156,6 +170,7 @@ struct af_alg_ctx {
bool more;
bool merge;
bool enc;
+ bool iiv;
unsigned int len;
};
@@ -252,5 +267,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
struct af_alg_async_req *areq, size_t maxsize,
size_t *outlen);
+int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
+struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
#endif /* _CRYPTO_IF_ALG_H */
diff --git a/include/uapi/linux/if_alg.h b/include/uapi/linux/if_alg.h
index bc2bcdec377b..76a4db392cb6 100644
--- a/include/uapi/linux/if_alg.h
+++ b/include/uapi/linux/if_alg.h
@@ -37,7 +37,9 @@ struct af_alg_iv {
#define ALG_SET_AEAD_AUTHSIZE 5
/* Operations */
-#define ALG_OP_DECRYPT 0
-#define ALG_OP_ENCRYPT 1
+#define ALG_OP_DECRYPT 0x0
+#define ALG_OP_ENCRYPT 0x1
+#define ALG_OP_CIPHER_MASK 0xf
+#define ALG_OP_INLINE_IV 0x10
#endif /* _LINUX_IF_ALG_H */
--
2.14.3
Am Freitag, 12. Januar 2018, 14:21:15 CET schrieb Stephan Mueller:
Hi,
>
> 1. Require that the cipher implementations serialize any AIO requests that
> have dependencies. I.e. for CBC, requests need to be serialized by the
> driver. For, say, ECB or XTS no serialization is necessary.
>
> 2. Change AF_ALG to require a per-request IV. This could be implemented by
> moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg
> code path would obtain the IV.
With the released patch, I found a third way that seems to be much less
intrusive: adding inline IV handling where the IV is sent as part of the data
to be processed.
In the libkcapi source code tree, I added a branch [1] that contains support
for the inline IV handling. It also has received numerous tests [2] verifying
that the inline IV kernel patch works.
[1] https://github.com/smuellerDD/libkcapi/tree/iiv
[2] https://github.com/smuellerDD/libkcapi/commit/
f56991ff2f975caf1aa6cb75f2b6fc104cc72f9c
Ciao
Stephan
On Fri, 12 Jan 2018 14:21:15 +0100
Stephan Mueller <[email protected]> wrote:
> Hi,
>
> The kernel crypto API requires the caller to set an IV in the request data
> structure. That request data structure shall define one particular cipher
> operation. During the cipher operation, the IV is read by the cipher
> implementation and eventually the potentially updated IV (e.g. in case of CBC)
> is written back to the memory location the request data structure points to.
Silly question, are we obliged to always write it back? In CBC it is obviously
the same as the last n bytes of the encrypted message. I guess for ease of
handling it makes sense to do so though.
>
> AF_ALG allows setting the IV with a sendmsg request, where the IV is stored in
> the AF_ALG context that is unique to one particular AF_ALG socket. Note the
> analogy: an AF_ALG socket is like a TFM where one recvmsg operation uses one
> request with the TFM from the socket.
>
> AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with one
> recvmsg call, multiple IOVECs can be specified. Each individual IOCB (derived
> from one IOVEC) implies that one request data structure is created with the
> data to be processed by the cipher implementation. The IV that was set with
> the sendmsg call is registered with the request data structure before the
> cipher operation.
>
> In case of an AIO operation, the cipher operation invocation returns
> immediately, queuing the request to the hardware. While the AIO request is
> processed by the hardware, recvmsg processes the next IOVEC for which another
> request is created. Again, the IV buffer from the AF_ALG socket context is
> registered with the new request and the cipher operation is invoked.
>
> You may now see that there is a potential race condition regarding the IV
> handling, because there is *no* separate IV buffer for the different requests.
> This is nicely demonstrated with libkcapi using the following command which
> creates an AIO request with two IOCBs each encrypting one AES block in CBC
> mode:
>
> kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
>
> When the first AIO request finishes before the 2nd AIO request is processed,
> the returned value is:
>
> 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
>
> I.e. two blocks where the IV output from the first request is the IV input to
> the 2nd block.
>
> In case the first AIO request is not completed before the 2nd request
> commences, the result is two identical AES blocks (i.e. both use the same IV):
>
> 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
>
> This inconsistent result may even lead to the conclusion that there can be a
> memory corruption in the IV buffer if both AIO requests write to the IV buffer
> at the same time.
>
> This needs to be solved somehow. I see the following options which I would
> like to have vetted by the community.
>
Taking some 'entirely hypothetical' hardware with the following structure
for all my responses - it's about as flexible as I think we'll see in the
near future - though I'm sure someone has something more complex out there :)
N hardware queues feeding M processing engines in a scheduler driven fashion.
Actually we might have P sets of these, but load balancing and tracking and
transferring contexts between these is a complexity I think we can ignore.
If you want to use more than one of these P you'll just have to handle it
yourself in userspace. Note messages may be shorter than IOCBs which
raises another question I've been meaning to ask. Are all crypto algorithms
obliged to run unlimited length IOCBs?
If there are M messages in a particular queue and none elsewhere it is
capable of processing them all at once (and perhaps returning out of order but
we can fudge them back in order in the driver to avoid that additional
complexity from an interface point of view).
So I'm going to look at this from the hardware point of view - you have
well addressed software management above.
Three ways context management can be handled (in CBC this is basically just
the IV).
1. Each 'work item' queued on a hardware queue has it's IV embedded with the
data. This requires external synchronization if we are chaining across
multiple 'work items' - note the hardware may have restrictions that mean
it has to split large pieces of data up to encrypt them. Not all hardware
may support per 'work item' IVs (I haven't done a survey to find out if
everyone does...)
2. Each queue has a context assigned. We get a new queue whenever we want
to have a different context. Runs out eventually but our hypothetical
hardware may support a lot of queues. Note this version could be 'faked'
by putting a cryptoengine queue on the front of the hardware queues.
3. The hardware supports IV dependency tracking in it's queues. That is,
it can check if the address pointing to the IV is in use by one of the
processing units which has not yet updated the IV ready for chaining with
the next message. Note it might use a magic token rather than the IV
pointer. For modes with out chaining (including counter modes) the IV
pointer will inherently always be different.
The hardware then simply schedules something else until it can safely
run that particular processing unit.
> 1. Require that the cipher implementations serialize any AIO requests that
> have dependencies. I.e. for CBC, requests need to be serialized by the driver.
> For, say, ECB or XTS no serialization is necessary.
There is a certain requirement to do this anyway as we may have a streaming
type situation and we don't want to have to do the chaining in userspace.
So we send first X MB block to HW but before it has come back we have more
data arrive that needs decrypting so we queue that behind it. The IV
then needs to be updated automatically (or the code needs to do it on the
first work item coming back). If you don't have option 3 above, you
have to do this. This is what I was planning to implement for our existing
hardware before you raised this question and I don't think we get around
it being necessary for performance in any case. Setting up IOMMUs etc is
costly so we want to be doing everything we can before the IV update is
ready.
>
> 2. Change AF_ALG to require a per-request IV. This could be implemented by
> moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the recvmsg
> code path would obtain the IV.
>
> I would tend to favor option 2 as this requires code change at only location.
> If option 2 is considered, I would recommend to still allow setting the IV via
> sendmsg CMSG (to keep the interface stable). If, however, the caller provides
> an IV via recvmsg, this takes precedence.
We definitely want to keep option 1 (which runs on the existing interface and
does the magic in driver) for those who want it.
So the only one left is the case 3 above where the hardware is capable
of doing the dependency tracking.
We can support that in two ways but one is rather heavyweight in terms of
resources.
1) Whenever we want to allocate a new context we spin up a new socket and
effectively associate a single IV with that (and it's chained updates) much
like we do in the existing interface.
2) We allow a token based tracking of IVs. So userspace code maintains
a counter and tags ever message and the initial IV setup with that counter.
As the socket typically belongs to a userspace process tag creation can
be in userspace and it can ensure it doesn't overlap tags (or it'll get
the wrong answer).
Kernel driver can then handle making sure any internal token / addresses
are correct. I haven't looked at in depth but would imagine this one
would be rather more invasive to support.
>
> If there are other options, please allow us to learn about them.
>
Glad we are addressing these usecases and that we have AIO support in
general. Makes for a better discussion around whether in kernel support
for these interfaces is actually as effective as moving to userspace
drivers...
Jonathan
> Ciao
> Stephan
>
>
Am Montag, 15. Januar 2018, 12:05:03 CET schrieb Jonathan Cameron:
Hi Jonathan,
> On Fri, 12 Jan 2018 14:21:15 +0100
>
> Stephan Mueller <[email protected]> wrote:
> > Hi,
> >
> > The kernel crypto API requires the caller to set an IV in the request data
> > structure. That request data structure shall define one particular cipher
> > operation. During the cipher operation, the IV is read by the cipher
> > implementation and eventually the potentially updated IV (e.g. in case of
> > CBC) is written back to the memory location the request data structure
> > points to.
> Silly question, are we obliged to always write it back?
Well, in general, yes. The AF_ALG interface should allow a "stream" mode of
operation:
socket
accept
setsockopt(setkey)
sendmsg(IV, data)
recvmsg(data)
sendmsg(data)
recvmsg(data)
..
For such synchronous operation, I guess it is clear that the IV needs to be
written back.
If you want to play with it, use the "stream" API of libkcapi and the
associated test cases.
> In CBC it is
> obviously the same as the last n bytes of the encrypted message. I guess
> for ease of handling it makes sense to do so though.
>
> > AF_ALG allows setting the IV with a sendmsg request, where the IV is
> > stored in the AF_ALG context that is unique to one particular AF_ALG
> > socket. Note the analogy: an AF_ALG socket is like a TFM where one
> > recvmsg operation uses one request with the TFM from the socket.
> >
> > AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with
> > one recvmsg call, multiple IOVECs can be specified. Each individual IOCB
> > (derived from one IOVEC) implies that one request data structure is
> > created with the data to be processed by the cipher implementation. The
> > IV that was set with the sendmsg call is registered with the request data
> > structure before the cipher operation.
> >
> > In case of an AIO operation, the cipher operation invocation returns
> > immediately, queuing the request to the hardware. While the AIO request is
> > processed by the hardware, recvmsg processes the next IOVEC for which
> > another request is created. Again, the IV buffer from the AF_ALG socket
> > context is registered with the new request and the cipher operation is
> > invoked.
> >
> > You may now see that there is a potential race condition regarding the IV
> > handling, because there is *no* separate IV buffer for the different
> > requests. This is nicely demonstrated with libkcapi using the following
> > command which creates an AIO request with two IOCBs each encrypting one
> > AES block in CBC mode:
> >
> > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
> >
> > When the first AIO request finishes before the 2nd AIO request is
> > processed, the returned value is:
> >
> > 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
> >
> > I.e. two blocks where the IV output from the first request is the IV input
> > to the 2nd block.
> >
> > In case the first AIO request is not completed before the 2nd request
> > commences, the result is two identical AES blocks (i.e. both use the same
> > IV):
> >
> > 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
> >
> > This inconsistent result may even lead to the conclusion that there can be
> > a memory corruption in the IV buffer if both AIO requests write to the IV
> > buffer at the same time.
> >
> > This needs to be solved somehow. I see the following options which I would
> > like to have vetted by the community.
>
> Taking some 'entirely hypothetical' hardware with the following structure
> for all my responses - it's about as flexible as I think we'll see in the
> near future - though I'm sure someone has something more complex out there
> :)
>
> N hardware queues feeding M processing engines in a scheduler driven
> fashion. Actually we might have P sets of these, but load balancing and
> tracking and transferring contexts between these is a complexity I think we
> can ignore. If you want to use more than one of these P you'll just have to
> handle it yourself in userspace. Note messages may be shorter than IOCBs
> which raises another question I've been meaning to ask. Are all crypto
> algorithms obliged to run unlimited length IOCBs?
There are instances where hardware may reject large data chunks. IIRC I have
seen some limits around 32k. But in this case, the driver must chunk up the
scatter-gather lists (SGLs) with the data and feed it to the hardware in the
chunk size necessary.
>From the kernel crypto API point of view, the driver must support unlimited
sized IOCBs / SGLs.
>
> If there are M messages in a particular queue and none elsewhere it is
> capable of processing them all at once (and perhaps returning out of order
> but we can fudge them back in order in the driver to avoid that additional
> complexity from an interface point of view).
>
> So I'm going to look at this from the hardware point of view - you have
> well addressed software management above.
>
> Three ways context management can be handled (in CBC this is basically just
> the IV).
>
> 1. Each 'work item' queued on a hardware queue has it's IV embedded with the
> data. This requires external synchronization if we are chaining across
> multiple 'work items' - note the hardware may have restrictions that mean
> it has to split large pieces of data up to encrypt them. Not all hardware
> may support per 'work item' IVs (I haven't done a survey to find out if
> everyone does...)
>
> 2. Each queue has a context assigned. We get a new queue whenever we want
> to have a different context. Runs out eventually but our hypothetical
> hardware may support a lot of queues. Note this version could be 'faked'
> by putting a cryptoengine queue on the front of the hardware queues.
>
> 3. The hardware supports IV dependency tracking in it's queues. That is,
> it can check if the address pointing to the IV is in use by one of the
> processing units which has not yet updated the IV ready for chaining with
> the next message. Note it might use a magic token rather than the IV
> pointer. For modes with out chaining (including counter modes) the IV
> pointer will inherently always be different.
> The hardware then simply schedules something else until it can safely
> run that particular processing unit.
The kernel crypto API has the following concept:
- a TFM holds the data that is stable for an entire cipher operation, such as
the key -- one cipher operation may consist of individual calls
- a request structure holds the volatile data, i.e. the data that is valid for
one particular call, such as the input plaintext or the IV
Thus, your hardware queue should expect one request and it must be capable of
handling that one request with the given data. If you want to split up the
request because you have sufficient hardware resources as you mentioned above,
your driver/hardware must process the request accordingly.
Coming back to the AF_ALG interface: in order to support the aforementioned
"stream" mode, the requests for each cipher call invoked by one recvmsg
syscall points to the same IV buffer.
In case of AIO with multiple IOCBs, user space conceptually calls:
sendmsg
recvmsg
recvmsg
..
where all recvmsg calls execute in parallel. As each recvmsg call has one
request associated with it, the question is what happens to a buffer that is
pointed to by multiple request structures in such parallel execution.
If your hardware is capable of serializing the recvmsg calls or tracking the
dependency, the current AF_ALG interface is fully sufficient.
But there may be hardware that cannot/will not track such dependencies. Yet,
it has multiple hardware queues. Such hardware can still handle parallel
requests when they are totally independent from each other. For such a case,
AF_ALG currently has no support, because it lacks the support for setting
multiple IVs for the multiple concurrent calls.
> > 1. Require that the cipher implementations serialize any AIO requests that
> > have dependencies. I.e. for CBC, requests need to be serialized by the
> > driver. For, say, ECB or XTS no serialization is necessary.
>
> There is a certain requirement to do this anyway as we may have a streaming
> type situation and we don't want to have to do the chaining in userspace.
Absolutely. If you have proper hardware/driver support, that would be
beneficial. This would be supported with the current AF_ALG interface.
But I guess there are also folks out there who simply want to offer multiple
hardware queues to allow independent cipher operations to be invoked
concurrently without any dependency handling. This is currently not supported
with AF_ALG.
>
> So we send first X MB block to HW but before it has come back we have more
> data arrive that needs decrypting so we queue that behind it. The IV
> then needs to be updated automatically (or the code needs to do it on the
> first work item coming back). If you don't have option 3 above, you
> have to do this. This is what I was planning to implement for our existing
> hardware before you raised this question and I don't think we get around
> it being necessary for performance in any case. Setting up IOMMUs etc is
> costly so we want to be doing everything we can before the IV update is
> ready.
>
> > 2. Change AF_ALG to require a per-request IV. This could be implemented by
> > moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the
> > recvmsg
> > code path would obtain the IV.
> >
> > I would tend to favor option 2 as this requires code change at only
> > location. If option 2 is considered, I would recommend to still allow
> > setting the IV via sendmsg CMSG (to keep the interface stable). If,
> > however, the caller provides an IV via recvmsg, this takes precedence.
>
> We definitely want to keep option 1 (which runs on the existing interface
> and does the magic in driver) for those who want it.
Agreed.
>
> So the only one left is the case 3 above where the hardware is capable
> of doing the dependency tracking.
>
> We can support that in two ways but one is rather heavyweight in terms of
> resources.
>
> 1) Whenever we want to allocate a new context we spin up a new socket and
> effectively associate a single IV with that (and it's chained updates) much
> like we do in the existing interface.
I would not like that because it is too heavyweight. Moreover, considering the
kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e.
for setting an IV, you do not need to re-instantiate a TFM.
>
> 2) We allow a token based tracking of IVs. So userspace code maintains
> a counter and tags ever message and the initial IV setup with that counter.
I think the option I offer with the patch, we have an even more lightweight
approach.
>
> As the socket typically belongs to a userspace process tag creation can
> be in userspace and it can ensure it doesn't overlap tags (or it'll get
> the wrong answer).
>
> Kernel driver can then handle making sure any internal token / addresses
> are correct. I haven't looked at in depth but would imagine this one
> would be rather more invasive to support.
>
> > If there are other options, please allow us to learn about them.
>
> Glad we are addressing these usecases and that we have AIO support in
> general. Makes for a better discussion around whether in kernel support
> for these interfaces is actually as effective as moving to userspace
> drivers...
:-)
I would like to have more code in user space than in kernel space...
Ciao
Stephan
On Mon, 15 Jan 2018 13:07:16 +0100
Stephan Mueller <[email protected]> wrote:
> Am Montag, 15. Januar 2018, 12:05:03 CET schrieb Jonathan Cameron:
>
> Hi Jonathan,
>
> > On Fri, 12 Jan 2018 14:21:15 +0100
> >
> > Stephan Mueller <[email protected]> wrote:
> > > Hi,
> > >
> > > The kernel crypto API requires the caller to set an IV in the request data
> > > structure. That request data structure shall define one particular cipher
> > > operation. During the cipher operation, the IV is read by the cipher
> > > implementation and eventually the potentially updated IV (e.g. in case of
> > > CBC) is written back to the memory location the request data structure
> > > points to.
> > Silly question, are we obliged to always write it back?
>
> Well, in general, yes. The AF_ALG interface should allow a "stream" mode of
> operation:
>
> socket
> accept
> setsockopt(setkey)
> sendmsg(IV, data)
> recvmsg(data)
> sendmsg(data)
> recvmsg(data)
> ..
>
> For such synchronous operation, I guess it is clear that the IV needs to be
> written back.
>
> If you want to play with it, use the "stream" API of libkcapi and the
> associated test cases.
Thanks for the pointer - will do.
>
> > In CBC it is
> > obviously the same as the last n bytes of the encrypted message. I guess
> > for ease of handling it makes sense to do so though.
> >
> > > AF_ALG allows setting the IV with a sendmsg request, where the IV is
> > > stored in the AF_ALG context that is unique to one particular AF_ALG
> > > socket. Note the analogy: an AF_ALG socket is like a TFM where one
> > > recvmsg operation uses one request with the TFM from the socket.
> > >
> > > AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with
> > > one recvmsg call, multiple IOVECs can be specified. Each individual IOCB
> > > (derived from one IOVEC) implies that one request data structure is
> > > created with the data to be processed by the cipher implementation. The
> > > IV that was set with the sendmsg call is registered with the request data
> > > structure before the cipher operation.
> > >
> > > In case of an AIO operation, the cipher operation invocation returns
> > > immediately, queuing the request to the hardware. While the AIO request is
> > > processed by the hardware, recvmsg processes the next IOVEC for which
> > > another request is created. Again, the IV buffer from the AF_ALG socket
> > > context is registered with the new request and the cipher operation is
> > > invoked.
> > >
> > > You may now see that there is a potential race condition regarding the IV
> > > handling, because there is *no* separate IV buffer for the different
> > > requests. This is nicely demonstrated with libkcapi using the following
> > > command which creates an AIO request with two IOCBs each encrypting one
> > > AES block in CBC mode:
> > >
> > > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> > > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> > > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
> > >
> > > When the first AIO request finishes before the 2nd AIO request is
> > > processed, the returned value is:
> > >
> > > 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
> > >
> > > I.e. two blocks where the IV output from the first request is the IV input
> > > to the 2nd block.
> > >
> > > In case the first AIO request is not completed before the 2nd request
> > > commences, the result is two identical AES blocks (i.e. both use the same
> > > IV):
> > >
> > > 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
> > >
> > > This inconsistent result may even lead to the conclusion that there can be
> > > a memory corruption in the IV buffer if both AIO requests write to the IV
> > > buffer at the same time.
> > >
> > > This needs to be solved somehow. I see the following options which I would
> > > like to have vetted by the community.
> >
> > Taking some 'entirely hypothetical' hardware with the following structure
> > for all my responses - it's about as flexible as I think we'll see in the
> > near future - though I'm sure someone has something more complex out there
> > :)
> >
> > N hardware queues feeding M processing engines in a scheduler driven
> > fashion. Actually we might have P sets of these, but load balancing and
> > tracking and transferring contexts between these is a complexity I think we
> > can ignore. If you want to use more than one of these P you'll just have to
> > handle it yourself in userspace. Note messages may be shorter than IOCBs
> > which raises another question I've been meaning to ask. Are all crypto
> > algorithms obliged to run unlimited length IOCBs?
>
> There are instances where hardware may reject large data chunks. IIRC I have
> seen some limits around 32k. But in this case, the driver must chunk up the
> scatter-gather lists (SGLs) with the data and feed it to the hardware in the
> chunk size necessary.
>
> From the kernel crypto API point of view, the driver must support unlimited
> sized IOCBs / SGLs.
Hmm. This can be somewhat of a pain to do but fair enough (though the limit
in question for us is a lot more than 32k and covers the vast majority of
real use cases)
> >
> > If there are M messages in a particular queue and none elsewhere it is
> > capable of processing them all at once (and perhaps returning out of order
> > but we can fudge them back in order in the driver to avoid that additional
> > complexity from an interface point of view).
> >
> > So I'm going to look at this from the hardware point of view - you have
> > well addressed software management above.
> >
> > Three ways context management can be handled (in CBC this is basically just
> > the IV).
> >
> > 1. Each 'work item' queued on a hardware queue has it's IV embedded with the
> > data. This requires external synchronization if we are chaining across
> > multiple 'work items' - note the hardware may have restrictions that mean
> > it has to split large pieces of data up to encrypt them. Not all hardware
> > may support per 'work item' IVs (I haven't done a survey to find out if
> > everyone does...)
> >
> > 2. Each queue has a context assigned. We get a new queue whenever we want
> > to have a different context. Runs out eventually but our hypothetical
> > hardware may support a lot of queues. Note this version could be 'faked'
> > by putting a cryptoengine queue on the front of the hardware queues.
> >
> > 3. The hardware supports IV dependency tracking in it's queues. That is,
> > it can check if the address pointing to the IV is in use by one of the
> > processing units which has not yet updated the IV ready for chaining with
> > the next message. Note it might use a magic token rather than the IV
> > pointer. For modes with out chaining (including counter modes) the IV
> > pointer will inherently always be different.
> > The hardware then simply schedules something else until it can safely
> > run that particular processing unit.
>
> The kernel crypto API has the following concept:
>
> - a TFM holds the data that is stable for an entire cipher operation, such as
> the key -- one cipher operation may consist of individual calls
>
> - a request structure holds the volatile data, i.e. the data that is valid for
> one particular call, such as the input plaintext or the IV
>
> Thus, your hardware queue should expect one request and it must be capable of
> handling that one request with the given data. If you want to split up the
> request because you have sufficient hardware resources as you mentioned above,
> your driver/hardware must process the request accordingly.
>
> Coming back to the AF_ALG interface: in order to support the aforementioned
> "stream" mode, the requests for each cipher call invoked by one recvmsg
> syscall points to the same IV buffer.
>
> In case of AIO with multiple IOCBs, user space conceptually calls:
>
> sendmsg
> recvmsg
> recvmsg
> ..
>
> where all recvmsg calls execute in parallel. As each recvmsg call has one
> request associated with it, the question is what happens to a buffer that is
> pointed to by multiple request structures in such parallel execution.
>
> If your hardware is capable of serializing the recvmsg calls or tracking the
> dependency, the current AF_ALG interface is fully sufficient.
>
> But there may be hardware that cannot/will not track such dependencies. Yet,
> it has multiple hardware queues. Such hardware can still handle parallel
> requests when they are totally independent from each other. For such a case,
> AF_ALG currently has no support, because it lacks the support for setting
> multiple IVs for the multiple concurrent calls.
Agreed, something like your new support is needed - I just suspect we need
a level between one socket one iv chain and every IOCB with own IV and
right now the only way to hit that balance is to have a separate socket
for each IV chain. Not exactly efficient use of resources though it will work.
>
> > > 1. Require that the cipher implementations serialize any AIO requests that
> > > have dependencies. I.e. for CBC, requests need to be serialized by the
> > > driver. For, say, ECB or XTS no serialization is necessary.
> >
> > There is a certain requirement to do this anyway as we may have a streaming
> > type situation and we don't want to have to do the chaining in userspace.
>
> Absolutely. If you have proper hardware/driver support, that would be
> beneficial. This would be supported with the current AF_ALG interface.
>
> But I guess there are also folks out there who simply want to offer multiple
> hardware queues to allow independent cipher operations to be invoked
> concurrently without any dependency handling. This is currently not supported
> with AF_ALG.
I'd argue the multiple hardware queues is supported fine in that it is easy to
have a driver supporting them via separate AF_ALG sockets. This is what
we are currently doing. It means that if two processes come along, they can
have their own queues that don't directly influence each other - one isn't
blocked behind another (though they are sharing the processing resources so
will run slower). One process can of course also open multiple sockets though
with your new patch that becomes less important.
The tricky case is the hardware queues where you want to run multiple contexts
through a single AF_ALG socket with iv chaining not effectively done in userspace.
> >
> > So we send first X MB block to HW but before it has come back we have more
> > data arrive that needs decrypting so we queue that behind it. The IV
> > then needs to be updated automatically (or the code needs to do it on the
> > first work item coming back). If you don't have option 3 above, you
> > have to do this. This is what I was planning to implement for our existing
> > hardware before you raised this question and I don't think we get around
> > it being necessary for performance in any case. Setting up IOMMUs etc is
> > costly so we want to be doing everything we can before the IV update is
> > ready.
> >
> > > 2. Change AF_ALG to require a per-request IV. This could be implemented by
> > > moving the IV submission via CMSG from sendmsg to recvmsg. I.e. the
> > > recvmsg
> > > code path would obtain the IV.
> > >
> > > I would tend to favor option 2 as this requires code change at only
> > > location. If option 2 is considered, I would recommend to still allow
> > > setting the IV via sendmsg CMSG (to keep the interface stable). If,
> > > however, the caller provides an IV via recvmsg, this takes precedence.
> >
> > We definitely want to keep option 1 (which runs on the existing interface
> > and does the magic in driver) for those who want it.
>
> Agreed.
> >
> > So the only one left is the case 3 above where the hardware is capable
> > of doing the dependency tracking.
> >
> > We can support that in two ways but one is rather heavyweight in terms of
> > resources.
> >
> > 1) Whenever we want to allocate a new context we spin up a new socket and
> > effectively associate a single IV with that (and it's chained updates) much
> > like we do in the existing interface.
>
> I would not like that because it is too heavyweight. Moreover, considering the
> kernel crypto API logic, a socket is the user space equivalent of a TFM. I.e.
> for setting an IV, you do not need to re-instantiate a TFM.
Agreed, though as I mention above if you have multiple processes you probably
want to give them their own resources anyway (own socket and probably hardware
queue if you can spare one) so as to avoid denial of service from one to another.
> >
> > 2) We allow a token based tracking of IVs. So userspace code maintains
> > a counter and tags ever message and the initial IV setup with that counter.
>
> I think the option I offer with the patch, we have an even more lightweight
> approach.
Except that I think you have to go all the way back to userspace - unless I
am missing the point - you can't have multiple elements of a stream queued up.
Performance will stink if you have a small number of contexts and can't keep
the processing engines busy. At the moment option 1 here is the only way
to implement this.
> >
> > As the socket typically belongs to a userspace process tag creation can
> > be in userspace and it can ensure it doesn't overlap tags (or it'll get
> > the wrong answer).
> >
> > Kernel driver can then handle making sure any internal token / addresses
> > are correct. I haven't looked at in depth but would imagine this one
> > would be rather more invasive to support.
> >
> > > If there are other options, please allow us to learn about them.
> >
> > Glad we are addressing these usecases and that we have AIO support in
> > general. Makes for a better discussion around whether in kernel support
> > for these interfaces is actually as effective as moving to userspace
> > drivers...
>
> :-)
>
> I would like to have more code in user space than in kernel space...
Probably not all of it though, which is basically the alternative here.
Jonathan
>
>
> Ciao
> Stephan
>
>
Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
Hi Jonathan,
> >
> > But there may be hardware that cannot/will not track such dependencies.
> > Yet, it has multiple hardware queues. Such hardware can still handle
> > parallel requests when they are totally independent from each other. For
> > such a case, AF_ALG currently has no support, because it lacks the
> > support for setting multiple IVs for the multiple concurrent calls.
>
> Agreed, something like your new support is needed - I just suspect we need
> a level between one socket one iv chain and every IOCB with own IV and
> right now the only way to hit that balance is to have a separate socket
> for each IV chain. Not exactly efficient use of resources though it will
> work.
How would you propose such support via AF_ALG? Wouldn't it be possible to
arrange the IOVEC array in user space appropriately before calling the AF_ALG
interface? In this case, I would still see that the current AF_ALG (plus the
patch) would support all use cases I am aware of.
> > > So the only one left is the case 3 above where the hardware is capable
> > > of doing the dependency tracking.
> > >
> > > We can support that in two ways but one is rather heavyweight in terms
> > > of
> > > resources.
> > >
> > > 1) Whenever we want to allocate a new context we spin up a new socket
> > > and
> > > effectively associate a single IV with that (and it's chained updates)
> > > much
> > > like we do in the existing interface.
> >
> > I would not like that because it is too heavyweight. Moreover, considering
> > the kernel crypto API logic, a socket is the user space equivalent of a
> > TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
>
> Agreed, though as I mention above if you have multiple processes you
> probably want to give them their own resources anyway (own socket and
> probably hardware queue if you can spare one) so as to avoid denial of
> service from one to another.
That is for sure, different processes shall never share a socket as otherwise
one can obtain data belonging to the other which would violate process
isolation.
> > > 2) We allow a token based tracking of IVs. So userspace code maintains
> > > a counter and tags ever message and the initial IV setup with that
> > > counter.
> >
> > I think the option I offer with the patch, we have an even more
> > lightweight
> > approach.
>
> Except that I think you have to go all the way back to userspace - unless I
> am missing the point - you can't have multiple elements of a stream queued
> up. Performance will stink if you have a small number of contexts and can't
> keep the processing engines busy. At the moment option 1 here is the only
> way to implement this.
What AF_ALG should do is to enable different vendors like yourself to use the
most appropriate solution. AF_ALG shall not limit users in any way.
Thus, AF_ALG allows multiple sockets, if desired. It allows a stream usage
with one setiv call applicable to multiple cipher operations. And with the
offered patch it would allow multiple concurrent and yet independent cipher
operations. Whichever use case is right for you, AF_ALG should not block you
from applying it. Yet, what is good for you may not be good for others. Thus,
these others may implement a different usage strategy for AF_ALG. The good
thing is that this strategy is defined by user space.
In case you see a use case that is prevented by AF_ALG, it would be great to
hear about it to see whether we can support it.
Ciao
Stephan
On Mon, 15 Jan 2018 14:15:42 +0100
Stephan Mueller <[email protected]> wrote:
> Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
>
> Hi Jonathan,
> > >
> > > But there may be hardware that cannot/will not track such dependencies.
> > > Yet, it has multiple hardware queues. Such hardware can still handle
> > > parallel requests when they are totally independent from each other. For
> > > such a case, AF_ALG currently has no support, because it lacks the
> > > support for setting multiple IVs for the multiple concurrent calls.
> >
> > Agreed, something like your new support is needed - I just suspect we need
> > a level between one socket one iv chain and every IOCB with own IV and
> > right now the only way to hit that balance is to have a separate socket
> > for each IV chain. Not exactly efficient use of resources though it will
> > work.
>
> How would you propose such support via AF_ALG?
> Wouldn't it be possible to
> arrange the IOVEC array in user space appropriately before calling the AF_ALG
> interface? In this case, I would still see that the current AF_ALG (plus the
> patch) would support all use cases I am aware of.
I'm not sure how that would work, but maybe I'm missing something - are you
suggesting we could contrive the situation where the kernel side can tell
it is getting the same IV multiple times and hence know that it should chain
it? We are talking streaming here - we don't have the data for the
later elements when the first ones are queued up.
One approach to handling token based IV - where the token refers to an IV without
being it's value would be to add another flag similar to the one you used for
inline IV.
You would then set the IV as you have done, but also provide a magic value by
which to track the chain. Later IOCBs using the same IV chain would just
provide the magic token.
You'd also need some way of retiring the IV eventually once you were done
with it or ultimately you would run out of resources.
>
> > > > So the only one left is the case 3 above where the hardware is capable
> > > > of doing the dependency tracking.
> > > >
> > > > We can support that in two ways but one is rather heavyweight in terms
> > > > of
> > > > resources.
> > > >
> > > > 1) Whenever we want to allocate a new context we spin up a new socket
> > > > and
> > > > effectively associate a single IV with that (and it's chained updates)
> > > > much
> > > > like we do in the existing interface.
> > >
> > > I would not like that because it is too heavyweight. Moreover, considering
> > > the kernel crypto API logic, a socket is the user space equivalent of a
> > > TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
> >
> > Agreed, though as I mention above if you have multiple processes you
> > probably want to give them their own resources anyway (own socket and
> > probably hardware queue if you can spare one) so as to avoid denial of
> > service from one to another.
>
> That is for sure, different processes shall never share a socket as otherwise
> one can obtain data belonging to the other which would violate process
> isolation.
>
> > > > 2) We allow a token based tracking of IVs. So userspace code maintains
> > > > a counter and tags ever message and the initial IV setup with that
> > > > counter.
> > >
> > > I think the option I offer with the patch, we have an even more
> > > lightweight
> > > approach.
> >
> > Except that I think you have to go all the way back to userspace - unless I
> > am missing the point - you can't have multiple elements of a stream queued
> > up. Performance will stink if you have a small number of contexts and can't
> > keep the processing engines busy. At the moment option 1 here is the only
> > way to implement this.
>
> What AF_ALG should do is to enable different vendors like yourself to use the
> most appropriate solution. AF_ALG shall not limit users in any way.
Agreed, but we also need to have some consistency for userspace to have some
awareness of what it should be using. Last thing we want is lots of higher
level software having to have knowledge of the encryption hardware underneath.
Hence I think we should keep the options to the minimum possible or put the
burden on drivers that must play well with all options (be it not as efficiently
for the ones that work badly for them).
>
> Thus, AF_ALG allows multiple sockets, if desired. It allows a stream usage
> with one setiv call applicable to multiple cipher operations. And with the
> offered patch it would allow multiple concurrent and yet independent cipher
> operations. Whichever use case is right for you, AF_ALG should not block you
> from applying it. Yet, what is good for you may not be good for others. Thus,
> these others may implement a different usage strategy for AF_ALG. The good
> thing is that this strategy is defined by user space.
>
> In case you see a use case that is prevented by AF_ALG, it would be great to
> hear about it to see whether we can support it.
>
The usecase isn't blocked, but if you have hardware that is doing the IV
management then it is not efficiently handled. Either
1) You move the chaining up to userspace - throughput on a given chain will
be awful - basically all the advantages of AIO are gone - fine if you know
you only care about bandwidth with lots of separate IV chains.
2) You open a socket per IV chain and eat resources.
Jonathan
> Ciao
> Stephan
>
>
Am Montag, 15. Januar 2018, 15:25:38 CET schrieb Jonathan Cameron:
Hi Jonathan,
> On Mon, 15 Jan 2018 14:15:42 +0100
>
> Stephan Mueller <[email protected]> wrote:
> > Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
> >
> > Hi Jonathan,
> >
> > > > But there may be hardware that cannot/will not track such
> > > > dependencies.
> > > > Yet, it has multiple hardware queues. Such hardware can still handle
> > > > parallel requests when they are totally independent from each other.
> > > > For
> > > > such a case, AF_ALG currently has no support, because it lacks the
> > > > support for setting multiple IVs for the multiple concurrent calls.
> > >
> > > Agreed, something like your new support is needed - I just suspect we
> > > need
> > > a level between one socket one iv chain and every IOCB with own IV and
> > > right now the only way to hit that balance is to have a separate socket
> > > for each IV chain. Not exactly efficient use of resources though it
> > > will
> > > work.
> >
> > How would you propose such support via AF_ALG?
> > Wouldn't it be possible to
> > arrange the IOVEC array in user space appropriately before calling the
> > AF_ALG interface? In this case, I would still see that the current AF_ALG
> > (plus the patch) would support all use cases I am aware of.
>
> I'm not sure how that would work, but maybe I'm missing something - are you
> suggesting we could contrive the situation where the kernel side can tell
> it is getting the same IV multiple times and hence know that it should chain
> it? We are talking streaming here - we don't have the data for the later
> elements when the first ones are queued up.
>
> One approach to handling token based IV - where the token refers to an IV
> without being it's value would be to add another flag similar to the one
> you used for inline IV.
What about:
sendmsg(IV, data)
sendmsg(data)
..
AIO recvmsg with multiple IOCBs
AIO recvmsg with multiple IOCBs
..
sendmsg(IV, data)
..
This implies, however, that before the sendmsg with the second IV is sent, all
AIO operations from the first invocation would need to be finished.
>
> You would then set the IV as you have done, but also provide a magic value
> by which to track the chain. Later IOCBs using the same IV chain would
> just provide the magic token.
>
> You'd also need some way of retiring the IV eventually once you were done
> with it or ultimately you would run out of resources.
Let me think about that approach a bit.
> >
> > What AF_ALG should do is to enable different vendors like yourself to use
> > the most appropriate solution. AF_ALG shall not limit users in any way.
> Agreed, but we also need to have some consistency for userspace to have some
> awareness of what it should be using. Last thing we want is lots of higher
> level software having to have knowledge of the encryption hardware
> underneath. Hence I think we should keep the options to the minimum
> possible or put the burden on drivers that must play well with all options
> (be it not as efficiently for the ones that work badly for them).
>
> > Thus, AF_ALG allows multiple sockets, if desired. It allows a stream usage
> > with one setiv call applicable to multiple cipher operations. And with the
> > offered patch it would allow multiple concurrent and yet independent
> > cipher
> > operations. Whichever use case is right for you, AF_ALG should not block
> > you from applying it. Yet, what is good for you may not be good for
> > others. Thus, these others may implement a different usage strategy for
> > AF_ALG. The good thing is that this strategy is defined by user space.
> >
> > In case you see a use case that is prevented by AF_ALG, it would be great
> > to hear about it to see whether we can support it.
>
> The usecase isn't blocked, but if you have hardware that is doing the IV
> management then it is not efficiently handled. Either
> 1) You move the chaining up to userspace - throughput on a given chain will
> be awful - basically all the advantages of AIO are gone - fine if you
> know you only care about bandwidth with lots of separate IV chains.
This sounds not like the right path.
>
> 2) You open a socket per IV chain and eat resources.
Ok, AF_ALG allows this.
>
> Jonathan
>
> > Ciao
> > Stephan
Ciao
Stephan
On Mon, 15 Jan 2018 14:25:38 +0000
Jonathan Cameron <[email protected]> wrote:
> On Mon, 15 Jan 2018 14:15:42 +0100
> Stephan Mueller <[email protected]> wrote:
>
> > Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
> >
> > Hi Jonathan,
> > > >
> > > > But there may be hardware that cannot/will not track such dependencies.
> > > > Yet, it has multiple hardware queues. Such hardware can still handle
> > > > parallel requests when they are totally independent from each other. For
> > > > such a case, AF_ALG currently has no support, because it lacks the
> > > > support for setting multiple IVs for the multiple concurrent calls.
> > >
> > > Agreed, something like your new support is needed - I just suspect we need
> > > a level between one socket one iv chain and every IOCB with own IV and
> > > right now the only way to hit that balance is to have a separate socket
> > > for each IV chain. Not exactly efficient use of resources though it will
> > > work.
> >
> > How would you propose such support via AF_ALG?
> > Wouldn't it be possible to
> > arrange the IOVEC array in user space appropriately before calling the AF_ALG
> > interface? In this case, I would still see that the current AF_ALG (plus the
> > patch) would support all use cases I am aware of.
>
> I'm not sure how that would work, but maybe I'm missing something - are you
> suggesting we could contrive the situation where the kernel side can tell
> it is getting the same IV multiple times and hence know that it should chain
> it? We are talking streaming here - we don't have the data for the
> later elements when the first ones are queued up.
>
> One approach to handling token based IV - where the token refers to an IV without
> being it's value would be to add another flag similar to the one you used for
> inline IV.
>
> You would then set the IV as you have done, but also provide a magic value by
> which to track the chain. Later IOCBs using the same IV chain would just
> provide the magic token.
>
> You'd also need some way of retiring the IV eventually once you were done
> with it or ultimately you would run out of resources.
>
> >
> > > > > So the only one left is the case 3 above where the hardware is capable
> > > > > of doing the dependency tracking.
> > > > >
> > > > > We can support that in two ways but one is rather heavyweight in terms
> > > > > of
> > > > > resources.
> > > > >
> > > > > 1) Whenever we want to allocate a new context we spin up a new socket
> > > > > and
> > > > > effectively associate a single IV with that (and it's chained updates)
> > > > > much
> > > > > like we do in the existing interface.
> > > >
> > > > I would not like that because it is too heavyweight. Moreover, considering
> > > > the kernel crypto API logic, a socket is the user space equivalent of a
> > > > TFM. I.e. for setting an IV, you do not need to re-instantiate a TFM.
> > >
> > > Agreed, though as I mention above if you have multiple processes you
> > > probably want to give them their own resources anyway (own socket and
> > > probably hardware queue if you can spare one) so as to avoid denial of
> > > service from one to another.
> >
> > That is for sure, different processes shall never share a socket as otherwise
> > one can obtain data belonging to the other which would violate process
> > isolation.
> >
> > > > > 2) We allow a token based tracking of IVs. So userspace code maintains
> > > > > a counter and tags ever message and the initial IV setup with that
> > > > > counter.
> > > >
> > > > I think the option I offer with the patch, we have an even more
> > > > lightweight
> > > > approach.
> > >
> > > Except that I think you have to go all the way back to userspace - unless I
> > > am missing the point - you can't have multiple elements of a stream queued
> > > up. Performance will stink if you have a small number of contexts and can't
> > > keep the processing engines busy. At the moment option 1 here is the only
> > > way to implement this.
> >
> > What AF_ALG should do is to enable different vendors like yourself to use the
> > most appropriate solution. AF_ALG shall not limit users in any way.
>
> Agreed, but we also need to have some consistency for userspace to have some
> awareness of what it should be using. Last thing we want is lots of higher
> level software having to have knowledge of the encryption hardware underneath.
> Hence I think we should keep the options to the minimum possible or put the
> burden on drivers that must play well with all options (be it not as efficiently
> for the ones that work badly for them).
>
> >
> > Thus, AF_ALG allows multiple sockets, if desired. It allows a stream usage
> > with one setiv call applicable to multiple cipher operations. And with the
> > offered patch it would allow multiple concurrent and yet independent cipher
> > operations. Whichever use case is right for you, AF_ALG should not block you
> > from applying it. Yet, what is good for you may not be good for others. Thus,
> > these others may implement a different usage strategy for AF_ALG. The good
> > thing is that this strategy is defined by user space.
> >
> > In case you see a use case that is prevented by AF_ALG, it would be great to
> > hear about it to see whether we can support it.
> >
>
> The usecase isn't blocked, but if you have hardware that is doing the IV
> management then it is not efficiently handled. Either
> 1) You move the chaining up to userspace - throughput on a given chain will
> be awful - basically all the advantages of AIO are gone - fine if you know
> you only care about bandwidth with lots of separate IV chains.
>
> 2) You open a socket per IV chain and eat resources.
Actually thinking more on this its worse, a given driver may need to know
in advance which of these two options it is working with. Is it expected
to do dependency tracking and serialization or not?
In case 1 you would just have a software queue and be able to assume
that whenever you got something back from the hardware you could stick in
the next IOCB and the IV would be right.
In case 2 you would want to push the whole lot to the hardware queue asap.
Given for 2 we are explicitly providing the IV I guess we could use that
info to conclude any chaining was up in userspace but that would only
work if it was very much and either or case which isn't nice.
Jonathan
>
> Jonathan
>
> > Ciao
> > Stephan
> >
> >
>
> _______________________________________________
> Linuxarm mailing list
> [email protected]
> http://hulk.huawei.com/mailman/listinfo/linuxarm
On Mon, 15 Jan 2018 15:31:42 +0100
Stephan Mueller <[email protected]> wrote:
> Am Montag, 15. Januar 2018, 15:25:38 CET schrieb Jonathan Cameron:
>
> Hi Jonathan,
>
> > On Mon, 15 Jan 2018 14:15:42 +0100
> >
> > Stephan Mueller <[email protected]> wrote:
> > > Am Montag, 15. Januar 2018, 13:59:27 CET schrieb Jonathan Cameron:
> > >
> > > Hi Jonathan,
> > >
> > > > > But there may be hardware that cannot/will not track such
> > > > > dependencies.
> > > > > Yet, it has multiple hardware queues. Such hardware can still handle
> > > > > parallel requests when they are totally independent from each other.
> > > > > For
> > > > > such a case, AF_ALG currently has no support, because it lacks the
> > > > > support for setting multiple IVs for the multiple concurrent calls.
> > > >
> > > > Agreed, something like your new support is needed - I just suspect we
> > > > need
> > > > a level between one socket one iv chain and every IOCB with own IV and
> > > > right now the only way to hit that balance is to have a separate socket
> > > > for each IV chain. Not exactly efficient use of resources though it
> > > > will
> > > > work.
> > >
> > > How would you propose such support via AF_ALG?
> > > Wouldn't it be possible to
> > > arrange the IOVEC array in user space appropriately before calling the
> > > AF_ALG interface? In this case, I would still see that the current AF_ALG
> > > (plus the patch) would support all use cases I am aware of.
> >
> > I'm not sure how that would work, but maybe I'm missing something - are you
> > suggesting we could contrive the situation where the kernel side can tell
> > it is getting the same IV multiple times and hence know that it should chain
> > it? We are talking streaming here - we don't have the data for the later
> > elements when the first ones are queued up.
> >
> > One approach to handling token based IV - where the token refers to an IV
> > without being it's value would be to add another flag similar to the one
> > you used for inline IV.
>
> What about:
>
> sendmsg(IV, data)
> sendmsg(data)
> ..
> AIO recvmsg with multiple IOCBs
> AIO recvmsg with multiple IOCBs
> ..
> sendmsg(IV, data)
> ..
>
> This implies, however, that before the sendmsg with the second IV is sent, all
> AIO operations from the first invocation would need to be finished.
Yes that works fine, but rather restricts the flow - you would end up waiting
until you could concatenate a bunch of data in userspace so as to trade
off against the slow down whenever you need to synchronize back up to userspace.
> >
> > You would then set the IV as you have done, but also provide a magic value
> > by which to track the chain. Later IOCBs using the same IV chain would
> > just provide the magic token.
> >
> > You'd also need some way of retiring the IV eventually once you were done
> > with it or ultimately you would run out of resources.
>
> Let me think about that approach a bit.
>
> > >
> > > What AF_ALG should do is to enable different vendors like yourself to use
> > > the most appropriate solution. AF_ALG shall not limit users in any way.
> > Agreed, but we also need to have some consistency for userspace to have some
> > awareness of what it should be using. Last thing we want is lots of higher
> > level software having to have knowledge of the encryption hardware
> > underneath. Hence I think we should keep the options to the minimum
> > possible or put the burden on drivers that must play well with all options
> > (be it not as efficiently for the ones that work badly for them).
> >
> > > Thus, AF_ALG allows multiple sockets, if desired. It allows a stream usage
> > > with one setiv call applicable to multiple cipher operations. And with the
> > > offered patch it would allow multiple concurrent and yet independent
> > > cipher
> > > operations. Whichever use case is right for you, AF_ALG should not block
> > > you from applying it. Yet, what is good for you may not be good for
> > > others. Thus, these others may implement a different usage strategy for
> > > AF_ALG. The good thing is that this strategy is defined by user space.
> > >
> > > In case you see a use case that is prevented by AF_ALG, it would be great
> > > to hear about it to see whether we can support it.
> >
> > The usecase isn't blocked, but if you have hardware that is doing the IV
> > management then it is not efficiently handled. Either
> > 1) You move the chaining up to userspace - throughput on a given chain will
> > be awful - basically all the advantages of AIO are gone - fine if you
> > know you only care about bandwidth with lots of separate IV chains.
>
> This sounds not like the right path.
> >
> > 2) You open a socket per IV chain and eat resources.
>
> Ok, AF_ALG allows this.
That was my plan before this discussion started. Ugly but works without
any AF_ALG changes.
We can probably play some internal games to make this not as bad as it
initially seems, but still not nice.
Jonathan
> >
> > Jonathan
> >
> > > Ciao
> > > Stephan
>
>
>
> Ciao
> Stephan
>
>
Am Montag, 15. Januar 2018, 15:42:58 CET schrieb Jonathan Cameron:
Hi Jonathan,
> > What about:
> >
> > sendmsg(IV, data)
> > sendmsg(data)
> > ..
> > AIO recvmsg with multiple IOCBs
> > AIO recvmsg with multiple IOCBs
> > ..
> > sendmsg(IV, data)
> > ..
> >
> > This implies, however, that before the sendmsg with the second IV is sent,
> > all AIO operations from the first invocation would need to be finished.
> Yes that works fine, but rather restricts the flow - you would end up
> waiting until you could concatenate a bunch of data in userspace so as to
> trade off against the slow down whenever you need to synchronize back up to
> userspace.
I think the solution is already present and even libkcapi's architecture is
set up to handle this scenario:
We have 2 types of FDs: one obtained from socket() and one from accept(). The
socket-FD is akin to the TFM. The accept FD is exactly what you want:
tfmfd = socket()
setkey(tfmfd)
opfd = accept()
opfd2 = accept()
sendmsg(opfd, IV, data)
recvmsg(opfd, data)
sendmsg(opfd2, IV, data)
recvmsg(opfd2, data)
sendmsg(opfd, data)
..
There can be multipe FDs from accept and these are the "identifiers" for your
cipher operation stream that belongs together.
libkcapi has already the architecture for this type of work, but it is not
exposed to the API yet. The internal calls for sendmsg/recvmsg all take an
(op)FD parameter. E.g. _kcapi_common_send_meta_fd has the fdptr variable.
internal.h currently wraps this call into _kcapi_common_send_meta where the
handle->opfd variable is used.
The idea why I implemented that is because the caller could maintain an array
of opfds. If we would expose these internal calls with the FD argument, you
can maintain multiple opfds implementing your use case.
The only change would be to expose the internal libkcapi calls.
Ciao
Stephan
On Tue, 16 Jan 2018 07:28:06 +0100
Stephan Mueller <[email protected]> wrote:
> Am Montag, 15. Januar 2018, 15:42:58 CET schrieb Jonathan Cameron:
>
> Hi Jonathan,
>
> > > What about:
> > >
> > > sendmsg(IV, data)
> > > sendmsg(data)
> > > ..
> > > AIO recvmsg with multiple IOCBs
> > > AIO recvmsg with multiple IOCBs
> > > ..
> > > sendmsg(IV, data)
> > > ..
> > >
> > > This implies, however, that before the sendmsg with the second IV is sent,
> > > all AIO operations from the first invocation would need to be finished.
> > Yes that works fine, but rather restricts the flow - you would end up
> > waiting until you could concatenate a bunch of data in userspace so as to
> > trade off against the slow down whenever you need to synchronize back up to
> > userspace.
>
> I think the solution is already present and even libkcapi's architecture is
> set up to handle this scenario:
>
> We have 2 types of FDs: one obtained from socket() and one from accept(). The
> socket-FD is akin to the TFM. The accept FD is exactly what you want:
>
> tfmfd = socket()
> setkey(tfmfd)
> opfd = accept()
> opfd2 = accept()
> sendmsg(opfd, IV, data)
> recvmsg(opfd, data)
>
> sendmsg(opfd2, IV, data)
> recvmsg(opfd2, data)
>
> sendmsg(opfd, data)
> ..
>
> There can be multipe FDs from accept and these are the "identifiers" for your
> cipher operation stream that belongs together.
>
> libkcapi has already the architecture for this type of work, but it is not
> exposed to the API yet. The internal calls for sendmsg/recvmsg all take an
> (op)FD parameter. E.g. _kcapi_common_send_meta_fd has the fdptr variable.
> internal.h currently wraps this call into _kcapi_common_send_meta where the
> handle->opfd variable is used.
>
> The idea why I implemented that is because the caller could maintain an array
> of opfds. If we would expose these internal calls with the FD argument, you
> can maintain multiple opfds implementing your use case.
>
> The only change would be to expose the internal libkcapi calls.
>
> Ciao
> Stephan
Thanks, I'll take a look at this soonish. Having a busy week.
Jonathan
>
>
Hi Herbert,
I tried to summarize the use cases of the AIO support at [1].
The use case covering the inline IV support is documented in section [2]. It
naturally would depend on this patch to be accepted. What is your take on this
use case?
What is your take on the issue outlined at [3]? Should the affected drivers be
updated?
Thanks
Stephan
[1] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11
[2] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11#diff-
dce7a00cbc610df94f0dc27eb769d01dR644
[3] https://github.com/smuellerDD/libkcapi/commit/
29132075b8f045f3f92367c0190add81ccf5da11#diff-
dce7a00cbc610df94f0dc27eb769d01dR575
On Mon, 15 Jan 2018 10:35:34 +0100
Stephan Mueller <[email protected]> wrote:
> The kernel crypto API requires the caller to set an IV in the request
> data structure. That request data structure shall define one particular
> cipher operation. During the cipher operation, the IV is read by the
> cipher implementation and eventually the potentially updated IV (e.g.
> in case of CBC) is written back to the memory location the request data
> structure points to.
>
> AF_ALG allows setting the IV with a sendmsg request, where the IV is
> stored in the AF_ALG context that is unique to one particular AF_ALG
> socket. Note the analogy: an AF_ALG socket is like a TFM where one
> recvmsg operation uses one request with the TFM from the socket.
>
> AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
> with one recvmsg call, multiple IOVECs can be specified. Each
> individual IOCB (derived from one IOVEC) implies that one request data
> structure is created with the data to be processed by the cipher
> implementation. The IV that was set with the sendmsg call is registered
> with the request data structure before the cipher operation.
>
> In case of an AIO operation, the cipher operation invocation returns
> immediately, queuing the request to the hardware. While the AIO request
> is processed by the hardware, recvmsg processes the next IOVEC for
> which another request is created. Again, the IV buffer from the AF_ALG
> socket context is registered with the new request and the cipher
> operation is invoked.
>
> You may now see that there is a potential race condition regarding the
> IV handling, because there is *no* separate IV buffer for the different
> requests. This is nicely demonstrated with libkcapi using the following
> command which creates an AIO request with two IOCBs each encrypting one
> AES block in CBC mode:
>
> kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
>
> When the first AIO request finishes before the 2nd AIO request is
> processed, the returned value is:
>
> 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
>
> I.e. two blocks where the IV output from the first request is the IV input
> to the 2nd block.
>
> In case the first AIO request is not completed before the 2nd request
> commences, the result is two identical AES blocks (i.e. both use the
> same IV):
>
> 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
>
> This inconsistent result may even lead to the conclusion that there can
> be a memory corruption in the IV buffer if both AIO requests write to
> the IV buffer at the same time.
>
> The solution is to allow providing the IV data supplied as part of the
> plaintext/ciphertext. To do so, the AF_ALG interface treats the
> ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
> cipher operation together with the flag whether the operation should
> enable support for inline IV handling.
>
> If inline IV handling is enabled, the IV is expected to be the first
> part of the input plaintext/ciphertext. This IV is only used for one
> cipher operation and will not retained in the kernel for subsequent
> cipher operations.
>
> The AEAD support required a slight re-arragning of the code, because
> obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
> access in _aead_recvmsg must be moved below the IV gathering.
>
> The AEAD code to find the first SG with data in the TX SGL is moved to a
> common function as it is required by the IV gathering function as well.
>
> This patch does not change the existing interface where user space is
> allowed to provide an IV via sendmsg. It only extends the interface by
> giving the user the choice to provide the IV either via sendmsg (the
> current approach) or as part of the data (the additional approach).
>
> Signed-off-by: Stephan Mueller <[email protected]>
Firstly it works
Tested-by: Jonathan Cameron <[email protected]>
Now to be really useful for my particular hardware (which doesn't do dependency
tracking in its out of order queues) what I need to know is whether I need to
let previous queued up entries finish before I can run this one or not.
Now there is an obvious 'hack' I can use, which is that the above only matters
is if the IV is still 'in flight'. Given it is still inflight the memory is
in use. Thus until it is finished it's address is clearly only being used by
'this IV'.
Hence I can just compare against the address of the IV for the previous packet.
If it changed we know there is no need to wait for previous packets to finish.
I'll still have to be a little careful to avoid DoS issues if we flip from
one mode to the other but I can do that by ensuring my software queue is empty
before pushing to the hardware queue.
Anyhow using the address does seem a little fragile, any suggestions for a
better way?
On the plus side, with 8k iovs I'm getting around a 50% performance boost by
not having to serialize the entry into the hardware queues. From a few back
of the envelope calculations the key thing is that I can coalesce interrupts.
For very small iovs the overhead of setting them us is enough to mean we
rarely end up with many in the queue at a time so the benefits aren't
seen.
On that note I have a hacked up libkcapi speed test that adds a parameter to
control this. I'll send you a pull request later so you can take a look.
I have only done the skcipher case though and I am sure it could be done
in a neater fashion!
Thanks,
Jonathan
> ---
> crypto/af_alg.c | 79 ++++++++++++++++++++++++++++++++++++++++++++-
> crypto/algif_aead.c | 54 ++++++++++++++-----------------
> crypto/algif_skcipher.c | 14 +++++---
> include/crypto/if_alg.h | 17 ++++++++++
> include/uapi/linux/if_alg.h | 6 ++--
> 5 files changed, 133 insertions(+), 37 deletions(-)
>
> diff --git a/crypto/af_alg.c b/crypto/af_alg.c
> index 5231f421ad00..87394dc90ac0 100644
> --- a/crypto/af_alg.c
> +++ b/crypto/af_alg.c
> @@ -14,6 +14,7 @@
>
> #include <linux/atomic.h>
> #include <crypto/if_alg.h>
> +#include <crypto/scatterwalk.h>
> #include <linux/crypto.h>
> #include <linux/init.h>
> #include <linux/kernel.h>
> @@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
> struct af_alg_control con = {};
> long copied = 0;
> bool enc = 0;
> + bool iiv = 0;
> bool init = 0;
> int err = 0;
>
> @@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
> return err;
>
> init = 1;
> - switch (con.op) {
> + switch (con.op & ALG_OP_CIPHER_MASK) {
> case ALG_OP_ENCRYPT:
> enc = 1;
> break;
> @@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
> return -EINVAL;
> }
>
> + if (con.op & ALG_OP_INLINE_IV)
> + iiv = 1;
> +
> if (con.iv && con.iv->ivlen != ivsize)
> return -EINVAL;
> }
> @@ -866,6 +871,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
>
> if (init) {
> ctx->enc = enc;
> + ctx->iiv = iiv;
> if (con.iv)
> memcpy(ctx->iv, con.iv->iv, ivsize);
>
> @@ -1031,6 +1037,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)
> struct sock *sk = areq->sk;
>
> af_alg_free_areq_sgls(areq);
> + if (areq->ivlen)
> + sock_kfree_s(sk, areq->iv, areq->ivlen);
> sock_kfree_s(sk, areq, areq->areqlen);
> }
> EXPORT_SYMBOL_GPL(af_alg_free_resources);
> @@ -1175,6 +1183,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> }
> EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
>
> +/**
> + * af_alg_get_tsgl_sg - get first SG entry with data from TX SGL
> + * @ctx [in] AF_ALG context with TX sgl
> + * @return pointer to SG with data or NULL if none found
> + */
> +struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx)
> +{
> + struct af_alg_tsgl *sgl, *tmp;
> + struct scatterlist *sg = NULL;
> + unsigned int i;
> +
> + list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
> + for (i = 0; i < sgl->cur; i++) {
> + struct scatterlist *process_sg = sgl->sg + i;
> +
> + if (!(process_sg->length) || !sg_page(process_sg))
> + continue;
> + sg = process_sg;
> + break;
> + }
> + if (sg)
> + break;
> + }
> +
> + return sg;
> +}
> +EXPORT_SYMBOL_GPL(af_alg_get_tsgl_sg);
> +
> +/**
> + * af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX
> + *
> + * @sk [in] AF_ALG socket
> + * @areq [in/out] request buffer that will receive the IV
> + * to be used for the cipher
> + * @return 0 on success, < 0 on error
> + */
> +int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
> +{
> + struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> + struct scatterlist *sg;
> +
> + /* No inline IV or cipher has no IV, use ctx IV buffer. */
> + if (!ctx->iiv || !ctx->ivlen) {
> + areq->iv = ctx->iv;
> + areq->ivlen = 0; // areq->iv will not be freed
> + return 0;
> + }
> +
> + /* There must be the IV data present. */
> + if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
> + return -EINVAL;
> +
> + areq->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
> + if (unlikely(!areq->iv))
> + return -ENOMEM;
> + areq->ivlen = ctx->ivlen; // areq->iv will be freed
> +
> + /* Get ivlen data from TX SGL and copy it into areq->iv. */
> + sg = af_alg_get_tsgl_sg(ctx);
> + if (!sg)
> + return -EFAULT;
> + scatterwalk_map_and_copy(areq->iv, sg, 0, ctx->ivlen, 0);
> + af_alg_pull_tsgl(sk, ctx->ivlen, NULL, 0);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(af_alg_get_iv);
> +
> static int __init af_alg_init(void)
> {
> int err = proto_register(&alg_proto, 0);
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 4b07edd5a9ff..7eb7cb132c09 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> struct aead_tfm *aeadc = pask->private;
> struct crypto_aead *tfm = aeadc->aead;
> struct crypto_skcipher *null_tfm = aeadc->null_tfm;
> - unsigned int i, as = crypto_aead_authsize(tfm);
> + unsigned int as = crypto_aead_authsize(tfm);
> struct af_alg_async_req *areq;
> - struct af_alg_tsgl *tsgl, *tmp;
> struct scatterlist *rsgl_src, *tsgl_src = NULL;
> int err = 0;
> size_t used = 0; /* [in] TX bufs to be en/decrypted */
> @@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> return err;
> }
>
> - /*
> - * Data length provided by caller via sendmsg/sendpage that has not
> - * yet been processed.
> - */
> - used = ctx->used;
> -
> /*
> * Make sure sufficient data is present -- note, the same check is
> * is also present in sendmsg/sendpage. The checks in sendpage/sendmsg
> @@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> if (!aead_sufficient_data(sk))
> return -EINVAL;
>
> + /* Allocate cipher request for current operation. */
> + areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
> + crypto_aead_reqsize(tfm));
> + if (IS_ERR(areq))
> + return PTR_ERR(areq);
> +
> + /* Set areq->iv. */
> + err = af_alg_get_iv(sk, areq);
> + if (err)
> + goto free;
> +
> + /*
> + * Data length provided by caller via sendmsg/sendpage that has not
> + * yet been processed.
> + */
> + used = ctx->used;
> +
> /*
> * Calculate the minimum output buffer size holding the result of the
> * cipher operation. When encrypting data, the receiving buffer is
> @@ -153,12 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> */
> used -= ctx->aead_assoclen;
>
> - /* Allocate cipher request for current operation. */
> - areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
> - crypto_aead_reqsize(tfm));
> - if (IS_ERR(areq))
> - return PTR_ERR(areq);
> -
> /* convert iovecs of output buffers into RX SGL */
> err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
> if (err)
> @@ -183,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> }
>
> processed = used + ctx->aead_assoclen;
> - list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) {
> - for (i = 0; i < tsgl->cur; i++) {
> - struct scatterlist *process_sg = tsgl->sg + i;
> -
> - if (!(process_sg->length) || !sg_page(process_sg))
> - continue;
> - tsgl_src = process_sg;
> - break;
> - }
> - if (tsgl_src)
> - break;
> - }
> + tsgl_src = af_alg_get_tsgl_sg(ctx);
> if (processed && !tsgl_src) {
> err = -EFAULT;
> goto free;
> @@ -282,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
>
> /* Initialize the crypto operation */
> aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src,
> - areq->first_rsgl.sgl.sg, used, ctx->iv);
> + areq->first_rsgl.sgl.sg, used, areq->iv);
> aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen);
> aead_request_set_tfm(&areq->cra_u.aead_req, tfm);
>
> @@ -549,19 +542,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
> struct aead_tfm *tfm = private;
> struct crypto_aead *aead = tfm->aead;
> unsigned int len = sizeof(*ctx);
> - unsigned int ivlen = crypto_aead_ivsize(aead);
>
> ctx = sock_kmalloc(sk, len, GFP_KERNEL);
> if (!ctx)
> return -ENOMEM;
> memset(ctx, 0, len);
>
> - ctx->iv = sock_kmalloc(sk, ivlen, GFP_KERNEL);
> + ctx->ivlen = crypto_aead_ivsize(aead);
> + ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
> if (!ctx->iv) {
> sock_kfree_s(sk, ctx, len);
> return -ENOMEM;
> }
> - memset(ctx->iv, 0, ivlen);
> + memset(ctx->iv, 0, ctx->ivlen);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> ctx->len = len;
> @@ -570,6 +563,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
> ctx->more = 0;
> ctx->merge = 0;
> ctx->enc = 0;
> + ctx->iiv = 0;
> ctx->aead_assoclen = 0;
> crypto_init_wait(&ctx->wait);
>
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index c88e5e4cd6a6..d40e1d6797d8 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -77,6 +77,11 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> if (IS_ERR(areq))
> return PTR_ERR(areq);
>
> + /* Set areq->iv */
> + err = af_alg_get_iv(sk, areq);
> + if (err)
> + goto free;
> +
> /* convert iovecs of output buffers into RX SGL */
> err = af_alg_get_rsgl(sk, msg, flags, areq, -1, &len);
> if (err)
> @@ -112,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> /* Initialize the crypto operation */
> skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm);
> skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl,
> - areq->first_rsgl.sgl.sg, len, ctx->iv);
> + areq->first_rsgl.sgl.sg, len, areq->iv);
>
> if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
> /* AIO operation */
> @@ -345,14 +350,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
> if (!ctx)
> return -ENOMEM;
>
> - ctx->iv = sock_kmalloc(sk, crypto_skcipher_ivsize(tfm),
> - GFP_KERNEL);
> + ctx->ivlen = crypto_skcipher_ivsize(tfm);
> + ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
> if (!ctx->iv) {
> sock_kfree_s(sk, ctx, len);
> return -ENOMEM;
> }
>
> - memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
> + memset(ctx->iv, 0, ctx->ivlen);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> ctx->len = len;
> @@ -361,6 +366,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
> ctx->more = 0;
> ctx->merge = 0;
> ctx->enc = 0;
> + ctx->iiv = 0;
> crypto_init_wait(&ctx->wait);
>
> ask->private = ctx;
> diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
> index f38227a78eae..ebc651ceb54a 100644
> --- a/include/crypto/if_alg.h
> +++ b/include/crypto/if_alg.h
> @@ -95,6 +95,14 @@ struct af_alg_rsgl {
> * @tsgl_entries: Number of entries in priv. TX SGL
> * @outlen: Number of output bytes generated by crypto op
> * @areqlen: Length of this data structure
> + * @iv: Buffer holding the IV for the cipher operation (this
> + * is either ctx->iv where ivlen is set to 0 or a newly
> + * allocated buffer where ivlen represents the size of
> + * this buffer).
> + * @ivlen: IV size -- if the IV buffer is to be freed during
> + * releasing of this data structure, ivlen is non-zero.
> + * If ivlen is zero, but iv is non-NULL, the iv buffer is
> + * not freed.
> * @cra_u: Cipher request
> */
> struct af_alg_async_req {
> @@ -111,6 +119,9 @@ struct af_alg_async_req {
> unsigned int outlen;
> unsigned int areqlen;
>
> + void *iv;
> + unsigned int ivlen;
> +
> union {
> struct aead_request aead_req;
> struct skcipher_request skcipher_req;
> @@ -127,6 +138,7 @@ struct af_alg_async_req {
> *
> * @tsgl_list: Link to TX SGL
> * @iv: IV for cipher operation
> + * @ivlen: IV size
> * @aead_assoclen: Length of AAD for AEAD cipher operations
> * @completion: Work queue for synchronous operation
> * @used: TX bytes sent to kernel. This variable is used to
> @@ -140,12 +152,14 @@ struct af_alg_async_req {
> * SG?
> * @enc: Cryptographic operation to be performed when
> * recvmsg is invoked.
> + * @iiv: Use inline IV: first part of TX data is IV
> * @len: Length of memory allocated for this data structure.
> */
> struct af_alg_ctx {
> struct list_head tsgl_list;
>
> void *iv;
> + unsigned int ivlen;
> size_t aead_assoclen;
>
> struct crypto_wait wait;
> @@ -156,6 +170,7 @@ struct af_alg_ctx {
> bool more;
> bool merge;
> bool enc;
> + bool iiv;
>
> unsigned int len;
> };
> @@ -252,5 +267,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
> int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> struct af_alg_async_req *areq, size_t maxsize,
> size_t *outlen);
> +int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
> +struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
>
> #endif /* _CRYPTO_IF_ALG_H */
> diff --git a/include/uapi/linux/if_alg.h b/include/uapi/linux/if_alg.h
> index bc2bcdec377b..76a4db392cb6 100644
> --- a/include/uapi/linux/if_alg.h
> +++ b/include/uapi/linux/if_alg.h
> @@ -37,7 +37,9 @@ struct af_alg_iv {
> #define ALG_SET_AEAD_AUTHSIZE 5
>
> /* Operations */
> -#define ALG_OP_DECRYPT 0
> -#define ALG_OP_ENCRYPT 1
> +#define ALG_OP_DECRYPT 0x0
> +#define ALG_OP_ENCRYPT 0x1
> +#define ALG_OP_CIPHER_MASK 0xf
> +#define ALG_OP_INLINE_IV 0x10
>
> #endif /* _LINUX_IF_ALG_H */
Am Montag, 22. Januar 2018, 15:11:53 CET schrieb Jonathan Cameron:
Hi Jonathan,
> On Mon, 15 Jan 2018 10:35:34 +0100
>
> Stephan Mueller <[email protected]> wrote:
> > The kernel crypto API requires the caller to set an IV in the request
> > data structure. That request data structure shall define one particular
> > cipher operation. During the cipher operation, the IV is read by the
> > cipher implementation and eventually the potentially updated IV (e.g.
> > in case of CBC) is written back to the memory location the request data
> > structure points to.
> >
> > AF_ALG allows setting the IV with a sendmsg request, where the IV is
> > stored in the AF_ALG context that is unique to one particular AF_ALG
> > socket. Note the analogy: an AF_ALG socket is like a TFM where one
> > recvmsg operation uses one request with the TFM from the socket.
> >
> > AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
> > with one recvmsg call, multiple IOVECs can be specified. Each
> > individual IOCB (derived from one IOVEC) implies that one request data
> > structure is created with the data to be processed by the cipher
> > implementation. The IV that was set with the sendmsg call is registered
> > with the request data structure before the cipher operation.
> >
> > In case of an AIO operation, the cipher operation invocation returns
> > immediately, queuing the request to the hardware. While the AIO request
> > is processed by the hardware, recvmsg processes the next IOVEC for
> > which another request is created. Again, the IV buffer from the AF_ALG
> > socket context is registered with the new request and the cipher
> > operation is invoked.
> >
> > You may now see that there is a potential race condition regarding the
> > IV handling, because there is *no* separate IV buffer for the different
> > requests. This is nicely demonstrated with libkcapi using the following
> > command which creates an AIO request with two IOCBs each encrypting one
> > AES block in CBC mode:
> >
> > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
> >
> > When the first AIO request finishes before the 2nd AIO request is
> > processed, the returned value is:
> >
> > 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
> >
> > I.e. two blocks where the IV output from the first request is the IV input
> > to the 2nd block.
> >
> > In case the first AIO request is not completed before the 2nd request
> > commences, the result is two identical AES blocks (i.e. both use the
> > same IV):
> >
> > 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
> >
> > This inconsistent result may even lead to the conclusion that there can
> > be a memory corruption in the IV buffer if both AIO requests write to
> > the IV buffer at the same time.
> >
> > The solution is to allow providing the IV data supplied as part of the
> > plaintext/ciphertext. To do so, the AF_ALG interface treats the
> > ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
> > cipher operation together with the flag whether the operation should
> > enable support for inline IV handling.
> >
> > If inline IV handling is enabled, the IV is expected to be the first
> > part of the input plaintext/ciphertext. This IV is only used for one
> > cipher operation and will not retained in the kernel for subsequent
> > cipher operations.
> >
> > The AEAD support required a slight re-arragning of the code, because
> > obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
> > access in _aead_recvmsg must be moved below the IV gathering.
> >
> > The AEAD code to find the first SG with data in the TX SGL is moved to a
> > common function as it is required by the IV gathering function as well.
> >
> > This patch does not change the existing interface where user space is
> > allowed to provide an IV via sendmsg. It only extends the interface by
> > giving the user the choice to provide the IV either via sendmsg (the
> > current approach) or as part of the data (the additional approach).
> >
> > Signed-off-by: Stephan Mueller <[email protected]>
>
> Firstly it works
> Tested-by: Jonathan Cameron <[email protected]>
Thank you.
>
> Now to be really useful for my particular hardware (which doesn't do
> dependency tracking in its out of order queues) what I need to know is
> whether I need to let previous queued up entries finish before I can run
> this one or not.
>
> Now there is an obvious 'hack' I can use, which is that the above only
> matters is if the IV is still 'in flight'. Given it is still inflight the
> memory is in use. Thus until it is finished it's address is clearly only
> being used by 'this IV'.
>
> Hence I can just compare against the address of the IV for the previous
> packet. If it changed we know there is no need to wait for previous packets
> to finish.
That is dangerous, because the IV is in a malloced location. It is possible
that a new malloc will return the same address.
What about simply applying the cipher operation as "one shot". I.e. one
invocation of a cipher operation is isolated from the next. Thus, queue one
request defined with one request data structure and handle each request
independent from the next.
>
> I'll still have to be a little careful to avoid DoS issues if we flip from
> one mode to the other but I can do that by ensuring my software queue is
> empty before pushing to the hardware queue.
>
> Anyhow using the address does seem a little fragile, any suggestions for a
> better way?
>
> On the plus side, with 8k iovs I'm getting around a 50% performance boost by
> not having to serialize the entry into the hardware queues. From a few back
> of the envelope calculations the key thing is that I can coalesce
> interrupts.
>
> For very small iovs the overhead of setting them us is enough to mean we
> rarely end up with many in the queue at a time so the benefits aren't
> seen.
>
> On that note I have a hacked up libkcapi speed test that adds a parameter to
> control this. I'll send you a pull request later so you can take a look. I
> have only done the skcipher case though and I am sure it could be done in a
> neater fashion!
Thank you.
Ciao
Stephan
On Mon, 22 Jan 2018 15:30:39 +0100
Stephan Mueller <[email protected]> wrote:
> Am Montag, 22. Januar 2018, 15:11:53 CET schrieb Jonathan Cameron:
>
> Hi Jonathan,
Hi Stephan,
>
> > On Mon, 15 Jan 2018 10:35:34 +0100
> >
> > Stephan Mueller <[email protected]> wrote:
> > > The kernel crypto API requires the caller to set an IV in the request
> > > data structure. That request data structure shall define one particular
> > > cipher operation. During the cipher operation, the IV is read by the
> > > cipher implementation and eventually the potentially updated IV (e.g.
> > > in case of CBC) is written back to the memory location the request data
> > > structure points to.
> > >
> > > AF_ALG allows setting the IV with a sendmsg request, where the IV is
> > > stored in the AF_ALG context that is unique to one particular AF_ALG
> > > socket. Note the analogy: an AF_ALG socket is like a TFM where one
> > > recvmsg operation uses one request with the TFM from the socket.
> > >
> > > AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
> > > with one recvmsg call, multiple IOVECs can be specified. Each
> > > individual IOCB (derived from one IOVEC) implies that one request data
> > > structure is created with the data to be processed by the cipher
> > > implementation. The IV that was set with the sendmsg call is registered
> > > with the request data structure before the cipher operation.
> > >
> > > In case of an AIO operation, the cipher operation invocation returns
> > > immediately, queuing the request to the hardware. While the AIO request
> > > is processed by the hardware, recvmsg processes the next IOVEC for
> > > which another request is created. Again, the IV buffer from the AF_ALG
> > > socket context is registered with the new request and the cipher
> > > operation is invoked.
> > >
> > > You may now see that there is a potential race condition regarding the
> > > IV handling, because there is *no* separate IV buffer for the different
> > > requests. This is nicely demonstrated with libkcapi using the following
> > > command which creates an AIO request with two IOCBs each encrypting one
> > > AES block in CBC mode:
> > >
> > > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> > > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> > > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
> > >
> > > When the first AIO request finishes before the 2nd AIO request is
> > > processed, the returned value is:
> > >
> > > 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
> > >
> > > I.e. two blocks where the IV output from the first request is the IV input
> > > to the 2nd block.
> > >
> > > In case the first AIO request is not completed before the 2nd request
> > > commences, the result is two identical AES blocks (i.e. both use the
> > > same IV):
> > >
> > > 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
> > >
> > > This inconsistent result may even lead to the conclusion that there can
> > > be a memory corruption in the IV buffer if both AIO requests write to
> > > the IV buffer at the same time.
> > >
> > > The solution is to allow providing the IV data supplied as part of the
> > > plaintext/ciphertext. To do so, the AF_ALG interface treats the
> > > ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
> > > cipher operation together with the flag whether the operation should
> > > enable support for inline IV handling.
> > >
> > > If inline IV handling is enabled, the IV is expected to be the first
> > > part of the input plaintext/ciphertext. This IV is only used for one
> > > cipher operation and will not retained in the kernel for subsequent
> > > cipher operations.
> > >
> > > The AEAD support required a slight re-arragning of the code, because
> > > obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
> > > access in _aead_recvmsg must be moved below the IV gathering.
> > >
> > > The AEAD code to find the first SG with data in the TX SGL is moved to a
> > > common function as it is required by the IV gathering function as well.
> > >
> > > This patch does not change the existing interface where user space is
> > > allowed to provide an IV via sendmsg. It only extends the interface by
> > > giving the user the choice to provide the IV either via sendmsg (the
> > > current approach) or as part of the data (the additional approach).
> > >
> > > Signed-off-by: Stephan Mueller <[email protected]>
> >
> > Firstly it works
> > Tested-by: Jonathan Cameron <[email protected]>
>
> Thank you.
> >
> > Now to be really useful for my particular hardware (which doesn't do
> > dependency tracking in its out of order queues) what I need to know is
> > whether I need to let previous queued up entries finish before I can run
> > this one or not.
> >
> > Now there is an obvious 'hack' I can use, which is that the above only
> > matters is if the IV is still 'in flight'. Given it is still inflight the
> > memory is in use. Thus until it is finished it's address is clearly only
> > being used by 'this IV'.
> >
> > Hence I can just compare against the address of the IV for the previous
> > packet. If it changed we know there is no need to wait for previous packets
> > to finish.
>
> That is dangerous, because the IV is in a malloced location. It is possible
> that a new malloc will return the same address.
True, but this can only happen once a free has occurred which means the
IV is done with from the previous point of view and we no longer need to
track the dependency.
So it doesn't tell us if the two messages use the same IV or not, but in the
case where it is wrong (and indeed I typically get the same address a lot
of times when running little else on the machine) it doesn't matter because
the element is already finished and so there is nothing to queue behind.
>
> What about simply applying the cipher operation as "one shot". I.e. one
> invocation of a cipher operation is isolated from the next. Thus, queue one
> request defined with one request data structure and handle each request
> independent from the next.
I do queue one request from one structure, but if chaining is going on
(the first one should save the chained IV for the second one) I have to wait
for the update to happen before I can put the second one on the queue.
This is done by having a front end software queue when in chaining modes.
The conditions for entry into the hardware queue are:
(hardware queue empty AND software feeder queue empty) OR
(no dependency AND software feeder queue empty)
The completion hander will then move a message from software to hardware
queues if there is one waiting.
It is this dependency tracking that indirectly takes the time as I can only
have one invocation in the hardware queues at a time - hence each interrupt
is for only one invocation. Given I have a number of hardware engines
behind the queue I really want to get as many in flight as possible.
To do this I need to know they are independent at ingress.
I need to get the driver through our internal processes, but hopefully can
post an RFC sometime later this week.
Jonathan
> >
> > I'll still have to be a little careful to avoid DoS issues if we flip from
> > one mode to the other but I can do that by ensuring my software queue is
> > empty before pushing to the hardware queue.
> >
> > Anyhow using the address does seem a little fragile, any suggestions for a
> > better way?
> >
> > On the plus side, with 8k iovs I'm getting around a 50% performance boost by
> > not having to serialize the entry into the hardware queues. From a few back
> > of the envelope calculations the key thing is that I can coalesce
> > interrupts.
> >
> > For very small iovs the overhead of setting them us is enough to mean we
> > rarely end up with many in the queue at a time so the benefits aren't
> > seen.
> >
> > On that note I have a hacked up libkcapi speed test that adds a parameter to
> > control this. I'll send you a pull request later so you can take a look. I
> > have only done the skcipher case though and I am sure it could be done in a
> > neater fashion!
>
> Thank you.
>
> Ciao
> Stephan
>
>
On 21-01-2018 17:44, Stephan Müller wrote:
> Hi Herbert,
>
> I tried to summarize the use cases of the AIO support at [1].
>
> The use case covering the inline IV support is documented in section [2]. It
> naturally would depend on this patch to be accepted. What is your take on this
> use case?
>
> What is your take on the issue outlined at [3]? Should the affected drivers be
> updated?
Till now driver is using content of req->info as IV for cipher operation and It treats each request independent of previous request.
>From description given at [1]. Following is the new requirement for Driver.
"The cipher implementations ensure that although the IOVECs cause parallel
invocation of the recvmsg handler, they are serialized such that the
one IV sent to the kernel is updated by the first cipher operation and
used as input to the second cipher operation, and so on. This implies,
for example, that the CBC block chaining operation is applied
for all IOVECs."
To make this work how driver will know that it has to delay the processing of current request because previous request is not complete(or IV is not available) .
> Thanks
> Stephan
>
> [1] https://github.com/smuellerDD/libkcapi/commit/
> 29132075b8f045f3f92367c0190add81ccf5da11
>
> [2] https://github.com/smuellerDD/libkcapi/commit/
> 29132075b8f045f3f92367c0190add81ccf5da11#diff-
> dce7a00cbc610df94f0dc27eb769d01dR644
>
> [3] https://github.com/smuellerDD/libkcapi/commit/
> 29132075b8f045f3f92367c0190add81ccf5da11#diff-
> dce7a00cbc610df94f0dc27eb769d01dR575
>
>
Hi Harsh,
may I as you to try the attached patch on your Chelsio driver? Please invoke
the following command from libkcapi:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
The result should now be a fully block-chained result.
Note, the patch goes on top of the inline IV patch.
Thanks
---8<---
In case of AIO where multiple IOCBs are sent to the kernel and inline IV
is not used, the ctx->iv is used for each IOCB. The ctx->iv is read for
the crypto operation and written back after the crypto operation. Thus,
processing of multiple IOCBs must be serialized.
As the AF_ALG interface is used by user space, a mutex provides the
serialization which puts the caller to sleep in case a previous IOCB
processing is not yet finished.
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
of processing arriving requests ensures that the blocked IOCBs are
unblocked in the right order.
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 22 ++++++++++++++++++++++
crypto/algif_aead.c | 2 ++
crypto/algif_skcipher.c | 2 ++
include/crypto/if_alg.h | 3 +++
4 files changed, 29 insertions(+)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 87394dc90ac0..3007c9d899da 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -1059,6 +1059,8 @@ void af_alg_async_cb(struct crypto_async_request *_req,
int err)
struct kiocb *iocb = areq->iocb;
unsigned int resultlen;
+ af_alg_put_iv(sk);
+
/* Buffer size written by crypto operation. */
resultlen = areq->outlen;
@@ -1227,6 +1229,11 @@ int af_alg_get_iv(struct sock *sk, struct
af_alg_async_req *areq)
/* No inline IV or cipher has no IV, use ctx IV buffer. */
if (!ctx->iiv || !ctx->ivlen) {
+ int ret = mutex_lock_interruptible(&ctx->ivlock);
+
+ if (ret)
+ return ret;
+
areq->iv = ctx->iv;
areq->ivlen = 0; // areq->iv will not be freed
return 0;
@@ -1252,6 +1259,21 @@ int af_alg_get_iv(struct sock *sk, struct
af_alg_async_req *areq)
}
EXPORT_SYMBOL_GPL(af_alg_get_iv);
+/**
+ * af_alg_put_iv - release lock on IV in case CTX IV is used
+ *
+ * @sk [in] AF_ALG socket
+ */
+void af_alg_put_iv(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+
+ if (!ctx->iiv || !ctx->ivlen)
+ mutex_unlock(&ctx->ivlock);
+}
+EXPORT_SYMBOL_GPL(af_alg_put_iv);
+
static int __init af_alg_init(void)
{
int err = proto_register(&alg_proto, 0);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 7eb7cb132c09..165b2ca82e51 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -309,6 +309,7 @@ static int _aead_recvmsg(struct socket *sock, struct
msghdr *msg,
&ctx->wait);
}
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -555,6 +556,7 @@ static int aead_accept_parent_nokey(void *private, struct
sock *sk)
return -ENOMEM;
}
memset(ctx->iv, 0, ctx->ivlen);
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index d40e1d6797d8..a759cec446b4 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -151,6 +151,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct
msghdr *msg,
&ctx->wait);
}
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -358,6 +359,7 @@ static int skcipher_accept_parent_nokey(void *private,
struct sock *sk)
}
memset(ctx->iv, 0, ctx->ivlen);
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index ebc651ceb54a..666be8ac683c 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -19,6 +19,7 @@
#include <linux/scatterlist.h>
#include <linux/types.h>
#include <linux/atomic.h>
+#include <linux/mutex.h>
#include <net/sock.h>
#include <crypto/aead.h>
@@ -160,6 +161,7 @@ struct af_alg_ctx {
void *iv;
unsigned int ivlen;
+ struct mutex ivlock;
size_t aead_assoclen;
struct crypto_wait wait;
@@ -268,6 +270,7 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg,
int flags,
struct af_alg_async_req *areq, size_t maxsize,
size_t *outlen);
int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
+void af_alg_put_iv(struct sock *sk);
struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
#endif /* _CRYPTO_IF_ALG_H */
--
Am Dienstag, 30. Januar 2018, 09:27:04 CET schrieb Stephan M?ller:
Hi,
> +/**
> + * af_alg_put_iv - release lock on IV in case CTX IV is used
> + *
> + * @sk [in] AF_ALG socket
> + */
> +void af_alg_put_iv(struct sock *sk)
> +{
> + struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> +
> + if (!ctx->iiv || !ctx->ivlen)
> + mutex_unlock(&ctx->ivlock);
> +}
> +EXPORT_SYMBOL_GPL(af_alg_put_iv);
Having this function implies that ctx->iiv must be set once at the beginning
and cannot be toggled for a CTX thereafter as otherwise the release of the
mutex is racy. This implies that the inline IV patch needs a slight revision
to prevent toggling the ctx->iiv value.
I will send a new revision of the inline IV and the lock context IV patch
covering this issue.
Ciao
Stephan
On Tue, 30 Jan 2018 09:27:04 +0100
Stephan M?ller <[email protected]> wrote:
> Hi Harsh,
>
> may I as you to try the attached patch on your Chelsio driver? Please invoke
> the following command from libkcapi:
>
> kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
>
> The result should now be a fully block-chained result.
>
> Note, the patch goes on top of the inline IV patch.
>
> Thanks
>
> ---8<---
>
> In case of AIO where multiple IOCBs are sent to the kernel and inline IV
> is not used, the ctx->iv is used for each IOCB. The ctx->iv is read for
> the crypto operation and written back after the crypto operation. Thus,
> processing of multiple IOCBs must be serialized.
>
> As the AF_ALG interface is used by user space, a mutex provides the
> serialization which puts the caller to sleep in case a previous IOCB
> processing is not yet finished.
>
> If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
> of processing arriving requests ensures that the blocked IOCBs are
> unblocked in the right order.
>
> Signed-off-by: Stephan Mueller <[email protected]>
As a reference point, holding outside the kernel (in at least
the case of our hardware with 8K CBC) drops us down to around 30%
of performance with separate IVs. Putting a queue in kernel
(and hence allowing most setup of descriptors / DMA etc) gets us
up to 50% of raw non chained performance.
So whilst this will work in principle I suspect anyone who
cares about the cost will be looking at adding their own
software queuing and dependency handling in driver. How
much that matters will depend on just how quick the hardware
is vs overheads.
Anyhow, just posted the driver as an RFC. There are a few corners
that need resolving and the outcome of this thread effects whether
we need to play games with IVs or not.
https://marc.info/?l=linux-crypto-vger&m=151732626428206&w=2
grep for softqueue.
We have a number of processing units that will grab requests
from the HW queues in parallel. So 'lots' of requests
from a given HW queue may be in flight at the same time (the problem
case).
Logic is fairly simple.
1. Only create a softqueue if the encryption mode requires chaining.
Currently in this driver that is CBC and CTR modes only.
Otherwise put everything straight in the hardware monitored queues.
2. If we have an IV embedded within the request it will have a different
address to the one used previously (guaranteed if that one is still
in flight and it doesn't matter if it isn't). If that occurs we
can safely add it to the hardware queue. To avoid DoS issues against
and earlier set of messages using the a chained IV we actually make
sure nothing is in the software queue (not needed for correctness)
3. If IV matches previous IV and there are existing elements in SW or HW
queues we need to hold it in the SW queue.
4. On receiving a response from the hardware and if the hardware queue
is empty, we can release an element from the software queue into the
hardware queue.
This maintains chaining when needed and gets a good chunk of the lost
performance back. I believe (though yet to verify) the remaining lost
performance is mostly down to the fact we can't coalesce interrupts if
chaining.
Note the driver is deliberately a touch 'simplistic' so there may be other
optimization opportunities to get some of the lost performance back or
it may be fundamental due to the fact the raw processing can't be in
parallel.
> ---
> crypto/af_alg.c | 22 ++++++++++++++++++++++
> crypto/algif_aead.c | 2 ++
> crypto/algif_skcipher.c | 2 ++
> include/crypto/if_alg.h | 3 +++
> 4 files changed, 29 insertions(+)
>
> diff --git a/crypto/af_alg.c b/crypto/af_alg.c
> index 87394dc90ac0..3007c9d899da 100644
> --- a/crypto/af_alg.c
> +++ b/crypto/af_alg.c
> @@ -1059,6 +1059,8 @@ void af_alg_async_cb(struct crypto_async_request *_req,
> int err)
> struct kiocb *iocb = areq->iocb;
> unsigned int resultlen;
>
> + af_alg_put_iv(sk);
> +
> /* Buffer size written by crypto operation. */
> resultlen = areq->outlen;
>
> @@ -1227,6 +1229,11 @@ int af_alg_get_iv(struct sock *sk, struct
> af_alg_async_req *areq)
>
> /* No inline IV or cipher has no IV, use ctx IV buffer. */
> if (!ctx->iiv || !ctx->ivlen) {
> + int ret = mutex_lock_interruptible(&ctx->ivlock);
> +
> + if (ret)
> + return ret;
> +
> areq->iv = ctx->iv;
> areq->ivlen = 0; // areq->iv will not be freed
> return 0;
> @@ -1252,6 +1259,21 @@ int af_alg_get_iv(struct sock *sk, struct
> af_alg_async_req *areq)
> }
> EXPORT_SYMBOL_GPL(af_alg_get_iv);
>
> +/**
> + * af_alg_put_iv - release lock on IV in case CTX IV is used
> + *
> + * @sk [in] AF_ALG socket
> + */
> +void af_alg_put_iv(struct sock *sk)
> +{
> + struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> +
> + if (!ctx->iiv || !ctx->ivlen)
> + mutex_unlock(&ctx->ivlock);
> +}
> +EXPORT_SYMBOL_GPL(af_alg_put_iv);
> +
> static int __init af_alg_init(void)
> {
> int err = proto_register(&alg_proto, 0);
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 7eb7cb132c09..165b2ca82e51 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -309,6 +309,7 @@ static int _aead_recvmsg(struct socket *sock, struct
> msghdr *msg,
> &ctx->wait);
> }
>
> + af_alg_put_iv(sk);
>
> free:
> af_alg_free_resources(areq);
> @@ -555,6 +556,7 @@ static int aead_accept_parent_nokey(void *private, struct
> sock *sk)
> return -ENOMEM;
> }
> memset(ctx->iv, 0, ctx->ivlen);
> + mutex_init(&ctx->ivlock);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> ctx->len = len;
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index d40e1d6797d8..a759cec446b4 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -151,6 +151,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct
> msghdr *msg,
> &ctx->wait);
> }
> + af_alg_put_iv(sk);
>
> free:
> af_alg_free_resources(areq);
> @@ -358,6 +359,7 @@ static int skcipher_accept_parent_nokey(void *private,
> struct sock *sk)
> }
> memset(ctx->iv, 0, ctx->ivlen);
> + mutex_init(&ctx->ivlock);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> ctx->len = len;
> diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
> index ebc651ceb54a..666be8ac683c 100644
> --- a/include/crypto/if_alg.h
> +++ b/include/crypto/if_alg.h
> @@ -19,6 +19,7 @@
> #include <linux/scatterlist.h>
> #include <linux/types.h>
> #include <linux/atomic.h>
> +#include <linux/mutex.h>
> #include <net/sock.h>
>
> #include <crypto/aead.h>
> @@ -160,6 +161,7 @@ struct af_alg_ctx {
>
> void *iv;
> unsigned int ivlen;
> + struct mutex ivlock;
> size_t aead_assoclen;
>
> struct crypto_wait wait;
> @@ -268,6 +270,7 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg,
> int flags,
> struct af_alg_async_req *areq, size_t maxsize,
> size_t *outlen);
> int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
> +void af_alg_put_iv(struct sock *sk);
> struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
>
> #endif /* _CRYPTO_IF_ALG_H */
On Tue, 30 Jan 2018 15:51:40 +0000
Jonathan Cameron <[email protected]> wrote:
> On Tue, 30 Jan 2018 09:27:04 +0100
> Stephan M?ller <[email protected]> wrote:
A few clarifications from me after discussions with Stephan this morning.
>
> > Hi Harsh,
> >
> > may I as you to try the attached patch on your Chelsio driver? Please invoke
> > the following command from libkcapi:
> >
> > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
> >
> > The result should now be a fully block-chained result.
> >
> > Note, the patch goes on top of the inline IV patch.
> >
> > Thanks
> >
> > ---8<---
> >
> > In case of AIO where multiple IOCBs are sent to the kernel and inline IV
> > is not used, the ctx->iv is used for each IOCB. The ctx->iv is read for
> > the crypto operation and written back after the crypto operation. Thus,
> > processing of multiple IOCBs must be serialized.
> >
> > As the AF_ALG interface is used by user space, a mutex provides the
> > serialization which puts the caller to sleep in case a previous IOCB
> > processing is not yet finished.
> >
> > If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
> > of processing arriving requests ensures that the blocked IOCBs are
> > unblocked in the right order.
> >
> > Signed-off-by: Stephan Mueller <[email protected]>
Firstly, as far as I can tell (not tested it yet) the patch does the
job and is about the best we can easily do in the AF_ALG code.
I'd suggest that this (or v2 anyway) is stable material as well
(which, as Stephan observed, will require reordering of the two patches).
>
> As a reference point, holding outside the kernel (in at least
> the case of our hardware with 8K CBC) drops us down to around 30%
> of performance with separate IVs. Putting a queue in kernel
> (and hence allowing most setup of descriptors / DMA etc) gets us
> up to 50% of raw non chained performance.
>
> So whilst this will work in principle I suspect anyone who
> cares about the cost will be looking at adding their own
> software queuing and dependency handling in driver. How
> much that matters will depend on just how quick the hardware
> is vs overheads.
>
> Anyhow, just posted the driver as an RFC. There are a few corners
> that need resolving and the outcome of this thread effects whether
> we need to play games with IVs or not.
>
> https://marc.info/?l=linux-crypto-vger&m=151732626428206&w=2
> grep for softqueue.
>
> We have a number of processing units that will grab requests
> from the HW queues in parallel. So 'lots' of requests
> from a given HW queue may be in flight at the same time (the problem
> case).
>
> Logic is fairly simple.
> 1. Only create a softqueue if the encryption mode requires chaining.
> Currently in this driver that is CBC and CTR modes only.
> Otherwise put everything straight in the hardware monitored queues.
> 2. If we have an IV embedded within the request it will have a different
> address to the one used previously (guaranteed if that one is still
> in flight and it doesn't matter if it isn't). If that occurs we
> can safely add it to the hardware queue. To avoid DoS issues against
> and earlier set of messages using the a chained IV we actually make
> sure nothing is in the software queue (not needed for correctness)
> 3. If IV matches previous IV and there are existing elements in SW or HW
> queues we need to hold it in the SW queue.
> 4. On receiving a response from the hardware and if the hardware queue
> is empty, we can release an element from the software queue into the
> hardware queue.
The differences are better shown with pictures...
To compare the two approaches. If we break up the data flow the
alternatives are
1) Mutex causes queuing in AF ALG code
The key thing is the [Build / Map HW Descs] for small packets,
up to perhaps 32K, is a significant task we can't avoid.
At 8k it looks like it takes perhaps 20-30% of the time
(though I only have end to end performance numbers so far)
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
NOTHING BLOCKING (lock mut) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (lock mut)
| [BUILD / MAP HW DESCS] * AFTER SERIALIZATION *
Don't care beyond here. |
[Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.
2) The queuing approach I did in the driver moves that serialization
to after the BUILD / MAP HW DESCs stage.
[REQUEST 1 from userspace]
| [REQUEST 2 from userspace]
[AF_ALG/SOCKET] |
| [AF_ALG/SOCKET]
[BUILD / MAP HW DESCS] |
| [BUILD / MAP HW DESCS] * BEFORE SERIALIZATION *
NOTHING BLOCKING |
| IV in flight (enqueue sw queue)
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| [Place in HW Queue]
Don't care beyond here. |
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.
>
> This maintains chaining when needed and gets a good chunk of the lost
> performance back. I believe (though yet to verify) the remaining lost
> performance is mostly down to the fact we can't coalesce interrupts if
> chaining.
>
> Note the driver is deliberately a touch 'simplistic' so there may be other
> optimization opportunities to get some of the lost performance back or
> it may be fundamental due to the fact the raw processing can't be in
> parallel.
>
Quoting a private email from Stephan (at Stephan's suggestion)
> What I however could fathom is that we introduce a flag where a driver
> implements its own queuing mechanism. In this case, the mutex handling would
> be disregarded.
Which works well for the sort of optimization I did and for hardware that
can do iv dependency tracking itself. If hardware dependency tracking was
avilable, you would be able to queue up requests with a chained IV
without taking any special precautions at all. The hardware would
handle the IV update dependencies.
So in conclusion, Stephan's approach should work and give us a nice
small patch suitable for stable inclusion.
However, if people know that their setup overhead can be put in parallel
with previous requests (even when the IV is not yet updated) then they will
probably want to do something inside their own driver and set the flag
that Stephan is proposing adding to bypass the mutex approach.
Jonathan
> > ---
> > crypto/af_alg.c | 22 ++++++++++++++++++++++
> > crypto/algif_aead.c | 2 ++
> > crypto/algif_skcipher.c | 2 ++
> > include/crypto/if_alg.h | 3 +++
> > 4 files changed, 29 insertions(+)
> >
> > diff --git a/crypto/af_alg.c b/crypto/af_alg.c
> > index 87394dc90ac0..3007c9d899da 100644
> > --- a/crypto/af_alg.c
> > +++ b/crypto/af_alg.c
> > @@ -1059,6 +1059,8 @@ void af_alg_async_cb(struct crypto_async_request *_req,
> > int err)
> > struct kiocb *iocb = areq->iocb;
> > unsigned int resultlen;
> >
> > + af_alg_put_iv(sk);
> > +
> > /* Buffer size written by crypto operation. */
> > resultlen = areq->outlen;
> >
> > @@ -1227,6 +1229,11 @@ int af_alg_get_iv(struct sock *sk, struct
> > af_alg_async_req *areq)
> >
> > /* No inline IV or cipher has no IV, use ctx IV buffer. */
> > if (!ctx->iiv || !ctx->ivlen) {
> > + int ret = mutex_lock_interruptible(&ctx->ivlock);
> > +
> > + if (ret)
> > + return ret;
> > +
> > areq->iv = ctx->iv;
> > areq->ivlen = 0; // areq->iv will not be freed
> > return 0;
> > @@ -1252,6 +1259,21 @@ int af_alg_get_iv(struct sock *sk, struct
> > af_alg_async_req *areq)
> > }
> > EXPORT_SYMBOL_GPL(af_alg_get_iv);
> >
> > +/**
> > + * af_alg_put_iv - release lock on IV in case CTX IV is used
> > + *
> > + * @sk [in] AF_ALG socket
> > + */
> > +void af_alg_put_iv(struct sock *sk)
> > +{
> > + struct alg_sock *ask = alg_sk(sk);
> > + struct af_alg_ctx *ctx = ask->private;
> > +
> > + if (!ctx->iiv || !ctx->ivlen)
> > + mutex_unlock(&ctx->ivlock);
> > +}
> > +EXPORT_SYMBOL_GPL(af_alg_put_iv);
> > +
> > static int __init af_alg_init(void)
> > {
> > int err = proto_register(&alg_proto, 0);
> > diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> > index 7eb7cb132c09..165b2ca82e51 100644
> > --- a/crypto/algif_aead.c
> > +++ b/crypto/algif_aead.c
> > @@ -309,6 +309,7 @@ static int _aead_recvmsg(struct socket *sock, struct
> > msghdr *msg,
> > &ctx->wait);
> > }
> >
> > + af_alg_put_iv(sk);
> >
> > free:
> > af_alg_free_resources(areq);
> > @@ -555,6 +556,7 @@ static int aead_accept_parent_nokey(void *private, struct
> > sock *sk)
> > return -ENOMEM;
> > }
> > memset(ctx->iv, 0, ctx->ivlen);
> > + mutex_init(&ctx->ivlock);
> >
> > INIT_LIST_HEAD(&ctx->tsgl_list);
> > ctx->len = len;
> > diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> > index d40e1d6797d8..a759cec446b4 100644
> > --- a/crypto/algif_skcipher.c
> > +++ b/crypto/algif_skcipher.c
> > @@ -151,6 +151,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct
> > msghdr *msg,
> > &ctx->wait);
> > }
> > + af_alg_put_iv(sk);
> >
> > free:
> > af_alg_free_resources(areq);
> > @@ -358,6 +359,7 @@ static int skcipher_accept_parent_nokey(void *private,
> > struct sock *sk)
> > }
> > memset(ctx->iv, 0, ctx->ivlen);
> > + mutex_init(&ctx->ivlock);
> >
> > INIT_LIST_HEAD(&ctx->tsgl_list);
> > ctx->len = len;
> > diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
> > index ebc651ceb54a..666be8ac683c 100644
> > --- a/include/crypto/if_alg.h
> > +++ b/include/crypto/if_alg.h
> > @@ -19,6 +19,7 @@
> > #include <linux/scatterlist.h>
> > #include <linux/types.h>
> > #include <linux/atomic.h>
> > +#include <linux/mutex.h>
> > #include <net/sock.h>
> >
> > #include <crypto/aead.h>
> > @@ -160,6 +161,7 @@ struct af_alg_ctx {
> >
> > void *iv;
> > unsigned int ivlen;
> > + struct mutex ivlock;
> > size_t aead_assoclen;
> >
> > struct crypto_wait wait;
> > @@ -268,6 +270,7 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg,
> > int flags,
> > struct af_alg_async_req *areq, size_t maxsize,
> > size_t *outlen);
> > int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
> > +void af_alg_put_iv(struct sock *sk);
> > struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
> >
> > #endif /* _CRYPTO_IF_ALG_H */
>
Hi,
On Wed, Jan 31, 2018 at 2:29 PM, Jonathan Cameron
<[email protected]> wrote:
> On Tue, 30 Jan 2018 15:51:40 +0000
> Jonathan Cameron <[email protected]> wrote:
>
>> On Tue, 30 Jan 2018 09:27:04 +0100
>> Stephan Müller <[email protected]> wrote:
>
> A few clarifications from me after discussions with Stephan this morning.
>
>>
>> > Hi Harsh,
>> >
>> > may I as you to try the attached patch on your Chelsio driver? Please invoke
>> > the following command from libkcapi:
>> >
>> > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
>> > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
>> > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
>> >
>> > The result should now be a fully block-chained result.
>> >
>> > Note, the patch goes on top of the inline IV patch.
>> >
>> > Thanks
>> >
>> > ---8<---
>> >
>> > In case of AIO where multiple IOCBs are sent to the kernel and inline IV
>> > is not used, the ctx->iv is used for each IOCB. The ctx->iv is read for
>> > the crypto operation and written back after the crypto operation. Thus,
>> > processing of multiple IOCBs must be serialized.
>> >
>> > As the AF_ALG interface is used by user space, a mutex provides the
>> > serialization which puts the caller to sleep in case a previous IOCB
>> > processing is not yet finished.
>> >
>> > If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
>> > of processing arriving requests ensures that the blocked IOCBs are
>> > unblocked in the right order.
>> >
>> > Signed-off-by: Stephan Mueller <[email protected]>
>
> Firstly, as far as I can tell (not tested it yet) the patch does the
> job and is about the best we can easily do in the AF_ALG code.
>
> I'd suggest that this (or v2 anyway) is stable material as well
> (which, as Stephan observed, will require reordering of the two patches).
>
>>
>> As a reference point, holding outside the kernel (in at least
>> the case of our hardware with 8K CBC) drops us down to around 30%
>> of performance with separate IVs. Putting a queue in kernel
>> (and hence allowing most setup of descriptors / DMA etc) gets us
>> up to 50% of raw non chained performance.
>>
>> So whilst this will work in principle I suspect anyone who
>> cares about the cost will be looking at adding their own
>> software queuing and dependency handling in driver. How
>> much that matters will depend on just how quick the hardware
>> is vs overheads.
>>
<snip>
>
> The differences are better shown with pictures...
> To compare the two approaches. If we break up the data flow the
> alternatives are
>
> 1) Mutex causes queuing in AF ALG code
>
> The key thing is the [Build / Map HW Descs] for small packets,
> up to perhaps 32K, is a significant task we can't avoid.
> At 8k it looks like it takes perhaps 20-30% of the time
> (though I only have end to end performance numbers so far)
>
>
> [REQUEST 1 from userspace]
> | [REQUEST 2 from userspace]
> [AF_ALG/SOCKET] |
> | [AF_ALG/SOCKET]
> NOTHING BLOCKING (lock mut) |
> | Queued on Mutex
> [BUILD / MAP HW DESCS] |
> | |
> [Place in HW Queue] |
> | |
> [Process] |
> | |
> [Interrupt] |
> | |
> [Respond and update IV] Nothing Blocking (lock mut)
> | [BUILD / MAP HW DESCS] * AFTER SERIALIZATION *
> Don't care beyond here. |
> [Place in HW Queue]
> |
> [Process]
> |
> [Interrupt]
> |
> [Respond and update IV]
> |
> Don't care beyond here.
>
>
> 2) The queuing approach I did in the driver moves that serialization
> to after the BUILD / MAP HW DESCs stage.
>
> [REQUEST 1 from userspace]
> | [REQUEST 2 from userspace]
> [AF_ALG/SOCKET] |
> | [AF_ALG/SOCKET]
> [BUILD / MAP HW DESCS] |
> | [BUILD / MAP HW DESCS] * BEFORE SERIALIZATION *
> NOTHING BLOCKING |
> | IV in flight (enqueue sw queue)
> [Place in HW Queue] |
> | |
> [Process] |
> | |
> [Interrupt] |
> | |
> [Respond and update IV] Nothing Blocking (dequeue sw queue)
> | [Place in HW Queue]
> Don't care beyond here. |
> [Process]
> |
> [Interrupt]
> |
> [Respond and update IV]
> |
> Don't care beyond here.
>
>>
>> This maintains chaining when needed and gets a good chunk of the lost
>> performance back. I believe (though yet to verify) the remaining lost
>> performance is mostly down to the fact we can't coalesce interrupts if
>> chaining.
>>
>> Note the driver is deliberately a touch 'simplistic' so there may be other
>> optimization opportunities to get some of the lost performance back or
>> it may be fundamental due to the fact the raw processing can't be in
>> parallel.
>>
>
> Quoting a private email from Stephan (at Stephan's suggestion)
>
>> What I however could fathom is that we introduce a flag where a driver
>> implements its own queuing mechanism. In this case, the mutex handling would
>> be disregarded.
>
> Which works well for the sort of optimization I did and for hardware that
> can do iv dependency tracking itself. If hardware dependency tracking was
> avilable, you would be able to queue up requests with a chained IV
> without taking any special precautions at all. The hardware would
> handle the IV update dependencies.
>
> So in conclusion, Stephan's approach should work and give us a nice
> small patch suitable for stable inclusion.
>
> However, if people know that their setup overhead can be put in parallel
> with previous requests (even when the IV is not yet updated) then they will
> probably want to do something inside their own driver and set the flag
> that Stephan is proposing adding to bypass the mutex approach.
>
The patches from Stephan looks good to me, but I think we can do better
for the long term approach you are discussing.
Consider that the issue we are dealing with is in no way unique to the AF_ALG
use case, it just happens to expose it.
The grander issue is, I believe, that we are missing an API for chained crypto
operations. One that allows submitting multiple concurrent but
dependent requests
to tfm providers.
Trying to second guess whether or not there is a dependency between calls from
the address of the IV is not a clean solution. Why don't we make it explicit?
For example, a flag that can be set on a tfm that states that the subsequent
series of requests are interdependent. If you need a different stream,
simply allocated anotehr
tfm object.
This will let each driver do it's best, be it a simple mutex, software
queuing or hardware
dependency tracking as the case m may be.
Than of course, AF_ALG code (or any other user for that matter) will
not need to handle
interdependence, just set the right flag.
Do you think this makes sense?
Thanks,
Gilad
--
Gilad Ben-Yossef
Chief Coffee Drinker
"If you take a class in large-scale robotics, can you end up in a
situation where the homework eats your dog?"
-- Jean-Baptiste Queru
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
> Hi,
>
> On Wed, Jan 31, 2018 at 2:29 PM, Jonathan Cameron
>
> <[email protected]> wrote:
> > On Tue, 30 Jan 2018 15:51:40 +0000
> >
> > Jonathan Cameron <[email protected]> wrote:
> >> On Tue, 30 Jan 2018 09:27:04 +0100
> >
> >> Stephan M?ller <[email protected]> wrote:
> > A few clarifications from me after discussions with Stephan this morning.
> >
> >> > Hi Harsh,
> >> >
> >> > may I as you to try the attached patch on your Chelsio driver? Please
> >> > invoke the following command from libkcapi:
> >> >
> >> > kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> >> > 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> >> > 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
> >> >
> >> > The result should now be a fully block-chained result.
> >> >
> >> > Note, the patch goes on top of the inline IV patch.
> >> >
> >> > Thanks
> >> >
> >> > ---8<---
> >> >
> >> > In case of AIO where multiple IOCBs are sent to the kernel and inline
> >> > IV
> >> > is not used, the ctx->iv is used for each IOCB. The ctx->iv is read for
> >> > the crypto operation and written back after the crypto operation. Thus,
> >> > processing of multiple IOCBs must be serialized.
> >> >
> >> > As the AF_ALG interface is used by user space, a mutex provides the
> >> > serialization which puts the caller to sleep in case a previous IOCB
> >> > processing is not yet finished.
> >> >
> >> > If multiple IOCBs arrive that all are blocked, the mutex' FIFO
> >> > operation
> >> > of processing arriving requests ensures that the blocked IOCBs are
> >> > unblocked in the right order.
> >> >
> >> > Signed-off-by: Stephan Mueller <[email protected]>
> >
> > Firstly, as far as I can tell (not tested it yet) the patch does the
> > job and is about the best we can easily do in the AF_ALG code.
> >
> > I'd suggest that this (or v2 anyway) is stable material as well
> > (which, as Stephan observed, will require reordering of the two patches).
> >
> >> As a reference point, holding outside the kernel (in at least
> >> the case of our hardware with 8K CBC) drops us down to around 30%
> >> of performance with separate IVs. Putting a queue in kernel
> >> (and hence allowing most setup of descriptors / DMA etc) gets us
> >> up to 50% of raw non chained performance.
> >>
> >> So whilst this will work in principle I suspect anyone who
> >> cares about the cost will be looking at adding their own
> >> software queuing and dependency handling in driver. How
> >> much that matters will depend on just how quick the hardware
> >> is vs overheads.
>
> <snip>
>
> > The differences are better shown with pictures...
> > To compare the two approaches. If we break up the data flow the
> > alternatives are
> >
> > 1) Mutex causes queuing in AF ALG code
> >
> > The key thing is the [Build / Map HW Descs] for small packets,
> > up to perhaps 32K, is a significant task we can't avoid.
> > At 8k it looks like it takes perhaps 20-30% of the time
> > (though I only have end to end performance numbers so far)
> >
> >
> > [REQUEST 1 from userspace]
> >
> > | [REQUEST 2 from userspace]
> >
> > [AF_ALG/SOCKET] |
> >
> > | [AF_ALG/SOCKET]
> >
> > NOTHING BLOCKING (lock mut) |
> >
> > | Queued on Mutex
> >
> > [BUILD / MAP HW DESCS] |
> >
> > [Place in HW Queue] |
> >
> > [Process] |
> >
> > [Interrupt] |
> >
> > [Respond and update IV] Nothing Blocking (lock mut)
> >
> > | [BUILD / MAP HW DESCS] * AFTER
> > | SERIALIZATION *
> >
> > Don't care beyond here. |
> >
> > [Place in HW Queue]
> >
> > [Process]
> >
> > [Interrupt]
> >
> > [Respond and update IV]
> >
> > Don't care beyond here.
> >
> > 2) The queuing approach I did in the driver moves that serialization
> > to after the BUILD / MAP HW DESCs stage.
> >
> > [REQUEST 1 from userspace]
> >
> > | [REQUEST 2 from userspace]
> >
> > [AF_ALG/SOCKET] |
> >
> > | [AF_ALG/SOCKET]
> >
> > [BUILD / MAP HW DESCS] |
> >
> > | [BUILD / MAP HW DESCS] * BEFORE
> > | SERIALIZATION *
> >
> > NOTHING BLOCKING |
> >
> > | IV in flight (enqueue sw queue)
> >
> > [Place in HW Queue] |
> >
> > [Process] |
> >
> > [Interrupt] |
> >
> > [Respond and update IV] Nothing Blocking (dequeue sw queue)
> >
> > | [Place in HW Queue]
> >
> > Don't care beyond here. |
> >
> > [Process]
> >
> > [Interrupt]
> >
> > [Respond and update IV]
> >
> > Don't care beyond here.
> >>
> >> This maintains chaining when needed and gets a good chunk of the lost
> >> performance back. I believe (though yet to verify) the remaining lost
> >> performance is mostly down to the fact we can't coalesce interrupts if
> >> chaining.
> >>
> >> Note the driver is deliberately a touch 'simplistic' so there may be
> >> other
> >> optimization opportunities to get some of the lost performance back or
> >> it may be fundamental due to the fact the raw processing can't be in
> >> parallel.
> >
> > Quoting a private email from Stephan (at Stephan's suggestion)
> >
> >> What I however could fathom is that we introduce a flag where a driver
> >> implements its own queuing mechanism. In this case, the mutex handling
> >> would be disregarded.
> >
> > Which works well for the sort of optimization I did and for hardware that
> > can do iv dependency tracking itself. If hardware dependency tracking was
> > avilable, you would be able to queue up requests with a chained IV
> > without taking any special precautions at all. The hardware would
> > handle the IV update dependencies.
> >
> > So in conclusion, Stephan's approach should work and give us a nice
> > small patch suitable for stable inclusion.
> >
> > However, if people know that their setup overhead can be put in parallel
> > with previous requests (even when the IV is not yet updated) then they
> > will
> > probably want to do something inside their own driver and set the flag
> > that Stephan is proposing adding to bypass the mutex approach.
>
> The patches from Stephan looks good to me, but I think we can do better
> for the long term approach you are discussing.
>
> Consider that the issue we are dealing with is in no way unique to the
> AF_ALG use case, it just happens to expose it.
>
> The grander issue is, I believe, that we are missing an API for chained
> crypto operations. One that allows submitting multiple concurrent but
> dependent requests
> to tfm providers.
>
> Trying to second guess whether or not there is a dependency between calls
> from the address of the IV is not a clean solution. Why don't we make it
> explicit?
>
> For example, a flag that can be set on a tfm that states that the subsequent
> series of requests are interdependent. If you need a different stream,
> simply allocated anotehr
> tfm object.
But this is already implemented, is it not considering the discussed patches?
Again, here are the use cases I see for AIO:
- multiple IOCBs which are totally separate and have no interdependencies: the
inline IV approach
- multiple IOCBs which are interdependent: the current API with the patches
that we are discussing here (note, a new patch set is coming after the merge
window)
- if you have several independent invocations of multiple interdependent
IOCBs: call accept() again on the TFM-FD to get another operations FD. This is
followed either option one or two above. Note, this is the idea behind the
kcapi_handle_reinit() call just added to libkcapi. So, you have multiple
operation-FDs on which you do either interdependent operation or inline IV
handling.
>
> This will let each driver do it's best, be it a simple mutex, software
> queuing or hardware
> dependency tracking as the case m may be.
Again, I am wondering that with the discussed patch set we have this
functionality already.
>
> Than of course, AF_ALG code (or any other user for that matter) will
> not need to handle
> interdependence, just set the right flag.
The issue is that some drivers simply do not handle this interdependency at
all -- see the bug report from Harsh.
Thus, the current patch set (again which will be updated after the merge
window) contains:
1. adding a mutex to AF_ALG to ensure serialization of interdependent calls
(option 2 from above) irrespective whether the driver implements support for
it or not.
2. add the inline IV handling (to serve option 1)
3. add a flag that can be set by the crypto implementation. If this flag is
set, then the mutex of AF_ALG is *not* set assuming that the crypto driver
handles the serialization.
Note, option 3 from above is implemented in the proper usage of the AF_ALG
interface.
>
> Do you think this makes sense?
>
> Thanks,
> Gilad
Ciao
Stephan
Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
> >
> > Which works well for the sort of optimization I did and for hardware that
> > can do iv dependency tracking itself. If hardware dependency tracking was
> > avilable, you would be able to queue up requests with a chained IV
> > without taking any special precautions at all. The hardware would
> > handle the IV update dependencies.
> >
> > So in conclusion, Stephan's approach should work and give us a nice
> > small patch suitable for stable inclusion.
> >
> > However, if people know that their setup overhead can be put in parallel
> > with previous requests (even when the IV is not yet updated) then they
> > will
> > probably want to do something inside their own driver and set the flag
> > that Stephan is proposing adding to bypass the mutex approach.
>
> The patches from Stephan looks good to me, but I think we can do better
> for the long term approach you are discussing.
What you made me think of is the following: shouldn't we relay the inline IV
flag on to the crypto drivers?
The reason is to allow a clean implementation of the enabling or disabling of
the dependency handling in the driver. Jonathan's driver, for example, decides
based on the pointer value of the IV buffer whether it is the same buffer and
thus dependency handling is to be applied. This is fragile.
As AF_ALG knows whether the inline IV with separate IVs per request or the
serialization with one IV buffer for all requests is requested, it should
relay this state on to the drivers. Thus, for example, Jonathan's driver can
be changed to rely on this flag instead on the buffer pointer value to decide
whether to enable its dependency handling.
Ciao
Stephan
On Thu, Feb 1, 2018 at 11:46 AM, Stephan Mueller <[email protected]> wrote:
> Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
>
> Hi Gilad,
>
>> Hi,
>>
<snip>
>> >
>> > Quoting a private email from Stephan (at Stephan's suggestion)
>> >
>> >> What I however could fathom is that we introduce a flag where a driver
>> >> implements its own queuing mechanism. In this case, the mutex handling
>> >> would be disregarded.
>> >
>> > Which works well for the sort of optimization I did and for hardware that
>> > can do iv dependency tracking itself. If hardware dependency tracking was
>> > avilable, you would be able to queue up requests with a chained IV
>> > without taking any special precautions at all. The hardware would
>> > handle the IV update dependencies.
>> >
>> > So in conclusion, Stephan's approach should work and give us a nice
>> > small patch suitable for stable inclusion.
>> >
>> > However, if people know that their setup overhead can be put in parallel
>> > with previous requests (even when the IV is not yet updated) then they
>> > will
>> > probably want to do something inside their own driver and set the flag
>> > that Stephan is proposing adding to bypass the mutex approach.
>>
>> The patches from Stephan looks good to me, but I think we can do better
>> for the long term approach you are discussing.
>>
>> Consider that the issue we are dealing with is in no way unique to the
>> AF_ALG use case, it just happens to expose it.
>>
>> The grander issue is, I believe, that we are missing an API for chained
>> crypto operations. One that allows submitting multiple concurrent but
>> dependent requests
>> to tfm providers.
>>
>> Trying to second guess whether or not there is a dependency between calls
>> from the address of the IV is not a clean solution. Why don't we make it
>> explicit?
>>
>> For example, a flag that can be set on a tfm that states that the subsequent
>> series of requests are interdependent. If you need a different stream,
>> simply allocated anotehr
>> tfm object.
>
> But this is already implemented, is it not considering the discussed patches?
>
> Again, here are the use cases I see for AIO:
>
> - multiple IOCBs which are totally separate and have no interdependencies: the
> inline IV approach
>
> - multiple IOCBs which are interdependent: the current API with the patches
> that we are discussing here (note, a new patch set is coming after the merge
> window)
>
> - if you have several independent invocations of multiple interdependent
> IOCBs: call accept() again on the TFM-FD to get another operations FD. This is
> followed either option one or two above. Note, this is the idea behind the
> kcapi_handle_reinit() call just added to libkcapi. So, you have multiple
> operation-FDs on which you do either interdependent operation or inline IV
> handling.
>>
>> This will let each driver do it's best, be it a simple mutex, software
>> queuing or hardware
>> dependency tracking as the case m may be.
>
> Again, I am wondering that with the discussed patch set we have this
> functionality already.
>>
>> Than of course, AF_ALG code (or any other user for that matter) will
>> not need to handle
>> interdependence, just set the right flag.
>
> The issue is that some drivers simply do not handle this interdependency at
> all -- see the bug report from Harsh.
>
> Thus, the current patch set (again which will be updated after the merge
> window) contains:
>
> 1. adding a mutex to AF_ALG to ensure serialization of interdependent calls
> (option 2 from above) irrespective whether the driver implements support for
> it or not.
>
> 2. add the inline IV handling (to serve option 1)
>
> 3. add a flag that can be set by the crypto implementation. If this flag is
> set, then the mutex of AF_ALG is *not* set assuming that the crypto driver
> handles the serialization.
>
> Note, option 3 from above is implemented in the proper usage of the AF_ALG
> interface.
Sorry for not communicating clearly enough. I was not objecting at
all. Let me try again.
What I was trying to say is, more succinctly hopefully this time:
1. Advocating for that 3rd option. I did not understand the plan
include adding it,
2. Pointing out that the problem solved (and rightfully so) by mutex in AF_ALG
AIO implementation must exists elsewhere as well - for example IPsec, and is
probably solved there too, so if we add the flag as suggested, it can be used
there as well to gain similar benefits to what Jonathan is suggesting.
Thanks,
Gilad
--
Gilad Ben-Yossef
Chief Coffee Drinker
"If you take a class in large-scale robotics, can you end up in a
situation where the homework eats your dog?"
-- Jean-Baptiste Queru
On Thu, Feb 1, 2018 at 12:04 PM, Stephan Mueller <[email protected]> wrote:
> Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
>
> Hi Gilad,
>
>> >
>> > Which works well for the sort of optimization I did and for hardware that
>> > can do iv dependency tracking itself. If hardware dependency tracking was
>> > avilable, you would be able to queue up requests with a chained IV
>> > without taking any special precautions at all. The hardware would
>> > handle the IV update dependencies.
>> >
>> > So in conclusion, Stephan's approach should work and give us a nice
>> > small patch suitable for stable inclusion.
>> >
>> > However, if people know that their setup overhead can be put in parallel
>> > with previous requests (even when the IV is not yet updated) then they
>> > will
>> > probably want to do something inside their own driver and set the flag
>> > that Stephan is proposing adding to bypass the mutex approach.
>>
>> The patches from Stephan looks good to me, but I think we can do better
>> for the long term approach you are discussing.
>
> What you made me think of is the following: shouldn't we relay the inline IV
> flag on to the crypto drivers?
>
> The reason is to allow a clean implementation of the enabling or disabling of
> the dependency handling in the driver. Jonathan's driver, for example, decides
> based on the pointer value of the IV buffer whether it is the same buffer and
> thus dependency handling is to be applied. This is fragile.
>
> As AF_ALG knows whether the inline IV with separate IVs per request or the
> serialization with one IV buffer for all requests is requested, it should
> relay this state on to the drivers. Thus, for example, Jonathan's driver can
> be changed to rely on this flag instead on the buffer pointer value to decide
> whether to enable its dependency handling.
Yes, that is exactly what I was trying to point out :-)
Thanks,
Gilad
--
Gilad Ben-Yossef
Chief Coffee Drinker
"If you take a class in large-scale robotics, can you end up in a
situation where the homework eats your dog?"
-- Jean-Baptiste Queru
Am Donnerstag, 1. Februar 2018, 11:06:30 CET schrieb Gilad Ben-Yossef:
Hi Gilad,
> 2. Pointing out that the problem solved (and rightfully so) by mutex in
> AF_ALG AIO implementation must exists elsewhere as well - for example
> IPsec, and is probably solved there too, so if we add the flag as
> suggested, it can be used there as well to gain similar benefits to what
> Jonathan is suggesting.
You are quite right. this patch could speed up things for IPSec and the disk
encryption logic too. So I think we should CC also the IPSec folks and the
disk crypto folks to review the patch.
Ciao
Stephan
On Thu, 1 Feb 2018 12:07:21 +0200
Gilad Ben-Yossef <[email protected]> wrote:
> On Thu, Feb 1, 2018 at 12:04 PM, Stephan Mueller <[email protected]> wrote:
> > Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
> >
> > Hi Gilad,
> >
> >> >
> >> > Which works well for the sort of optimization I did and for hardware that
> >> > can do iv dependency tracking itself. If hardware dependency tracking was
> >> > avilable, you would be able to queue up requests with a chained IV
> >> > without taking any special precautions at all. The hardware would
> >> > handle the IV update dependencies.
> >> >
> >> > So in conclusion, Stephan's approach should work and give us a nice
> >> > small patch suitable for stable inclusion.
> >> >
> >> > However, if people know that their setup overhead can be put in parallel
> >> > with previous requests (even when the IV is not yet updated) then they
> >> > will
> >> > probably want to do something inside their own driver and set the flag
> >> > that Stephan is proposing adding to bypass the mutex approach.
> >>
> >> The patches from Stephan looks good to me, but I think we can do better
> >> for the long term approach you are discussing.
> >
> > What you made me think of is the following: shouldn't we relay the inline IV
> > flag on to the crypto drivers?
> >
> > The reason is to allow a clean implementation of the enabling or disabling of
> > the dependency handling in the driver. Jonathan's driver, for example, decides
> > based on the pointer value of the IV buffer whether it is the same buffer and
> > thus dependency handling is to be applied. This is fragile.
I agree it's inelegant and a flag would be better than pointer tricks (though
they are safe currently - we never know what might change in future)
It was really a minimal example rather than a suggestion of the ultimate
solution ;) I was planning on suggesting a flag myself once the basic
discussion concluded the approach was worthwhile.
> >
> > As AF_ALG knows whether the inline IV with separate IVs per request or the
> > serialization with one IV buffer for all requests is requested, it should
> > relay this state on to the drivers. Thus, for example, Jonathan's driver can
> > be changed to rely on this flag instead on the buffer pointer value to decide
> > whether to enable its dependency handling.
>
> Yes, that is exactly what I was trying to point out :-)
Agreed - make things explicit rather than basically relying on knowing how
the above layers are working.
Thanks,
Jonathan
On 01-02-2018 15:55, Jonathan Cameron wrote:
> On Thu, 1 Feb 2018 12:07:21 +0200
> Gilad Ben-Yossef <[email protected]> wrote:
>
>> On Thu, Feb 1, 2018 at 12:04 PM, Stephan Mueller <[email protected]> wrote:
>>> Am Donnerstag, 1. Februar 2018, 10:35:07 CET schrieb Gilad Ben-Yossef:
>>>
>>> Hi Gilad,
>>>
>>>>> Which works well for the sort of optimization I did and for hardware that
>>>>> can do iv dependency tracking itself. If hardware dependency tracking was
>>>>> avilable, you would be able to queue up requests with a chained IV
>>>>> without taking any special precautions at all. The hardware would
>>>>> handle the IV update dependencies.
>>>>>
>>>>> So in conclusion, Stephan's approach should work and give us a nice
>>>>> small patch suitable for stable inclusion.
>>>>>
>>>>> However, if people know that their setup overhead can be put in parallel
>>>>> with previous requests (even when the IV is not yet updated) then they
>>>>> will
>>>>> probably want to do something inside their own driver and set the flag
>>>>> that Stephan is proposing adding to bypass the mutex approach.
>>>> The patches from Stephan looks good to me, but I think we can do better
>>>> for the long term approach you are discussing.
>>> What you made me think of is the following: shouldn't we relay the inline IV
>>> flag on to the crypto drivers?
>>>
>>> The reason is to allow a clean implementation of the enabling or disabling of
>>> the dependency handling in the driver. Jonathan's driver, for example, decides
>>> based on the pointer value of the IV buffer whether it is the same buffer and
>>> thus dependency handling is to be applied. This is fragile.
> I agree it's inelegant and a flag would be better than pointer tricks (though
> they are safe currently - we never know what might change in future)
> It was really a minimal example rather than a suggestion of the ultimate
> solution ;) I was planning on suggesting a flag myself once the basic
> discussion concluded the approach was worthwhile.
>
>>> As AF_ALG knows whether the inline IV with separate IVs per request or the
>>> serialization with one IV buffer for all requests is requested, it should
>>> relay this state on to the drivers. Thus, for example, Jonathan's driver can
>>> be changed to rely on this flag instead on the buffer pointer value to decide
>>> whether to enable its dependency handling.
>> Yes, that is exactly what I was trying to point out :-)
> Agreed - make things explicit rather than basically relying on knowing how
> the above layers are working.
IPSec layer may not get benefit from this because they send complete sg list in single request only. They don't need partial mode support.
>
> Thanks,
>
> Jonathan
>
Hi Herbert,
Herbert, the patch 1 is meant for stable. However, this patch as is
only applies to the new AF_ALG interface implementation. Though,
the issue goes back to the first implementation of AIO support.
Shall I try prepare a patch for the old AF_ALG implementation
as well?
Changes from v1:
* integrate the inline IV and locking patch into one patch set
* reverse the order of lock context IV patch and inline IV patch --
the reason is to allow the first patch to be back-ported to stable
* mark the first patch (locking of the context IV) as applicable to
stable as there is an existing inconsistency which was demonstrated
by Harsh with the Chelsio driver vs the AES-NI driver
* modify the inline IV patch to have proper unlocking of the mutex
in case of errors
* prevent locking if no IV is defined by cipher
* add a patch to allow crypto drivers to report whether they support
serialization -- in this case the locking in AF_ALG shall be
disabled
* add a patch to inform the crypto drivers that their serialization
support should actually be enabled and used because AF_ALG does not
serialize the interdependent parallel AIO requests
* streamline the code in patch 1 and 2 slightly
I would like to ask the folks with real AIO hardware (Harsh, Jonathan)
to test the patches. Especially, is the locking patch should be tested
by Harsh as you have seen the issue with your hardware.
Thanks.
Stephan Mueller (4):
crypto: AF_ALG AIO - lock context IV
crypto: AF_ALG - inline IV support
crypto: AF_ALG - allow driver to serialize IV access
crypto: add CRYPTO_TFM_REQ_PARALLEL flag
crypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++-
crypto/algif_aead.c | 86 +++++++++++++++++---------------
crypto/algif_skcipher.c | 38 ++++++++++----
include/crypto/if_alg.h | 37 ++++++++++++++
include/linux/crypto.h | 16 ++++++
include/uapi/linux/if_alg.h | 6 ++-
6 files changed, 249 insertions(+), 53 deletions(-)
--
2.14.3
The kernel crypto API requires the caller to set an IV in the request
data structure. That request data structure shall define one particular
cipher operation. During the cipher operation, the IV is read by the
cipher implementation and eventually the potentially updated IV (e.g.
in case of CBC) is written back to the memory location the request data
structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is
stored in the AF_ALG context that is unique to one particular AF_ALG
socket. Note the analogy: an AF_ALG socket is like a TFM where one
recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
with one recvmsg call, multiple IOVECs can be specified. Each
individual IOCB (derived from one IOVEC) implies that one request data
structure is created with the data to be processed by the cipher
implementation. The IV that was set with the sendmsg call is registered
with the request data structure before the cipher operation.
As of now, the individual IOCBs are serialized with respect to the IV
handling. This implies that the kernel does not perform a truly parallel
invocation of the cipher implementations. However, if the user wants to
perform cryptographic operations on multiple IOCBs where each IOCB is
truly independent from the other, parallel invocations are possible.
This would require that each IOCB provides its own IV to ensure true
separation of the IOCBs.
The solution is to allow providing the IV data supplied as part of the
plaintext/ciphertext. To do so, the AF_ALG interface treats the
ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
cipher operation together with the flag whether the operation should
enable support for inline IV handling.
If inline IV handling is enabled, the IV is expected to be the first
part of the input plaintext/ciphertext. This IV is only used for one
cipher operation and will not retained in the kernel for subsequent
cipher operations.
The inline IV handling support is only allowed to be enabled during
the first sendmsg call for a context. Any subsequent sendmsg calls are
not allowed to change the setting of the inline IV handling (either
enable or disable it) as this would open up a race condition with the
locking and unlocking of the ctx->ivlock mutex.
The AEAD support required a slight re-arragning of the code, because
obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
access in _aead_recvmsg must be moved below the IV gathering.
The AEAD code to find the first SG with data in the TX SGL is moved to a
common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is
allowed to provide an IV via sendmsg. It only extends the interface by
giving the user the choice to provide the IV either via sendmsg (the
current approach) or as part of the data (the additional approach).
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 93 ++++++++++++++++++++++++++++++++++++++++++---
crypto/algif_aead.c | 58 ++++++++++++----------------
crypto/algif_skcipher.c | 12 +++---
include/crypto/if_alg.h | 21 +++++++++-
include/uapi/linux/if_alg.h | 6 ++-
5 files changed, 143 insertions(+), 47 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index e7887621aa44..973233000d18 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -14,6 +14,7 @@
#include <linux/atomic.h>
#include <crypto/if_alg.h>
+#include <crypto/scatterwalk.h>
#include <linux/crypto.h>
#include <linux/init.h>
#include <linux/kernel.h>
@@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
+ int iiv = ALG_IIV_DISABLE;
bool init = 0;
int err = 0;
@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
return err;
init = 1;
- switch (con.op) {
+ switch (con.op & ALG_OP_CIPHER_MASK) {
case ALG_OP_ENCRYPT:
enc = 1;
break;
@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
return -EINVAL;
}
+ if (con.op & ALG_OP_INLINE_IV)
+ iiv = ALG_IIV_USE;
+
if (con.iv && con.iv->ivlen != ivsize)
return -EINVAL;
}
@@ -866,6 +871,19 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
if (init) {
ctx->enc = enc;
+
+ /*
+ * IIV can only be enabled once with the first sendmsg call.
+ * This prevents a race in locking and unlocking the
+ * ctx->ivlock mutex.
+ */
+ if (ctx->iiv == ALG_IIV_UNSET) {
+ ctx->iiv = iiv;
+ } else if (iiv == ALG_IIV_USE) {
+ err = -EINVAL;
+ goto unlock;
+ }
+
if (con.iv)
memcpy(ctx->iv, con.iv->iv, ivsize);
@@ -1031,6 +1049,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)
struct sock *sk = areq->sk;
af_alg_free_areq_sgls(areq);
+ if (areq->ivlen)
+ sock_kfree_s(sk, areq->iv, areq->ivlen);
sock_kfree_s(sk, areq, areq->areqlen);
}
EXPORT_SYMBOL_GPL(af_alg_free_resources);
@@ -1178,17 +1198,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
/**
- * af_alg_get_iv
+ * af_alg_get_tsgl_sg - get first SG entry with data from TX SGL
+ * @ctx [in] AF_ALG context with TX sgl
+ * @return pointer to SG with data or NULL if none found
+ */
+struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx)
+{
+ struct af_alg_tsgl *sgl, *tmp;
+ struct scatterlist *sg = NULL;
+ unsigned int i;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ for (i = 0; i < sgl->cur; i++) {
+ struct scatterlist *process_sg = sgl->sg + i;
+
+ if (!(process_sg->length) || !sg_page(process_sg))
+ continue;
+ sg = process_sg;
+ break;
+ }
+ if (sg)
+ break;
+ }
+
+ return sg;
+}
+EXPORT_SYMBOL_GPL(af_alg_get_tsgl_sg);
+
+/**
+ * af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX
*
* @sk [in] AF_ALG socket
+ * @areq [in/out] request buffer that will receive the IV
+ * to be used for the cipher
* @return 0 on success, < 0 on error
*/
-int af_alg_get_iv(struct sock *sk)
+int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
{
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
+ struct scatterlist *sg;
+
+ areq->iv = ctx->iv;
+ areq->ivlen = 0; // areq->iv will not be freed
+
+ /* If cipher has no IV, no need to lock it or do IIV processing */
+ if (!ctx->ivlen)
+ return 0;
- return mutex_lock_interruptible(&ctx->ivlock);
+ /* No inline IV, use ctx IV buffer and lock it */
+ if (ctx->iiv == ALG_IIV_DISABLE)
+ return mutex_lock_interruptible(&ctx->ivlock);
+
+ /* Inline IV handling: There must be the IV data present. */
+ if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
+ return -EINVAL;
+
+ areq->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
+ if (unlikely(!areq->iv))
+ return -ENOMEM;
+ areq->ivlen = ctx->ivlen; // areq->iv will be freed
+
+ /* Get ivlen data from TX SGL and copy it into areq->iv. */
+ sg = af_alg_get_tsgl_sg(ctx);
+ if (!sg)
+ return -EFAULT;
+ scatterwalk_map_and_copy(areq->iv, sg, 0, ctx->ivlen, 0);
+ af_alg_pull_tsgl(sk, ctx->ivlen, NULL, 0);
+
+ return 0;
}
EXPORT_SYMBOL_GPL(af_alg_get_iv);
@@ -1202,7 +1280,12 @@ void af_alg_put_iv(struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
- mutex_unlock(&ctx->ivlock);
+ /* To resemble af_alg_get_iv, do not merge the two branches. */
+ if (!ctx->ivlen)
+ return;
+
+ if (ctx->iiv == ALG_IIV_DISABLE)
+ mutex_unlock(&ctx->ivlock);
}
EXPORT_SYMBOL_GPL(af_alg_put_iv);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 402de50d4a58..623a0fc2b535 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
struct aead_tfm *aeadc = pask->private;
struct crypto_aead *tfm = aeadc->aead;
struct crypto_skcipher *null_tfm = aeadc->null_tfm;
- unsigned int i, as = crypto_aead_authsize(tfm);
+ unsigned int as = crypto_aead_authsize(tfm);
struct af_alg_async_req *areq;
- struct af_alg_tsgl *tsgl, *tmp;
struct scatterlist *rsgl_src, *tsgl_src = NULL;
int err = 0;
size_t used = 0; /* [in] TX bufs to be en/decrypted */
@@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
return err;
}
- /*
- * Data length provided by caller via sendmsg/sendpage that has not
- * yet been processed.
- */
- used = ctx->used;
-
/*
* Make sure sufficient data is present -- note, the same check is
* is also present in sendmsg/sendpage. The checks in sendpage/sendmsg
@@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (!aead_sufficient_data(sk))
return -EINVAL;
+ /* Allocate cipher request for current operation. */
+ areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
+ crypto_aead_reqsize(tfm));
+ if (IS_ERR(areq))
+ return PTR_ERR(areq);
+
+ /* Set areq->iv. */
+ err = af_alg_get_iv(sk, areq);
+ if (err)
+ goto free;
+
+ /*
+ * Data length provided by caller via sendmsg/sendpage that has not
+ * yet been processed.
+ */
+ used = ctx->used;
+
/*
* Calculate the minimum output buffer size holding the result of the
* cipher operation. When encrypting data, the receiving buffer is
@@ -153,16 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
*/
used -= ctx->aead_assoclen;
- /* Allocate cipher request for current operation. */
- areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
- crypto_aead_reqsize(tfm));
- if (IS_ERR(areq))
- return PTR_ERR(areq);
-
- err = af_alg_get_iv(sk);
- if (err)
- goto free;
-
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
if (err)
@@ -187,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
}
processed = used + ctx->aead_assoclen;
- list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) {
- for (i = 0; i < tsgl->cur; i++) {
- struct scatterlist *process_sg = tsgl->sg + i;
-
- if (!(process_sg->length) || !sg_page(process_sg))
- continue;
- tsgl_src = process_sg;
- break;
- }
- if (tsgl_src)
- break;
- }
+ tsgl_src = af_alg_get_tsgl_sg(ctx);
if (processed && !tsgl_src) {
err = -EFAULT;
goto unlock;
@@ -286,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
/* Initialize the crypto operation */
aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src,
- areq->first_rsgl.sgl.sg, used, ctx->iv);
+ areq->first_rsgl.sgl.sg, used, areq->iv);
aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen);
aead_request_set_tfm(&areq->cra_u.aead_req, tfm);
@@ -554,19 +543,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
struct aead_tfm *tfm = private;
struct crypto_aead *aead = tfm->aead;
unsigned int len = sizeof(*ctx);
- unsigned int ivlen = crypto_aead_ivsize(aead);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
return -ENOMEM;
memset(ctx, 0, len);
- ctx->iv = sock_kmalloc(sk, ivlen, GFP_KERNEL);
+ ctx->ivlen = crypto_aead_ivsize(aead);
+ ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
if (!ctx->iv) {
sock_kfree_s(sk, ctx, len);
return -ENOMEM;
}
- memset(ctx->iv, 0, ivlen);
+ memset(ctx->iv, 0, ctx->ivlen);
mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
@@ -576,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->iiv = ALG_IIV_UNSET;
ctx->aead_assoclen = 0;
crypto_init_wait(&ctx->wait);
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index b65b9a89d6bb..8e4d996ecb63 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -77,7 +77,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
- err = af_alg_get_iv(sk);
+ /* Set areq->iv */
+ err = af_alg_get_iv(sk, areq);
if (err)
goto free;
@@ -116,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
/* Initialize the crypto operation */
skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm);
skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl,
- areq->first_rsgl.sgl.sg, len, ctx->iv);
+ areq->first_rsgl.sgl.sg, len, areq->iv);
if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
/* AIO operation */
@@ -350,14 +351,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
if (!ctx)
return -ENOMEM;
- ctx->iv = sock_kmalloc(sk, crypto_skcipher_ivsize(tfm),
- GFP_KERNEL);
+ ctx->ivlen = crypto_skcipher_ivsize(tfm);
+ ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
if (!ctx->iv) {
sock_kfree_s(sk, ctx, len);
return -ENOMEM;
}
- memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
+ memset(ctx->iv, 0, ctx->ivlen);
mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
@@ -367,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->iiv = ALG_IIV_UNSET;
crypto_init_wait(&ctx->wait);
ask->private = ctx;
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index c48eff27e5a9..fb128f953b36 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -96,6 +96,14 @@ struct af_alg_rsgl {
* @tsgl_entries: Number of entries in priv. TX SGL
* @outlen: Number of output bytes generated by crypto op
* @areqlen: Length of this data structure
+ * @iv: Buffer holding the IV for the cipher operation (this
+ * is either ctx->iv where ivlen is set to 0 or a newly
+ * allocated buffer where ivlen represents the size of
+ * this buffer).
+ * @ivlen: IV size -- if the IV buffer is to be freed during
+ * releasing of this data structure, ivlen is non-zero.
+ * If ivlen is zero, but iv is non-NULL, the iv buffer is
+ * not freed.
* @cra_u: Cipher request
*/
struct af_alg_async_req {
@@ -112,6 +120,9 @@ struct af_alg_async_req {
unsigned int outlen;
unsigned int areqlen;
+ void *iv;
+ unsigned int ivlen;
+
union {
struct aead_request aead_req;
struct skcipher_request skcipher_req;
@@ -128,6 +139,7 @@ struct af_alg_async_req {
*
* @tsgl_list: Link to TX SGL
* @iv: IV for cipher operation
+ * @ivlen: IV size
* @ivlock: Mutex to serialize access to the IV
* @aead_assoclen: Length of AAD for AEAD cipher operations
* @completion: Work queue for synchronous operation
@@ -142,12 +154,14 @@ struct af_alg_async_req {
* SG?
* @enc: Cryptographic operation to be performed when
* recvmsg is invoked.
+ * @iiv: Use inline IV: first part of TX data is IV
* @len: Length of memory allocated for this data structure.
*/
struct af_alg_ctx {
struct list_head tsgl_list;
void *iv;
+ unsigned int ivlen;
struct mutex ivlock;
size_t aead_assoclen;
@@ -159,9 +173,13 @@ struct af_alg_ctx {
bool more;
bool merge;
bool enc;
+ int iiv;
unsigned int len;
};
+#define ALG_IIV_UNSET (-1)
+#define ALG_IIV_DISABLE (0)
+#define ALG_IIV_USE (1)
int af_alg_register_type(const struct af_alg_type *type);
int af_alg_unregister_type(const struct af_alg_type *type);
@@ -255,7 +273,8 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
struct af_alg_async_req *areq, size_t maxsize,
size_t *outlen);
-int af_alg_get_iv(struct sock *sk);
+int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
void af_alg_put_iv(struct sock *sk);
+struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
#endif /* _CRYPTO_IF_ALG_H */
diff --git a/include/uapi/linux/if_alg.h b/include/uapi/linux/if_alg.h
index bc2bcdec377b..76a4db392cb6 100644
--- a/include/uapi/linux/if_alg.h
+++ b/include/uapi/linux/if_alg.h
@@ -37,7 +37,9 @@ struct af_alg_iv {
#define ALG_SET_AEAD_AUTHSIZE 5
/* Operations */
-#define ALG_OP_DECRYPT 0
-#define ALG_OP_ENCRYPT 1
+#define ALG_OP_DECRYPT 0x0
+#define ALG_OP_ENCRYPT 0x1
+#define ALG_OP_CIPHER_MASK 0xf
+#define ALG_OP_INLINE_IV 0x10
#endif /* _LINUX_IF_ALG_H */
--
2.14.3
The kernel crypto API requires the caller to set an IV in the request data
structure. That request data structure shall define one particular cipher
operation. During the cipher operation, the IV is read by the cipher
implementation and eventually the potentially updated IV (e.g. in case of
CBC) is written back to the memory location the request data structure
points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored
in the AF_ALG context that is unique to one particular AF_ALG socket. Note
the analogy: an AF_ALG socket is like a TFM where one recvmsg operation
uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with
one recvmsg call, multiple IOVECs can be specified. Each individual IOCB
(derived from one IOVEC) implies that one request data structure is
created with the data to be processed by the cipher implementation. The
IV that was set with the sendmsg call is registered with the request data
structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns
immediately, queuing the request to the hardware. While the AIO request is
processed by the hardware, recvmsg processes the next IOVEC for which
another request is created. Again, the IV buffer from the AF_ALG socket
context is registered with the new request and the cipher operation is
invoked.
You may now see that there is a potential race condition regarding the IV
handling, because there is *no* separate IV buffer for the different
requests. This is nicely demonstrated with libkcapi using the following
command which creates an AIO request with two IOCBs each encrypting one
AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is
processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input
to the 2nd block.
In case the first AIO request is not completed before the 2nd request
commences, the result is two identical AES blocks (i.e. both use the same
IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be
a memory corruption in the IV buffer if both AIO requests write to the IV
buffer at the same time.
As the AF_ALG interface is used by user space, a mutex provides the
serialization which puts the caller to sleep in case a previous IOCB
processing is not yet finished.
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
of processing arriving requests ensures that the blocked IOCBs are
unblocked in the right order.
CC: <[email protected]> #4.14
Fixes: e870456d8e7c8 ("crypto: algif_skcipher - overhaul memory management")
Fixes: d887c52d6ae43 ("crypto: algif_aead - overhaul memory management")
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 31 +++++++++++++++++++++++++++++++
crypto/algif_aead.c | 20 +++++++++++++-------
crypto/algif_skcipher.c | 12 +++++++++---
include/crypto/if_alg.h | 5 +++++
4 files changed, 58 insertions(+), 10 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 5231f421ad00..e7887621aa44 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -1051,6 +1051,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)
struct kiocb *iocb = areq->iocb;
unsigned int resultlen;
+ af_alg_put_iv(sk);
+
/* Buffer size written by crypto operation. */
resultlen = areq->outlen;
@@ -1175,6 +1177,35 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
}
EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
+/**
+ * af_alg_get_iv
+ *
+ * @sk [in] AF_ALG socket
+ * @return 0 on success, < 0 on error
+ */
+int af_alg_get_iv(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+
+ return mutex_lock_interruptible(&ctx->ivlock);
+}
+EXPORT_SYMBOL_GPL(af_alg_get_iv);
+
+/**
+ * af_alg_put_iv - release lock on IV in case CTX IV is used
+ *
+ * @sk [in] AF_ALG socket
+ */
+void af_alg_put_iv(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+
+ mutex_unlock(&ctx->ivlock);
+}
+EXPORT_SYMBOL_GPL(af_alg_put_iv);
+
static int __init af_alg_init(void)
{
int err = proto_register(&alg_proto, 0);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 4b07edd5a9ff..402de50d4a58 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -159,10 +159,14 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
+ err = af_alg_get_iv(sk);
+ if (err)
+ goto free;
+
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
if (err)
- goto free;
+ goto unlock;
/*
* Ensure output buffer is sufficiently large. If the caller provides
@@ -176,7 +180,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (used < less) {
err = -EINVAL;
- goto free;
+ goto unlock;
}
used -= less;
outlen -= less;
@@ -197,7 +201,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
}
if (processed && !tsgl_src) {
err = -EFAULT;
- goto free;
+ goto unlock;
}
/*
@@ -230,7 +234,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
err = crypto_aead_copy_sgl(null_tfm, tsgl_src,
areq->first_rsgl.sgl.sg, processed);
if (err)
- goto free;
+ goto unlock;
af_alg_pull_tsgl(sk, processed, NULL, 0);
} else {
/*
@@ -248,7 +252,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
err = crypto_aead_copy_sgl(null_tfm, tsgl_src,
areq->first_rsgl.sgl.sg, outlen);
if (err)
- goto free;
+ goto unlock;
/* Create TX SGL for tag and chain it to RX SGL. */
areq->tsgl_entries = af_alg_count_tsgl(sk, processed,
@@ -260,7 +264,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
GFP_KERNEL);
if (!areq->tsgl) {
err = -ENOMEM;
- goto free;
+ goto unlock;
}
sg_init_table(areq->tsgl, areq->tsgl_entries);
@@ -316,7 +320,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
&ctx->wait);
}
-
+unlock:
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -562,6 +567,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
return -ENOMEM;
}
memset(ctx->iv, 0, ivlen);
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index c88e5e4cd6a6..b65b9a89d6bb 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -77,10 +77,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
+ err = af_alg_get_iv(sk);
+ if (err)
+ goto free;
+
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, -1, &len);
if (err)
- goto free;
+ goto unlock;
/* Process only as much RX buffers for which we have TX data */
if (len > ctx->used)
@@ -104,7 +108,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
GFP_KERNEL);
if (!areq->tsgl) {
err = -ENOMEM;
- goto free;
+ goto unlock;
}
sg_init_table(areq->tsgl, areq->tsgl_entries);
af_alg_pull_tsgl(sk, len, areq->tsgl, 0);
@@ -146,7 +150,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
&ctx->wait);
}
-
+unlock:
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -353,6 +358,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
}
memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index f38227a78eae..c48eff27e5a9 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -19,6 +19,7 @@
#include <linux/scatterlist.h>
#include <linux/types.h>
#include <linux/atomic.h>
+#include <linux/mutex.h>
#include <net/sock.h>
#include <crypto/aead.h>
@@ -127,6 +128,7 @@ struct af_alg_async_req {
*
* @tsgl_list: Link to TX SGL
* @iv: IV for cipher operation
+ * @ivlock: Mutex to serialize access to the IV
* @aead_assoclen: Length of AAD for AEAD cipher operations
* @completion: Work queue for synchronous operation
* @used: TX bytes sent to kernel. This variable is used to
@@ -146,6 +148,7 @@ struct af_alg_ctx {
struct list_head tsgl_list;
void *iv;
+ struct mutex ivlock;
size_t aead_assoclen;
struct crypto_wait wait;
@@ -252,5 +255,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
struct af_alg_async_req *areq, size_t maxsize,
size_t *outlen);
+int af_alg_get_iv(struct sock *sk);
+void af_alg_put_iv(struct sock *sk);
#endif /* _CRYPTO_IF_ALG_H */
--
2.14.3
Crypto drivers may implement a streamlined serialization support for AIO
requests that is reported by the CRYPTO_TFM_REQ_PARALLEL flag to the
crypto user. When the user decides that he wants to send multiple AIO
requests concurrently and wants the crypto driver to handle the
serialization, the caller has to set CRYPTO_TFM_REQ_PARALLEL to notify
the crypto driver.
Only when this flag is enabled, the crypto driver shall apply its
serialization logic for handling IV updates between requests. If this
flag is not provided, the serialization logic shall not be applied by
the driver as the caller decides that it does not need it (because no
parallel AIO requests are sent) or that it performs its own
serialization.
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/algif_aead.c | 15 ++++++++++++---
crypto/algif_skcipher.c | 15 ++++++++++++---
include/linux/crypto.h | 1 +
3 files changed, 25 insertions(+), 6 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 3970ad7f6fd0..da010405eea0 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
struct sock *psk = ask->parent;
struct alg_sock *pask = alg_sk(psk);
struct aead_tfm *aeadc = pask->private;
- struct crypto_aead *tfm = aeadc->aead;
- unsigned int ivsize = crypto_aead_ivsize(tfm);
+ struct crypto_aead *aead = aeadc->aead;
+ struct crypto_tfm *tfm = crypto_aead_tfm(aead);
+ unsigned int ivsize = crypto_aead_ivsize(aead);
+ int ret = af_alg_sendmsg(sock, msg, size, ivsize);
+
+ if (ret < 0)
+ return ret;
- return af_alg_sendmsg(sock, msg, size, ivsize);
+ if (ctx->iiv == ALG_IIV_USE)
+ tfm->crt_flags |= CRYPTO_TFM_REQ_PARALLEL;
+
+ return ret;
}
static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index aee602a3ec24..225546666cf7 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
struct sock *psk = ask->parent;
struct alg_sock *pask = alg_sk(psk);
- struct crypto_skcipher *tfm = pask->private;
- unsigned ivsize = crypto_skcipher_ivsize(tfm);
+ struct crypto_skcipher *skc = pask->private;
+ struct crypto_tfm *tfm = crypto_skcipher_tfm(skc);
+ unsigned int ivsize = crypto_skcipher_ivsize(skc);
+ int ret = af_alg_sendmsg(sock, msg, size, ivsize);
+
+ if (ret < 0)
+ return ret;
- return af_alg_sendmsg(sock, msg, size, ivsize);
+ if (ctx->iiv == ALG_IIV_USE)
+ tfm->crt_flags |= CRYPTO_TFM_REQ_PARALLEL;
+
+ return ret;
}
static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 4860aa2c9be4..95c9d6e8fb70 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -133,6 +133,7 @@
#define CRYPTO_TFM_REQ_WEAK_KEY 0x00000100
#define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200
#define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400
+#define CRYPTO_TFM_REQ_PARALLEL 0x00000800
#define CRYPTO_TFM_RES_WEAK_KEY 0x00100000
#define CRYPTO_TFM_RES_BAD_KEY_LEN 0x00200000
#define CRYPTO_TFM_RES_BAD_KEY_SCHED 0x00400000
--
2.14.3
The mutex in AF_ALG to serialize access to the IV ensures full
serialization of requests sent to the crypto driver.
However, the hardware may implement serialization to the IV such that
preparation work without touching the IV can already happen while the IV
is processed by another operation. This may speed up the AIO processing.
The following ASCII art demonstrates this.
AF_ALG mutex handling implements the following logic:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
NOTHING BLOCKING (lock mutex) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] |
| |
[unlock mutex] Nothing Blocking (lock mutex)
| |
Don't care beyond here. [BUILD / MAP HW DESCS]
|
[Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.
A crypto driver may implement the following serialization:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
[BUILD / MAP HW DESCS] [BUILD / MAP HW DESCS]
| |
NOTHING BLOCKING IV in flight (enqueue sw queue)
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| |
Don't care beyond here. [Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.
If the driver implements its own serialization (i.e. AF_ALG does not
serialize the access to the IV), the crypto implementation must set the
flag CRYPTO_ALG_SERIALIZES_IV_ACCESS.
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 13 ++++++++-----
crypto/algif_aead.c | 1 +
crypto/algif_skcipher.c | 1 +
include/crypto/if_alg.h | 13 +++++++++++++
include/linux/crypto.h | 15 +++++++++++++++
5 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 973233000d18..8e634163b9ba 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -1247,8 +1247,10 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
return 0;
/* No inline IV, use ctx IV buffer and lock it */
- if (ctx->iiv == ALG_IIV_DISABLE)
- return mutex_lock_interruptible(&ctx->ivlock);
+ if (ctx->iiv == ALG_IIV_DISABLE) {
+ return (ctx->lock_iv) ?
+ mutex_lock_interruptible(&ctx->ivlock) : 0;
+ }
/* Inline IV handling: There must be the IV data present. */
if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
@@ -1280,12 +1282,13 @@ void af_alg_put_iv(struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
- /* To resemble af_alg_get_iv, do not merge the two branches. */
if (!ctx->ivlen)
return;
- if (ctx->iiv == ALG_IIV_DISABLE)
- mutex_unlock(&ctx->ivlock);
+ if (ctx->iiv == ALG_IIV_DISABLE) {
+ if (ctx->lock_iv)
+ mutex_unlock(&ctx->ivlock);
+ }
}
EXPORT_SYMBOL_GPL(af_alg_put_iv);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 623a0fc2b535..3970ad7f6fd0 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -565,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->lock_iv = af_alg_lock_iv(crypto_aead_tfm(aead));
ctx->iiv = ALG_IIV_UNSET;
ctx->aead_assoclen = 0;
crypto_init_wait(&ctx->wait);
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index 8e4d996ecb63..aee602a3ec24 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -368,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->lock_iv = af_alg_lock_iv(crypto_skcipher_tfm(tfm));
ctx->iiv = ALG_IIV_UNSET;
crypto_init_wait(&ctx->wait);
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index fb128f953b36..8cc74ba1160e 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -154,6 +154,7 @@ struct af_alg_async_req {
* SG?
* @enc: Cryptographic operation to be performed when
* recvmsg is invoked.
+ * @lock_iv: Shall IV be locked?
* @iiv: Use inline IV: first part of TX data is IV
* @len: Length of memory allocated for this data structure.
*/
@@ -173,6 +174,7 @@ struct af_alg_ctx {
bool more;
bool merge;
bool enc;
+ bool lock_iv;
int iiv;
unsigned int len;
@@ -251,6 +253,17 @@ static inline bool af_alg_readable(struct sock *sk)
return PAGE_SIZE <= af_alg_rcvbuf(sk);
}
+/**
+ * Shall access to the ctx->iv be serialized using a mutex?
+ *
+ * @tfm TFM of to be used for cipher operation
+ * @return true => lock, false => do not lock
+ */
+static inline bool af_alg_lock_iv(struct crypto_tfm *tfm)
+{
+ return !(crypto_tfm_alg_flags(tfm) & CRYPTO_ALG_SERIALIZES_IV_ACCESS);
+}
+
int af_alg_alloc_tsgl(struct sock *sk);
unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset);
void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst,
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 7e6e84cf6383..4860aa2c9be4 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -112,6 +112,16 @@
*/
#define CRYPTO_ALG_OPTIONAL_KEY 0x00004000
+/*
+ * Set if the algorithm implementation provides its own serialization
+ * of IV accesses. This prevents the AF_ALG interface to serialize
+ * cipher requests. This flag allows the implementation to implement
+ * a more streamlined IV serialization logic, such as only serializing
+ * the submission of a request to hardware whereas the preparation steps
+ * for sending data to hardware are not serialized.
+ */
+#define CRYPTO_ALG_SERIALIZES_IV_ACCESS 0x00008000
+
/*
* Transform masks and values (for crt_flags).
*/
@@ -674,6 +684,11 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK;
}
+static inline u32 crypto_tfm_alg_flags(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags;
+}
+
static inline unsigned int crypto_tfm_alg_blocksize(struct crypto_tfm *tfm)
{
return tfm->__crt_alg->cra_blocksize;
--
2.14.3
On 07-02-2018 13:12, Stephan Müller wrote:
> Hi Herbert,
>
> Herbert, the patch 1 is meant for stable. However, this patch as is
> only applies to the new AF_ALG interface implementation. Though,
> the issue goes back to the first implementation of AIO support.
> Shall I try prepare a patch for the old AF_ALG implementation
> as well?
>
> Changes from v1:
>
> * integrate the inline IV and locking patch into one patch set
>
> * reverse the order of lock context IV patch and inline IV patch --
> the reason is to allow the first patch to be back-ported to stable
>
> * mark the first patch (locking of the context IV) as applicable to
> stable as there is an existing inconsistency which was demonstrated
> by Harsh with the Chelsio driver vs the AES-NI driver
>
> * modify the inline IV patch to have proper unlocking of the mutex
> in case of errors
>
> * prevent locking if no IV is defined by cipher
>
> * add a patch to allow crypto drivers to report whether they support
> serialization -- in this case the locking in AF_ALG shall be
> disabled
>
> * add a patch to inform the crypto drivers that their serialization
> support should actually be enabled and used because AF_ALG does not
> serialize the interdependent parallel AIO requests
>
> * streamline the code in patch 1 and 2 slightly
>
> I would like to ask the folks with real AIO hardware (Harsh, Jonathan)
> to test the patches. Especially, is the locking patch should be tested
> by Harsh as you have seen the issue with your hardware.
Sure I will test the patch and let you know.
>
> Thanks.
>
> Stephan Mueller (4):
> crypto: AF_ALG AIO - lock context IV
> crypto: AF_ALG - inline IV support
> crypto: AF_ALG - allow driver to serialize IV access
> crypto: add CRYPTO_TFM_REQ_PARALLEL flag
>
> crypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++-
> crypto/algif_aead.c | 86 +++++++++++++++++---------------
> crypto/algif_skcipher.c | 38 ++++++++++----
> include/crypto/if_alg.h | 37 ++++++++++++++
> include/linux/crypto.h | 16 ++++++
> include/uapi/linux/if_alg.h | 6 ++-
> 6 files changed, 249 insertions(+), 53 deletions(-)
>
Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan M?ller:
Hi,
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 3970ad7f6fd0..da010405eea0 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct
> msghdr *msg, size_t size) {
> struct sock *sk = sock->sk;
> struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> struct sock *psk = ask->parent;
> struct alg_sock *pask = alg_sk(psk);
> struct aead_tfm *aeadc = pask->private;
> - struct crypto_aead *tfm = aeadc->aead;
> - unsigned int ivsize = crypto_aead_ivsize(tfm);
> + struct crypto_aead *aead = aeadc->aead;
> + struct crypto_tfm *tfm = crypto_aead_tfm(aead);
> + unsigned int ivsize = crypto_aead_ivsize(aead);
> + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> +
> + if (ret < 0)
> + return ret;
>
> - return af_alg_sendmsg(sock, msg, size, ivsize);
> + if (ctx->iiv == ALG_IIV_USE)
This should be ALG_IIV_DISABLE of course.
> + tfm->crt_flags |= CRYPTO_TFM_REQ_PARALLEL;
> +
> + return ret;
> }
>
> static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index aee602a3ec24..225546666cf7 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct
> msghdr *msg, {
> struct sock *sk = sock->sk;
> struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> struct sock *psk = ask->parent;
> struct alg_sock *pask = alg_sk(psk);
> - struct crypto_skcipher *tfm = pask->private;
> - unsigned ivsize = crypto_skcipher_ivsize(tfm);
> + struct crypto_skcipher *skc = pask->private;
> + struct crypto_tfm *tfm = crypto_skcipher_tfm(skc);
> + unsigned int ivsize = crypto_skcipher_ivsize(skc);
> + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> +
> + if (ret < 0)
> + return ret;
>
> - return af_alg_sendmsg(sock, msg, size, ivsize);
> + if (ctx->iiv == ALG_IIV_USE)
Dto.
I will wait for some more comments before sending a new patch.
Ciao
Stephan
On Wed, 7 Feb 2018 08:43:32 +0100
Stephan M?ller <[email protected]> wrote:
> The kernel crypto API requires the caller to set an IV in the request
> data structure. That request data structure shall define one particular
> cipher operation. During the cipher operation, the IV is read by the
> cipher implementation and eventually the potentially updated IV (e.g.
> in case of CBC) is written back to the memory location the request data
> structure points to.
>
> AF_ALG allows setting the IV with a sendmsg request, where the IV is
> stored in the AF_ALG context that is unique to one particular AF_ALG
> socket. Note the analogy: an AF_ALG socket is like a TFM where one
> recvmsg operation uses one request with the TFM from the socket.
>
> AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
> with one recvmsg call, multiple IOVECs can be specified. Each
> individual IOCB (derived from one IOVEC) implies that one request data
> structure is created with the data to be processed by the cipher
> implementation. The IV that was set with the sendmsg call is registered
> with the request data structure before the cipher operation.
>
> As of now, the individual IOCBs are serialized with respect to the IV
> handling. This implies that the kernel does not perform a truly parallel
> invocation of the cipher implementations. However, if the user wants to
> perform cryptographic operations on multiple IOCBs where each IOCB is
> truly independent from the other, parallel invocations are possible.
> This would require that each IOCB provides its own IV to ensure true
> separation of the IOCBs.
>
> The solution is to allow providing the IV data supplied as part of the
> plaintext/ciphertext. To do so, the AF_ALG interface treats the
> ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
> cipher operation together with the flag whether the operation should
> enable support for inline IV handling.
>
> If inline IV handling is enabled, the IV is expected to be the first
> part of the input plaintext/ciphertext. This IV is only used for one
> cipher operation and will not retained in the kernel for subsequent
> cipher operations.
>
> The inline IV handling support is only allowed to be enabled during
> the first sendmsg call for a context. Any subsequent sendmsg calls are
> not allowed to change the setting of the inline IV handling (either
> enable or disable it) as this would open up a race condition with the
> locking and unlocking of the ctx->ivlock mutex.
>
> The AEAD support required a slight re-arragning of the code, because
> obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
> access in _aead_recvmsg must be moved below the IV gathering.
>
> The AEAD code to find the first SG with data in the TX SGL is moved to a
> common function as it is required by the IV gathering function as well.
>
> This patch does not change the existing interface where user space is
> allowed to provide an IV via sendmsg. It only extends the interface by
> giving the user the choice to provide the IV either via sendmsg (the
> current approach) or as part of the data (the additional approach).
>
> Signed-off-by: Stephan Mueller <[email protected]>
It's not a 'bug' as such, but this does currently break the crypto-perf
test for aio and iiv in the libkcapi branch. We can perhaps make it
more forgiving...
I suggest we let noop cases where we set IIV on when it is already on
to not result in an error but to just be ignored.
Jonathan
> ---
> crypto/af_alg.c | 93 ++++++++++++++++++++++++++++++++++++++++++---
> crypto/algif_aead.c | 58 ++++++++++++----------------
> crypto/algif_skcipher.c | 12 +++---
> include/crypto/if_alg.h | 21 +++++++++-
> include/uapi/linux/if_alg.h | 6 ++-
> 5 files changed, 143 insertions(+), 47 deletions(-)
>
> diff --git a/crypto/af_alg.c b/crypto/af_alg.c
> index e7887621aa44..973233000d18 100644
> --- a/crypto/af_alg.c
> +++ b/crypto/af_alg.c
> @@ -14,6 +14,7 @@
>
> #include <linux/atomic.h>
> #include <crypto/if_alg.h>
> +#include <crypto/scatterwalk.h>
> #include <linux/crypto.h>
> #include <linux/init.h>
> #include <linux/kernel.h>
> @@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
> struct af_alg_control con = {};
> long copied = 0;
> bool enc = 0;
> + int iiv = ALG_IIV_DISABLE;
> bool init = 0;
> int err = 0;
>
> @@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
> return err;
>
> init = 1;
> - switch (con.op) {
> + switch (con.op & ALG_OP_CIPHER_MASK) {
> case ALG_OP_ENCRYPT:
> enc = 1;
> break;
> @@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
> return -EINVAL;
> }
>
> + if (con.op & ALG_OP_INLINE_IV)
> + iiv = ALG_IIV_USE;
> +
> if (con.iv && con.iv->ivlen != ivsize)
> return -EINVAL;
> }
> @@ -866,6 +871,19 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
>
> if (init) {
> ctx->enc = enc;
> +
> + /*
> + * IIV can only be enabled once with the first sendmsg call.
> + * This prevents a race in locking and unlocking the
> + * ctx->ivlock mutex.
> + */
> + if (ctx->iiv == ALG_IIV_UNSET) {
> + ctx->iiv = iiv;
> + } else if (iiv == ALG_IIV_USE) {
So the issue in kcapi is that we are running a tight loop around a test case
that does a certain number of aio messages. At the start of each run of
that performance loop we reinitialize.
There is an optmisation in there though which avoids an accept call (which
would have given a new context) if we have one available already.
The result is that we are reusing the context which already has this
set.
So, can we be a little less restrictive and let things through
if and only iff iiv = ctx->iiv - i.e. the set is a noop?
That will make life nicer for performance testing and other code
which just sets this without thinking on repeat calls.
> + err = -EINVAL;
> + goto unlock;
> + }
> +
> if (con.iv)
> memcpy(ctx->iv, con.iv->iv, ivsize);
>
> @@ -1031,6 +1049,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)
> struct sock *sk = areq->sk;
>
> af_alg_free_areq_sgls(areq);
> + if (areq->ivlen)
> + sock_kfree_s(sk, areq->iv, areq->ivlen);
> sock_kfree_s(sk, areq, areq->areqlen);
> }
> EXPORT_SYMBOL_GPL(af_alg_free_resources);
> @@ -1178,17 +1198,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
>
> /**
> - * af_alg_get_iv
> + * af_alg_get_tsgl_sg - get first SG entry with data from TX SGL
> + * @ctx [in] AF_ALG context with TX sgl
> + * @return pointer to SG with data or NULL if none found
> + */
> +struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx)
> +{
> + struct af_alg_tsgl *sgl, *tmp;
> + struct scatterlist *sg = NULL;
> + unsigned int i;
> +
> + list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
> + for (i = 0; i < sgl->cur; i++) {
> + struct scatterlist *process_sg = sgl->sg + i;
> +
> + if (!(process_sg->length) || !sg_page(process_sg))
> + continue;
> + sg = process_sg;
> + break;
> + }
> + if (sg)
> + break;
> + }
> +
> + return sg;
> +}
> +EXPORT_SYMBOL_GPL(af_alg_get_tsgl_sg);
> +
> +/**
> + * af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX
> *
> * @sk [in] AF_ALG socket
> + * @areq [in/out] request buffer that will receive the IV
> + * to be used for the cipher
> * @return 0 on success, < 0 on error
> */
> -int af_alg_get_iv(struct sock *sk)
> +int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
> {
> struct alg_sock *ask = alg_sk(sk);
> struct af_alg_ctx *ctx = ask->private;
> + struct scatterlist *sg;
> +
> + areq->iv = ctx->iv;
> + areq->ivlen = 0; // areq->iv will not be freed
> +
> + /* If cipher has no IV, no need to lock it or do IIV processing */
> + if (!ctx->ivlen)
> + return 0;
>
> - return mutex_lock_interruptible(&ctx->ivlock);
> + /* No inline IV, use ctx IV buffer and lock it */
> + if (ctx->iiv == ALG_IIV_DISABLE)
> + return mutex_lock_interruptible(&ctx->ivlock);
> +
> + /* Inline IV handling: There must be the IV data present. */
> + if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
> + return -EINVAL;
> +
> + areq->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
> + if (unlikely(!areq->iv))
> + return -ENOMEM;
> + areq->ivlen = ctx->ivlen; // areq->iv will be freed
> +
> + /* Get ivlen data from TX SGL and copy it into areq->iv. */
> + sg = af_alg_get_tsgl_sg(ctx);
> + if (!sg)
> + return -EFAULT;
> + scatterwalk_map_and_copy(areq->iv, sg, 0, ctx->ivlen, 0);
> + af_alg_pull_tsgl(sk, ctx->ivlen, NULL, 0);
> +
> + return 0;
> }
> EXPORT_SYMBOL_GPL(af_alg_get_iv);
>
> @@ -1202,7 +1280,12 @@ void af_alg_put_iv(struct sock *sk)
> struct alg_sock *ask = alg_sk(sk);
> struct af_alg_ctx *ctx = ask->private;
>
> - mutex_unlock(&ctx->ivlock);
> + /* To resemble af_alg_get_iv, do not merge the two branches. */
> + if (!ctx->ivlen)
> + return;
> +
> + if (ctx->iiv == ALG_IIV_DISABLE)
> + mutex_unlock(&ctx->ivlock);
> }
> EXPORT_SYMBOL_GPL(af_alg_put_iv);
>
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 402de50d4a58..623a0fc2b535 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> struct aead_tfm *aeadc = pask->private;
> struct crypto_aead *tfm = aeadc->aead;
> struct crypto_skcipher *null_tfm = aeadc->null_tfm;
> - unsigned int i, as = crypto_aead_authsize(tfm);
> + unsigned int as = crypto_aead_authsize(tfm);
> struct af_alg_async_req *areq;
> - struct af_alg_tsgl *tsgl, *tmp;
> struct scatterlist *rsgl_src, *tsgl_src = NULL;
> int err = 0;
> size_t used = 0; /* [in] TX bufs to be en/decrypted */
> @@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> return err;
> }
>
> - /*
> - * Data length provided by caller via sendmsg/sendpage that has not
> - * yet been processed.
> - */
> - used = ctx->used;
> -
> /*
> * Make sure sufficient data is present -- note, the same check is
> * is also present in sendmsg/sendpage. The checks in sendpage/sendmsg
> @@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> if (!aead_sufficient_data(sk))
> return -EINVAL;
>
> + /* Allocate cipher request for current operation. */
> + areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
> + crypto_aead_reqsize(tfm));
> + if (IS_ERR(areq))
> + return PTR_ERR(areq);
> +
> + /* Set areq->iv. */
> + err = af_alg_get_iv(sk, areq);
> + if (err)
> + goto free;
> +
> + /*
> + * Data length provided by caller via sendmsg/sendpage that has not
> + * yet been processed.
> + */
> + used = ctx->used;
> +
> /*
> * Calculate the minimum output buffer size holding the result of the
> * cipher operation. When encrypting data, the receiving buffer is
> @@ -153,16 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> */
> used -= ctx->aead_assoclen;
>
> - /* Allocate cipher request for current operation. */
> - areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
> - crypto_aead_reqsize(tfm));
> - if (IS_ERR(areq))
> - return PTR_ERR(areq);
> -
> - err = af_alg_get_iv(sk);
> - if (err)
> - goto free;
> -
> /* convert iovecs of output buffers into RX SGL */
> err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
> if (err)
> @@ -187,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> }
>
> processed = used + ctx->aead_assoclen;
> - list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) {
> - for (i = 0; i < tsgl->cur; i++) {
> - struct scatterlist *process_sg = tsgl->sg + i;
> -
> - if (!(process_sg->length) || !sg_page(process_sg))
> - continue;
> - tsgl_src = process_sg;
> - break;
> - }
> - if (tsgl_src)
> - break;
> - }
> + tsgl_src = af_alg_get_tsgl_sg(ctx);
> if (processed && !tsgl_src) {
> err = -EFAULT;
> goto unlock;
> @@ -286,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
>
> /* Initialize the crypto operation */
> aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src,
> - areq->first_rsgl.sgl.sg, used, ctx->iv);
> + areq->first_rsgl.sgl.sg, used, areq->iv);
> aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen);
> aead_request_set_tfm(&areq->cra_u.aead_req, tfm);
>
> @@ -554,19 +543,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
> struct aead_tfm *tfm = private;
> struct crypto_aead *aead = tfm->aead;
> unsigned int len = sizeof(*ctx);
> - unsigned int ivlen = crypto_aead_ivsize(aead);
>
> ctx = sock_kmalloc(sk, len, GFP_KERNEL);
> if (!ctx)
> return -ENOMEM;
> memset(ctx, 0, len);
>
> - ctx->iv = sock_kmalloc(sk, ivlen, GFP_KERNEL);
> + ctx->ivlen = crypto_aead_ivsize(aead);
> + ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
> if (!ctx->iv) {
> sock_kfree_s(sk, ctx, len);
> return -ENOMEM;
> }
> - memset(ctx->iv, 0, ivlen);
> + memset(ctx->iv, 0, ctx->ivlen);
> mutex_init(&ctx->ivlock);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> @@ -576,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
> ctx->more = 0;
> ctx->merge = 0;
> ctx->enc = 0;
> + ctx->iiv = ALG_IIV_UNSET;
> ctx->aead_assoclen = 0;
> crypto_init_wait(&ctx->wait);
>
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index b65b9a89d6bb..8e4d996ecb63 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -77,7 +77,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> if (IS_ERR(areq))
> return PTR_ERR(areq);
>
> - err = af_alg_get_iv(sk);
> + /* Set areq->iv */
> + err = af_alg_get_iv(sk, areq);
> if (err)
> goto free;
>
> @@ -116,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> /* Initialize the crypto operation */
> skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm);
> skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl,
> - areq->first_rsgl.sgl.sg, len, ctx->iv);
> + areq->first_rsgl.sgl.sg, len, areq->iv);
>
> if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
> /* AIO operation */
> @@ -350,14 +351,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
> if (!ctx)
> return -ENOMEM;
>
> - ctx->iv = sock_kmalloc(sk, crypto_skcipher_ivsize(tfm),
> - GFP_KERNEL);
> + ctx->ivlen = crypto_skcipher_ivsize(tfm);
> + ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
> if (!ctx->iv) {
> sock_kfree_s(sk, ctx, len);
> return -ENOMEM;
> }
>
> - memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
> + memset(ctx->iv, 0, ctx->ivlen);
> mutex_init(&ctx->ivlock);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> @@ -367,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
> ctx->more = 0;
> ctx->merge = 0;
> ctx->enc = 0;
> + ctx->iiv = ALG_IIV_UNSET;
> crypto_init_wait(&ctx->wait);
>
> ask->private = ctx;
> diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
> index c48eff27e5a9..fb128f953b36 100644
> --- a/include/crypto/if_alg.h
> +++ b/include/crypto/if_alg.h
> @@ -96,6 +96,14 @@ struct af_alg_rsgl {
> * @tsgl_entries: Number of entries in priv. TX SGL
> * @outlen: Number of output bytes generated by crypto op
> * @areqlen: Length of this data structure
> + * @iv: Buffer holding the IV for the cipher operation (this
> + * is either ctx->iv where ivlen is set to 0 or a newly
> + * allocated buffer where ivlen represents the size of
> + * this buffer).
> + * @ivlen: IV size -- if the IV buffer is to be freed during
> + * releasing of this data structure, ivlen is non-zero.
> + * If ivlen is zero, but iv is non-NULL, the iv buffer is
> + * not freed.
> * @cra_u: Cipher request
> */
> struct af_alg_async_req {
> @@ -112,6 +120,9 @@ struct af_alg_async_req {
> unsigned int outlen;
> unsigned int areqlen;
>
> + void *iv;
> + unsigned int ivlen;
> +
> union {
> struct aead_request aead_req;
> struct skcipher_request skcipher_req;
> @@ -128,6 +139,7 @@ struct af_alg_async_req {
> *
> * @tsgl_list: Link to TX SGL
> * @iv: IV for cipher operation
> + * @ivlen: IV size
> * @ivlock: Mutex to serialize access to the IV
> * @aead_assoclen: Length of AAD for AEAD cipher operations
> * @completion: Work queue for synchronous operation
> @@ -142,12 +154,14 @@ struct af_alg_async_req {
> * SG?
> * @enc: Cryptographic operation to be performed when
> * recvmsg is invoked.
> + * @iiv: Use inline IV: first part of TX data is IV
> * @len: Length of memory allocated for this data structure.
> */
> struct af_alg_ctx {
> struct list_head tsgl_list;
>
> void *iv;
> + unsigned int ivlen;
> struct mutex ivlock;
> size_t aead_assoclen;
>
> @@ -159,9 +173,13 @@ struct af_alg_ctx {
> bool more;
> bool merge;
> bool enc;
> + int iiv;
>
> unsigned int len;
> };
> +#define ALG_IIV_UNSET (-1)
> +#define ALG_IIV_DISABLE (0)
> +#define ALG_IIV_USE (1)
>
> int af_alg_register_type(const struct af_alg_type *type);
> int af_alg_unregister_type(const struct af_alg_type *type);
> @@ -255,7 +273,8 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
> int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> struct af_alg_async_req *areq, size_t maxsize,
> size_t *outlen);
> -int af_alg_get_iv(struct sock *sk);
> +int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
> void af_alg_put_iv(struct sock *sk);
> +struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
>
> #endif /* _CRYPTO_IF_ALG_H */
> diff --git a/include/uapi/linux/if_alg.h b/include/uapi/linux/if_alg.h
> index bc2bcdec377b..76a4db392cb6 100644
> --- a/include/uapi/linux/if_alg.h
> +++ b/include/uapi/linux/if_alg.h
> @@ -37,7 +37,9 @@ struct af_alg_iv {
> #define ALG_SET_AEAD_AUTHSIZE 5
>
> /* Operations */
> -#define ALG_OP_DECRYPT 0
> -#define ALG_OP_ENCRYPT 1
> +#define ALG_OP_DECRYPT 0x0
> +#define ALG_OP_ENCRYPT 0x1
> +#define ALG_OP_CIPHER_MASK 0xf
> +#define ALG_OP_INLINE_IV 0x10
>
> #endif /* _LINUX_IF_ALG_H */
Am Mittwoch, 7. Februar 2018, 14:54:08 CET schrieb Jonathan Cameron:
Hi Jonathan,
>
> It's not a 'bug' as such, but this does currently break the crypto-perf
> test for aio and iiv in the libkcapi branch. We can perhaps make it
> more forgiving...
I just pushed an update for the IIV branch of libkcapi that handles it
appropriately.
>
> I suggest we let noop cases where we set IIV on when it is already on
> to not result in an error but to just be ignored.
I do not like the kernel to behave differently on one and the same flag.
Though, the check could be more forgiving indeed: if the flag from user space
is set again and identical to what we already have, we would not report an
error. Only if it is different from what was already set, we report the error.
> >
> > +
> > + /*
> > + * IIV can only be enabled once with the first sendmsg call.
> > + * This prevents a race in locking and unlocking the
> > + * ctx->ivlock mutex.
> > + */
> > + if (ctx->iiv == ALG_IIV_UNSET) {
> > + ctx->iiv = iiv;
> > + } else if (iiv == ALG_IIV_USE) {
>
> So the issue in kcapi is that we are running a tight loop around a test case
> that does a certain number of aio messages. At the start of each run of
> that performance loop we reinitialize.
>
> There is an optmisation in there though which avoids an accept call (which
> would have given a new context) if we have one available already.
>
> The result is that we are reusing the context which already has this
> set.
>
> So, can we be a little less restrictive and let things through
> if and only iff iiv = ctx->iiv - i.e. the set is a noop?
Precisely, will do in the next version of the patch set.
Ciao
Stephan
On Wed, 7 Feb 2018 08:42:59 +0100
Stephan M?ller <[email protected]> wrote:
> Hi Herbert,
>
> Herbert, the patch 1 is meant for stable. However, this patch as is
> only applies to the new AF_ALG interface implementation. Though,
> the issue goes back to the first implementation of AIO support.
> Shall I try prepare a patch for the old AF_ALG implementation
> as well?
>
> Changes from v1:
>
> * integrate the inline IV and locking patch into one patch set
>
> * reverse the order of lock context IV patch and inline IV patch --
> the reason is to allow the first patch to be back-ported to stable
>
> * mark the first patch (locking of the context IV) as applicable to
> stable as there is an existing inconsistency which was demonstrated
> by Harsh with the Chelsio driver vs the AES-NI driver
>
> * modify the inline IV patch to have proper unlocking of the mutex
> in case of errors
>
> * prevent locking if no IV is defined by cipher
>
> * add a patch to allow crypto drivers to report whether they support
> serialization -- in this case the locking in AF_ALG shall be
> disabled
>
> * add a patch to inform the crypto drivers that their serialization
> support should actually be enabled and used because AF_ALG does not
> serialize the interdependent parallel AIO requests
>
> * streamline the code in patch 1 and 2 slightly
>
> I would like to ask the folks with real AIO hardware (Harsh, Jonathan)
> to test the patches. Especially, is the locking patch should be tested
> by Harsh as you have seen the issue with your hardware.
>
I've updated my driver to take advantage of this and it is working
pretty much the same as it was with the dirty hacks I was running before.
So Tested-by: Jonathan Cameron <[email protected]> with
that minor change to sensitivity to having it set to IIV multiple times.
Hmm. Naming of the parallel request probably needs some thought though.
Will reply to that patch.
> Thanks.
>
> Stephan Mueller (4):
> crypto: AF_ALG AIO - lock context IV
> crypto: AF_ALG - inline IV support
> crypto: AF_ALG - allow driver to serialize IV access
> crypto: add CRYPTO_TFM_REQ_PARALLEL flag
>
> crypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++-
> crypto/algif_aead.c | 86 +++++++++++++++++---------------
> crypto/algif_skcipher.c | 38 ++++++++++----
> include/crypto/if_alg.h | 37 ++++++++++++++
> include/linux/crypto.h | 16 ++++++
> include/uapi/linux/if_alg.h | 6 ++-
> 6 files changed, 249 insertions(+), 53 deletions(-)
>
On Wed, 7 Feb 2018 13:48:32 +0100
Stephan Mueller <[email protected]> wrote:
> Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan M?ller:
>
> Hi,
>
>
> > diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> > index 3970ad7f6fd0..da010405eea0 100644
> > --- a/crypto/algif_aead.c
> > +++ b/crypto/algif_aead.c
> > @@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct
> > msghdr *msg, size_t size) {
> > struct sock *sk = sock->sk;
> > struct alg_sock *ask = alg_sk(sk);
> > + struct af_alg_ctx *ctx = ask->private;
> > struct sock *psk = ask->parent;
> > struct alg_sock *pask = alg_sk(psk);
> > struct aead_tfm *aeadc = pask->private;
> > - struct crypto_aead *tfm = aeadc->aead;
> > - unsigned int ivsize = crypto_aead_ivsize(tfm);
> > + struct crypto_aead *aead = aeadc->aead;
> > + struct crypto_tfm *tfm = crypto_aead_tfm(aead);
> > + unsigned int ivsize = crypto_aead_ivsize(aead);
> > + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> > +
> > + if (ret < 0)
> > + return ret;
> >
> > - return af_alg_sendmsg(sock, msg, size, ivsize);
> > + if (ctx->iiv == ALG_IIV_USE)
>
> This should be ALG_IIV_DISABLE of course.
You say that, but my initial reading was that the core
was requesting that the driver do things in parallel
irrespective of whether the driver thought it was safe.
So I would think this was correct.
Definitely needs some documentation or a clearer name.
>
> > + tfm->crt_flags |= CRYPTO_TFM_REQ_PARALLEL;
> > +
> > + return ret;
> > }
> >
> > static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
> > diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> > index aee602a3ec24..225546666cf7 100644
> > --- a/crypto/algif_skcipher.c
> > +++ b/crypto/algif_skcipher.c
> > @@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct
> > msghdr *msg, {
> > struct sock *sk = sock->sk;
> > struct alg_sock *ask = alg_sk(sk);
> > + struct af_alg_ctx *ctx = ask->private;
> > struct sock *psk = ask->parent;
> > struct alg_sock *pask = alg_sk(psk);
> > - struct crypto_skcipher *tfm = pask->private;
> > - unsigned ivsize = crypto_skcipher_ivsize(tfm);
> > + struct crypto_skcipher *skc = pask->private;
> > + struct crypto_tfm *tfm = crypto_skcipher_tfm(skc);
> > + unsigned int ivsize = crypto_skcipher_ivsize(skc);
> > + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> > +
> > + if (ret < 0)
> > + return ret;
> >
> > - return af_alg_sendmsg(sock, msg, size, ivsize);
> > + if (ctx->iiv == ALG_IIV_USE)
>
> Dto.
>
> I will wait for some more comments before sending a new patch.
>
> Ciao
> Stephan
>
>
Am Mittwoch, 7. Februar 2018, 16:39:11 CET schrieb Jonathan Cameron:
Hi Jonathan,
> On Wed, 7 Feb 2018 13:48:32 +0100
>
> Stephan Mueller <[email protected]> wrote:
> > Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan M?ller:
> >
> > Hi,
> >
> > > diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> > > index 3970ad7f6fd0..da010405eea0 100644
> > > --- a/crypto/algif_aead.c
> > > +++ b/crypto/algif_aead.c
> > > @@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct
> > > msghdr *msg, size_t size) {
> > >
> > > struct sock *sk = sock->sk;
> > > struct alg_sock *ask = alg_sk(sk);
> > >
> > > + struct af_alg_ctx *ctx = ask->private;
> > >
> > > struct sock *psk = ask->parent;
> > > struct alg_sock *pask = alg_sk(psk);
> > > struct aead_tfm *aeadc = pask->private;
> > >
> > > - struct crypto_aead *tfm = aeadc->aead;
> > > - unsigned int ivsize = crypto_aead_ivsize(tfm);
> > > + struct crypto_aead *aead = aeadc->aead;
> > > + struct crypto_tfm *tfm = crypto_aead_tfm(aead);
> > > + unsigned int ivsize = crypto_aead_ivsize(aead);
> > > + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> > > +
> > > + if (ret < 0)
> > > + return ret;
> > >
> > > - return af_alg_sendmsg(sock, msg, size, ivsize);
> > > + if (ctx->iiv == ALG_IIV_USE)
> >
> > This should be ALG_IIV_DISABLE of course.
>
> You say that, but my initial reading was that the core
> was requesting that the driver do things in parallel
> irrespective of whether the driver thought it was safe.
> So I would think this was correct.
>
> Definitely needs some documentation or a clearer name.
How about:
ALG_IV_SERIAL_PROCESSING (was ALG_IIV_DISABLE)
ALG_IV_PARALLEL_PROCESSING (was ALG_IIV_USE)
Ciao
Stephan
On Wed, 7 Feb 2018 16:43:10 +0100
Stephan Mueller <[email protected]> wrote:
> Am Mittwoch, 7. Februar 2018, 16:39:11 CET schrieb Jonathan Cameron:
>
> Hi Jonathan,
>
> > On Wed, 7 Feb 2018 13:48:32 +0100
> >
> > Stephan Mueller <[email protected]> wrote:
> > > Am Mittwoch, 7. Februar 2018, 08:44:04 CET schrieb Stephan M?ller:
> > >
> > > Hi,
> > >
> > > > diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> > > > index 3970ad7f6fd0..da010405eea0 100644
> > > > --- a/crypto/algif_aead.c
> > > > +++ b/crypto/algif_aead.c
> > > > @@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct
> > > > msghdr *msg, size_t size) {
> > > >
> > > > struct sock *sk = sock->sk;
> > > > struct alg_sock *ask = alg_sk(sk);
> > > >
> > > > + struct af_alg_ctx *ctx = ask->private;
> > > >
> > > > struct sock *psk = ask->parent;
> > > > struct alg_sock *pask = alg_sk(psk);
> > > > struct aead_tfm *aeadc = pask->private;
> > > >
> > > > - struct crypto_aead *tfm = aeadc->aead;
> > > > - unsigned int ivsize = crypto_aead_ivsize(tfm);
> > > > + struct crypto_aead *aead = aeadc->aead;
> > > > + struct crypto_tfm *tfm = crypto_aead_tfm(aead);
> > > > + unsigned int ivsize = crypto_aead_ivsize(aead);
> > > > + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> > > > +
> > > > + if (ret < 0)
> > > > + return ret;
> > > >
> > > > - return af_alg_sendmsg(sock, msg, size, ivsize);
> > > > + if (ctx->iiv == ALG_IIV_USE)
> > >
> > > This should be ALG_IIV_DISABLE of course.
> >
> > You say that, but my initial reading was that the core
> > was requesting that the driver do things in parallel
> > irrespective of whether the driver thought it was safe.
> > So I would think this was correct.
> >
> > Definitely needs some documentation or a clearer name.
>
> How about:
>
> ALG_IV_SERIAL_PROCESSING (was ALG_IIV_DISABLE)
> ALG_IV_PARALLEL_PROCESSING (was ALG_IIV_USE)
>
Actually those were fine on the basis that inline iv is
obvious enough, it was CRYPTO_TFM_REQ_PARALLEL that
was causing me confusion.
Sorry, wasn't terribly clear on that!
Jonathan
> Ciao
> Stephan
>
>
Am Mittwoch, 7. Februar 2018, 17:14:12 CET schrieb Jonathan Cameron:
Hi Jonathan,
> Actually those were fine on the basis that inline iv is
> obvious enough, it was CRYPTO_TFM_REQ_PARALLEL that
> was causing me confusion.
>
> Sorry, wasn't terribly clear on that!
Ok, I will add some comments around that new REQ flag.
Ciao
Stephan
Crypto drivers may implement a streamlined serialization support for AIO
requests that is reported by the CRYPTO_ALG_SERIALIZES_IV_ACCESS flag to
the crypto user. When the user decides that he wants to send multiple
AIO requests concurrently and wants the crypto driver to handle the
serialization, the caller has to set CRYPTO_TFM_REQ_IV_SERIALIZE to notify
the crypto driver.
Only when this flag is enabled, the crypto driver shall apply its
serialization logic for handling IV updates between requests. If this
flag is not provided, the serialization logic shall not be applied by
the driver as the caller decides that it does not need it (because no
parallel AIO requests are sent) or that it performs its own
serialization.
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/algif_aead.c | 15 ++++++++++++---
crypto/algif_skcipher.c | 15 ++++++++++++---
include/linux/crypto.h | 1 +
3 files changed, 25 insertions(+), 6 deletions(-)
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 619147792cc9..5ec4dec6e6a1 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
struct sock *psk = ask->parent;
struct alg_sock *pask = alg_sk(psk);
struct aead_tfm *aeadc = pask->private;
- struct crypto_aead *tfm = aeadc->aead;
- unsigned int ivsize = crypto_aead_ivsize(tfm);
+ struct crypto_aead *aead = aeadc->aead;
+ struct crypto_tfm *tfm = crypto_aead_tfm(aead);
+ unsigned int ivsize = crypto_aead_ivsize(aead);
+ int ret = af_alg_sendmsg(sock, msg, size, ivsize);
+
+ if (ret < 0)
+ return ret;
- return af_alg_sendmsg(sock, msg, size, ivsize);
+ if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
+ tfm->crt_flags |= CRYPTO_TFM_REQ_IV_SERIALIZE;
+
+ return ret;
}
static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index cf27dda6a181..fd2a0ba32feb 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
{
struct sock *sk = sock->sk;
struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
struct sock *psk = ask->parent;
struct alg_sock *pask = alg_sk(psk);
- struct crypto_skcipher *tfm = pask->private;
- unsigned ivsize = crypto_skcipher_ivsize(tfm);
+ struct crypto_skcipher *skc = pask->private;
+ struct crypto_tfm *tfm = crypto_skcipher_tfm(skc);
+ unsigned int ivsize = crypto_skcipher_ivsize(skc);
+ int ret = af_alg_sendmsg(sock, msg, size, ivsize);
+
+ if (ret < 0)
+ return ret;
- return af_alg_sendmsg(sock, msg, size, ivsize);
+ if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
+ tfm->crt_flags |= CRYPTO_TFM_REQ_IV_SERIALIZE;
+
+ return ret;
}
static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 4860aa2c9be4..4d54f2b30692 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -133,6 +133,7 @@
#define CRYPTO_TFM_REQ_WEAK_KEY 0x00000100
#define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200
#define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400
+#define CRYPTO_TFM_REQ_IV_SERIALIZE 0x00000800
#define CRYPTO_TFM_RES_WEAK_KEY 0x00100000
#define CRYPTO_TFM_RES_BAD_KEY_LEN 0x00200000
#define CRYPTO_TFM_RES_BAD_KEY_SCHED 0x00400000
--
2.14.3
The kernel crypto API requires the caller to set an IV in the request
data structure. That request data structure shall define one particular
cipher operation. During the cipher operation, the IV is read by the
cipher implementation and eventually the potentially updated IV (e.g.
in case of CBC) is written back to the memory location the request data
structure points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is
stored in the AF_ALG context that is unique to one particular AF_ALG
socket. Note the analogy: an AF_ALG socket is like a TFM where one
recvmsg operation uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e.
with one recvmsg call, multiple IOVECs can be specified. Each
individual IOCB (derived from one IOVEC) implies that one request data
structure is created with the data to be processed by the cipher
implementation. The IV that was set with the sendmsg call is registered
with the request data structure before the cipher operation.
As of now, the individual IOCBs are serialized with respect to the IV
handling. This implies that the kernel does not perform a truly parallel
invocation of the cipher implementations. However, if the user wants to
perform cryptographic operations on multiple IOCBs where each IOCB is
truly independent from the other, parallel invocations are possible.
This would require that each IOCB provides its own IV to ensure true
separation of the IOCBs.
The solution is to allow providing the IV data supplied as part of the
plaintext/ciphertext. To do so, the AF_ALG interface treats the
ALG_SET_OP flag usable with sendmsg as a bit-array allowing to set the
cipher operation together with the flag whether the operation should
enable support for inline IV handling.
If inline IV handling is enabled, the IV is expected to be the first
part of the input plaintext/ciphertext. This IV is only used for one
cipher operation and will not retained in the kernel for subsequent
cipher operations.
The inline IV handling support is only allowed to be enabled during
the first sendmsg call for a context. Any subsequent sendmsg calls are
not allowed to change the setting of the inline IV handling (either
enable or disable it) as this would open up a race condition with the
locking and unlocking of the ctx->ivlock mutex.
The AEAD support required a slight re-arragning of the code, because
obtaining the IV implies that ctx->used is updated. Thus, the ctx->used
access in _aead_recvmsg must be moved below the IV gathering.
The AEAD code to find the first SG with data in the TX SGL is moved to a
common function as it is required by the IV gathering function as well.
This patch does not change the existing interface where user space is
allowed to provide an IV via sendmsg. It only extends the interface by
giving the user the choice to provide the IV either via sendmsg (the
current approach) or as part of the data (the additional approach).
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 93 ++++++++++++++++++++++++++++++++++++++++++---
crypto/algif_aead.c | 58 ++++++++++++----------------
crypto/algif_skcipher.c | 12 +++---
include/crypto/if_alg.h | 21 +++++++++-
include/uapi/linux/if_alg.h | 6 ++-
5 files changed, 143 insertions(+), 47 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index e7887621aa44..7f80dcfc12a6 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -14,6 +14,7 @@
#include <linux/atomic.h>
#include <crypto/if_alg.h>
+#include <crypto/scatterwalk.h>
#include <linux/crypto.h>
#include <linux/init.h>
#include <linux/kernel.h>
@@ -834,6 +835,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
struct af_alg_control con = {};
long copied = 0;
bool enc = 0;
+ int iiv = ALG_IV_SERIAL_PROCESSING;
bool init = 0;
int err = 0;
@@ -843,7 +845,7 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
return err;
init = 1;
- switch (con.op) {
+ switch (con.op & ALG_OP_CIPHER_MASK) {
case ALG_OP_ENCRYPT:
enc = 1;
break;
@@ -854,6 +856,9 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
return -EINVAL;
}
+ if (con.op & ALG_OP_INLINE_IV)
+ iiv = ALG_IV_PARALLEL_PROCESSING;
+
if (con.iv && con.iv->ivlen != ivsize)
return -EINVAL;
}
@@ -866,6 +871,19 @@ int af_alg_sendmsg(struct socket *sock, struct msghdr *msg, size_t size,
if (init) {
ctx->enc = enc;
+
+ /*
+ * IIV can only be enabled once with the first sendmsg call.
+ * This prevents a race in locking and unlocking the
+ * ctx->ivlock mutex.
+ */
+ if (ctx->iiv == ALG_IV_PROCESSING_UNDEF) {
+ ctx->iiv = iiv;
+ } else if (ctx->iiv != iiv) {
+ err = -EINVAL;
+ goto unlock;
+ }
+
if (con.iv)
memcpy(ctx->iv, con.iv->iv, ivsize);
@@ -1031,6 +1049,8 @@ void af_alg_free_resources(struct af_alg_async_req *areq)
struct sock *sk = areq->sk;
af_alg_free_areq_sgls(areq);
+ if (areq->ivlen)
+ sock_kfree_s(sk, areq->iv, areq->ivlen);
sock_kfree_s(sk, areq, areq->areqlen);
}
EXPORT_SYMBOL_GPL(af_alg_free_resources);
@@ -1178,17 +1198,75 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
/**
- * af_alg_get_iv
+ * af_alg_get_tsgl_sg - get first SG entry with data from TX SGL
+ * @ctx [in] AF_ALG context with TX sgl
+ * @return pointer to SG with data or NULL if none found
+ */
+struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx)
+{
+ struct af_alg_tsgl *sgl, *tmp;
+ struct scatterlist *sg = NULL;
+ unsigned int i;
+
+ list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) {
+ for (i = 0; i < sgl->cur; i++) {
+ struct scatterlist *process_sg = sgl->sg + i;
+
+ if (!(process_sg->length) || !sg_page(process_sg))
+ continue;
+ sg = process_sg;
+ break;
+ }
+ if (sg)
+ break;
+ }
+
+ return sg;
+}
+EXPORT_SYMBOL_GPL(af_alg_get_tsgl_sg);
+
+/**
+ * af_alg_get_iv - get IV from either TX inline IV data or from IV set at CTX
*
* @sk [in] AF_ALG socket
+ * @areq [in/out] request buffer that will receive the IV
+ * to be used for the cipher
* @return 0 on success, < 0 on error
*/
-int af_alg_get_iv(struct sock *sk)
+int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
{
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
+ struct scatterlist *sg;
+
+ areq->iv = ctx->iv;
+ areq->ivlen = 0; // areq->iv will not be freed
+
+ /* If cipher has no IV, no need to lock it or do IIV processing */
+ if (!ctx->ivlen)
+ return 0;
- return mutex_lock_interruptible(&ctx->ivlock);
+ /* No inline IV, use ctx IV buffer and lock it */
+ if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
+ return mutex_lock_interruptible(&ctx->ivlock);
+
+ /* Inline IV handling: There must be the IV data present. */
+ if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
+ return -EINVAL;
+
+ areq->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
+ if (unlikely(!areq->iv))
+ return -ENOMEM;
+ areq->ivlen = ctx->ivlen; // areq->iv will be freed
+
+ /* Get ivlen data from TX SGL and copy it into areq->iv. */
+ sg = af_alg_get_tsgl_sg(ctx);
+ if (!sg)
+ return -EFAULT;
+ scatterwalk_map_and_copy(areq->iv, sg, 0, ctx->ivlen, 0);
+ af_alg_pull_tsgl(sk, ctx->ivlen, NULL, 0);
+
+ return 0;
}
EXPORT_SYMBOL_GPL(af_alg_get_iv);
@@ -1202,7 +1280,12 @@ void af_alg_put_iv(struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
- mutex_unlock(&ctx->ivlock);
+ /* To resemble af_alg_get_iv, do not merge the two branches. */
+ if (!ctx->ivlen)
+ return;
+
+ if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
+ mutex_unlock(&ctx->ivlock);
}
EXPORT_SYMBOL_GPL(af_alg_put_iv);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 402de50d4a58..4c425effc5a5 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -100,9 +100,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
struct aead_tfm *aeadc = pask->private;
struct crypto_aead *tfm = aeadc->aead;
struct crypto_skcipher *null_tfm = aeadc->null_tfm;
- unsigned int i, as = crypto_aead_authsize(tfm);
+ unsigned int as = crypto_aead_authsize(tfm);
struct af_alg_async_req *areq;
- struct af_alg_tsgl *tsgl, *tmp;
struct scatterlist *rsgl_src, *tsgl_src = NULL;
int err = 0;
size_t used = 0; /* [in] TX bufs to be en/decrypted */
@@ -116,12 +115,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
return err;
}
- /*
- * Data length provided by caller via sendmsg/sendpage that has not
- * yet been processed.
- */
- used = ctx->used;
-
/*
* Make sure sufficient data is present -- note, the same check is
* is also present in sendmsg/sendpage. The checks in sendpage/sendmsg
@@ -134,6 +127,23 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (!aead_sufficient_data(sk))
return -EINVAL;
+ /* Allocate cipher request for current operation. */
+ areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
+ crypto_aead_reqsize(tfm));
+ if (IS_ERR(areq))
+ return PTR_ERR(areq);
+
+ /* Set areq->iv. */
+ err = af_alg_get_iv(sk, areq);
+ if (err)
+ goto free;
+
+ /*
+ * Data length provided by caller via sendmsg/sendpage that has not
+ * yet been processed.
+ */
+ used = ctx->used;
+
/*
* Calculate the minimum output buffer size holding the result of the
* cipher operation. When encrypting data, the receiving buffer is
@@ -153,16 +163,6 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
*/
used -= ctx->aead_assoclen;
- /* Allocate cipher request for current operation. */
- areq = af_alg_alloc_areq(sk, sizeof(struct af_alg_async_req) +
- crypto_aead_reqsize(tfm));
- if (IS_ERR(areq))
- return PTR_ERR(areq);
-
- err = af_alg_get_iv(sk);
- if (err)
- goto free;
-
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
if (err)
@@ -187,18 +187,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
}
processed = used + ctx->aead_assoclen;
- list_for_each_entry_safe(tsgl, tmp, &ctx->tsgl_list, list) {
- for (i = 0; i < tsgl->cur; i++) {
- struct scatterlist *process_sg = tsgl->sg + i;
-
- if (!(process_sg->length) || !sg_page(process_sg))
- continue;
- tsgl_src = process_sg;
- break;
- }
- if (tsgl_src)
- break;
- }
+ tsgl_src = af_alg_get_tsgl_sg(ctx);
if (processed && !tsgl_src) {
err = -EFAULT;
goto unlock;
@@ -286,7 +275,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
/* Initialize the crypto operation */
aead_request_set_crypt(&areq->cra_u.aead_req, rsgl_src,
- areq->first_rsgl.sgl.sg, used, ctx->iv);
+ areq->first_rsgl.sgl.sg, used, areq->iv);
aead_request_set_ad(&areq->cra_u.aead_req, ctx->aead_assoclen);
aead_request_set_tfm(&areq->cra_u.aead_req, tfm);
@@ -554,19 +543,19 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
struct aead_tfm *tfm = private;
struct crypto_aead *aead = tfm->aead;
unsigned int len = sizeof(*ctx);
- unsigned int ivlen = crypto_aead_ivsize(aead);
ctx = sock_kmalloc(sk, len, GFP_KERNEL);
if (!ctx)
return -ENOMEM;
memset(ctx, 0, len);
- ctx->iv = sock_kmalloc(sk, ivlen, GFP_KERNEL);
+ ctx->ivlen = crypto_aead_ivsize(aead);
+ ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
if (!ctx->iv) {
sock_kfree_s(sk, ctx, len);
return -ENOMEM;
}
- memset(ctx->iv, 0, ivlen);
+ memset(ctx->iv, 0, ctx->ivlen);
mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
@@ -576,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->iiv = ALG_IV_PROCESSING_UNDEF;
ctx->aead_assoclen = 0;
crypto_init_wait(&ctx->wait);
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index b65b9a89d6bb..f2a671b6ddf3 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -77,7 +77,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
- err = af_alg_get_iv(sk);
+ /* Set areq->iv */
+ err = af_alg_get_iv(sk, areq);
if (err)
goto free;
@@ -116,7 +117,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
/* Initialize the crypto operation */
skcipher_request_set_tfm(&areq->cra_u.skcipher_req, tfm);
skcipher_request_set_crypt(&areq->cra_u.skcipher_req, areq->tsgl,
- areq->first_rsgl.sgl.sg, len, ctx->iv);
+ areq->first_rsgl.sgl.sg, len, areq->iv);
if (msg->msg_iocb && !is_sync_kiocb(msg->msg_iocb)) {
/* AIO operation */
@@ -350,14 +351,14 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
if (!ctx)
return -ENOMEM;
- ctx->iv = sock_kmalloc(sk, crypto_skcipher_ivsize(tfm),
- GFP_KERNEL);
+ ctx->ivlen = crypto_skcipher_ivsize(tfm);
+ ctx->iv = sock_kmalloc(sk, ctx->ivlen, GFP_KERNEL);
if (!ctx->iv) {
sock_kfree_s(sk, ctx, len);
return -ENOMEM;
}
- memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
+ memset(ctx->iv, 0, ctx->ivlen);
mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
@@ -367,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->iiv = ALG_IV_PROCESSING_UNDEF;
crypto_init_wait(&ctx->wait);
ask->private = ctx;
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index c48eff27e5a9..ef114b72813a 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -96,6 +96,14 @@ struct af_alg_rsgl {
* @tsgl_entries: Number of entries in priv. TX SGL
* @outlen: Number of output bytes generated by crypto op
* @areqlen: Length of this data structure
+ * @iv: Buffer holding the IV for the cipher operation (this
+ * is either ctx->iv where ivlen is set to 0 or a newly
+ * allocated buffer where ivlen represents the size of
+ * this buffer).
+ * @ivlen: IV size -- if the IV buffer is to be freed during
+ * releasing of this data structure, ivlen is non-zero.
+ * If ivlen is zero, but iv is non-NULL, the iv buffer is
+ * not freed.
* @cra_u: Cipher request
*/
struct af_alg_async_req {
@@ -112,6 +120,9 @@ struct af_alg_async_req {
unsigned int outlen;
unsigned int areqlen;
+ void *iv;
+ unsigned int ivlen;
+
union {
struct aead_request aead_req;
struct skcipher_request skcipher_req;
@@ -128,6 +139,7 @@ struct af_alg_async_req {
*
* @tsgl_list: Link to TX SGL
* @iv: IV for cipher operation
+ * @ivlen: IV size
* @ivlock: Mutex to serialize access to the IV
* @aead_assoclen: Length of AAD for AEAD cipher operations
* @completion: Work queue for synchronous operation
@@ -142,12 +154,14 @@ struct af_alg_async_req {
* SG?
* @enc: Cryptographic operation to be performed when
* recvmsg is invoked.
+ * @iiv: Use inline IV: first part of TX data is IV
* @len: Length of memory allocated for this data structure.
*/
struct af_alg_ctx {
struct list_head tsgl_list;
void *iv;
+ unsigned int ivlen;
struct mutex ivlock;
size_t aead_assoclen;
@@ -159,9 +173,13 @@ struct af_alg_ctx {
bool more;
bool merge;
bool enc;
+ int iiv;
unsigned int len;
};
+#define ALG_IV_PROCESSING_UNDEF (-1)
+#define ALG_IV_SERIAL_PROCESSING (0)
+#define ALG_IV_PARALLEL_PROCESSING (1)
int af_alg_register_type(const struct af_alg_type *type);
int af_alg_unregister_type(const struct af_alg_type *type);
@@ -255,7 +273,8 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
struct af_alg_async_req *areq, size_t maxsize,
size_t *outlen);
-int af_alg_get_iv(struct sock *sk);
+int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq);
void af_alg_put_iv(struct sock *sk);
+struct scatterlist *af_alg_get_tsgl_sg(struct af_alg_ctx *ctx);
#endif /* _CRYPTO_IF_ALG_H */
diff --git a/include/uapi/linux/if_alg.h b/include/uapi/linux/if_alg.h
index bc2bcdec377b..76a4db392cb6 100644
--- a/include/uapi/linux/if_alg.h
+++ b/include/uapi/linux/if_alg.h
@@ -37,7 +37,9 @@ struct af_alg_iv {
#define ALG_SET_AEAD_AUTHSIZE 5
/* Operations */
-#define ALG_OP_DECRYPT 0
-#define ALG_OP_ENCRYPT 1
+#define ALG_OP_DECRYPT 0x0
+#define ALG_OP_ENCRYPT 0x1
+#define ALG_OP_CIPHER_MASK 0xf
+#define ALG_OP_INLINE_IV 0x10
#endif /* _LINUX_IF_ALG_H */
--
2.14.3
The mutex in AF_ALG to serialize access to the IV ensures full
serialization of requests sent to the crypto driver.
However, the hardware may implement serialization to the IV such that
preparation work without touching the IV can already happen while the IV
is processed by another operation. This may speed up the AIO processing.
The following ASCII art demonstrates this.
AF_ALG mutex handling implements the following logic:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
NOTHING BLOCKING (lock mutex) |
| Queued on Mutex
[BUILD / MAP HW DESCS] |
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] |
| |
[unlock mutex] Nothing Blocking (lock mutex)
| |
Don't care beyond here. [BUILD / MAP HW DESCS]
|
[Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.
A crypto driver may implement the following serialization:
[REQUEST 1 from userspace] [REQUEST 2 from userspace]
| |
[AF_ALG/SOCKET] [AF_ALG/SOCKET]
| |
[BUILD / MAP HW DESCS] [BUILD / MAP HW DESCS]
| |
NOTHING BLOCKING IV in flight (enqueue sw queue)
| |
[Place in HW Queue] |
| |
[Process] |
| |
[Interrupt] |
| |
[Respond and update IV] Nothing Blocking (dequeue sw queue)
| |
Don't care beyond here. [Place in HW Queue]
|
[Process]
|
[Interrupt]
|
[Respond and update IV]
|
Don't care beyond here.
If the driver implements its own serialization (i.e. AF_ALG does not
serialize the access to the IV), the crypto implementation must set the
flag CRYPTO_ALG_SERIALIZES_IV_ACCESS.
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 13 ++++++++-----
crypto/algif_aead.c | 1 +
crypto/algif_skcipher.c | 1 +
include/crypto/if_alg.h | 13 +++++++++++++
include/linux/crypto.h | 15 +++++++++++++++
5 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 7f80dcfc12a6..56b4da65025a 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -1247,8 +1247,10 @@ int af_alg_get_iv(struct sock *sk, struct af_alg_async_req *areq)
return 0;
/* No inline IV, use ctx IV buffer and lock it */
- if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
- return mutex_lock_interruptible(&ctx->ivlock);
+ if (ctx->iiv == ALG_IV_SERIAL_PROCESSING) {
+ return (ctx->lock_iv) ?
+ mutex_lock_interruptible(&ctx->ivlock) : 0;
+ }
/* Inline IV handling: There must be the IV data present. */
if (ctx->used < ctx->ivlen || list_empty(&ctx->tsgl_list))
@@ -1280,12 +1282,13 @@ void af_alg_put_iv(struct sock *sk)
struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private;
- /* To resemble af_alg_get_iv, do not merge the two branches. */
if (!ctx->ivlen)
return;
- if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
- mutex_unlock(&ctx->ivlock);
+ if (ctx->iiv == ALG_IV_SERIAL_PROCESSING) {
+ if (ctx->lock_iv)
+ mutex_unlock(&ctx->ivlock);
+ }
}
EXPORT_SYMBOL_GPL(af_alg_put_iv);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 4c425effc5a5..619147792cc9 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -565,6 +565,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->lock_iv = af_alg_lock_iv(crypto_aead_tfm(aead));
ctx->iiv = ALG_IV_PROCESSING_UNDEF;
ctx->aead_assoclen = 0;
crypto_init_wait(&ctx->wait);
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index f2a671b6ddf3..cf27dda6a181 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -368,6 +368,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
ctx->more = 0;
ctx->merge = 0;
ctx->enc = 0;
+ ctx->lock_iv = af_alg_lock_iv(crypto_skcipher_tfm(tfm));
ctx->iiv = ALG_IV_PROCESSING_UNDEF;
crypto_init_wait(&ctx->wait);
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index ef114b72813a..4b48a48464c3 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -154,6 +154,7 @@ struct af_alg_async_req {
* SG?
* @enc: Cryptographic operation to be performed when
* recvmsg is invoked.
+ * @lock_iv: Shall IV be locked?
* @iiv: Use inline IV: first part of TX data is IV
* @len: Length of memory allocated for this data structure.
*/
@@ -173,6 +174,7 @@ struct af_alg_ctx {
bool more;
bool merge;
bool enc;
+ bool lock_iv;
int iiv;
unsigned int len;
@@ -251,6 +253,17 @@ static inline bool af_alg_readable(struct sock *sk)
return PAGE_SIZE <= af_alg_rcvbuf(sk);
}
+/**
+ * Shall access to the ctx->iv be serialized using a mutex?
+ *
+ * @tfm TFM of to be used for cipher operation
+ * @return true => lock, false => do not lock
+ */
+static inline bool af_alg_lock_iv(struct crypto_tfm *tfm)
+{
+ return !(crypto_tfm_alg_flags(tfm) & CRYPTO_ALG_SERIALIZES_IV_ACCESS);
+}
+
int af_alg_alloc_tsgl(struct sock *sk);
unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset);
void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst,
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 7e6e84cf6383..4860aa2c9be4 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -112,6 +112,16 @@
*/
#define CRYPTO_ALG_OPTIONAL_KEY 0x00004000
+/*
+ * Set if the algorithm implementation provides its own serialization
+ * of IV accesses. This prevents the AF_ALG interface to serialize
+ * cipher requests. This flag allows the implementation to implement
+ * a more streamlined IV serialization logic, such as only serializing
+ * the submission of a request to hardware whereas the preparation steps
+ * for sending data to hardware are not serialized.
+ */
+#define CRYPTO_ALG_SERIALIZES_IV_ACCESS 0x00008000
+
/*
* Transform masks and values (for crt_flags).
*/
@@ -674,6 +684,11 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK;
}
+static inline u32 crypto_tfm_alg_flags(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags;
+}
+
static inline unsigned int crypto_tfm_alg_blocksize(struct crypto_tfm *tfm)
{
return tfm->__crt_alg->cra_blocksize;
--
2.14.3
The kernel crypto API requires the caller to set an IV in the request data
structure. That request data structure shall define one particular cipher
operation. During the cipher operation, the IV is read by the cipher
implementation and eventually the potentially updated IV (e.g. in case of
CBC) is written back to the memory location the request data structure
points to.
AF_ALG allows setting the IV with a sendmsg request, where the IV is stored
in the AF_ALG context that is unique to one particular AF_ALG socket. Note
the analogy: an AF_ALG socket is like a TFM where one recvmsg operation
uses one request with the TFM from the socket.
AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with
one recvmsg call, multiple IOVECs can be specified. Each individual IOCB
(derived from one IOVEC) implies that one request data structure is
created with the data to be processed by the cipher implementation. The
IV that was set with the sendmsg call is registered with the request data
structure before the cipher operation.
In case of an AIO operation, the cipher operation invocation returns
immediately, queuing the request to the hardware. While the AIO request is
processed by the hardware, recvmsg processes the next IOVEC for which
another request is created. Again, the IV buffer from the AF_ALG socket
context is registered with the new request and the cipher operation is
invoked.
You may now see that there is a potential race condition regarding the IV
handling, because there is *no* separate IV buffer for the different
requests. This is nicely demonstrated with libkcapi using the following
command which creates an AIO request with two IOCBs each encrypting one
AES block in CBC mode:
kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
When the first AIO request finishes before the 2nd AIO request is
processed, the returned value is:
8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
I.e. two blocks where the IV output from the first request is the IV input
to the 2nd block.
In case the first AIO request is not completed before the 2nd request
commences, the result is two identical AES blocks (i.e. both use the same
IV):
8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
This inconsistent result may even lead to the conclusion that there can be
a memory corruption in the IV buffer if both AIO requests write to the IV
buffer at the same time.
As the AF_ALG interface is used by user space, a mutex provides the
serialization which puts the caller to sleep in case a previous IOCB
processing is not yet finished.
If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
of processing arriving requests ensures that the blocked IOCBs are
unblocked in the right order.
CC: <[email protected]> #4.14
Fixes: e870456d8e7c8 ("crypto: algif_skcipher - overhaul memory management")
Fixes: d887c52d6ae43 ("crypto: algif_aead - overhaul memory management")
Signed-off-by: Stephan Mueller <[email protected]>
---
crypto/af_alg.c | 31 +++++++++++++++++++++++++++++++
crypto/algif_aead.c | 20 +++++++++++++-------
crypto/algif_skcipher.c | 12 +++++++++---
include/crypto/if_alg.h | 5 +++++
4 files changed, 58 insertions(+), 10 deletions(-)
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 5231f421ad00..e7887621aa44 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -1051,6 +1051,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)
struct kiocb *iocb = areq->iocb;
unsigned int resultlen;
+ af_alg_put_iv(sk);
+
/* Buffer size written by crypto operation. */
resultlen = areq->outlen;
@@ -1175,6 +1177,35 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
}
EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
+/**
+ * af_alg_get_iv
+ *
+ * @sk [in] AF_ALG socket
+ * @return 0 on success, < 0 on error
+ */
+int af_alg_get_iv(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+
+ return mutex_lock_interruptible(&ctx->ivlock);
+}
+EXPORT_SYMBOL_GPL(af_alg_get_iv);
+
+/**
+ * af_alg_put_iv - release lock on IV in case CTX IV is used
+ *
+ * @sk [in] AF_ALG socket
+ */
+void af_alg_put_iv(struct sock *sk)
+{
+ struct alg_sock *ask = alg_sk(sk);
+ struct af_alg_ctx *ctx = ask->private;
+
+ mutex_unlock(&ctx->ivlock);
+}
+EXPORT_SYMBOL_GPL(af_alg_put_iv);
+
static int __init af_alg_init(void)
{
int err = proto_register(&alg_proto, 0);
diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
index 4b07edd5a9ff..402de50d4a58 100644
--- a/crypto/algif_aead.c
+++ b/crypto/algif_aead.c
@@ -159,10 +159,14 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
+ err = af_alg_get_iv(sk);
+ if (err)
+ goto free;
+
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
if (err)
- goto free;
+ goto unlock;
/*
* Ensure output buffer is sufficiently large. If the caller provides
@@ -176,7 +180,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
if (used < less) {
err = -EINVAL;
- goto free;
+ goto unlock;
}
used -= less;
outlen -= less;
@@ -197,7 +201,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
}
if (processed && !tsgl_src) {
err = -EFAULT;
- goto free;
+ goto unlock;
}
/*
@@ -230,7 +234,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
err = crypto_aead_copy_sgl(null_tfm, tsgl_src,
areq->first_rsgl.sgl.sg, processed);
if (err)
- goto free;
+ goto unlock;
af_alg_pull_tsgl(sk, processed, NULL, 0);
} else {
/*
@@ -248,7 +252,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
err = crypto_aead_copy_sgl(null_tfm, tsgl_src,
areq->first_rsgl.sgl.sg, outlen);
if (err)
- goto free;
+ goto unlock;
/* Create TX SGL for tag and chain it to RX SGL. */
areq->tsgl_entries = af_alg_count_tsgl(sk, processed,
@@ -260,7 +264,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
GFP_KERNEL);
if (!areq->tsgl) {
err = -ENOMEM;
- goto free;
+ goto unlock;
}
sg_init_table(areq->tsgl, areq->tsgl_entries);
@@ -316,7 +320,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
&ctx->wait);
}
-
+unlock:
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -562,6 +567,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
return -ENOMEM;
}
memset(ctx->iv, 0, ivlen);
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index c88e5e4cd6a6..b65b9a89d6bb 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -77,10 +77,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
if (IS_ERR(areq))
return PTR_ERR(areq);
+ err = af_alg_get_iv(sk);
+ if (err)
+ goto free;
+
/* convert iovecs of output buffers into RX SGL */
err = af_alg_get_rsgl(sk, msg, flags, areq, -1, &len);
if (err)
- goto free;
+ goto unlock;
/* Process only as much RX buffers for which we have TX data */
if (len > ctx->used)
@@ -104,7 +108,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
GFP_KERNEL);
if (!areq->tsgl) {
err = -ENOMEM;
- goto free;
+ goto unlock;
}
sg_init_table(areq->tsgl, areq->tsgl_entries);
af_alg_pull_tsgl(sk, len, areq->tsgl, 0);
@@ -146,7 +150,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
&ctx->wait);
}
-
+unlock:
+ af_alg_put_iv(sk);
free:
af_alg_free_resources(areq);
@@ -353,6 +358,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
}
memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
+ mutex_init(&ctx->ivlock);
INIT_LIST_HEAD(&ctx->tsgl_list);
ctx->len = len;
diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
index f38227a78eae..c48eff27e5a9 100644
--- a/include/crypto/if_alg.h
+++ b/include/crypto/if_alg.h
@@ -19,6 +19,7 @@
#include <linux/scatterlist.h>
#include <linux/types.h>
#include <linux/atomic.h>
+#include <linux/mutex.h>
#include <net/sock.h>
#include <crypto/aead.h>
@@ -127,6 +128,7 @@ struct af_alg_async_req {
*
* @tsgl_list: Link to TX SGL
* @iv: IV for cipher operation
+ * @ivlock: Mutex to serialize access to the IV
* @aead_assoclen: Length of AAD for AEAD cipher operations
* @completion: Work queue for synchronous operation
* @used: TX bytes sent to kernel. This variable is used to
@@ -146,6 +148,7 @@ struct af_alg_ctx {
struct list_head tsgl_list;
void *iv;
+ struct mutex ivlock;
size_t aead_assoclen;
struct crypto_wait wait;
@@ -252,5 +255,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
struct af_alg_async_req *areq, size_t maxsize,
size_t *outlen);
+int af_alg_get_iv(struct sock *sk);
+void af_alg_put_iv(struct sock *sk);
#endif /* _CRYPTO_IF_ALG_H */
--
2.14.3
Hi,
Herbert, the patch 1 is meant for stable. However, this patch as is
only applies to the new AF_ALG interface implementation. Though,
the issue goes back to the first implementation of AIO support.
Shall I try prepare a patch for the old AF_ALG implementation
as well?
Changes from v2:
* rename ALG_IIV flags into ALG_IV_...
* rename CRYPTO_TFM_REQ_PARALLEL into CRYPTO_TFM_REQ_IV_SERIALIZE
* fix branch in patch 4 to add CRYPTO_TFM_REQ_IV_SERIALIZE flag when
ctx->iiv == ALG_IV_SERIAL_PROCESSING
* fix patch description of patch 4
Stephan Mueller (4):
crypto: AF_ALG AIO - lock context IV
crypto: AF_ALG - inline IV support
crypto: AF_ALG - allow driver to serialize IV access
crypto: add CRYPTO_TFM_REQ_IV_SERIALIZE flag
crypto/af_alg.c | 119 +++++++++++++++++++++++++++++++++++++++++++-
crypto/algif_aead.c | 86 +++++++++++++++++---------------
crypto/algif_skcipher.c | 38 ++++++++++----
include/crypto/if_alg.h | 37 ++++++++++++++
include/linux/crypto.h | 16 ++++++
include/uapi/linux/if_alg.h | 6 ++-
6 files changed, 249 insertions(+), 53 deletions(-)
--
2.14.3
On 10-02-2018 03:33, Stephan Müller wrote:
> The kernel crypto API requires the caller to set an IV in the request data
> structure. That request data structure shall define one particular cipher
> operation. During the cipher operation, the IV is read by the cipher
> implementation and eventually the potentially updated IV (e.g. in case of
> CBC) is written back to the memory location the request data structure
> points to.
>
> AF_ALG allows setting the IV with a sendmsg request, where the IV is stored
> in the AF_ALG context that is unique to one particular AF_ALG socket. Note
> the analogy: an AF_ALG socket is like a TFM where one recvmsg operation
> uses one request with the TFM from the socket.
>
> AF_ALG these days supports AIO operations with multiple IOCBs. I.e. with
> one recvmsg call, multiple IOVECs can be specified. Each individual IOCB
> (derived from one IOVEC) implies that one request data structure is
> created with the data to be processed by the cipher implementation. The
> IV that was set with the sendmsg call is registered with the request data
> structure before the cipher operation.
>
> In case of an AIO operation, the cipher operation invocation returns
> immediately, queuing the request to the hardware. While the AIO request is
> processed by the hardware, recvmsg processes the next IOVEC for which
> another request is created. Again, the IV buffer from the AF_ALG socket
> context is registered with the new request and the cipher operation is
> invoked.
>
> You may now see that there is a potential race condition regarding the IV
> handling, because there is *no* separate IV buffer for the different
> requests. This is nicely demonstrated with libkcapi using the following
> command which creates an AIO request with two IOCBs each encrypting one
> AES block in CBC mode:
>
> kcapi -d 2 -x 9 -e -c "cbc(aes)" -k
> 8d7dd9b0170ce0b5f2f8e1aa768e01e91da8bfc67fd486d081b28254c99eb423 -i
> 7fbc02ebf5b93322329df9bfccb635af -p 48981da18e4bb9ef7e2e3162d16b1910
>
> When the first AIO request finishes before the 2nd AIO request is
> processed, the returned value is:
>
> 8b19050f66582cb7f7e4b6c873819b7108afa0eaa7de29bac7d903576b674c32
>
> I.e. two blocks where the IV output from the first request is the IV input
> to the 2nd block.
>
> In case the first AIO request is not completed before the 2nd request
> commences, the result is two identical AES blocks (i.e. both use the same
> IV):
>
> 8b19050f66582cb7f7e4b6c873819b718b19050f66582cb7f7e4b6c873819b71
>
> This inconsistent result may even lead to the conclusion that there can be
> a memory corruption in the IV buffer if both AIO requests write to the IV
> buffer at the same time.
>
> As the AF_ALG interface is used by user space, a mutex provides the
> serialization which puts the caller to sleep in case a previous IOCB
> processing is not yet finished.
>
> If multiple IOCBs arrive that all are blocked, the mutex' FIFO operation
> of processing arriving requests ensures that the blocked IOCBs are
> unblocked in the right order.
Hi Stephen,
Patch set is working fine with chelsio Driver.
Do we really need IV locking mechanism for AEAD algo because AEAD algo's don't support Partial mode operation and Driver are not updating(atleast Chelsio) IV's on AEAD request completions.
>
> CC: <[email protected]> #4.14
> Fixes: e870456d8e7c8 ("crypto: algif_skcipher - overhaul memory management")
> Fixes: d887c52d6ae43 ("crypto: algif_aead - overhaul memory management")
> Signed-off-by: Stephan Mueller <[email protected]>
> ---
> crypto/af_alg.c | 31 +++++++++++++++++++++++++++++++
> crypto/algif_aead.c | 20 +++++++++++++-------
> crypto/algif_skcipher.c | 12 +++++++++---
> include/crypto/if_alg.h | 5 +++++
> 4 files changed, 58 insertions(+), 10 deletions(-)
>
> diff --git a/crypto/af_alg.c b/crypto/af_alg.c
> index 5231f421ad00..e7887621aa44 100644
> --- a/crypto/af_alg.c
> +++ b/crypto/af_alg.c
> @@ -1051,6 +1051,8 @@ void af_alg_async_cb(struct crypto_async_request *_req, int err)
> struct kiocb *iocb = areq->iocb;
> unsigned int resultlen;
>
> + af_alg_put_iv(sk);
> +
> /* Buffer size written by crypto operation. */
> resultlen = areq->outlen;
>
> @@ -1175,6 +1177,35 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> }
> EXPORT_SYMBOL_GPL(af_alg_get_rsgl);
>
> +/**
> + * af_alg_get_iv
> + *
> + * @sk [in] AF_ALG socket
> + * @return 0 on success, < 0 on error
> + */
> +int af_alg_get_iv(struct sock *sk)
> +{
> + struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> +
> + return mutex_lock_interruptible(&ctx->ivlock);
> +}
> +EXPORT_SYMBOL_GPL(af_alg_get_iv);
> +
> +/**
> + * af_alg_put_iv - release lock on IV in case CTX IV is used
> + *
> + * @sk [in] AF_ALG socket
> + */
> +void af_alg_put_iv(struct sock *sk)
> +{
> + struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> +
> + mutex_unlock(&ctx->ivlock);
> +}
> +EXPORT_SYMBOL_GPL(af_alg_put_iv);
> +
> static int __init af_alg_init(void)
> {
> int err = proto_register(&alg_proto, 0);
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 4b07edd5a9ff..402de50d4a58 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -159,10 +159,14 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> if (IS_ERR(areq))
> return PTR_ERR(areq);
>
> + err = af_alg_get_iv(sk);
> + if (err)
> + goto free;
> +
> /* convert iovecs of output buffers into RX SGL */
> err = af_alg_get_rsgl(sk, msg, flags, areq, outlen, &usedpages);
> if (err)
> - goto free;
> + goto unlock;
>
> /*
> * Ensure output buffer is sufficiently large. If the caller provides
> @@ -176,7 +180,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
>
> if (used < less) {
> err = -EINVAL;
> - goto free;
> + goto unlock;
> }
> used -= less;
> outlen -= less;
> @@ -197,7 +201,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> }
> if (processed && !tsgl_src) {
> err = -EFAULT;
> - goto free;
> + goto unlock;
> }
>
> /*
> @@ -230,7 +234,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> err = crypto_aead_copy_sgl(null_tfm, tsgl_src,
> areq->first_rsgl.sgl.sg, processed);
> if (err)
> - goto free;
> + goto unlock;
> af_alg_pull_tsgl(sk, processed, NULL, 0);
> } else {
> /*
> @@ -248,7 +252,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> err = crypto_aead_copy_sgl(null_tfm, tsgl_src,
> areq->first_rsgl.sgl.sg, outlen);
> if (err)
> - goto free;
> + goto unlock;
>
> /* Create TX SGL for tag and chain it to RX SGL. */
> areq->tsgl_entries = af_alg_count_tsgl(sk, processed,
> @@ -260,7 +264,7 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> GFP_KERNEL);
> if (!areq->tsgl) {
> err = -ENOMEM;
> - goto free;
> + goto unlock;
> }
> sg_init_table(areq->tsgl, areq->tsgl_entries);
>
> @@ -316,7 +320,8 @@ static int _aead_recvmsg(struct socket *sock, struct msghdr *msg,
> &ctx->wait);
> }
>
> -
> +unlock:
> + af_alg_put_iv(sk);
> free:
> af_alg_free_resources(areq);
>
> @@ -562,6 +567,7 @@ static int aead_accept_parent_nokey(void *private, struct sock *sk)
> return -ENOMEM;
> }
> memset(ctx->iv, 0, ivlen);
> + mutex_init(&ctx->ivlock);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> ctx->len = len;
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index c88e5e4cd6a6..b65b9a89d6bb 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -77,10 +77,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> if (IS_ERR(areq))
> return PTR_ERR(areq);
>
> + err = af_alg_get_iv(sk);
> + if (err)
> + goto free;
> +
> /* convert iovecs of output buffers into RX SGL */
> err = af_alg_get_rsgl(sk, msg, flags, areq, -1, &len);
> if (err)
> - goto free;
> + goto unlock;
>
> /* Process only as much RX buffers for which we have TX data */
> if (len > ctx->used)
> @@ -104,7 +108,7 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> GFP_KERNEL);
> if (!areq->tsgl) {
> err = -ENOMEM;
> - goto free;
> + goto unlock;
> }
> sg_init_table(areq->tsgl, areq->tsgl_entries);
> af_alg_pull_tsgl(sk, len, areq->tsgl, 0);
> @@ -146,7 +150,8 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> &ctx->wait);
> }
>
> -
> +unlock:
> + af_alg_put_iv(sk);
> free:
> af_alg_free_resources(areq);
>
> @@ -353,6 +358,7 @@ static int skcipher_accept_parent_nokey(void *private, struct sock *sk)
> }
>
> memset(ctx->iv, 0, crypto_skcipher_ivsize(tfm));
> + mutex_init(&ctx->ivlock);
>
> INIT_LIST_HEAD(&ctx->tsgl_list);
> ctx->len = len;
> diff --git a/include/crypto/if_alg.h b/include/crypto/if_alg.h
> index f38227a78eae..c48eff27e5a9 100644
> --- a/include/crypto/if_alg.h
> +++ b/include/crypto/if_alg.h
> @@ -19,6 +19,7 @@
> #include <linux/scatterlist.h>
> #include <linux/types.h>
> #include <linux/atomic.h>
> +#include <linux/mutex.h>
> #include <net/sock.h>
>
> #include <crypto/aead.h>
> @@ -127,6 +128,7 @@ struct af_alg_async_req {
> *
> * @tsgl_list: Link to TX SGL
> * @iv: IV for cipher operation
> + * @ivlock: Mutex to serialize access to the IV
> * @aead_assoclen: Length of AAD for AEAD cipher operations
> * @completion: Work queue for synchronous operation
> * @used: TX bytes sent to kernel. This variable is used to
> @@ -146,6 +148,7 @@ struct af_alg_ctx {
> struct list_head tsgl_list;
>
> void *iv;
> + struct mutex ivlock;
> size_t aead_assoclen;
>
> struct crypto_wait wait;
> @@ -252,5 +255,7 @@ struct af_alg_async_req *af_alg_alloc_areq(struct sock *sk,
> int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
> struct af_alg_async_req *areq, size_t maxsize,
> size_t *outlen);
> +int af_alg_get_iv(struct sock *sk);
> +void af_alg_put_iv(struct sock *sk);
>
> #endif /* _CRYPTO_IF_ALG_H */
On 10-02-2018 03:34, Stephan Müller wrote:
> Crypto drivers may implement a streamlined serialization support for AIO
> requests that is reported by the CRYPTO_ALG_SERIALIZES_IV_ACCESS flag to
> the crypto user. When the user decides that he wants to send multiple
> AIO requests concurrently and wants the crypto driver to handle the
> serialization, the caller has to set CRYPTO_TFM_REQ_IV_SERIALIZE to notify
> the crypto driver.
Will crypto_alloc_* API takes cares of this flag?. For Kernel Crypto user IV Synchronization logic depends on weather tfm allocated supports IV Serialisation or not.
>
> Only when this flag is enabled, the crypto driver shall apply its
> serialization logic for handling IV updates between requests. If this
> flag is not provided, the serialization logic shall not be applied by
> the driver as the caller decides that it does not need it (because no
> parallel AIO requests are sent) or that it performs its own
> serialization.
>
> Signed-off-by: Stephan Mueller <[email protected]>
> ---
> crypto/algif_aead.c | 15 ++++++++++++---
> crypto/algif_skcipher.c | 15 ++++++++++++---
> include/linux/crypto.h | 1 +
> 3 files changed, 25 insertions(+), 6 deletions(-)
>
> diff --git a/crypto/algif_aead.c b/crypto/algif_aead.c
> index 619147792cc9..5ec4dec6e6a1 100644
> --- a/crypto/algif_aead.c
> +++ b/crypto/algif_aead.c
> @@ -66,13 +66,22 @@ static int aead_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
> {
> struct sock *sk = sock->sk;
> struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> struct sock *psk = ask->parent;
> struct alg_sock *pask = alg_sk(psk);
> struct aead_tfm *aeadc = pask->private;
> - struct crypto_aead *tfm = aeadc->aead;
> - unsigned int ivsize = crypto_aead_ivsize(tfm);
> + struct crypto_aead *aead = aeadc->aead;
> + struct crypto_tfm *tfm = crypto_aead_tfm(aead);
> + unsigned int ivsize = crypto_aead_ivsize(aead);
> + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> +
> + if (ret < 0)
> + return ret;
>
> - return af_alg_sendmsg(sock, msg, size, ivsize);
> + if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
> + tfm->crt_flags |= CRYPTO_TFM_REQ_IV_SERIALIZE;
> +
> + return ret;
> }
>
> static int crypto_aead_copy_sgl(struct crypto_skcipher *null_tfm,
> diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
> index cf27dda6a181..fd2a0ba32feb 100644
> --- a/crypto/algif_skcipher.c
> +++ b/crypto/algif_skcipher.c
> @@ -43,12 +43,21 @@ static int skcipher_sendmsg(struct socket *sock, struct msghdr *msg,
> {
> struct sock *sk = sock->sk;
> struct alg_sock *ask = alg_sk(sk);
> + struct af_alg_ctx *ctx = ask->private;
> struct sock *psk = ask->parent;
> struct alg_sock *pask = alg_sk(psk);
> - struct crypto_skcipher *tfm = pask->private;
> - unsigned ivsize = crypto_skcipher_ivsize(tfm);
> + struct crypto_skcipher *skc = pask->private;
> + struct crypto_tfm *tfm = crypto_skcipher_tfm(skc);
> + unsigned int ivsize = crypto_skcipher_ivsize(skc);
> + int ret = af_alg_sendmsg(sock, msg, size, ivsize);
> +
> + if (ret < 0)
> + return ret;
>
> - return af_alg_sendmsg(sock, msg, size, ivsize);
> + if (ctx->iiv == ALG_IV_SERIAL_PROCESSING)
> + tfm->crt_flags |= CRYPTO_TFM_REQ_IV_SERIALIZE;
> +
> + return ret;
> }
>
> static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
> diff --git a/include/linux/crypto.h b/include/linux/crypto.h
> index 4860aa2c9be4..4d54f2b30692 100644
> --- a/include/linux/crypto.h
> +++ b/include/linux/crypto.h
> @@ -133,6 +133,7 @@
> #define CRYPTO_TFM_REQ_WEAK_KEY 0x00000100
> #define CRYPTO_TFM_REQ_MAY_SLEEP 0x00000200
> #define CRYPTO_TFM_REQ_MAY_BACKLOG 0x00000400
> +#define CRYPTO_TFM_REQ_IV_SERIALIZE 0x00000800
> #define CRYPTO_TFM_RES_WEAK_KEY 0x00100000
> #define CRYPTO_TFM_RES_BAD_KEY_LEN 0x00200000
> #define CRYPTO_TFM_RES_BAD_KEY_SCHED 0x00400000
Am Mittwoch, 14. Februar 2018, 06:50:38 CET schrieb Harsh Jain:
Hi Harsh,
> On 10-02-2018 03:34, Stephan M?ller wrote:
> > Crypto drivers may implement a streamlined serialization support for AIO
> > requests that is reported by the CRYPTO_ALG_SERIALIZES_IV_ACCESS flag to
> > the crypto user. When the user decides that he wants to send multiple
> > AIO requests concurrently and wants the crypto driver to handle the
> > serialization, the caller has to set CRYPTO_TFM_REQ_IV_SERIALIZE to notify
> > the crypto driver.
>
> Will crypto_alloc_* API takes cares of this flag?. For Kernel Crypto user IV
> Synchronization logic depends on weather tfm allocated supports IV
> Serialisation or not.
The alloc API calls are not related to this flag. This flag is set in the
request sent to the driver. If the driver sees this flag, it shall perform its
serialization.
The idea is the following: the driver reports CRYPTO_ALG_SERIALIZES_IV_ACCESS
if it can serialize requests. In this case, the higher level functions (like
AF_ALG) would not serialize the request. Now, the higher level functions must
inform the driver that its serialization function shall be performed which
implemented with CRYPTO_TFM_REQ_IV_SERIALIZE.
Note, the higher level functions may decide that no serialization is necessary
(e.g. in the case the inline IV handling is followed by AF_ALG). This implies
that the CRYPTO_TFM_REQ_IV_SERIALIZE flag would not be set even though the
driver is capable of serializing (and thus would report
CRYPTO_ALG_SERIALIZES_IV_ACCESS).
Ciao
Stephan
Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
Hi Harsh,
>
> Patch set is working fine with chelsio Driver.
Thank you.
> Do we really need IV locking mechanism for AEAD algo because AEAD algo's
> don't support Partial mode operation and Driver are not updating(atleast
> Chelsio) IV's on AEAD request completions.
Yes, I think we would need it. It is technically possible to have multiple
IOCBs for AEAD ciphers. Even though your implementation may not write the IV
back, others may do that. At least I do not see a guarantee that the IV is
*not* written back by a driver.
In case your driver does not write the IV back and thus does not need to
serialize, the driver can report CRYPTO_ALG_SERIALIZES_IV_ACCESS. In this
case, the higher level functions would not serialize as the driver serializes
the requests (or the driver deems it appropriate that no serialization is
needed as is the case with your driver).
Ciao
Stephan
On 14-02-2018 18:22, Stephan Mueller wrote:
> Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
>
> Hi Harsh,
>
>> Patch set is working fine with chelsio Driver.
> Thank you.
>
>> Do we really need IV locking mechanism for AEAD algo because AEAD algo's
>> don't support Partial mode operation and Driver are not updating(atleast
>> Chelsio) IV's on AEAD request completions.
> Yes, I think we would need it. It is technically possible to have multiple
> IOCBs for AEAD ciphers. Even though your implementation may not write the IV
> back, others may do that. At least I do not see a guarantee that the IV is
> *not* written back by a driver.
There is no use of writing IV back in AEAD algo till Framework starts supporting Partial mode.
Even if Driver starts updating IV for AEAD, Multiple IOCB's in both cases will yield wrong results only.
Case 1 : If we have AEAD IV serialization applied, Encryption will be wrong if same IV gets used.
Case 2: If we do not have IV serialization for AEAD. Encryption will be fine but user will have multiple Authentication tag (that too with final block processed). Its like 2nd Block encryption is based on IV received from 1st block and Authentication Tag value is based on 2nd block content only.
>
> In case your driver does not write the IV back and thus does not need to
> serialize, the driver can report CRYPTO_ALG_SERIALIZES_IV_ACCESS. In this
> case, the higher level functions would not serialize as the driver serializes
> the requests (or the driver deems it appropriate that no serialization is
> needed as is the case with your driver).
>
> Ciao
> Stephan
>
>
Am Donnerstag, 15. Februar 2018, 06:30:36 CET schrieb Harsh Jain:
Hi Harsh,
> On 14-02-2018 18:22, Stephan Mueller wrote:
> > Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
> >
> > Hi Harsh,
> >
> >> Patch set is working fine with chelsio Driver.
> >
> > Thank you.
> >
> >> Do we really need IV locking mechanism for AEAD algo because AEAD algo's
> >> don't support Partial mode operation and Driver are not updating(atleast
> >> Chelsio) IV's on AEAD request completions.
> >
> > Yes, I think we would need it. It is technically possible to have multiple
> > IOCBs for AEAD ciphers. Even though your implementation may not write the
> > IV back, others may do that. At least I do not see a guarantee that the
> > IV is *not* written back by a driver.
>
> There is no use of writing IV back in AEAD algo till Framework starts
> supporting Partial mode.
I agree.
> Even if Driver starts updating IV for AEAD,
> Multiple IOCB's in both cases will yield wrong results only.
This would only be the case if the driver would not implicitly or explicitly
serialize the requests.
>
> Case 1 : If we have AEAD IV serialization applied, Encryption will be
> wrong if same IV gets used.
Agreed.
> Case 2: If we do not have IV serialization for
> AEAD. Encryption will be fine but user will have multiple Authentication
> tag (that too with final block processed). Its like 2nd Block encryption
> is based on IV received from 1st block and Authentication Tag value is
> based on 2nd block content only.
Agreed.
But are we sure that all drivers behave correctly? Before you notified us of
the issue, I was not even aware of the fact that this serialization may not be
done in the driver. And we only have seen that issue with AF_ALG where we test
for multiple concurrent AIO operations.
Besides, when we do not have the locking for AEAD, what would we gain: one
less lock to take vs. guarantee that the AEAD operation is always properly
serialized.
Ciao
Stephan
On 15-02-2018 11:58, Stephan Mueller wrote:
> Am Donnerstag, 15. Februar 2018, 06:30:36 CET schrieb Harsh Jain:
>
> Hi Harsh,
>
>> On 14-02-2018 18:22, Stephan Mueller wrote:
>>> Am Mittwoch, 14. Februar 2018, 06:43:53 CET schrieb Harsh Jain:
>>>
>>> Hi Harsh,
>>>
>>>> Patch set is working fine with chelsio Driver.
>>> Thank you.
>>>
>>>> Do we really need IV locking mechanism for AEAD algo because AEAD algo's
>>>> don't support Partial mode operation and Driver are not updating(atleast
>>>> Chelsio) IV's on AEAD request completions.
>>> Yes, I think we would need it. It is technically possible to have multiple
>>> IOCBs for AEAD ciphers. Even though your implementation may not write the
>>> IV back, others may do that. At least I do not see a guarantee that the
>>> IV is *not* written back by a driver.
>> There is no use of writing IV back in AEAD algo till Framework starts
>> supporting Partial mode.
> I agree.
>
>> Even if Driver starts updating IV for AEAD,
>> Multiple IOCB's in both cases will yield wrong results only.
> This would only be the case if the driver would not implicitly or explicitly
> serialize the requests.
>> Case 1 : If we have AEAD IV serialization applied, Encryption will be
>> wrong if same IV gets used.
> Agreed.
>
>> Case 2: If we do not have IV serialization for
>> AEAD. Encryption will be fine but user will have multiple Authentication
>> tag (that too with final block processed). Its like 2nd Block encryption
>> is based on IV received from 1st block and Authentication Tag value is
>> based on 2nd block content only.
> Agreed.
>
> But are we sure that all drivers behave correctly? Before you notified us of
> the issue, I was not even aware of the fact that this serialization may not be
> done in the driver. And we only have seen that issue with AF_ALG where we test
> for multiple concurrent AIO operations.
I am sure other H/W will have similar problem, It's just that we tested it first.
>
> Besides, when we do not have the locking for AEAD, what would we gain: one
> less lock to take vs. guarantee that the AEAD operation is always properly
> serialized.
Even after guarantee of serialization, In the end we will get wrong result as mentioned above. which destination side cannot decrypt it.
What I feel is scenario of sending 2 of more IOCB in case of AEAD itself is wrong. We should not allow this type of requests for AEAD.
Can you think of any use case it is going to solve?
Can receiver decrypt(with 2 IOCB) the same request successfully without knowing sender has done the operation in 2 request with size "x" each?
>
> Ciao
> Stephan
>
>
Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
Hi Harsh,
> Even after guarantee of serialization, In the end we will get wrong result
> as mentioned above. which destination side cannot decrypt it. What I feel
> is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
Without the inline IV handling, I would concur.
> We
> should not allow this type of requests for AEAD.
"Not allow" as in "technically block"? As a user would only shoot itself when
he does that not knowing the consequences, I am not in favor of such an
artificial block.
> Can you think of any use
> case it is going to solve?
Well, I could fathom a use case of this. In FIPS 140-2 (yes, a term not well
received by some here), NIST insists for GCM that the IV is handled by the
cryptographic implementation.
So, when using GCM for TLS, for example, the GCM implementation would know a
bit about how the IV is updated as a session ID. I.e. after the end of one
AEAD operation, the IV is written back but modified such to comply with the
rules of some higher level proto. Thus, if such a scenarios is implemented by
a driver here, multiple IOCBs could be used with such "TLSified" GCM, for
example.
And such "TLSification" could be as simple as implementing an IV generator
that can be used with every (AEAD) cipher implementation.
> Can receiver decrypt(with 2 IOCB) the same request successfully without
> knowing sender has done the operation in 2 request with size "x" each?
> > Ciao
> > Stephan
Ciao
Stephan
On 15-02-2018 12:47, Stephan Mueller wrote:
> Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
>
> Hi Harsh,
>
>> Even after guarantee of serialization, In the end we will get wrong result
>> as mentioned above. which destination side cannot decrypt it. What I feel
>> is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
> Without the inline IV handling, I would concur.
Even with Inline IV, We will have 2 Auth Tag. can we authenticate the data with 2 Auth Tags?
>
>> We
>> should not allow this type of requests for AEAD.
> "Not allow" as in "technically block"? As a user would only shoot itself when
> he does that not knowing the consequences, I am not in favor of such an
> artificial block.
Agreed, It may not be right thing to do, but if we allowed it, What he will do with Auth( each processed with final Block) tags received in each request.
I personally feels AEAD IV serialization logic is incomplete without partial tag support.
>> Can you think of any use
>> case it is going to solve?
> Well, I could fathom a use case of this. In FIPS 140-2 (yes, a term not well
> received by some here), NIST insists for GCM that the IV is handled by the
> cryptographic implementation.
>
> So, when using GCM for TLS, for example, the GCM implementation would know a
> bit about how the IV is updated as a session ID. I.e. after the end of one
> AEAD operation, the IV is written back but modified such to comply with the
> rules of some higher level proto. Thus, if such a scenarios is implemented by
> a driver here, multiple IOCBs could be used with such "TLSified" GCM, for
> example.
>
> And such "TLSification" could be as simple as implementing an IV generator
> that can be used with every (AEAD) cipher implementation.
>
>> Can receiver decrypt(with 2 IOCB) the same request successfully without
>> knowing sender has done the operation in 2 request with size "x" each?
>>> Ciao
>>> Stephan
>
>
> Ciao
> Stephan
>
>
Am Donnerstag, 15. Februar 2018, 12:38:12 CET schrieb Harsh Jain:
Hi Harsh,
> On 15-02-2018 12:47, Stephan Mueller wrote:
> > Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
> >
> > Hi Harsh,
> >
> >> Even after guarantee of serialization, In the end we will get wrong
> >> result
> >> as mentioned above. which destination side cannot decrypt it. What I feel
> >> is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
> >
> > Without the inline IV handling, I would concur.
>
> Even with Inline IV, We will have 2 Auth Tag. can we authenticate the data
> with 2 Auth Tags?
The AAD and the tag are both sent to the kernel like in the inline IV case as
part of the data via sendmsg.
Thus, when you use inline IV support, an entire self-contained AEAD cipher
operation can be defined with one IOCB. Thus, if you have multiple IOCBs,
inline IV allow fully parallel execution of different AEAD requests.
See the following kernel comment:
/*
* Encryption operation - The in-place cipher operation is
* achieved by the following operation:
*
* TX SGL: AAD || PT
* | |
* | copy |
* v v
* RX SGL: AAD || PT || Tag
/*
* Decryption operation - To achieve an in-place cipher
* operation, the following SGL structure is used:
*
* TX SGL: AAD || CT || Tag
* | | ^
* | copy | | Create SGL link.
* v v |
* RX SGL: AAD || CT ----+
*/
Note, the TX SGL is what the caller submitted via sendmsg and the RX SGL is
the data the caller obtains via recvmsg.
Hence, in inline IV support, the caller sends the following:
encryption: IV || AAD || PT
decryption: IV || AAD || CT || Tag
> >> We
> >> should not allow this type of requests for AEAD.
> >
> > "Not allow" as in "technically block"? As a user would only shoot itself
> > when he does that not knowing the consequences, I am not in favor of such
> > an artificial block.
>
> Agreed, It may not be right thing to do, but if we allowed it, What he will
> do with Auth( each processed with final Block) tags received in each
> request.
Each tag is returned as part of the recvmsg data. Thus, AEAD cipher operations
can commence fully parallel if inline IV handling is used.
>
> I personally feels AEAD IV serialization logic is incomplete without partial
> tag support.
Could you please elaborate what you mean with "partial tag" support?
Ciao
Stephan
On 15-02-2018 17:15, Stephan Mueller wrote:
> Am Donnerstag, 15. Februar 2018, 12:38:12 CET schrieb Harsh Jain:
>
> Hi Harsh,
>
>> On 15-02-2018 12:47, Stephan Mueller wrote:
>>> Am Donnerstag, 15. Februar 2018, 08:03:20 CET schrieb Harsh Jain:
>>>
>>> Hi Harsh,
>>>
>>>> Even after guarantee of serialization, In the end we will get wrong
>>>> result
>>>> as mentioned above. which destination side cannot decrypt it. What I feel
>>>> is scenario of sending 2 of more IOCB in case of AEAD itself is wrong.
>>> Without the inline IV handling, I would concur.
>> Even with Inline IV, We will have 2 Auth Tag. can we authenticate the data
>> with 2 Auth Tags?
> The AAD and the tag are both sent to the kernel like in the inline IV case as
> part of the data via sendmsg.
>
> Thus, when you use inline IV support, an entire self-contained AEAD cipher
> operation can be defined with one IOCB. Thus, if you have multiple IOCBs,
> inline IV allow fully parallel execution of different AEAD requests.
>
> See the following kernel comment:
>
> /*
> * Encryption operation - The in-place cipher operation is
> * achieved by the following operation:
> *
> * TX SGL: AAD || PT
> * | |
> * | copy |
> * v v
> * RX SGL: AAD || PT || Tag
>
> /*
> * Decryption operation - To achieve an in-place cipher
> * operation, the following SGL structure is used:
> *
> * TX SGL: AAD || CT || Tag
> * | | ^
> * | copy | | Create SGL link.
> * v v |
> * RX SGL: AAD || CT ----+
> */
>
> Note, the TX SGL is what the caller submitted via sendmsg and the RX SGL is
> the data the caller obtains via recvmsg.
>
> Hence, in inline IV support, the caller sends the following:
>
> encryption: IV || AAD || PT
>
> decryption: IV || AAD || CT || Tag
>
>>>> We
>>>> should not allow this type of requests for AEAD.
>>> "Not allow" as in "technically block"? As a user would only shoot itself
>>> when he does that not knowing the consequences, I am not in favor of such
>>> an artificial block.
>> Agreed, It may not be right thing to do, but if we allowed it, What he will
>> do with Auth( each processed with final Block) tags received in each
>> request.
> Each tag is returned as part of the recvmsg data. Thus, AEAD cipher operations
> can commence fully parallel if inline IV handling is used.
>> I personally feels AEAD IV serialization logic is incomplete without partial
>> tag support.
> Could you please elaborate what you mean with "partial tag" support?
Here is the catch, Calculation of tag depends on total payload length atleast for shaX, gcm,ccm mode on which I have worked.
If we take an example of shaX. It appends 1 special block at the end of user data which includes total input length in bit. Refer "sha1_base_do_finalize"
Suppose we have 32 byte and we break this in 2 IOCB of 16 bytes each.
Expected result : 32 encrypted bytes + sha auth tag considering length 32 bytes.
What we will get : 16 bytes + sha auth tag considering length 16 bytes + 16 encrypted bytes + another sha tag considering 16 bytes.
>
> Ciao
> Stephan
>
>
Am Donnerstag, 15. Februar 2018, 13:45:53 CET schrieb Harsh Jain:
Hi Harsh,
> > Could you please elaborate what you mean with "partial tag" support?
>
> Here is the catch, Calculation of tag depends on total payload length
> atleast for shaX, gcm,ccm mode on which I have worked.
>
> If we take an example of shaX. It appends 1 special block at the end of user
> data which includes total input length in bit. Refer
> "sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2 IOCB
> of 16 bytes each. Expected result : 32 encrypted bytes + sha auth tag
> considering length 32 bytes. What we will get : 16 bytes + sha auth tag
> considering length 16 bytes + 16 encrypted bytes + another sha tag
> considering 16 bytes.
As AF_ALG for AEAD is implemented, there is no stream support where the hash
is calculated at the end. This is even not supported in the current AEAD API
of the kernel crypto API as far as I see. The only "stream-like" support is
that you can invoke multiple separate sendmsg calls to provide the input data
for the AEAD. But once you call recvmsg, the ciphertext and the tag is
calculated and thus the recvmsg is akin to a hash_final operation.
Complete parallel execution of multiple independent AEAD cipher operations
with AIO is possible using the inline IV as suggested in the patch set.
The locking of the ctx->iv is only there to support a driver not to whack the
IV buffer. However, if others also agree that the ctx->ivlock should not be
taken in the AEAD code path because a potential whacking the IV buffer by a
driver does not need to be prevented, I would change the patch.
Ciao
Stephan
On Thu, Feb 15, 2018 at 8:04 AM, Stephan Mueller <[email protected]> wrote:
> Am Donnerstag, 15. Februar 2018, 13:45:53 CET schrieb Harsh Jain:
>
>> > Could you please elaborate what you mean with "partial tag" support?
>>
>> Here is the catch, Calculation of tag depends on total payload length
>> atleast for shaX, gcm,ccm mode on which I have worked.
>>
>> If we take an example of shaX. It appends 1 special block at the end of user
>> data which includes total input length in bit. Refer
>> "sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2 IOCB
>> of 16 bytes each. Expected result : 32 encrypted bytes + sha auth tag
>> considering length 32 bytes. What we will get : 16 bytes + sha auth tag
>> considering length 16 bytes + 16 encrypted bytes + another sha tag
>> considering 16 bytes.
>
> As AF_ALG for AEAD is implemented, there is no stream support where the hash
> is calculated at the end. This is even not supported in the current AEAD API
> of the kernel crypto API as far as I see. The only "stream-like" support is
> that you can invoke multiple separate sendmsg calls to provide the input data
> for the AEAD. But once you call recvmsg, the ciphertext and the tag is
> calculated and thus the recvmsg is akin to a hash_final operation.
If you follow Bernstein's protocol design philosophy, then messages
should be no larger than about 4K in size. From
https://nacl.cr.yp.to/valid.html:
This is one of several reasons [1] that callers should (1) split
all data into packets sent through the network; (2) put a
small global limit on packet length; and (3) separately
encrypt and authenticate each packet.
With the [1] link being
https://groups.google.com/forum/#!original/boring-crypto/BpUmNMXKMYQ/EEwAIeQdjacJ
Jeff
Am Donnerstag, 15. Februar 2018, 14:04:53 CET schrieb Stephan Mueller:
Hi Stephan,
> Am Donnerstag, 15. Februar 2018, 13:45:53 CET schrieb Harsh Jain:
>
> Hi Harsh,
>
> > > Could you please elaborate what you mean with "partial tag" support?
> >
> > Here is the catch, Calculation of tag depends on total payload length
> > atleast for shaX, gcm,ccm mode on which I have worked.
> >
> > If we take an example of shaX. It appends 1 special block at the end of
> > user data which includes total input length in bit. Refer
> > "sha1_base_do_finalize" Suppose we have 32 byte and we break this in 2
> > IOCB
> > of 16 bytes each. Expected result : 32 encrypted bytes + sha auth tag
> > considering length 32 bytes. What we will get : 16 bytes + sha auth tag
> > considering length 16 bytes + 16 encrypted bytes + another sha tag
> > considering 16 bytes.
>
> As AF_ALG for AEAD is implemented, there is no stream support where the hash
> is calculated at the end. This is even not supported in the current AEAD
> API of the kernel crypto API as far as I see. The only "stream-like"
> support is that you can invoke multiple separate sendmsg calls to provide
> the input data for the AEAD. But once you call recvmsg, the ciphertext and
> the tag is calculated and thus the recvmsg is akin to a hash_final
> operation.
>
> Complete parallel execution of multiple independent AEAD cipher operations
> with AIO is possible using the inline IV as suggested in the patch set.
>
> The locking of the ctx->iv is only there to support a driver not to whack
> the IV buffer. However, if others also agree that the ctx->ivlock should
> not be taken in the AEAD code path because a potential whacking the IV
> buffer by a driver does not need to be prevented, I would change the patch.
I was about to change the implementation. However, I already found an
implementation where the driver changes the IV of the request buffer:
crypto_ccm_init_crypt modifies the original buffer of the IV. I expect that
there are more.
Therefore, I would think that keeping the patch set as it is right now is the
right thing to do. Thus, we should cover the AEAD ciphers with the lock and
thus serialize the AEAD request. This guarantees that the IV buffer is at
least not mangled while the cipher operation is ongoing.
Again, if a particular AEAD driver guarantees the IV is not mangled, it can
very well set CRYPTO_ALG_SERIALIZES_IV_ACCESS in its flag set. This would
imply the lock is not set.
Ciao
Stephan