Added support for executing multiple, independent or not, requests
for crypto engine based on a retry mechanism. If hardware was unable
to execute a backlog request, enqueue it back in front of crypto-engine
queue, to keep the order of requests.
Now do_one_request() returns:
>= 0: hardware executed the request successfully;
< 0: this is the old error path. If hardware has support for retry
mechanism, the backlog request is put back in front of crypto-engine
queue. For backwards compatibility, if the retry support is not available,
the crypto-engine will work as before.
If hardware queue is full (-ENOSPC), requeue request regardless
of MAY_BACKLOG flag.
If hardware throws any other error code (like -EIO, -EINVAL,
-ENOMEM, etc.) only MAY_BACKLOG requests are enqueued back into
crypto-engine's queue, since the others can be dropped.
If hardware supports batch requests, crypto-engine can handle this use-case
through do_batch_requests callback.
Since, these new features, cannot be supported by all hardware,
the crypto-engine framework is backward compatible:
- by using the crypto_engine_alloc_init function, to initialize
crypto-engine, the new callback is NULL and retry mechanism is
disabled, so crypto-engine will work as before these changes;
- to support multiple requests, in parallel, retry_support variable
must be set on true, in driver.
- to support batch requests, do_batch_requests callback must be
implemented in driver, to execute a batch of requests. The link
between the requests, is expected to be done in driver, in
do_one_request().
---
Changes since V5:
- updated, in algapi, the crypto_enqueue_request_head function to
enqueue requests regardless of backlog flag;
- requeue request based on the error code returned by do_one_request().
Changes since V4:
- added, in algapi a function to add a request in front of queue;
- added a retry mechanism: if hardware is unable to execute
a backlog request, enqueue it back in front of crypto-engine
queue, to keep the order of requests.
Changes since V3:
- removed can_enqueue_hardware callback and added a start-stop
mechanism based on the on the return value of do_one_request().
Changes since V2:
- readded cur_req in crypto-engine, to keep, the exact behavior as before
these changes, if can_enqueue_more is not implemented: send requests
to hardware, _one-by-one_, on crypto_pump_requests, and complete it,
on crypto_finalize_request, and so on.
- do_batch_requests is available only with can_enqueue_more.
Changes since V1:
- changed the name of can_enqueue_hardware callback to can_enqueue_more, and
the argument of this callback to crypto_engine structure (for cases when more
than ore crypto-engine is used).
- added a new patch with support for batch requests.
Changes since V0 (RFC):
- removed max_no_req and no_req, as the number of request that can be
processed in parallel;
- added a new callback, can_enqueue_more, to check whether the hardware
can process a new request.
Iuliana Prodan (3):
crypto: algapi - create function to add request in front of queue
crypto: engine - support for parallel requests based on retry
mechanism
crypto: engine - support for batch requests
crypto/algapi.c | 8 +++
crypto/crypto_engine.c | 171 +++++++++++++++++++++++++++++++++++++++---------
include/crypto/algapi.h | 2 +
include/crypto/engine.h | 15 ++++-
4 files changed, 164 insertions(+), 32 deletions(-)
--
2.1.0
Added support for executing multiple requests, in parallel,
for crypto engine based on a retry mechanism.
If hardware was unable to execute a backlog request, enqueue it
back in front of crypto-engine queue, to keep the order
of requests.
A new variable is added, retry_support (this is to keep the
backward compatibility of crypto-engine) , which keeps track
whether the hardware has support for retry mechanism and,
also, if can run multiple requests.
If do_one_request() returns:
>= 0: hardware executed the request successfully;
< 0: this is the old error path. If hardware has support for retry
mechanism, the request is put back in front of crypto-engine queue.
For backwards compatibility, if the retry support is not available,
the crypto-engine will work as before.
If hardware queue is full (-ENOSPC), requeue request regardless
of MAY_BACKLOG flag.
If hardware throws any other error code (like -EIO, -EINVAL,
-ENOMEM, etc.) only MAY_BACKLOG requests are enqueued back into
crypto-engine's queue, since the others can be dropped.
The new crypto_engine_alloc_init_and_set function, initializes
crypto-engine, sets the maximum size for crypto-engine software
queue (not hardcoded anymore) and the retry_support variable
is set, by default, to false.
On crypto_pump_requests(), if do_one_request() returns >= 0,
a new request is send to hardware, until there is no space in
hardware and do_one_request() returns < 0.
By default, retry_support is false and crypto-engine will
work as before - will send requests to hardware,
one-by-one, on crypto_pump_requests(), and complete it, on
crypto_finalize_request(), and so on.
To support multiple requests, in each driver, retry_support
must be set on true, and if do_one_request() returns an error
the request must not be freed, since it will be enqueued back
into crypto-engine's queue.
When all drivers, that use crypto-engine now, will be updated for
retry mechanism, the retry_support variable can be removed.
Signed-off-by: Iuliana Prodan <[email protected]>
---
crypto/crypto_engine.c | 146 ++++++++++++++++++++++++++++++++++++++----------
include/crypto/engine.h | 10 +++-
2 files changed, 124 insertions(+), 32 deletions(-)
diff --git a/crypto/crypto_engine.c b/crypto/crypto_engine.c
index eb029ff..ee19273 100644
--- a/crypto/crypto_engine.c
+++ b/crypto/crypto_engine.c
@@ -22,32 +22,36 @@
* @err: error number
*/
static void crypto_finalize_request(struct crypto_engine *engine,
- struct crypto_async_request *req, int err)
+ struct crypto_async_request *req, int err)
{
unsigned long flags;
- bool finalize_cur_req = false;
+ bool finalize_req = false;
int ret;
struct crypto_engine_ctx *enginectx;
- spin_lock_irqsave(&engine->queue_lock, flags);
- if (engine->cur_req == req)
- finalize_cur_req = true;
- spin_unlock_irqrestore(&engine->queue_lock, flags);
+ /*
+ * If hardware cannot enqueue more requests
+ * and retry mechanism is not supported
+ * make sure we are completing the current request
+ */
+ if (!engine->retry_support) {
+ spin_lock_irqsave(&engine->queue_lock, flags);
+ if (engine->cur_req == req) {
+ finalize_req = true;
+ engine->cur_req = NULL;
+ }
+ spin_unlock_irqrestore(&engine->queue_lock, flags);
+ }
- if (finalize_cur_req) {
+ if (finalize_req || engine->retry_support) {
enginectx = crypto_tfm_ctx(req->tfm);
- if (engine->cur_req_prepared &&
+ if (enginectx->op.prepare_request &&
enginectx->op.unprepare_request) {
ret = enginectx->op.unprepare_request(engine, req);
if (ret)
dev_err(engine->dev, "failed to unprepare request\n");
}
- spin_lock_irqsave(&engine->queue_lock, flags);
- engine->cur_req = NULL;
- engine->cur_req_prepared = false;
- spin_unlock_irqrestore(&engine->queue_lock, flags);
}
-
req->complete(req, err);
kthread_queue_work(engine->kworker, &engine->pump_requests);
@@ -74,7 +78,7 @@ static void crypto_pump_requests(struct crypto_engine *engine,
spin_lock_irqsave(&engine->queue_lock, flags);
/* Make sure we are not already running a request */
- if (engine->cur_req)
+ if (!engine->retry_support && engine->cur_req)
goto out;
/* If another context is idling then defer */
@@ -108,13 +112,21 @@ static void crypto_pump_requests(struct crypto_engine *engine,
goto out;
}
+start_request:
/* Get the fist request from the engine queue to handle */
backlog = crypto_get_backlog(&engine->queue);
async_req = crypto_dequeue_request(&engine->queue);
if (!async_req)
goto out;
- engine->cur_req = async_req;
+ /*
+ * If hardware doesn't support the retry mechanism,
+ * keep track of the request we are processing now.
+ * We'll need it on completion (crypto_finalize_request).
+ */
+ if (!engine->retry_support)
+ engine->cur_req = async_req;
+
if (backlog)
backlog->complete(backlog, -EINPROGRESS);
@@ -130,7 +142,7 @@ static void crypto_pump_requests(struct crypto_engine *engine,
ret = engine->prepare_crypt_hardware(engine);
if (ret) {
dev_err(engine->dev, "failed to prepare crypt hardware\n");
- goto req_err;
+ goto req_err_2;
}
}
@@ -141,28 +153,81 @@ static void crypto_pump_requests(struct crypto_engine *engine,
if (ret) {
dev_err(engine->dev, "failed to prepare request: %d\n",
ret);
- goto req_err;
+ goto req_err_2;
}
- engine->cur_req_prepared = true;
}
if (!enginectx->op.do_one_request) {
dev_err(engine->dev, "failed to do request\n");
ret = -EINVAL;
- goto req_err;
+ goto req_err_1;
}
+
ret = enginectx->op.do_one_request(engine, async_req);
- if (ret) {
- dev_err(engine->dev, "Failed to do one request from queue: %d\n", ret);
- goto req_err;
+
+ /* Request unsuccessfully executed by hardware */
+ if (ret < 0) {
+ /*
+ * If hardware queue is full (-ENOSPC), requeue request
+ * regardless of backlog flag.
+ * If hardware throws any other error code,
+ * requeue only backlog requests.
+ * Otherwise, unprepare and complete the request.
+ */
+ if (!engine->retry_support ||
+ ((ret != -ENOSPC) &&
+ !(async_req->flags & CRYPTO_TFM_REQ_MAY_BACKLOG))) {
+ dev_err(engine->dev,
+ "Failed to do one request from queue: %d\n",
+ ret);
+ goto req_err_1;
+ }
+ /*
+ * If retry mechanism is supported,
+ * unprepare current request and
+ * enqueue it back into crypto-engine queue.
+ */
+ if (enginectx->op.unprepare_request) {
+ ret = enginectx->op.unprepare_request(engine,
+ async_req);
+ if (ret)
+ dev_err(engine->dev,
+ "failed to unprepare request\n");
+ }
+ spin_lock_irqsave(&engine->queue_lock, flags);
+ /*
+ * If hardware was unable to execute request, enqueue it
+ * back in front of crypto-engine queue, to keep the order
+ * of requests.
+ */
+ crypto_enqueue_request_head(&engine->queue, async_req);
+
+ kthread_queue_work(engine->kworker, &engine->pump_requests);
+ goto out;
}
- return;
-req_err:
- crypto_finalize_request(engine, async_req, ret);
+ goto retry;
+
+req_err_1:
+ if (enginectx->op.unprepare_request) {
+ ret = enginectx->op.unprepare_request(engine, async_req);
+ if (ret)
+ dev_err(engine->dev, "failed to unprepare request\n");
+ }
+
+req_err_2:
+ async_req->complete(async_req, ret);
+
+retry:
+ /* If retry mechanism is supported, send new requests to engine */
+ if (engine->retry_support) {
+ spin_lock_irqsave(&engine->queue_lock, flags);
+ goto start_request;
+ }
return;
out:
spin_unlock_irqrestore(&engine->queue_lock, flags);
+ return;
}
static void crypto_pump_work(struct kthread_work *work)
@@ -386,15 +451,20 @@ int crypto_engine_stop(struct crypto_engine *engine)
EXPORT_SYMBOL_GPL(crypto_engine_stop);
/**
- * crypto_engine_alloc_init - allocate crypto hardware engine structure and
- * initialize it.
+ * crypto_engine_alloc_init_and_set - allocate crypto hardware engine structure
+ * and initialize it by setting the maximum number of entries in the software
+ * crypto-engine queue.
* @dev: the device attached with one hardware engine
+ * @retry_support: whether hardware has support for retry mechanism
* @rt: whether this queue is set to run as a realtime task
+ * @qlen: maximum size of the crypto-engine queue
*
* This must be called from context that can sleep.
* Return: the crypto engine structure on success, else NULL.
*/
-struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
+struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev,
+ bool retry_support,
+ bool rt, int qlen)
{
struct sched_param param = { .sched_priority = MAX_RT_PRIO / 2 };
struct crypto_engine *engine;
@@ -411,12 +481,12 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
engine->running = false;
engine->busy = false;
engine->idling = false;
- engine->cur_req_prepared = false;
+ engine->retry_support = retry_support;
engine->priv_data = dev;
snprintf(engine->name, sizeof(engine->name),
"%s-engine", dev_name(dev));
- crypto_init_queue(&engine->queue, CRYPTO_ENGINE_MAX_QLEN);
+ crypto_init_queue(&engine->queue, qlen);
spin_lock_init(&engine->queue_lock);
engine->kworker = kthread_create_worker(0, "%s", engine->name);
@@ -433,6 +503,22 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
return engine;
}
+EXPORT_SYMBOL_GPL(crypto_engine_alloc_init_and_set);
+
+/**
+ * crypto_engine_alloc_init - allocate crypto hardware engine structure and
+ * initialize it.
+ * @dev: the device attached with one hardware engine
+ * @rt: whether this queue is set to run as a realtime task
+ *
+ * This must be called from context that can sleep.
+ * Return: the crypto engine structure on success, else NULL.
+ */
+struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
+{
+ return crypto_engine_alloc_init_and_set(dev, false, rt,
+ CRYPTO_ENGINE_MAX_QLEN);
+}
EXPORT_SYMBOL_GPL(crypto_engine_alloc_init);
/**
diff --git a/include/crypto/engine.h b/include/crypto/engine.h
index e29cd67..b92d7ff 100644
--- a/include/crypto/engine.h
+++ b/include/crypto/engine.h
@@ -24,7 +24,9 @@
* @idling: the engine is entering idle state
* @busy: request pump is busy
* @running: the engine is on working
- * @cur_req_prepared: current request is prepared
+ * @retry_support: indication that the hardware allows re-execution
+ * of a failed backlog request
+ * crypto-engine, in head position to keep order
* @list: link with the global crypto engine list
* @queue_lock: spinlock to syncronise access to request queue
* @queue: the crypto queue of the engine
@@ -45,7 +47,8 @@ struct crypto_engine {
bool idling;
bool busy;
bool running;
- bool cur_req_prepared;
+
+ bool retry_support;
struct list_head list;
spinlock_t queue_lock;
@@ -102,6 +105,9 @@ void crypto_finalize_skcipher_request(struct crypto_engine *engine,
int crypto_engine_start(struct crypto_engine *engine);
int crypto_engine_stop(struct crypto_engine *engine);
struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt);
+struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev,
+ bool retry_support,
+ bool rt, int qlen);
int crypto_engine_exit(struct crypto_engine *engine);
#endif /* _CRYPTO_ENGINE_H */
--
2.1.0
Added support for batch requests, per crypto engine.
A new callback is added, do_batch_requests, which executes a
batch of requests. This has the crypto_engine structure as argument
(for cases when more than one crypto-engine is used).
The crypto_engine_alloc_init_and_set function, initializes
crypto-engine, but also, sets the do_batch_requests callback.
On crypto_pump_requests, if do_batch_requests callback is
implemented in a driver, this will be executed. The link between
the requests will be done in driver, if possible.
do_batch_requests is available only if the hardware has support
for multiple request.
Signed-off-by: Iuliana Prodan <[email protected]>
---
crypto/crypto_engine.c | 27 ++++++++++++++++++++++++++-
include/crypto/engine.h | 5 +++++
2 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/crypto/crypto_engine.c b/crypto/crypto_engine.c
index ee19273..412149e 100644
--- a/crypto/crypto_engine.c
+++ b/crypto/crypto_engine.c
@@ -227,6 +227,18 @@ static void crypto_pump_requests(struct crypto_engine *engine,
out:
spin_unlock_irqrestore(&engine->queue_lock, flags);
+
+ /*
+ * Batch requests is possible only if
+ * hardware can enqueue multiple requests
+ */
+ if (engine->do_batch_requests) {
+ ret = engine->do_batch_requests(engine);
+ if (ret)
+ dev_err(engine->dev, "failed to do batch requests: %d\n",
+ ret);
+ }
+
return;
}
@@ -456,6 +468,12 @@ EXPORT_SYMBOL_GPL(crypto_engine_stop);
* crypto-engine queue.
* @dev: the device attached with one hardware engine
* @retry_support: whether hardware has support for retry mechanism
+ * @cbk_do_batch: pointer to a callback function to be invoked when executing a
+ * a batch of requests.
+ * This has the form:
+ * callback(struct crypto_engine *engine)
+ * where:
+ * @engine: the crypto engine structure.
* @rt: whether this queue is set to run as a realtime task
* @qlen: maximum size of the crypto-engine queue
*
@@ -464,6 +482,7 @@ EXPORT_SYMBOL_GPL(crypto_engine_stop);
*/
struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev,
bool retry_support,
+ int (*cbk_do_batch)(struct crypto_engine *engine),
bool rt, int qlen)
{
struct sched_param param = { .sched_priority = MAX_RT_PRIO / 2 };
@@ -483,6 +502,12 @@ struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev,
engine->idling = false;
engine->retry_support = retry_support;
engine->priv_data = dev;
+ /*
+ * Batch requests is possible only if
+ * hardware has support for retry mechanism.
+ */
+ engine->do_batch_requests = retry_support ? cbk_do_batch : NULL;
+
snprintf(engine->name, sizeof(engine->name),
"%s-engine", dev_name(dev));
@@ -516,7 +541,7 @@ EXPORT_SYMBOL_GPL(crypto_engine_alloc_init_and_set);
*/
struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
{
- return crypto_engine_alloc_init_and_set(dev, false, rt,
+ return crypto_engine_alloc_init_and_set(dev, false, NULL, rt,
CRYPTO_ENGINE_MAX_QLEN);
}
EXPORT_SYMBOL_GPL(crypto_engine_alloc_init);
diff --git a/include/crypto/engine.h b/include/crypto/engine.h
index b92d7ff..3f06e40 100644
--- a/include/crypto/engine.h
+++ b/include/crypto/engine.h
@@ -37,6 +37,8 @@
* @unprepare_crypt_hardware: there are currently no more requests on the
* queue so the subsystem notifies the driver that it may relax the
* hardware by issuing this call
+ * @do_batch_requests: execute a batch of requests. Depends on multiple
+ * requests support.
* @kworker: kthread worker struct for request pump
* @pump_requests: work struct for scheduling work to the request pump
* @priv_data: the engine private data
@@ -59,6 +61,8 @@ struct crypto_engine {
int (*prepare_crypt_hardware)(struct crypto_engine *engine);
int (*unprepare_crypt_hardware)(struct crypto_engine *engine);
+ int (*do_batch_requests)(struct crypto_engine *engine);
+
struct kthread_worker *kworker;
struct kthread_work pump_requests;
@@ -107,6 +111,7 @@ int crypto_engine_stop(struct crypto_engine *engine);
struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt);
struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev,
bool retry_support,
+ int (*cbk_do_batch)(struct crypto_engine *engine),
bool rt, int qlen);
int crypto_engine_exit(struct crypto_engine *engine);
--
2.1.0
On Tue, Apr 28, 2020 at 06:49:02PM +0300, Iuliana Prodan wrote:
> Added support for executing multiple, independent or not, requests
> for crypto engine based on a retry mechanism. If hardware was unable
> to execute a backlog request, enqueue it back in front of crypto-engine
> queue, to keep the order of requests.
>
> Now do_one_request() returns:
> >= 0: hardware executed the request successfully;
> < 0: this is the old error path. If hardware has support for retry
> mechanism, the backlog request is put back in front of crypto-engine
> queue. For backwards compatibility, if the retry support is not available,
> the crypto-engine will work as before.
> If hardware queue is full (-ENOSPC), requeue request regardless
> of MAY_BACKLOG flag.
> If hardware throws any other error code (like -EIO, -EINVAL,
> -ENOMEM, etc.) only MAY_BACKLOG requests are enqueued back into
> crypto-engine's queue, since the others can be dropped.
>
> If hardware supports batch requests, crypto-engine can handle this use-case
> through do_batch_requests callback.
>
> Since, these new features, cannot be supported by all hardware,
> the crypto-engine framework is backward compatible:
> - by using the crypto_engine_alloc_init function, to initialize
> crypto-engine, the new callback is NULL and retry mechanism is
> disabled, so crypto-engine will work as before these changes;
> - to support multiple requests, in parallel, retry_support variable
> must be set on true, in driver.
> - to support batch requests, do_batch_requests callback must be
> implemented in driver, to execute a batch of requests. The link
> between the requests, is expected to be done in driver, in
> do_one_request().
>
> ---
> Changes since V5:
> - updated, in algapi, the crypto_enqueue_request_head function to
> enqueue requests regardless of backlog flag;
> - requeue request based on the error code returned by do_one_request().
>
> Changes since V4:
> - added, in algapi a function to add a request in front of queue;
> - added a retry mechanism: if hardware is unable to execute
> a backlog request, enqueue it back in front of crypto-engine
> queue, to keep the order of requests.
>
> Changes since V3:
> - removed can_enqueue_hardware callback and added a start-stop
> mechanism based on the on the return value of do_one_request().
>
> Changes since V2:
> - readded cur_req in crypto-engine, to keep, the exact behavior as before
> these changes, if can_enqueue_more is not implemented: send requests
> to hardware, _one-by-one_, on crypto_pump_requests, and complete it,
> on crypto_finalize_request, and so on.
> - do_batch_requests is available only with can_enqueue_more.
>
> Changes since V1:
> - changed the name of can_enqueue_hardware callback to can_enqueue_more, and
> the argument of this callback to crypto_engine structure (for cases when more
> than ore crypto-engine is used).
> - added a new patch with support for batch requests.
>
> Changes since V0 (RFC):
> - removed max_no_req and no_req, as the number of request that can be
> processed in parallel;
> - added a new callback, can_enqueue_more, to check whether the hardware
> can process a new request.
>
>
> Iuliana Prodan (3):
> crypto: algapi - create function to add request in front of queue
> crypto: engine - support for parallel requests based on retry
> mechanism
> crypto: engine - support for batch requests
>
> crypto/algapi.c | 8 +++
> crypto/crypto_engine.c | 171 +++++++++++++++++++++++++++++++++++++++---------
> include/crypto/algapi.h | 2 +
> include/crypto/engine.h | 15 ++++-
> 4 files changed, 164 insertions(+), 32 deletions(-)
All applied. Thanks.
--
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt