Received: by 2002:a25:8b91:0:0:0:0:0 with SMTP id j17csp3983925ybl; Mon, 27 Jan 2020 14:18:08 -0800 (PST) X-Google-Smtp-Source: APXvYqzS/JNeMn3E+RMS4/sE80MItEJxB1yBgIHA8+bkO5tueiBpfacHeN9OGCOCJvmNflcvAA70 X-Received: by 2002:aca:4306:: with SMTP id q6mr962681oia.54.1580163488145; Mon, 27 Jan 2020 14:18:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1580163488; cv=none; d=google.com; s=arc-20160816; b=aOOaqFphEjoiaY4iphErri5rMSgiN0Gec2tqG+GOlVWP2YLMla61FrFiBA9hMYZShv 28JIwKTFbZjncGIcE2E5qHQrCBSNteo+2qRz4B7sDjDMiA2K3iFsh/fm1SyYFtxSJbr/ rKPAxfSLOzhNsyCEEc0KUFR5Ahu0fdY8ARjGMEMPgTvv1zxEobZfeUz4SGkDvnIv33TB JMLIZZsL8+3RO0ewKWOMvyPRugHGJ4I8B2pp4ZcRGKtoHFO3yQmX+8197X02pNRIPjl9 PXkpq9uXiJKb8lY6bvnvozSBb8rmofU1vgugPFH4QQ45t0qzc4KqYP78I+U7C3kQCwhZ gKLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from; bh=NmQGly8QVzEN7hxLLQBBusKDMXrAtexQEwwjs1LMPRg=; b=eVs7CQrZXgmJKycQt7Xl3qvlX9TQBjhc0XYvvz6803rhjFEUq7d91CX5lswp0nzyA+ U3aWKE42Q1qaiD1py1H7gkjAv9e6dgMHt4S6JFNesC7cW+nqjg77AuKy/Magyv2/P/Du HSnT/zFxSJy3gErejId4xJavFh9TcdAUSdxGHJesGiOKaLNDdbD0R1mXMGMSft3889ks AkK3sZY/8gSB+ZppH/XqAoWF2wOFjvrLqqoo0pdrZII/cC4pklSxyQDu67ojkb0I1PJj Zt2NDr1CtrENGRyKGWlfPwd8HDxq6jlyUfJBzxh+afB1RdG4+zlxBXh/v5TEnnIKj92w 4grw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w24si3722646oic.260.2020.01.27.14.17.34; Mon, 27 Jan 2020 14:18:08 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726173AbgA0WRY (ORCPT + 99 others); Mon, 27 Jan 2020 17:17:24 -0500 Received: from inva021.nxp.com ([92.121.34.21]:37894 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726101AbgA0WRY (ORCPT ); Mon, 27 Jan 2020 17:17:24 -0500 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id AE9C521B9E6; Mon, 27 Jan 2020 23:17:20 +0100 (CET) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id A066121BA0A; Mon, 27 Jan 2020 23:17:20 +0100 (CET) Received: from lorenz.ea.freescale.net (lorenz.ea.freescale.net [10.171.71.5]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 082B9205A3; Mon, 27 Jan 2020 23:17:19 +0100 (CET) From: Iuliana Prodan To: Herbert Xu , Baolin Wang , Ard Biesheuvel , Corentin Labbe , Horia Geanta , Maxime Coquelin , Alexandre Torgue , Maxime Ripard Cc: Aymen Sghaier , "David S. Miller" , Silvano Di Ninno , Franck Lenormand , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, linux-imx , Iuliana Prodan Subject: [PATCH] crypto: engine - support for parallel requests Date: Tue, 28 Jan 2020 00:17:05 +0200 Message-Id: <1580163425-13266-1-git-send-email-iuliana.prodan@nxp.com> X-Mailer: git-send-email 2.1.0 X-Virus-Scanned: ClamAV using ClamSMTP Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Added support for executing multiple requests, in parallel, for crypto engine. A new callback is added, can_enqueue_hardware, which asks the driver if the hardware has free space, to enqueue a new request. The new crypto_engine_alloc_init_and_set function, initialize crypto-engine, sets the maximum size for crypto-engine software queue (not hardcoded anymore) and the can_enqueue_hardware callback. On crypto_pump_requests, if can_enqueue_hardware callback returns true, a new request is send to hardware, until there is no space and the callback returns false. Signed-off-by: Iuliana Prodan --- Changes since V0 (RFC): - removed max_no_req and no_req, as the number of request that can be processed in parallel; - added a new callback, can_enqueue_hardware, to check whether the hardware can process a new request. --- crypto/crypto_engine.c | 105 ++++++++++++++++++++++++++++++------------------ include/crypto/engine.h | 10 +++-- 2 files changed, 71 insertions(+), 44 deletions(-) diff --git a/crypto/crypto_engine.c b/crypto/crypto_engine.c index eb029ff..ee3a610 100644 --- a/crypto/crypto_engine.c +++ b/crypto/crypto_engine.c @@ -22,32 +22,18 @@ * @err: error number */ static void crypto_finalize_request(struct crypto_engine *engine, - struct crypto_async_request *req, int err) + struct crypto_async_request *req, int err) { - unsigned long flags; - bool finalize_cur_req = false; int ret; struct crypto_engine_ctx *enginectx; - spin_lock_irqsave(&engine->queue_lock, flags); - if (engine->cur_req == req) - finalize_cur_req = true; - spin_unlock_irqrestore(&engine->queue_lock, flags); - - if (finalize_cur_req) { - enginectx = crypto_tfm_ctx(req->tfm); - if (engine->cur_req_prepared && - enginectx->op.unprepare_request) { - ret = enginectx->op.unprepare_request(engine, req); - if (ret) - dev_err(engine->dev, "failed to unprepare request\n"); - } - spin_lock_irqsave(&engine->queue_lock, flags); - engine->cur_req = NULL; - engine->cur_req_prepared = false; - spin_unlock_irqrestore(&engine->queue_lock, flags); + enginectx = crypto_tfm_ctx(req->tfm); + if (enginectx->op.prepare_request && + enginectx->op.unprepare_request) { + ret = enginectx->op.unprepare_request(engine, req); + if (ret) + dev_err(engine->dev, "failed to unprepare request\n"); } - req->complete(req, err); kthread_queue_work(engine->kworker, &engine->pump_requests); @@ -73,10 +59,6 @@ static void crypto_pump_requests(struct crypto_engine *engine, spin_lock_irqsave(&engine->queue_lock, flags); - /* Make sure we are not already running a request */ - if (engine->cur_req) - goto out; - /* If another context is idling then defer */ if (engine->idling) { kthread_queue_work(engine->kworker, &engine->pump_requests); @@ -108,13 +90,18 @@ static void crypto_pump_requests(struct crypto_engine *engine, goto out; } +start_request: + /* If hw is busy, do not send any request */ + if (engine->can_enqueue_hardware && + !engine->can_enqueue_hardware(engine->dev)) + goto out; + /* Get the fist request from the engine queue to handle */ backlog = crypto_get_backlog(&engine->queue); async_req = crypto_dequeue_request(&engine->queue); if (!async_req) goto out; - engine->cur_req = async_req; if (backlog) backlog->complete(backlog, -EINPROGRESS); @@ -130,7 +117,7 @@ static void crypto_pump_requests(struct crypto_engine *engine, ret = engine->prepare_crypt_hardware(engine); if (ret) { dev_err(engine->dev, "failed to prepare crypt hardware\n"); - goto req_err; + goto req_err_2; } } @@ -141,26 +128,38 @@ static void crypto_pump_requests(struct crypto_engine *engine, if (ret) { dev_err(engine->dev, "failed to prepare request: %d\n", ret); - goto req_err; + goto req_err_2; } - engine->cur_req_prepared = true; } if (!enginectx->op.do_one_request) { dev_err(engine->dev, "failed to do request\n"); ret = -EINVAL; - goto req_err; + goto req_err_1; } + ret = enginectx->op.do_one_request(engine, async_req); if (ret) { dev_err(engine->dev, "Failed to do one request from queue: %d\n", ret); - goto req_err; + goto req_err_1; } - return; -req_err: - crypto_finalize_request(engine, async_req, ret); - return; + goto retry; + +req_err_1: + if (enginectx->op.unprepare_request) { + ret = enginectx->op.unprepare_request(engine, async_req); + if (ret) + dev_err(engine->dev, "failed to unprepare request\n"); + } +req_err_2: + async_req->complete(async_req, ret); +retry: + if (engine->can_enqueue_hardware) { + spin_lock_irqsave(&engine->queue_lock, flags); + goto start_request; + } + return; out: spin_unlock_irqrestore(&engine->queue_lock, flags); } @@ -386,15 +385,25 @@ int crypto_engine_stop(struct crypto_engine *engine) EXPORT_SYMBOL_GPL(crypto_engine_stop); /** - * crypto_engine_alloc_init - allocate crypto hardware engine structure and - * initialize it. + * crypto_engine_alloc_init_and_set - allocate crypto hardware engine structure + * and initialize it by setting the maximum number of entries in the software + * crypto-engine queue. * @dev: the device attached with one hardware engine + * @cbk: pointer to a callback function to be invoked when pumping requests + * to check whether the hardware can process a new request. + * This has the form: + * callback(struct device *dev) + * where: + * @dev: contains the device that processed this response. * @rt: whether this queue is set to run as a realtime task + * @qlen: maximum size of the crypto-engine queue * * This must be called from context that can sleep. * Return: the crypto engine structure on success, else NULL. */ -struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) +struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev, + bool (*cbk)(struct device *dev), + bool rt, int qlen) { struct sched_param param = { .sched_priority = MAX_RT_PRIO / 2 }; struct crypto_engine *engine; @@ -411,12 +420,12 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) engine->running = false; engine->busy = false; engine->idling = false; - engine->cur_req_prepared = false; engine->priv_data = dev; + engine->can_enqueue_hardware = cbk; snprintf(engine->name, sizeof(engine->name), "%s-engine", dev_name(dev)); - crypto_init_queue(&engine->queue, CRYPTO_ENGINE_MAX_QLEN); + crypto_init_queue(&engine->queue, qlen); spin_lock_init(&engine->queue_lock); engine->kworker = kthread_create_worker(0, "%s", engine->name); @@ -433,6 +442,22 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) return engine; } +EXPORT_SYMBOL_GPL(crypto_engine_alloc_init_and_set); + +/** + * crypto_engine_alloc_init - allocate crypto hardware engine structure and + * initialize it. + * @dev: the device attached with one hardware engine + * @rt: whether this queue is set to run as a realtime task + * + * This must be called from context that can sleep. + * Return: the crypto engine structure on success, else NULL. + */ +struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) +{ + return crypto_engine_alloc_init_and_set(dev, NULL, rt, + CRYPTO_ENGINE_MAX_QLEN); +} EXPORT_SYMBOL_GPL(crypto_engine_alloc_init); /** diff --git a/include/crypto/engine.h b/include/crypto/engine.h index e29cd67..15d1150 100644 --- a/include/crypto/engine.h +++ b/include/crypto/engine.h @@ -24,7 +24,6 @@ * @idling: the engine is entering idle state * @busy: request pump is busy * @running: the engine is on working - * @cur_req_prepared: current request is prepared * @list: link with the global crypto engine list * @queue_lock: spinlock to syncronise access to request queue * @queue: the crypto queue of the engine @@ -35,17 +34,17 @@ * @unprepare_crypt_hardware: there are currently no more requests on the * queue so the subsystem notifies the driver that it may relax the * hardware by issuing this call + * @can_enqueue_hardware: callback to check whether the hardware can process + * a new request * @kworker: kthread worker struct for request pump * @pump_requests: work struct for scheduling work to the request pump * @priv_data: the engine private data - * @cur_req: the current request which is on processing */ struct crypto_engine { char name[ENGINE_NAME_LEN]; bool idling; bool busy; bool running; - bool cur_req_prepared; struct list_head list; spinlock_t queue_lock; @@ -56,12 +55,12 @@ struct crypto_engine { int (*prepare_crypt_hardware)(struct crypto_engine *engine); int (*unprepare_crypt_hardware)(struct crypto_engine *engine); + bool (*can_enqueue_hardware)(struct device *dev); struct kthread_worker *kworker; struct kthread_work pump_requests; void *priv_data; - struct crypto_async_request *cur_req; }; /* @@ -102,6 +101,9 @@ void crypto_finalize_skcipher_request(struct crypto_engine *engine, int crypto_engine_start(struct crypto_engine *engine); int crypto_engine_stop(struct crypto_engine *engine); struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt); +struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev, + bool (*cbk)(struct device *dev), + bool rt, int qlen); int crypto_engine_exit(struct crypto_engine *engine); #endif /* _CRYPTO_ENGINE_H */ -- 2.1.0