Received: by 2002:a25:e7d8:0:0:0:0:0 with SMTP id e207csp1878327ybh; Sun, 8 Mar 2020 15:52:25 -0700 (PDT) X-Google-Smtp-Source: ADFU+vuaHeAASlRmS2C/cHhtLtlOK9qVOIDfS9j+Sd2Qy4NnpjrUKQk9/bOA5zyJu9Fzz5kd0huL X-Received: by 2002:a9d:4b15:: with SMTP id q21mr2446217otf.84.1583707945075; Sun, 08 Mar 2020 15:52:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1583707945; cv=none; d=google.com; s=arc-20160816; b=CeIgnJ1hNJ8wNfpm6uM/CojjOLyxdhAguwEqUaDqrJ3ClZJrYKDFHyFj00Ze3iEUpQ hoDAEyYQzUIw5tPes03LcDhOa3oXpTIaUytwIl5ynN0DeWH6lR1J/jOvBce0TtRfyWs6 0MpM6f3bN52JzZcMdB2BBUk4dxjIKbv6EzNpVvRkJHAV0OvmybTQbedYgccDX7OP3yqp IJuO67kfyK2kPQUdWR4WmcN7Lg1XMQdTuKzY18IP+O9Sm/0m94EkUcsKpE34fB5MbmzG dkVjsfzyo8Dq8J/VIEgyV9PotlsF3ti6kt5yeiXqE453cZko3af5mdYX85ibT/QLOAAL mO2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=5abTrMEoZgsvUwWddeJSFlwWFIGl9BaAu2Eden9f/lM=; b=aPbsoijuD8rAPldeCTq4TyIw9zwYpXe3zGGjQfg9y+uktHT98XG+Pz+R9AkOUq7gVc 0UTQYV/32aHtS7ou4H7sQXchKD6ElTC2z8YHVJhz80iguWNsH3gtcIRKIwRxKmo4Bk1t /yv8z+ZlmHlOo6IH9g/uohp/1+wEaZHvl+cfAmHn4sGenebr3PSMMPttyrUnukbrppT1 LLPGVTrLmi1jhg0i5Voyr010c/3BXTwTXkUlV8JyNRky2Ezpn8e1RlsTV+jvf4uyfaja OhZRuAypyYMtRMN2JvKh/C9I3mBDcEqalGY3z7V58FDO25USK/Qc3ZSjAbA1Pk8LYYcP yU3g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n14si4918820otr.241.2020.03.08.15.52.02; Sun, 08 Mar 2020 15:52:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726442AbgCHWvy (ORCPT + 99 others); Sun, 8 Mar 2020 18:51:54 -0400 Received: from inva021.nxp.com ([92.121.34.21]:57088 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726346AbgCHWvy (ORCPT ); Sun, 8 Mar 2020 18:51:54 -0400 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 8BEF9200A7B; Sun, 8 Mar 2020 23:51:51 +0100 (CET) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 7CDDE20020F; Sun, 8 Mar 2020 23:51:51 +0100 (CET) Received: from lorenz.ea.freescale.net (lorenz.ea.freescale.net [10.171.71.5]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id E1DD7204CC; Sun, 8 Mar 2020 23:51:50 +0100 (CET) From: Iuliana Prodan To: Herbert Xu , Baolin Wang , Ard Biesheuvel , Corentin Labbe , Horia Geanta , Maxime Coquelin , Alexandre Torgue , Maxime Ripard Cc: Aymen Sghaier , "David S. Miller" , Silvano Di Ninno , Franck Lenormand , linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, linux-imx , Iuliana Prodan Subject: [PATCH v4 1/2] crypto: engine - support for parallel requests Date: Mon, 9 Mar 2020 00:51:32 +0200 Message-Id: <1583707893-23699-2-git-send-email-iuliana.prodan@nxp.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1583707893-23699-1-git-send-email-iuliana.prodan@nxp.com> References: <1583707893-23699-1-git-send-email-iuliana.prodan@nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Added support for executing multiple requests, in parallel, for crypto engine. Two new variables are added, cnt_do_req (number of requests accepted by hardware) and cnt_finalize (number of completed/ finalized requests), which keeps track whether the hardware can enqueue new requests. cnt_do_req will be set based on the return value of do_one_request(), which is expected to be: > 0: if hardware still has space in its queue. A driver can implement do_one_request() to return the number of free entries in hardware queue; 0: if the request was accepted, by hardware, with success, but the hardware doesn't support multiple requests or there is no space left in the hardware queue. This is to keep the backward compatibility of crypto-engine. < 0: error. cnt_finalize will be increased in crypto_finalize_request. The new crypto_engine_alloc_init_and_set function, initialize crypto-engine, sets the maximum size for crypto-engine software queue (not hardcoded anymore) and the cnt_do_req and cnt_finalize variables will be set, by default, to 0. On crypto_pump_requests(), if do_one_request() returns > 0, a new request is send to hardware, until there is no space and do_one_request() returns 0. By default, if do_one_request() returns 0, crypto-engine will work as before - will send requests to hardware, one-by-one, on crypto_pump_requests(), and complete it, on crypto_finalize_request(), and so on. To support multiple requests, in each driver, do_one_request() needs to be updated to return > 0, if there is space in hardware queue, otherwise will work as before. Signed-off-by: Iuliana Prodan --- crypto/crypto_engine.c | 122 ++++++++++++++++++++++++++++++++++-------------- include/crypto/engine.h | 11 +++-- 2 files changed, 94 insertions(+), 39 deletions(-) diff --git a/crypto/crypto_engine.c b/crypto/crypto_engine.c index eb029ff..dbfd53c2 100644 --- a/crypto/crypto_engine.c +++ b/crypto/crypto_engine.c @@ -22,30 +22,27 @@ * @err: error number */ static void crypto_finalize_request(struct crypto_engine *engine, - struct crypto_async_request *req, int err) + struct crypto_async_request *req, int err) { unsigned long flags; - bool finalize_cur_req = false; int ret; struct crypto_engine_ctx *enginectx; spin_lock_irqsave(&engine->queue_lock, flags); - if (engine->cur_req == req) - finalize_cur_req = true; + /* + * Increment the number of requests completed. + * We'll need it to start the engine on pump_requests, + * if hardware can enqueue new requests. + */ + engine->cnt_finalize++; spin_unlock_irqrestore(&engine->queue_lock, flags); - if (finalize_cur_req) { - enginectx = crypto_tfm_ctx(req->tfm); - if (engine->cur_req_prepared && - enginectx->op.unprepare_request) { - ret = enginectx->op.unprepare_request(engine, req); - if (ret) - dev_err(engine->dev, "failed to unprepare request\n"); - } - spin_lock_irqsave(&engine->queue_lock, flags); - engine->cur_req = NULL; - engine->cur_req_prepared = false; - spin_unlock_irqrestore(&engine->queue_lock, flags); + enginectx = crypto_tfm_ctx(req->tfm); + if (enginectx->op.prepare_request && + enginectx->op.unprepare_request) { + ret = enginectx->op.unprepare_request(engine, req); + if (ret) + dev_err(engine->dev, "failed to unprepare request\n"); } req->complete(req, err); @@ -69,12 +66,19 @@ static void crypto_pump_requests(struct crypto_engine *engine, unsigned long flags; bool was_busy = false; int ret; + int can_enq_more = 0; struct crypto_engine_ctx *enginectx; spin_lock_irqsave(&engine->queue_lock, flags); - /* Make sure we are not already running a request */ - if (engine->cur_req) + /* + * If hardware cannot enqueue new requests, + * stop the engine, until requests are processed and + * hardware can execute new requests. + * We'll start the engine on request completion + * (crypto_finalize_request). + */ + if (engine->cnt_finalize != engine->cnt_do_req) goto out; /* If another context is idling then defer */ @@ -108,13 +112,13 @@ static void crypto_pump_requests(struct crypto_engine *engine, goto out; } +start_request: /* Get the fist request from the engine queue to handle */ backlog = crypto_get_backlog(&engine->queue); async_req = crypto_dequeue_request(&engine->queue); if (!async_req) goto out; - engine->cur_req = async_req; if (backlog) backlog->complete(backlog, -EINPROGRESS); @@ -130,7 +134,7 @@ static void crypto_pump_requests(struct crypto_engine *engine, ret = engine->prepare_crypt_hardware(engine); if (ret) { dev_err(engine->dev, "failed to prepare crypt hardware\n"); - goto req_err; + goto req_err_2; } } @@ -141,25 +145,53 @@ static void crypto_pump_requests(struct crypto_engine *engine, if (ret) { dev_err(engine->dev, "failed to prepare request: %d\n", ret); - goto req_err; + goto req_err_2; } - engine->cur_req_prepared = true; } if (!enginectx->op.do_one_request) { dev_err(engine->dev, "failed to do request\n"); ret = -EINVAL; - goto req_err; + goto req_err_1; } + ret = enginectx->op.do_one_request(engine, async_req); - if (ret) { - dev_err(engine->dev, "Failed to do one request from queue: %d\n", ret); - goto req_err; + can_enq_more = ret; + if (can_enq_more < 0) { + dev_err(engine->dev, "Failed to do one request from queue: %d\n", + ret); + goto req_err_1; + } + + goto retry; + +req_err_1: + if (enginectx->op.unprepare_request) { + ret = enginectx->op.unprepare_request(engine, async_req); + if (ret) + dev_err(engine->dev, "failed to unprepare request\n"); } - return; -req_err: - crypto_finalize_request(engine, async_req, ret); - return; +req_err_2: + async_req->complete(async_req, ret); + +retry: + spin_lock_irqsave(&engine->queue_lock, flags); + + /* + * If hardware can still enqueue requests, + * increment the number of requests accepted by hardware. + * We'll need it to start the engine on pump_requests. + */ + if (can_enq_more >= 0) + engine->cnt_do_req++; + + /* + * We'll send new requests to engine, if there is space. + * If the 2 counters are equal, that means that all requests + * were executed, so we can send new requests. + */ + if (engine->cnt_finalize == engine->cnt_do_req || can_enq_more > 0) + goto start_request; out: spin_unlock_irqrestore(&engine->queue_lock, flags); @@ -386,15 +418,18 @@ int crypto_engine_stop(struct crypto_engine *engine) EXPORT_SYMBOL_GPL(crypto_engine_stop); /** - * crypto_engine_alloc_init - allocate crypto hardware engine structure and - * initialize it. + * crypto_engine_alloc_init_and_set - allocate crypto hardware engine structure + * and initialize it by setting the maximum number of entries in the software + * crypto-engine queue. * @dev: the device attached with one hardware engine * @rt: whether this queue is set to run as a realtime task + * @qlen: maximum size of the crypto-engine queue * * This must be called from context that can sleep. * Return: the crypto engine structure on success, else NULL. */ -struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) +struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev, + bool rt, int qlen) { struct sched_param param = { .sched_priority = MAX_RT_PRIO / 2 }; struct crypto_engine *engine; @@ -411,12 +446,13 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) engine->running = false; engine->busy = false; engine->idling = false; - engine->cur_req_prepared = false; + engine->cnt_do_req = 0; + engine->cnt_finalize = 0; engine->priv_data = dev; snprintf(engine->name, sizeof(engine->name), "%s-engine", dev_name(dev)); - crypto_init_queue(&engine->queue, CRYPTO_ENGINE_MAX_QLEN); + crypto_init_queue(&engine->queue, qlen); spin_lock_init(&engine->queue_lock); engine->kworker = kthread_create_worker(0, "%s", engine->name); @@ -433,6 +469,22 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) return engine; } +EXPORT_SYMBOL_GPL(crypto_engine_alloc_init_and_set); + +/** + * crypto_engine_alloc_init - allocate crypto hardware engine structure and + * initialize it. + * @dev: the device attached with one hardware engine + * @rt: whether this queue is set to run as a realtime task + * + * This must be called from context that can sleep. + * Return: the crypto engine structure on success, else NULL. + */ +struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt) +{ + return crypto_engine_alloc_init_and_set(dev, rt, + CRYPTO_ENGINE_MAX_QLEN); +} EXPORT_SYMBOL_GPL(crypto_engine_alloc_init); /** diff --git a/include/crypto/engine.h b/include/crypto/engine.h index e29cd67..33a5be2 100644 --- a/include/crypto/engine.h +++ b/include/crypto/engine.h @@ -24,7 +24,8 @@ * @idling: the engine is entering idle state * @busy: request pump is busy * @running: the engine is on working - * @cur_req_prepared: current request is prepared + * @cnt_finalize: number of completed/finalized requests + * @cnt_do_req: number of requests accepted by hardware * @list: link with the global crypto engine list * @queue_lock: spinlock to syncronise access to request queue * @queue: the crypto queue of the engine @@ -38,14 +39,15 @@ * @kworker: kthread worker struct for request pump * @pump_requests: work struct for scheduling work to the request pump * @priv_data: the engine private data - * @cur_req: the current request which is on processing */ struct crypto_engine { char name[ENGINE_NAME_LEN]; bool idling; bool busy; bool running; - bool cur_req_prepared; + + u32 cnt_finalize; + u32 cnt_do_req; struct list_head list; spinlock_t queue_lock; @@ -61,7 +63,6 @@ struct crypto_engine { struct kthread_work pump_requests; void *priv_data; - struct crypto_async_request *cur_req; }; /* @@ -102,6 +103,8 @@ void crypto_finalize_skcipher_request(struct crypto_engine *engine, int crypto_engine_start(struct crypto_engine *engine); int crypto_engine_stop(struct crypto_engine *engine); struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt); +struct crypto_engine *crypto_engine_alloc_init_and_set(struct device *dev, + bool rt, int qlen); int crypto_engine_exit(struct crypto_engine *engine); #endif /* _CRYPTO_ENGINE_H */ -- 2.1.0