Received: by 2002:ab2:1347:0:b0:1f4:ac9d:b246 with SMTP id g7csp403607lqg; Thu, 11 Apr 2024 06:41:14 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCU5G/A/FpRFu8bn6zbER20jWCDMHshKqg4m/UcQDC2D5zE7Wa+7dXxbE0y5RmICmVVeQYQX/Gqzfq3+lG8pdSRRZWJFa9dVRFTUfk3BHA== X-Google-Smtp-Source: AGHT+IH5NoPvgaTuemPR+B9mu+d+cTyw522nyfKzUyjEHwor+A0Tp2xcwIM3A3A6KIngxBGHEpI/ X-Received: by 2002:a17:902:ec83:b0:1e2:9ac2:4a03 with SMTP id x3-20020a170902ec8300b001e29ac24a03mr5222667plg.28.1712842874139; Thu, 11 Apr 2024 06:41:14 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712842874; cv=pass; d=google.com; s=arc-20160816; b=xkhgwrHm8DLlpmamzXPZ/8CxDnB2psS4pG7PshBELtJDOeUzOIxhNhhO9EaABij7uQ XtTDodCgkEr4A8D0eYmnpJbBOH2kSonjeyHRv6jaXbcRkK0kg7fCfj/WiDdGuUf9pjoO i+tyMGBAnRlIBDX0yfY4k6SeLUX62eNddqXx/c64QxtN5DYu6YtzCzERM/FuynBOtRnW 47qT6Y1svVkZm9gg3JksU2lIN6xt6JX2S5q7DJ+tSl1mBB1+u6RR1POmJHRGYviUJbc1 MnSz5WwITkOyFMLdaUGX+AC8zckQHOpI6JLH/c1o4hUT+6FhD2QUYJi+BSHJKTsQZEcd wyHw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature:dkim-filter; bh=oSSG1nFN48Pp27lrPpipw+gU7eoF+L7UQCvM42iFT9k=; fh=7+QqR+U+m0B5ryyywou1kgMZVOfT/GgVHz0ZH2Lvlxg=; b=f4XuYjucijD3V83YhJyxfgLvjL/U+tNOP/0eD+DwG8HHCpoI+3RLl1VXrDhlkB9VdM h6T+NvRIniKh7WWmUhgu1RG0pIcfk4s3bQugIH9YNuiErZ+Tb2mM0VvjYtH+h1FWklOF J0UkaZLmaHG0E9riUZm2smRtnATvHt4jQwsvtkNx5BVGso0vhx6DPnehVSBHJ8owlTLO +iQBX9xC4poovZSJqy7j15JFW6CmdzgDzk0gysU6eSIipxYzopZV7ta+t8U2XJG9/e4Z GRlentvzEa2FAFfSATD6oZ/rk9OqccmxYJZCDWdR1dLgv2kkkMt16UGy2F9L4bqMjfSA Felw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@salutedevices.com header.s=mail header.b=VGjYg8sY; arc=pass (i=1 spf=pass spfdomain=salutedevices.com dkim=pass dkdomain=salutedevices.com dmarc=pass fromdomain=salutedevices.com); spf=pass (google.com: domain of linux-crypto+bounces-3462-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-crypto+bounces-3462-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=salutedevices.com Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id d7-20020a170902b70700b001e44412d17fsi1210795pls.525.2024.04.11.06.41.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Apr 2024 06:41:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-crypto+bounces-3462-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@salutedevices.com header.s=mail header.b=VGjYg8sY; arc=pass (i=1 spf=pass spfdomain=salutedevices.com dkim=pass dkdomain=salutedevices.com dmarc=pass fromdomain=salutedevices.com); spf=pass (google.com: domain of linux-crypto+bounces-3462-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-crypto+bounces-3462-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=salutedevices.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id C185B283D5B for ; Thu, 11 Apr 2024 13:41:13 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B49FD152E0A; Thu, 11 Apr 2024 13:39:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=salutedevices.com header.i=@salutedevices.com header.b="VGjYg8sY" X-Original-To: linux-crypto@vger.kernel.org Received: from mx1.sberdevices.ru (mx1.sberdevices.ru [37.18.73.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66A65152522; Thu, 11 Apr 2024 13:39:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=37.18.73.165 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712842749; cv=none; b=MM8vm2Tvm8Mnh6MvkdSxkUSH++ZetnoE37RofhY8qs0kRbLmN2kRcgPkKluyQxRXaR75nM9XbJ4tYLjT5mxrgFgKqioY/Fc+8L52TbERyBNShVxdIwS9Og3mmGoglIWzMYLDOFix0CnO88OL57Gc5OABIs27Mue27Pif4h9yJu4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712842749; c=relaxed/simple; bh=sRVzDtLuAYid7bpL1vEiJDXSCoFwNdDImg0HLGp1GxI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=jZr5dpmx6Reef1jhvwsjEUCKfdyfTEl9ixYdcG5+TTqI6Lr0UyzogvQ7szoSVOdsdQQTgKZA9xF193qH9GZJcoFy4qedMiBOb0JPSkkmhXBWhSogXPTN7cMKhJzuA6B+Zm5zQbzW8eRDJJOjIbDk7WuP4cx4f54yMUQQ2VzjdyM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=salutedevices.com; spf=pass smtp.mailfrom=salutedevices.com; dkim=pass (2048-bit key) header.d=salutedevices.com header.i=@salutedevices.com header.b=VGjYg8sY; arc=none smtp.client-ip=37.18.73.165 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=salutedevices.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=salutedevices.com Received: from p-infra-ksmg-sc-msk01 (localhost [127.0.0.1]) by mx1.sberdevices.ru (Postfix) with ESMTP id AA3A210003E; Thu, 11 Apr 2024 16:39:05 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.sberdevices.ru AA3A210003E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=salutedevices.com; s=mail; t=1712842745; bh=oSSG1nFN48Pp27lrPpipw+gU7eoF+L7UQCvM42iFT9k=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:From; b=VGjYg8sYmM/LjkpQJGvLAj/wildQxch++DddYiGkld5GOUKm7wMqxO4+83Cy3Gpie 9W/x8iruZc824NvQ6Qidbg788Nu1cXGHh4DlF8YytzkY/Tl1oXBdN7BKkX16Vsx1go 0V3b+UmYOSRPZk+gudL8xbHO9nTK1BcUyRQr3u62MUEXgfNwEGI8XU4Y+BrjcyJ198 3QK4XhABU+GQ+M9vJxUEpzUoc5MdGXSezLOnW34oxhz4fNqOk0DzN/QQD+V9igcU8r bhUbjkVo3Y5D1w+YuCQ/91DX9tifZzfWRbHw2CAOCvdIZIA7OePgKau5VFHoJWed3P yLIM777szh0AA== Received: from smtp.sberdevices.ru (p-i-exch-sc-m02.sberdevices.ru [172.16.192.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sberdevices.ru (Postfix) with ESMTPS; Thu, 11 Apr 2024 16:39:05 +0300 (MSK) Received: from user-A520M-DS3H.sberdevices.ru (100.64.160.123) by p-i-exch-sc-m02.sberdevices.ru (172.16.192.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Thu, 11 Apr 2024 16:39:03 +0300 From: Alexey Romanov To: , , , , , , , , , , CC: , , , , , , Alexey Romanov Subject: [PATCH v7 09/23] drivers: crypto: meson: process more than MAXDESCS descriptors Date: Thu, 11 Apr 2024 16:38:18 +0300 Message-ID: <20240411133832.2896463-10-avromanov@salutedevices.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411133832.2896463-1-avromanov@salutedevices.com> References: <20240411133832.2896463-1-avromanov@salutedevices.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: p-i-exch-sc-m01.sberdevices.ru (172.16.192.107) To p-i-exch-sc-m02.sberdevices.ru (172.16.192.103) X-KSMG-Rule-ID: 10 X-KSMG-Message-Action: clean X-KSMG-AntiSpam-Lua-Profiles: 184672 [Apr 11 2024] X-KSMG-AntiSpam-Version: 6.1.0.4 X-KSMG-AntiSpam-Envelope-From: avromanov@salutedevices.com X-KSMG-AntiSpam-Rate: 0 X-KSMG-AntiSpam-Status: not_detected X-KSMG-AntiSpam-Method: none X-KSMG-AntiSpam-Auth: dkim=none X-KSMG-AntiSpam-Info: LuaCore: 16 0.3.16 6e64c33514fcbd07e515710c86ba61de7f56194e, {Tracking_from_domain_doesnt_match_to}, smtp.sberdevices.ru:5.0.1,7.1.1;100.64.160.123:7.1.2;d41d8cd98f00b204e9800998ecf8427e.com:7.1.1;127.0.0.199:7.1.2;salutedevices.com:7.1.1, FromAlignment: s, ApMailHostAddress: 100.64.160.123 X-MS-Exchange-Organization-SCL: -1 X-KSMG-AntiSpam-Interceptor-Info: scan successful X-KSMG-AntiPhishing: Clean X-KSMG-LinksScanning: Clean X-KSMG-AntiVirus: Kaspersky Secure Mail Gateway, version 2.0.1.6960, bases: 2024/04/11 09:22:00 #24744908 X-KSMG-AntiVirus-Status: Clean, skipped 1. The old alhorithm was not designed to process a large amount of memory, and therefore gave incorrect results. 2. Not all Amlogic SoC's use 3 KEY/IV descriptors. Add keyiv descriptors count parameter to platform data. Signed-off-by: Alexey Romanov --- drivers/crypto/amlogic/amlogic-gxl-cipher.c | 445 ++++++++++++-------- drivers/crypto/amlogic/amlogic-gxl-core.c | 1 + drivers/crypto/amlogic/amlogic-gxl.h | 2 + 3 files changed, 283 insertions(+), 165 deletions(-) diff --git a/drivers/crypto/amlogic/amlogic-gxl-cipher.c b/drivers/crypto/amlogic/amlogic-gxl-cipher.c index c662c4b86e97..dd2478ae7c54 100644 --- a/drivers/crypto/amlogic/amlogic-gxl-cipher.c +++ b/drivers/crypto/amlogic/amlogic-gxl-cipher.c @@ -17,35 +17,41 @@ #include #include "amlogic-gxl.h" -static bool meson_cipher_need_fallback(struct skcipher_request *areq) +static bool meson_cipher_need_fallback_sg(struct skcipher_request *areq, + struct scatterlist *sg) { - struct scatterlist *src_sg = areq->src; - struct scatterlist *dst_sg = areq->dst; + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); + unsigned int blocksize = crypto_skcipher_blocksize(tfm); + unsigned int cryptlen = areq->cryptlen; + + while (cryptlen) { + unsigned int len = min(cryptlen, sg->length); + + if (!IS_ALIGNED(sg->offset, sizeof(u32))) + return true; + if (len % blocksize != 0) + return true; + + cryptlen -= len; + sg = sg_next(sg); + } + + return false; +} +static bool meson_cipher_need_fallback(struct skcipher_request *areq) +{ if (areq->cryptlen == 0) return true; - if (sg_nents(src_sg) != sg_nents(dst_sg)) + if (meson_cipher_need_fallback_sg(areq, areq->src)) return true; - /* KEY/IV descriptors use 3 desc */ - if (sg_nents(src_sg) > MAXDESC - 3 || sg_nents(dst_sg) > MAXDESC - 3) - return true; + if (areq->dst == areq->src) + return false; - while (src_sg && dst_sg) { - if ((src_sg->length % 16) != 0) - return true; - if ((dst_sg->length % 16) != 0) - return true; - if (src_sg->length != dst_sg->length) - return true; - if (!IS_ALIGNED(src_sg->offset, sizeof(u32))) - return true; - if (!IS_ALIGNED(dst_sg->offset, sizeof(u32))) - return true; - src_sg = sg_next(src_sg); - dst_sg = sg_next(dst_sg); - } + if (meson_cipher_need_fallback_sg(areq, areq->dst)) + return true; return false; } @@ -76,184 +82,293 @@ static int meson_cipher_do_fallback(struct skcipher_request *areq) return err; } -static int meson_cipher(struct skcipher_request *areq) +struct cipher_ctx { + struct { + dma_addr_t addr; + unsigned int len; + } keyiv; + + struct skcipher_request *areq; + struct scatterlist *src_sg; + struct scatterlist *dst_sg; + void *bkeyiv; + + unsigned int src_offset; + unsigned int dst_offset; + unsigned int cryptlen; + unsigned int tloffset; +}; + +static int meson_map_scatterlist(struct skcipher_request *areq, struct meson_dev *mc) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); - struct meson_cipher_tfm_ctx *op = crypto_skcipher_ctx(tfm); - struct meson_cipher_req_ctx *rctx = skcipher_request_ctx(areq); - struct meson_dev *mc = op->mc; - struct skcipher_alg *alg = crypto_skcipher_alg(tfm); - struct meson_alg_template *algt; - int flow = rctx->flow; - unsigned int todo, eat, len; - struct scatterlist *src_sg = areq->src; - struct scatterlist *dst_sg = areq->dst; - struct meson_desc *desc; int nr_sgs, nr_sgd; - int i, err = 0; - unsigned int keyivlen, ivsize, offset, tloffset; - dma_addr_t phykeyiv; - void *backup_iv = NULL, *bkeyiv; - u32 v; - - algt = container_of(alg, struct meson_alg_template, alg.skcipher.base); - - dev_dbg(mc->dev, "%s %s %u %x IV(%u) key=%u flow=%d\n", __func__, - crypto_tfm_alg_name(areq->base.tfm), - areq->cryptlen, - rctx->op_dir, crypto_skcipher_ivsize(tfm), - op->keylen, flow); - -#ifdef CONFIG_CRYPTO_DEV_AMLOGIC_GXL_DEBUG - algt->stat_req++; - mc->chanlist[flow].stat_req++; -#endif - - /* - * The hardware expect a list of meson_desc structures. - * The 2 first structures store key - * The third stores IV - */ - bkeyiv = kzalloc(48, GFP_KERNEL | GFP_DMA); - if (!bkeyiv) - return -ENOMEM; - - memcpy(bkeyiv, op->key, op->keylen); - keyivlen = op->keylen; - - ivsize = crypto_skcipher_ivsize(tfm); - if (areq->iv && ivsize > 0) { - if (ivsize > areq->cryptlen) { - dev_err(mc->dev, "invalid ivsize=%d vs len=%d\n", ivsize, areq->cryptlen); - err = -EINVAL; - goto theend; - } - memcpy(bkeyiv + 32, areq->iv, ivsize); - keyivlen = 48; - if (rctx->op_dir == MESON_DECRYPT) { - backup_iv = kzalloc(ivsize, GFP_KERNEL); - if (!backup_iv) { - err = -ENOMEM; - goto theend; - } - offset = areq->cryptlen - ivsize; - scatterwalk_map_and_copy(backup_iv, areq->src, offset, - ivsize, 0); - } - } - if (keyivlen == AES_KEYSIZE_192) - keyivlen = AES_MAX_KEY_SIZE; - - phykeyiv = dma_map_single(mc->dev, bkeyiv, keyivlen, - DMA_TO_DEVICE); - err = dma_mapping_error(mc->dev, phykeyiv); - if (err) { - dev_err(mc->dev, "Cannot DMA MAP KEY IV\n"); - goto theend; - } - - tloffset = 0; - eat = 0; - i = 0; - while (keyivlen > eat) { - desc = &mc->chanlist[flow].tl[tloffset]; - memset(desc, 0, sizeof(struct meson_desc)); - todo = min(keyivlen - eat, 16u); - desc->t_src = cpu_to_le32(phykeyiv + i * 16); - desc->t_dst = cpu_to_le32(i * 16); - v = DESC_MODE_KEY | DESC_OWN | 16; - desc->t_status = cpu_to_le32(v); - - eat += todo; - i++; - tloffset++; - } if (areq->src == areq->dst) { nr_sgs = dma_map_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_BIDIRECTIONAL); if (!nr_sgs) { dev_err(mc->dev, "Invalid SG count %d\n", nr_sgs); - err = -EINVAL; - goto theend; + return -EINVAL; } - nr_sgd = nr_sgs; } else { nr_sgs = dma_map_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_TO_DEVICE); - if (!nr_sgs || nr_sgs > MAXDESC - 3) { + if (!nr_sgs) { dev_err(mc->dev, "Invalid SG count %d\n", nr_sgs); - err = -EINVAL; - goto theend; + return -EINVAL; } + nr_sgd = dma_map_sg(mc->dev, areq->dst, sg_nents(areq->dst), DMA_FROM_DEVICE); - if (!nr_sgd || nr_sgd > MAXDESC - 3) { + if (!nr_sgd) { + dma_unmap_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_TO_DEVICE); dev_err(mc->dev, "Invalid SG count %d\n", nr_sgd); - err = -EINVAL; - goto theend; + return -EINVAL; } } - src_sg = areq->src; - dst_sg = areq->dst; - len = areq->cryptlen; - while (src_sg) { - desc = &mc->chanlist[flow].tl[tloffset]; - memset(desc, 0, sizeof(struct meson_desc)); - - desc->t_src = cpu_to_le32(sg_dma_address(src_sg)); - desc->t_dst = cpu_to_le32(sg_dma_address(dst_sg)); - todo = min(len, sg_dma_len(src_sg)); - v = op->keymode | DESC_OWN | todo | algt->blockmode; - if (rctx->op_dir) - v |= DESC_ENCRYPTION; - len -= todo; - - if (!sg_next(src_sg)) - v |= DESC_LAST; - desc->t_status = cpu_to_le32(v); - tloffset++; - src_sg = sg_next(src_sg); - dst_sg = sg_next(dst_sg); + return 0; +} + +static void meson_unmap_scatterlist(struct skcipher_request *areq, struct meson_dev *mc) +{ + if (areq->src == areq->dst) { + dma_unmap_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_BIDIRECTIONAL); + } else { + dma_unmap_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_TO_DEVICE); + dma_unmap_sg(mc->dev, areq->dst, sg_nents(areq->dst), DMA_FROM_DEVICE); } +} - reinit_completion(&mc->chanlist[flow].complete); - meson_dma_start(mc, flow); +static void meson_setup_keyiv_descs(struct cipher_ctx *ctx) +{ + struct meson_cipher_req_ctx *rctx = skcipher_request_ctx(ctx->areq); + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(ctx->areq); + struct skcipher_alg *alg = crypto_skcipher_alg(tfm); + struct meson_alg_template *algt = container_of(alg, + struct meson_alg_template, alg.skcipher.base); + struct meson_cipher_tfm_ctx *op = crypto_skcipher_ctx(tfm); + struct meson_dev *mc = op->mc; + unsigned int ivsize = crypto_skcipher_ivsize(tfm); + unsigned int blockmode = algt->blockmode; + int i; + + if (ctx->tloffset) + return; + + if (blockmode == DESC_OPMODE_CBC) { + memcpy(ctx->bkeyiv + AES_MAX_KEY_SIZE, ctx->areq->iv, ivsize); + dma_sync_single_for_device(mc->dev, ctx->keyiv.addr, + ctx->keyiv.len, DMA_TO_DEVICE); + } + + for (i = 0; i < mc->pdata->setup_desc_cnt; i++) { + struct meson_desc *desc = + &mc->chanlist[rctx->flow].tl[ctx->tloffset]; + int offset = i * 16; + + desc->t_src = cpu_to_le32(ctx->keyiv.addr + offset); + desc->t_dst = cpu_to_le32(offset); + desc->t_status = cpu_to_le32(DESC_OWN | DESC_MODE_KEY | ctx->keyiv.len); + + ctx->tloffset++; + } +} - err = wait_for_completion_interruptible_timeout(&mc->chanlist[flow].complete, +static bool meson_setup_data_descs(struct cipher_ctx *ctx) +{ + struct meson_cipher_req_ctx *rctx = skcipher_request_ctx(ctx->areq); + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(ctx->areq); + struct meson_cipher_tfm_ctx *op = crypto_skcipher_ctx(tfm); + struct skcipher_alg *alg = crypto_skcipher_alg(tfm); + struct meson_alg_template *algt = container_of(alg, + struct meson_alg_template, + alg.skcipher.base); + struct meson_dev *mc = op->mc; + struct meson_desc *desc = &mc->chanlist[rctx->flow].tl[ctx->tloffset]; + unsigned int blocksize = crypto_skcipher_blocksize(tfm); + unsigned int blockmode = algt->blockmode; + unsigned int maxlen = rounddown(DESC_MAXLEN, blocksize); + unsigned int todo; + u32 v; + + ctx->tloffset++; + + todo = min(ctx->cryptlen, maxlen); + todo = min(todo, ctx->cryptlen); + todo = min(todo, sg_dma_len(ctx->src_sg) - ctx->src_offset); + todo = min(todo, sg_dma_len(ctx->dst_sg) - ctx->dst_offset); + + desc->t_src = cpu_to_le32(sg_dma_address(ctx->src_sg) + ctx->src_offset); + desc->t_dst = cpu_to_le32(sg_dma_address(ctx->dst_sg) + ctx->dst_offset); + + ctx->cryptlen -= todo; + ctx->src_offset += todo; + ctx->dst_offset += todo; + + v = DESC_OWN | blockmode | op->keymode | todo; + if (rctx->op_dir == MESON_ENCRYPT) + v |= DESC_ENCRYPTION; + + if (!ctx->cryptlen || ctx->tloffset == MAXDESC) + v |= DESC_LAST; + + desc->t_status = cpu_to_le32(v); + + return v & DESC_LAST; +} + +static int meson_kick_hardware(struct cipher_ctx *ctx) +{ + struct meson_cipher_req_ctx *rctx = skcipher_request_ctx(ctx->areq); + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(ctx->areq); + struct meson_cipher_tfm_ctx *op = crypto_skcipher_ctx(tfm); + struct skcipher_alg *alg = crypto_skcipher_alg(tfm); + struct meson_alg_template *algt = container_of(alg, + struct meson_alg_template, + alg.skcipher.base); + struct meson_dev *mc = op->mc; + unsigned int ivsize = crypto_skcipher_ivsize(tfm); + unsigned int blockmode = algt->blockmode; + enum dma_data_direction new_iv_dir; + dma_addr_t new_iv_phys; + void *new_iv; + int err; + + if (blockmode == DESC_OPMODE_CBC) { + struct scatterlist *sg; + unsigned int offset; + + if (rctx->op_dir == MESON_ENCRYPT) { + sg = ctx->dst_sg; + offset = ctx->dst_offset; + new_iv_dir = DMA_FROM_DEVICE; + } else { + sg = ctx->src_sg; + offset = ctx->src_offset; + new_iv_dir = DMA_TO_DEVICE; + } + + if (ctx->areq->src == ctx->areq->dst) + new_iv_dir = DMA_BIDIRECTIONAL; + + offset -= ivsize; + new_iv = sg_virt(sg) + offset; + new_iv_phys = sg_dma_address(sg) + offset; + } + + if (blockmode == DESC_OPMODE_CBC && + rctx->op_dir == MESON_DECRYPT) { + dma_sync_single_for_cpu(mc->dev, new_iv_phys, + ivsize, new_iv_dir); + memcpy(ctx->areq->iv, new_iv, ivsize); + } + + reinit_completion(&mc->chanlist[rctx->flow].complete); + meson_dma_start(mc, rctx->flow); + err = wait_for_completion_interruptible_timeout(&mc->chanlist[rctx->flow].complete, msecs_to_jiffies(500)); if (err == 0) { - dev_err(mc->dev, "DMA timeout for flow %d\n", flow); - err = -EINVAL; + dev_err(mc->dev, "DMA timeout for flow %d\n", rctx->flow); + return -EINVAL; } else if (err < 0) { dev_err(mc->dev, "Waiting for DMA completion is failed (%d)\n", err); - } else { - /* No error */ - err = 0; + return err; } - dma_unmap_single(mc->dev, phykeyiv, keyivlen, DMA_TO_DEVICE); + if (blockmode == DESC_OPMODE_CBC && + rctx->op_dir == MESON_ENCRYPT) { + dma_sync_single_for_cpu(mc->dev, new_iv_phys, + ivsize, new_iv_dir); + memcpy(ctx->areq->iv, new_iv, ivsize); + } - if (areq->src == areq->dst) { - dma_unmap_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_BIDIRECTIONAL); - } else { - dma_unmap_sg(mc->dev, areq->src, sg_nents(areq->src), DMA_TO_DEVICE); - dma_unmap_sg(mc->dev, areq->dst, sg_nents(areq->dst), DMA_FROM_DEVICE); + ctx->tloffset = 0; + + return 0; +} + +static int meson_cipher(struct skcipher_request *areq) +{ + struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(areq); + struct meson_cipher_tfm_ctx *op = crypto_skcipher_ctx(tfm); + struct meson_cipher_req_ctx *rctx = skcipher_request_ctx(areq); + struct meson_dev *mc = op->mc; + struct skcipher_alg *alg = crypto_skcipher_alg(tfm); + struct meson_alg_template *algt; + struct cipher_ctx ctx = { + .areq = areq, + .src_offset = 0, + .dst_offset = 0, + .src_sg = areq->src, + .dst_sg = areq->dst, + .cryptlen = areq->cryptlen, + }; + int err; + + dev_dbg(mc->dev, "%s %s %u %x IV(%u) key=%u ctx.flow=%d\n", __func__, + crypto_tfm_alg_name(areq->base.tfm), + areq->cryptlen, + rctx->op_dir, crypto_skcipher_ivsize(tfm), + op->keylen, rctx->flow); + + algt = container_of(alg, struct meson_alg_template, alg.skcipher.base); + +#ifdef CONFIG_CRYPTO_DEV_AMLOGIC_GXL_DEBUG + algt->stat_req++; + mc->chanlist[rctx->flow].stat_req++; +#endif + + ctx.bkeyiv = kzalloc(48, GFP_KERNEL | GFP_DMA); + if (!ctx.bkeyiv) + return -ENOMEM; + + memcpy(ctx.bkeyiv, op->key, op->keylen); + ctx.keyiv.len = op->keylen; + if (ctx.keyiv.len == AES_KEYSIZE_192) + ctx.keyiv.len = AES_MAX_KEY_SIZE; + + ctx.keyiv.addr = dma_map_single(mc->dev, ctx.bkeyiv, ctx.keyiv.len, + DMA_TO_DEVICE); + err = dma_mapping_error(mc->dev, ctx.keyiv.addr); + if (err) { + dev_err(mc->dev, "Cannot DMA MAP KEY IV\n"); + goto free_keyiv; } - if (areq->iv && ivsize > 0) { - if (rctx->op_dir == MESON_DECRYPT) { - memcpy(areq->iv, backup_iv, ivsize); - } else { - scatterwalk_map_and_copy(areq->iv, areq->dst, - areq->cryptlen - ivsize, - ivsize, 0); + err = meson_map_scatterlist(areq, mc); + if (err) + goto unmap_keyiv; + + ctx.tloffset = 0; + + while (ctx.cryptlen) { + meson_setup_keyiv_descs(&ctx); + + if (meson_setup_data_descs(&ctx)) { + err = meson_kick_hardware(&ctx); + if (err) + break; + } + + if (ctx.src_offset == sg_dma_len(ctx.src_sg)) { + ctx.src_offset = 0; + ctx.src_sg = sg_next(ctx.src_sg); + } + + if (ctx.dst_offset == sg_dma_len(ctx.dst_sg)) { + ctx.dst_offset = 0; + ctx.dst_sg = sg_next(ctx.dst_sg); } } -theend: - kfree_sensitive(bkeyiv); - kfree_sensitive(backup_iv); + + meson_unmap_scatterlist(areq, mc); + +unmap_keyiv: + dma_unmap_single(mc->dev, ctx.keyiv.addr, ctx.keyiv.len, DMA_TO_DEVICE); + +free_keyiv: + kfree_sensitive(ctx.bkeyiv); return err; } diff --git a/drivers/crypto/amlogic/amlogic-gxl-core.c b/drivers/crypto/amlogic/amlogic-gxl-core.c index 98e63e67aa6e..5fea95e9876b 100644 --- a/drivers/crypto/amlogic/amlogic-gxl-core.c +++ b/drivers/crypto/amlogic/amlogic-gxl-core.c @@ -258,6 +258,7 @@ static void meson_crypto_remove(struct platform_device *pdev) static const struct meson_pdata meson_gxl_pdata = { .descs_reg = 0x0, .status_reg = 0x4, + .setup_desc_cnt = 3, }; static const struct of_device_id meson_crypto_of_match_table[] = { diff --git a/drivers/crypto/amlogic/amlogic-gxl.h b/drivers/crypto/amlogic/amlogic-gxl.h index 1ab3462dea42..f3455babb52a 100644 --- a/drivers/crypto/amlogic/amlogic-gxl.h +++ b/drivers/crypto/amlogic/amlogic-gxl.h @@ -82,10 +82,12 @@ struct meson_flow { * struct meson_pdata - SoC series dependent data. * @reg_descs: offset to descriptors register * @reg_status: offset to status register + * @setup_desc_cnt: number of setup descriptor to configure. */ struct meson_pdata { u32 descs_reg; u32 status_reg; + u32 setup_desc_cnt; }; /* -- 2.34.1