Received-SPF: softfail (google.com: domain of transitioning linux-crypto-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Ard Biesheuvel <ardb@kernel.org>
To:     linux-crypto@vger.kernel.org
Cc:     herbert@gondor.apana.org.au, keescook@chromium.org,
        Ard Biesheuvel <ardb@kernel.org>
Subject: [PATCH 6/8] crypto: aead - avoid DMA alignment for request structures unless needed
Date:   Wed,  6 Apr 2022 16:27:13 +0200
Message-Id: <20220406142715.2270256-7-ardb@kernel.org>
In-Reply-To: <20220406142715.2270256-1-ardb@kernel.org>
References: <20220406142715.2270256-1-ardb@kernel.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

AEAD request structures are currently aligned to minimal DMA alignment
for the arcitecture, which defaults to 128 bytes on arm64. This is
excessive, and rarely needed, i.e., only when doing non-coherent inbound
DMA on the contents of the request context buffer. So let's relax this
requirement, and only use this alignment if the
CRYPTO_ALG_NEED_DMA_ALIGNMENT flag is set by the implementation.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 include/crypto/aead.h          |  2 +-
 include/crypto/internal/aead.h | 13 +++++++++++--
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/include/crypto/aead.h b/include/crypto/aead.h
index 14db3bee0519..3a09f540c38a 100644
--- a/include/crypto/aead.h
+++ b/include/crypto/aead.h
@@ -97,7 +97,7 @@ struct aead_request {
 	struct scatterlist *src;
 	struct scatterlist *dst;
 
-	void *__ctx[] CRYPTO_MINALIGN_ATTR;
+	void *__ctx[] CRYPTO_REQ_MINALIGN_ATTR;
 };
 
 /**
diff --git a/include/crypto/internal/aead.h b/include/crypto/internal/aead.h
index 27b7b0224ea6..b30bc33a0c75 100644
--- a/include/crypto/internal/aead.h
+++ b/include/crypto/internal/aead.h
@@ -62,7 +62,8 @@ static inline void *aead_instance_ctx(struct aead_instance *inst)
 
 static inline void *aead_request_ctx(struct aead_request *req)
 {
-	return req->__ctx;
+	return PTR_ALIGN(&req->__ctx,
+			 crypto_tfm_alg_req_alignmask(req->base.tfm) + 1);
 }
 
 static inline void aead_request_complete(struct aead_request *req, int err)
@@ -105,7 +106,15 @@ static inline struct crypto_aead *crypto_spawn_aead(
 static inline void crypto_aead_set_reqsize(struct crypto_aead *aead,
 					   unsigned int reqsize)
 {
-	aead->reqsize = reqsize;
+	unsigned int align = crypto_tfm_alg_req_alignmask(&aead->base) + 1;
+
+	/*
+	 * The request structure itself is only aligned to CRYPTO_REQ_MINALIGN,
+	 * so we need to add some headroom, allowing us to return a suitably
+	 * aligned context buffer pointer. We also need to round up the size so
+	 * we don't end up sharing a cacheline at the end of the buffer.
+	 */
+	aead->reqsize = ALIGN(reqsize, align) + align - CRYPTO_REQ_MINALIGN;
 }
 
 static inline void aead_init_queue(struct aead_queue *queue,
-- 
2.30.2