Received: by 2002:a05:7412:b10a:b0:f3:1519:9f41 with SMTP id az10csp2759758rdb; Mon, 4 Dec 2023 06:56:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IHyHB+bCR6IZsp/srpZ/2upjUxESwK5UBVPsodVzT1GluSMkPMg5W4cyZEUalCSu6A0/VJs X-Received: by 2002:a05:6a20:a10d:b0:18c:eae:bd86 with SMTP id q13-20020a056a20a10d00b0018c0eaebd86mr31463851pzk.32.1701701787709; Mon, 04 Dec 2023 06:56:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701701787; cv=none; d=google.com; s=arc-20160816; b=pUNQnLvKk0TOHpXVmqnG71intGif1vSqlUNlM46S0ExjIK9EatI1JQ8ugT77ckY7km XR4R9mWZ27zUyDoF68qsn/o3O+2j2pIvjP3PxORy/FWGjwLSRBwgeDQ9DL7QEo1WIVS0 ADjxjypiTdYHkL3/B6HSTDpo/tYEwztsoThSo1sWUPzfrwXyqfQLopNrrU7U8egy2a7U qya8qizVFNLDJTjso994XFyaUfLPlsTKbsY9Bp3F8bQxtCSwd8ZnaRzt3J2EoKuwjFgb cxlhwCRcn5H3I40uDZy6/9JzL3kOmFMJ4VwsHy5G1KbSsF3jlhHvOPKxChmB/toZU5j7 RaDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:cc:to:from:subject:dkim-signature; bh=NN03Wz2SdyBrOnlGSaviqMkTPuXqWYSY141oMdaZlUg=; fh=WfGVbliYpqgzl5GprTVUBQR8rnRPHqjfX8LKlqDYYGk=; b=U6FsHuuhpbqCDkPPybghqrCZl5yTZ+Ou89gDZ2SJ5ggPt3Iwff+34GcK0RbvvmFs61 +wsFq8++VLXrwi2zHY/+Ssf+5IR+JdlUulnXcxBbRM6y6e7xD8MQYqq0mlFilueo5Gwv uxcYfyxB3ZSHhojjjRu6CnEXc7fRVuc9gInTf74NSkajFD3wfz4NdMc/rLq7RD9YNyWV GpcixNTm4b/iCsoiFtQtcEzUZ++4LQuRcLRSNUGiW6I3BaTbbntntyOiGGDMXamJX11B WLeuPt3QIk/xdST360vV4jHhEIdQfaFarHQjoGw1m5Rz8O6weNmhYSuWgeFarCnEe/yZ vEQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="gXvI/0pQ"; spf=pass (google.com: domain of linux-nfs+bounces-276-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-nfs+bounces-276-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id g71-20020a636b4a000000b005bd39be60b3si7846193pgc.390.2023.12.04.06.56.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:56:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs+bounces-276-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="gXvI/0pQ"; spf=pass (google.com: domain of linux-nfs+bounces-276-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-nfs+bounces-276-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 5DC312811B7 for ; Mon, 4 Dec 2023 14:56:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CB680286B3; Mon, 4 Dec 2023 14:56:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gXvI/0pQ" X-Original-To: linux-nfs@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AADE422EEE; Mon, 4 Dec 2023 14:56:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB3E9C433C7; Mon, 4 Dec 2023 14:56:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701701786; bh=hOWTxUDjA+zelWg+rXMPFmZj3aTXBB7jCFyHmHc2LFQ=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=gXvI/0pQRwvNR8oq2oAeYR0JWT+U213ehTldb0foyrsW6+vANphRKJmtJVi5mNvig GVCXrxOEnDb7YiQrXmHZTNDj/PrfEsKKBgGKEc7Oalgz7/xYJphZ3+Ob1g3hj1L1lg 8FAonkJjSD5qnpYmoVZYZpq9cNOaGpjzygVXk2rmb7laM43uaaqkzUXnD6J7/NaggX ywlSysJJ1vbtCTWGhNh8s5bNKoYoOQm10Me14hYP6B0auaKb0gknqT5rk4huudAVSC GRWDI7uJ6okDFTa80O0vnEaLsJ5KQvdIiTzXWK62tgGJ3XpksPTias/P0qf3AIuwji /f5lrVNs5G35Q== Subject: [PATCH v1 01/21] svcrdma: Reduce size of struct svc_rdma_rw_ctxt From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Cc: tom@talpey.com Date: Mon, 04 Dec 2023 09:56:24 -0500 Message-ID: <170170178494.54779.7160706939335739704.stgit@bazille.1015granger.net> In-Reply-To: <170170144201.54779.9877683240030806819.stgit@bazille.1015granger.net> References: <170170144201.54779.9877683240030806819.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Chuck Lever SG_CHUNK_SIZE is 128, making struct svc_rdma_rw_ctxt + the first SGL array more than 4200 bytes in length, pushing the memory allocation well into order 1. Even so, the RDMA rw core doesn't seem to use more than max_send_sge entries in that array (typically 32 or less), so that is all wasted space. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/svc_rdma_rw.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/net/sunrpc/xprtrdma/svc_rdma_rw.c b/net/sunrpc/xprtrdma/svc_rdma_rw.c index c06676714417..69010ab7f0c3 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_rw.c +++ b/net/sunrpc/xprtrdma/svc_rdma_rw.c @@ -39,6 +39,7 @@ struct svc_rdma_rw_ctxt { struct list_head rw_list; struct rdma_rw_ctx rw_ctx; unsigned int rw_nents; + unsigned int rw_first_sgl_nents; struct sg_table rw_sg_table; struct scatterlist rw_first_sgl[]; }; @@ -53,6 +54,8 @@ svc_rdma_next_ctxt(struct list_head *list) static struct svc_rdma_rw_ctxt * svc_rdma_get_rw_ctxt(struct svcxprt_rdma *rdma, unsigned int sges) { + struct ib_device *dev = rdma->sc_cm_id->device; + unsigned int first_sgl_nents = dev->attrs.max_send_sge; struct svc_rdma_rw_ctxt *ctxt; struct llist_node *node; @@ -62,18 +65,19 @@ svc_rdma_get_rw_ctxt(struct svcxprt_rdma *rdma, unsigned int sges) if (node) { ctxt = llist_entry(node, struct svc_rdma_rw_ctxt, rw_node); } else { - ctxt = kmalloc_node(struct_size(ctxt, rw_first_sgl, SG_CHUNK_SIZE), - GFP_KERNEL, ibdev_to_node(rdma->sc_cm_id->device)); + ctxt = kmalloc_node(struct_size(ctxt, rw_first_sgl, first_sgl_nents), + GFP_KERNEL, ibdev_to_node(dev)); if (!ctxt) goto out_noctx; INIT_LIST_HEAD(&ctxt->rw_list); + ctxt->rw_first_sgl_nents = first_sgl_nents; } ctxt->rw_sg_table.sgl = ctxt->rw_first_sgl; if (sg_alloc_table_chained(&ctxt->rw_sg_table, sges, ctxt->rw_sg_table.sgl, - SG_CHUNK_SIZE)) + first_sgl_nents)) goto out_free; return ctxt; @@ -87,7 +91,7 @@ svc_rdma_get_rw_ctxt(struct svcxprt_rdma *rdma, unsigned int sges) static void __svc_rdma_put_rw_ctxt(struct svc_rdma_rw_ctxt *ctxt, struct llist_head *list) { - sg_free_table_chained(&ctxt->rw_sg_table, SG_CHUNK_SIZE); + sg_free_table_chained(&ctxt->rw_sg_table, ctxt->rw_first_sgl_nents); llist_add(&ctxt->rw_node, list); }