Received: by 2002:a05:7412:8d10:b0:f3:1519:9f41 with SMTP id bj16csp4044027rdb; Mon, 11 Dec 2023 07:24:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IGT/udN581VWPhX66V/TC+HqlqSgRmBV5F8jtuAf5ad9wNrcBeHo8GcDQxZNvs65RaZE4Lv X-Received: by 2002:a50:f699:0:b0:54c:aee4:8b95 with SMTP id d25-20020a50f699000000b0054caee48b95mr4223515edn.35.1702308297372; Mon, 11 Dec 2023 07:24:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702308297; cv=none; d=google.com; s=arc-20160816; b=JxbyJx+HI8OtCQ0i/8j+L9ZLIZRQVZYuioLJHhkDPheQJ5pS5VPzsZjSx7ZfPrYF4F y8G/ndbluYRbcDFNTf05U0Doww9I/t+lfQsz+ZmFSGWfPo8vN1aOu075T/c6L5F+axQy z49I342nH90Pwe1zTmyEuYEQro6YK65IYevlQcYRO5+um1t6Ye3V/ZE/PMSQ3FNN84HF M+pDkvH7E4CGndMfgacZ1UesUE2i2JAxfetmiSktdct12HVkp/OZpCBB1Ajtfk86ZOXv RmmUu41/4MCZOFAuHGdqmMtMThIn/FTDpvkLXqmF/j3uirGQtTgPnEZB7Ilnx63QbQ9j llHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:references:in-reply-to :message-id:date:cc:to:from:subject:dkim-signature; bh=Vslrqt81uEW6YNJiq1QP4hE91728Me0TkYKgglbperE=; fh=WfGVbliYpqgzl5GprTVUBQR8rnRPHqjfX8LKlqDYYGk=; b=O7sG2L3DYhnI3oyfcSq+EO2CxkHGqeVvqgWc/kcFzxXwPehJPzjM3elo6ewvW+r9Kh A1LullJfEYtGNbYdDvY9JqJAl0qivfM9e0aLdhHyg3LXm2dORfwCLEaiAEGkx4cCNqGZ jKZLHpWHAQDqGdM411Z/WPTi3vpAZeuGK+W2D8PCPlIsvhuU8AiPVdVfe+qaH+s5x8fK YG53BBx9Tyhe844m9SYNcW3qjUQXrEI+qDMqPu7f8/993hqOUn26R07tUPUoqCFzoNjt Q5dzuMAYOOLr+MTg+73PBHImPnBqJfKDYzN9kQBcWF6o/8pT3tsGwolW6igKCaVufiLR lEBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LMVQ+r4P; spf=pass (google.com: domain of linux-nfs+bounces-490-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-nfs+bounces-490-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id h14-20020a05640250ce00b0054b126c8d00si3742871edb.59.2023.12.11.07.24.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:24:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs+bounces-490-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LMVQ+r4P; spf=pass (google.com: domain of linux-nfs+bounces-490-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-nfs+bounces-490-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 181A91F217D5 for ; Mon, 11 Dec 2023 15:24:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 898014AF87; Mon, 11 Dec 2023 15:24:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LMVQ+r4P" X-Original-To: linux-nfs@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 685BD3B184; Mon, 11 Dec 2023 15:24:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A4BFCC433C7; Mon, 11 Dec 2023 15:24:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702308294; bh=jdchVgDmsZtECwiFg6aF5OPF0Qz+GiqmuHEdpGSnXtE=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=LMVQ+r4PcRjJfACop+BPqRzcPR4AiDY7LSQeZNUzqHVZs0sFie/08jBpstRqYFJPz rWYLIByCaMe6gUKdjkjkmBW4mFUngugfrVMf+ckcTcyzzojtikcVcFgWSYew4tZQd7 LbK9zq/fMoXi8vvC89qQ93w5IyQ9vK4VXmgec9Td8FBU5EDdtjdZ+fNt5rbFtQpWn9 LTWLH+xqQSC5+2WgJJzH8pDvGvS8ve+gNTVb0mk/K0yz2KL5IuRlbIJ3ER6Pk8Lk6C y92m3icsxa3m57RHuELpCvRiRfriTdE7n67i/zg8O9Z/qPcl9rfYtyqvDbmRtRwSvM sfOiDgNGKSuew== Subject: [PATCH v1 8/8] svcrdma: Increase the per-transport rw_ctx count From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Cc: tom@talpey.com Date: Mon, 11 Dec 2023 10:24:53 -0500 Message-ID: <170230829373.90242.11114271955743181616.stgit@bazille.1015granger.net> In-Reply-To: <170230788373.90242.9421368360904462120.stgit@bazille.1015granger.net> References: <170230788373.90242.9421368360904462120.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Chuck Lever rdma_rw_mr_factor() returns the smallest number of MRs needed to move a particular number of pages. svcrdma currently asks for the number of MRs needed to move RPCSVC_MAXPAGES (a little over one megabyte), as that is the number of pages in the largest r/wsize the server supports. This call assumes that the client's NIC can bundle a full one megabyte payload in a single rdma_segment. In fact, most NICs cannot handle a full megabyte with a single rkey / rdma_segment. Clients will typically split even a single Read chunk into many segments. The server needs one MR to read each rdma_segment in a Read chunk, and thus each one needs an rw_ctx. svcrdma has been vastly underestimating the number of rw_ctxs needed to handle 64 RPC requests with large Read chunks using small rdma_segments. Unfortunately there doesn't seem to be a good way to estimate this number without knowing the client NIC's capabilities. Even then, the client RPC/RDMA implementation is still free to split a chunk into smaller segments (for example, it might be using physical registration, which needs an rdma_segment per page). The best we can do for now is choose a number that will guarantee forward progress in the worst case (one page per segment). At some later point, we could add some mechanisms to make this much less of a problem: - Add a core API to add more rw_ctxs to an already-established QP - svcrdma could treat rw_ctx exhaustion as a temporary error and try again - Limit the number of Reads in flight Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/svc_rdma_transport.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index 790841864153..0ceb2817ca4d 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -422,8 +422,12 @@ static struct svc_xprt *svc_rdma_accept(struct svc_xprt *xprt) newxprt->sc_max_bc_requests = 2; } - ctxts = rdma_rw_mr_factor(dev, newxprt->sc_port_num, RPCSVC_MAXPAGES); - ctxts *= newxprt->sc_max_requests; + /* Arbitrarily estimate the number of rw_ctxs needed for + * this transport. This is enough rw_ctxs to make forward + * progress even if the client is using one rkey per page + * in each Read chunk. + */ + ctxts = 3 * RPCSVC_MAXPAGES; sq_depth = newxprt->sc_max_requests + newxprt->sc_max_bc_requests + 1; if (sq_depth > dev->attrs.max_qp_wr)