Received: by 2002:a05:7412:8598:b0:f9:33c2:5753 with SMTP id n24csp144375rdh; Mon, 18 Dec 2023 14:31:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IEzgSs+yjIpWswPGi9uWZHFSQvj0BwNiT/Qje2wZ2JXrvGF3ri+RY1/gAC9y0MTeGrpZXkR X-Received: by 2002:a17:903:44d:b0:1d3:ae2c:3ed8 with SMTP id iw13-20020a170903044d00b001d3ae2c3ed8mr1383093plb.29.1702938705949; Mon, 18 Dec 2023 14:31:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702938705; cv=none; d=google.com; s=arc-20160816; b=U/+AH8bRZq+3+2LTQSnnyVSY+Eej9eKu6Mpw8nx+r7YCgV2+1o1P6AmOWM8V7rfGkV FMsUPHSmz1PyWpAaEuuFZAvoHKKAKbDCPANt70bQGyQ8s55SHLuA6RpJ5RXVvZ+2bzcz pPXN9jtFwloSG3sWlnHZ8owqXEjHJGP+kNpy+sxTmwBlRcMb9nKLPyQq7alfrGcSaJaX UsF5+5fS5lEAE1wHGrOpSQNuOm0WvovFYTOM7w0BtuE+QKGQAe1mQQ6sSOMGN/zeDzOb G1wiSuCQN8XU4E5Mxq32X5XqX1waF0kk3fhJ8CyLMxpg5wYo0nH8nUdb0lKWjRi5JXLo 5UJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:message-id:date:cc:to :from:subject:dkim-signature; bh=FIvWmiCqVFoU8qr8dgT95GWGthXiTwDYmGvxD0r6DNk=; fh=WfGVbliYpqgzl5GprTVUBQR8rnRPHqjfX8LKlqDYYGk=; b=ga9vq6kLZfpBoJIqnIIeE+LEr5mqlcH/c0Zo+rXcc5uXkFbOBpneW/XwmOjGWMwapg 0DtkGqmIuKG7C/a3q+i1eLlGIpijGg1xsKOAe3Zgh8W7vHcqjMzHlwQZTh1PAdY9sOrS DWCl09/Bydhpk40bS5HPXeWw1Cy7RjovEZEicHnl1KAOmTnc2Xtmr5ZpqyLmqi8SzzuX u10SwmvUCU0Z/RCPcP5eABbpXHdYFit3LarV+IzhXZ85OTbCRamcNoxYEoa9SJ4MfDYj +x2Xq1i0VJzJb79uWqjXwTswdyn9YEycc3FebEPcsFP8WHuKmqjE/NMy5B643vbSZjOj aY/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=aNVKXfmC; spf=pass (google.com: domain of linux-nfs+bounces-699-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-699-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id j9-20020a17090276c900b001cfa17ec332si18417480plt.616.2023.12.18.14.31.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 14:31:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs+bounces-699-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=aNVKXfmC; spf=pass (google.com: domain of linux-nfs+bounces-699-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-699-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 6B28028516E for ; Mon, 18 Dec 2023 22:31:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6D5D4129EFB; Mon, 18 Dec 2023 22:31:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aNVKXfmC" X-Original-To: linux-nfs@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EF9D1DFC1; Mon, 18 Dec 2023 22:31:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 98CDBC433C7; Mon, 18 Dec 2023 22:31:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702938702; bh=3LgK3enpU/ETsSDkGpEPMOM8mfdkEA5i46IzdazB8QY=; h=Subject:From:To:Cc:Date:From; b=aNVKXfmCX1jkLQpMlcf1I2pJarQbLxMSINnobFbboKWxn0uykV7rox99tWkCXx2j3 /52jYAan1xgYdYZO57JGXYSCNrXY9ACT5IOODN+9IIdRIl0C97SOgqaUqY+SY7dstO 5fdH+Gv27KIe3IJnvMuCS97O2jXsK1ljR+E46rbY12Y5LhgMBdP9iUOqzXiH04SugV ZnD4oxCuRPyCZsanNMGs4ahzBN7lRSw5lmKo3K9HZRHTbNlMrUaQ1hd4wr3EHopmya hAPvELy6OpBFwx3yUlcBSWjFjUKCMcG1dalnKkXSPLpHE4/rWB1DL7q2Tbba3IOKVF ofwpXmSZviDAg== Subject: [PATCH v1 0/4] svcrdma: Go back to multi-staged RDMA Reads From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Cc: tom@talpey.com Date: Mon, 18 Dec 2023 17:31:41 -0500 Message-ID: <170293795877.4604.12721267378032407419.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit In 2020, commit 7d81ee8722d6 ("svcrdma: Single-stage RDMA Read") changed svcrdma's Read chunk handler to wait, in nfsd thread context, for the completion of RDMA Reads from the client. The thought was that fewer context switches should make for more efficient Read chunk processing, since RDMA Read completion is typically very fast. What I neglected to observe at the time is that if a client should stop responding to RDMA Read requests or the RDMA transport should fail to convey them (say, due to congestion), the herd of waiting nfsd threads could result in a denial-of-service. This is why the original svcrdma design was multi-staged: the server schedules the RDMA Reads and then the nfsd thread is released for other work; then the Read completions wake up another nfsd thread to finish assembling the incoming RPC. This series of patches reverts commit 7d81ee8722d6 ("svcrdma: Single-stage RDMA Read") by replacing the current single-stage Read mechanism with a reimplementation of the original multi-stage design. Throughput and latency tests show a slight improvement with the new handler. --- Chuck Lever (4): svcrdma: Add back svc_rdma_recv_ctxt::rc_pages svcrdma: Add back svcxprt_rdma::sc_read_complete_q svcrdma: Copy construction of svc_rqst::rq_arg to rdma_read_complete() svcrdma: Implement multi-stage Read completion again include/linux/sunrpc/svc_rdma.h | 11 +- include/trace/events/rpcrdma.h | 1 + net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 167 +++++++++++++++++++++-- net/sunrpc/xprtrdma/svc_rdma_rw.c | 149 +++++++------------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 5 files changed, 217 insertions(+), 112 deletions(-) -- Chuck Lever