Received: by 2002:a05:7412:40d:b0:e2:908c:2ebd with SMTP id 13csp517478rdf; Tue, 21 Nov 2023 08:40:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IHZi14QtA3/ERR7wBBar/y5qD511mH9+dg7umHvqjwqAtCzEM668Y3U2qsmGP3Oz5qR4Ra/ X-Received: by 2002:a05:6a00:134f:b0:692:b6e8:ce88 with SMTP id k15-20020a056a00134f00b00692b6e8ce88mr10978662pfu.17.1700584811230; Tue, 21 Nov 2023 08:40:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700584811; cv=none; d=google.com; s=arc-20160816; b=QaIvTwb/ZHFZJNSXeOKQ6Z0GPN58BZGQZreNp5ZTvSJa3EAdbSN6GLBk6qjlqDEgem rdK1S1QHyUO6RSVnaa5U2M/t7FWzdLO7oQkOTcjvw/4ZHK5HYCMmU4+yKLkd54fsYm+V 8BQ9YsGvEErVjbX919xkDaviLV7KG6DzHOdEBaC8EKA2mrpIgpzUJHu+q15EDaOiPO2h z9g00mBRXPegnob6zq2kweDqlMZhhM/BPpkmp5seTEcXdWQOpX1+jpzH9Tuke120UeXe pT5r6R/apjPiMqvvZIRSEtyEv73ZpGGBbEgQICE06mpvyB6WOQI4Uz2ECyEQiDaeDVlk o2+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:message-id:date:cc:to :from:subject:dkim-signature; bh=6DzdU1fhZwXNoXvQgmuYb3z10watINtUXQALhCeCZoo=; fh=ckZOnuavOce67CxFsxbWAkDFDgnVrD8nvvkVrKxmLnw=; b=oYc87EqwUb8NVwRk3yHoNQFaeuQfHQk08Mq3WHQyya8XEc8W8kfJj+W/LSMIhB5KVO eJOWNq6+6PtesQ+xH2G8VqnQXr0UQ3+rW4mDTL4IgcoGpUIny4ijDh8ZMzAnhOnl0maG /p/vqtnUw/5ehGJOLXkte1jZw7V39PloDr3iuwqn04qqxtiEzpHTTxuDE7yyysQRBClt w/f70siE0UvcJUA1LLSdUW2UbdJyGOMgwKHEU6Ugbwmhub8IYWBHfHPj4nrwztTSEfxc UhJ8p8xPqwOyF3SB1i7ZhFakd+49CHu64LP06jebmckKoGHiyBhgUunxuxO5S43rha3u guzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AgukbjsF; spf=pass (google.com: domain of linux-nfs+bounces-11-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-11-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id 15-20020a63104f000000b005b909e678dfsi10084877pgq.450.2023.11.21.08.40.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 08:40:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs+bounces-11-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AgukbjsF; spf=pass (google.com: domain of linux-nfs+bounces-11-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-11-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 8DDF3283077 for ; Tue, 21 Nov 2023 16:40:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1B59B18E02; Tue, 21 Nov 2023 16:40:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AgukbjsF" X-Original-To: linux-nfs@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F25E95674C; Tue, 21 Nov 2023 16:40:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F4BAC433C8; Tue, 21 Nov 2023 16:40:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700584808; bh=d5n9vNStvOceqXEbN7a2X4urtfXmU7o7L1d/cHedV5g=; h=Subject:From:To:Cc:Date:From; b=AgukbjsFECOrP1eXArlfv4HzgZG4q0FiJ8dsCy4I5y+ulWyh+KOA7ZdVD/Egut4EW er63tibTe3YMvQZkRDpbp5PDEY6tCWCgz7BpV5WIdNOqs17/OzfM93YJ8E3wPXZ0xn P72dmYmWOTcJvLoJgMnN34fi+lVceeB9oj45iC75NpejoBvRIVtS0BygKu0wuTHtsz rnkpJ3gR7bLdbKF1EHugjiWdAwp9mII6bdtECvJWJIhiFnO0V1KWM9V4+H+eQJuRp1 +UvET2WCvY8BWz7MoQgO8CRRtYgsFKzuh8DlBS5oDZNDOHrjXdA2HhElBqE0ZL0ErR qDUtViGt6iJ4w== Subject: [PATCH v2 0/6] Was: "Switch NFS server CQs to use soft IRQ" From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Cc: Chuck Lever , tom@talpey.com Date: Tue, 21 Nov 2023 11:40:07 -0500 Message-ID: <170058462629.4504.17663192195815644972.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit After testing v1 "Switch NFS server CQs to use soft IRQ", I found that using soft IRQ for completion handling actually reduced throughput on a 70/30 8KB read/write test. So I've tossed those patches out of this series. Also, Jason reminded me that DMA unmapping is pretty costly, so that has been moved out of Send completion handling on the NFS server. --- Chuck Lever (6): svcrdma: Eliminate allocation of recv_ctxt objects in backchannel svcrdma: Pre-allocate svc_rdma_recv_ctxt objects svcrdma: Add a utility workqueue to svcrdma svcrdma: Add an async version of svc_rdma_send_ctxt_put() svcrdma: Add an async version of svc_rdma_write_info_free() svcrdma: Clean up locking include/linux/sunrpc/svc_rdma.h | 6 +- net/sunrpc/xprtrdma/svc_rdma.c | 32 +++++++--- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 11 ++-- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 32 ++++++---- net/sunrpc/xprtrdma/svc_rdma_rw.c | 12 +++- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 69 ++++++++++++++-------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 7 files changed, 110 insertions(+), 53 deletions(-) -- Chuck Lever