Received: by 2002:a05:7412:8d10:b0:f3:1519:9f41 with SMTP id bj16csp4043498rdb; Mon, 11 Dec 2023 07:24:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IHKjqYcYVdzGmK7ctHUco8dnnAUM5CJ1BvgpeHNoJBd+9tBClDuipHbclfqNJC11dAmZ/cp X-Received: by 2002:a17:90a:590c:b0:286:6cc1:2cd6 with SMTP id k12-20020a17090a590c00b002866cc12cd6mr1782549pji.96.1702308249775; Mon, 11 Dec 2023 07:24:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702308249; cv=none; d=google.com; s=arc-20160816; b=wOasxhKOZh7olYdn9a4stwW743OywQcx0zjvLWEiKkolhjS08l4BiK1XqRWOtbYmsP tJvgswQfYwqNFlnBP8BRTgE+xi7fGa1/VrWMpCAja80elc8L+pM+jUgvS6jMyo3eQEna e4OntFJtOA+KReQ1xGq61QqvyKN8oh8e248uX+PfEKiFKI0jBtdR9Ot2o4OBsbOiJ0xz LXoq/NmEq7GLoPyDeOLRZ+FOqDVxZcuVq+8gn3yOvbcsrnQIEiUNwaneKiFP48OLUb/I Tg1CEVOZizSJkHe/QiP6UeZtU+TsoWEoTFPNo/32t+PC5/xS53IbdFdkcqWZH2lHrSnq +o8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:user-agent:message-id:date:cc:to :from:subject:dkim-signature; bh=2SgH9xMDHD1eNID+CVg8tLah+sBBmu1NmwuzSaT7rIQ=; fh=WfGVbliYpqgzl5GprTVUBQR8rnRPHqjfX8LKlqDYYGk=; b=xDkonA9anIHUVpkN2euTmP6VCdjMHvBnDGF3EhO1Bf15PK25hzjJ+e3aTi6/O7T1wc 3DgEQcStISD0imn6bJnB0r1j7DUYz3QXL+rfenyIb+AvgSTKoPV1LbAL9psIiHBS2XJM GarpLIQPmG/8XgqsfIlzIkakI+cnPoDmODVy6DDQfJfxo3FB0ykVawmDiP/QqDJSy4sz uFAIdrni16uiy3XxlozA/bCpwvlkQaw1Y/IRriHhZlCONjpgcfJlYgLIke35XEUHDHor fWf2qKrfNwpUihCQ66bKVUjrBnVoI1tzJEmg9G3H+1lHrNG93pFg/VQ0I9q+kJ14qJ10 ZoTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=vIUvV22K; spf=pass (google.com: domain of linux-nfs+bounces-482-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-482-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id o8-20020a17090ac70800b0028ab0c61b77si123769pjt.112.2023.12.11.07.24.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:24:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-nfs+bounces-482-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=vIUvV22K; spf=pass (google.com: domain of linux-nfs+bounces-482-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-nfs+bounces-482-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 950CFB20FDC for ; Mon, 11 Dec 2023 15:24:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 19FF84AF78; Mon, 11 Dec 2023 15:24:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vIUvV22K" X-Original-To: linux-nfs@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E76303B184; Mon, 11 Dec 2023 15:24:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E249C433C7; Mon, 11 Dec 2023 15:24:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702308243; bh=l/lZQDrBjfmqVGnHnbpv7lJzJFSf5AE1xC6CokFuDL0=; h=Subject:From:To:Cc:Date:From; b=vIUvV22KEG/OgyxxdHcv8QyBowKyvQW4kgwB9s6o0gVQU7kB3FVMJCvMC4HWQGRpH zN3UVsK78BH4T1gDmQFp3npYM093XI6Er/ykGGnxXEOEDS4iYD3U+xL+ROj1fkil2m GZfsJh6yJVsGOG3gMcBtCen2R0ZcJ92p34j3hWvLCRuS5XV4SHKx6tnj3tkrnhsB0i i5cWU7bmGX0LeEsy3+1hu/oi17iKmLXpXbrPOdDC/QpmBUWPxyy8XCEw8LpJZYzyW3 OMx7vDS/f1YCW5ivpXQUlIejOQ5ZRnmHNRxtbLyCXgbgUrw9+F8JlTB1KnMiObTjNf KgrimwRAQyEvA== Subject: [PATCH v1 0/8] More svcrdma improvements From: Chuck Lever To: linux-nfs@vger.kernel.org, linux-rdma@vger.kernel.org Cc: tom@talpey.com Date: Mon, 11 Dec 2023 10:24:02 -0500 Message-ID: <170230788373.90242.9421368360904462120.stgit@bazille.1015granger.net> User-Agent: StGit/1.5 Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Here are some clean-ups and minor optimizations for svcrdma, in addition to two patches that reduce the likelihood of connection loss on heavy workloads (the final two patches in the series). svcrdma appears to under-allocate Send Queue resources, which will cause NFSD to drop the connection occasionally. This results in a burp in the Send pipeline and a loss of throughput. The series (including these patches) is in the svcrdma-next branch of: https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git --- Chuck Lever (8): svcrdma: De-duplicate completion ID initialization helpers svcrdma: Optimize svc_rdma_cc_init() svcrdma: Remove pointer addresses shown in dprintk() svcrdma: Remove queue-shortening warnings svcrdma: Clean up comment in svc_rdma_accept() svcrdma: Reserve an extra WQE for ib_drain_rq() svcrdma: Use all allocated Send Queue entries svcrdma: Increase the per-transport rw_ctx count include/linux/sunrpc/svc_rdma.h | 24 ++++++++ net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 9 +-- net/sunrpc/xprtrdma/svc_rdma_rw.c | 16 ++--- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 9 +-- net/sunrpc/xprtrdma/svc_rdma_transport.c | 75 ++++++++++++++---------- 5 files changed, 74 insertions(+), 59 deletions(-) -- Chuck Lever