Received-SPF: pass (google.com: domain of linux-nfs+bounces-2211-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Tue, 5 Mar 2024 09:02:25 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: Trond Myklebust <trondmy@hammerspace.com>
Cc: "aglo@umich.edu" <aglo@umich.edu>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH] NFS: enable nconnect for RDMA
Message-ID: <Zecl8WT1MAXQq0N0@manet.1015granger.net>
References: <20240228213523.35819-1-trondmy@kernel.org>
 <ZeTC2h59TXUTuCh_@manet.1015granger.net>
 <CAN-5tyH2dkLbt9dMau46ebrU=PfvLgo2=mr8u3_C1gUnAGM_ow@mail.gmail.com>
 <51189037-DD10-4806-8596-D4374067FDFB@oracle.com>
 <c88bda54d8eb5a78a94b426e23cc8048d804927c.camel@hammerspace.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <c88bda54d8eb5a78a94b426e23cc8048d804927c.camel@hammerspace.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?b2dnNVo5eU5OdnYwYm1JWklBZ2NGRGkxakJYYkdqazgyOHhMckF6em05Q2Jl?=
 =?utf-8?B?T0s0MEFYRnc4NEFZdTdXYWhUMDdSMjJuNGNEWmozWGJOZUU2eWJOY0NVVUty?=
 =?utf-8?B?dDZKdmd0VTRqSktZS2xZNnZYRlFLb2pnOVpLenFZdjVRNjlLb29QaC9YWDZo?=
 =?utf-8?B?OVNtMktPT3F6NkxRS2drYnlFUGlqRFFxOXQxRzhtWk9qdlBWRDhBc1o1bkYy?=
 =?utf-8?B?Z0ROcG5mNjk5MFFNclFLbGY5N2REYkJGYmxmOXRESWhKQnh0U1VHUGtvOThT?=
 =?utf-8?B?K1UyajcvdUxsNG56VkJLdnpZb3ZUWVZOd3A0TWJVdytPSGFLYTBsOFhIQW5U?=
 =?utf-8?B?OXVzUmhVYTJKbmlpOEI4VHVaL2lxUkl1R2JZQkRVZWlMZnRHeDgrbDZaSS80?=
 =?utf-8?B?MW1DT2kwQWh1bUhGS2YwOWZyKzJJNEx3aXI4WE9WNVpVSjhOK1ovekpwSVB0?=
 =?utf-8?B?Ym4xaEpqeXFDNFhxQyt5dVpRdjNtSGl4YnQrNGtRbHdRb3VLUkFRK2l5YUpB?=
 =?utf-8?B?VXpTcWlsd1ZVZmMwYVAvWXRqaVpocy9uOEkzUFJnVy9qdHZKVGdmQWFjZWFr?=
 =?utf-8?B?U3ZtZ09uSTZsMjE3b0VrTk43VFhlcTRwY1NmTzVPVXdFejlud0hONVZKd0ZZ?=
 =?utf-8?B?VmFwVnpSNC9CS3pZSTcvZ1hHNkw1aG1UYmpEbk0vbmRJZ0tyN252Y08zOTZ4?=
 =?utf-8?B?OHJiM29WWFptVitFL3NoOGlGK0IySnNBVDRrdk5mZTNqV0NwVUxlcEJUajRt?=
 =?utf-8?B?anM3b3RTTUpwL1NsSDZ6Rm4rLzl0a1BVSm1XNmk2ZG1iZDJ5OHNoOEVFQTQ4?=
 =?utf-8?B?dlRsNkdMaENJU1p6bDNIc3Bqdm50OXd3U0UybE81aW1WUzVJeGJwRUp3dHFu?=
 =?utf-8?B?SEdzSXdsRlFSczNZRStkU3VCZElpWU1BUjBPbVlFSXpqZGxRZkw3czVWblBM?=
 =?utf-8?B?NjVqQ0RZVzR0akM0SlpodFRXSkh4YlAwb3RhNEl5UkMwbTFmT1J1RWpEK0Za?=
 =?utf-8?B?T1g0M1hudzAxbitrTFpDM3ZjazlCM3BzenQ2MkN4WjZYYXpEa245TUdVNTg2?=
 =?utf-8?B?MjBrQ1Jram5WQ0tFR1JMMzcwcTRycitCT0pJTUZlUTBhMzU1Zm5UTnF2Rzlr?=
 =?utf-8?B?d3lObDM5TjdQOWdHRzFGQkkyWWJCTmZvVEVBS080eGZUeEd3UlkzZExxNWZW?=
 =?utf-8?B?WVFiTHlOdkduK21KZ2xvTDY1L2cwWTBicXlEQUltNHBBYW8wRmQyTWE1dmJi?=
 =?utf-8?B?U2ZQUU1NVGt4d2M0Q0ErcHNWUDZRUVp5MTRwczNEck5Ca3FEQ1lPaCtDelIw?=
 =?utf-8?B?VE85SlBYNjFWSm5uYnBjMHlveHplclBoZkg0aHhScEtzK3VWZ3JCNUdWNHVB?=
 =?utf-8?B?aHZqMlJacWJjZlhMN1JXME80cGVWS0laZ3JZSHo3dEluWnh4K2hENG95L3JQ?=
 =?utf-8?B?ZVc4ZTZJcVoxaG5OY3BQdUw4WWJHY2srb3NFc1gvNFA4NU9pd1J1TkpNME5H?=
 =?utf-8?B?UlhjeFIxSmdGaTZ6WDYzcVJYZGhsM1VYVzB0UzU1NzBWR0c5cDN3Z3E4cXRQ?=
 =?utf-8?B?dEZQOHFRem9JMlNyQ1A5NVVSY0d1U3pBR25tVlpmL1VoRGorczFoWkkwMTBX?=
 =?utf-8?B?bkE3aWkxa1ROTWxtOXFSVlhqTEU3c25pZkJrV2xNb2VHK1NWV1ZhdW9ieC8z?=
 =?utf-8?B?QWhTTVFQWGQ5UlpPeG95QzZ3SGNXdnJKWUxtU25GUDhYT0IzUnVReDFJTWJO?=
 =?utf-8?B?QlRMNWJZUk5yZzF5UjVzUWpQZDNnei8yTG45cVk4VXJ0Z0ZjMlVQVUt2VkpV?=
 =?utf-8?B?TVladmdpVFYwaW9XK1BIWXo0MXpVSWgvVWxSYVFSWlNtQW0xeWJHdnk4cmF0?=
 =?utf-8?B?WmhhZm1xaFk3K3BmNEtnRENLaTk0TGgza0pxeVBMTTdqNmh2aVplZXpTRUhP?=
 =?utf-8?B?RVB5Q2hITnhHWk5Od1EwOGY1RnJ2ak1GWSs0enUyZ1I1TVNkUnVkbjVMMXlT?=
 =?utf-8?B?QkdISExib3p2RVNyNkozVXVPWS9SNmtSb012VVBlRVVTVnZqeExoTHdYRVlZ?=
 =?utf-8?B?Rkx6S3NQRjJCQnowTjlTWmV3VW5ITE1DZDI2NUlYSmM2WE5tSHYydnhoRmww?=
 =?utf-8?B?ZzRONEFBYTdQOENxTkNycG9FWThCSTc1NVMxbWo1M0c1dHMyV1k2OURWYS93?=
 =?utf-8?B?d3c9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	Nq5mavYB7KgTp3WoM6jzM2j06ucAsjuSYEQc51LIGcH4YWgqZwWL8LzhFIjjm5x7/jMQ5nSPz++TthCu/Ow90x8C7aIvc2MVACK/LFOUG+n4j2zP9jIUgvFQNFynWFy3WFRP64OQXfPeuiy+eie5QOrKjiRaMKifftgDIqiK47r3lZyxiNFQnhrS8TaUc7ShGK4NtWsf3dzU3VogWPQLDlQoiyTKjT/GvM50iiPsrEDDWxWOA17uiaDygDQttAIsgoKLYu0P49VG+h/qNO2E2PKvdx0AK68U/XCXhpozQgqRePBeUxwu0kdV5AbmRAXsbG9yCi73saiQtmB6wXVgMyfjnQEQ3C7ISkac1XaqKGC6njA5E8xR0fVU0jwXQAc/iVWVvpRENGarmdfPot0ZtoQWjAIqhz7q9LxXB2nH5mHD4EACYGRURrgvFY3VzCJua5SW36IDj0bIDZp210Wacd6okzO/v/MpYYVCVDVRkVjd+L/IT+o2UsSSWIPWjzf9jyoRlFE1EQRosUsMS4WXDpJeZd4OHv+3k3a6n7qDWktD8Mfhp3VJKJeu5pIFsnQ1OUXspL17bbwR5O0U5ZbofPlH1H+ahZUHgR5k9eh5MFo=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7493d17e-130a-4435-8582-08dc3d1ce483
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2024 14:02:28.3397
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: w29M46JjGqejUVBsLPhiJ8WWLlc/dbJNxhytb82MohZ/nz0I451dVgBFT3Nq60xpNbvXcKISgFLqEIDrJIXtnw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB5051
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-05_11,2024-03-05_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999
 suspectscore=0 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2403050113
X-Proofpoint-ORIG-GUID: L89eLBJs6poi3iw8sO6i12XR_ejmmAVP
X-Proofpoint-GUID: L89eLBJs6poi3iw8sO6i12XR_ejmmAVP

On Mon, Mar 04, 2024 at 11:08:00PM +0000, Trond Myklebust wrote:
> On Mon, 2024-03-04 at 19:32 +0000, Chuck Lever III wrote:
> > 
> > 
> > > On Mar 4, 2024, at 2:01 PM, Olga Kornievskaia <aglo@umich.edu>
> > > wrote:
> > > 
> > > On Sun, Mar 3, 2024 at 1:35 PM Chuck Lever <chuck.lever@oracle.com>
> > > wrote:
> > > > 
> > > > On Wed, Feb 28, 2024 at 04:35:23PM -0500,
> > > > trondmy@kernel.org wrote:
> > > > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > > 
> > > > > It appears that in certain cases, RDMA capable transports can
> > > > > benefit
> > > > > from the ability to establish multiple connections to increase
> > > > > their
> > > > > throughput. This patch therefore enables the use of the
> > > > > "nconnect" mount
> > > > > option for those use cases.
> > > > > 
> > > > > Signed-off-by: Trond Myklebust
> > > > > <trond.myklebust@hammerspace.com>
> > > > 
> > > > No objection to this patch.
> > > > 
> > > > You don't mention here if you have root-caused the throughput
> > > > issue.
> > > > One thing I've noticed is that contention for the transport's
> > > > queue_lock is holding back the RPC/RDMA Receive completion
> > > > handler,
> > > > which is single-threaded per transport.
> > > 
> > > Curious: how does a queue_lock per transport is a problem for
> > > nconnect? nconnect would create its own transport, would it now and
> > > so
> > > it would have its own queue_lock (per nconnect).
> > 
> > I did not mean to imply that queue_lock contention is a
> > problem for nconnect or would increase when there are
> > multiple transports.
> > 
> > But there is definitely lock contention between the send and
> > receive code paths, and that could be one source of the relief
> > that Trond saw by adding more transports. IMO that contention
> > should be addressed at some point.
> > 
> > I'm not asking for a change to the proposed patch. But I am
> > suggesting some possible future work.
> > 
> 
> We were comparing NFS/RDMA performance to that of NFS/TCP, and it was
> clear that the nconnect value was giving the latter a major boost. Once
> we enabled nconnect for the RDMA channel, then the values evened out a
> lot more.
> Once we fixed the nconnect issue, what we were seeing when the RDMA
> code maxed out was actually that the CPU got pegged running the IB
> completion work queues on writes.
> 
> We can certainly look into improving the performance of
> xprt_lookup_rqst() if we have evidence that is slow, but I'm not yet
> sure that was what we were seeing.

One observation: the Receive completion handler doesn't do anything
that is CPU-intensive. If ib_comp_wq is hot, that's an indication of
lock contention.

I've found there are typically two contended locks when handling
RPC/RDMA Receive completions:

 - The workqueue pool lock. Tejun mitigated that issue in v6.7.

 - The queue_lock, as described above.

A flame graph might narrow the issue.

-- 
Chuck Lever