Received-SPF: pass (google.com: domain of linux-nfs+bounces-2147-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Sun, 3 Mar 2024 13:35:06 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: trondmy@kernel.org
Cc: linux-nfs@vger.kernel.org
Subject: Re: [PATCH] NFS: enable nconnect for RDMA
Message-ID: <ZeTC2h59TXUTuCh_@manet.1015granger.net>
References: <20240228213523.35819-1-trondmy@kernel.org>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240228213523.35819-1-trondmy@kernel.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?7WICtEQ7CLhAWTCTWnSCd0o04eS0o56hoYVbztM9ltPhmoeTnH4w5r6p0XZr?=
 =?us-ascii?Q?2UCzNmHsuQ+0z4zSjtTCMTjg3XqpUpKOYmH9ATfKGH6qDcTA4JUWSgdPRSmd?=
 =?us-ascii?Q?j0HqOW3FueDWDOztIewANhkqfRTidDnvMR9CUtKFadcAATFQx4+zSUeNv7Ew?=
 =?us-ascii?Q?OZAvV434cJRAFKd1bNl+1jbO1AEsMTNPqYuFpf4teVfyU4WadJWLtRzyYb4f?=
 =?us-ascii?Q?SmpUr1kVlsS5pS21FTUCiDYQTRyJYTBWQRgYhKOd5//tJRdlkByQktha9Eoa?=
 =?us-ascii?Q?6TXGEF//Di4619wnRxVZzAwRTBBI6vlUT8nfki7hBjUuk/wvLDV5mM6A+2EY?=
 =?us-ascii?Q?Pfu+O8A3ax++V9FdiJVHVclQiIaZE+FIiDwundRs7qMExLlimfoBifCBdgVw?=
 =?us-ascii?Q?l4zz6E5AIqVknSGOErCwFHh9dU2nYt8SWvBr1tJbKp7BTiWBG6208xQPhMeY?=
 =?us-ascii?Q?MOLJxCiY8JhFN3ZkLtVyGCcsB9RuUWVg3pbew+DvIqwFe08AsxL3FBmpvpTB?=
 =?us-ascii?Q?lrS8hZR2XO1UccOLhLjbzA4CAVnw6VK1ONgIIbXWhrKP3EyfpN67JO4meSYs?=
 =?us-ascii?Q?YK3lBRCcVkoyOP1rMsNhN6ZlJBsL7FY6TSWxST8KvD5nT7pL9IlSQyUNKLWN?=
 =?us-ascii?Q?V+dCwZSA8syv4zNpgM/PemqmRNVJSgOxJeNbc0IzY1xCKhgGYqylnx22td+D?=
 =?us-ascii?Q?DsOxo7bTCslu+hlS7Hwi7H5PzxpynB+PhY4sAepmeN3Q5aaJfouPkzRKjGFu?=
 =?us-ascii?Q?ZIcVIxl0M+LAutqBKJz8oPQx0PNmfoOCS+1mMYnrqXdbowA83kd/6ItAxXNO?=
 =?us-ascii?Q?6qaBsO/w3SpBgOtHcBzWTM+cN8N10eAINqyw+vTsmiMx4K2gwKbtFUMIqod/?=
 =?us-ascii?Q?kp1q/DfmOmNqPNwlqYoi2u8qg1h36KXFCz5sufqzfu0y/SaymoR/fWcc2Ida?=
 =?us-ascii?Q?BD5BlDeu5K0XHbgiJ6heE66Pr+AjB9QYRWqpazVMer+T0brz6Ko7gTILwoGF?=
 =?us-ascii?Q?Slj1E9ezCrRb98OHNY+qJXDJenmHS3tK65FqwcM2Ghemn0Pvb2HOzqGTtUFd?=
 =?us-ascii?Q?UiLeGV95kLWsuCUFK393XpulwaIkDRrsPZjzO/kde4wyTSw9ulvpI5r1hJGU?=
 =?us-ascii?Q?MzDEfirg7AG+JusX/XrEcs6zIZALBjhyHEe5L3+h50f5x6rhbH/4ljgkzUWy?=
 =?us-ascii?Q?t7IRUVOc7F6haaDf7cNAvi+nb4/63il20kJIEQ6YgGM7mETNpfz8bNLqGcXS?=
 =?us-ascii?Q?2zLDbkiALQQ76IGUYkxNNfBUlRvwxFPyPqgvVJj9Zjx/XRP2ZIm1ViEcFNs5?=
 =?us-ascii?Q?ZD9ww9zeBoYiobmIspBxm5dPau2xIcqw56N0QhY9oqI9VO+agAf6/LsvyJHi?=
 =?us-ascii?Q?hXBp7JgF7JmOEMitW3u76GCTNntaX0jMTKG/u6Afgz8Pqsc4R9P7gmE33upj?=
 =?us-ascii?Q?G/Se76alupEDmTSk9pJy9j1AkvrN6+dTw4cm7ChFowPR7rwQRugsQUTMusY9?=
 =?us-ascii?Q?acImknqQe+7He8koekLtUIUwO7i83zolxH4N+BiHnYgms0UmDhCOg6CkPgn7?=
 =?us-ascii?Q?m3gG6DEoIX8aosZiqyo9yxhwYZTSJyOH7BYF528YDd3GfpRB4ZeZRXjERtWw?=
 =?us-ascii?Q?+g=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	wqX1paPsHmRbl4hXJ3vPXV7Cej767s9fVACF5PfgUjhDmPRBgo3dShbi6Y+igE6mW0/oRrBULDWkLDEp+W4Qsvj+zMx4zSj8RDS2HbGLOU0ykx/4xMOAtPiCz2ew1YctNzbaGUq1OkgREkY5WF0TApOJDnD88r7+ygmTRDwlMMiTjRI4DXtGqjj/62nLuZkiO1ihz3pwnTqZNfTboGiWHBx8EbO6awYVcPmYvLrUaUQA/gZsPfbEujfFl/ouQ4J9q8eiYch2fzcZOodgBSc6P+jyqRLMVTsWy91pktFiJ9Pmb58ohmm00MgY8ly33dPdSap63LjL5gHOdncdKGNBlZfmGRJpOEB2pxJk3ISTi08P9l6gtjdIcT5GD2/9JHKIxcmuycOHsioLAzxWUa/yNLgBxHKHcvQc/HUnhlRSUuNXcKanKiWvy6RNbt7j/bexrXq0NCZEdC9F6eFMeq62bIM4Spnq+YPiKvao9TL4188Z6mzUrZJyX678y9Ib6CO4u/O+dxsqdsC5ifuLnO8dAJH3gNsPliFboijh5pquC61lK2Guy324XCQz0ttosoBic0RkV1OKvzNLjH0+cBijHsa7OcFYQlU9yyhuv06cAXA=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a65b3906-7998-45ee-732c-08dc3bb0aa6b
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2024 18:35:13.9967
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: rSyRU5zFG3amxxufQnWDgO/BtihY75ytQKR7o22yD9kQGgwIAM8HJaYiyR0xbqLbLzvumVJ7UoPKCmg01gFvxQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR10MB6667
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-03_08,2024-03-01_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0
 adultscore=0 bulkscore=0 spamscore=0 mlxlogscore=999 malwarescore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2403030156
X-Proofpoint-ORIG-GUID: C1luGpSc1s0e6xqh_BOKyWmRdJ21yD0F
X-Proofpoint-GUID: C1luGpSc1s0e6xqh_BOKyWmRdJ21yD0F

On Wed, Feb 28, 2024 at 04:35:23PM -0500, trondmy@kernel.org wrote:
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
> 
> It appears that in certain cases, RDMA capable transports can benefit
> from the ability to establish multiple connections to increase their
> throughput. This patch therefore enables the use of the "nconnect" mount
> option for those use cases.
> 
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>

No objection to this patch.

You don't mention here if you have root-caused the throughput issue.
One thing I've noticed is that contention for the transport's
queue_lock is holding back the RPC/RDMA Receive completion handler,
which is single-threaded per transport.

A way to mitigate this would be to replace the recv_queue
R-B tree with a data structure that can perform a lookup while
holding only the RCU read lock. I have experimented with using an
xarray for the recv_queue, and saw improvement.

The workload on that data structure is different for TCP versus
RDMA, though: on RDMA, the number of RPCs in flight is significantly
smaller. For tens of thousands of RPCs in flight, an xarray might
not scale well. The newer Maple tree or rose bush hash, or maybe a
more classic data structure like rhashtable, might handle this
workload better.

It's also worth considering deleting each RPC from the recv_queue
in a less performance-sensitive context, for example, xprt_release,
rather than in xprt_complete_rqst.


> ---
>  fs/nfs/nfs3client.c | 1 +
>  fs/nfs/nfs4client.c | 2 ++
>  2 files changed, 3 insertions(+)
> 
> diff --git a/fs/nfs/nfs3client.c b/fs/nfs/nfs3client.c
> index 674c012868b1..b0c8a39c2bbd 100644
> --- a/fs/nfs/nfs3client.c
> +++ b/fs/nfs/nfs3client.c
> @@ -111,6 +111,7 @@ struct nfs_client *nfs3_set_ds_client(struct nfs_server *mds_srv,
>  	cl_init.hostname = buf;
>  
>  	switch (ds_proto) {
> +	case XPRT_TRANSPORT_RDMA:
>  	case XPRT_TRANSPORT_TCP:
>  	case XPRT_TRANSPORT_TCP_TLS:
>  		if (mds_clp->cl_nconnect > 1)
> diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
> index 11e3a285594c..84573df5cf5a 100644
> --- a/fs/nfs/nfs4client.c
> +++ b/fs/nfs/nfs4client.c
> @@ -924,6 +924,7 @@ static int nfs4_set_client(struct nfs_server *server,
>  	else
>  		cl_init.max_connect = max_connect;
>  	switch (proto) {
> +	case XPRT_TRANSPORT_RDMA:
>  	case XPRT_TRANSPORT_TCP:
>  	case XPRT_TRANSPORT_TCP_TLS:
>  		cl_init.nconnect = nconnect;
> @@ -1000,6 +1001,7 @@ struct nfs_client *nfs4_set_ds_client(struct nfs_server *mds_srv,
>  	cl_init.hostname = buf;
>  
>  	switch (ds_proto) {
> +	case XPRT_TRANSPORT_RDMA:
>  	case XPRT_TRANSPORT_TCP:
>  	case XPRT_TRANSPORT_TCP_TLS:
>  		if (mds_clp->cl_nconnect > 1) {
> -- 
> 2.44.0
> 
> 

-- 
Chuck Lever