Received-SPF: pass (google.com: domain of linux-nfs+bounces-3396-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <CAJiE4OkE5=6Jw3kf+vrfDYsR5ybJDKUffWGXAQd2R2AJO=4Fwg@mail.gmail.com>
 <CAJiE4Okdie0u0YCxHj6XsQOcxTYecqQ=P-R=iuzn-iipphkwHQ@mail.gmail.com>
 <04BECFE4-7BC0-40CE-90DE-9EEF0DB973BF@oracle.com> <CAJiE4Ok5FLgk0Asq92m7HhDwAgbiTuWo57ff=i_zaOxhnV6YvA@mail.gmail.com>
 <CAJiE4Om+HWMCuQYsxsaOCnTAZAuLxA4B+3phqeTJNT81gp_mJw@mail.gmail.com>
 <CAJiE4OnNpnJb7EYfujgcRbaUPqruo2j3hOdmmt3LHNHEJ8bw6g@mail.gmail.com>
 <ZcecJNwQnXPpJ3fT@tissot.1015granger.net> <CAJiE4O=7C=6rAfkzw4dT9bs_VrxLCqYhp1tszn2h62P2ZVuiGg@mail.gmail.com>
 <36C42C4D-E296-41B0-8418-1CFC5AB61107@oracle.com> <CAJiE4Okpwnh4aQvfaVqaSt=vHo6cVuid4w9Xd4_yGEaa3ZkGQQ@mail.gmail.com>
In-Reply-To: <CAJiE4Okpwnh4aQvfaVqaSt=vHo6cVuid4w9Xd4_yGEaa3ZkGQQ@mail.gmail.com>
From: gaurav gangalwar <gaurav.gangalwar@gmail.com>
Date: Mon, 27 May 2024 10:45:59 +0530
Message-ID: <CAJiE4OnAwurNwF+jRB84H3sfM2VK5+4Ci3a6ovrZk75zmV2Hdw@mail.gmail.com>
Subject: Re: NFS4.0 rdma with referal
To: Chuck Lever III <chuck.lever@oracle.com>, 
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi,
Facing one more issue while using referrals with RDMA
If RDMA is enabled and supported on both client and server and If we
mount parent with TCP. Then referral/submount mount will be done over
RDMA instead of TCP, since for referral/submount mount the client
tries with RDMA first and then TCP only of RDMA connections fails.

As we can see here parent /home  .160, mounted with tcp, t1 is
referral mount mounted with rdma

> /root/tcp-mnt1 from 10.53.87.160:/home
>  Flags: rw,relatime,vers=3D4.0,rsize=3D1048576,wsize=3D1048576,namlen=3D2=
55,hard,proto=3Dtcp,timeo=3D600,retrans=3D2,sec=3Dsys,clientaddr=3D10.53.87=
.158,local_lock=3Dnone,addr=3D10.53.87.160
>
> /root/tcp-mnt1/t1 from 10.53.87.157:/:home/t1
>  Flags: rw,relatime,vers=3D4.0,rsize=3D1048576,wsize=3D1048576,namlen=3D2=
55,hard,proto=3Drdma,port=3D20049,timeo=3D600,retrans=3D2,sec=3Dsys,clienta=
ddr=3D10.53.87.158,local_lock=3Dnone,addr=3D10.53.87.157


Code which tries RDMA first, can we get transport type from parent and
use the same?

> #if IS_ENABLED(CONFIG_SUNRPC_XPRT_RDMA)
>
>         rpc_set_port(&ctx->nfs_server.address, NFS_RDMA_PORT);
>
>         error =3D nfs4_set_client(server,
>
>                                 ctx->nfs_server.hostname,
>
>                                 &ctx->nfs_server._address,
>
>                                 ctx->nfs_server.addrlen,
>
>                                 parent_client->cl_ipaddr,
>
>                                 XPRT_TRANSPORT_RDMA,
>
>                                 parent_server->client->cl_timeout,
>
>                                 parent_client->cl_mvops->minor_version,
>
>                                 parent_client->cl_nconnect,
>
>                                 parent_client->cl_max_connect,
>
>                                 parent_client->cl_net,
>
>                                 &parent_client->cl_xprtsec);
>
>         if (!error)
>
>                 goto init_server;
>
> #endif  /* IS_ENABLED(CONFIG_SUNRPC_XPRT_RDMA) */
>
>

Regards,
Gaurav Gangalwar

On Tue, Feb 27, 2024 at 6:22=E2=80=AFPM gaurav gangalwar
<gaurav.gangalwar@gmail.com> wrote:
>
> One more issue with referral code is there is no retry on connection fail=
ure
>
>> Feb 26 01:49:32 rbt-el7-1 kernel: nfs_create_rpc_client: cannot create R=
PC client. Error =3D -111
>> Feb 26 01:49:32 rbt-el7-1 kernel: NFS4: Couldn't follow remote path
>> Feb 26 01:49:32 rbt-el7-1 kernel: <-- nfs4_get_referral_tree() =3D -111 =
[error]
>
>
> I was expecting retries from the client if submount fails if it's a hard =
mount on parent, but it fails submount.
> I can understand we will be stuck in a loop if fs info is not valid, then=
 connection will always fail.
>
> Regards,
> Gaurav Gangalwar
>
> On Mon, Feb 12, 2024 at 7:23=E2=80=AFPM Chuck Lever III <chuck.lever@orac=
le.com> wrote:
>>
>>
>>
>> > On Feb 12, 2024, at 12:51=E2=80=AFAM, gaurav gangalwar <gaurav.gangalw=
ar@gmail.com> wrote:
>> >
>> > I think I was using an older kernel version on a client which doesn't =
have your fix.
>> > I tried with the newer version v5.10, it worked fine.
>> >
>> > The only issue I see is we are not inheriting port from the parent in =
nfs4_create_referral_server, so even if I use port=3D20047 in mount it will=
 try referral submount on 20049 only.
>> >
>> > rpc_set_port(data->addr, NFS_RDMA_PORT);
>> >
>> > We could inherit this also from parent?
>>
>> The client is supposed to use the port number information contained
>> in the referral. There's nothing that mandates that the two servers
>> will use the same alternate port.
>>
>> Using a constant here is probably wrong for both the TCP and RDMA
>> cases, though.
>>
>>
>> --
>> Chuck Lever
>>
>>