Hi,
I had one query with NFS rdma support, details mentioned in the mail, but
couldn't reach out to [email protected]
Please check if this is the right mailing list to ask this question, else
point me to the correct mailing list.
Thanks,
Gaurav Gangalwar
---------- Forwarded message ---------
From: gaurav gangalwar <[email protected]>
Date: Fri, Jan 26, 2024 at 3:14 PM
Subject: NFS4.0 rdma with referal
To: <[email protected]>
Hello,
I was experimenting with Linux nfs kernel rdma with nfsref.
With direct mount NFSv4.0 it works fine with rdma
But when I add a referral using nfsref, it uses tcp for referral mount.
[root@uvm-ca102ba ~]# sudo mount -vvv -o rdma,port=20049,vers=4.0
10.53.65.28:/expdir rdma-kernel
mount.nfs: timeout set for Fri Jan 26 01:28:40 2024
mount.nfs: trying text-based options
'rdma,port=20049,vers=4.0,addr=10.53.65.28,clientaddr=10.51.43.103'
[root@uvm-ca102ba ~]# cd rdma-kernel
[root@uvm-ca102ba rdma-kernel]# find .
.
./expdir1
[root@uvm-ca102ba rdma-kernel]# nfsstat -m
/root/rdma-kernel from 10.53.65.28:/expdir
Flags:
rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=
*rdma*
,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.51.43.103,local_lock=none,addr=10.53.65.28
/root/rdma-kernel/expdir1 from 10.53.65.36:/expdir1
Flags:
rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=*tcp*
,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.51.43.103,local_lock=none,addr=10.53.65.36
I am using SoftROCE and both the ips have a rdma port added for nfs.
~$ cat /proc/fs/nfsd/portlist
rdma 20049
rdma 20049
udp 2049
tcp 2049
udp 2049
tcp 2049
Attaching tcpdump also for same
Why is the client not using rdma to mount referral fs location?
> On Jan 26, 2024, at 4:53 AM, gaurav gangalwar <[email protected]> wrote:
>
> Hi,
> I had one query with NFS rdma support, details mentioned in the mail, but couldn't reach out to [email protected]
> Please check if this is the right mailing list to ask this question, else point me to the correct mailing list.
>
> Thanks,
> Gaurav Gangalwar
>
> ---------- Forwarded message ---------
> From: gaurav gangalwar <[email protected]>
> Date: Fri, Jan 26, 2024 at 3:14 PM
> Subject: NFS4.0 rdma with referal
> To: <[email protected]>
>
>
> Hello,
> I was experimenting with Linux nfs kernel rdma with nfsref.
> With direct mount NFSv4.0 it works fine with rdma
> But when I add a referral using nfsref, it uses tcp for referral mount.
>
> [root@uvm-ca102ba ~]# sudo mount -vvv -o rdma,port=20049,vers=4.0 10.53.65.28:/expdir rdma-kernel
> mount.nfs: timeout set for Fri Jan 26 01:28:40 2024
> mount.nfs: trying text-based options 'rdma,port=20049,vers=4.0,addr=10.53.65.28,clientaddr=10.51.43.103'
> [root@uvm-ca102ba ~]# cd rdma-kernel
> [root@uvm-ca102ba rdma-kernel]# find .
> .
> ./expdir1
> [root@uvm-ca102ba rdma-kernel]# nfsstat -m
> /root/rdma-kernel from 10.53.65.28:/expdir
> Flags: rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.51.43.103,local_lock=none,addr=10.53.65.28
>
> /root/rdma-kernel/expdir1 from 10.53.65.36:/expdir1
> Flags: rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=10.51.43.103,local_lock=none,addr=10.53.65.36
>
>
> I am using SoftROCE and both the ips have a rdma port added for nfs.
>
> ~$ cat /proc/fs/nfsd/portlist
> rdma 20049
> rdma 20049
> udp 2049
> tcp 2049
> udp 2049
> tcp 2049
>
> Attaching tcpdump also for same
>
> Why is the client not using rdma to mount referral fs location?
This is only speculation.
The client would need to negotiate the transport setting if
it wants to try rdma. It can't assume that a referral that
it encounters on an proto=rdma mount will redirect to a
server that also has RDMA.
I don't think the Linux NFS client does that negotiating yet.
--
Chuck Lever