Hi,
I know NFSv4 is definitely a connection oriented protocol, but noting
prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead
of UDP could be of great interest, especially in a HPC context where
thousands of clients will operate at the same time. If a nfs mount point
is made over udp, what would be the side effects ? I tried to run the
connectathon on such a udp based mount point and I could find no errors.
Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?
Thanks in advance for your answer.
Regards
Philippe
Hi
> The Linux server allows it, but I've been considering that a
> (low-priority) bug, so it wouldn't be safe to assume it will continue
> working.
>
I am mostly thinking at the client side using as well udp and tcp. Would
the "NFSv4 client over UDP" behaves differently then the same client
over TCP ?
> That aside, if you have a perfect network,
I can trust my network. It's not a WAN, it's located into a very massive
cluster (it's kind of "internal LAN"). It's a very high throughput
network (IB based) so I believe there are less "hardware based reason"
to loose packets.
> NFSv4.0 at least will
> probably work.
OK.
> (Not 4.1 since backchannel setup will fail?)
>
What is erroneous in using UDP for NFSv4.1 backchannels ?
> Are you really sure that you can't make tcp scale to thousands of
> clients?
>
I am a bit afraid of a "No more file descriptors" effect. If I have one
TCP socket per client and thousands of clients, I have less remaining
fds for other purposes. Another point : UDP is a "cheap" protocol. I can
have bunches of clients without overloading the server (a new client
will almost cost nothing to the server, just the cost of a new clientid
negotiation) . I was wondering if it could be reliable to use it for
NFSv4 inside a large cluster.
Philippe
On Tue, Dec 14, 2010 at 08:35:14AM +0100, DENIEL Philippe wrote:
> Hi
> >The Linux server allows it, but I've been considering that a
> >(low-priority) bug, so it wouldn't be safe to assume it will continue
> >working.
> I am mostly thinking at the client side using as well udp and tcp.
> Would the "NFSv4 client over UDP" behaves differently then the same
> client over TCP ?
>
> >That aside, if you have a perfect network,
> I can trust my network. It's not a WAN, it's located into a very
> massive cluster (it's kind of "internal LAN"). It's a very high
> throughput network (IB based) so I believe there are less "hardware
> based reason" to loose packets.
>
> >NFSv4.0 at least will
> >probably work.
> OK.
> >(Not 4.1 since backchannel setup will fail?)
> What is erroneous in using UDP for NFSv4.1 backchannels ?
I assumed it would require some stronger notion of a "connection" than
UDP could provide. But maybe it could be made to work somehow.
> >Are you really sure that you can't make tcp scale to thousands of
> >clients?
> I am a bit afraid of a "No more file descriptors" effect. If I have
> one TCP socket per client and thousands of clients, I have less
> remaining fds for other purposes. Another point : UDP is a "cheap"
> protocol. I can have bunches of clients without overloading the
> server (a new client will almost cost nothing to the server, just
> the cost of a new clientid negotiation) .
Is the tcp state all that much more?
> I was wondering if it
> could be reliable to use it for NFSv4 inside a large cluster.
Well, it'd require some spec changes in any case.
--b.
On Mon, 2010-12-13 at 14:16 -0500, J. Bruce Fields wrote:
> On Mon, Dec 13, 2010 at 01:04:28PM -0600, Thomas Haynes wrote:
> >
> > On Dec 13, 2010, at 1:46 AM, DENIEL Philippe wrote:
> >
> > > Hi,
> > >
> > > I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?
> >
> >
> > You might find that some implementations have UDP disabled and it might then make
> > it hard to triage what is going on.
> >
> > I believe both NetApp's OnTap and Oracle's Solaris have it turned off.
>
> The Linux server allows it, but I've been considering that a
> (low-priority) bug, so it wouldn't be safe to assume it will continue
> working.
>
> That aside, if you have a perfect network, NFSv4.0 at least will
> probably work. (Not 4.1 since backchannel setup will fail?)
>
> Are you really sure that you can't make tcp scale to thousands of
> clients?
Maybe SCTP would be the way to go then? We've often talked about it over
the years, but nobody has yet (to my knowledge) written an ONC RPC
transport for it.
Cheers
Trond
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
On Mon, Dec 13, 2010 at 01:04:28PM -0600, Thomas Haynes wrote:
>
> On Dec 13, 2010, at 1:46 AM, DENIEL Philippe wrote:
>
> > Hi,
> >
> > I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?
>
>
> You might find that some implementations have UDP disabled and it might then make
> it hard to triage what is going on.
>
> I believe both NetApp's OnTap and Oracle's Solaris have it turned off.
The Linux server allows it, but I've been considering that a
(low-priority) bug, so it wouldn't be safe to assume it will continue
working.
That aside, if you have a perfect network, NFSv4.0 at least will
probably work. (Not 4.1 since backchannel setup will fail?)
Are you really sure that you can't make tcp scale to thousands of
clients?
--b.
>
>
> >
> > Thanks in advance for your answer.
> >
> > Regards
> >
> > Philippe
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Dec 13, 2010, at 1:46 AM, DENIEL Philippe wrote:
> Hi,
>
> I know NFSv4 is definitely a connection oriented protocol, but noting prevent from using "mount -t nfs4 -o proto=udp <...>". Using TCP instead of UDP could be of great interest, especially in a HPC context where thousands of clients will operate at the same time. If a nfs mount point is made over udp, what would be the side effects ? I tried to run the connectathon on such a udp based mount point and I could find no errors. Is NFSv4/UDP a suitable solution ? Or does it make no sense to use udp ?
You might find that some implementations have UDP disabled and it might then make
it hard to triage what is going on.
I believe both NetApp's OnTap and Oracle's Solaris have it turned off.
>
> Thanks in advance for your answer.
>
> Regards
>
> Philippe
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html