I've been thinking about what might be required for NFS to run over
QUIC.
Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
I don't have real plans. For Linux, I don't even know if there's a
kernel QUIC implementation planned yet.
QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:
https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/
For example, section 4.3, which explains how to authenticate on top of
an already-encrypted session, should also apply to QUIC.
QUIC runs over UDP, so I think all that would be required to negotiate
support would be to attempt a QUIC connection to port 2049.
The "Transport Layers" section in the NFS RFCs:
https://tools.ietf.org/html/rfc5661#section-2.9
requires transports support reliable and in-order transmission, forbids
clients from retrying a request unless a connection is lost, and forbids
servers from dropping a request without closing a connection. I'm still
vague on how those requirements interact with QUIC's connection
management and 0-RTT reconnection.
https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks
useful, as a guide for applications running over QUIC. It warns that
connections can time out fairly quickly. For timely callbacks over NFS
sessions, that means we need the client to ping the server regularly.
Sounds like that's what they do for HTTP/QUIC to make server push
notifications work:
https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5
HTTP clients are expected to use QUIC PING frames to keep
connections open. Servers SHOULD NOT use PING frames to keep a
connection open. A client SHOULD NOT use PING frames for this
purpose unless there are responses outstanding for requests or
server pushes.
QUIC allows multiple streams per connection--I wonder how we might use
that. RFC 5661 justifies the requirement for an ordered transport with:
Ordered delivery simplifies detection of transmit errors, and
simplifies the sending of arbitrary sized requests and responses
via the record marking protocol.
So as long as we don't try to split a single RPC among streams, I think
we're OK. Would a stream per session slot be reasonable? I'm not sure
what the cost of a stream is.
Do we need to add a new universal address type so the protocol can
specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs
file layouts, fs_locations, etc.) All QUIC needs is an IP address and
maybe a port, so maybe the existing UDP/TCP addresses are enough?
--b.
Hi Bruce-
> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <[email protected]> wrote:
>
> I've been thinking about what might be required for NFS to run over
> QUIC.
>
> Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
>
> I don't have real plans. For Linux, I don't even know if there's a
> kernel QUIC implementation planned yet.
>
> QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:
>
> https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/
The link to the latest version of that document is
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/
> For example, section 4.3, which explains how to authenticate on top of
> an already-encrypted session, should also apply to QUIC.
Most of the document's content will be re-used for defining
RPC-over-QUIC, for example the ALPN defined in Section 8.2.
Lars Eggert, a chair of the QUIC WG, has been helping guide
the RPC-over-TLS effort with an eye towards using QUIC for
RPC when QUIC becomes more mature.
I thought the plan was to write a specification of RPC-over-
QUIC as a new RPC transport type with a netid and uaddr along
with a definition of the transport semantics (a la TI-RPC).
The document would need to explain record marking, peer
authentication, how to use multi-path and multi-stream support,
and so on.
Making NFS work on that transport should then be straightforward
enough that perhaps additional standards work wouldn't be
necessary.
> QUIC runs over UDP, so I think all that would be required to negotiate
> support would be to attempt a QUIC connection to port 2049.
>
> The "Transport Layers" section in the NFS RFCs:
>
> https://tools.ietf.org/html/rfc5661#section-2.9
>
> requires transports support reliable and in-order transmission, forbids
> clients from retrying a request unless a connection is lost, and forbids
> servers from dropping a request without closing a connection. I'm still
> vague on how those requirements interact with QUIC's connection
> management and 0-RTT reconnection.
>
> https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks
> useful, as a guide for applications running over QUIC. It warns that
> connections can time out fairly quickly. For timely callbacks over NFS
> sessions, that means we need the client to ping the server regularly.
> Sounds like that's what they do for HTTP/QUIC to make server push
> notifications work:
>
> https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5
>
> HTTP clients are expected to use QUIC PING frames to keep
> connections open. Servers SHOULD NOT use PING frames to keep a
> connection open. A client SHOULD NOT use PING frames for this
> purpose unless there are responses outstanding for requests or
> server pushes.
>
> QUIC allows multiple streams per connection--I wonder how we might use
> that. RFC 5661 justifies the requirement for an ordered transport with:
>
> Ordered delivery simplifies detection of transmit errors, and
> simplifies the sending of arbitrary sized requests and responses
> via the record marking protocol.
>
> So as long as we don't try to split a single RPC among streams, I think
> we're OK. Would a stream per session slot be reasonable? I'm not sure
> what the cost of a stream is.
>
> Do we need to add a new universal address type so the protocol can
> specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs
> file layouts, fs_locations, etc.) All QUIC needs is an IP address and
> maybe a port, so maybe the existing UDP/TCP addresses are enough?
--
Chuck Lever
[email protected]
On Thu, Sep 03, 2020 at 07:48:19PM -0400, Chuck Lever wrote:
> Hi Bruce-
>
> > On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <[email protected]> wrote:
> >
> > I've been thinking about what might be required for NFS to run over
> > QUIC.
> >
> > Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
> >
> > I don't have real plans. For Linux, I don't even know if there's a
> > kernel QUIC implementation planned yet.
> >
> > QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:
> >
> > https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/
>
> The link to the latest version of that document is
>
> https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/
>
> > For example, section 4.3, which explains how to authenticate on top of
> > an already-encrypted session, should also apply to QUIC.
>
> Most of the document's content will be re-used for defining
> RPC-over-QUIC, for example the ALPN defined in Section 8.2.
> Lars Eggert, a chair of the QUIC WG, has been helping guide
> the RPC-over-TLS effort with an eye towards using QUIC for
> RPC when QUIC becomes more mature.
>
> I thought the plan was to write a specification of RPC-over-
> QUIC as a new RPC transport type with a netid and uaddr along
> with a definition of the transport semantics (a la TI-RPC).
> The document would need to explain record marking, peer
> authentication, how to use multi-path and multi-stream support,
> and so on.
>
> Making NFS work on that transport should then be straightforward
> enough that perhaps additional standards work wouldn't be
> necessary.
Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know
there was a plan.
--b.
> > QUIC runs over UDP, so I think all that would be required to negotiate
> > support would be to attempt a QUIC connection to port 2049.
> >
> > The "Transport Layers" section in the NFS RFCs:
> >
> > https://tools.ietf.org/html/rfc5661#section-2.9
> >
> > requires transports support reliable and in-order transmission, forbids
> > clients from retrying a request unless a connection is lost, and forbids
> > servers from dropping a request without closing a connection. I'm still
> > vague on how those requirements interact with QUIC's connection
> > management and 0-RTT reconnection.
> >
> > https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks
> > useful, as a guide for applications running over QUIC. It warns that
> > connections can time out fairly quickly. For timely callbacks over NFS
> > sessions, that means we need the client to ping the server regularly.
> > Sounds like that's what they do for HTTP/QUIC to make server push
> > notifications work:
> >
> > https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5
> >
> > HTTP clients are expected to use QUIC PING frames to keep
> > connections open. Servers SHOULD NOT use PING frames to keep a
> > connection open. A client SHOULD NOT use PING frames for this
> > purpose unless there are responses outstanding for requests or
> > server pushes.
> >
> > QUIC allows multiple streams per connection--I wonder how we might use
> > that. RFC 5661 justifies the requirement for an ordered transport with:
> >
> > Ordered delivery simplifies detection of transmit errors, and
> > simplifies the sending of arbitrary sized requests and responses
> > via the record marking protocol.
> >
> > So as long as we don't try to split a single RPC among streams, I think
> > we're OK. Would a stream per session slot be reasonable? I'm not sure
> > what the cost of a stream is.
> >
> > Do we need to add a new universal address type so the protocol can
> > specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs
> > file layouts, fs_locations, etc.) All QUIC needs is an IP address and
> > maybe a port, so maybe the existing UDP/TCP addresses are enough?
>
> --
> Chuck Lever
> [email protected]
>
>
> On Sep 3, 2020, at 8:32 PM, Bruce Fields <[email protected]> wrote:
>
> On Thu, Sep 03, 2020 at 07:48:19PM -0400, Chuck Lever wrote:
>> Hi Bruce-
>>
>>> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <[email protected]> wrote:
>>>
>>> I've been thinking about what might be required for NFS to run over
>>> QUIC.
>>>
>>> Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
>>>
>>> I don't have real plans. For Linux, I don't even know if there's a
>>> kernel QUIC implementation planned yet.
>>>
>>> QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:
>>>
>>> https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpc-tls/
>>
>> The link to the latest version of that document is
>>
>> https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rpc-tls/
>>
>>> For example, section 4.3, which explains how to authenticate on top of
>>> an already-encrypted session, should also apply to QUIC.
>>
>> Most of the document's content will be re-used for defining
>> RPC-over-QUIC, for example the ALPN defined in Section 8.2.
>> Lars Eggert, a chair of the QUIC WG, has been helping guide
>> the RPC-over-TLS effort with an eye towards using QUIC for
>> RPC when QUIC becomes more mature.
>>
>> I thought the plan was to write a specification of RPC-over-
>> QUIC as a new RPC transport type with a netid and uaddr along
>> with a definition of the transport semantics (a la TI-RPC).
>> The document would need to explain record marking, peer
>> authentication, how to use multi-path and multi-stream support,
>> and so on.
>>
>> Making NFS work on that transport should then be straightforward
>> enough that perhaps additional standards work wouldn't be
>> necessary.
>
> Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know
> there was a plan.
That's all there is for the moment! :-)
> --b.
>
>>> QUIC runs over UDP, so I think all that would be required to negotiate
>>> support would be to attempt a QUIC connection to port 2049.
>>>
>>> The "Transport Layers" section in the NFS RFCs:
>>>
>>> https://tools.ietf.org/html/rfc5661#section-2.9
>>>
>>> requires transports support reliable and in-order transmission, forbids
>>> clients from retrying a request unless a connection is lost, and forbids
>>> servers from dropping a request without closing a connection. I'm still
>>> vague on how those requirements interact with QUIC's connection
>>> management and 0-RTT reconnection.
>>>
>>> https://www.ietf.org/id/draft-ietf-quic-applicability-07.txt looks
>>> useful, as a guide for applications running over QUIC. It warns that
>>> connections can time out fairly quickly. For timely callbacks over NFS
>>> sessions, that means we need the client to ping the server regularly.
>>> Sounds like that's what they do for HTTP/QUIC to make server push
>>> notifications work:
>>>
>>> https://tools.ietf.org/html/draft-ietf-quic-http-09#section-5
>>>
>>> HTTP clients are expected to use QUIC PING frames to keep
>>> connections open. Servers SHOULD NOT use PING frames to keep a
>>> connection open. A client SHOULD NOT use PING frames for this
>>> purpose unless there are responses outstanding for requests or
>>> server pushes.
>>>
>>> QUIC allows multiple streams per connection--I wonder how we might use
>>> that. RFC 5661 justifies the requirement for an ordered transport with:
>>>
>>> Ordered delivery simplifies detection of transmit errors, and
>>> simplifies the sending of arbitrary sized requests and responses
>>> via the record marking protocol.
>>>
>>> So as long as we don't try to split a single RPC among streams, I think
>>> we're OK. Would a stream per session slot be reasonable? I'm not sure
>>> what the cost of a stream is.
>>>
>>> Do we need to add a new universal address type so the protocol can
>>> specify QUIC endpoints when necessary? (For server-to-server-copy, pnfs
>>> file layouts, fs_locations, etc.) All QUIC needs is an IP address and
>>> maybe a port, so maybe the existing UDP/TCP addresses are enough?
>>
>> --
>> Chuck Lever
>> [email protected]
--
Chuck Lever
[email protected]
We (open source SMB3 developers) have been thinking a lot about SMB3.1.1 over QUIC especially after the interesting talks on this at the last few SDC conferences e.g.
https://www.snia.org/sites/default/files/SDC/2019/presentations/SMB/Li_Frank_Surfing_the_World_Wide_File_SMB3_Improvements_for_Safe_and_Efficient_Internet_Access.pdf (slides 8 and following).
And some of the broader coverage:
https://redmondmag.com/articles/2020/03/02/microsoft-smb-over-quic-to-windows.aspx
I have gotten a chance to talk with some of the developers of the recently open sourced quic libraries (https://github.com/microsoft/msquic) and there are various code style issues that would prevent it being merged into Linux kernel without significant changes (not just to fix camelCase) but ... it is a very very promising place to start.
One of the ideas being bounced around is to do this in stages - first try to cleanup the kernel style issues with the open sourced msquic project and merge it experimentally into the Linux client (cifs.ko) since there should be SMB3.1.1. servers to test it against reasonably soon (presumably we will know a lot more after the 2020 SNIA SDC talk on SMB3.1.1 over QUIC in a few weeks updating us on status https://www.snia.org/events/storage-developer/2020/abstracts#smb-Dantuluri) after we are convinced that the quic driver interoperates ok then move it into the Linux networking stack and let the networking experts clean it up more and remove the 'experimental' tag from it when satisfied.
More discussion is needed on this so we should add samba-technical and some of the network experts as well.
I am very excited about the possibilities opened up by a decent quic kernel driver, but the current cross platform open sourced driver is fairly large - ~90KLOC so will have to be trimmed down (and a lot of boring changes to convert to kernel style needed).
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Thursday, September 3, 2020 7:53 PM
To: Bruce Fields <[email protected]>
Cc: Linux NFS Mailing List <[email protected]>; NFSv4 <[email protected]>; Steven French <[email protected]>
Subject: [EXTERNAL] Re: [nfsv4] NFS over QUIC
> On Sep 3, 2020, at 8:32 PM, Bruce Fields <[email protected]> wrote:
>
> On Thu, Sep 03, 2020 at 07:48:19PM -0400, Chuck Lever wrote:
>> Hi Bruce-
>>
>>> On Sep 3, 2020, at 5:52 PM, J. Bruce Fields <[email protected]> wrote:
>>>
>>> I've been thinking about what might be required for NFS to run over
>>> QUIC.
>>>
>>> Also cc'ing Steve French in case he's thought about this for CIFS/SMB.
>>>
>>> I don't have real plans. For Linux, I don't even know if there's a
>>> kernel QUIC implementation planned yet.
>>>
>>> QUIC uses TLS so we'd probably steal some stuff from the NFS/TLS draft:
>>>
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fda
>>> tatracker.ietf.org%2Fdoc%2Fdraft-cel-nfsv4-rpc-tls%2F&data=02%7C
>>> 01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee6
>>> 3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&am
>>> p;sdata=pOnlPJcTkyJiqQm%2BCbo7dbOAqHMKFIi01qrShVZTH%2Bk%3D&reser
>>> ved=0
>>
>> The link to the latest version of that document is
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdat
>> atracker.ietf.org%2Fdoc%2Fdraft-ietf-nfsv4-rpc-tls%2F&data=02%7C0
>> 1%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%
>> 7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075883604&s
>> data=YYBPryN5sFJApWm%2BkQvmjYzGXxjxI1fXRUWVXvVGE24%3D&reserved=0
>>
>>> For example, section 4.3, which explains how to authenticate on top
>>> of an already-encrypted session, should also apply to QUIC.
>>
>> Most of the document's content will be re-used for defining
>> RPC-over-QUIC, for example the ALPN defined in Section 8.2.
>> Lars Eggert, a chair of the QUIC WG, has been helping guide the
>> RPC-over-TLS effort with an eye towards using QUIC for RPC when QUIC
>> becomes more mature.
>>
>> I thought the plan was to write a specification of RPC-over- QUIC as
>> a new RPC transport type with a netid and uaddr along with a
>> definition of the transport semantics (a la TI-RPC).
>> The document would need to explain record marking, peer
>> authentication, how to use multi-path and multi-stream support, and
>> so on.
>>
>> Making NFS work on that transport should then be straightforward
>> enough that perhaps additional standards work wouldn't be necessary.
>
> Oh, OK, good. Sounds like you're way ahead of me, then, I didn't know
> there was a plan.
That's all there is for the moment! :-)
> --b.
>
>>> QUIC runs over UDP, so I think all that would be required to
>>> negotiate support would be to attempt a QUIC connection to port 2049.
>>>
>>> The "Transport Layers" section in the NFS RFCs:
>>>
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>> ols.ietf.org%2Fhtml%2Frfc5661%23section-2.9&data=02%7C01%7CSteve
>>> n.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee63%7C72f988
>>> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&sdata=ej
>>> gN6OuHrcDMzejeEKbaP5UiD1GAxHfQcPyLZZ45vbo%3D&reserved=0
>>>
>>> requires transports support reliable and in-order transmission,
>>> forbids clients from retrying a request unless a connection is lost,
>>> and forbids servers from dropping a request without closing a
>>> connection. I'm still vague on how those requirements interact with
>>> QUIC's connection management and 0-RTT reconnection.
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>> w.ietf.org%2Fid%2Fdraft-ietf-quic-applicability-07.txt&data=02%7
>>> C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506cee
>>> 63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637347776075893604&sdata=TZeoR%2FT22hg%2BkG0LllDEttlgdzv%2FAsH8PeXbdEHQ9Mk%3D&reserved=0 looks useful, as a guide for applications running over QUIC. It warns that connections can time out fairly quickly. For timely callbacks over NFS sessions, that means we need the client to ping the server regularly.
>>> Sounds like that's what they do for HTTP/QUIC to make server push
>>> notifications work:
>>>
>>>
>>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fto
>>> ols.ietf.org%2Fhtml%2Fdraft-ietf-quic-http-09%23section-5&data=0
>>> 2%7C01%7CSteven.French%40microsoft.com%7C9bab515f15bc49fbbe2c08d8506
>>> cee63%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63734777607589360
>>> 4&sdata=rF91Tq8OXHeN7W5iuSbtYotq1XBC4Zc8DGEGzwPUuTg%3D&reser
>>> ved=0
>>>
>>> HTTP clients are expected to use QUIC PING frames to keep
>>> connections open. Servers SHOULD NOT use PING frames to keep a
>>> connection open. A client SHOULD NOT use PING frames for this
>>> purpose unless there are responses outstanding for requests or
>>> server pushes.
>>>
>>> QUIC allows multiple streams per connection--I wonder how we might
>>> use that. RFC 5661 justifies the requirement for an ordered transport with:
>>>
>>> Ordered delivery simplifies detection of transmit errors, and
>>> simplifies the sending of arbitrary sized requests and responses
>>> via the record marking protocol.
>>>
>>> So as long as we don't try to split a single RPC among streams, I
>>> think we're OK. Would a stream per session slot be reasonable? I'm
>>> not sure what the cost of a stream is.
>>>
>>> Do we need to add a new universal address type so the protocol can
>>> specify QUIC endpoints when necessary? (For server-to-server-copy,
>>> pnfs file layouts, fs_locations, etc.) All QUIC needs is an IP
>>> address and maybe a port, so maybe the existing UDP/TCP addresses are enough?
>>
>> --
>> Chuck Lever
>> [email protected]
--
Chuck Lever
[email protected]