Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
NFSv3 client can mount with "xprtsec=none" a filesystem exported with
"xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
mount to its mount table - it can't actually access the mount).
Here's an example using machines from the recent Bakeathon.
Mounting a server with TLS enabled:
# mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
# mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
Yet a v3 mount without "xprtsec=tls" works:
# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# umount /mnt
and a mount with no explicit version and without "xprtsec=tls" falls back to
v3 and also "works":
# mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
# grep ora /proc/mounts
oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
+rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
Even though the filesystem is mounted, the client can't do anything with it:
# ls /mnt
ls: cannot open directory '/mnt': Permission denied
When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
The client compares that list with its own list of auth flavors parsed from the
mount request and returns -EACCES if no match is found (see
nfs_verify_authflavors()).
Perhaps we should be doing something similar with xprtsec policies? Should
there be an errata to RFC 9289 and a request from IANA for assigned numbers for
pseudo-flavors corresponding to xprtsec policies?
If not, this behavior should at least be documented in the man pages.
-Scott
> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
>
> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> mount to its mount table - it can't actually access the mount).
>
> Here's an example using machines from the recent Bakeathon.
>
> Mounting a server with TLS enabled:
>
> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> # umount /mnt
>
> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
>
> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
>
> Yet a v3 mount without "xprtsec=tls" works:
>
> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> # umount /mnt
>
> and a mount with no explicit version and without "xprtsec=tls" falls back to
> v3 and also "works":
>
> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> # grep ora /proc/mounts
> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
>
> Even though the filesystem is mounted, the client can't do anything with it:
>
> # ls /mnt
> ls: cannot open directory '/mnt': Permission denied
>
> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> The client compares that list with its own list of auth flavors parsed from the
> mount request and returns -EACCES if no match is found (see
> nfs_verify_authflavors()).
>
> Perhaps we should be doing something similar with xprtsec policies?
The problem might be in how you've set up the exports. With NFSv3,
the parent export needs the "crossmnt" export option in order for
NFSv3 to behave like NFSv4 in this regard, although I could have
missed something.
> Should
> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> pseudo-flavors corresponding to xprtsec policies?
No. Transport-layer security is not an RPC security flavor or
pseudo-flavor. These two things are not related.
(And in fact, I proposed something like this for NFSv4 SECINFO,
but it was rejected).
> If not, this behavior should at least be documented in the man pages.
"crossmnt", and it's kin "nohide", are explained in exports(5).
--
Chuck Lever
On Thu, 02 May 2024, Chuck Lever III wrote:
>
>
> > On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> >
> > Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > mount to its mount table - it can't actually access the mount).
> >
> > Here's an example using machines from the recent Bakeathon.
> >
> > Mounting a server with TLS enabled:
> >
> > # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > # umount /mnt
> >
> > Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> >
> > # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> >
> > Yet a v3 mount without "xprtsec=tls" works:
> >
> > # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > # umount /mnt
> >
> > and a mount with no explicit version and without "xprtsec=tls" falls back to
> > v3 and also "works":
> >
> > # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > # grep ora /proc/mounts
> > oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> >
> > Even though the filesystem is mounted, the client can't do anything with it:
> >
> > # ls /mnt
> > ls: cannot open directory '/mnt': Permission denied
> >
> > When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > The client compares that list with its own list of auth flavors parsed from the
> > mount request and returns -EACCES if no match is found (see
> > nfs_verify_authflavors()).
> >
> > Perhaps we should be doing something similar with xprtsec policies?
>
> The problem might be in how you've set up the exports. With NFSv3,
> the parent export needs the "crossmnt" export option in order for
> NFSv3 to behave like NFSv4 in this regard, although I could have
> missed something.
I was mounting your server though :)
>
>
> > Should
> > there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > pseudo-flavors corresponding to xprtsec policies?
>
> No. Transport-layer security is not an RPC security flavor or
> pseudo-flavor. These two things are not related.
>
> (And in fact, I proposed something like this for NFSv4 SECINFO,
> but it was rejected).
I thought it might be a stretch to try to use mountres3.auth_flavors for
this, but since RFC 9289 does refer to AUTH_TLS as an authentication
flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
also lists TLS under the Flavor Name column I thought it might make
sense to treat xprtsec policies as if they were pseudo-flavors even
though they're not, if only to give the client a way to determine that
the mount should fail.
>
>
> > If not, this behavior should at least be documented in the man pages.
>
> "crossmnt", and it's kin "nohide", are explained in exports(5).
rpc.mountd doesn't do any access checking based on xprtsec policies on
the export (or krb5 pseudo-flavors, for that matter), so I don't see how
"crossmount" or "nohide" would have any effect here.
-Scott
>
>
> --
> Chuck Lever
>
>
> On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
>
> On Thu, 02 May 2024, Chuck Lever III wrote:
>
>>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
>>>
>>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
>>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
>>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
>>> mount to its mount table - it can't actually access the mount).
>>>
>>> Here's an example using machines from the recent Bakeathon.
>>>
>>> Mounting a server with TLS enabled:
>>>
>>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
>>> # umount /mnt
>>>
>>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
>>>
>>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
>>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
>>>
>>> Yet a v3 mount without "xprtsec=tls" works:
>>>
>>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
>>> # umount /mnt
>>>
>>> and a mount with no explicit version and without "xprtsec=tls" falls back to
>>> v3 and also "works":
>>>
>>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
>>> # grep ora /proc/mounts
>>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
>>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
>>>
>>> Even though the filesystem is mounted, the client can't do anything with it:
>>>
>>> # ls /mnt
>>> ls: cannot open directory '/mnt': Permission denied
>>>
>>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
>>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
>>> The client compares that list with its own list of auth flavors parsed from the
>>> mount request and returns -EACCES if no match is found (see
>>> nfs_verify_authflavors()).
>>>
>>> Perhaps we should be doing something similar with xprtsec policies?
>>
>> The problem might be in how you've set up the exports. With NFSv3,
>> the parent export needs the "crossmnt" export option in order for
>> NFSv3 to behave like NFSv4 in this regard, although I could have
>> missed something.
>
> I was mounting your server though :)
OK, then not the same bug that Olga found last year.
We should find out what FreeBSD does in this case.
>>> Should
>>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
>>> pseudo-flavors corresponding to xprtsec policies?
>>
>> No. Transport-layer security is not an RPC security flavor or
>> pseudo-flavor. These two things are not related.
>>
>> (And in fact, I proposed something like this for NFSv4 SECINFO,
>> but it was rejected).
>
> I thought it might be a stretch to try to use mountres3.auth_flavors for
> this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> also lists TLS under the Flavor Name column I thought it might make
> sense to treat xprtsec policies as if they were pseudo-flavors even
> though they're not, if only to give the client a way to determine that
> the mount should fail.
RPC_AUTH_TLS is used only when a client probes a server to see if
it supports RPC-with-TLS. At all other times, the client uses one
of the normal, legitimate flavors. It does not represent a security
flavor that can be used during regular operation.
NFSv3 mount failover logic is still open for discussion (ie, incomplete).
Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
list? I think clients that don't recognize it should ignore it,
but I'm not sure. What should a client do if it sees that flavor in
the list? It's not one that can be used for any other procedure than
a NULL RPC.
>>> If not, this behavior should at least be documented in the man pages.
>>
>> "crossmnt", and it's kin "nohide", are explained in exports(5).
>
> rpc.mountd doesn't do any access checking based on xprtsec policies on
> the export (or krb5 pseudo-flavors, for that matter), so I don't see how
> "crossmount" or "nohide" would have any effect here.
No, they don't, you are correct.
--
Chuck Lever
On Thu, 02 May 2024, Chuck Lever III wrote:
>
>
> > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> >
> > On Thu, 02 May 2024, Chuck Lever III wrote:
> >
> >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> >>>
> >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> >>> mount to its mount table - it can't actually access the mount).
> >>>
> >>> Here's an example using machines from the recent Bakeathon.
> >>>
> >>> Mounting a server with TLS enabled:
> >>>
> >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> >>> # umount /mnt
> >>>
> >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> >>>
> >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> >>>
> >>> Yet a v3 mount without "xprtsec=tls" works:
> >>>
> >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> >>> # umount /mnt
> >>>
> >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> >>> v3 and also "works":
> >>>
> >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> >>> # grep ora /proc/mounts
> >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> >>>
> >>> Even though the filesystem is mounted, the client can't do anything with it:
> >>>
> >>> # ls /mnt
> >>> ls: cannot open directory '/mnt': Permission denied
> >>>
> >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> >>> The client compares that list with its own list of auth flavors parsed from the
> >>> mount request and returns -EACCES if no match is found (see
> >>> nfs_verify_authflavors()).
> >>>
> >>> Perhaps we should be doing something similar with xprtsec policies?
> >>
> >> The problem might be in how you've set up the exports. With NFSv3,
> >> the parent export needs the "crossmnt" export option in order for
> >> NFSv3 to behave like NFSv4 in this regard, although I could have
> >> missed something.
> >
> > I was mounting your server though :)
>
> OK, then not the same bug that Olga found last year.
>
> We should find out what FreeBSD does in this case.
I thought about that. Rick's servers from the BAT are offline, and I
don't think he was exporting v3 anyway.
>
>
> >>> Should
> >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> >>> pseudo-flavors corresponding to xprtsec policies?
> >>
> >> No. Transport-layer security is not an RPC security flavor or
> >> pseudo-flavor. These two things are not related.
> >>
> >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> >> but it was rejected).
> >
> > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > also lists TLS under the Flavor Name column I thought it might make
> > sense to treat xprtsec policies as if they were pseudo-flavors even
> > though they're not, if only to give the client a way to determine that
> > the mount should fail.
>
> RPC_AUTH_TLS is used only when a client probes a server to see if
> it supports RPC-with-TLS. At all other times, the client uses one
> of the normal, legitimate flavors. It does not represent a security
> flavor that can be used during regular operation.
>
> NFSv3 mount failover logic is still open for discussion (ie, incomplete).
>
> Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> list? I think clients that don't recognize it should ignore it,
> but I'm not sure. What should a client do if it sees that flavor in
> the list? It's not one that can be used for any other procedure than
> a NULL RPC.
Maybe? After the client gets the filehandle it's calling FSINFO and
PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
checking for a negative return code from the PATHCONF operation. If it
did, it could maybe use the -EACCES coupled with the knowledge that the
server had RPC_AUTH_TLS enabled to emit an error message saying to check
the xprtsec policies (but I don't think that would be as definitive as
what I had in mind) and to fail the mount.
-Scott
>
>
> >>> If not, this behavior should at least be documented in the man pages.
> >>
> >> "crossmnt", and it's kin "nohide", are explained in exports(5).
> >
> > rpc.mountd doesn't do any access checking based on xprtsec policies on
> > the export (or krb5 pseudo-flavors, for that matter), so I don't see how
> > "crossmount" or "nohide" would have any effect here.
>
> No, they don't, you are correct.
>
>
> --
> Chuck Lever
>
>
On Thu, 2024-05-02 at 14:51 -0400, Scott Mayhew wrote:
> On Thu, 02 May 2024, Chuck Lever III wrote:
>
> >
> >
> > > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> > >
> > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > >
> > > > > On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> > > > >
> > > > > Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > > > > NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > > > > "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > > > > mount to its mount table - it can't actually access the mount).
> > > > >
> > > > > Here's an example using machines from the recent Bakeathon.
> > > > >
> > > > > Mounting a server with TLS enabled:
> > > > >
> > > > > # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > # umount /mnt
> > > > >
> > > > > Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > > > >
> > > > > # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > > > >
> > > > > Yet a v3 mount without "xprtsec=tls" works:
> > > > >
> > > > > # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > # umount /mnt
> > > > >
> > > > > and a mount with no explicit version and without "xprtsec=tls" falls back to
> > > > > v3 and also "works":
> > > > >
> > > > > # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > # grep ora /proc/mounts
> > > > > oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > > > > +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > > > >
> > > > > Even though the filesystem is mounted, the client can't do anything with it:
> > > > >
> > > > > # ls /mnt
> > > > > ls: cannot open directory '/mnt': Permission denied
> > > > >
> > > > > When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > > > > mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > > > > The client compares that list with its own list of auth flavors parsed from the
> > > > > mount request and returns -EACCES if no match is found (see
> > > > > nfs_verify_authflavors()).
> > > > >
> > > > > Perhaps we should be doing something similar with xprtsec policies?
> > > >
> > > > The problem might be in how you've set up the exports. With NFSv3,
> > > > the parent export needs the "crossmnt" export option in order for
> > > > NFSv3 to behave like NFSv4 in this regard, although I could have
> > > > missed something.
> > >
> > > I was mounting your server though :)
> >
> > OK, then not the same bug that Olga found last year.
> >
> > We should find out what FreeBSD does in this case.
>
> I thought about that. Rick's servers from the BAT are offline, and I
> don't think he was exporting v3 anyway.
>
> >
> >
> > > > > Should
> > > > > there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > > > > pseudo-flavors corresponding to xprtsec policies?
> > > >
> > > > No. Transport-layer security is not an RPC security flavor or
> > > > pseudo-flavor. These two things are not related.
> > > >
> > > > (And in fact, I proposed something like this for NFSv4 SECINFO,
> > > > but it was rejected).
> > >
> > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > also lists TLS under the Flavor Name column I thought it might make
> > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > though they're not, if only to give the client a way to determine that
> > > the mount should fail.
> >
> > RPC_AUTH_TLS is used only when a client probes a server to see if
> > it supports RPC-with-TLS. At all other times, the client uses one
> > of the normal, legitimate flavors. It does not represent a security
> > flavor that can be used during regular operation.
> >
> > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> >
> > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > list? I think clients that don't recognize it should ignore it,
> > but I'm not sure. What should a client do if it sees that flavor in
> > the list? It's not one that can be used for any other procedure than
> > a NULL RPC.
>
> Maybe? After the client gets the filehandle it's calling FSINFO and
> PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> checking for a negative return code from the PATHCONF operation. If it
> did, it could maybe use the -EACCES coupled with the knowledge that the
> server had RPC_AUTH_TLS enabled to emit an error message saying to check
> the xprtsec policies (but I don't think that would be as definitive as
> what I had in mind) and to fail the mount.
>
That sounds reasonable if it can be made to work.
One could argue that to properly implement NFSv3 over TLS, that the
sidecar protocols (including mountd) should be TLS-enabled as well. If
that were the case, then we could just make the mount error out when
the TLS handshake doesn't work.
That _is_ a bit draconian though, and I don't see anyone lining up to
do the mountd work anytime soon.
> >
> >
> > > > > If not, this behavior should at least be documented in the man pages.
> > > >
> > > > "crossmnt", and it's kin "nohide", are explained in exports(5).
> > >
> > > rpc.mountd doesn't do any access checking based on xprtsec policies on
> > > the export (or krb5 pseudo-flavors, for that matter), so I don't see how
> > > "crossmount" or "nohide" would have any effect here.
> >
> > No, they don't, you are correct.
> >
> >
> > --
> > Chuck Lever
> >
> >
>
>
--
Jeffrey Layton <[email protected]>
On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote:
> On Thu, 02 May 2024, Chuck Lever III wrote:
>
> >
> >
> > > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> > >
> > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > >
> > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> > >>>
> > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > >>> mount to its mount table - it can't actually access the mount).
> > >>>
> > >>> Here's an example using machines from the recent Bakeathon.
> > >>>
> > >>> Mounting a server with TLS enabled:
> > >>>
> > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> # umount /mnt
> > >>>
> > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > >>>
> > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > >>>
> > >>> Yet a v3 mount without "xprtsec=tls" works:
> > >>>
> > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> # umount /mnt
> > >>>
> > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> > >>> v3 and also "works":
> > >>>
> > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > >>> # grep ora /proc/mounts
> > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > >>>
> > >>> Even though the filesystem is mounted, the client can't do anything with it:
> > >>>
> > >>> # ls /mnt
> > >>> ls: cannot open directory '/mnt': Permission denied
> > >>>
> > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > >>> The client compares that list with its own list of auth flavors parsed from the
> > >>> mount request and returns -EACCES if no match is found (see
> > >>> nfs_verify_authflavors()).
> > >>>
> > >>> Perhaps we should be doing something similar with xprtsec policies?
> > >>
> > >> The problem might be in how you've set up the exports. With NFSv3,
> > >> the parent export needs the "crossmnt" export option in order for
> > >> NFSv3 to behave like NFSv4 in this regard, although I could have
> > >> missed something.
> > >
> > > I was mounting your server though :)
> >
> > OK, then not the same bug that Olga found last year.
> >
> > We should find out what FreeBSD does in this case.
>
> I thought about that. Rick's servers from the BAT are offline, and I
> don't think he was exporting v3 anyway.
>
> >
> >
> > >>> Should
> > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > >>> pseudo-flavors corresponding to xprtsec policies?
> > >>
> > >> No. Transport-layer security is not an RPC security flavor or
> > >> pseudo-flavor. These two things are not related.
> > >>
> > >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> > >> but it was rejected).
> > >
> > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > also lists TLS under the Flavor Name column I thought it might make
> > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > though they're not, if only to give the client a way to determine that
> > > the mount should fail.
> >
> > RPC_AUTH_TLS is used only when a client probes a server to see if
> > it supports RPC-with-TLS. At all other times, the client uses one
> > of the normal, legitimate flavors. It does not represent a security
> > flavor that can be used during regular operation.
> >
> > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> >
> > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > list? I think clients that don't recognize it should ignore it,
> > but I'm not sure. What should a client do if it sees that flavor in
> > the list? It's not one that can be used for any other procedure than
> > a NULL RPC.
>
> Maybe? After the client gets the filehandle it's calling FSINFO and
> PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> checking for a negative return code from the PATHCONF operation. If it
> did, it could maybe use the -EACCES coupled with the knowledge that the
> server had RPC_AUTH_TLS enabled to emit an error message saying to check
> the xprtsec policies (but I don't think that would be as definitive as
> what I had in mind) and to fail the mount.
If Linux is the only implementation of NFSv3 with TLS so far, then
we have some latitude for innovation.
I would like to hear from the client maintainers about what they
would prefer the client user experience to look like. Then NFSD's
behavior can be adjusted to accommodate.
In this case, Steve would have to sign off on an rpc.mountd change
to return AUTH_TLS in the auth_flavors list.
--
Chuck Lever
On Fri, 03 May 2024, Chuck Lever wrote:
> On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote:
> > On Thu, 02 May 2024, Chuck Lever III wrote:
> >
> > >
> > >
> > > > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> > > >
> > > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > > >
> > > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> > > >>>
> > > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > > >>> mount to its mount table - it can't actually access the mount).
> > > >>>
> > > >>> Here's an example using machines from the recent Bakeathon.
> > > >>>
> > > >>> Mounting a server with TLS enabled:
> > > >>>
> > > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> # umount /mnt
> > > >>>
> > > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > > >>>
> > > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > > >>>
> > > >>> Yet a v3 mount without "xprtsec=tls" works:
> > > >>>
> > > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> # umount /mnt
> > > >>>
> > > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> > > >>> v3 and also "works":
> > > >>>
> > > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> # grep ora /proc/mounts
> > > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > > >>>
> > > >>> Even though the filesystem is mounted, the client can't do anything with it:
> > > >>>
> > > >>> # ls /mnt
> > > >>> ls: cannot open directory '/mnt': Permission denied
> > > >>>
> > > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > > >>> The client compares that list with its own list of auth flavors parsed from the
> > > >>> mount request and returns -EACCES if no match is found (see
> > > >>> nfs_verify_authflavors()).
> > > >>>
> > > >>> Perhaps we should be doing something similar with xprtsec policies?
> > > >>
> > > >> The problem might be in how you've set up the exports. With NFSv3,
> > > >> the parent export needs the "crossmnt" export option in order for
> > > >> NFSv3 to behave like NFSv4 in this regard, although I could have
> > > >> missed something.
> > > >
> > > > I was mounting your server though :)
> > >
> > > OK, then not the same bug that Olga found last year.
> > >
> > > We should find out what FreeBSD does in this case.
> >
> > I thought about that. Rick's servers from the BAT are offline, and I
> > don't think he was exporting v3 anyway.
> >
> > >
> > >
> > > >>> Should
> > > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > > >>> pseudo-flavors corresponding to xprtsec policies?
> > > >>
> > > >> No. Transport-layer security is not an RPC security flavor or
> > > >> pseudo-flavor. These two things are not related.
> > > >>
> > > >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> > > >> but it was rejected).
> > > >
> > > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > > also lists TLS under the Flavor Name column I thought it might make
> > > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > > though they're not, if only to give the client a way to determine that
> > > > the mount should fail.
> > >
> > > RPC_AUTH_TLS is used only when a client probes a server to see if
> > > it supports RPC-with-TLS. At all other times, the client uses one
> > > of the normal, legitimate flavors. It does not represent a security
> > > flavor that can be used during regular operation.
> > >
> > > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> > >
> > > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > > list? I think clients that don't recognize it should ignore it,
> > > but I'm not sure. What should a client do if it sees that flavor in
> > > the list? It's not one that can be used for any other procedure than
> > > a NULL RPC.
> >
> > Maybe? After the client gets the filehandle it's calling FSINFO and
> > PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> > checking for a negative return code from the PATHCONF operation. If it
> > did, it could maybe use the -EACCES coupled with the knowledge that the
> > server had RPC_AUTH_TLS enabled to emit an error message saying to check
> > the xprtsec policies (but I don't think that would be as definitive as
> > what I had in mind) and to fail the mount.
>
> If Linux is the only implementation of NFSv3 with TLS so far, then
> we have some latitude for innovation.
>
> I would like to hear from the client maintainers about what they
> would prefer the client user experience to look like. Then NFSD's
> behavior can be adjusted to accommodate.
>
> In this case, Steve would have to sign off on an rpc.mountd change
> to return AUTH_TLS in the auth_flavors list.
Maybe instead of messing with the auth_flavors list, we just have the
client check the status from the PATHCONF operation and leave it at
that.
---8<---
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 44eca51b2808..09d28dae0f06 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -867,8 +867,10 @@ static int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *mntfh, str
pathinfo.fattr = fattr;
nfs_fattr_init(fattr);
- if (clp->rpc_ops->pathconf(server, mntfh, &pathinfo) >= 0)
- server->namelen = pathinfo.max_namelen;
+ error = clp->rpc_ops->pathconf(server, mntfh, &pathinfo);
+ if (error < 0)
+ return error;
+ server->namelen = pathinfo.max_namelen;
}
if (clp->rpc_ops->discover_trunking != NULL &&
---8<---
That's sufficient to make the mount fail with EACCES, which should be
enough of a clue for someone that they should compare the client's mount
options with the server's export options.
[root@rawhide-client ~]# mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
mount.nfs: access denied by server while mounting oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls
-Scott
>
>
> --
> Chuck Lever
>
On Fri, May 3, 2024 at 11:45 AM Chuck Lever <[email protected]> wrote:
>
> On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote:
> > On Thu, 02 May 2024, Chuck Lever III wrote:
> >
> > >
> > >
> > > > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> > > >
> > > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > > >
> > > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> > > >>>
> > > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > > >>> mount to its mount table - it can't actually access the mount).
> > > >>>
> > > >>> Here's an example using machines from the recent Bakeathon.
> > > >>>
> > > >>> Mounting a server with TLS enabled:
> > > >>>
> > > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> # umount /mnt
> > > >>>
> > > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > > >>>
> > > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > > >>>
> > > >>> Yet a v3 mount without "xprtsec=tls" works:
> > > >>>
> > > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> # umount /mnt
> > > >>>
> > > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> > > >>> v3 and also "works":
> > > >>>
> > > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > >>> # grep ora /proc/mounts
> > > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > > >>>
> > > >>> Even though the filesystem is mounted, the client can't do anything with it:
> > > >>>
> > > >>> # ls /mnt
> > > >>> ls: cannot open directory '/mnt': Permission denied
> > > >>>
> > > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > > >>> The client compares that list with its own list of auth flavors parsed from the
> > > >>> mount request and returns -EACCES if no match is found (see
> > > >>> nfs_verify_authflavors()).
> > > >>>
> > > >>> Perhaps we should be doing something similar with xprtsec policies?
> > > >>
> > > >> The problem might be in how you've set up the exports. With NFSv3,
> > > >> the parent export needs the "crossmnt" export option in order for
> > > >> NFSv3 to behave like NFSv4 in this regard, although I could have
> > > >> missed something.
> > > >
> > > > I was mounting your server though :)
> > >
> > > OK, then not the same bug that Olga found last year.
> > >
> > > We should find out what FreeBSD does in this case.
> >
> > I thought about that. Rick's servers from the BAT are offline, and I
> > don't think he was exporting v3 anyway.
If you want me to leave the FreeBSD server up on tailscale configured to
allow NFSv3/RPC-with-TLS mounts, just email.
I did a quick test using the FreeBSD client and the mount fails, but I know
that failure happens when it tries to access the server file system in
the kernel.
The replies fail with RPC Msg_denied.
(I've attached a packet trace, in case you are interested.)
I haven't done anything with the userspace mountd daemon, which means it
will allow a Mount protocol RPC without TLS and does not return any different
info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard,
but I have tried hard to avoid adding RPC-with-TLS to the userspace
RPC libraries.
(Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.)
> >
> > >
> > >
> > > >>> Should
> > > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > > >>> pseudo-flavors corresponding to xprtsec policies?
> > > >>
> > > >> No. Transport-layer security is not an RPC security flavor or
> > > >> pseudo-flavor. These two things are not related.
> > > >>
> > > >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> > > >> but it was rejected).
> > > >
> > > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > > also lists TLS under the Flavor Name column I thought it might make
> > > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > > though they're not, if only to give the client a way to determine that
> > > > the mount should fail.
> > >
> > > RPC_AUTH_TLS is used only when a client probes a server to see if
> > > it supports RPC-with-TLS. At all other times, the client uses one
> > > of the normal, legitimate flavors. It does not represent a security
> > > flavor that can be used during regular operation.
> > >
> > > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> > >
> > > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > > list? I think clients that don't recognize it should ignore it,
> > > but I'm not sure. What should a client do if it sees that flavor in
> > > the list? It's not one that can be used for any other procedure than
> > > a NULL RPC.
> >
> > Maybe? After the client gets the filehandle it's calling FSINFO and
> > PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> > checking for a negative return code from the PATHCONF operation. If it
> > did, it could maybe use the -EACCES coupled with the knowledge that the
> > server had RPC_AUTH_TLS enabled to emit an error message saying to check
> > the xprtsec policies (but I don't think that would be as definitive as
> > what I had in mind) and to fail the mount.
>
> If Linux is the only implementation of NFSv3 with TLS so far, then
> we have some latitude for innovation.
At this point, FreeBSD can change to whatever you guys think works best,
although I'd like to avoid adding RPC-with-TLS support to the Mount protocol.
Have fun with it, rick
>
> I would like to hear from the client maintainers about what they
> would prefer the client user experience to look like. Then NFSD's
> behavior can be adjusted to accommodate.
>
> In this case, Steve would have to sign off on an rpc.mountd change
> to return AUTH_TLS in the auth_flavors list.
>
>
> --
> Chuck Lever
>
On Fri, 03 May 2024, Rick Macklem wrote:
> On Fri, May 3, 2024 at 11:45 AM Chuck Lever <[email protected]> wrote:
> >
> > On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote:
> > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > >
> > > >
> > > >
> > > > > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> > > > >
> > > > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > > > >
> > > > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> > > > >>>
> > > > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > > > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > > > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > > > >>> mount to its mount table - it can't actually access the mount).
> > > > >>>
> > > > >>> Here's an example using machines from the recent Bakeathon.
> > > > >>>
> > > > >>> Mounting a server with TLS enabled:
> > > > >>>
> > > > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > >>> # umount /mnt
> > > > >>>
> > > > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > > > >>>
> > > > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > > > >>>
> > > > >>> Yet a v3 mount without "xprtsec=tls" works:
> > > > >>>
> > > > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > >>> # umount /mnt
> > > > >>>
> > > > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> > > > >>> v3 and also "works":
> > > > >>>
> > > > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > >>> # grep ora /proc/mounts
> > > > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > > > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > > > >>>
> > > > >>> Even though the filesystem is mounted, the client can't do anything with it:
> > > > >>>
> > > > >>> # ls /mnt
> > > > >>> ls: cannot open directory '/mnt': Permission denied
> > > > >>>
> > > > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > > > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > > > >>> The client compares that list with its own list of auth flavors parsed from the
> > > > >>> mount request and returns -EACCES if no match is found (see
> > > > >>> nfs_verify_authflavors()).
> > > > >>>
> > > > >>> Perhaps we should be doing something similar with xprtsec policies?
> > > > >>
> > > > >> The problem might be in how you've set up the exports. With NFSv3,
> > > > >> the parent export needs the "crossmnt" export option in order for
> > > > >> NFSv3 to behave like NFSv4 in this regard, although I could have
> > > > >> missed something.
> > > > >
> > > > > I was mounting your server though :)
> > > >
> > > > OK, then not the same bug that Olga found last year.
> > > >
> > > > We should find out what FreeBSD does in this case.
> > >
> > > I thought about that. Rick's servers from the BAT are offline, and I
> > > don't think he was exporting v3 anyway.
> If you want me to leave the FreeBSD server up on tailscale configured to
> allow NFSv3/RPC-with-TLS mounts, just email.
>
> I did a quick test using the FreeBSD client and the mount fails, but I know
> that failure happens when it tries to access the server file system in
> the kernel.
> The replies fail with RPC Msg_denied.
> (I've attached a packet trace, in case you are interested.)
Thanks, Rick. Looking at that trace, freebsd is rejecting the GETATTR
and FSINFO requests at the RPC layer, whereas Linux knfsd isn't.
Digging a little further, both nfsd3_proc_fsinfo() and nfsd3_proc_getattr()
are calling fh_verify() with a flag called NFSD_MAY_BYPASS_GSS_ON_ROOT
---8<---
resp->status = fh_verify(rqstp, &resp->fh, 0,
NFSD_MAY_NOP | NFSD_MAY_BYPASS_GSS_ON_ROOT);
---8<---
and fh_verify() has
---8<---
/*
* Clients may expect to be able to use auth_sys during mount,
* even if they use gss for everything else; see section 2.3.2
* of rfc 2623.
*/
if (access & NFSD_MAY_BYPASS_GSS_ON_ROOT
&& exp->ex_path.dentry == dentry)
goto skip_pseudoflavor_check;
error = check_nfsd_access(exp, rqstp);
---8<---
so it's skipping check_nfsd_access(), which is where the xprtsec policy
checking occurs.
Looking at RFC 2623 section 2.3.2, it basically says a server can choose
to not require authentication for GETATTR and FSINFO. Obviously, RFC
2623 predates RPC-with-TLS by 20+ years, so there's nothing there to
dictate what to do here... but as Chuck reiterated, TLS isn't a security
flavor so maybe knfsd shouldn't be bypassing those xprtsec policy checks
at all.
-Scott
>
> I haven't done anything with the userspace mountd daemon, which means it
> will allow a Mount protocol RPC without TLS and does not return any different
> info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard,
> but I have tried hard to avoid adding RPC-with-TLS to the userspace
> RPC libraries.
> (Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.)
>
> > >
> > > >
> > > >
> > > > >>> Should
> > > > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > > > >>> pseudo-flavors corresponding to xprtsec policies?
> > > > >>
> > > > >> No. Transport-layer security is not an RPC security flavor or
> > > > >> pseudo-flavor. These two things are not related.
> > > > >>
> > > > >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> > > > >> but it was rejected).
> > > > >
> > > > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > > > also lists TLS under the Flavor Name column I thought it might make
> > > > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > > > though they're not, if only to give the client a way to determine that
> > > > > the mount should fail.
> > > >
> > > > RPC_AUTH_TLS is used only when a client probes a server to see if
> > > > it supports RPC-with-TLS. At all other times, the client uses one
> > > > of the normal, legitimate flavors. It does not represent a security
> > > > flavor that can be used during regular operation.
> > > >
> > > > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> > > >
> > > > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > > > list? I think clients that don't recognize it should ignore it,
> > > > but I'm not sure. What should a client do if it sees that flavor in
> > > > the list? It's not one that can be used for any other procedure than
> > > > a NULL RPC.
> > >
> > > Maybe? After the client gets the filehandle it's calling FSINFO and
> > > PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> > > checking for a negative return code from the PATHCONF operation. If it
> > > did, it could maybe use the -EACCES coupled with the knowledge that the
> > > server had RPC_AUTH_TLS enabled to emit an error message saying to check
> > > the xprtsec policies (but I don't think that would be as definitive as
> > > what I had in mind) and to fail the mount.
> >
> > If Linux is the only implementation of NFSv3 with TLS so far, then
> > we have some latitude for innovation.
> At this point, FreeBSD can change to whatever you guys think works best,
> although I'd like to avoid adding RPC-with-TLS support to the Mount protocol.
>
> Have fun with it, rick
>
> >
> > I would like to hear from the client maintainers about what they
> > would prefer the client user experience to look like. Then NFSD's
> > behavior can be adjusted to accommodate.
> >
> > In this case, Steve would have to sign off on an rpc.mountd change
> > to return AUTH_TLS in the auth_flavors list.
> >
> >
> > --
> > Chuck Lever
> >
On Mon, May 6, 2024 at 9:47 AM Scott Mayhew <[email protected]> wrote:
>
> On Fri, 03 May 2024, Rick Macklem wrote:
>
> > On Fri, May 3, 2024 at 11:45 AM Chuck Lever <[email protected]> wrote:
> > >
> > > On Thu, May 02, 2024 at 02:51:23PM -0400, Scott Mayhew wrote:
> > > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > > >
> > > > >
> > > > >
> > > > > > On May 2, 2024, at 1:37 PM, Scott Mayhew <[email protected]> wrote:
> > > > > >
> > > > > > On Thu, 02 May 2024, Chuck Lever III wrote:
> > > > > >
> > > > > >>> On May 2, 2024, at 11:54 AM, Scott Mayhew <[email protected]> wrote:
> > > > > >>>
> > > > > >>> Red Hat QE identified an "interesting" issue with NFSv3 and TLS, in that an
> > > > > >>> NFSv3 client can mount with "xprtsec=none" a filesystem exported with
> > > > > >>> "xprtsec=tls:mtls" (in the sense that the client gets the filehandle and adds a
> > > > > >>> mount to its mount table - it can't actually access the mount).
> > > > > >>>
> > > > > >>> Here's an example using machines from the recent Bakeathon.
> > > > > >>>
> > > > > >>> Mounting a server with TLS enabled:
> > > > > >>>
> > > > > >>> # mount -o v4.2,sec=sys,xprtsec=tls oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > >>> # umount /mnt
> > > > > >>>
> > > > > >>> Trying to mount without "xprtsec=tls" shows that the filesystem is not exported with "xprtsec=none":
> > > > > >>>
> > > > > >>> # mount -o v4.2,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > >>> mount.nfs: Operation not permitted for oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls on /mnt
> > > > > >>>
> > > > > >>> Yet a v3 mount without "xprtsec=tls" works:
> > > > > >>>
> > > > > >>> # mount -o v3,sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > >>> # umount /mnt
> > > > > >>>
> > > > > >>> and a mount with no explicit version and without "xprtsec=tls" falls back to
> > > > > >>> v3 and also "works":
> > > > > >>>
> > > > > >>> # mount -o sec=sys oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt
> > > > > >>> # grep ora /proc/mounts
> > > > > >>> oracle-102.chuck.lever.oracle.com.nfsv4.dev:/export/tls /mnt nfs
> > > > > >>> +rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=100.64.0.49,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=100.64.0.49 0 0
> > > > > >>>
> > > > > >>> Even though the filesystem is mounted, the client can't do anything with it:
> > > > > >>>
> > > > > >>> # ls /mnt
> > > > > >>> ls: cannot open directory '/mnt': Permission denied
> > > > > >>>
> > > > > >>> When krb5 is used with NFSv3, the server returns a list of pseudoflavors in
> > > > > >>> mountres3_ok (https://datatracker.ietf.org/doc/html/rfc1813#section-5.2.1).
> > > > > >>> The client compares that list with its own list of auth flavors parsed from the
> > > > > >>> mount request and returns -EACCES if no match is found (see
> > > > > >>> nfs_verify_authflavors()).
> > > > > >>>
> > > > > >>> Perhaps we should be doing something similar with xprtsec policies?
> > > > > >>
> > > > > >> The problem might be in how you've set up the exports. With NFSv3,
> > > > > >> the parent export needs the "crossmnt" export option in order for
> > > > > >> NFSv3 to behave like NFSv4 in this regard, although I could have
> > > > > >> missed something.
> > > > > >
> > > > > > I was mounting your server though :)
> > > > >
> > > > > OK, then not the same bug that Olga found last year.
> > > > >
> > > > > We should find out what FreeBSD does in this case.
> > > >
> > > > I thought about that. Rick's servers from the BAT are offline, and I
> > > > don't think he was exporting v3 anyway.
> > If you want me to leave the FreeBSD server up on tailscale configured to
> > allow NFSv3/RPC-with-TLS mounts, just email.
> >
> > I did a quick test using the FreeBSD client and the mount fails, but I know
> > that failure happens when it tries to access the server file system in
> > the kernel.
> > The replies fail with RPC Msg_denied.
> > (I've attached a packet trace, in case you are interested.)
>
> Thanks, Rick. Looking at that trace, freebsd is rejecting the GETATTR
> and FSINFO requests at the RPC layer, whereas Linux knfsd isn't.
>
> Digging a little further, both nfsd3_proc_fsinfo() and nfsd3_proc_getattr()
> are calling fh_verify() with a flag called NFSD_MAY_BYPASS_GSS_ON_ROOT
>
> ---8<---
> resp->status = fh_verify(rqstp, &resp->fh, 0,
> NFSD_MAY_NOP | NFSD_MAY_BYPASS_GSS_ON_ROOT);
> ---8<---
>
> and fh_verify() has
>
> ---8<---
> /*
> * Clients may expect to be able to use auth_sys during mount,
> * even if they use gss for everything else; see section 2.3.2
> * of rfc 2623.
> */
> if (access & NFSD_MAY_BYPASS_GSS_ON_ROOT
> && exp->ex_path.dentry == dentry)
> goto skip_pseudoflavor_check;
>
> error = check_nfsd_access(exp, rqstp);
> ---8<---
>
> so it's skipping check_nfsd_access(), which is where the xprtsec policy
> checking occurs.
>
> Looking at RFC 2623 section 2.3.2, it basically says a server can choose
> to not require authentication for GETATTR and FSINFO.
My guess would be that this was done to allow root to do a mount without
having a TGT. I don't think the concept of a host/fqdn@realm keytab entry
was invented until NFSv4 came along.
> Obviously, RFC
> 2623 predates RPC-with-TLS by 20+ years, so there's nothing there to
> dictate what to do here... but as Chuck reiterated, TLS isn't a security
> flavor so maybe knfsd shouldn't be bypassing those xprtsec policy checks
> at all.
I'd agree with Chuck and do not see why a mount being done by root cannot
do GETATTR/FSINFO after doing a TLS handshake.
But, obviously, it is up to you guys.
Have fun with it, rick
>
> -Scott
>
>
> >
> > I haven't done anything with the userspace mountd daemon, which means it
> > will allow a Mount protocol RPC without TLS and does not return any different
> > info (such as a new pseudo flavour). Adding a pseudo-flavour wouldn't be hard,
> > but I have tried hard to avoid adding RPC-with-TLS to the userspace
> > RPC libraries.
> > (Mainly avoiding the need to link the userspace stuff to the OpenSSL libraries.)
> >
> > > >
> > > > >
> > > > >
> > > > > >>> Should
> > > > > >>> there be an errata to RFC 9289 and a request from IANA for assigned numbers for
> > > > > >>> pseudo-flavors corresponding to xprtsec policies?
> > > > > >>
> > > > > >> No. Transport-layer security is not an RPC security flavor or
> > > > > >> pseudo-flavor. These two things are not related.
> > > > > >>
> > > > > >> (And in fact, I proposed something like this for NFSv4 SECINFO,
> > > > > >> but it was rejected).
> > > > > >
> > > > > > I thought it might be a stretch to try to use mountres3.auth_flavors for
> > > > > > this, but since RFC 9289 does refer to AUTH_TLS as an authentication
> > > > > > flavor and https://www.iana.org/assignments/rpc-authentication-numbers/rpc-authentication-numbers.xhtml
> > > > > > also lists TLS under the Flavor Name column I thought it might make
> > > > > > sense to treat xprtsec policies as if they were pseudo-flavors even
> > > > > > though they're not, if only to give the client a way to determine that
> > > > > > the mount should fail.
> > > > >
> > > > > RPC_AUTH_TLS is used only when a client probes a server to see if
> > > > > it supports RPC-with-TLS. At all other times, the client uses one
> > > > > of the normal, legitimate flavors. It does not represent a security
> > > > > flavor that can be used during regular operation.
> > > > >
> > > > > NFSv3 mount failover logic is still open for discussion (ie, incomplete).
> > > > >
> > > > > Would it help if rpc.mountd stuck RPC_AUTH_TLS in the auth_flavors
> > > > > list? I think clients that don't recognize it should ignore it,
> > > > > but I'm not sure. What should a client do if it sees that flavor in
> > > > > the list? It's not one that can be used for any other procedure than
> > > > > a NULL RPC.
> > > >
> > > > Maybe? After the client gets the filehandle it's calling FSINFO and
> > > > PATHCONF. The latter get NFS3ERR_ACCES, but nfs_probe_fsinfo() isn't
> > > > checking for a negative return code from the PATHCONF operation. If it
> > > > did, it could maybe use the -EACCES coupled with the knowledge that the
> > > > server had RPC_AUTH_TLS enabled to emit an error message saying to check
> > > > the xprtsec policies (but I don't think that would be as definitive as
> > > > what I had in mind) and to fail the mount.
> > >
> > > If Linux is the only implementation of NFSv3 with TLS so far, then
> > > we have some latitude for innovation.
> > At this point, FreeBSD can change to whatever you guys think works best,
> > although I'd like to avoid adding RPC-with-TLS support to the Mount protocol.
> >
> > Have fun with it, rick
> >
> > >
> > > I would like to hear from the client maintainers about what they
> > > would prefer the client user experience to look like. Then NFSD's
> > > behavior can be adjusted to accommodate.
> > >
> > > In this case, Steve would have to sign off on an rpc.mountd change
> > > to return AUTH_TLS in the auth_flavors list.
> > >
> > >
> > > --
> > > Chuck Lever
> > >
>
>