Greetings list,
I have been scouring the Web (and nfs-utils, kernel, and libmount source trees) for several days now to try to understand what happens during the mount procedure when the (NFSv4) share is authenticated by GSS (or rather, Kerberos). What I am trying to do is mount an NFS share as myself (a regular user) with my own Kerberos credentials. What I am seeing is an insistence on the part of some part of the system to populate the $RPC_PIPEFS/nfs/$CLIENT/krb5 pseudo-file with “mech=krb5 uid=0 service=* enctypes=…”, which then gets ferried on to rpc.gssd, which dutifully goes looking for machine credentials that do not exist. Instead (at least by my reading of the source code for what kind of outcome I want), that pseudo-file should say “mech=krb5 uid=1000 enctypes=…” (ie no service=…) etc. If it said that then rpc.gssd would (likely) do the right thing.
My question then: what is populating that pseudo-file in the rpc_pipefs filesystem? (and when is it doing it?) How come it insists on directing rpc.gssd to look for machine credentials for root instead of the uid of the caller (me)? I have been unable to locate any information on the role of rpc_pipefs beyond a blurb in the kernel source code, nor have I been able to locate anything that looks remotely like a protocol diagram for the NFSv4(+gss/krb5) mounting process, so I guess my question reduces to: where do I go looking for a solution to this problem?
(Note this is all recent Ubuntu, 20.04 and newer, and I already have Mac clients connecting to the server. More context and details here: https://askubuntu.com/questions/1382702/21-10-client-gssd-cant-seem-to-see-user-credentials-cache-when-mounting-nfsv4)
Thanks in advance for any insight,
--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com
> On Dec 23, 2021, at 5:27 PM, Dorian Taylor <[email protected]> wrote:
>
> Greetings list,
>
> I have been scouring the Web (and nfs-utils, kernel, and libmount source trees) for several days now to try to understand what happens during the mount procedure when the (NFSv4) share is authenticated by GSS (or rather, Kerberos). What I am trying to do is mount an NFS share as myself (a regular user) with my own Kerberos credentials. What I am seeing is an insistence on the part of some part of the system to populate the $RPC_PIPEFS/nfs/$CLIENT/krb5 pseudo-file with “mech=krb5 uid=0 service=* enctypes=…”, which then gets ferried on to rpc.gssd, which dutifully goes looking for machine credentials that do not exist. Instead (at least by my reading of the source code for what kind of outcome I want), that pseudo-file should say “mech=krb5 uid=1000 enctypes=…” (ie no service=…) etc. If it said that then rpc.gssd would (likely) do the right thing.
>
> My question then: what is populating that pseudo-file in the rpc_pipefs filesystem? (and when is it doing it?) How come it insists on directing rpc.gssd to look for machine credentials for root instead of the uid of the caller (me)? I have been unable to locate any information on the role of rpc_pipefs beyond a blurb in the kernel source code, nor have I been able to locate anything that looks remotely like a protocol diagram for the NFSv4(+gss/krb5) mounting process, so I guess my question reduces to: where do I go looking for a solution to this problem?
man 8 rpc.gssd
The "-n" option might be helpful.
> (Note this is all recent Ubuntu, 20.04 and newer, and I already have Mac clients connecting to the server. More context and details here: https://askubuntu.com/questions/1382702/21-10-client-gssd-cant-seem-to-see-user-credentials-cache-when-mounting-nfsv4)
--
Chuck Lever
> On Dec 24, 2021, at 12:28 PM, Chuck Lever III <[email protected]> wrote:
>
> man 8 rpc.gssd
>
> The "-n" option might be helpful.
Interesting, thanks. I tried it, but predictably it complained that my ccache was (correctly) owned by uid 1000, not 0. What I’m wondering about is why the uid in the $RPC_PIPEFS/nfs/$CLIENT/krb5 pseudofile is 0 when it should be 1000. Like I’m trying to determine if I have something misconfigured vs whether something is calling geteuid() when it should be calling getuid() (or whatever). Restating my situation:
* I run `kinit` as myself and get my TGT
* I run `mount -t nfs4 -o sec=krb5p host:/listed/in/fstab/with/user/flag /desired/target`
* I get EPERM with additional information saying the mount was denied by the server (actually a red herring; wireshark shows nothing coherent makes it to the server so the request is summarily denied)
* rpc.gssd -f -vvv shows that the failure is because it can’t find a keytab for various service principals
* problem is I am expecting it to use *my* *user* principal
* I know the mount should work because my Mac is already doing it; it’s the Linux client that’s failing
I have tracked the failure down as far as the pseudo-file $RPC_PIPEFS/nfs/$CLIENT/krb5 containing incorrect information (namely it says “mech=krb5 uid=0 service=*” where it should be saying “mech=krb5 uid=1000”). If that pseudo-file contained the correct information then by all accounts rpc.gssd should do the right thing. Thing is, I don’t know what (the kernel? something else?) populates that pseudo-file, and I have zero leads as to what creates it short of exhaustively poring over the kernel and nfs-utils (and other?) source code.
(I should also note that I have the debug output on rpc.idmapd also cranked up, and it does report that it interacts with that rpc_pipefs pseudo-filesystem but *not* that krb5 pseudo-file.)
(I even ran mount.nfs4 in gdb, but since it’s suid and because of that I have to run gdb as root, I’m necessarily going to miss the exact thing I’m looking for.)
I have two hypotheses:
* I have misconfigured something, but if I have, I can’t find the documentation needed to un-misconfigure it, because no documentation I have found so far mentions troubleshooting nfsv4+gss/krb5 with user (not machine) credentials, at least with non-root users (despite appearing to be supported in the source code)
* something is short-circuiting whatever mechanism is supposed to correctly report my (real) uid to whatever other mechanism makes it show up in that krb5 pseudo-file which subsequently directs rpc.gssd’s behaviour, and there isn’t a test case to catch it.
Either way, if there is a flow diagram kicking around showing all of the parts and pieces of the nfsv4+krb5 mounting process, I would be eternally grateful to see it.
--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com
> On Dec 24, 2021, at 2:15 PM, Dorian Taylor (Lists) <[email protected]> wrote:
>
>
>> On Dec 24, 2021, at 12:28 PM, Chuck Lever III <[email protected]> wrote:
>>
>> man 8 rpc.gssd
>>
>> The "-n" option might be helpful.
>
> Interesting, thanks. I tried it, but predictably it complained that my ccache was (correctly) owned by uid 1000, not 0. What I’m wondering about is why the uid in the $RPC_PIPEFS/nfs/$CLIENT/krb5 pseudofile is 0 when it should be 1000. Like I’m trying to determine if I have something misconfigured vs whether something is calling geteuid() when it should be calling getuid() (or whatever). Restating my situation:
>
> * I run `kinit` as myself and get my TGT
> * I run `mount -t nfs4 -o sec=krb5p host:/listed/in/fstab/with/user/flag /desired/target`
> * I get EPERM with additional information saying the mount was denied by the server (actually a red herring; wireshark shows nothing coherent makes it to the server so the request is summarily denied)
> * rpc.gssd -f -vvv shows that the failure is because it can’t find a keytab for various service principals
> * problem is I am expecting it to use *my* *user* principal
> * I know the mount should work because my Mac is already doing it; it’s the Linux client that’s failing
IIRC Linux requires that a mount operation be done by root. If you run gssd with "-n", become root, then kinit as yourself, I think it should work.
There has been some discussion about enabling a non-privileged user to perform a mount... it's a bit tricky because the function of mount is to alter the file namespace, which traditionally requires extra privilege to do.
Mac OS has had this functionality for ages to enable basic Finder operation. Linux doesn't have it yet.
> I have tracked the failure down as far as the pseudo-file $RPC_PIPEFS/nfs/$CLIENT/krb5 containing incorrect information (namely it says “mech=krb5 uid=0 service=*” where it should be saying “mech=krb5 uid=1000”). If that pseudo-file contained the correct information then by all accounts rpc.gssd should do the right thing. Thing is, I don’t know what (the kernel? something else?) populates that pseudo-file, and I have zero leads as to what creates it short of exhaustively poring over the kernel and nfs-utils (and other?) source code.
>
> (I should also note that I have the debug output on rpc.idmapd also cranked up, and it does report that it interacts with that rpc_pipefs pseudo-filesystem but *not* that krb5 pseudo-file.)
>
> (I even ran mount.nfs4 in gdb, but since it’s suid and because of that I have to run gdb as root, I’m necessarily going to miss the exact thing I’m looking for.)
>
> I have two hypotheses:
>
> * I have misconfigured something, but if I have, I can’t find the documentation needed to un-misconfigure it, because no documentation I have found so far mentions troubleshooting nfsv4+gss/krb5 with user (not machine) credentials, at least with non-root users (despite appearing to be supported in the source code)
>
> * something is short-circuiting whatever mechanism is supposed to correctly report my (real) uid to whatever other mechanism makes it show up in that krb5 pseudo-file which subsequently directs rpc.gssd’s behaviour, and there isn’t a test case to catch it.
AFAIK you are not doing anything wrong. It just isn't supported on Linux at this time.
> Either way, if there is a flow diagram kicking around showing all of the parts and pieces of the nfsv4+krb5 mounting process, I would be eternally grateful to see it.
I'm not aware of such a diagram.
--
Chuck Lever
> On Dec 25, 2021, at 5:53 PM, Chuck Lever III <[email protected]> wrote:
>
> IIRC Linux requires that a mount operation be done by root. If you run gssd with "-n", become root, then kinit as yourself, I think it should work.
>
> There has been some discussion about enabling a non-privileged user to perform a mount... it's a bit tricky because the function of mount is to alter the file namespace, which traditionally requires extra privilege to do.
>
> Mac OS has had this functionality for ages to enable basic Finder operation. Linux doesn't have it yet.
I mean, you’re the expert, though it looks a heck of a lot like the functionality is present: mount(8) is setuid (and so is mount.nfs), there is the `user` (and separate `users`) mount option in fstab, and I am pretty sure I have mounted things like optical drives and USB keys from a Linux desktop without e.g. entering a password (though I suppose that could have been the work of FUSE or GVFS or something).
> AFAIK you are not doing anything wrong. It just isn't supported on Linux at this time.
So, here is something interesting:
* I run `rpc.gssd -fn -vvv`
* in a root shell, I `kinit` as myself
* I `mount remotehome:/in/fstab`
* miraculously, that $RPC_PIPEFS/nfs/$CLIENT/krb5 pseudo-file now reports “mech=krb5 uid=1000…”
* after some carping about credential caches, rpc.gssd works correctly and the share is mounted as me.
I didn’t mount the NFS share though, root did, with my Kerberos TGT. From this I can deduce that that uid=1000 must be coming from rpc.idmapd, because where else could it come from? That’s the only thing that “knows” the relationship between the Kerberos principals and the user IDs on the system.
Moreover, what looks like what happened with the credential caches is that even though doing a kinit for my ticket as root, the ccache that rpc.gssd actually *used* to authenticate the share was a different one owned by me (uid 1000) that was also in /tmp at the time.
So /tmp/krb5cc_0 is used (presumably by rpc.idmapd) to find out that dorian@REALM -> 1000, but then /tmp/krb5cc_1000_XoaBV1 (by rpc.gssd) to actually authenticate the mount.
What this is telling me is that there is no reason in principle why a non-root user shouldn't be able to mount an NFSv4 share authenticated by GSS/Kerberos (as both `mount` and `mount.nfs` are setuid, and the fstab entry has `user` in the options; by all means the code to do the job sure looks like it’s in there), but rather the information about the *initial* mapping from Kerberos principal to system uid is not getting transmitted to rpc.gssd. To recap:
* when I run `mount remotehome:/in/fstab` as myself, gssd reports reading uid=0 in that pseudo-file;
* when I run `mount remotehome:/in/fstab` as root with my ticket, gssd reports uid=1000.
although, actually:
* it turns out that the only parts that matter are a) rpc.gssd -n, and b) root having a ccache with my ticket in it, in addition to me having one as well. I also appear not to need to be root to do the actual mount though, as `mount` is suid.
(aside: it looks like the `noresvport` mount option is ignored, as port <1024 is used to connect whether the real uid invoking `mount` is root or myself.)
This is looking more and more like a bug. I wonder what it would take to get it to work properly?
--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com
On Sat, Dec 25, 2021 at 10:53:33PM +0000, Chuck Lever III wrote:
> IIRC Linux requires that a mount operation be done by root. If you run
> gssd with "-n", become root, then kinit as yourself, I think it should
> work.
>
> There has been some discussion about enabling a non-privileged user to
> perform a mount... it's a bit tricky because the function of mount is
> to alter the file namespace, which traditionally requires extra
> privilege to do.
The core VFS code is quite happy to allow you to make unprivileged
mounts in your own namespace, but the particular filesystem being
mounted also gets a veto.
I think we're expecting NFS will be patched to allow unprivileged mounts
some time. See e.g.
https://lore.kernel.org/linux-nfs/[email protected]/
--b.
On Mon, 2022-01-03 at 16:32 -0500, J. Bruce Fields wrote:
> On Sat, Dec 25, 2021 at 10:53:33PM +0000, Chuck Lever III wrote:
> > IIRC Linux requires that a mount operation be done by root. If you
> > run
> > gssd with "-n", become root, then kinit as yourself, I think it
> > should
> > work.
> >
> > There has been some discussion about enabling a non-privileged user
> > to
> > perform a mount... it's a bit tricky because the function of mount
> > is
> > to alter the file namespace, which traditionally requires extra
> > privilege to do.
>
> The core VFS code is quite happy to allow you to make unprivileged
> mounts in your own namespace, but the particular filesystem being
> mounted also gets a veto.
>
> I think we're expecting NFS will be patched to allow unprivileged
> mounts
> some time. See e.g.
>
>
> https://lore.kernel.org/linux-nfs/[email protected]
> /
>
> --b.
As noted, the main issue is the bind() privileges needed for AUTH_SYS.
When using AUTH_GSS, the knfsd server doesn't care about the
originating port, which would allow unprivileged mounts to go ahead
provided that the user specifies the 'noresvport' mount option on the
client. Isn't that working?
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
On Mon, Jan 03, 2022 at 09:45:45PM +0000, Trond Myklebust wrote:
> On Mon, 2022-01-03 at 16:32 -0500, J. Bruce Fields wrote:
> > On Sat, Dec 25, 2021 at 10:53:33PM +0000, Chuck Lever III wrote:
> > > IIRC Linux requires that a mount operation be done by root. If you
> > > run
> > > gssd with "-n", become root, then kinit as yourself, I think it
> > > should
> > > work.
> > >
> > > There has been some discussion about enabling a non-privileged user
> > > to
> > > perform a mount... it's a bit tricky because the function of mount
> > > is
> > > to alter the file namespace, which traditionally requires extra
> > > privilege to do.
> >
> > The core VFS code is quite happy to allow you to make unprivileged
> > mounts in your own namespace, but the particular filesystem being
> > mounted also gets a veto.
> >
> > I think we're expecting NFS will be patched to allow unprivileged
> > mounts
> > some time. See e.g.
> >
> >
> > https://lore.kernel.org/linux-nfs/[email protected]
> > /
> >
> > --b.
>
> As noted, the main issue is the bind() privileges needed for AUTH_SYS.
>
> When using AUTH_GSS, the knfsd server doesn't care about the
> originating port, which would allow unprivileged mounts to go ahead
> provided that the user specifies the 'noresvport' mount option on the
> client. Isn't that working?
Oh, I remembered you'd said that was one of the issues, but didn't
understand that that was literally the only check remaining in the
code.... In which case, you could also test this by using setcap on
/usr/bin/mount or capsh to give the mount process CAP_NET_BIND_SERVICE?
(If you also set up the right namespaces first.)
--b.
On Mon, 2022-01-03 at 16:58 -0500, [email protected] wrote:
> On Mon, Jan 03, 2022 at 09:45:45PM +0000, Trond Myklebust wrote:
> > On Mon, 2022-01-03 at 16:32 -0500, J. Bruce Fields wrote:
> > > On Sat, Dec 25, 2021 at 10:53:33PM +0000, Chuck Lever III wrote:
> > > > IIRC Linux requires that a mount operation be done by root. If
> > > > you
> > > > run
> > > > gssd with "-n", become root, then kinit as yourself, I think it
> > > > should
> > > > work.
> > > >
> > > > There has been some discussion about enabling a non-privileged
> > > > user
> > > > to
> > > > perform a mount... it's a bit tricky because the function of
> > > > mount
> > > > is
> > > > to alter the file namespace, which traditionally requires extra
> > > > privilege to do.
> > >
> > > The core VFS code is quite happy to allow you to make
> > > unprivileged
> > > mounts in your own namespace, but the particular filesystem being
> > > mounted also gets a veto.
> > >
> > > I think we're expecting NFS will be patched to allow unprivileged
> > > mounts
> > > some time. See e.g.
> > >
> > >
> > > https://lore.kernel.org/linux-nfs/[email protected]
> > > /
> > >
> > > --b.
> >
> > As noted, the main issue is the bind() privileges needed for
> > AUTH_SYS.
> >
> > When using AUTH_GSS, the knfsd server doesn't care about the
> > originating port, which would allow unprivileged mounts to go ahead
> > provided that the user specifies the 'noresvport' mount option on
> > the
> > client. Isn't that working?
>
> Oh, I remembered you'd said that was one of the issues, but didn't
> understand that that was literally the only check remaining in the
> code.... In which case, you could also test this by using setcap on
> /usr/bin/mount or capsh to give the mount process
> CAP_NET_BIND_SERVICE?
> (If you also set up the right namespaces first.)
>
You'd have to give the container a CAP_NET_BIND_SERVICE privilege. With
docker, you can do that using the '--cap-add=NET_BIND_SERVICE' option.
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
Apologies, I had missed the additional activity on this topic.
TL;DR: `mount` of an nfsv4 volume as an unprivileged user works with Kerberos, for some definition of “works”, as long as root also has a copy of the unprivileged user’s TGT.
> On Jan 3, 2022, at 4:45 PM, Trond Myklebust <[email protected]> wrote:
>
> As noted, the main issue is the bind() privileges needed for AUTH_SYS.
>
> When using AUTH_GSS, the knfsd server doesn't care about the
> originating port, which would allow unprivileged mounts to go ahead
> provided that the user specifies the 'noresvport' mount option on the
> client. Isn't that working?
Indeed, mounting an nfs volume as an unprivileged user seems to work in principle; it looks like what’s happening is the mounting process fails to transmit (or fails to make use of a successful transmission of) the correct uid at some point along the line.
consider the relevant part of the server’s /etc/exports:
/srv/home/dorian *(rw,nohide,insecure,no_subtree_check,async,pnfs,sec=krb5p:krb5i:krb5:sys)
and the client’s /etc/fstab:
deuce:/home/dorian /net/dorian nfs4 sec=krb5p,soft,noresvport,noauto,user 0 0
Then the procedure:
* start rpc.gssd with -n to prevent it from looking for service principals
* `kinit` as myself, also `sudo kinit dorian` to get a ticket for me owned by root
* then just execute `mount deuce:/home/dorian` as myself.
Both my user and root have to have a TGT for dorian@REALM or it won’t work. Same with rpc.gssd suppressing service principals, but I suspect that is just incidental. It’s also worth noting that the connection appears to be unconditionally made on a privileged port even though `insecure` was set on the server and `noresvport` was specified on the client.
I am afraid I do not have any additional visibility into the mounting procedure save to note that that `krb5` pipefs pseudo-file looks like it’s getting its information from rpc.idmapd, which in turn is getting its own information from Kerberos principals. I don’t know how that pipefs mechanism works though. If I were to hazard a guess as to what was happening, it’s that mount.nfs(8) internals are losing track of the caller’s real uid when escalating privileges to do the mount. That is consistent with a) root also needing an unprivileged TGT and b) the client unconditionally connecting on port <1024.
--
Dorian Taylor
Make things. Make sense.
https://doriantaylor.com