Hello everyone, I am stuck trying to figure out why I cannot get
sec=krb5 Linux clients working after upgrading from Ubuntu 10.04 LTS
(Lucid) to 12.04 (Precise)
I suspect the same problem is with the newer nfs-utils, but cannot be sure.
On the old (working) Lucid system, I think the important software is:
# dpkg -l |grep nfs-common
ii nfs-common 1:1.2.0-4ubuntu4.2 NFS
support files common to client and serve
# uname -a
Linux cselin3 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC
2011 x86_64 GNU/Linux
And on the newer (sec=krb5 mounts fail) system, the important software is:
# dpkg -l |grep nfs-common
ii nfs-common 1:1.2.5-3ubuntu3.1 NFS support files
common to client and server
# uname -a
Linux c27-00 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux
The NFS server we are using is a Hitachi BlueARC, and like I said, older
Linux clients work fine. After upgrading to new kernel and
nfs-utils, any attempt to mount yields an error:
# mount hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5
mount.nfs: access denied by server while mounting
hnas.engr.uconn.edu:/EngrUser/users/rohitm
I've reproduced the same behavior with both -t nfs4 and -t nfs. (Both
nfsv3 and nfsv4 work with kerberos security in our configuration with
Lucid, but not Precise) I've checked the Kerberos credential cache:
root@c27-00:~# klist -e -f -c /tmp/krb5cc_machine_ENGR.UCONN.EDU
Ticket cache: FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU
Default principal: nfs/[email protected]
Valid starting Expires Service principal
01/08/2013 15:40 02/08/2013 01:40 krbtgt/[email protected]
renew until 02/08/2013 15:40, Flags: FRI
Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
01/08/2013 15:40 02/08/2013 01:40 nfs/[email protected]
renew until 02/08/2013 15:40, Flags: FRT
Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
I also have rpc.idmapd and rpc.gssd running with extra verbosity. I
don't see anything blatantly wrong. This looks slightly suspicious:
Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating tcp client for server
hnas.engr.uconn.edu
Aug 1 16:32:50 c27-00 rpc.gssd[780]: DEBUG: port already set to 2049
Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating context with server
[email protected]
Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create krb5
context for user with uid 0 for server hnas.engr.uconn.edu
Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine
krb5 context with credentials cache
FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU for server hnas.engr.uconn.edu
Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine
krb5 context with any credentials cache for server hnas.engr.uconn.edu
Aug 1 16:32:50 c27-00 rpc.gssd[780]: doing error downcall
Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si
0x7fffdf0135b0 data 0x7fffdf013480
Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si
0x7fffdf0135b0 data 0x7fffdf013480
Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si
0x7fffdf0134f0 data 0x7fffdf0133c0
Aug 1 16:32:50 rpc.gssd[780]: last message repeated 4 times
Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client
/run/rpc_pipefs/nfs/clnt5
Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client
/run/rpc_pipefs/nfs/clnt4
I am able to successfuly get the nfs principal for the client from
/etc/krb5.keytab "nfs/c27-00.engr.uconn.edu" and I can see the principal
for the server "nfs/hnas.engr.uconn.edu" in cache
/tmp/krb5cc_machine_ENGR.UCONN.EDU.
I appreciate any advice or assistance. Thanks in advance!
Rohit
--
Rohit Mehta
Computer Engineer
University of Connecticut
Engineering Computing Services
371 Fairfield Road Unit 4031
Storrs, CT 06269-4031
Office: (860) 486 - 2331
Fax: (860) 486 - 1273
Thanks Simo, I apologize for the copy and paste error. You are
correct about it being "ldd /usr/sbin/rpc.gssd" command.
The actual error message we get is from the mount command:
mount.nfs: access denied by server while mounting
hnas.engr.uconn.edu:/EngrUser/users/rohitm
I got a little more output with mount -v
root@c27-00:~# mount -vvv hnas.engr.uconn.edu:/EngrUser/users/rohitm
/foo -o sec=krb5
mount: fstab path: "/etc/fstab"
mount: mtab path: "/etc/mtab"
mount: lock path: "/etc/mtab~"
mount: temp path: "/etc/mtab.tmp"
mount: UID: 0
mount: eUID: 0
mount: no type was given - I'll assume nfs because of the colon
mount: spec: "hnas.engr.uconn.edu:/EngrUser/users/rohitm"
mount: node: "/foo"
mount: types: "nfs"
mount: opts: "sec=krb5"
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] =
"hnas.engr.uconn.edu:/EngrUser/users/rohitm"
mount: external mount: argv[2] = "/foo"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sec=krb5"
mount.nfs: timeout set for Fri Aug 2 08:04:08 2013
mount.nfs: trying text-based options
'sec=krb5,vers=4,addr=137.99.203.4,clientaddr=137.99.2.29'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting
hnas.engr.uconn.edu:/EngrUser/users/rohitm
So there is unfortunately no more info available from mount command and
rpc.gssd output did not have any red flags for me. I'm trying to find
out if there is a way to get more information from the NFS server
itself, but as of now I'm not sure how to do that.
Thanks,
Rohit
On 08/01/2013 10:33 PM, Simo Sorce wrote:
> On Thu, 2013-08-01 at 21:20 -0400, Rohit Mehta wrote:
>> Thanks Simo, it doesn't look like libtirpc was available in previous
>> (10.04) release of Ubuntu.
>> root@c27-00:~# ldd /usr/sbin/
> I assume this had ^^ rpc.gssd at the end and is just a copy&paste error.
>
>> linux-vdso.so.1 => (0x00007fffeedff000)
>> libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
>> libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3
>> (0x00007fb0674c0000)
>> libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
>> (0x00007fb0672bb000)
>> libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1
>> (0x00007fb067093000)
>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
>> libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
>> (0x00007fb0668a7000)
>> libkrb5support.so.0 =>
>> /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
>> libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
>> (0x00007fb06649a000)
>> libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
>> (0x00007fb06627e000)
>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x00007fb066061000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
>> root@c27-00:~# locate libtirpc.so
>> /lib/x86_64-linux-gnu/libtirpc.so.1
>> /lib/x86_64-linux-gnu/libtirpc.so.1.0.10
>> root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
>> linux-vdso.so.1 => (0x00007fff01dc1000)
>> libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x00007fb701b37000)
>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
>> root@c27-00:~#
> Both are built against libgssglue so at least they are consistent, and
> that is not the source of the problem as I suspected.
>
> More information on the actual error would help.
>
> Simo.
>
>
>> On 08/01/2013 05:06 PM, Simo Sorce wrote:
>>> Rohit
>>> Was libtirpc also updated ?
>>> There has beena change recently where we eliminated the use of
>>> libgssglue and you will get issues if both nfs-utils and libtirpc are
>>> not compiled to use the same gssapi library.
>>>
>>> Please post ldd output on rpc.gssd and libtirpc.so to verify if this is
>>> the issue.
>>>
>>> Simo.
>>>
>>
>
--
Rohit Mehta
Computer Engineer
University of Connecticut
Engineering Computing Services
371 Fairfield Road Unit 2031
Storrs, CT 06269-2031
On Fri, 2013-08-02 at 08:15 -0400, Rohit Mehta wrote:
> Thanks Simo, I apologize for the copy and paste error. You are
> correct about it being "ldd /usr/sbin/rpc.gssd" command.
>
> The actual error message we get is from the mount command:
> mount.nfs: access denied by server while mounting
> hnas.engr.uconn.edu:/EngrUser/users/rohitm
>
> I got a little more output with mount -v
> root@c27-00:~# mount -vvv hnas.engr.uconn.edu:/EngrUser/users/rohitm
> /foo -o sec=krb5
> mount: fstab path: "/etc/fstab"
> mount: mtab path: "/etc/mtab"
> mount: lock path: "/etc/mtab~"
> mount: temp path: "/etc/mtab.tmp"
> mount: UID: 0
> mount: eUID: 0
> mount: no type was given - I'll assume nfs because of the colon
> mount: spec: "hnas.engr.uconn.edu:/EngrUser/users/rohitm"
> mount: node: "/foo"
> mount: types: "nfs"
> mount: opts: "sec=krb5"
> mount: external mount: argv[0] = "/sbin/mount.nfs"
> mount: external mount: argv[1] =
> "hnas.engr.uconn.edu:/EngrUser/users/rohitm"
> mount: external mount: argv[2] = "/foo"
> mount: external mount: argv[3] = "-v"
> mount: external mount: argv[4] = "-o"
> mount: external mount: argv[5] = "rw,sec=krb5"
> mount.nfs: timeout set for Fri Aug 2 08:04:08 2013
> mount.nfs: trying text-based options
> 'sec=krb5,vers=4,addr=137.99.203.4,clientaddr=137.99.2.29'
> mount.nfs: mount(2): Permission denied
> mount.nfs: access denied by server while mounting
> hnas.engr.uconn.edu:/EngrUser/users/rohitm
>
> So there is unfortunately no more info available from mount command and
> rpc.gssd output did not have any red flags for me. I'm trying to find
> out if there is a way to get more information from the NFS server
> itself, but as of now I'm not sure how to do that.
It looks like you have to look at the server side here.
Bump up rpc.svcgssd (or whatever is used on your server) debug and maybe
even the kernel log with the rpcdebug command and see why it is refusing
your authentication.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
On Thu, 2013-08-01 at 16:46 -0400, Rohit Kumar Mehta wrote:
> Hello everyone, I am stuck trying to figure out why I cannot get
> sec=krb5 Linux clients working after upgrading from Ubuntu 10.04 LTS
> (Lucid) to 12.04 (Precise)
>
> I suspect the same problem is with the newer nfs-utils, but cannot be sure.
>
> On the old (working) Lucid system, I think the important software is:
> # dpkg -l |grep nfs-common
> ii nfs-common 1:1.2.0-4ubuntu4.2 NFS
> support files common to client and serve
> # uname -a
> Linux cselin3 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC
> 2011 x86_64 GNU/Linux
>
> And on the newer (sec=krb5 mounts fail) system, the important software is:
> # dpkg -l |grep nfs-common
> ii nfs-common 1:1.2.5-3ubuntu3.1 NFS support files
> common to client and server
> # uname -a
> Linux c27-00 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC
> 2013 x86_64 x86_64 x86_64 GNU/Linux
>
>
> The NFS server we are using is a Hitachi BlueARC, and like I said, older
> Linux clients work fine. After upgrading to new kernel and
>
> nfs-utils, any attempt to mount yields an error:
> # mount hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5
> mount.nfs: access denied by server while mounting
> hnas.engr.uconn.edu:/EngrUser/users/rohitm
>
> I've reproduced the same behavior with both -t nfs4 and -t nfs. (Both
> nfsv3 and nfsv4 work with kerberos security in our configuration with
> Lucid, but not Precise) I've checked the Kerberos credential cache:
>
> root@c27-00:~# klist -e -f -c /tmp/krb5cc_machine_ENGR.UCONN.EDU
> Ticket cache: FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU
> Default principal: nfs/[email protected]
>
> Valid starting Expires Service principal
> 01/08/2013 15:40 02/08/2013 01:40 krbtgt/[email protected]
> renew until 02/08/2013 15:40, Flags: FRI
> Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
> 01/08/2013 15:40 02/08/2013 01:40 nfs/[email protected]
> renew until 02/08/2013 15:40, Flags: FRT
> Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
>
> I also have rpc.idmapd and rpc.gssd running with extra verbosity. I
> don't see anything blatantly wrong. This looks slightly suspicious:
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating tcp client for server
> hnas.engr.uconn.edu
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: DEBUG: port already set to 2049
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating context with server
> [email protected]
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create krb5
> context for user with uid 0 for server hnas.engr.uconn.edu
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine
> krb5 context with credentials cache
> FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU for server hnas.engr.uconn.edu
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine
> krb5 context with any credentials cache for server hnas.engr.uconn.edu
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: doing error downcall
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si
> 0x7fffdf0135b0 data 0x7fffdf013480
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si
> 0x7fffdf0135b0 data 0x7fffdf013480
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si
> 0x7fffdf0134f0 data 0x7fffdf0133c0
> Aug 1 16:32:50 rpc.gssd[780]: last message repeated 4 times
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client
> /run/rpc_pipefs/nfs/clnt5
> Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client
> /run/rpc_pipefs/nfs/clnt4
>
> I am able to successfuly get the nfs principal for the client from
> /etc/krb5.keytab "nfs/c27-00.engr.uconn.edu" and I can see the principal
> for the server "nfs/hnas.engr.uconn.edu" in cache
> /tmp/krb5cc_machine_ENGR.UCONN.EDU.
>
> I appreciate any advice or assistance. Thanks in advance!
> Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of
libgssglue and you will get issues if both nfs-utils and libtirpc are
not compiled to use the same gssapi library.
Please post ldd output on rpc.gssd and libtirpc.so to verify if this is
the issue.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
Thanks Simo, it doesn't look like libtirpc was available in previous
(10.04) release of Ubuntu.
root@c27-00:~# ldd /usr/sbin/
linux-vdso.so.1 => (0x00007fffeedff000)
libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3
(0x00007fb0674c0000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
(0x00007fb0672bb000)
libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1
(0x00007fb067093000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007fb0668a7000)
libkrb5support.so.0 =>
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007fb06649a000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007fb06627e000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb066061000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
root@c27-00:~# locate libtirpc.so
/lib/x86_64-linux-gnu/libtirpc.so.1
/lib/x86_64-linux-gnu/libtirpc.so.1.0.10
root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
linux-vdso.so.1 => (0x00007fff01dc1000)
libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb701b37000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
root@c27-00:~#
On 08/01/2013 05:06 PM, Simo Sorce wrote:
>
> Rohit
> Was libtirpc also updated ?
> There has beena change recently where we eliminated the use of
> libgssglue and you will get issues if both nfs-utils and libtirpc are
> not compiled to use the same gssapi library.
>
> Please post ldd output on rpc.gssd and libtirpc.so to verify if this is
> the issue.
>
> Simo.
>
--
Rohit Mehta
Computer Engineer
University of Connecticut
Engineering Computing Services
371 Fairfield Road Unit 2031
Storrs, CT 06269-2031
On Thu, 2013-08-01 at 21:20 -0400, Rohit Mehta wrote:
> Thanks Simo, it doesn't look like libtirpc was available in previous
> (10.04) release of Ubuntu.
> root@c27-00:~# ldd /usr/sbin/
I assume this had ^^ rpc.gssd at the end and is just a copy&paste error.
> linux-vdso.so.1 => (0x00007fffeedff000)
> libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
> libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3
> (0x00007fb0674c0000)
> libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
> (0x00007fb0672bb000)
> libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1
> (0x00007fb067093000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
> libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
> (0x00007fb0668a7000)
> libkrb5support.so.0 =>
> /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
> libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
> (0x00007fb06649a000)
> libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
> (0x00007fb06627e000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007fb066061000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
> root@c27-00:~# locate libtirpc.so
> /lib/x86_64-linux-gnu/libtirpc.so.1
> /lib/x86_64-linux-gnu/libtirpc.so.1.0.10
> root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
> linux-vdso.so.1 => (0x00007fff01dc1000)
> libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007fb701b37000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
> root@c27-00:~#
Both are built against libgssglue so at least they are consistent, and
that is not the source of the problem as I suspected.
More information on the actual error would help.
Simo.
> On 08/01/2013 05:06 PM, Simo Sorce wrote:
> >
> > Rohit
> > Was libtirpc also updated ?
> > There has beena change recently where we eliminated the use of
> > libgssglue and you will get issues if both nfs-utils and libtirpc are
> > not compiled to use the same gssapi library.
> >
> > Please post ldd output on rpc.gssd and libtirpc.so to verify if this is
> > the issue.
> >
> > Simo.
> >
>
>
--
Simo Sorce * Red Hat, Inc * New York