We are starting to add some EL9 clients into the mix on our network. Autofs
mounts are working fine when initiated by a regular user, but they are failing
when initiated by root. This works fine from our EL8 clients.
Client:
kernel 5.14.0-362.18.1.el9_3.x86_64
nfs-utils-2.5.4-20.el9.x86_64
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
1 host/[email protected] (aes256-cts-hmac-sha1-96)
1 host/[email protected] (aes128-cts-hmac-sha1-96)
Server:
kernel 4.18.0-513.18.1.el8_9.x86_64
nfs-utils-2.3.3-59.el8.x86_64
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
1 host/[email protected] (aes256-cts-hmac-sha1-96)
1 host/[email protected] (aes128-cts-hmac-sha1-96)
1 nfs/[email protected] (aes256-cts-hmac-sha1-96)
1 nfs/[email protected] (aes128-cts-hmac-sha1-96)
Client rpc.gssd:
rpc.gssd[778]:
handle_gssd_upcall(0x7f15a0299840): 'mech=krb5
uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt3)
rpc.gssd[778]: start_upcall_thread(0x7f15a0299840): created thread id
0x7f159f1fe640
rpc.gssd[778]: krb5_use_machine_creds(0x7f159f1fe640): uid 0 tgtname (null)
rpc.gssd[778]: No key table entry found for [email protected] while getting
keytab entry for '[email protected]'
rpc.gssd[778]: No key table entry found for [email protected] while getting
keytab entry for '[email protected]'
rpc.gssd[778]: No key table entry found for root/[email protected] while
getting keytab entry for 'root/[email protected]'
rpc.gssd[778]: No key table entry found for nfs/[email protected] while
getting keytab entry for 'nfs/[email protected]'
rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
for 'host/[email protected]'
rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
server server.fqdn
rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
server [email protected]
rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
server [email protected]
rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
rpc.gssd[778]: WARNING: Machine cache prematurely expired or corrupted trying
to recreate cache for server server.fqdn
rpc.gssd[778]: No key table entry found for [email protected] while getting
keytab entry for '[email protected]'
rpc.gssd[778]: No key table entry found for [email protected] while getting
keytab entry for '[email protected]'
rpc.gssd[778]: No key table entry found for root/[email protected] while
getting keytab entry for 'root/[email protected]'
rpc.gssd[778]: No key table entry found for nfs/[email protected] while
getting keytab entry for 'nfs/[email protected]'
rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
for 'host/[email protected]'
rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
server server.fqdn
rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
server [email protected]
rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
server [email protected]
rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
rpc.gssd[778]: ERROR: Failed to create machine krb5 context with any
credentials cache for server server.fqdn
rpc.gssd[778]: do_error_downcall(0x7f159f1fe640): uid 0 err -13
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting
I don't seem to be getting any useful information from rpc.gssd on the server.
Please include me in replies.
--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of IT Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane [email protected]
Boulder, CO 80301 https://www.nwra.com/
On Thu, 29 Feb 2024, Orion Poplawski wrote:
> We are starting to add some EL9 clients into the mix on our network. Autofs
> mounts are working fine when initiated by a regular user, but they are failing
> when initiated by root. This works fine from our EL8 clients.
>
> Client:
> kernel 5.14.0-362.18.1.el9_3.x86_64
> nfs-utils-2.5.4-20.el9.x86_64
>
> Keytab name: FILE:/etc/krb5.keytab
> KVNO Principal
> ---- --------------------------------------------------------------------------
> 1 host/[email protected] (aes256-cts-hmac-sha1-96)
> 1 host/[email protected] (aes128-cts-hmac-sha1-96)
>
> Server:
> kernel 4.18.0-513.18.1.el8_9.x86_64
> nfs-utils-2.3.3-59.el8.x86_64
>
> Keytab name: FILE:/etc/krb5.keytab
> KVNO Principal
> ---- --------------------------------------------------------------------------
> 1 host/[email protected] (aes256-cts-hmac-sha1-96)
> 1 host/[email protected] (aes128-cts-hmac-sha1-96)
> 1 nfs/[email protected] (aes256-cts-hmac-sha1-96)
> 1 nfs/[email protected] (aes128-cts-hmac-sha1-96)
>
> Client rpc.gssd:
>
> rpc.gssd[778]:
> handle_gssd_upcall(0x7f15a0299840): 'mech=krb5
> uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt3)
> rpc.gssd[778]: start_upcall_thread(0x7f15a0299840): created thread id
> 0x7f159f1fe640
> rpc.gssd[778]: krb5_use_machine_creds(0x7f159f1fe640): uid 0 tgtname (null)
> rpc.gssd[778]: No key table entry found for [email protected] while getting
> keytab entry for '[email protected]'
> rpc.gssd[778]: No key table entry found for [email protected] while getting
> keytab entry for '[email protected]'
> rpc.gssd[778]: No key table entry found for root/[email protected] while
> getting keytab entry for 'root/[email protected]'
> rpc.gssd[778]: No key table entry found for nfs/[email protected] while
> getting keytab entry for 'nfs/[email protected]'
> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
> for 'host/[email protected]'
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
> server server.fqdn
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
> server [email protected]
> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
> server [email protected]
> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
> rpc.gssd[778]: WARNING: Machine cache prematurely expired or corrupted trying
> to recreate cache for server server.fqdn
> rpc.gssd[778]: No key table entry found for [email protected] while getting
> keytab entry for '[email protected]'
> rpc.gssd[778]: No key table entry found for [email protected] while getting
> keytab entry for '[email protected]'
> rpc.gssd[778]: No key table entry found for root/[email protected] while
> getting keytab entry for 'root/[email protected]'
> rpc.gssd[778]: No key table entry found for nfs/[email protected] while
> getting keytab entry for 'nfs/[email protected]'
> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
> for 'host/[email protected]'
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
> server server.fqdn
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
> server [email protected]
> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
> server [email protected]
> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
> rpc.gssd[778]: ERROR: Failed to create machine krb5 context with any
> credentials cache for server server.fqdn
> rpc.gssd[778]: do_error_downcall(0x7f159f1fe640): uid 0 err -13
>
> mount.nfs4: mount(2): Permission denied
> mount.nfs4: access denied by server while mounting
>
> I don't seem to be getting any useful information from rpc.gssd on the server.
>
> Please include me in replies.
I'm pretty sure this is the same issue that would be addressed by the
patches that I sent to the list yesterday would address:
https://lore.kernel.org/linux-nfs/[email protected]/T/#t
There's a couple things you could check.
1. On the NFS server, increase the rpc debug logging just a little:
# rpcdebug -m rpc -s auth
and then after a failure, look for lines like this in the journal:
Feb 29 18:14:44 rhel8.smayhew.test kernel: svc: svc_authenticate (6)
Feb 29 18:14:44 rhel8.smayhew.test kernel: gss_kerberos_mech: unsupported krb5 enctype 20
Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_import_sec_context_kerberos: returning -22
Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_delete_sec_context deleting 0000000090f401ca
2. On the NFS client, increase the rpc-verbosity in the gssd stanza in
/etc/nfs.conf (I have mine set to 3 but I think 1 would suffice) and
then 'systemctl restart rpc-gssd'.
then after a failure, look for a line like this in the journal:
Feb 29 18:14:44 rhel9.smayhew.test rpc.gssd[1700]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
If you see both of those, then it's almost certainly the same issue.
A quick solution would be to do this on your NFS server:
# echo "mac@Kerberos = -HMAC-SHA2-*" >/usr/share/crypto-policies/policies/modules/NFS.pmod
# update-crypto-policies --set DEFAULT:NFS
# systemctl restart gssproxy
but note that would be turning off the SHA2 enctypes for everything
krb5-related, not just NFS. Note you can always revert that later
simply by:
# update-crypto-policies --set DEFAULT
# systemctl restart gssproxy
Or, you could test the patches I sent to the list yesterday (this would
be on the client, not the server). The problem is those patches don't
apply cleanly to the current version of nfs-utils shipped in EL9. At a
quick glance, it looks like nfs-utils would need:
49567e7d configure: check for rpc_gss_seccreate
15cd5666 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials
2bfb59c6 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials
3abf6b52 gssd: switch to using rpc_gss_seccreate()
f05af7d9 gssd: revert commit 513630d720bd
20c07979 gssd: revert commit a5f3b7ccb01c
14ee4878 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials
4b272471 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials
75b04a9b gssd: fix handling DNS lookup failure
f066f87b gssd: enable forcing cred renewal using the keytab
and you'd also need to patch libtirpc to include:
22b1c0c gssapi: fix rpc_gss_seccreate passed in cred
-Scott
>
> --
> Orion Poplawski
> he/him/his - surely the least important thing about me
> Manager of IT Systems 720-772-5637
> NWRA, Boulder/CoRA Office FAX: 303-415-9702
> 3380 Mitchell Lane [email protected]
> Boulder, CO 80301 https://www.nwra.com/
On 2/29/24 16:52, Scott Mayhew wrote:
> On Thu, 29 Feb 2024, Orion Poplawski wrote:
>
>> We are starting to add some EL9 clients into the mix on our network. Autofs
>> mounts are working fine when initiated by a regular user, but they are failing
>> when initiated by root. This works fine from our EL8 clients.
>>
>> Client:
>> kernel 5.14.0-362.18.1.el9_3.x86_64
>> nfs-utils-2.5.4-20.el9.x86_64
>>
>> Keytab name: FILE:/etc/krb5.keytab
>> KVNO Principal
>> ---- --------------------------------------------------------------------------
>> 1 host/[email protected] (aes256-cts-hmac-sha1-96)
>> 1 host/[email protected] (aes128-cts-hmac-sha1-96)
>>
>> Server:
>> kernel 4.18.0-513.18.1.el8_9.x86_64
>> nfs-utils-2.3.3-59.el8.x86_64
>>
>> Keytab name: FILE:/etc/krb5.keytab
>> KVNO Principal
>> ---- --------------------------------------------------------------------------
>> 1 host/[email protected] (aes256-cts-hmac-sha1-96)
>> 1 host/[email protected] (aes128-cts-hmac-sha1-96)
>> 1 nfs/[email protected] (aes256-cts-hmac-sha1-96)
>> 1 nfs/[email protected] (aes128-cts-hmac-sha1-96)
>>
>> Client rpc.gssd:
>>
>> rpc.gssd[778]:
>> handle_gssd_upcall(0x7f15a0299840): 'mech=krb5
>> uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt3)
>> rpc.gssd[778]: start_upcall_thread(0x7f15a0299840): created thread id
>> 0x7f159f1fe640
>> rpc.gssd[778]: krb5_use_machine_creds(0x7f159f1fe640): uid 0 tgtname (null)
>> rpc.gssd[778]: No key table entry found for [email protected] while getting
>> keytab entry for '[email protected]'
>> rpc.gssd[778]: No key table entry found for [email protected] while getting
>> keytab entry for '[email protected]'
>> rpc.gssd[778]: No key table entry found for root/[email protected] while
>> getting keytab entry for 'root/[email protected]'
>> rpc.gssd[778]: No key table entry found for nfs/[email protected] while
>> getting keytab entry for 'nfs/[email protected]'
>> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
>> for 'host/[email protected]'
>> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
>> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
>> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
>> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
>> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
>> server server.fqdn
>> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
>> server [email protected]
>> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
>> server [email protected]
>> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
>> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
>> rpc.gssd[778]: WARNING: Machine cache prematurely expired or corrupted trying
>> to recreate cache for server server.fqdn
>> rpc.gssd[778]: No key table entry found for [email protected] while getting
>> keytab entry for '[email protected]'
>> rpc.gssd[778]: No key table entry found for [email protected] while getting
>> keytab entry for '[email protected]'
>> rpc.gssd[778]: No key table entry found for root/[email protected] while
>> getting keytab entry for 'root/[email protected]'
>> rpc.gssd[778]: No key table entry found for nfs/[email protected] while
>> getting keytab entry for 'nfs/[email protected]'
>> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
>> for 'host/[email protected]'
>> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
>> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
>> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
>> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar 1 10:39:34 2024
>> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
>> server server.fqdn
>> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
>> server [email protected]
>> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
>> server [email protected]
>> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
>> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
>> rpc.gssd[778]: ERROR: Failed to create machine krb5 context with any
>> credentials cache for server server.fqdn
>> rpc.gssd[778]: do_error_downcall(0x7f159f1fe640): uid 0 err -13
>>
>> mount.nfs4: mount(2): Permission denied
>> mount.nfs4: access denied by server while mounting
>>
>> I don't seem to be getting any useful information from rpc.gssd on the server.
>>
>> Please include me in replies.
>
> I'm pretty sure this is the same issue that would be addressed by the
> patches that I sent to the list yesterday would address:
> https://lore.kernel.org/linux-nfs/[email protected]/T/#t
>
> There's a couple things you could check.
>
> 1. On the NFS server, increase the rpc debug logging just a little:
> # rpcdebug -m rpc -s auth
>
> and then after a failure, look for lines like this in the journal:
> Feb 29 18:14:44 rhel8.smayhew.test kernel: svc: svc_authenticate (6)
> Feb 29 18:14:44 rhel8.smayhew.test kernel: gss_kerberos_mech: unsupported krb5 enctype 20
> Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_import_sec_context_kerberos: returning -22
> Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC: gss_delete_sec_context deleting 0000000090f401ca
>
> 2. On the NFS client, increase the rpc-verbosity in the gssd stanza in
> /etc/nfs.conf (I have mine set to 3 but I think 1 would suffice) and
> then 'systemctl restart rpc-gssd'.
>
> then after a failure, look for a line like this in the journal:
> Feb 29 18:14:44 rhel9.smayhew.test rpc.gssd[1700]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
>
> If you see both of those, then it's almost certainly the same issue.
Thank you! Yes, I see both of these messages. I guess this also explains why
I was able to mount from an EL7 server.
> A quick solution would be to do this on your NFS server:
>
> # echo "mac@Kerberos = -HMAC-SHA2-*" >/usr/share/crypto-policies/policies/modules/NFS.pmod
> # update-crypto-policies --set DEFAULT:NFS
> # systemctl restart gssproxy
>
> but note that would be turning off the SHA2 enctypes for everything
> krb5-related, not just NFS. Note you can always revert that later
> simply by:
>
> # update-crypto-policies --set DEFAULT
> # systemctl restart gssproxy
>
> Or, you could test the patches I sent to the list yesterday (this would
> be on the client, not the server). The problem is those patches don't
> apply cleanly to the current version of nfs-utils shipped in EL9. At a
> quick glance, it looks like nfs-utils would need:
>
> 49567e7d configure: check for rpc_gss_seccreate
> 15cd5666 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials
> 2bfb59c6 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials
> 3abf6b52 gssd: switch to using rpc_gss_seccreate()
> f05af7d9 gssd: revert commit 513630d720bd
> 20c07979 gssd: revert commit a5f3b7ccb01c
> 14ee4878 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials
> 4b272471 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials
> 75b04a9b gssd: fix handling DNS lookup failure
> f066f87b gssd: enable forcing cred renewal using the keytab
>
> and you'd also need to patch libtirpc to include:
>
> 22b1c0c gssapi: fix rpc_gss_seccreate passed in cred
I'll take a look at these more tomorrow. Since I see you have a @redhat.com
address, I am hoping that these fixes will flow down to EL9 sooner rather than
later. Is there an open issue I can subscribe to to follow along?
--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of IT Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane [email protected]
Boulder, CO 80301 https://www.nwra.com/
On 2/29/24 16:52, Scott Mayhew wrote:
> A quick solution would be to do this on your NFS server:
>
> # echo "mac@Kerberos = -HMAC-SHA2-*" >/usr/share/crypto-policies/policies/modules/NFS.pmod
> # update-crypto-policies --set DEFAULT:NFS
> # systemctl restart gssproxy
>
> but note that would be turning off the SHA2 enctypes for everything
> krb5-related, not just NFS.
That has worked well, thank you again!
> Or, you could test the patches I sent to the list yesterday (this would
> be on the client, not the server). The problem is those patches don't
> apply cleanly to the current version of nfs-utils shipped in EL9.
That's a bit much for me at the moment. Especially since the above workaround
is acceptable to me.
--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of IT Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane [email protected]
Boulder, CO 80301 https://www.nwra.com/