Received-SPF: pass (google.com: domain of linux-nfs+bounces-2128-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Thu, 29 Feb 2024 18:52:22 -0500
From: Scott Mayhew <smayhew@redhat.com>
To: Orion Poplawski <orion@nwra.com>
Cc: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: Cannot initiate mount with sec=krb5 as root from EL9 clients
Message-ID: <ZeEYttbCcrqTUO9z@aion>
References: <8b16410b-6b2b-406e-a669-41abee137932@nwra.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8b16410b-6b2b-406e-a669-41abee137932@nwra.com>

On Thu, 29 Feb 2024, Orion Poplawski wrote:

> We are starting to add some EL9 clients into the mix on our network.  Autofs
> mounts are working fine when initiated by a regular user, but they are failing
> when initiated by root.  This works fine from our EL8 clients.
> 
> Client:
> kernel 5.14.0-362.18.1.el9_3.x86_64
> nfs-utils-2.5.4-20.el9.x86_64
> 
> Keytab name: FILE:/etc/krb5.keytab
> KVNO Principal
> ---- --------------------------------------------------------------------------
>    1 host/client.fqdn@NWRA.COM (aes256-cts-hmac-sha1-96)
>    1 host/client.fqdn@NWRA.COM (aes128-cts-hmac-sha1-96)
> 
> Server:
> kernel 4.18.0-513.18.1.el8_9.x86_64
> nfs-utils-2.3.3-59.el8.x86_64
> 
> Keytab name: FILE:/etc/krb5.keytab
> KVNO Principal
> ---- --------------------------------------------------------------------------
>    1 host/server.fqdn@NWRA.COM (aes256-cts-hmac-sha1-96)
>    1 host/server.fqdn@NWRA.COM (aes128-cts-hmac-sha1-96)
>    1 nfs/server.fqdn@NWRA.COM (aes256-cts-hmac-sha1-96)
>    1 nfs/server.fqdn@NWRA.COM (aes128-cts-hmac-sha1-96)
> 
> Client rpc.gssd:
> 
> rpc.gssd[778]:
>                                handle_gssd_upcall(0x7f15a0299840): 'mech=krb5
> uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt3)
> rpc.gssd[778]: start_upcall_thread(0x7f15a0299840): created thread id
> 0x7f159f1fe640
> rpc.gssd[778]: krb5_use_machine_creds(0x7f159f1fe640): uid 0 tgtname (null)
> rpc.gssd[778]: No key table entry found for client$@NWRA.COM while getting
> keytab entry for 'client$@NWRA.COM'
> rpc.gssd[778]: No key table entry found for CLIENT$@NWRA.COM while getting
> keytab entry for 'SRV-MRY01$@NWRA.COM'
> rpc.gssd[778]: No key table entry found for root/client.fqdn@NWRA.COM while
> getting keytab entry for 'root/client.fqdn@NWRA.COM'
> rpc.gssd[778]: No key table entry found for nfs/client.fqdn@NWRA.COM while
> getting keytab entry for 'nfs/client.fqdn@NWRA.COM'
> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
> for 'host/client.fqdn@NWRA.COM'
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar  1 10:39:34 2024
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar  1 10:39:34 2024
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
> server server.fqdn
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
> server nfs@server.fqdn
> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
> server nfs@server.fqdn
> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
> rpc.gssd[778]: WARNING: Machine cache prematurely expired or corrupted trying
> to recreate cache for server server.fqdn
> rpc.gssd[778]: No key table entry found for client$@NWRA.COM while getting
> keytab entry for 'client$@NWRA.COM'
> rpc.gssd[778]: No key table entry found for CLIENT$@NWRA.COM while getting
> keytab entry for 'SRV-MRY01$@NWRA.COM'
> rpc.gssd[778]: No key table entry found for root/client.fqdn@NWRA.COM while
> getting keytab entry for 'root/client.fqdn@NWRA.COM'
> rpc.gssd[778]: No key table entry found for nfs/client.fqdn@NWRA.COM while
> getting keytab entry for 'nfs/client.fqdn@NWRA.COM'
> rpc.gssd[778]: find_keytab_entry(0x7f159f1fe640): Success getting keytab entry
> for 'host/client.fqdn@NWRA.COM'
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar  1 10:39:34 2024
> rpc.gssd[778]: gssd_get_single_krb5_cred(0x7f159f1fe640): Credentials in CC
> 'FILE:/tmp/krb5ccmachine_NWRA.COM' are good until Fri Mar  1 10:39:34 2024
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating tcp client for
> server server.fqdn
> rpc.gssd[778]: create_auth_rpc_client(0x7f159f1fe640): creating context with
> server nfs@server.fqdn
> rpc.gssd[778]: WARNING: Failed to create krb5 context for user with uid 0 for
> server nfs@server.fqdn
> rpc.gssd[778]: WARNING: Failed to create machine krb5 context with cred cache
> FILE:/tmp/krb5ccmachine_NWRA.COM for server server.fqdn
> rpc.gssd[778]: ERROR: Failed to create machine krb5 context with any
> credentials cache for server server.fqdn
> rpc.gssd[778]: do_error_downcall(0x7f159f1fe640): uid 0 err -13
> 
> mount.nfs4: mount(2): Permission denied
> mount.nfs4: access denied by server while mounting
> 
> I don't seem to be getting any useful information from rpc.gssd on the server.
> 
> Please include me in replies.

I'm pretty sure this is the same issue that would be addressed by the
patches that I sent to the list yesterday would address:
https://lore.kernel.org/linux-nfs/20240228222253.1080880-1-smayhew@redhat.com/T/#t

There's a couple things you could check.

1. On the NFS server, increase the rpc debug logging just a little:
# rpcdebug -m rpc -s auth

and then after a failure, look for lines like this in the journal:
Feb 29 18:14:44 rhel8.smayhew.test kernel: svc: svc_authenticate (6)
Feb 29 18:14:44 rhel8.smayhew.test kernel: gss_kerberos_mech: unsupported krb5 enctype 20
Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC:       gss_import_sec_context_kerberos: returning -22
Feb 29 18:14:44 rhel8.smayhew.test kernel: RPC:       gss_delete_sec_context deleting 0000000090f401ca

2. On the NFS client, increase the rpc-verbosity in the gssd stanza in
/etc/nfs.conf (I have mine set to 3 but I think 1 would suffice) and
then 'systemctl restart rpc-gssd'.

then after a failure, look for a line like this in the journal:
Feb 29 18:14:44 rhel9.smayhew.test rpc.gssd[1700]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer

If you see both of those, then it's almost certainly the same issue.

A quick solution would be to do this on your NFS server:

# echo "mac@Kerberos = -HMAC-SHA2-*" >/usr/share/crypto-policies/policies/modules/NFS.pmod
# update-crypto-policies --set DEFAULT:NFS
# systemctl restart gssproxy

but note that would be turning off the SHA2 enctypes for everything
krb5-related, not just NFS.  Note you can always revert that later
simply by:

# update-crypto-policies --set DEFAULT
# systemctl restart gssproxy

Or, you could test the patches I sent to the list yesterday (this would
be on the client, not the server).  The problem is those patches don't
apply cleanly to the current version of nfs-utils shipped in EL9.  At a
quick glance, it looks like nfs-utils would need:

49567e7d configure: check for rpc_gss_seccreate
15cd5666 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials
2bfb59c6 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials
3abf6b52 gssd: switch to using rpc_gss_seccreate()
f05af7d9 gssd: revert commit 513630d720bd
20c07979 gssd: revert commit a5f3b7ccb01c
14ee4878 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for user credentials
4b272471 gssd: handle KRB5_AP_ERR_BAD_INTEGRITY for machine credentials
75b04a9b gssd: fix handling DNS lookup failure
f066f87b gssd: enable forcing cred renewal using the keytab

and you'd also need to patch libtirpc to include:

22b1c0c gssapi: fix rpc_gss_seccreate passed in cred

-Scott

> 
> -- 
> Orion Poplawski
> he/him/his  - surely the least important thing about me
> Manager of IT Systems                      720-772-5637
> NWRA, Boulder/CoRA Office             FAX: 303-415-9702
> 3380 Mitchell Lane                       orion@nwra.com
> Boulder, CO 80301                 https://www.nwra.com/