Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: Fedora 32 rpc.gssd misbehavior
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <0A4076D1-46BD-4B48-9667-60D679221210@oracle.com>
Date:   Mon, 10 Aug 2020 11:28:37 -0400
Cc:     Robbie Harwood <rharwood@redhat.com>,
        Jeff Layton <jlayton@redhat.com>,
        Bruce Fields <bfields@fieldses.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <47856272-28CE-4790-AD30-06B3A2A1A7C0@oracle.com>
References: <83856C49-309A-4AD6-9B27-9F93FDDE00DF@oracle.com>
 <48B9E144-41CA-4DF0-A88D-2F6652A0EBF1@oracle.com>
 <5ae72c7b0afa65d509db23686d72a1055f7cc6b4.camel@redhat.com>
 <jlg7dulylq6.fsf@redhat.com>
 <4EB4AE01-F6D4-4E8F-86BF-C8BB07E63517@oracle.com>
 <ee4b7c47bc37a53afd751159ae39d01d7cd3ee34.camel@redhat.com>
 <0A4076D1-46BD-4B48-9667-60D679221210@oracle.com>
To:     Simo Sorce <simo@redhat.com>, Steve Dickson <SteveD@redhat.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Jul 30, 2020, at 3:39 PM, Chuck Lever <chuck.lever@oracle.com> =
wrote:
>=20
>=20
>=20
>> On Jul 30, 2020, at 3:10 PM, Simo Sorce <simo@redhat.com> wrote:
>>=20
>> On Thu, 2020-07-30 at 13:59 -0400, Chuck Lever wrote:
>>>> On Jul 30, 2020, at 1:08 PM, Robbie Harwood <rharwood@redhat.com> =
wrote:
>>>>=20
>>>> Simo Sorce <simo@redhat.com> writes:
>>>>=20
>>>>> On Wed, 2020-07-29 at 14:27 -0400, Chuck Lever wrote:
>>>>>>> On Jul 29, 2020, at 1:19 PM, Chuck Lever =
<chuck.lever@oracle.com> wrote:
>>>>>>>=20
>>>>>>> Hi!
>>>>>>>=20
>>>>>>> I recently updated my test systems from EL7 to Fedora 32, and
>>>>>>> NFSv4.0 with Kerberos has stopped working.
>>>>>>>=20
>>>>>>> I mount with "klimt.ib" as before. The client workload stops
>>>>>>> dead when the server tries to perform its first CB_RECALL.
>>>>>>>=20
>>>>>>> I added some client instrumentation:
>>>>>>>=20
>>>>>>> kernel: NFSv4: Callback principal (nfs@klimt.ib.1015granger.net) =
does not match acceptor (nfs@klimt.ib).
>>>>>>> kernel: NFS: NFSv4 callback contains invalid cred
>>>>>>>=20
>>>>>>> I boosted gssd verbosity, and it says:
>>>>>>>=20
>>>>>>> rpc.gssd[986]: doing downcall: lifetime_rec=3D72226 =
acceptor=3Dnfs@klimt.ib
>>>>>>>=20
>>>>>>> But it knows the full hostname for the server:
>>>>>>>=20
>>>>>>> rpc.gssd[986]: Full hostname for 'klimt.ib' is =
'klimt.ib.1015granger.net'
>>>>>>>=20
>>>>>>>=20
>>>>>>> The acceptor appears to come from the Kerberos library. =
Shouldn't
>>>>>>> it be canonicalized? If so, should the Kerberos library do it, =
or
>>>>>>> should gssd? Since this behavior appeared after an upgrade, I
>>>>>>> suspect a Kerberos library regression. But it could be config-
>>>>>>> related, since both systems were re-imaged from the ground up.
>>>>>>>=20
>>>>>>> Also noticing some other problems on the server (missing =
hostname
>>>>>>> strings in debug messages, sssd_kcm infinite loops, and gssd
>>>>>>> sending garbage to the client after the NULL request that
>>>>>>> establishes the callback context).
>>>>>>>=20
>>>>>>> But let's look at the client acceptor problem first.
>>>>>>=20
>>>>>> I believe I found the problem.
>>>>>>=20
>>>>>> 8bffe8c5ec1a ("gssd: add /etc/nfs.conf support") added a number =
of gssd config
>>>>>> options to /etc/nfs.conf, including "avoid-dns". The default =
setting of avoid-
>>>>>> dns is 1. When I set this option on my client system explicitly =
to 0, NFSv4.0
>>>>>> with Kerberos works again.
>>>>>>=20
>>>>>> Is there a reason the default setting is 1?
>>>>>>=20
>>>>>=20
>>>>> Now that you mention DNS, this may be an interaction between a new
>>>>> default in Fedora 32 and how your environment is setup re DNS.
>>>>>=20
>>>>> In F32 we changed the option dns_canonicalize_hostname from 'true' =
to
>>>>> 'fallback'.
>>>>> This is a transitional state to eventually move it to 'false' at =
some
>>>>> point in the future.
>>>>>=20
>>>>> What it changes in practice is that it will first try the name =
passed
>>>>> in *as is* and only as a fallback try a CNAME if the name passed =
is not
>>>>> resolved as an A name. If you have principals in the KDC for both
>>>>> names, but you do not have keys in the keytab for both, you can =
have
>>>>> transitional issues.
>>>>>=20
>>>>> Additionally we discovered a bug that causes non qualified names =
to
>>>>> fail resolution with the 'fallback' option.
>>>>> If your name in the principal is really not qualified it will try =
to
>>>>> qualify it anyway, so if your principal is literally nfs/foo@FOO
>>>>> libgssapi may try to use nfs/foo.my.domdain@FOO, where "my.domain" =
is
>>>>> what is defined in resolv.conf search path.
>>>>>=20
>>>>> We are trying to address this regression.
>>>>>=20
>>>>> So try to set dns_canonicalize_hostname to true to see if that may
>>>>> influence your issue. If so, please let me know, as we still need =
to
>>>>> address this where possible.
>>>>=20
>>>> Also, please try setting `qualify_shortname =3D ""`.  (I did update =
the
>>>> config file we ship with Fedora, but upstream's default turns that =
on.
>>>> This is a temporary workaround while we merge something better
>>>> upstream.)
>>>=20
>>> For completeness, I tried:
>>>=20
>>> avoid-dns =3D 1
>>> dns_canonicalize_hostname =3D fallback
>>> qualify_shortname =3D ""
>>>=20
>>> which is the default configuration out of the shrink wrap.
>>>=20
>>> The workload hangs as before, and the acceptor is unqualified:
>>>=20
>>> rpc.gssd[985]: doing downcall: lifetime_rec=3D84046 =
acceptor=3Dnfs@klimt.ib
>>>=20
>>>=20
>>> The test is:
>>>=20
>>> Configured domain name is "1015granger.net"
>>>=20
>>> Fully-qualified client hostname is "manet.ib.granger.net"
>>>=20
>>> Fully-qualified server hostname is "klimt.ib.granger.net"
>>>=20
>>> mount command is "mount -o vers=3D4.0,sec=3Dsys klimt.ib:/export =
/mnt"
>>>=20
>>> In this case, both systems have keytabs and service principals, so
>>> the client automatically attempts to establish a GSS context for
>>> lease management and callback operations. The failure occurs because
>>> the server's principal is nfs@klimt.ib.1015granger.net but the
>>> acceptor now matches the server hostname from the mount command =
line,
>>> which is not always fully qualified.
>>=20
>> Ok, TBH I personally consider the syntax you  are currently using as
>> working by accident and that you should really sue the FQDN on the
>> command line (I assume it works that way, right?), however I =
understand
>> this is also technically a regression, that said I do not think we =
can
>> really fix this case because your "shortname" is not short (it has a
>> dot in it) so the heuristicts won't trigger to qualify it even when =
you
>> set qualify_shortname=3D"".
>>=20
>> I have the feeling we'll break this case, and our answer will have to
>> be "use the fqdn on the command line".
>=20
> See previous e-mail. Using the shrink wrap default settings, which
> includes qualify_shortname=3D"", results in a hang on callback, as
> originally observed.
>=20
> Users will notice this and complain: klimt.ib works for the NFSv3
> case and the NFSv4.1 case, and for NFSv4.0 when there is no keytab,
> but NFSv4.0,sec=3D* with a keytab eventually hangs.

I filed

https://bugzilla.redhat.com/show_bug.cgi?id=3D1867719

to document the behavior regression and some possible workarounds.
Further discussion about addressing the issue (possibly in nfs-utils)
can happen there.

Thanks!


--
Chuck Lever