Subject: CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

Hi there,

sorry for bothering here, but I did not fing anything useful on
Google... occasianally searching for serveral months.

I'm running a CentOs 8.2.2004 file server on KVM, kernel version
4.18.0-193.6.3.el8_2. That file server VM is part of a FreeeIPA domain.
Domain users (=extended family) can log in via SSH: this works very
reliable.

At home, I have set up some CentOS 8 clients, two for my kids and one
for myself. I'll focus on my own client for now the kids one's are the
same/similar. My personal client is a HP z420 with 12 cores and 64GB of
RAM, so pleanty of resources. Kernel 4.18.0-193.14-2. This machine has a
1GBit link to the file server.

I registered the client machine in the domain: id <domainuser> works
just fine: the userid and the associated groups are returned as would be
expected.

There is no clock skew between the client and the NFS & Kerberos
servers: I have a Stratum 1 Meinberg NTP appliance providing accurate
GPS time to the entire network.

I export the home directories on the NFS server to the local network
(rw,sec=krb5p,root_squash,async,subtree_check) and configured autofs on
the clients to mount the respective user home directory when a user logs
into a client:

/etc/auto.master:
/home /etc/auto.home --timeout=600

/etc/auto.home
* -rw,nfs4,nfsvers=4.1,soft fileserver:/home/&

When a user logs into a client, the user automatically gets a 24h
Kerberos ticket that can be displayed with klist.

Upon login, AutoFS does its thing by mounting the user's home directory
in the expected location. ID mappiung works as expected: ls -l therefore
shows usernames/groupnames instead of numeric UIDs/GIDs. Write access to
the exported FS works: For instance

touch test
rm test

in the user's home all just work fine.

BUT starting a normal GNOME session after accepting the login
credentials takes VERY long (2-3 minutes). Opening a simple GNOME
terminal takes long (~1 minute), starting Nautilus takes ages (firing it
up from the terminal until an empty window appear takes ~1 minute, then
I get a spinning wheel in the lower right corner saying "Loading" for
another couple of minutes before the home directory contents is
displayed in the window). Lauching programs in GNOME frequently makes
the mouse&keyboard freeze for several minutes.

SELinux does not seem to be the culprit: I tried with "setenforce 0" on
both the server and the client without success.

My feeling is that something with NFS is wrong here, but I don't see
what it might be. Both, the client and the server run on quite powerful
hardware, so I would expect sec=krb5p not to be an issue.

I have a few questions:

* What is the right approach to debug this?
* Are there any known issues with kerberized NFS and exported home
directories similar to what I described above?

Any pointers / help would be greatly appreciated.

--
Cheers,
Ray


2020-08-13 12:08:32

by Benjamin Coddington

[permalink] [raw]
Subject: Re: CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

On 12 Aug 2020, at 16:05, NFS Traumatized Linux User wrote:

> Hi there,
..
> Any pointers / help would be greatly appreciated.

Delays in the UI on the order of several minutes that are suspected to be
NFS related should be visible in a wirecapture. Have you tried using
vanilla krb5 and if the delays are still there, what does a wirecapture of
the NFS traffic look like? Are there long delays between each RPC call and
reply? That's the place I would start.

I can say that this problem sounds a bit like a name resolution problem
somewhere. Long delays that produce no errors at random times often turn
out to be systems repeatedly trying to use a name server that doesn't reply.
Perhaps either the client or server's name services are trying to look up an
identifier and there's a DNS server record that isn't replying - but that's
a wild guess.

Ben

2020-08-18 19:07:53

by J. Bruce Fields

[permalink] [raw]
Subject: Re: CentOS 8 + Kerberized NFS homes / AutoFS -> can't get this to run properly, many actions heavily delayed, freq. mouse/kbd freezes -> NFS issue?

On Wed, Aug 12, 2020 at 10:05:08PM +0200, NFS Traumatized Linux User wrote:
> I export the home directories on the NFS server to the local network
> (rw,sec=krb5p,root_squash,async,subtree_check) and configured autofs

Unrelated to your problem, but I wouldn't recommend "async".

Like Ben I'd start by watching the traffic in wireshark.

--b.