Hello,
I’m seeing two different behaviours between kernel NFS server versions in AlmaLinux 8 and Ubuntu 20. The following Perl demonstrates the issue:
--------
perl -MFile::Temp -Mautodie -Mstrict -e'my $fh = File::Temp::tempfile( DIR => "/the/nfs/mount" ); my $mailgid = getgrnam "mail"; my ($uid, $gid) = (getpwnam "bin")[2,3]; chown $uid, $gid, $fh; $) = "$gid $mailgid"; $> = $uid; chown -1, $mailgid, $fh'
--------
What this does, as root, is:
1) Creates a file under /mnt, then deletes it, leaving the Linux file descriptor open.
2) chowns the file to bin:bin.
3) Sets the process’s EUID & GUID to bin & bin/mail.
4) Does fchown( fd, -1, mailgid ).
When the server is AlmaLinux 8, the above works. When the server is Ubuntu 20, it fails with EPERM. (The client is AlmaLinux 8 in both cases.) Both are configured identically.
Does anyone know of anything that changed fairly recently in the kernel’s NFS server that might affect this? I’ve done a packet capture and confirmed that in both cases there’s an NFS SETATTR sent in an RPC 2.4 packet whose UID & GIDs match the process.
Thank you in advance!
cheers,
-Felipe
> On Jul 23, 2022, at 11:53 AM, Felipe Gasper <[email protected]> wrote:
>
> Hello,
>
> I’m seeing two different behaviours between kernel NFS server versions in AlmaLinux 8 and Ubuntu 20. The following Perl demonstrates the issue:
>
> --------
> perl -MFile::Temp -Mautodie -Mstrict -e'my $fh = File::Temp::tempfile( DIR => "/the/nfs/mount" ); my $mailgid = getgrnam "mail"; my ($uid, $gid) = (getpwnam "bin")[2,3]; chown $uid, $gid, $fh; $) = "$gid $mailgid"; $> = $uid; chown -1, $mailgid, $fh'
> --------
>
> What this does, as root, is:
>
> 1) Creates a file under /mnt, then deletes it, leaving the Linux file descriptor open.
>
> 2) chowns the file to bin:bin.
>
> 3) Sets the process’s EUID & GUID to bin & bin/mail.
>
> 4) Does fchown( fd, -1, mailgid ).
>
> When the server is AlmaLinux 8, the above works. When the server is Ubuntu 20, it fails with EPERM. (The client is AlmaLinux 8 in both cases.) Both are configured identically.
On each NFS sever, can you run 'uname -a' and show us the output?
On on the NFS client, can you show us the output of 'nfsstat -m'
during each test run?
> Does anyone know of anything that changed fairly recently in the kernel’s NFS server that might affect this? I’ve done a packet capture and confirmed that in both cases there’s an NFS SETATTR sent in an RPC 2.4 packet whose UID & GIDs match the process.
>
> Thank you in advance!
>
> cheers,
> -Felipe
--
Chuck Lever
> On Jul 25, 2022, at 09:23, Chuck Lever III <[email protected]> wrote:
>
>
>
>> On Jul 23, 2022, at 11:53 AM, Felipe Gasper <[email protected]> wrote:
>>
>> Hello,
>>
>> I’m seeing two different behaviours between kernel NFS server versions in AlmaLinux 8 and Ubuntu 20. The following Perl demonstrates the issue:
>>
>> --------
>> perl -MFile::Temp -Mautodie -Mstrict -e'my $fh = File::Temp::tempfile( DIR => "/the/nfs/mount" ); my $mailgid = getgrnam "mail"; my ($uid, $gid) = (getpwnam "bin")[2,3]; chown $uid, $gid, $fh; $) = "$gid $mailgid"; $> = $uid; chown -1, $mailgid, $fh'
>> --------
>>
>> What this does, as root, is:
>>
>> 1) Creates a file under /mnt, then deletes it, leaving the Linux file descriptor open.
>>
>> 2) chowns the file to bin:bin.
>>
>> 3) Sets the process’s EUID & GUID to bin & bin/mail.
>>
>> 4) Does fchown( fd, -1, mailgid ).
>>
>> When the server is AlmaLinux 8, the above works. When the server is Ubuntu 20, it fails with EPERM. (The client is AlmaLinux 8 in both cases.) Both are configured identically.
>
> On each NFS sever, can you run 'uname -a' and show us the output?
Ubuntu 20 (the “bad” one):
root@kvm-demo-support:~# uname -a
Linux kvm-demo-support 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
AlmaLinux 8 (the “good” one):
[root@10-2-71-6 ~]# uname -a
Linux 10-2-71-6.cprapid.com 4.18.0-372.16.1.el8_6.x86_64 #1 SMP Wed Jul 13 03:56:16 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> On on the NFS client, can you show us the output of 'nfsstat -m'
> during each test run?
> nfsstat -m
/mnt/phil from kvm-demo-support.dev.cpanel.net:/volumes/kvm-demo
Flags: rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.2.65.75,local_lock=none,addr=10.0.32.83
/mnt/felipe from 10.2.71.6:/home
Flags: rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.2.65.75,local_lock=none,addr=10.2.71.6
Thank you!
-FG
> On Jul 25, 2022, at 09:29, Felipe Gasper <[email protected]> wrote:
>
>
>
>> On Jul 25, 2022, at 09:23, Chuck Lever III <[email protected]> wrote:
>>
>>
>>
>>> On Jul 23, 2022, at 11:53 AM, Felipe Gasper <[email protected]> wrote:
>>>
>>> Hello,
>>>
>>> I’m seeing two different behaviours between kernel NFS server versions in AlmaLinux 8 and Ubuntu 20. The following Perl demonstrates the issue:
>>>
>>> --------
>>> perl -MFile::Temp -Mautodie -Mstrict -e'my $fh = File::Temp::tempfile( DIR => "/the/nfs/mount" ); my $mailgid = getgrnam "mail"; my ($uid, $gid) = (getpwnam "bin")[2,3]; chown $uid, $gid, $fh; $) = "$gid $mailgid"; $> = $uid; chown -1, $mailgid, $fh'
>>> --------
>>>
>>> What this does, as root, is:
>>>
>>> 1) Creates a file under /mnt, then deletes it, leaving the Linux file descriptor open.
>>>
>>> 2) chowns the file to bin:bin.
>>>
>>> 3) Sets the process’s EUID & GUID to bin & bin/mail.
>>>
>>> 4) Does fchown( fd, -1, mailgid ).
>>>
>>> When the server is AlmaLinux 8, the above works. When the server is Ubuntu 20, it fails with EPERM. (The client is AlmaLinux 8 in both cases.) Both are configured identically.
>>
>> On each NFS sever, can you run 'uname -a' and show us the output?
>
> Ubuntu 20 (the “bad” one):
> root@kvm-demo-support:~# uname -a
> Linux kvm-demo-support 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> AlmaLinux 8 (the “good” one):
> [root@10-2-71-6 ~]# uname -a
> Linux 10-2-71-6.cprapid.com 4.18.0-372.16.1.el8_6.x86_64 #1 SMP Wed Jul 13 03:56:16 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
>
>>
>> On on the NFS client, can you show us the output of 'nfsstat -m'
>> during each test run?
>
>> nfsstat -m
> /mnt/phil from kvm-demo-support.dev.cpanel.net:/volumes/kvm-demo
> Flags: rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.2.65.75,local_lock=none,addr=10.0.32.83
>
> /mnt/felipe from 10.2.71.6:/home
> Flags: rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.2.65.75,local_lock=none,addr=10.2.71.6
Update on this: it appears to have been a result of Ubuntu 20’s /etc/default/nfs-kernel-server’s containing `RPCMOUNTDOPTS="--manage-gids"`. Once I commented that line out the issue went away. I also confirmed that AlmaLinux 8 doesn’t enable `--manage-gids`. The varlink traffic in the Ubuntu server to/from rpc.mountd showed that it was asking systemd for the user’s groups and getting an empty response.
Sorry for the noise, and thank you for your time.
cheers,
-Felipe
On Sun, 24 Jul 2022, Felipe Gasper wrote:
> Hello,
>
> I’m seeing two different behaviours between kernel NFS server versions in AlmaLinux 8 and Ubuntu 20. The following Perl demonstrates the issue:
>
> --------
> perl -MFile::Temp -Mautodie -Mstrict -e'my $fh = File::Temp::tempfile( DIR => "/the/nfs/mount" ); my $mailgid = getgrnam "mail"; my ($uid, $gid) = (getpwnam "bin")[2,3]; chown $uid, $gid, $fh; $) = "$gid $mailgid"; $> = $uid; chown -1, $mailgid, $fh'
> --------
>
> What this does, as root, is:
>
> 1) Creates a file under /mnt, then deletes it, leaving the Linux file descriptor open.
>
> 2) chowns the file to bin:bin.
>
> 3) Sets the process’s EUID & GUID to bin & bin/mail.
>
> 4) Does fchown( fd, -1, mailgid ).
>
> When the server is AlmaLinux 8, the above works. When the server is Ubuntu 20, it fails with EPERM. (The client is AlmaLinux 8 in both cases.) Both are configured identically.
>
> Does anyone know of anything that changed fairly recently in the kernel’s NFS server that might affect this? I’ve done a packet capture and confirmed that in both cases there’s an NFS SETATTR sent in an RPC 2.4 packet whose UID & GIDs match the process.
>
Is mountd on Ubuntu running with "--manage-gids"?? And is mountd on
AlmaLinux running without that flag?
That would explain the difference.
NeilBrown
> On Jul 26, 2022, at 01:02, NeilBrown <[email protected]> wrote:
>
> On Sun, 24 Jul 2022, Felipe Gasper wrote:
>> Hello,
>>
>> I’m seeing two different behaviours between kernel NFS server versions in AlmaLinux 8 and Ubuntu 20. The following Perl demonstrates the issue:
>>
>> --------
>> perl -MFile::Temp -Mautodie -Mstrict -e'my $fh = File::Temp::tempfile( DIR => "/the/nfs/mount" ); my $mailgid = getgrnam "mail"; my ($uid, $gid) = (getpwnam "bin")[2,3]; chown $uid, $gid, $fh; $) = "$gid $mailgid"; $> = $uid; chown -1, $mailgid, $fh'
>> --------
>>
>> What this does, as root, is:
>>
>> 1) Creates a file under /mnt, then deletes it, leaving the Linux file descriptor open.
>>
>> 2) chowns the file to bin:bin.
>>
>> 3) Sets the process’s EUID & GUID to bin & bin/mail.
>>
>> 4) Does fchown( fd, -1, mailgid ).
>>
>> When the server is AlmaLinux 8, the above works. When the server is Ubuntu 20, it fails with EPERM. (The client is AlmaLinux 8 in both cases.) Both are configured identically.
>>
>> Does anyone know of anything that changed fairly recently in the kernel’s NFS server that might affect this? I’ve done a packet capture and confirmed that in both cases there’s an NFS SETATTR sent in an RPC 2.4 packet whose UID & GIDs match the process.
>>
>
> Is mountd on Ubuntu running with "--manage-gids"?? And is mountd on
> AlmaLinux running without that flag?
>
> That would explain the difference.
Hi Neil,
That is precisely the difference—thank you!
cheers,
-Felipe