Hi-
We've found an unexpected behavior with mount security
negotiation in the current Linux NFS client.
Given two real shares on an NFS server: one is a sys-only
share, and the other is a krb5-only share. When we try to
mount the sys-only share without specifying a sec= option,
it fails. Specifying sec=sys is successful.
What is seen on the wire:
1. The client attempts to access the pseudofs, and negotiates
krb5
2. The client walks down the pseudofs namespace to the
sys-only share
3. The client attempts to access the sys-only share with krb5
and gets WRONGSEC
4. The client negotiates sys, and continues setting up the
mount
5. nfs_fs_mount_common() invokes nfs_get_root(), but it
uses the pseudofs superblock, so it does a GETATTR on the
share's root directory with krb5, and that fails
At this point the client gives up, and the mount attempt
fails.
We could alter the server to allow a GETATTR with the
same flavor as the underlying directory. But seems like
the problem is on the client: it should use the negotiated
flavor that is appropriate to the share, not the flavor
appropriate for the pseudofs.
Any thoughts?
--
Chuck Lever
> We've found an unexpected behavior with mount security negotiation in the
> current Linux NFS client.
>
> Given two real shares on an NFS server: one is a sys-only share, and the
> other is a krb5-only share. When we try to mount the sys-only share
without
> specifying a sec= option, it fails. Specifying sec=sys is successful.
>
> What is seen on the wire:
>
> 1. The client attempts to access the pseudofs, and negotiates
> krb5
>
> 2. The client walks down the pseudofs namespace to the sys-only share
>
> 3. The client attempts to access the sys-only share with krb5 and gets
> WRONGSEC
>
> 4. The client negotiates sys, and continues setting up the mount
>
> 5. nfs_fs_mount_common() invokes nfs_get_root(), but it uses the
> pseudofs superblock, so it does a GETATTR on the share's root directory
with
> krb5, and that fails
>
> At this point the client gives up, and the mount attempt fails.
>
> We could alter the server to allow a GETATTR with the same flavor as the
> underlying directory. But seems like the problem is on the client: it
should
> use the negotiated flavor that is appropriate to the share, not the flavor
> appropriate for the pseudofs.
>
> Any thoughts?
Hmm, I thought maybe this was a scenario I had not tested, but I think I'm
misunderstanding the sequence. Could you summarize the ops and response for
each COMPOUND request?
I don't think the server should have to do anything special here.
Frank
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
On Sep 15, 2015, at 12:33 PM, Frank Filz <[email protected]> wrote:
>> We've found an unexpected behavior with mount security negotiation in the
>> current Linux NFS client.
>>
>> Given two real shares on an NFS server: one is a sys-only share, and the
>> other is a krb5-only share. When we try to mount the sys-only share
> without
>> specifying a sec= option, it fails. Specifying sec=sys is successful.
>>
>> What is seen on the wire:
>>
>> 1. The client attempts to access the pseudofs, and negotiates
>> krb5
>>
>> 2. The client walks down the pseudofs namespace to the sys-only share
>>
>> 3. The client attempts to access the sys-only share with krb5 and gets
>> WRONGSEC
>>
>> 4. The client negotiates sys, and continues setting up the mount
>>
>> 5. nfs_fs_mount_common() invokes nfs_get_root(), but it uses the
>> pseudofs superblock, so it does a GETATTR on the share's root directory
> with
>> krb5, and that fails
>>
>> At this point the client gives up, and the mount attempt fails.
>>
>> We could alter the server to allow a GETATTR with the same flavor as the
>> underlying directory. But seems like the problem is on the client: it
> should
>> use the negotiated flavor that is appropriate to the share, not the flavor
>> appropriate for the pseudofs.
>>
>> Any thoughts?
>
> Hmm, I thought maybe this was a scenario I had not tested, but I think I'm
> misunderstanding the sequence. Could you summarize the ops and response for
> each COMPOUND request?
There are two shares on the server.
The share being mounted is /export/CTHON. It is a sys-only
share. No "sec=" option is used on the mount command. The
expected outcome is that the mount will succeed and use
sec=sys.
The share that is not being mounted here is /export/KRB5.
It is shared with at least krb5i, which means the set of
flavors the server accepts for the pseudofs includes
both krb5i and sys.
The client has already negotiated krb5i to access the
pseudofs. I see a PUTROOTFH using krb5i and it is
successful. The negotiation is not in the trace I have.
I see a number of GETATTRs then a LOOKUP /export, all with
krb5i, all successful.
Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
Client recovers with SECINFO on /CTHON using krb5i. The
server responds with a list containing just AUTH_UNIX.
Client tries the LOOKUP /CTHON again, now with sys. It
works.
Similar GETATTR activity using sys as the client mounts
this share.
Then one last GETATTR, this time using krb5i. The FH is
the /export/CTHON directory. This fails with WRONGSEC.
The attr mask here is Supported Attrs, FH_Expire_type,
Link_Support, Symlink_Support, ACLSupport. This is
from nfs4_server_capabilities(), which is invoked only
in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
The next operation OTW is a RENEW that happens a minute
later.
--
Chuck Lever
> On Sep 15, 2015, at 12:33 PM, Frank Filz <[email protected]> wrote:
>
> >> We've found an unexpected behavior with mount security negotiation in
> >> the current Linux NFS client.
> >>
> >> Given two real shares on an NFS server: one is a sys-only share, and
> >> the other is a krb5-only share. When we try to mount the sys-only
> >> share
> > without
> >> specifying a sec= option, it fails. Specifying sec=sys is successful.
> >>
> >> What is seen on the wire:
> >>
> >> 1. The client attempts to access the pseudofs, and negotiates
> >> krb5
> >>
> >> 2. The client walks down the pseudofs namespace to the sys-only share
> >>
> >> 3. The client attempts to access the sys-only share with krb5 and
> >> gets WRONGSEC
> >>
> >> 4. The client negotiates sys, and continues setting up the mount
> >>
> >> 5. nfs_fs_mount_common() invokes nfs_get_root(), but it uses the
> >> pseudofs superblock, so it does a GETATTR on the share's root
> >> directory
> > with
> >> krb5, and that fails
> >>
> >> At this point the client gives up, and the mount attempt fails.
> >>
> >> We could alter the server to allow a GETATTR with the same flavor as
> >> the underlying directory. But seems like the problem is on the
> >> client: it
> > should
> >> use the negotiated flavor that is appropriate to the share, not the
> >> flavor appropriate for the pseudofs.
> >>
> >> Any thoughts?
> >
> > Hmm, I thought maybe this was a scenario I had not tested, but I think
> > I'm misunderstanding the sequence. Could you summarize the ops and
> > response for each COMPOUND request?
>
> There are two shares on the server.
>
> The share being mounted is /export/CTHON. It is a sys-only share. No
"sec="
> option is used on the mount command. The expected outcome is that the
> mount will succeed and use sec=sys.
>
> The share that is not being mounted here is /export/KRB5.
> It is shared with at least krb5i, which means the set of flavors the
server
> accepts for the pseudofs includes both krb5i and sys.
>
> The client has already negotiated krb5i to access the pseudofs. I see a
> PUTROOTFH using krb5i and it is successful. The negotiation is not in the
trace
> I have.
>
> I see a number of GETATTRs then a LOOKUP /export, all with krb5i, all
> successful.
>
> Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
>
> Client recovers with SECINFO on /CTHON using krb5i. The server responds
> with a list containing just AUTH_UNIX.
>
> Client tries the LOOKUP /CTHON again, now with sys. It works.
>
> Similar GETATTR activity using sys as the client mounts this share.
>
> Then one last GETATTR, this time using krb5i. The FH is the /export/CTHON
> directory. This fails with WRONGSEC.
>
> The attr mask here is Supported Attrs, FH_Expire_type, Link_Support,
> Symlink_Support, ACLSupport. This is from nfs4_server_capabilities(),
which
> is invoked only in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
>
> The next operation OTW is a RENEW that happens a minute later.
On Sep 15, 2015, at 1:41 PM, Frank Filz <[email protected]> wrote:
>> There are two shares on the server.
>>
>> The share being mounted is /export/CTHON. It is a sys-only share. No
> "sec="
>> option is used on the mount command. The expected outcome is that the
>> mount will succeed and use sec=sys.
>>
>> The share that is not being mounted here is /export/KRB5.
>> It is shared with at least krb5i, which means the set of flavors the
> server
>> accepts for the pseudofs includes both krb5i and sys.
>>
>> The client has already negotiated krb5i to access the pseudofs. I see a
>> PUTROOTFH using krb5i and it is successful. The negotiation is not in the
> trace
>> I have.
>>
>> I see a number of GETATTRs then a LOOKUP /export, all with krb5i, all
>> successful.
>>
>> Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
>>
>> Client recovers with SECINFO on /CTHON using krb5i. The server responds
>> with a list containing just AUTH_UNIX.
>>
>> Client tries the LOOKUP /CTHON again, now with sys. It works.
>>
>> Similar GETATTR activity using sys as the client mounts this share.
>>
>> Then one last GETATTR, this time using krb5i. The FH is the /export/CTHON
>> directory. This fails with WRONGSEC.
>>
>> The attr mask here is Supported Attrs, FH_Expire_type, Link_Support,
>> Symlink_Support, ACLSupport. This is from nfs4_server_capabilities(),
> which
>> is invoked only in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
>>
>> The next operation OTW is a RENEW that happens a minute later.
>
> From that sequence, I'm pretty sure we want to fix this in the client not
> the server.
>
> Hmm, did fsid change?
I'm not sure how to tell. The working LOOKUP operations on
parent/CTHON always return the same FSID.
> Does the mount work if sec=sys is specified?
Yes.
--
Chuck Lever
> On Sep 15, 2015, at 1:41 PM, Frank Filz <[email protected]> wrote:
>
> >> There are two shares on the server.
> >>
> >> The share being mounted is /export/CTHON. It is a sys-only share. No
> > "sec="
> >> option is used on the mount command. The expected outcome is that the
> >> mount will succeed and use sec=sys.
> >>
> >> The share that is not being mounted here is /export/KRB5.
> >> It is shared with at least krb5i, which means the set of flavors the
> > server
> >> accepts for the pseudofs includes both krb5i and sys.
> >>
> >> The client has already negotiated krb5i to access the pseudofs. I see
> >> a PUTROOTFH using krb5i and it is successful. The negotiation is not
> >> in the
> > trace
> >> I have.
> >>
> >> I see a number of GETATTRs then a LOOKUP /export, all with krb5i, all
> >> successful.
> >>
> >> Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
> >>
> >> Client recovers with SECINFO on /CTHON using krb5i. The server
> >> responds with a list containing just AUTH_UNIX.
> >>
> >> Client tries the LOOKUP /CTHON again, now with sys. It works.
> >>
> >> Similar GETATTR activity using sys as the client mounts this share.
> >>
> >> Then one last GETATTR, this time using krb5i. The FH is the
> >> /export/CTHON directory. This fails with WRONGSEC.
> >>
> >> The attr mask here is Supported Attrs, FH_Expire_type, Link_Support,
> >> Symlink_Support, ACLSupport. This is from nfs4_server_capabilities(),
> > which
> >> is invoked only in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
> >>
> >> The next operation OTW is a RENEW that happens a minute later.
> >
> > From that sequence, I'm pretty sure we want to fix this in the client
> > not the server.
> >
> > Hmm, did fsid change?
>
> I'm not sure how to tell. The working LOOKUP operations on parent/CTHON
> always return the same FSID.
I should have been more clear... Is the fsid for /export different from the
fsid for /export/CTHON? If it changes that should produce a new superblock.
> > Does the mount work if sec=sys is specified?
>
> Yes.
And with different mount options, a different superblock should definitely
be created.
I wonder if there is some way we can push up the security negotiation to
help identify a new superblock is necessary, equivalent to having a
different mount option?
Frank
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
On Sep 15, 2015, at 2:45 PM, Frank Filz <[email protected]> wrote:
>> On Sep 15, 2015, at 1:41 PM, Frank Filz <[email protected]> wrote:
>>
>>>> There are two shares on the server.
>>>>
>>>> The share being mounted is /export/CTHON. It is a sys-only share. No
>>> "sec="
>>>> option is used on the mount command. The expected outcome is that the
>>>> mount will succeed and use sec=sys.
>>>>
>>>> The share that is not being mounted here is /export/KRB5.
>>>> It is shared with at least krb5i, which means the set of flavors the
>>> server
>>>> accepts for the pseudofs includes both krb5i and sys.
>>>>
>>>> The client has already negotiated krb5i to access the pseudofs. I see
>>>> a PUTROOTFH using krb5i and it is successful. The negotiation is not
>>>> in the
>>> trace
>>>> I have.
>>>>
>>>> I see a number of GETATTRs then a LOOKUP /export, all with krb5i, all
>>>> successful.
>>>>
>>>> Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
>>>>
>>>> Client recovers with SECINFO on /CTHON using krb5i. The server
>>>> responds with a list containing just AUTH_UNIX.
>>>>
>>>> Client tries the LOOKUP /CTHON again, now with sys. It works.
>>>>
>>>> Similar GETATTR activity using sys as the client mounts this share.
>>>>
>>>> Then one last GETATTR, this time using krb5i. The FH is the
>>>> /export/CTHON directory. This fails with WRONGSEC.
>>>>
>>>> The attr mask here is Supported Attrs, FH_Expire_type, Link_Support,
>>>> Symlink_Support, ACLSupport. This is from nfs4_server_capabilities(),
>>> which
>>>> is invoked only in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
>>>>
>>>> The next operation OTW is a RENEW that happens a minute later.
>>>
>>> From that sequence, I'm pretty sure we want to fix this in the client
>>> not the server.
>>>
>>> Hmm, did fsid change?
>>
>> I'm not sure how to tell. The working LOOKUP operations on parent/CTHON
>> always return the same FSID.
>
> I should have been more clear... Is the fsid for /export different from the
> fsid for /export/CTHON? If it changes that should produce a new superblock.
The PUTROOTFH, GETATTR reply returns an FSID of
fsid4.major = 13172185619750249631
fsid4.minor = 0
The LOOKUP reply for /export returns an FSID of
fsid4.major = 5354380871752655416
fsid4.minor = 0
The LOOKUP reply for /export/CTHON returns an FSID of
fsid4.major = 5354380871752655416
fsid4.minor = 0
But I thought the security policy boundaries on the server
did not have to correspond with FSID boundaries.
> Does the mount work if sec=sys is specified?
>>
>> Yes.
>
> And with different mount options, a different superblock should definitely
> be created.
>
> I wonder if there is some way we can push up the security negotiation to
> help identify a new superblock is necessary, equivalent to having a
> different mount option?
The question, to me, is whether the client should use the
correct security flavor for the share right off the bat
(because the server has told it to use sys already), or
whether it should try to recover from the second WRONGSEC
instead of failing.
--
Chuck Lever
> >> On Sep 15, 2015, at 1:41 PM, Frank Filz <[email protected]>
wrote:
> >>
> >>>> There are two shares on the server.
> >>>>
> >>>> The share being mounted is /export/CTHON. It is a sys-only share.
> >>>> No
> >>> "sec="
> >>>> option is used on the mount command. The expected outcome is that
> >>>> the mount will succeed and use sec=sys.
> >>>>
> >>>> The share that is not being mounted here is /export/KRB5.
> >>>> It is shared with at least krb5i, which means the set of flavors
> >>>> the
> >>> server
> >>>> accepts for the pseudofs includes both krb5i and sys.
> >>>>
> >>>> The client has already negotiated krb5i to access the pseudofs. I
> >>>> see a PUTROOTFH using krb5i and it is successful. The negotiation
> >>>> is not in the
> >>> trace
> >>>> I have.
> >>>>
> >>>> I see a number of GETATTRs then a LOOKUP /export, all with krb5i,
> >>>> all successful.
> >>>>
> >>>> Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
> >>>>
> >>>> Client recovers with SECINFO on /CTHON using krb5i. The server
> >>>> responds with a list containing just AUTH_UNIX.
> >>>>
> >>>> Client tries the LOOKUP /CTHON again, now with sys. It works.
> >>>>
> >>>> Similar GETATTR activity using sys as the client mounts this share.
> >>>>
> >>>> Then one last GETATTR, this time using krb5i. The FH is the
> >>>> /export/CTHON directory. This fails with WRONGSEC.
> >>>>
> >>>> The attr mask here is Supported Attrs, FH_Expire_type,
> >>>> Link_Support, Symlink_Support, ACLSupport. This is from
> >>>> nfs4_server_capabilities(),
> >>> which
> >>>> is invoked only in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
> >>>>
> >>>> The next operation OTW is a RENEW that happens a minute later.
> >>>
> >>> From that sequence, I'm pretty sure we want to fix this in the
> >>> client not the server.
> >>>
> >>> Hmm, did fsid change?
> >>
> >> I'm not sure how to tell. The working LOOKUP operations on
> >> parent/CTHON always return the same FSID.
> >
> > I should have been more clear... Is the fsid for /export different
> > from the fsid for /export/CTHON? If it changes that should produce a new
> superblock.
>
> The PUTROOTFH, GETATTR reply returns an FSID of
>
> fsid4.major = 13172185619750249631
> fsid4.minor = 0
>
> The LOOKUP reply for /export returns an FSID of
>
> fsid4.major = 5354380871752655416
> fsid4.minor = 0
>
> The LOOKUP reply for /export/CTHON returns an FSID of
>
> fsid4.major = 5354380871752655416
> fsid4.minor = 0
>
> But I thought the security policy boundaries on the server did not have to
> correspond with FSID boundaries.
I think that's a reasonable expectation, though of course there's no
guarantee filehandle guessing couldn't be used to access files in a
different portion of the filesystem.
> > Does the mount work if sec=sys is specified?
> >>
> >> Yes.
> >
> > And with different mount options, a different superblock should
> > definitely be created.
> >
> > I wonder if there is some way we can push up the security negotiation
> > to help identify a new superblock is necessary, equivalent to having a
> > different mount option?
>
> The question, to me, is whether the client should use the correct security
> flavor for the share right off the bat (because the server has told it to
use sys
> already), or whether it should try to recover from the second WRONGSEC
> instead of failing.
I think the problem you are having is that the client is trying to re-use
the superblock because FSID and mount options are the same. The problem is
since the security policy is different, a second superblock is required in
this case. When security flavor is specified on mount, that forces a
different superblock because of the difference in mount options.
I think it should use the correct flavor, but maybe recovery from the
WRONGSEC is a path to creating a different superblock in this case?
Hmm, I think my original security negotiation testing might have been before
we added superblock sharing (back in 2006, and really mostly on RHEL 5 as my
testing was primarily in service of backporting security negotiation to RHEL
5), so I might not have run into this issue even if I had a test that would
trip over it now...
Frank
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
On Sep 15, 2015, at 4:28 PM, Frank Filz <[email protected]> wrote:
>>>> On Sep 15, 2015, at 1:41 PM, Frank Filz <[email protected]>
> wrote:
>>>>
>>>>>> There are two shares on the server.
>>>>>>
>>>>>> The share being mounted is /export/CTHON. It is a sys-only share.
>>>>>> No
>>>>> "sec="
>>>>>> option is used on the mount command. The expected outcome is that
>>>>>> the mount will succeed and use sec=sys.
>>>>>>
>>>>>> The share that is not being mounted here is /export/KRB5.
>>>>>> It is shared with at least krb5i, which means the set of flavors
>>>>>> the
>>>>> server
>>>>>> accepts for the pseudofs includes both krb5i and sys.
>>>>>>
>>>>>> The client has already negotiated krb5i to access the pseudofs. I
>>>>>> see a PUTROOTFH using krb5i and it is successful. The negotiation
>>>>>> is not in the
>>>>> trace
>>>>>> I have.
>>>>>>
>>>>>> I see a number of GETATTRs then a LOOKUP /export, all with krb5i,
>>>>>> all successful.
>>>>>>
>>>>>> Using krb5i, LOOKUP /CTHON, which fails with WRONGSEC.
>>>>>>
>>>>>> Client recovers with SECINFO on /CTHON using krb5i. The server
>>>>>> responds with a list containing just AUTH_UNIX.
>>>>>>
>>>>>> Client tries the LOOKUP /CTHON again, now with sys. It works.
>>>>>>
>>>>>> Similar GETATTR activity using sys as the client mounts this share.
>>>>>>
>>>>>> Then one last GETATTR, this time using krb5i. The FH is the
>>>>>> /export/CTHON directory. This fails with WRONGSEC.
>>>>>>
>>>>>> The attr mask here is Supported Attrs, FH_Expire_type,
>>>>>> Link_Support, Symlink_Support, ACLSupport. This is from
>>>>>> nfs4_server_capabilities(),
>>>>> which
>>>>>> is invoked only in nfs4_proc_get_rootfh() and nfs4_proc_get_root().
>>>>>>
>>>>>> The next operation OTW is a RENEW that happens a minute later.
>>>>>
>>>>> From that sequence, I'm pretty sure we want to fix this in the
>>>>> client not the server.
>>>>>
>>>>> Hmm, did fsid change?
>>>>
>>>> I'm not sure how to tell. The working LOOKUP operations on
>>>> parent/CTHON always return the same FSID.
>>>
>>> I should have been more clear... Is the fsid for /export different
>>> from the fsid for /export/CTHON? If it changes that should produce a new
>> superblock.
>>
>> The PUTROOTFH, GETATTR reply returns an FSID of
>>
>> fsid4.major = 13172185619750249631
>> fsid4.minor = 0
>>
>> The LOOKUP reply for /export returns an FSID of
>>
>> fsid4.major = 5354380871752655416
>> fsid4.minor = 0
>>
>> The LOOKUP reply for /export/CTHON returns an FSID of
>>
>> fsid4.major = 5354380871752655416
>> fsid4.minor = 0
>>
>> But I thought the security policy boundaries on the server did not have to
>> correspond with FSID boundaries.
>
> I think that's a reasonable expectation, though of course there's no
> guarantee filehandle guessing couldn't be used to access files in a
> different portion of the filesystem.
>
>>> Does the mount work if sec=sys is specified?
>>>>
>>>> Yes.
>>>
>>> And with different mount options, a different superblock should
>>> definitely be created.
>>>
>>> I wonder if there is some way we can push up the security negotiation
>>> to help identify a new superblock is necessary, equivalent to having a
>>> different mount option?
>>
>> The question, to me, is whether the client should use the correct security
>> flavor for the share right off the bat (because the server has told it to
> use sys
>> already), or whether it should try to recover from the second WRONGSEC
>> instead of failing.
>
> I think the problem you are having is that the client is trying to re-use
> the superblock because FSID and mount options are the same. The problem is
> since the security policy is different, a second superblock is required in
> this case. When security flavor is specified on mount, that forces a
> different superblock because of the difference in mount options.
As far as I can see, there are already two super blocks in play.
The first is for the pseudofs, which is using krb5i, and one is
for the real share, and that is using sys.
If nfs4_proc_get_root() used the super block for the real share,
the right OTW behavior would occur.
> I think it should use the correct flavor, but maybe recovery from the
> WRONGSEC is a path to creating a different superblock in this case?
>
> Hmm, I think my original security negotiation testing might have been before
> we added superblock sharing (back in 2006, and really mostly on RHEL 5 as my
> testing was primarily in service of backporting security negotiation to RHEL
> 5), so I might not have run into this issue even if I had a test that would
> trip over it now...
--
Chuck Lever