Hi-
Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
mounts where the client is not running rpc.gssd.
After reverting the "use krb5i for SETCLIENTID" patch, I've added
the AUTH_SYS fallback in the EACCES case in
nfs4_discover_server_trunking(). I'm not sure whether we need to
supplement what's there now, or replace it.
"case -ENOKEY:" is added so the kernel will recognize that when gssd
is changed to return that instead of EACCES in this case. If the
second patch is appled to 3.7 stable and following, it might be a way
to address the same regression in older kernels.
I've been focused on another bug this week, so this has seen very
light testing only. Looking for comments.
---
Chuck Lever (2):
NFS: Revert commit 4edaa308 and follow-on fixes
NFS: Fall back to AUTH_SYS for SETCLIENTID (take 2)
fs/nfs/nfs4client.c | 4 +--
fs/nfs/nfs4state.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++----
2 files changed, 61 insertions(+), 8 deletions(-)
--
Chuck Lever
On May 15, 2013, at 12:40 PM, "Myklebust, Trond" <[email protected]> wrote:
> On Wed, 2013-05-15 at 12:30 -0400, Chuck Lever wrote:
>> On May 15, 2013, at 12:24 PM, "Myklebust, Trond" <[email protected]> wrote:
>>
>>> On Wed, 2013-05-15 at 12:22 -0400, Chuck Lever wrote:
>>>> On May 15, 2013, at 12:18 PM, "Myklebust, Trond" <[email protected]> wrote:
>>>>
>>>>> On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
>>>>>> Hi-
>>>>>>
>>>>>> Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
>>>>>> mounts where the client is not running rpc.gssd.
>>>>>>
>>>>>> After reverting the "use krb5i for SETCLIENTID" patch, I've added
>>>>>> the AUTH_SYS fallback in the EACCES case in
>>>>>> nfs4_discover_server_trunking(). I'm not sure whether we need to
>>>>>> supplement what's there now, or replace it.
>>>>>>
>>>>>> "case -ENOKEY:" is added so the kernel will recognize that when gssd
>>>>>> is changed to return that instead of EACCES in this case. If the
>>>>>> second patch is appled to 3.7 stable and following, it might be a way
>>>>>> to address the same regression in older kernels.
>>>>>>
>>>>>> I've been focused on another bug this week, so this has seen very
>>>>>> light testing only. Looking for comments.
>>>>>
>>>>> I'd like to propose a different approach: we can set up rpc_pipefs files
>>>>> clnt/gssd and clnt/krb5 as "honeypots" that rpc.gssd will connect to,
>>>>> but that won't do any upcalls. When gssd connects, we set a
>>>>> per-rpc_net_ns variable that tells us 'gssd' is up and running. That
>>>>> variable only gets cleared if we see a timeout.
>>>>
>>>> Note my solution is a short term gap filler. Bruce and Jeff seem to want something that can fix current kernels without requiring user space changes, and I need something that will allow sec=krb5 mounts to work without a client keytab on kernels since 3.7.
>>>>
>>>> I see your proposal as a long term fix, and not something that we can expect to apply without deploying gssd support at the same time.
>>>
>>> How does it require gssd modifications?
>>>
>>> The whole point is that it requires kernel-only changes, and only minor
>>> changes at that...
>>
>> You'll have to be more specific then. The impression I was left with last week was that this solution was a non-starter because one of the two end points wipes all the directories at certain times.
>>
> No. The problem was the gssd behaviour when it receives a directory
> notification due to a client creation/destroy event: it disconnects from
> all rpc pipes, and then reconnects to them.
>
> That is solved by using the strategy that the variable is set on the
> first connection by gssd, and is only cleared if we see a timeout. It
> means that we can quickly detect whether or not gssd has been started
> (which is what we need here).
It seems to me we want to leave that per-ns flag set. An upcall timeout can result even when gssd is running, so should we consider a timeout reason to clear the flag?
Are you trying to detect the case where gssd was started but then is stopped (due to administrative action or daemon crash)?
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Wed, 2013-05-15 at 12:22 -0400, Chuck Lever wrote:
> On May 15, 2013, at 12:18 PM, "Myklebust, Trond" <[email protected]> wrote:
>
> > On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
> >> Hi-
> >>
> >> Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
> >> mounts where the client is not running rpc.gssd.
> >>
> >> After reverting the "use krb5i for SETCLIENTID" patch, I've added
> >> the AUTH_SYS fallback in the EACCES case in
> >> nfs4_discover_server_trunking(). I'm not sure whether we need to
> >> supplement what's there now, or replace it.
> >>
> >> "case -ENOKEY:" is added so the kernel will recognize that when gssd
> >> is changed to return that instead of EACCES in this case. If the
> >> second patch is appled to 3.7 stable and following, it might be a way
> >> to address the same regression in older kernels.
> >>
> >> I've been focused on another bug this week, so this has seen very
> >> light testing only. Looking for comments.
> >
> > I'd like to propose a different approach: we can set up rpc_pipefs files
> > clnt/gssd and clnt/krb5 as "honeypots" that rpc.gssd will connect to,
> > but that won't do any upcalls. When gssd connects, we set a
> > per-rpc_net_ns variable that tells us 'gssd' is up and running. That
> > variable only gets cleared if we see a timeout.
>
> Note my solution is a short term gap filler. Bruce and Jeff seem to want something that can fix current kernels without requiring user space changes, and I need something that will allow sec=krb5 mounts to work without a client keytab on kernels since 3.7.
>
> I see your proposal as a long term fix, and not something that we can expect to apply without deploying gssd support at the same time.
How does it require gssd modifications?
The whole point is that it requires kernel-only changes, and only minor
changes at that...
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
On Wed, 2013-05-15 at 15:52 -0400, J. Bruce Fields wrote:
> On Wed, May 15, 2013 at 03:28:27PM -0400, Chuck Lever wrote:
> >
> > On May 15, 2013, at 1:48 PM, J. Bruce Fields <[email protected]> wrote:
> >
> > > On Wed, May 15, 2013 at 01:42:58PM -0400, Chuck Lever wrote:
> > >>
> > >> On May 15, 2013, at 1:39 PM, "J. Bruce Fields" <[email protected]> wrote:
> > >>
> > >>> By the way, have you looked at SP4_MACH_CRED at all yet? It's a
> > >>> selfish question (I could use something to test against), but I
> > >>> think it's also what you want if you want krb5i-protected 4.1 state.
> > >>
> > >> I asked about that recently and was told SP4_MACH_CRED was going the
> > >> way of the do do
> > >
> > > Do you remember who said that? Is the discussion on line somewhere?
> >
> > No, I mis-remembered. I was thinking of SP4_SSV.
> >
> > >
> > >> (or did I misunderstand the response from the floor?).
> > >>
> > >> I'm certainly open to exploring other solutions, but I do want to be
> > >> practical about it. Will it be supported on other servers besides
> > >> Linux? Does SP4_MACH_CRED help for NFSv4.0?
> > >
> > > I haven't tested other servers. It's a 4.1-only feature.
> >
> > SP4_MACH_CRED for 4.1 appears useful, but I think we would need to consider:
> >
> > o whether SP4_MACH_CRED is a broadly implemented feature where Linux
> > clients can rely on it being there in typical environments
>
> I'm assuming it's mandatory for servers to implement. If there is any
> example of a released server not implementing SP4_MACH_CRED, I'd like to
> know.
>
> > o how to address the "no keytab" issue for NFSv4.0, which does not
> > have SP4_MACH_CRED (that I am aware of)
> >
> > Andy is probably more interested in seeing SP4_MACH_CRED implemented in the Linux client, as it is one solution for the "user cred expired while there is still dirty data in the client's page cache" problem, I think.
>
> OK.
>
> The ability to perform writes using the machine credential is built on
> top of SP4_MACH_CRED, but is optional for servers to support.
Right, but given that they solve a real problem with secure NFS, I
expect that most servers will implement it eventually. For that reason,
we will be implementing it on the client. I just need to get the basic
NFSv4.1 state management stuff debugged first.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
On Wed, May 15, 2013 at 03:28:27PM -0400, Chuck Lever wrote:
>
> On May 15, 2013, at 1:48 PM, J. Bruce Fields <[email protected]> wrote:
>
> > On Wed, May 15, 2013 at 01:42:58PM -0400, Chuck Lever wrote:
> >>
> >> On May 15, 2013, at 1:39 PM, "J. Bruce Fields" <[email protected]> wrote:
> >>
> >>> By the way, have you looked at SP4_MACH_CRED at all yet? It's a
> >>> selfish question (I could use something to test against), but I
> >>> think it's also what you want if you want krb5i-protected 4.1 state.
> >>
> >> I asked about that recently and was told SP4_MACH_CRED was going the
> >> way of the do do
> >
> > Do you remember who said that? Is the discussion on line somewhere?
>
> No, I mis-remembered. I was thinking of SP4_SSV.
>
> >
> >> (or did I misunderstand the response from the floor?).
> >>
> >> I'm certainly open to exploring other solutions, but I do want to be
> >> practical about it. Will it be supported on other servers besides
> >> Linux? Does SP4_MACH_CRED help for NFSv4.0?
> >
> > I haven't tested other servers. It's a 4.1-only feature.
>
> SP4_MACH_CRED for 4.1 appears useful, but I think we would need to consider:
>
> o whether SP4_MACH_CRED is a broadly implemented feature where Linux
> clients can rely on it being there in typical environments
I'm assuming it's mandatory for servers to implement. If there is any
example of a released server not implementing SP4_MACH_CRED, I'd like to
know.
> o how to address the "no keytab" issue for NFSv4.0, which does not
> have SP4_MACH_CRED (that I am aware of)
>
> Andy is probably more interested in seeing SP4_MACH_CRED implemented in the Linux client, as it is one solution for the "user cred expired while there is still dirty data in the client's page cache" problem, I think.
OK.
The ability to perform writes using the machine credential is built on
top of SP4_MACH_CRED, but is optional for servers to support.
--b.
On Mon, May 13, 2013 at 12:25:28PM -0400, Chuck Lever wrote:
> Now that commit 4edaa308 "NFS: Use "krb5i" to establish NFSv4 state
> whenever possible" is reverted...
>
> NFSv4 server trunking detection was added with commit 05f4c350 "NFS:
> Discover NFSv4 server trunking when mounting" in v3.7. One of the
> issues that the reverted commit 4edaa308 tried to address was that
> when server trunking detection was added, we broke the ability to
> mount krb5 exports when the client has no keytab.
And that happened because we started asking gssd explicitly for a
context for a machine principal instead of for uid=0? Or for some other
reason?
> We still need to
> address that regression.
>
> Let's try a narrow approach to re-enabling sec=krb5 mounts without a
> local system keytab, and worry about the other issues when we can
> plan concomitant user space and kernel changes.
>
> When a GSS security flavor is used on a mount, allow the security
> flavor for state management to fall back to AUTH_SYS if there is
> a problem setting up a GSS context for it.
This brings back the delay in the case gssd isn't running, doesn't it?
Or am I missing something?
--b.
> The existing situation is that the first NFSv4 mount of a server
> determines the state management security flavor. So, if a sec=sys
> mount occurs first, all subsequent mounts will continue to use
> AUTH_SYS for state management, even if they specified sec=krb5 or
> higher. The risk is allowing a malicious or broken client to purge
> this client's open and lock state on the server; data is not
> exposed.
>
> Falling back if any error occurs while establishing a GSS context
> for state management has its own risks. A network intermediary can
> prevent the establishment of a GSS context for state management,
> forcing the client to use AUTH_SYS for this task.
>
> However, given that the order of NFS mounts is non-deterministic
> anyway, we already fail to provide robust security guarantees for
> state management.
>
> This will be remedied eventually, but changes co-ordinated with user
> space are required for that. When a gssd upcall fails, it currently
> does not distinguish between "I don't have a keytab" and "some other
> problem occurred". To close this security hole, the kernel must
> fall back only when there is no local keytab.
>
> Signed-off-by: Chuck Lever <[email protected]>
> Cc: [email protected] [>=3.7]
> ---
> fs/nfs/nfs4state.c | 14 +++++++++++++-
> 1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
> index b5137c9..a2482c6 100644
> --- a/fs/nfs/nfs4state.c
> +++ b/fs/nfs/nfs4state.c
> @@ -1911,11 +1911,23 @@ again:
> switch (status) {
> case 0:
> break;
> -
> + case -ENOKEY:
> case -EACCES:
> if (clp->cl_machine_cred == NULL)
> break;
> /* Handle case where the user hasn't set up machine creds */
> + if (clnt->cl_auth->au_flavor > RPC_AUTH_MAXFLAVOR) {
> + clnt = rpc_clone_client_set_auth(clnt, RPC_AUTH_UNIX);
> + if (IS_ERR(clnt)) {
> + status = PTR_ERR(clnt);
> + break;
> + }
> + clnt = xchg(&clp->cl_rpcclient, clnt);
> + rpc_shutdown_client(clnt);
> + clnt = clp->cl_rpcclient;
> + dprintk("NFS: retrying trunking detection with AUTH_SYS\n");
> + goto again;
> + }
> nfs4_clear_machine_cred(clp);
> case -NFS4ERR_DELAY:
> case -ETIMEDOUT:
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On May 15, 2013, at 1:48 PM, J. Bruce Fields <[email protected]> wrote:
> On Wed, May 15, 2013 at 01:42:58PM -0400, Chuck Lever wrote:
>>
>> On May 15, 2013, at 1:39 PM, "J. Bruce Fields" <[email protected]> wrote:
>>
>>> By the way, have you looked at SP4_MACH_CRED at all yet? It's a
>>> selfish question (I could use something to test against), but I
>>> think it's also what you want if you want krb5i-protected 4.1 state.
>>
>> I asked about that recently and was told SP4_MACH_CRED was going the
>> way of the do do
>
> Do you remember who said that? Is the discussion on line somewhere?
No, I mis-remembered. I was thinking of SP4_SSV.
>
>> (or did I misunderstand the response from the floor?).
>>
>> I'm certainly open to exploring other solutions, but I do want to be
>> practical about it. Will it be supported on other servers besides
>> Linux? Does SP4_MACH_CRED help for NFSv4.0?
>
> I haven't tested other servers. It's a 4.1-only feature.
SP4_MACH_CRED for 4.1 appears useful, but I think we would need to consider:
o whether SP4_MACH_CRED is a broadly implemented feature where Linux
clients can rely on it being there in typical environments
o how to address the "no keytab" issue for NFSv4.0, which does not
have SP4_MACH_CRED (that I am aware of)
Andy is probably more interested in seeing SP4_MACH_CRED implemented in the Linux client, as it is one solution for the "user cred expired while there is still dirty data in the client's page cache" problem, I think.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On May 15, 2013, at 12:18 PM, "Myklebust, Trond" <[email protected]> wrote:
> On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
>> Hi-
>>
>> Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
>> mounts where the client is not running rpc.gssd.
>>
>> After reverting the "use krb5i for SETCLIENTID" patch, I've added
>> the AUTH_SYS fallback in the EACCES case in
>> nfs4_discover_server_trunking(). I'm not sure whether we need to
>> supplement what's there now, or replace it.
>>
>> "case -ENOKEY:" is added so the kernel will recognize that when gssd
>> is changed to return that instead of EACCES in this case. If the
>> second patch is appled to 3.7 stable and following, it might be a way
>> to address the same regression in older kernels.
>>
>> I've been focused on another bug this week, so this has seen very
>> light testing only. Looking for comments.
>
> I'd like to propose a different approach: we can set up rpc_pipefs files
> clnt/gssd and clnt/krb5 as "honeypots" that rpc.gssd will connect to,
> but that won't do any upcalls. When gssd connects, we set a
> per-rpc_net_ns variable that tells us 'gssd' is up and running. That
> variable only gets cleared if we see a timeout.
Note my solution is a short term gap filler. Bruce and Jeff seem to want something that can fix current kernels without requiring user space changes, and I need something that will allow sec=krb5 mounts to work without a client keytab on kernels since 3.7.
I see your proposal as a long term fix, and not something that we can expect to apply without deploying gssd support at the same time.
>
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On May 15, 2013, at 12:04 PM, "J. Bruce Fields" <[email protected]> wrote:
> On Mon, May 13, 2013 at 12:25:28PM -0400, Chuck Lever wrote:
>> Now that commit 4edaa308 "NFS: Use "krb5i" to establish NFSv4 state
>> whenever possible" is reverted...
>>
>> NFSv4 server trunking detection was added with commit 05f4c350 "NFS:
>> Discover NFSv4 server trunking when mounting" in v3.7. One of the
>> issues that the reverted commit 4edaa308 tried to address was that
>> when server trunking detection was added, we broke the ability to
>> mount krb5 exports when the client has no keytab.
>
> And that happened because we started asking gssd explicitly for a
> context for a machine principal instead of for uid=0? Or for some other
> reason?
We've asked gssd for a machine credential for SETCLIENTID for a while now, so "some other reason." I honestly don't know exactly why it stopped working with my change, but someone bisected it to that one.
But we've always had some issue with the "no keytab" case because we do require a machine credential for SETCLIENTID. I'm suggesting a fix for the broad class of problems using "sec=krb5?" that arise due to a missing keytab on the client.
In this case we need the fix sooner rather than later because it stopped working in 3.7.
>
>> We still need to
>> address that regression.
>>
>> Let's try a narrow approach to re-enabling sec=krb5 mounts without a
>> local system keytab, and worry about the other issues when we can
>> plan concomitant user space and kernel changes.
>>
>> When a GSS security flavor is used on a mount, allow the security
>> flavor for state management to fall back to AUTH_SYS if there is
>> a problem setting up a GSS context for it.
>
> This brings back the delay in the case gssd isn't running, doesn't it?
No. "When a GSS flavor is used" means ONLY when a GSS flavor is used. We make the minimal assumption that using a GSS flavor for a mount means the administrator either has gssd running or wants to know when it isn't, and can tolerate a delay in that case.
When a non-GSS flavor is used, we behave as we did before 3.10, and simply use the same security flavor that the mount point used. So if sec=sys is specified, then SETCLIENTID will use AUTH_SYS, and a gssd upcall is avoided.
> Or am I missing something?
> --b.
>
>> The existing situation is that the first NFSv4 mount of a server
>> determines the state management security flavor. So, if a sec=sys
>> mount occurs first, all subsequent mounts will continue to use
>> AUTH_SYS for state management, even if they specified sec=krb5 or
>> higher. The risk is allowing a malicious or broken client to purge
>> this client's open and lock state on the server; data is not
>> exposed.
>>
>> Falling back if any error occurs while establishing a GSS context
>> for state management has its own risks. A network intermediary can
>> prevent the establishment of a GSS context for state management,
>> forcing the client to use AUTH_SYS for this task.
>>
>> However, given that the order of NFS mounts is non-deterministic
>> anyway, we already fail to provide robust security guarantees for
>> state management.
>>
>> This will be remedied eventually, but changes co-ordinated with user
>> space are required for that. When a gssd upcall fails, it currently
>> does not distinguish between "I don't have a keytab" and "some other
>> problem occurred". To close this security hole, the kernel must
>> fall back only when there is no local keytab.
>>
>> Signed-off-by: Chuck Lever <[email protected]>
>> Cc: [email protected] [>=3.7]
>> ---
>> fs/nfs/nfs4state.c | 14 +++++++++++++-
>> 1 file changed, 13 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
>> index b5137c9..a2482c6 100644
>> --- a/fs/nfs/nfs4state.c
>> +++ b/fs/nfs/nfs4state.c
>> @@ -1911,11 +1911,23 @@ again:
>> switch (status) {
>> case 0:
>> break;
>> -
>> + case -ENOKEY:
>> case -EACCES:
>> if (clp->cl_machine_cred == NULL)
>> break;
>> /* Handle case where the user hasn't set up machine creds */
>> + if (clnt->cl_auth->au_flavor > RPC_AUTH_MAXFLAVOR) {
>> + clnt = rpc_clone_client_set_auth(clnt, RPC_AUTH_UNIX);
>> + if (IS_ERR(clnt)) {
>> + status = PTR_ERR(clnt);
>> + break;
>> + }
>> + clnt = xchg(&clp->cl_rpcclient, clnt);
>> + rpc_shutdown_client(clnt);
>> + clnt = clp->cl_rpcclient;
>> + dprintk("NFS: retrying trunking detection with AUTH_SYS\n");
>> + goto again;
>> + }
>> nfs4_clear_machine_cred(clp);
>> case -NFS4ERR_DELAY:
>> case -ETIMEDOUT:
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
By the way, have you looked at SP4_MACH_CRED at all yet? It's a selfish
question (I could use something to test against), but I think it's also
what you want if you want krb5i-protected 4.1 state.
--b.
On May 15, 2013, at 12:24 PM, "Myklebust, Trond" <[email protected]> wrote:
> On Wed, 2013-05-15 at 12:22 -0400, Chuck Lever wrote:
>> On May 15, 2013, at 12:18 PM, "Myklebust, Trond" <[email protected]> wrote:
>>
>>> On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
>>>> Hi-
>>>>
>>>> Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
>>>> mounts where the client is not running rpc.gssd.
>>>>
>>>> After reverting the "use krb5i for SETCLIENTID" patch, I've added
>>>> the AUTH_SYS fallback in the EACCES case in
>>>> nfs4_discover_server_trunking(). I'm not sure whether we need to
>>>> supplement what's there now, or replace it.
>>>>
>>>> "case -ENOKEY:" is added so the kernel will recognize that when gssd
>>>> is changed to return that instead of EACCES in this case. If the
>>>> second patch is appled to 3.7 stable and following, it might be a way
>>>> to address the same regression in older kernels.
>>>>
>>>> I've been focused on another bug this week, so this has seen very
>>>> light testing only. Looking for comments.
>>>
>>> I'd like to propose a different approach: we can set up rpc_pipefs files
>>> clnt/gssd and clnt/krb5 as "honeypots" that rpc.gssd will connect to,
>>> but that won't do any upcalls. When gssd connects, we set a
>>> per-rpc_net_ns variable that tells us 'gssd' is up and running. That
>>> variable only gets cleared if we see a timeout.
>>
>> Note my solution is a short term gap filler. Bruce and Jeff seem to want something that can fix current kernels without requiring user space changes, and I need something that will allow sec=krb5 mounts to work without a client keytab on kernels since 3.7.
>>
>> I see your proposal as a long term fix, and not something that we can expect to apply without deploying gssd support at the same time.
>
> How does it require gssd modifications?
>
> The whole point is that it requires kernel-only changes, and only minor
> changes at that...
You'll have to be more specific then. The impression I was left with last week was that this solution was a non-starter because one of the two end points wipes all the directories at certain times.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Wed, May 15, 2013 at 01:42:58PM -0400, Chuck Lever wrote:
>
> On May 15, 2013, at 1:39 PM, "J. Bruce Fields" <[email protected]> wrote:
>
> > By the way, have you looked at SP4_MACH_CRED at all yet? It's a
> > selfish question (I could use something to test against), but I
> > think it's also what you want if you want krb5i-protected 4.1 state.
>
> I asked about that recently and was told SP4_MACH_CRED was going the
> way of the do do
Do you remember who said that? Is the discussion on line somewhere?
> (or did I misunderstand the response from the floor?).
>
> I'm certainly open to exploring other solutions, but I do want to be
> practical about it. Will it be supported on other servers besides
> Linux? Does SP4_MACH_CRED help for NFSv4.0?
I haven't tested other servers. It's a 4.1-only feature.
--b.
Now that commit 4edaa308 "NFS: Use "krb5i" to establish NFSv4 state
whenever possible" is reverted...
NFSv4 server trunking detection was added with commit 05f4c350 "NFS:
Discover NFSv4 server trunking when mounting" in v3.7. One of the
issues that the reverted commit 4edaa308 tried to address was that
when server trunking detection was added, we broke the ability to
mount krb5 exports when the client has no keytab. We still need to
address that regression.
Let's try a narrow approach to re-enabling sec=krb5 mounts without a
local system keytab, and worry about the other issues when we can
plan concomitant user space and kernel changes.
When a GSS security flavor is used on a mount, allow the security
flavor for state management to fall back to AUTH_SYS if there is
a problem setting up a GSS context for it.
The existing situation is that the first NFSv4 mount of a server
determines the state management security flavor. So, if a sec=sys
mount occurs first, all subsequent mounts will continue to use
AUTH_SYS for state management, even if they specified sec=krb5 or
higher. The risk is allowing a malicious or broken client to purge
this client's open and lock state on the server; data is not
exposed.
Falling back if any error occurs while establishing a GSS context
for state management has its own risks. A network intermediary can
prevent the establishment of a GSS context for state management,
forcing the client to use AUTH_SYS for this task.
However, given that the order of NFS mounts is non-deterministic
anyway, we already fail to provide robust security guarantees for
state management.
This will be remedied eventually, but changes co-ordinated with user
space are required for that. When a gssd upcall fails, it currently
does not distinguish between "I don't have a keytab" and "some other
problem occurred". To close this security hole, the kernel must
fall back only when there is no local keytab.
Signed-off-by: Chuck Lever <[email protected]>
Cc: [email protected] [>=3.7]
---
fs/nfs/nfs4state.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index b5137c9..a2482c6 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1911,11 +1911,23 @@ again:
switch (status) {
case 0:
break;
-
+ case -ENOKEY:
case -EACCES:
if (clp->cl_machine_cred == NULL)
break;
/* Handle case where the user hasn't set up machine creds */
+ if (clnt->cl_auth->au_flavor > RPC_AUTH_MAXFLAVOR) {
+ clnt = rpc_clone_client_set_auth(clnt, RPC_AUTH_UNIX);
+ if (IS_ERR(clnt)) {
+ status = PTR_ERR(clnt);
+ break;
+ }
+ clnt = xchg(&clp->cl_rpcclient, clnt);
+ rpc_shutdown_client(clnt);
+ clnt = clp->cl_rpcclient;
+ dprintk("NFS: retrying trunking detection with AUTH_SYS\n");
+ goto again;
+ }
nfs4_clear_machine_cred(clp);
case -NFS4ERR_DELAY:
case -ETIMEDOUT:
On Wed, May 15, 2013 at 04:18:03PM +0000, Myklebust, Trond wrote:
> On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
> > Hi-
> >
> > Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
> > mounts where the client is not running rpc.gssd.
> >
> > After reverting the "use krb5i for SETCLIENTID" patch, I've added
> > the AUTH_SYS fallback in the EACCES case in
> > nfs4_discover_server_trunking(). I'm not sure whether we need to
> > supplement what's there now, or replace it.
> >
> > "case -ENOKEY:" is added so the kernel will recognize that when gssd
> > is changed to return that instead of EACCES in this case. If the
> > second patch is appled to 3.7 stable and following, it might be a way
> > to address the same regression in older kernels.
> >
> > I've been focused on another bug this week, so this has seen very
> > light testing only. Looking for comments.
>
> I'd like to propose a different approach: we can set up rpc_pipefs files
> clnt/gssd and clnt/krb5
I'm not sure what paths you mean exactly? Is "clnt" a top-level
rpc-pipefs directory, or a dummy client at the level of other rpc
clients (like rpc_pipefs/nfs/clnt/gssd).
I'm assuming the latter--seems like it might work.
Long-term, it might also be nice to add an interface to allow the kernel
to determine whether gssd is running, without depending on timeouts. It
could be something very trivial.
--b.
> as "honeypots" that rpc.gssd will connect to,
> but that won't do any upcalls. When gssd connects, we set a
> per-rpc_net_ns variable that tells us 'gssd' is up and running. That
> variable only gets cleared if we see a timeout.
>
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
After applying commit 4edaa308 "NFS: Use "krb5i" to establish
NFSv4 state whenever possible", some report a 15 second delay when
attempting to mount the first NFS export on a server, if their
client does not run rpc.gssd. This could be onerous for
automounting or following referrals.
The immediate work around is to insist that such clients be
configured to start rpc.gssd. Some feel that requiring this kind
of change after a kernel upgrade without warning should be
considered a regression.
The desired fix is to avoid the 15 second gssd upcall timeout when
rpc.gssd is not running. Despite everyone's initial opinion, it
appears that it is impossible to eliminate this timeout securely and
reliably without co-operation from user space. Either rpc.gssd
changes, or changes to system start up, or both, are required.
To restore previous behavior, revert:
* commit 79d852bf NFS: Retry SETCLIENTID with AUTH_SYS instead
of AUTH_NONE
* commit 845cbceb NFSv4: Don't clear the machine cred when
client establish returns EACCES
* commit 23631227 NFSv4: Fix the fallback to AUTH_NULL if krb5i
is not available
* commit 4edaa308 NFS: Use "krb5i" to establish NFSv4 state
whenever possible
Signed-off-by: Chuck Lever <[email protected]>
---
fs/nfs/nfs4client.c | 4 +---
fs/nfs/nfs4state.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 49 insertions(+), 8 deletions(-)
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 947b0c9..fdf519c 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -201,9 +201,7 @@ struct nfs_client *nfs4_init_client(struct nfs_client *clp,
if (clp->cl_minorversion != 0)
__set_bit(NFS_CS_INFINITE_SLOTS, &clp->cl_flags);
__set_bit(NFS_CS_DISCRTRY, &clp->cl_flags);
- error = nfs_create_rpc_client(clp, timeparms, RPC_AUTH_GSS_KRB5I);
- if (error == -EINVAL)
- error = nfs_create_rpc_client(clp, timeparms, RPC_AUTH_NULL);
+ error = nfs_create_rpc_client(clp, timeparms, authflavour);
if (error < 0)
goto error;
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 300d17d..b5137c9 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -154,6 +154,18 @@ struct rpc_cred *nfs4_get_machine_cred_locked(struct nfs_client *clp)
return cred;
}
+static void nfs4_clear_machine_cred(struct nfs_client *clp)
+{
+ struct rpc_cred *cred;
+
+ spin_lock(&clp->cl_lock);
+ cred = clp->cl_machine_cred;
+ clp->cl_machine_cred = NULL;
+ spin_unlock(&clp->cl_lock);
+ if (cred != NULL)
+ put_rpccred(cred);
+}
+
static struct rpc_cred *
nfs4_get_renew_cred_server_locked(struct nfs_server *server)
{
@@ -1767,6 +1779,10 @@ static int nfs4_handle_reclaim_lease_error(struct nfs_client *clp, int status)
clear_bit(NFS4CLNT_LEASE_CONFIRM, &clp->cl_state);
return -EPERM;
case -EACCES:
+ if (clp->cl_machine_cred == NULL)
+ return -EACCES;
+ /* Handle case where the user hasn't set up machine creds */
+ nfs4_clear_machine_cred(clp);
case -NFS4ERR_DELAY:
case -ETIMEDOUT:
case -EAGAIN:
@@ -1861,13 +1877,26 @@ int nfs4_discover_server_trunking(struct nfs_client *clp,
{
const struct nfs4_state_recovery_ops *ops =
clp->cl_mvops->reboot_recovery_ops;
+ rpc_authflavor_t *flavors, flav, save;
struct rpc_clnt *clnt;
struct rpc_cred *cred;
- int i, status;
+ int i, len, status;
dprintk("NFS: %s: testing '%s'\n", __func__, clp->cl_hostname);
+ len = NFS_MAX_SECFLAVORS;
+ flavors = kcalloc(len, sizeof(*flavors), GFP_KERNEL);
+ if (flavors == NULL) {
+ status = -ENOMEM;
+ goto out;
+ }
+ len = rpcauth_list_flavors(flavors, len);
+ if (len < 0) {
+ status = len;
+ goto out_free;
+ }
clnt = clp->cl_rpcclient;
+ save = clnt->cl_auth->au_flavor;
i = 0;
mutex_lock(&nfs_clid_init_mutex);
@@ -1882,6 +1911,12 @@ again:
switch (status) {
case 0:
break;
+
+ case -EACCES:
+ if (clp->cl_machine_cred == NULL)
+ break;
+ /* Handle case where the user hasn't set up machine creds */
+ nfs4_clear_machine_cred(clp);
case -NFS4ERR_DELAY:
case -ETIMEDOUT:
case -EAGAIN:
@@ -1890,12 +1925,17 @@ again:
dprintk("NFS: %s after status %d, retrying\n",
__func__, status);
goto again;
- case -EACCES:
- if (i++)
- break;
+
case -NFS4ERR_CLID_INUSE:
case -NFS4ERR_WRONGSEC:
- clnt = rpc_clone_client_set_auth(clnt, RPC_AUTH_UNIX);
+ status = -EPERM;
+ if (i >= len)
+ break;
+
+ flav = flavors[i++];
+ if (flav == save)
+ flav = flavors[i++];
+ clnt = rpc_clone_client_set_auth(clnt, flav);
if (IS_ERR(clnt)) {
status = PTR_ERR(clnt);
break;
@@ -1926,6 +1966,9 @@ again:
out_unlock:
mutex_unlock(&nfs_clid_init_mutex);
+out_free:
+ kfree(flavors);
+out:
dprintk("NFS: %s: status = %d\n", __func__, status);
return status;
}
On Wed, May 15, 2013 at 12:23:12PM -0400, Chuck Lever wrote:
>
> On May 15, 2013, at 12:04 PM, "J. Bruce Fields" <[email protected]> wrote:
>
> > On Mon, May 13, 2013 at 12:25:28PM -0400, Chuck Lever wrote:
> >> Now that commit 4edaa308 "NFS: Use "krb5i" to establish NFSv4 state
> >> whenever possible" is reverted...
> >>
> >> NFSv4 server trunking detection was added with commit 05f4c350 "NFS:
> >> Discover NFSv4 server trunking when mounting" in v3.7. One of the
> >> issues that the reverted commit 4edaa308 tried to address was that
> >> when server trunking detection was added, we broke the ability to
> >> mount krb5 exports when the client has no keytab.
> >
> > And that happened because we started asking gssd explicitly for a
> > context for a machine principal instead of for uid=0? Or for some other
> > reason?
>
> We've asked gssd for a machine credential for SETCLIENTID for a while now, so "some other reason." I honestly don't know exactly why it stopped working with my change, but someone bisected it to that one.
>
> But we've always had some issue with the "no keytab" case because we do require a machine credential for SETCLIENTID. I'm suggesting a fix for the broad class of problems using "sec=krb5?" that arise due to a missing keytab on the client.
>
> In this case we need the fix sooner rather than later because it stopped working in 3.7.
>
> >
> >> We still need to
> >> address that regression.
> >>
> >> Let's try a narrow approach to re-enabling sec=krb5 mounts without a
> >> local system keytab, and worry about the other issues when we can
> >> plan concomitant user space and kernel changes.
> >>
> >> When a GSS security flavor is used on a mount, allow the security
> >> flavor for state management to fall back to AUTH_SYS if there is
> >> a problem setting up a GSS context for it.
> >
> > This brings back the delay in the case gssd isn't running, doesn't it?
>
> No. "When a GSS flavor is used" means ONLY when a GSS flavor is used. We make the minimal assumption that using a GSS flavor for a mount means the administrator either has gssd running or wants to know when it isn't, and can tolerate a delay in that case.
Oh, got it, thanks.--b.
On May 15, 2013, at 1:39 PM, "J. Bruce Fields" <[email protected]> wrote:
> By the way, have you looked at SP4_MACH_CRED at all yet? It's a selfish
> question (I could use something to test against), but I think it's also
> what you want if you want krb5i-protected 4.1 state.
I asked about that recently and was told SP4_MACH_CRED was going the way of the do do (or did I misunderstand the response from the floor?).
I'm certainly open to exploring other solutions, but I do want to be practical about it. Will it be supported on other servers besides Linux? Does SP4_MACH_CRED help for NFSv4.0?
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Wed, 2013-05-15 at 12:30 -0400, Chuck Lever wrote:
> On May 15, 2013, at 12:24 PM, "Myklebust, Trond" <[email protected]> wrote:
>
> > On Wed, 2013-05-15 at 12:22 -0400, Chuck Lever wrote:
> >> On May 15, 2013, at 12:18 PM, "Myklebust, Trond" <[email protected]> wrote:
> >>
> >>> On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
> >>>> Hi-
> >>>>
> >>>> Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
> >>>> mounts where the client is not running rpc.gssd.
> >>>>
> >>>> After reverting the "use krb5i for SETCLIENTID" patch, I've added
> >>>> the AUTH_SYS fallback in the EACCES case in
> >>>> nfs4_discover_server_trunking(). I'm not sure whether we need to
> >>>> supplement what's there now, or replace it.
> >>>>
> >>>> "case -ENOKEY:" is added so the kernel will recognize that when gssd
> >>>> is changed to return that instead of EACCES in this case. If the
> >>>> second patch is appled to 3.7 stable and following, it might be a way
> >>>> to address the same regression in older kernels.
> >>>>
> >>>> I've been focused on another bug this week, so this has seen very
> >>>> light testing only. Looking for comments.
> >>>
> >>> I'd like to propose a different approach: we can set up rpc_pipefs files
> >>> clnt/gssd and clnt/krb5 as "honeypots" that rpc.gssd will connect to,
> >>> but that won't do any upcalls. When gssd connects, we set a
> >>> per-rpc_net_ns variable that tells us 'gssd' is up and running. That
> >>> variable only gets cleared if we see a timeout.
> >>
> >> Note my solution is a short term gap filler. Bruce and Jeff seem to want something that can fix current kernels without requiring user space changes, and I need something that will allow sec=krb5 mounts to work without a client keytab on kernels since 3.7.
> >>
> >> I see your proposal as a long term fix, and not something that we can expect to apply without deploying gssd support at the same time.
> >
> > How does it require gssd modifications?
> >
> > The whole point is that it requires kernel-only changes, and only minor
> > changes at that...
>
> You'll have to be more specific then. The impression I was left with last week was that this solution was a non-starter because one of the two end points wipes all the directories at certain times.
>
No. The problem was the gssd behaviour when it receives a directory
notification due to a client creation/destroy event: it disconnects from
all rpc pipes, and then reconnects to them.
That is solved by using the strategy that the variable is set on the
first connection by gssd, and is only cleared if we see a timeout. It
means that we can quickly detect whether or not gssd has been started
(which is what we need here).
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
On Mon, 2013-05-13 at 12:25 -0400, Chuck Lever wrote:
> Hi-
>
> Here's a stab at addressing the 15 second wait for some 3.10 sec=sys
> mounts where the client is not running rpc.gssd.
>
> After reverting the "use krb5i for SETCLIENTID" patch, I've added
> the AUTH_SYS fallback in the EACCES case in
> nfs4_discover_server_trunking(). I'm not sure whether we need to
> supplement what's there now, or replace it.
>
> "case -ENOKEY:" is added so the kernel will recognize that when gssd
> is changed to return that instead of EACCES in this case. If the
> second patch is appled to 3.7 stable and following, it might be a way
> to address the same regression in older kernels.
>
> I've been focused on another bug this week, so this has seen very
> light testing only. Looking for comments.
I'd like to propose a different approach: we can set up rpc_pipefs files
clnt/gssd and clnt/krb5 as "honeypots" that rpc.gssd will connect to,
but that won't do any upcalls. When gssd connects, we set a
per-rpc_net_ns variable that tells us 'gssd' is up and running. That
variable only gets cleared if we see a timeout.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com