On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
wrote:
> This should probably be cc'd to the mailing list.
Agreed!
>
> On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
>> I have a pretty functional client-side SP4_MACH_CRED implementation
>> and I'm trying to implement the server side so I can fully test the
>> client code. I was testing against another server, but that didn't
>> implement any useful set of operations other than the required ones.
>
> The latest Linux server implements SP4_MACH_CRED but only for the
> required operations. (But that hasn't really been tested--any testing
> even to make sure that basic stuff works would be welcomed.)
>
Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
>> I'm currently trying to figure out the best way to set nfsd's current cred to that of the user really doing the operation.
>>
>> i.e.:
>>
>> - WRITE, CLOSE, LOCKT, LOCKU are supported by SP4_MACH_CRED.
>
> "are supported by" means they're included in the spo_must_allow list
> returned by the server?
>
Yes, what I meant was "the client thinks it can use the machine cred for these ops".
>> - OPEN calls use the user cred
>> - When cred expires, SP4_MACH_CRED kicks in and cached WRITE calls are sent out using the machine cred.
>> - nfsd handles request with the machine cred as the current cred
>>
>> Currently, these writes fail in nfsd_permission (after I made a few other
>> changes) because the current cred (the machine cred of the client) maps to
>> a UID that is not the owner of the file (even though the io was done as the
>> user).
>>
>> I think what we need is a way to look up the cred associated with the state owner of the current operation's stateid and use that as the current credential when the machine cred is in use.
>
> Is there a rule that says a stateowner or stateid can only be used with
> one credential? If not, I'm not sure what you mean by "the cred
> associated with the state owner of the current operation's stateid".
>
> Maybe we could just pick "a cred" instead of "the cred"? So stash the
> last cred used with a given stateid in the stateid, and use that in this
> case?
Yeah... I was a bit confused. This sounds like a good approach. Thanks!
>
>> Does that make sense to you? Is there something like this already? If
>> not, any pointers on what you'd like to see?
>
> As a user I'd normally expect that once my credentials have expired, the
> client will no longer be able to write to my files.
That's true with unbuffered writes, but what about buffered writes that are already considered written by the application?
The problem I'm trying to solve initially with SP4_MACH_CRED is when there are a bunch of buffered writes and the user's credentials expire. Any client side operations initiated after the credentials expire will fail (as expected), but operations that have already been seen as successful still need to be flushed.
Andy is addressing this in a different way by falling back to unbuffered writes and flushing all data right before a cred expires, but this method is better when available - and it's Trond requested.
>
> This violates that assumption, so I wonder if it needs to be
> configurable, and even off by default.
>
This will fix potential data corruption in a corner case of the way things currently work and doesn't change expected behavior from a user's perspective.
-dros
On Wed, Jul 31, 2013 at 10:22:22PM +0000, Adamson, Dros wrote:
>
> On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
> wrote:
>
> > This should probably be cc'd to the mailing list.
>
> Agreed!
>
> >
> > On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
> >> I have a pretty functional client-side SP4_MACH_CRED implementation
> >> and I'm trying to implement the server side so I can fully test the
> >> client code. I was testing against another server, but that didn't
> >> implement any useful set of operations other than the required ones.
> >
> > The latest Linux server implements SP4_MACH_CRED but only for the
> > required operations. (But that hasn't really been tested--any testing
> > even to make sure that basic stuff works would be welcomed.)
> >
>
> Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
>
> >> I'm currently trying to figure out the best way to set nfsd's current cred to that of the user really doing the operation.
> >>
> >> i.e.:
> >>
> >> - WRITE, CLOSE, LOCKT, LOCKU are supported by SP4_MACH_CRED.
> >
> > "are supported by" means they're included in the spo_must_allow list
> > returned by the server?
> >
>
> Yes, what I meant was "the client thinks it can use the machine cred for these ops".
>
> >> - OPEN calls use the user cred
> >> - When cred expires, SP4_MACH_CRED kicks in and cached WRITE calls are sent out using the machine cred.
> >> - nfsd handles request with the machine cred as the current cred
> >>
> >> Currently, these writes fail in nfsd_permission (after I made a few other
> >> changes) because the current cred (the machine cred of the client) maps to
> >> a UID that is not the owner of the file (even though the io was done as the
> >> user).
> >>
> >> I think what we need is a way to look up the cred associated with the state owner of the current operation's stateid and use that as the current credential when the machine cred is in use.
> >
> > Is there a rule that says a stateowner or stateid can only be used with
> > one credential? If not, I'm not sure what you mean by "the cred
> > associated with the state owner of the current operation's stateid".
> >
> > Maybe we could just pick "a cred" instead of "the cred"? So stash the
> > last cred used with a given stateid in the stateid, and use that in this
> > case?
>
> Yeah... I was a bit confused. This sounds like a good approach. Thanks!
>
> >
> >> Does that make sense to you? Is there something like this already? If
> >> not, any pointers on what you'd like to see?
> >
> > As a user I'd normally expect that once my credentials have expired, the
> > client will no longer be able to write to my files.
>
> That's true with unbuffered writes, but what about buffered writes that are already considered written by the application?
Well, the server can't distinguish between these two, all it can do is
say that once you've allowed the client to open a file as you, it's
going to continue allowing the client to access the file without needing
your credentials.
> The problem I'm trying to solve initially with SP4_MACH_CRED is when there are a bunch of buffered writes and the user's credentials expire. Any client side operations initiated after the credentials expire will fail (as expected), but operations that have already been seen as successful still need to be flushed.
>
> Andy is addressing this in a different way by falling back to unbuffered writes and flushing all data right before a cred expires, but this method is better when available - and it's Trond requested.
I understand the problem it solves, but it is a bit of a change to the
security model and I just wonder what the right thing is to do by
default.
--b.
>
> >
> > This violates that assumption, so I wonder if it needs to be
> > configurable, and even off by default.
> >
>
> This will fix potential data corruption in a corner case of the way things currently work and doesn't change expected behavior from a user's perspective.
>
> -dros
>
On Thu, Aug 01, 2013 at 03:24:05PM +0000, Adamson, Dros wrote:
>
> On Aug 1, 2013, at 10:56 AM, J. Bruce Fields <[email protected]> wrote:
>
> > On Thu, Aug 01, 2013 at 02:09:49PM +0000, Adamson, Dros wrote:
> >>
> >> On Aug 1, 2013, at 10:03 AM, "Adamson, Dros" <[email protected]>
> >> wrote:
> >>
> >>>
> >>> On Jul 31, 2013, at 7:48 PM, J. Bruce Fields <[email protected]> wrote:
> >>>
> >>>> On Wed, Jul 31, 2013 at 10:22:22PM +0000, Adamson, Dros wrote:
> >>>>>
> >>>>> On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
> >>>>> wrote:
> >>>>>
> >>>>>> This should probably be cc'd to the mailing list.
> >>>>>
> >>>>> Agreed!
> >>>>>
> >>>>>>
> >>>>>> On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
> >>>>>>> I have a pretty functional client-side SP4_MACH_CRED implementation
> >>>>>>> and I'm trying to implement the server side so I can fully test the
> >>>>>>> client code. I was testing against another server, but that didn't
> >>>>>>> implement any useful set of operations other than the required ones.
> >>>>>>
> >>>>>> The latest Linux server implements SP4_MACH_CRED but only for the
> >>>>>> required operations. (But that hasn't really been tested--any testing
> >>>>>> even to make sure that basic stuff works would be welcomed.)
> >>>>>>
> >>>>>
> >>>>> Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
> >
> > (And I think I forgot to say thanks here! It's useful to have that
> > tested.)
>
> I should note it's somewhat useless to use SP4_MACH_CRED in this way from the linux client's perspective. We already use the machine cred with krb5i if possible for state manager ops even in SP4_NONE mode.
Without SP4_MACH_CRED anyone can e.g. destroy your client using just
auth_sys.
On its own all the use of krb5i really does is reassure you that your
rpc replies are from the server.
> > - user credentials go away before they're expected to expire.
> > (I wonder how this would typically happen?)
>
> I don't believe this can happen yet. IIRC a kdestroy isn't noticed by
> gssd, but this is probably going to change soon with Andy's keyring
> work, so it'd be nice to plan for it.
Yes. I wonder if you could also take advantage of Andy's expiring-cred
emergency mode here? So when you get the kdestroy notification, you
could try to flush writes before destroying contexts.
--b.
On Thu, Aug 01, 2013 at 03:24:05PM +0000, Adamson, Dros wrote:
>
> On Aug 1, 2013, at 10:56 AM, J. Bruce Fields <[email protected]> wrote:
>
> > So the choices are:
> > 1. allow those problems, or
> > 2. change the nfs/krb5 security model from "once you give a
> > client your credential, you trust it with your data till the
> > credential expires" to "once you give a client your
> > credential, you trust it indefinitely". (Well, until its
> > state expires, anyway.)
> >
> > And we can implement either 1, or 2, or both, and if we implement both,
> > we get to choose which of 1 or 2 is the default.
> >
> > So I don't have a strong opinion yet, but I do fear that if we only
> > implement #2 there will be users who see that as a serious regression in
> > security.
>
> Perhaps you're right - the server is essentially trusting the client machine to do the right thing here, but the server is already trusting the client machine (not users) to do the right thing. A rouge client implementation could create all types of havoc as it stands now, i.e. using another user's credentials to bypass permissions, etc.
Right, but normally that capacity for havoc is time-limited. For
example, you're not affected if a client is compromised after your creds
expire.
I think that's pretty important. So let's make this optional. I'm
willing to argue about the default policy.
Do all the operations that might conceivably be affected by this take
filehandles? If so an export option might make sense. Something like
trust_client_writeback
NFSv4.1 and higher clients who have accessed a file
using a user's credential will be permitted further
write access to that file using only their machine
credential. Some clients can take advantage of this to
write back cached data after user credentials expire,
preventing possible data loss in some cases.
Maybe there's a better name!
--b.
On Thu, Aug 01, 2013 at 02:03:29PM +0000, Adamson, Dros wrote:
> The other case we are aiming for with the client is to be able to use
> the machine cred in recovery when the user's creds expire so that
> flushing of buffers can happen once recovery is complete. Of course,
> if you don't want to see the flushing changes, this is useless.
Just so I understand: addressing this case would require allowing more
operations, right? (OPEN to do the reclaim, for example.)
--b.
On Jul 31, 2013, at 7:48 PM, J. Bruce Fields <[email protected]> wrote:
> On Wed, Jul 31, 2013 at 10:22:22PM +0000, Adamson, Dros wrote:
>>
>> On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
>> wrote:
>>
>>> This should probably be cc'd to the mailing list.
>>
>> Agreed!
>>
>>>
>>> On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
>>>> I have a pretty functional client-side SP4_MACH_CRED implementation
>>>> and I'm trying to implement the server side so I can fully test the
>>>> client code. I was testing against another server, but that didn't
>>>> implement any useful set of operations other than the required ones.
>>>
>>> The latest Linux server implements SP4_MACH_CRED but only for the
>>> required operations. (But that hasn't really been tested--any testing
>>> even to make sure that basic stuff works would be welcomed.)
>>>
>>
>> Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
>>
>>>> I'm currently trying to figure out the best way to set nfsd's current cred to that of the user really doing the operation.
>>>>
>>>> i.e.:
>>>>
>>>> - WRITE, CLOSE, LOCKT, LOCKU are supported by SP4_MACH_CRED.
>>>
>>> "are supported by" means they're included in the spo_must_allow list
>>> returned by the server?
>>>
>>
>> Yes, what I meant was "the client thinks it can use the machine cred for these ops".
>>
>>>> - OPEN calls use the user cred
>>>> - When cred expires, SP4_MACH_CRED kicks in and cached WRITE calls are sent out using the machine cred.
>>>> - nfsd handles request with the machine cred as the current cred
>>>>
>>>> Currently, these writes fail in nfsd_permission (after I made a few other
>>>> changes) because the current cred (the machine cred of the client) maps to
>>>> a UID that is not the owner of the file (even though the io was done as the
>>>> user).
>>>>
>>>> I think what we need is a way to look up the cred associated with the state owner of the current operation's stateid and use that as the current credential when the machine cred is in use.
>>>
>>> Is there a rule that says a stateowner or stateid can only be used with
>>> one credential? If not, I'm not sure what you mean by "the cred
>>> associated with the state owner of the current operation's stateid".
>>>
>>> Maybe we could just pick "a cred" instead of "the cred"? So stash the
>>> last cred used with a given stateid in the stateid, and use that in this
>>> case?
>>
>> Yeah... I was a bit confused. This sounds like a good approach. Thanks!
>>
>>>
>>>> Does that make sense to you? Is there something like this already? If
>>>> not, any pointers on what you'd like to see?
>>>
>>> As a user I'd normally expect that once my credentials have expired, the
>>> client will no longer be able to write to my files.
>>
>> That's true with unbuffered writes, but what about buffered writes that are already considered written by the application?
>
> Well, the server can't distinguish between these two, all it can do is
> say that once you've allowed the client to open a file as you, it's
> going to continue allowing the client to access the file without needing
> your credentials.
>
Right! And using state protection here extends the trust to the client machine so it's not just allowing continued access without needing user credentials.
>> The problem I'm trying to solve initially with SP4_MACH_CRED is when there are a bunch of buffered writes and the user's credentials expire. Any client side operations initiated after the credentials expire will fail (as expected), but operations that have already been seen as successful still need to be flushed.
>>
>> Andy is addressing this in a different way by falling back to unbuffered writes and flushing all data right before a cred expires, but this method is better when available - and it's Trond requested.
>
> I understand the problem it solves, but it is a bit of a change to the
> security model and I just wonder what the right thing is to do by
> default.
So you think the default behavior should be that buffered writes just hang around forever in the client's buffer cache when a user cred expires? That seems really wrong to me.
If this isn't something you want, I'm not sure that we want any SP4_MACH_CRED support other than the required operations, which is kind of a moot point since we now use krb5i for state manager operations by default (whether in SP4_MACH_CRED or SP4_NONE).
The other case we are aiming for with the client is to be able to use the machine cred in recovery when the user's creds expire so that flushing of buffers can happen once recovery is complete. Of course, if you don't want to see the flushing changes, this is useless.
So? Is SP4_MACH_CRED support even worth the effort?
-dros
>
> --b.
>
>>
>>>
>>> This violates that assumption, so I wonder if it needs to be
>>> configurable, and even off by default.
>>>
>>
>> This will fix potential data corruption in a corner case of the way things currently work and doesn't change expected behavior from a user's perspective.
>>
>> -dros
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Aug 01, 2013 at 02:09:49PM +0000, Adamson, Dros wrote:
>
> On Aug 1, 2013, at 10:03 AM, "Adamson, Dros" <[email protected]>
> wrote:
>
> >
> > On Jul 31, 2013, at 7:48 PM, J. Bruce Fields <[email protected]> wrote:
> >
> >> On Wed, Jul 31, 2013 at 10:22:22PM +0000, Adamson, Dros wrote:
> >>>
> >>> On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
> >>> wrote:
> >>>
> >>>> This should probably be cc'd to the mailing list.
> >>>
> >>> Agreed!
> >>>
> >>>>
> >>>> On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
> >>>>> I have a pretty functional client-side SP4_MACH_CRED implementation
> >>>>> and I'm trying to implement the server side so I can fully test the
> >>>>> client code. I was testing against another server, but that didn't
> >>>>> implement any useful set of operations other than the required ones.
> >>>>
> >>>> The latest Linux server implements SP4_MACH_CRED but only for the
> >>>> required operations. (But that hasn't really been tested--any testing
> >>>> even to make sure that basic stuff works would be welcomed.)
> >>>>
> >>>
> >>> Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
(And I think I forgot to say thanks here! It's useful to have that
tested.)
> >>> The problem I'm trying to solve initially with SP4_MACH_CRED is when there are a bunch of buffered writes and the user's credentials expire. Any client side operations initiated after the credentials expire will fail (as expected), but operations that have already been seen as successful still need to be flushed.
> >>>
> >>> Andy is addressing this in a different way by falling back to unbuffered writes and flushing all data right before a cred expires, but this method is better when available - and it's Trond requested.
> >>
> >> I understand the problem it solves, but it is a bit of a change to the
> >> security model and I just wonder what the right thing is to do by
> >> default.
> >
> > So you think
I'm not staking out a position, just trying to understand the tradeoffs:
> the default behavior should be that buffered writes just hang around
> forever in the client's buffer cache when a user cred expires? That
> seems really wrong to me.
So once Andy's work is also done, I guess the client can attempt to:
- try to get creds renewed well in advance.
- if they aren't, flush everything and start writing
synchronously.
But problems could still happen, for example, if:
- an unfortunately timed network outage prevents the above.
(Then again, MACH_CHRED may not help here if that also means
the client's state expires?)
- an unfortunately timed server reboot prevents the above.
- user credentials go away before they're expected to expire.
(I wonder how this would typically happen?)
And then the client may be forced to choose between throwing away
buffered data and allowing users to DOS it by filling memory with data
it can't write back.
So the choices are:
1. allow those problems, or
2. change the nfs/krb5 security model from "once you give a
client your credential, you trust it with your data till the
credential expires" to "once you give a client your
credential, you trust it indefinitely". (Well, until its
state expires, anyway.)
And we can implement either 1, or 2, or both, and if we implement both,
we get to choose which of 1 or 2 is the default.
So I don't have a strong opinion yet, but I do fear that if we only
implement #2 there will be users who see that as a serious regression in
security.
--b.
> >
> > If this isn't something you want, I'm not sure that we want any SP4_MACH_CRED support other than the required operations, which is kind of a moot point since we now use krb5i for state manager operations by default (whether in SP4_MACH_CRED or SP4_NONE).
> >
> > The other case we are aiming for with the client is to be able to use the machine cred in recovery when the user's creds expire so that flushing of buffers can happen once recovery is complete. Of course, if you don't want to see the flushing changes, this is useless.
> >
> > So… Is SP4_MACH_CRED support even worth the effort?
> >
> > -dros
> >
> >>
> >> --b.
> >>
> >>>
> >>>>
> >>>> This violates that assumption, so I wonder if it needs to be
> >>>> configurable, and even off by default.
> >>>>
> >>>
> >>> This will fix potential data corruption in a corner case of the way things currently work and doesn't change expected behavior from a user's perspective.
> >>>
> >>> -dros
> >>>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >> the body of a message to [email protected]
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Aug 1, 2013, at 10:03 AM, "Adamson, Dros" <[email protected]>
wrote:
>
> On Jul 31, 2013, at 7:48 PM, J. Bruce Fields <[email protected]> wrote:
>
>> On Wed, Jul 31, 2013 at 10:22:22PM +0000, Adamson, Dros wrote:
>>>
>>> On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
>>> wrote:
>>>
>>>> This should probably be cc'd to the mailing list.
>>>
>>> Agreed!
>>>
>>>>
>>>> On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
>>>>> I have a pretty functional client-side SP4_MACH_CRED implementation
>>>>> and I'm trying to implement the server side so I can fully test the
>>>>> client code. I was testing against another server, but that didn't
>>>>> implement any useful set of operations other than the required ones.
>>>>
>>>> The latest Linux server implements SP4_MACH_CRED but only for the
>>>> required operations. (But that hasn't really been tested--any testing
>>>> even to make sure that basic stuff works would be welcomed.)
>>>>
>>>
>>> Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
>>>
>>>>> I'm currently trying to figure out the best way to set nfsd's current cred to that of the user really doing the operation.
>>>>>
>>>>> i.e.:
>>>>>
>>>>> - WRITE, CLOSE, LOCKT, LOCKU are supported by SP4_MACH_CRED.
>>>>
>>>> "are supported by" means they're included in the spo_must_allow list
>>>> returned by the server?
>>>>
>>>
>>> Yes, what I meant was "the client thinks it can use the machine cred for these ops".
>>>
>>>>> - OPEN calls use the user cred
>>>>> - When cred expires, SP4_MACH_CRED kicks in and cached WRITE calls are sent out using the machine cred.
>>>>> - nfsd handles request with the machine cred as the current cred
>>>>>
>>>>> Currently, these writes fail in nfsd_permission (after I made a few other
>>>>> changes) because the current cred (the machine cred of the client) maps to
>>>>> a UID that is not the owner of the file (even though the io was done as the
>>>>> user).
>>>>>
>>>>> I think what we need is a way to look up the cred associated with the state owner of the current operation's stateid and use that as the current credential when the machine cred is in use.
>>>>
>>>> Is there a rule that says a stateowner or stateid can only be used with
>>>> one credential? If not, I'm not sure what you mean by "the cred
>>>> associated with the state owner of the current operation's stateid".
>>>>
>>>> Maybe we could just pick "a cred" instead of "the cred"? So stash the
>>>> last cred used with a given stateid in the stateid, and use that in this
>>>> case?
>>>
>>> Yeah... I was a bit confused. This sounds like a good approach. Thanks!
>>>
>>>>
>>>>> Does that make sense to you? Is there something like this already? If
>>>>> not, any pointers on what you'd like to see?
>>>>
>>>> As a user I'd normally expect that once my credentials have expired, the
>>>> client will no longer be able to write to my files.
>>>
>>> That's true with unbuffered writes, but what about buffered writes that are already considered written by the application?
>>
>> Well, the server can't distinguish between these two, all it can do is
>> say that once you've allowed the client to open a file as you, it's
>> going to continue allowing the client to access the file without needing
>> your credentials.
>>
>
> Right! And using state protection here extends the trust to the client machine so it's not just allowing continued access without needing user credentials.
s/user //
>
>>> The problem I'm trying to solve initially with SP4_MACH_CRED is when there are a bunch of buffered writes and the user's credentials expire. Any client side operations initiated after the credentials expire will fail (as expected), but operations that have already been seen as successful still need to be flushed.
>>>
>>> Andy is addressing this in a different way by falling back to unbuffered writes and flushing all data right before a cred expires, but this method is better when available - and it's Trond requested.
>>
>> I understand the problem it solves, but it is a bit of a change to the
>> security model and I just wonder what the right thing is to do by
>> default.
>
> So you think the default behavior should be that buffered writes just hang around forever in the client's buffer cache when a user cred expires? That seems really wrong to me.
>
> If this isn't something you want, I'm not sure that we want any SP4_MACH_CRED support other than the required operations, which is kind of a moot point since we now use krb5i for state manager operations by default (whether in SP4_MACH_CRED or SP4_NONE).
>
> The other case we are aiming for with the client is to be able to use the machine cred in recovery when the user's creds expire so that flushing of buffers can happen once recovery is complete. Of course, if you don't want to see the flushing changes, this is useless.
>
> So? Is SP4_MACH_CRED support even worth the effort?
>
> -dros
>
>>
>> --b.
>>
>>>
>>>>
>>>> This violates that assumption, so I wonder if it needs to be
>>>> configurable, and even off by default.
>>>>
>>>
>>> This will fix potential data corruption in a corner case of the way things currently work and doesn't change expected behavior from a user's perspective.
>>>
>>> -dros
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Aug 1, 2013, at 3:05 PM, J. Bruce Fields <[email protected]> wrote:
> On Thu, Aug 01, 2013 at 03:24:05PM +0000, Adamson, Dros wrote:
>>
>> On Aug 1, 2013, at 10:56 AM, J. Bruce Fields <[email protected]> wrote:
>>
>>> So the choices are:
>>> 1. allow those problems, or
>>> 2. change the nfs/krb5 security model from "once you give a
>>> client your credential, you trust it with your data till the
>>> credential expires" to "once you give a client your
>>> credential, you trust it indefinitely". (Well, until its
>>> state expires, anyway.)
>>>
>>> And we can implement either 1, or 2, or both, and if we implement both,
>>> we get to choose which of 1 or 2 is the default.
>>>
>>> So I don't have a strong opinion yet, but I do fear that if we only
>>> implement #2 there will be users who see that as a serious regression in
>>> security.
>>
>> Perhaps you're right - the server is essentially trusting the client machine to do the right thing here, but the server is already trusting the client machine (not users) to do the right thing. A rouge client implementation could create all types of havoc as it stands now, i.e. using another user's credentials to bypass permissions, etc.
>
> Right, but normally that capacity for havoc is time-limited. For
> example, you're not affected if a client is compromised after your creds
> expire.
>
> I think that's pretty important. So let's make this optional. I'm
> willing to argue about the default policy.
>
> Do all the operations that might conceivably be affected by this take
> filehandles? If so an export option might make sense. Something like
>
> trust_client_writeback
>
> NFSv4.1 and higher clients who have accessed a file
> using a user's credential will be permitted further
> write access to that file using only their machine
> credential. Some clients can take advantage of this to
> write back cached data after user credentials expire,
> preventing possible data loss in some cases.
>
> Maybe there's a better name!
Making an option is fine with me. I think we'll eventually want it to be the default, but we'll still need a way to disable it.
-dros
>
> --b.
On Fri, Aug 02, 2013 at 04:51:28PM +0000, Adamson, Dros wrote:
> On Aug 2, 2013, at 11:32 AM, J. Bruce Fields <[email protected]> wrote:
>
> > On Thu, Aug 01, 2013 at 02:03:29PM +0000, Adamson, Dros wrote:
> >> The other case we are aiming for with the client is to be able to use
> >> the machine cred in recovery when the user's creds expire so that
> >> flushing of buffers can happen once recovery is complete. Of course,
> >> if you don't want to see the flushing changes, this is useless.
> >
> > Just so I understand: addressing this case would require allowing more
> > operations, right? (OPEN to do the reclaim, for example.)
>
> Correct.
That one seems bigger--I'm not sure how to keep it from becoming
"clients can write anywhere they want" without a lot of work (like, say,
storing on disk some information about which users have written from
that client recently).
--b.
On Aug 2, 2013, at 11:32 AM, J. Bruce Fields <[email protected]> wrote:
> On Thu, Aug 01, 2013 at 02:03:29PM +0000, Adamson, Dros wrote:
>> The other case we are aiming for with the client is to be able to use
>> the machine cred in recovery when the user's creds expire so that
>> flushing of buffers can happen once recovery is complete. Of course,
>> if you don't want to see the flushing changes, this is useless.
>
> Just so I understand: addressing this case would require allowing more
> operations, right? (OPEN to do the reclaim, for example.)
Correct.
-dros
>
> --b.
On Aug 1, 2013, at 10:56 AM, J. Bruce Fields <[email protected]> wrote:
> On Thu, Aug 01, 2013 at 02:09:49PM +0000, Adamson, Dros wrote:
>>
>> On Aug 1, 2013, at 10:03 AM, "Adamson, Dros" <[email protected]>
>> wrote:
>>
>>>
>>> On Jul 31, 2013, at 7:48 PM, J. Bruce Fields <[email protected]> wrote:
>>>
>>>> On Wed, Jul 31, 2013 at 10:22:22PM +0000, Adamson, Dros wrote:
>>>>>
>>>>> On Jul 31, 2013, at 5:39 PM, "J. Bruce Fields" <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> This should probably be cc'd to the mailing list.
>>>>>
>>>>> Agreed!
>>>>>
>>>>>>
>>>>>> On Wed, Jul 31, 2013 at 09:25:29PM +0000, Adamson, Dros wrote:
>>>>>>> I have a pretty functional client-side SP4_MACH_CRED implementation
>>>>>>> and I'm trying to implement the server side so I can fully test the
>>>>>>> client code. I was testing against another server, but that didn't
>>>>>>> implement any useful set of operations other than the required ones.
>>>>>>
>>>>>> The latest Linux server implements SP4_MACH_CRED but only for the
>>>>>> required operations. (But that hasn't really been tested--any testing
>>>>>> even to make sure that basic stuff works would be welcomed.)
>>>>>>
>>>>>
>>>>> Right, I'm happy to report that the initial implementation of the required operations seems to work with (at least) one small patch I'll send your way soon.
>
> (And I think I forgot to say thanks here! It's useful to have that
> tested.)
I should note it's somewhat useless to use SP4_MACH_CRED in this way from the linux client's perspective. We already use the machine cred with krb5i if possible for state manager ops even in SP4_NONE mode.
>
>>>>> The problem I'm trying to solve initially with SP4_MACH_CRED is when there are a bunch of buffered writes and the user's credentials expire. Any client side operations initiated after the credentials expire will fail (as expected), but operations that have already been seen as successful still need to be flushed.
>>>>>
>>>>> Andy is addressing this in a different way by falling back to unbuffered writes and flushing all data right before a cred expires, but this method is better when available - and it's Trond requested.
>>>>
>>>> I understand the problem it solves, but it is a bit of a change to the
>>>> security model and I just wonder what the right thing is to do by
>>>> default.
>>>
>>> So you think
>
> I'm not staking out a position, just trying to understand the tradeoffs:
>
>> the default behavior should be that buffered writes just hang around
>> forever in the client's buffer cache when a user cred expires? That
>> seems really wrong to me.
>
> So once Andy's work is also done, I guess the client can attempt to:
> - try to get creds renewed well in advance.
This is still up to the user / distro / config.
> - if they aren't, flush everything and start writing
> synchronously.
>
> But problems could still happen, for example, if:
> - an unfortunately timed network outage prevents the above.
> (Then again, MACH_CHRED may not help here if that also means
> the client's state expires?)
Right. Using MACH_CRED has a much better chance of being able to flush the data.
> - an unfortunately timed server reboot prevents the above.
Yep, and using the machine creds should help here.
> - user credentials go away before they're expected to expire.
> (I wonder how this would typically happen?)
I don't believe this can happen yet. IIRC a kdestroy isn't noticed by gssd, but this is probably going to change soon with Andy's keyring work, so it'd be nice to plan for it.
>
> And then the client may be forced to choose between throwing away
> buffered data and allowing users to DOS it by filling memory with data
> it can't write back.
Right, and this leads to data corruption - the application believes it has written bytes to storage that never really made it.
>
> So the choices are:
> 1. allow those problems, or
> 2. change the nfs/krb5 security model from "once you give a
> client your credential, you trust it with your data till the
> credential expires" to "once you give a client your
> credential, you trust it indefinitely". (Well, until its
> state expires, anyway.)
>
> And we can implement either 1, or 2, or both, and if we implement both,
> we get to choose which of 1 or 2 is the default.
>
> So I don't have a strong opinion yet, but I do fear that if we only
> implement #2 there will be users who see that as a serious regression in
> security.
Perhaps you're right - the server is essentially trusting the client machine to do the right thing here, but the server is already trusting the client machine (not users) to do the right thing. A rouge client implementation could create all types of havoc as it stands now, i.e. using another user's credentials to bypass permissions, etc.
I argue that from a user's perspective, #2 is the right way to go. If my credentials aren't expired when I write some bytes, and the application is told that the write succeeded, I expect those bytes next time I open the file.
-dros
>
> --b.
>
>>>
>>> If this isn't something you want, I'm not sure that we want any SP4_MACH_CRED support other than the required operations, which is kind of a moot point since we now use krb5i for state manager operations by default (whether in SP4_MACH_CRED or SP4_NONE).
>>>
>>> The other case we are aiming for with the client is to be able to use the machine cred in recovery when the user's creds expire so that flushing of buffers can happen once recovery is complete. Of course, if you don't want to see the flushing changes, this is useless.
>>>
>>> So? Is SP4_MACH_CRED support even worth the effort?
>>>
>>> -dros
>>>
>>>>
>>>> --b.
>>>>
>>>>>
>>>>>>
>>>>>> This violates that assumption, so I wonder if it needs to be
>>>>>> configurable, and even off by default.
>>>>>>
>>>>>
>>>>> This will fix potential data corruption in a corner case of the way things currently work and doesn't change expected behavior from a user's perspective.
>>>>>
>>>>> -dros
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html