Hi folks,
Problem:
The upstream nfs4.0 client has problem where it will go into an
infinite loop of re-sending an OPEN when it's trying to recover from
receiving a BAD_STATEID error on an IO operation such READ or WRITE.
How to easily reproduce (by using fault injection):
1. Do nfs4.0 mount to a server.
2. Open a file such that the server gives you a write delegation.
3. Do a write. Have a server return a BAD_STATEID. One way to do so is
by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
WRITE.
4. And off it goes with the loop.
Here’s why….
IO op like WRITE receives a BAD_STATEID.
1. for this error, in async handle error we call
nfs4_schedule_stateid_recover()
2. that in turn will call nfs4_state_mark_reclaim_nograce() that will
set a RECLAIM_NOGRACE in the state flags.
3. state manager thread will run and call nfs4_do_reclaim() to recover.
4. that will call nfs4_reclaim_open_state()
in that function:
restart:
for open states in state
test if RECLAIM_NOGRACE is set in state flags, if so clear it (it’s
set and we’ll clear it)
check open_stateid (checks if RECOVERY_FAILED is not set) (it’s not)
checks if we have state
calls ops->recover_open()
for nfs4.0, it’ll call nfs40_open_expired()
it’ll call nfs40_clear_delegation_stateid()
it’ll call nfs_finish_clear_delegation_stateid()
it’ll call nfs_remove_bad_delegation()
it’ll call nfs_inode_find_state_and_recover()
it’ll call nfs4_state_mark_reclaim_nograce() **** this will set
RECLAIM_NOGRACE in state flags
we return from recover_open() with status 0
call nfs4_reclaim_locks() returns 0 then
goto restart; ************** what happens is since we reset the flag
in the state flags the whole loop starts again.
Solution:
nfs_remove_bad_delegation() is only called from
nfs_finish_clear_delegation_stateid() which is called from either 4.0
or 4.1 recover open functions in nograce case. In both cases, this is
already state manager doing recovery based on the RECLAIM_NOGRACE flag
set and it's going thru opens that need to be recovered.
I propose to correct the loop by removing the call:
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 4711d04..b322823 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -632,10 +632,8 @@ void nfs_remove_bad_delegation(struct inode *inode)
nfs_revoke_delegation(inode);
delegation = nfs_inode_detach_delegation(inode);
- if (delegation) {
- nfs_inode_find_state_and_recover(inode, &delegation->stateid);
+ if (delegation)
nfs_free_delegation(delegation);
- }
}
EXPORT_SYMBOL_GPL(nfs_remove_bad_delegation);
Hi Olga,
I believe we see the same infinite loop without delegation with RHEL6/7
kernels, but without delegation being involved. Currently, on the server
side, if client looping is detected, we return RESOURCE. This breaks the
loop and application gets an IO error.
Your fix is only covers the delegation case isn't it?
Tigran.
----- Original Message -----
> From: "Olga Kornievskaia" <[email protected]>
> To: "Trond Myklebust" <[email protected]>, "linux-nfs" <[email protected]>
> Sent: Thursday, May 7, 2015 7:04:58 PM
> Subject: 4.0 NFS client in infinite loop in state recovery after getting BAD_STATEID
> Hi folks,
>
> Problem:
> The upstream nfs4.0 client has problem where it will go into an
> infinite loop of re-sending an OPEN when it's trying to recover from
> receiving a BAD_STATEID error on an IO operation such READ or WRITE.
>
> How to easily reproduce (by using fault injection):
> 1. Do nfs4.0 mount to a server.
> 2. Open a file such that the server gives you a write delegation.
> 3. Do a write. Have a server return a BAD_STATEID. One way to do so is
> by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
> WRITE.
> 4. And off it goes with the loop.
>
> Here’s why….
>
> IO op like WRITE receives a BAD_STATEID.
> 1. for this error, in async handle error we call
> nfs4_schedule_stateid_recover()
> 2. that in turn will call nfs4_state_mark_reclaim_nograce() that will
> set a RECLAIM_NOGRACE in the state flags.
> 3. state manager thread will run and call nfs4_do_reclaim() to recover.
> 4. that will call nfs4_reclaim_open_state()
>
> in that function:
>
> restart:
> for open states in state
> test if RECLAIM_NOGRACE is set in state flags, if so clear it (it’s
> set and we’ll clear it)
> check open_stateid (checks if RECOVERY_FAILED is not set) (it’s not)
> checks if we have state
> calls ops->recover_open()
>
> for nfs4.0, it’ll call nfs40_open_expired()
> it’ll call nfs40_clear_delegation_stateid()
> it’ll call nfs_finish_clear_delegation_stateid()
> it’ll call nfs_remove_bad_delegation()
> it’ll call nfs_inode_find_state_and_recover()
> it’ll call nfs4_state_mark_reclaim_nograce() **** this will set
> RECLAIM_NOGRACE in state flags
>
> we return from recover_open() with status 0
> call nfs4_reclaim_locks() returns 0 then
> goto restart; ************** what happens is since we reset the flag
> in the state flags the whole loop starts again.
>
> Solution:
> nfs_remove_bad_delegation() is only called from
> nfs_finish_clear_delegation_stateid() which is called from either 4.0
> or 4.1 recover open functions in nograce case. In both cases, this is
> already state manager doing recovery based on the RECLAIM_NOGRACE flag
> set and it's going thru opens that need to be recovered.
>
> I propose to correct the loop by removing the call:
> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> index 4711d04..b322823 100644
> --- a/fs/nfs/delegation.c
> +++ b/fs/nfs/delegation.c
> @@ -632,10 +632,8 @@ void nfs_remove_bad_delegation(struct inode *inode)
>
> nfs_revoke_delegation(inode);
> delegation = nfs_inode_detach_delegation(inode);
> - if (delegation) {
> - nfs_inode_find_state_and_recover(inode, &delegation->stateid);
> + if (delegation)
> nfs_free_delegation(delegation);
> - }
> }
> EXPORT_SYMBOL_GPL(nfs_remove_bad_delegation);
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
> Hi Olga,
>
> I believe we see the same infinite loop without delegation with RHEL6/7
> kernels, but without delegation being involved. Currently, on the server
> side, if client looping is detected, we return RESOURCE. This breaks the
> loop and application gets an IO error.
>
> Your fix is only covers the delegation case isn't it?
>
> Tigran.
Tigran, do you have a BZ or can you tell me how to reproduce this?
Ben
Hi Ben,
probably you can simulate, as Olga has suggested, with fault injection.
I can I can prepare a snadalone version of your server, which returns
BAD_STATEID on any IO request.
Tigran.
----- Original Message -----
> From: "Benjamin Coddington" <[email protected]>
> To: "Mkrtchyan, Tigran" <[email protected]>
> Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust" <[email protected]>, "linux-nfs"
> <[email protected]>
> Sent: Friday, May 8, 2015 2:25:19 PM
> Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting BAD_STATEID
> On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
>
>> Hi Olga,
>>
>> I believe we see the same infinite loop without delegation with RHEL6/7
>> kernels, but without delegation being involved. Currently, on the server
>> side, if client looping is detected, we return RESOURCE. This breaks the
>> loop and application gets an IO error.
>>
>> Your fix is only covers the delegation case isn't it?
>>
>> Tigran.
>
> Tigran, do you have a BZ or can you tell me how to reproduce this?
>
> Ben
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Yes, I've done that, and the client's behavior correctly cycles through
OPEN, then READ with the new stateid. Are you able to create what Olga's
talking about -- which is (I believe) a loop of just OPENs?
Ben
On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
> Hi Ben,
>
> probably you can simulate, as Olga has suggested, with fault injection.
> I can I can prepare a snadalone version of your server, which returns
> BAD_STATEID on any IO request.
>
> Tigran.
>
> ----- Original Message -----
> > From: "Benjamin Coddington" <[email protected]>
> > To: "Mkrtchyan, Tigran" <[email protected]>
> > Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust" <[email protected]>, "linux-nfs"
> > <[email protected]>
> > Sent: Friday, May 8, 2015 2:25:19 PM
> > Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting BAD_STATEID
>
> > On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
> >
> >> Hi Olga,
> >>
> >> I believe we see the same infinite loop without delegation with RHEL6/7
> >> kernels, but without delegation being involved. Currently, on the server
> >> side, if client looping is detected, we return RESOURCE. This breaks the
> >> loop and application gets an IO error.
> >>
> >> Your fix is only covers the delegation case isn't it?
> >>
> >> Tigran.
> >
> > Tigran, do you have a BZ or can you tell me how to reproduce this?
> >
> > Ben
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to [email protected]
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
No, Olga's case includes delegation, which our server does not supports.
Tigran.
----- Original Message -----
> From: "Benjamin Coddington" <[email protected]>
> To: "Mkrtchyan, Tigran" <[email protected]>
> Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust" <[email protected]>, "linux-nfs"
> <[email protected]>
> Sent: Friday, May 8, 2015 3:08:03 PM
> Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting BAD_STATEID
> Yes, I've done that, and the client's behavior correctly cycles through
> OPEN, then READ with the new stateid. Are you able to create what Olga's
> talking about -- which is (I believe) a loop of just OPENs?
>
> Ben
>
> On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
>
>> Hi Ben,
>>
>> probably you can simulate, as Olga has suggested, with fault injection.
>> I can I can prepare a snadalone version of your server, which returns
>> BAD_STATEID on any IO request.
>>
>> Tigran.
>>
>> ----- Original Message -----
>> > From: "Benjamin Coddington" <[email protected]>
>> > To: "Mkrtchyan, Tigran" <[email protected]>
>> > Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust"
>> > <[email protected]>, "linux-nfs"
>> > <[email protected]>
>> > Sent: Friday, May 8, 2015 2:25:19 PM
>> > Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting
>> > BAD_STATEID
>>
>> > On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
>> >
>> >> Hi Olga,
>> >>
>> >> I believe we see the same infinite loop without delegation with RHEL6/7
>> >> kernels, but without delegation being involved. Currently, on the server
>> >> side, if client looping is detected, we return RESOURCE. This breaks the
>> >> loop and application gets an IO error.
>> >>
>> >> Your fix is only covers the delegation case isn't it?
>> >>
>> >> Tigran.
>> >
>> > Tigran, do you have a BZ or can you tell me how to reproduce this?
>> >
>> > Ben
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> > the body of a message to [email protected]
>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Tigran,
I think you are hitting something else as mine is as you mentioned has
to do with delegation. Also, RHEL6.6 code is very different from
RHEL7.1 (or upstream). I haven't tested my test case on RHEL6.6 but
just looking at it, I don't think it has the same OPEN loop problem.
Ben,
I didn't include this in my original message but we do have BZ open
for the problem by Jorge Mora,
Bug 1219184:
Infinite OPEN loop on NFSv4.0 when I/O receives NFS4ERR_BAD_STATEID
https://bugzilla.redhat.com/show_bug.cgi?id=1219184
On Fri, May 8, 2015 at 9:13 AM, Mkrtchyan, Tigran
<[email protected]> wrote:
> No, Olga's case includes delegation, which our server does not supports.
>
> Tigran.
>
> ----- Original Message -----
>> From: "Benjamin Coddington" <[email protected]>
>> To: "Mkrtchyan, Tigran" <[email protected]>
>> Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust" <[email protected]>, "linux-nfs"
>> <[email protected]>
>> Sent: Friday, May 8, 2015 3:08:03 PM
>> Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting BAD_STATEID
>
>> Yes, I've done that, and the client's behavior correctly cycles through
>> OPEN, then READ with the new stateid. Are you able to create what Olga's
>> talking about -- which is (I believe) a loop of just OPENs?
>>
>> Ben
>>
>> On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
>>
>>> Hi Ben,
>>>
>>> probably you can simulate, as Olga has suggested, with fault injection.
>>> I can I can prepare a snadalone version of your server, which returns
>>> BAD_STATEID on any IO request.
>>>
>>> Tigran.
>>>
>>> ----- Original Message -----
>>> > From: "Benjamin Coddington" <[email protected]>
>>> > To: "Mkrtchyan, Tigran" <[email protected]>
>>> > Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust"
>>> > <[email protected]>, "linux-nfs"
>>> > <[email protected]>
>>> > Sent: Friday, May 8, 2015 2:25:19 PM
>>> > Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting
>>> > BAD_STATEID
>>>
>>> > On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
>>> >
>>> >> Hi Olga,
>>> >>
>>> >> I believe we see the same infinite loop without delegation with RHEL6/7
>>> >> kernels, but without delegation being involved. Currently, on the server
>>> >> side, if client looping is detected, we return RESOURCE. This breaks the
>>> >> loop and application gets an IO error.
>>> >>
>>> >> Your fix is only covers the delegation case isn't it?
>>> >>
>>> >> Tigran.
>>> >
>>> > Tigran, do you have a BZ or can you tell me how to reproduce this?
>>> >
>>> > Ben
>>> > --
>>> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> > the body of a message to [email protected]
>>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, 8 May 2015, Olga Kornievskaia wrote:
> Hi Tigran,
>
> I think you are hitting something else as mine is as you mentioned has
> to do with delegation. Also, RHEL6.6 code is very different from
> RHEL7.1 (or upstream). I haven't tested my test case on RHEL6.6 but
> just looking at it, I don't think it has the same OPEN loop problem.
>
> Ben,
>
> I didn't include this in my original message but we do have BZ open
> for the problem by Jorge Mora,
> Bug 1219184:
> Infinite OPEN loop on NFSv4.0 when I/O receives NFS4ERR_BAD_STATEID
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1219184
Hi Olga,
Yep, I'm working that one. My ONTAP sims got purged, so I've been setting
them back up this morning.
Ben
> On Fri, May 8, 2015 at 9:13 AM, Mkrtchyan, Tigran
> <[email protected]> wrote:
> > No, Olga's case includes delegation, which our server does not supports.
> >
> > Tigran.
> >
> > ----- Original Message -----
> >> From: "Benjamin Coddington" <[email protected]>
> >> To: "Mkrtchyan, Tigran" <[email protected]>
> >> Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust" <[email protected]>, "linux-nfs"
> >> <[email protected]>
> >> Sent: Friday, May 8, 2015 3:08:03 PM
> >> Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting BAD_STATEID
> >
> >> Yes, I've done that, and the client's behavior correctly cycles through
> >> OPEN, then READ with the new stateid. Are you able to create what Olga's
> >> talking about -- which is (I believe) a loop of just OPENs?
> >>
> >> Ben
> >>
> >> On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
> >>
> >>> Hi Ben,
> >>>
> >>> probably you can simulate, as Olga has suggested, with fault injection.
> >>> I can I can prepare a snadalone version of your server, which returns
> >>> BAD_STATEID on any IO request.
> >>>
> >>> Tigran.
> >>>
> >>> ----- Original Message -----
> >>> > From: "Benjamin Coddington" <[email protected]>
> >>> > To: "Mkrtchyan, Tigran" <[email protected]>
> >>> > Cc: "Olga Kornievskaia" <[email protected]>, "Trond Myklebust"
> >>> > <[email protected]>, "linux-nfs"
> >>> > <[email protected]>
> >>> > Sent: Friday, May 8, 2015 2:25:19 PM
> >>> > Subject: Re: 4.0 NFS client in infinite loop in state recovery after getting
> >>> > BAD_STATEID
> >>>
> >>> > On Fri, 8 May 2015, Mkrtchyan, Tigran wrote:
> >>> >
> >>> >> Hi Olga,
> >>> >>
> >>> >> I believe we see the same infinite loop without delegation with RHEL6/7
> >>> >> kernels, but without delegation being involved. Currently, on the server
> >>> >> side, if client looping is detected, we return RESOURCE. This breaks the
> >>> >> loop and application gets an IO error.
> >>> >>
> >>> >> Your fix is only covers the delegation case isn't it?
> >>> >>
> >>> >> Tigran.
> >>> >
> >>> > Tigran, do you have a BZ or can you tell me how to reproduce this?
> >>> >
> >>> > Ben
> >>> > --
> >>> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>> > the body of a message to [email protected]
> >>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >>> the body of a message to [email protected]
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >> the body of a message to [email protected]
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
Here's another version:
Problem: When an operation like WRITE receives a BAD_STATEID, even though
recovery code clears the RECLAIM_NOGRACE recovery flag before recovering
the open state, because of clearing delegation state for the associated
inode, nfs_inode_find_state_and_recover() gets called and it marks the
same state with RECLAIM_NOGRACE flag again. As a results, when we restart
looking over the open states, we end up in the infinite loop instead of
breaking out in the next test of state flags.
Solution: unset the RECLAIM_NOGRACE set because of
calling of nfs_inode_find_state_and_recover() after returning from calling
recover_open() function.
Signed-off-by: Olga Kornievskaia <[email protected]>
---
fs/nfs/nfs4state.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 2782cfc..e99e1f0 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1482,6 +1482,8 @@ restart:
spin_unlock(&state->state_lock);
}
nfs4_put_open_state(state);
+ test_and_clear_bit(NFS4CLNT_RECLAIM_NOGRACE,
+ &state->flags);
spin_lock(&sp->so_lock);
goto restart;
}
--
On Thu, May 7, 2015 at 1:04 PM, Olga Kornievskaia <[email protected]> wrote:
> Hi folks,
>
> Problem:
> The upstream nfs4.0 client has problem where it will go into an
> infinite loop of re-sending an OPEN when it's trying to recover from
> receiving a BAD_STATEID error on an IO operation such READ or WRITE.
>
> How to easily reproduce (by using fault injection):
> 1. Do nfs4.0 mount to a server.
> 2. Open a file such that the server gives you a write delegation.
> 3. Do a write. Have a server return a BAD_STATEID. One way to do so is
> by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
> WRITE.
> 4. And off it goes with the loop.
>
> Here’s why….
>
> IO op like WRITE receives a BAD_STATEID.
> 1. for this error, in async handle error we call
> nfs4_schedule_stateid_recover()
> 2. that in turn will call nfs4_state_mark_reclaim_nograce() that will
> set a RECLAIM_NOGRACE in the state flags.
> 3. state manager thread will run and call nfs4_do_reclaim() to recover.
> 4. that will call nfs4_reclaim_open_state()
>
> in that function:
>
> restart:
> for open states in state
> test if RECLAIM_NOGRACE is set in state flags, if so clear it (it’s
> set and we’ll clear it)
> check open_stateid (checks if RECOVERY_FAILED is not set) (it’s not)
> checks if we have state
> calls ops->recover_open()
>
> for nfs4.0, it’ll call nfs40_open_expired()
> it’ll call nfs40_clear_delegation_stateid()
> it’ll call nfs_finish_clear_delegation_stateid()
> it’ll call nfs_remove_bad_delegation()
> it’ll call nfs_inode_find_state_and_recover()
> it’ll call nfs4_state_mark_reclaim_nograce() **** this will set
> RECLAIM_NOGRACE in state flags
>
> we return from recover_open() with status 0
> call nfs4_reclaim_locks() returns 0 then
> goto restart; ************** what happens is since we reset the flag
> in the state flags the whole loop starts again.
>
> Solution:
> nfs_remove_bad_delegation() is only called from
> nfs_finish_clear_delegation_stateid() which is called from either 4.0
> or 4.1 recover open functions in nograce case. In both cases, this is
> already state manager doing recovery based on the RECLAIM_NOGRACE flag
> set and it's going thru opens that need to be recovered.
>
> I propose to correct the loop by removing the call:
> diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
> index 4711d04..b322823 100644
> --- a/fs/nfs/delegation.c
> +++ b/fs/nfs/delegation.c
> @@ -632,10 +632,8 @@ void nfs_remove_bad_delegation(struct inode *inode)
>
> nfs_revoke_delegation(inode);
> delegation = nfs_inode_detach_delegation(inode);
> - if (delegation) {
> - nfs_inode_find_state_and_recover(inode, &delegation->stateid);
> + if (delegation)
> nfs_free_delegation(delegation);
> - }
> }
> EXPORT_SYMBOL_GPL(nfs_remove_bad_delegation);
On Thu, 7 May 2015, Olga Kornievskaia wrote:
> Hi folks,
>
> Problem:
> The upstream nfs4.0 client has problem where it will go into an
> infinite loop of re-sending an OPEN when it's trying to recover from
> receiving a BAD_STATEID error on an IO operation such READ or WRITE.
>
> How to easily reproduce (by using fault injection):
> 1. Do nfs4.0 mount to a server.
> 2. Open a file such that the server gives you a write delegation.
> 3. Do a write. Have a server return a BAD_STATEID. One way to do so is
> by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
> WRITE.
> 4. And off it goes with the loop.
Hi Olga,
I've been trying to reproduce it, and I'm frustratingly unable. It sounds
fairly easy to produce.. What version of the client produces this?
Ben
On Fri, May 29, 2015 at 9:44 AM, Benjamin Coddington
<[email protected]> wrote:
> On Thu, 7 May 2015, Olga Kornievskaia wrote:
>
>> Hi folks,
>>
>> Problem:
>> The upstream nfs4.0 client has problem where it will go into an
>> infinite loop of re-sending an OPEN when it's trying to recover from
>> receiving a BAD_STATEID error on an IO operation such READ or WRITE.
>>
>> How to easily reproduce (by using fault injection):
>> 1. Do nfs4.0 mount to a server.
>> 2. Open a file such that the server gives you a write delegation.
>> 3. Do a write. Have a server return a BAD_STATEID. One way to do so is
>> by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
>> WRITE.
>> 4. And off it goes with the loop.
>
> Hi Olga,
>
> I've been trying to reproduce it, and I'm frustratingly unable. It sounds
> fairly easy to produce.. What version of the client produces this?
>
Hi Ben,
Problem exists in the upstream kernels as well but we noticed the
problem on RHEL7.1 distro (RedHat's 2.6.32-229.el7 kernel I think).
> Ben
I meant to say 3.10.0-229. Mixing my RHEL6 and RHEL7 prefixes...
On Fri, May 29, 2015 at 12:51 PM, Olga Kornievskaia <[email protected]> wrote:
> On Fri, May 29, 2015 at 9:44 AM, Benjamin Coddington
> <[email protected]> wrote:
>> On Thu, 7 May 2015, Olga Kornievskaia wrote:
>>
>>> Hi folks,
>>>
>>> Problem:
>>> The upstream nfs4.0 client has problem where it will go into an
>>> infinite loop of re-sending an OPEN when it's trying to recover from
>>> receiving a BAD_STATEID error on an IO operation such READ or WRITE.
>>>
>>> How to easily reproduce (by using fault injection):
>>> 1. Do nfs4.0 mount to a server.
>>> 2. Open a file such that the server gives you a write delegation.
>>> 3. Do a write. Have a server return a BAD_STATEID. One way to do so is
>>> by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
>>> WRITE.
>>> 4. And off it goes with the loop.
>>
>> Hi Olga,
>>
>> I've been trying to reproduce it, and I'm frustratingly unable. It sounds
>> fairly easy to produce.. What version of the client produces this?
>>
>
> Hi Ben,
>
> Problem exists in the upstream kernels as well but we noticed the
> problem on RHEL7.1 distro (RedHat's 2.6.32-229.el7 kernel I think).
>
>> Ben
On Fri, 29 May 2015, Olga Kornievskaia wrote:
> I meant to say 3.10.0-229. Mixing my RHEL6 and RHEL7 prefixes...
>
> On Fri, May 29, 2015 at 12:51 PM, Olga Kornievskaia <[email protected]> wrote:
> > On Fri, May 29, 2015 at 9:44 AM, Benjamin Coddington
> > <[email protected]> wrote:
> >> On Thu, 7 May 2015, Olga Kornievskaia wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Problem:
> >>> The upstream nfs4.0 client has problem where it will go into an
> >>> infinite loop of re-sending an OPEN when it's trying to recover from
> >>> receiving a BAD_STATEID error on an IO operation such READ or WRITE.
> >>>
> >>> How to easily reproduce (by using fault injection):
> >>> 1. Do nfs4.0 mount to a server.
> >>> 2. Open a file such that the server gives you a write delegation.
> >>> 3. Do a write. Have a server return a BAD_STATEID. One way to do so is
> >>> by using a python proxy, nfs4proxy, and inject BAD_STATEID error on
> >>> WRITE.
> >>> 4. And off it goes with the loop.
> >>
> >> Hi Olga,
> >>
> >> I've been trying to reproduce it, and I'm frustratingly unable. It sounds
> >> fairly easy to produce.. What version of the client produces this?
> >>
> >
> > Hi Ben,
> >
> > Problem exists in the upstream kernels as well but we noticed the
> > problem on RHEL7.1 distro (RedHat's 2.6.32-229.el7 kernel I think).
I've now been able to reproduce this upstream, and 7.1.. and just today on a
6.7 client. The 6.7 client seems to self-limit the OPEN storm to around 7k
OPENs.. That's interesting enough to look into further..
Taking this issue to the BZs for now, just wanted to let the list know that
we see this now, too.
Ben