2012-12-26 15:17:07

by Andriy Skulysh

Subject: [PATCH 1/1] sunrpc: Fix lockd sleeping until timeout

There is a race in enqueueing thread to a pool and
waking up a thread.
lockd doesn't wake up on reception of lock granted callback
if svc_wake_up() is called before lockd's thread is added
to a pool.

Signed-off-by: Andriy Skulysh <[email protected]>
---
include/linux/sunrpc/svc.h | 1 +
net/sunrpc/svc_xprt.c | 9 ++++++++-
2 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h
index 676ddf5..1f0216b 100644
--- a/include/linux/sunrpc/svc.h
+++ b/include/linux/sunrpc/svc.h
@@ -50,6 +50,7 @@ struct svc_pool {
unsigned int sp_nrthreads; /* # of threads in pool */
struct list_head sp_all_threads; /* all server threads */
struct svc_pool_stats sp_stats; /* statistics on pool operation */
+ int sp_task_pending;/* has pending task */
} ____cacheline_aligned_in_smp;

/*
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index b8e47fa..c7ab6f5 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -499,7 +499,8 @@ void svc_wake_up(struct svc_serv *serv)
rqstp->rq_xprt = NULL;
*/
wake_up(&rqstp->rq_wait);
- }
+ } else
+ pool->sp_task_pending = 1;
spin_unlock_bh(&pool->sp_lock);
}
}
@@ -634,7 +635,13 @@ struct svc_xprt *svc_get_next_xprt(struct
svc_rqst *rqstp, long timeout)
* long for cache updates.
*/
rqstp->rq_chandle.thread_wait = 1*HZ;
+ pool->sp_task_pending = 0;
} else {
+ if (pool->sp_task_pending) {
+ pool->sp_task_pending = 0;
+ spin_unlock_bh(&pool->sp_lock);
+ return -EAGAIN;
+ }
/* No data pending. Go to sleep */
svc_thread_enqueue(pool, rqstp);

--
1.7.1

2013-01-04 00:53:48

by J. Bruce Fields

Subject: Re: [PATCH 1/1] sunrpc: Fix lockd sleeping until timeout

On Wed, Dec 26, 2012 at 05:09:07PM +0200, Andriy Skulysh wrote:
> There is a race in enqueueing thread to a pool and
> waking up a thread.
> lockd doesn't wake up on reception of lock granted callback
> if svc_wake_up() is called before lockd's thread is added
> to a pool.
>
> Signed-off-by: Andriy Skulysh <[email protected]>
> ---
> include/linux/sunrpc/svc.h | 1 +
> net/sunrpc/svc_xprt.c | 9 ++++++++-
> 2 files changed, 9 insertions(+), 1 deletions(-)
>
> diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h
> index 676ddf5..1f0216b 100644
> --- a/include/linux/sunrpc/svc.h
> +++ b/include/linux/sunrpc/svc.h
> @@ -50,6 +50,7 @@ struct svc_pool {
> unsigned int sp_nrthreads; /* # of threads in pool */
> struct list_head sp_all_threads; /* all server threads */
> struct svc_pool_stats sp_stats; /* statistics on pool operation */
> + int sp_task_pending;/* has pending task */
> } ____cacheline_aligned_in_smp;
>
> /*
> diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> index b8e47fa..c7ab6f5 100644
> --- a/net/sunrpc/svc_xprt.c
> +++ b/net/sunrpc/svc_xprt.c
> @@ -499,7 +499,8 @@ void svc_wake_up(struct svc_serv *serv)
> rqstp->rq_xprt = NULL;
> */
> wake_up(&rqstp->rq_wait);
> - }
> + } else
> + pool->sp_task_pending = 1;
> spin_unlock_bh(&pool->sp_lock);
> }
> }
> @@ -634,7 +635,13 @@ struct svc_xprt *svc_get_next_xprt(struct
> svc_rqst *rqstp, long timeout)
> * long for cache updates.
> */
> rqstp->rq_chandle.thread_wait = 1*HZ;
> + pool->sp_task_pending = 0;
> } else {
> + if (pool->sp_task_pending) {
> + pool->sp_task_pending = 0;
> + spin_unlock_bh(&pool->sp_lock);
> + return -EAGAIN;

That should be ERR_PTR(-EAGAIN).

Other than this this looks right to me....

Out of curiosity: how did you run across this problem, and how did you
test the fix?

--b.

> + }
> /* No data pending. Go to sleep */
> svc_thread_enqueue(pool, rqstp);
>
> --
> 1.7.1

2013-01-08 21:47:45

by J. Bruce Fields

Subject: Re: [PATCH 1/1] sunrpc: Fix lockd sleeping until timeout

On Mon, Jan 07, 2013 at 12:12:15AM +0200, Andriy Skulysh wrote:
> There is a race in enqueueing thread to a pool and
> waking up a thread.
> lockd doesn't wake up on reception of lock granted callback
> if svc_wake_up() is called before lockd's thread is added
> to a pool.
>
> Signed-off-by: Andriy Skulysh <[email protected]>
> ---
> include/linux/sunrpc/svc.h | 1 +
> net/sunrpc/svc_xprt.c | 9 ++++++++-
> 2 files changed, 9 insertions(+), 1 deletions(-)
>
> diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h
> index 676ddf5..1f0216b 100644
> --- a/include/linux/sunrpc/svc.h
> +++ b/include/linux/sunrpc/svc.h
> @@ -50,6 +50,7 @@ struct svc_pool {
> unsigned int sp_nrthreads; /* # of threads in pool */
> struct list_head sp_all_threads; /* all server threads */
> struct svc_pool_stats sp_stats; /* statistics on pool operation */
> + int sp_task_pending;/* has pending task */
> } ____cacheline_aligned_in_smp;
>
> /*
> diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
> index b8e47fa..5a9d40c 100644
> --- a/net/sunrpc/svc_xprt.c
> +++ b/net/sunrpc/svc_xprt.c
> @@ -499,7 +499,8 @@ void svc_wake_up(struct svc_serv *serv)
> rqstp->rq_xprt = NULL;
> */
> wake_up(&rqstp->rq_wait);
> - }
> + } else
> + pool->sp_task_pending = 1;
> spin_unlock_bh(&pool->sp_lock);
> }
> }
> @@ -634,7 +635,13 @@ struct svc_xprt *svc_get_next_xprt(struct
> svc_rqst *rqstp, long timeout)
> * long for cache updates.
> */
> rqstp->rq_chandle.thread_wait = 1*HZ;
> + pool->sp_task_pending = 0;
> } else {
> + if (pool->sp_task_pending) {
> + pool->sp_task_pending = 0;
> + spin_unlock_bh(&pool->sp_lock);
> + return ERR_PTR(-EAGAIN);
> + }
> /* No data pending. Go to sleep */
> svc_thread_enqueue(pool, rqstp);
>
> --
> 1.7.1
>
>
> On 4 January 2013 02:53, J. Bruce Fields <[email protected]> wrote:
> > That should be ERR_PTR(-EAGAIN).
> fixed.
> >
> > Other than this this looks right to me....
> >
> > Out of curiosity: how did you run across this problem, and how did you
> > test the fix?
> I can reproduce it with single ping_pong with nfs on top of Lustre filesystem.

Got it, thanks; applying for 3.9.--b.

2013-01-06 22:12:16

by Andriy Skulysh

Subject: Re: [PATCH 1/1] sunrpc: Fix lockd sleeping until timeout

There is a race in enqueueing thread to a pool and
waking up a thread.
lockd doesn't wake up on reception of lock granted callback
if svc_wake_up() is called before lockd's thread is added
to a pool.

Signed-off-by: Andriy Skulysh <[email protected]>
---
include/linux/sunrpc/svc.h | 1 +
net/sunrpc/svc_xprt.c | 9 ++++++++-
2 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h
index 676ddf5..1f0216b 100644
--- a/include/linux/sunrpc/svc.h
+++ b/include/linux/sunrpc/svc.h
@@ -50,6 +50,7 @@ struct svc_pool {
unsigned int sp_nrthreads; /* # of threads in pool */
struct list_head sp_all_threads; /* all server threads */
struct svc_pool_stats sp_stats; /* statistics on pool operation */
+ int sp_task_pending;/* has pending task */
} ____cacheline_aligned_in_smp;

/*
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index b8e47fa..5a9d40c 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -499,7 +499,8 @@ void svc_wake_up(struct svc_serv *serv)
rqstp->rq_xprt = NULL;
*/
wake_up(&rqstp->rq_wait);
- }
+ } else
+ pool->sp_task_pending = 1;
spin_unlock_bh(&pool->sp_lock);
}
}
@@ -634,7 +635,13 @@ struct svc_xprt *svc_get_next_xprt(struct
svc_rqst *rqstp, long timeout)
* long for cache updates.
*/
rqstp->rq_chandle.thread_wait = 1*HZ;
+ pool->sp_task_pending = 0;
} else {
+ if (pool->sp_task_pending) {
+ pool->sp_task_pending = 0;
+ spin_unlock_bh(&pool->sp_lock);
+ return ERR_PTR(-EAGAIN);
+ }
/* No data pending. Go to sleep */
svc_thread_enqueue(pool, rqstp);

--
1.7.1


On 4 January 2013 02:53, J. Bruce Fields <[email protected]> wrote:
> That should be ERR_PTR(-EAGAIN).
fixed.
>
> Other than this this looks right to me....
>
> Out of curiosity: how did you run across this problem, and how did you
> test the fix?
I can reproduce it with single ping_pong with nfs on top of Lustre filesystem.

Andriy.