2019-11-29 14:13:23

by Vasily Averin

Subject: unsafe req->rq_xprt using inside bc_svc_process() ?

OpenVz team got complain on crash in bc_svc_process().
Crashed node had 15 running containers with active nfsv4.1 mounts,
single nfsv4.1-svc thread was processed its back-channel requests.
In our case nfs41_callback_svc() took rpc_rqst *req from serv->sv_cb_list
started its processing but found that req->rq_xprt points to already freed
struct rpc_xprt aka part of struct sock_xprt transport.

Back-channel request was submitted via xprt_complete_bc_request(),
its processing uses req->rq_xprt reference in many times,
however I did not found who keeps this reference.
It seems sock_xprt or even whole its net namespace can be freed before bc_svc_process() will start processing of submitted back-channel request, and req->rq_xprt using is unsafe in bc_svc_process()

Am I missed something probably?

Thank you,
Vasily Averin

2019-12-05 08:40:46

by Vasily Averin

Subject: Re: unsafe req->rq_xprt using inside bc_svc_process() ?

I've found that Trond fixed it recently in commit 875f0706acc
"SUNRPC: The TCP back channel mustn't disappear while requests are outstanding"

On 11/29/19 5:10 PM, Vasily Averin wrote:
> OpenVz team got complain on crash in bc_svc_process().
> Crashed node had 15 running containers with active nfsv4.1 mounts,
> single nfsv4.1-svc thread was processed its back-channel requests.
> In our case nfs41_callback_svc() took rpc_rqst *req from serv->sv_cb_list
> started its processing but found that req->rq_xprt points to already freed
> struct rpc_xprt aka part of struct sock_xprt transport.
>
> Back-channel request was submitted via xprt_complete_bc_request(),
> its processing uses req->rq_xprt reference in many times,
> however I did not found who keeps this reference.
> It seems sock_xprt or even whole its net namespace can be freed before bc_svc_process() will start processing of submitted back-channel request, and req->rq_xprt using is unsafe in bc_svc_process()
>
> Am I missed something probably?

2019-12-06 21:21:15

by J. Bruce Fields

Subject: Re: unsafe req->rq_xprt using inside bc_svc_process() ?

On Thu, Dec 05, 2019 at 11:40:04AM +0300, Vasily Averin wrote:
> I've found that Trond fixed it recently in commit 875f0706acc
> "SUNRPC: The TCP back channel mustn't disappear while requests are outstanding"

Thanks for following up!--b.

>
> On 11/29/19 5:10 PM, Vasily Averin wrote:
> >OpenVz team got complain on crash in bc_svc_process().
> >Crashed node had 15 running containers with active nfsv4.1 mounts,
> >single nfsv4.1-svc thread was processed its back-channel requests.
> >In our case nfs41_callback_svc() took rpc_rqst *req from serv->sv_cb_list
> >started its processing but found that req->rq_xprt points to already freed
> >struct rpc_xprt aka part of struct sock_xprt transport.
> >
> >Back-channel request was submitted via xprt_complete_bc_request(),
> >its processing uses req->rq_xprt reference in many times,
> >however I did not found who keeps this reference.
> >It seems sock_xprt or even whole its net namespace can be freed before bc_svc_process() will start processing of submitted back-channel request, and req->rq_xprt using is unsafe in bc_svc_process()
> >
> >Am I missed something probably?