2021-04-16 15:55:25

by Pradeep P V K

[permalink] [raw]
Subject: [PATCH V1] fuse: Set fuse request error upon fuse abort connection

There is a minor race in setting the fuse out request error
between fuse_abort_conn() and fuse_dev_do_read() as explained
below.

Thread-1 Thread-2
======== ========
->fuse_simple_request() ->shutdown
->__fuse_request_send()
->queue_request() ->fuse_abort_conn()
->fuse_dev_do_read() ->acquire(fpq->lock)
->wait_for(fpq->lock) ->set err to all req's in fpq->io
->release(fpq->lock)
->acquire(fpq->lock)
->add req to fpq->io

The above scenario may cause Thread-1 request to add into
fpq->io list after Thread-2 sets -ECONNABORTED err to all
its requests in fpq->io list. This leaves Thread-1 request
with unset err and this further misleads as a completed
request without an err set upon request_end().

Handle this by setting the err appropriately.

Signed-off-by: Pradeep P V K <[email protected]>
---
fs/fuse/dev.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index a5ceccc..102c56f 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1283,6 +1283,7 @@ static ssize_t fuse_dev_do_read(struct fuse_dev *fud, struct file *file,
clear_bit(FR_LOCKED, &req->flags);
if (!fpq->connected) {
err = fc->aborted ? -ECONNABORTED : -ENODEV;
+ req->out.h.error = err;
goto out_end;
}
if (err) {
--
2.7.4


2021-05-10 07:56:05

by Pradeep P V K

[permalink] [raw]
Subject: Re: [PATCH V1] fuse: Set fuse request error upon fuse abort connection

Hi Miklos,

Did you get a chance to look on the below change ?
could you please review and provide your comments if any.

Thanks and Regards,
Pradeep

On 2021-04-16 19:49, Pradeep P V K wrote:
> There is a minor race in setting the fuse out request error
> between fuse_abort_conn() and fuse_dev_do_read() as explained
> below.
>
> Thread-1 Thread-2
> ======== ========
> ->fuse_simple_request() ->shutdown
> ->__fuse_request_send()
> ->queue_request() ->fuse_abort_conn()
> ->fuse_dev_do_read() ->acquire(fpq->lock)
> ->wait_for(fpq->lock) ->set err to all req's in fpq->io
> ->release(fpq->lock)
> ->acquire(fpq->lock)
> ->add req to fpq->io
>
> The above scenario may cause Thread-1 request to add into
> fpq->io list after Thread-2 sets -ECONNABORTED err to all
> its requests in fpq->io list. This leaves Thread-1 request
> with unset err and this further misleads as a completed
> request without an err set upon request_end().
>
> Handle this by setting the err appropriately.
>
> Signed-off-by: Pradeep P V K <[email protected]>
> ---
> fs/fuse/dev.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
> index a5ceccc..102c56f 100644
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -1283,6 +1283,7 @@ static ssize_t fuse_dev_do_read(struct fuse_dev
> *fud, struct file *file,
> clear_bit(FR_LOCKED, &req->flags);
> if (!fpq->connected) {
> err = fc->aborted ? -ECONNABORTED : -ENODEV;
> + req->out.h.error = err;
> goto out_end;
> }
> if (err) {

2021-05-10 14:50:39

by Miklos Szeredi

[permalink] [raw]
Subject: Re: [PATCH V1] fuse: Set fuse request error upon fuse abort connection

On Mon, May 10, 2021 at 9:52 AM <[email protected]> wrote:
>
> Hi Miklos,
>
> Did you get a chance to look on the below change ?
> could you please review and provide your comments if any.

Hi Pradeep,

Patch looks good at first glance. Will review properly.

Thanks,
Miklos

2021-06-21 07:50:08

by Miklos Szeredi

[permalink] [raw]
Subject: Re: [PATCH V1] fuse: Set fuse request error upon fuse abort connection

On Fri, Apr 16, 2021 at 07:49:12PM +0530, Pradeep P V K wrote:
> There is a minor race in setting the fuse out request error
> between fuse_abort_conn() and fuse_dev_do_read() as explained
> below.
>
> Thread-1 Thread-2
> ======== ========
> ->fuse_simple_request() ->shutdown
> ->__fuse_request_send()
> ->queue_request() ->fuse_abort_conn()
> ->fuse_dev_do_read() ->acquire(fpq->lock)
> ->wait_for(fpq->lock) ->set err to all req's in fpq->io
> ->release(fpq->lock)
> ->acquire(fpq->lock)
> ->add req to fpq->io
>
> The above scenario may cause Thread-1 request to add into
> fpq->io list after Thread-2 sets -ECONNABORTED err to all
> its requests in fpq->io list. This leaves Thread-1 request
> with unset err and this further misleads as a completed
> request without an err set upon request_end().
>
> Handle this by setting the err appropriately.

The fix looks good, but still allows the request to block during the copy phase
after being aborted, which the FR_LOCKED/FR_ABORT bits are meant to prevent.

Here's an updated fix. It does not allow the request to be queued on the
fpq->io list after fuse_abort_conn() has aborted requests on that list.

Can you verify that it fixes the race you reported?

Thanks,
Miklos


--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1272,6 +1272,15 @@ static ssize_t fuse_dev_do_read(struct f
goto restart;
}
spin_lock(&fpq->lock);
+ /*
+ * Must not put request on fpq->io queue after having been shut down by
+ * fuse_abort_conn()
+ */
+ if (!fpq->connected) {
+ req->out.h.error = err = -ECONNABORTED;
+ goto out_end;
+
+ }
list_add(&req->list, &fpq->io);
spin_unlock(&fpq->lock);
cs->req = req;