2022-06-30 17:28:45

by Chuck Lever III

[permalink] [raw]
Subject: [PATCH v1] SUNRPC: Fix READ_PLUS crasher

Looks like there are still cases when "space_left - frag1bytes" can
legitimately exceed PAGE_SIZE. Ensure that xdr->end always remains
within the current encode buffer.

Reported-by: Bruce Fields <[email protected]>
Reported-by: Zorro Lang <[email protected]>
Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in xdr_get_next_encode_buffer()")
Signed-off-by: Chuck Lever <[email protected]>
---
net/sunrpc/xdr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Hi-

I had a few minutes yesterday afternoon to look into this one. The
following one-liner seems to address the issue and passes my smoke
tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?


diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index f87a2d8f23a7..916659be2774 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -987,7 +987,7 @@ static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
if (space_left - nbytes >= PAGE_SIZE)
xdr->end = p + PAGE_SIZE;
else
- xdr->end = p + space_left - frag1bytes;
+ xdr->end = p + min_t(int, space_left - frag1bytes, PAGE_SIZE);

xdr->buf->page_len += frag2bytes;
xdr->buf->len += nbytes;



2022-06-30 18:03:39

by Trond Myklebust

[permalink] [raw]
Subject: Re: [PATCH v1] SUNRPC: Fix READ_PLUS crasher

On Thu, 2022-06-30 at 13:24 -0400, Chuck Lever wrote:
> Looks like there are still cases when "space_left - frag1bytes" can
> legitimately exceed PAGE_SIZE. Ensure that xdr->end always remains
> within the current encode buffer.
>
> Reported-by: Bruce Fields <[email protected]>
> Reported-by: Zorro Lang <[email protected]>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
> Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in
> xdr_get_next_encode_buffer()")
> Signed-off-by: Chuck Lever <[email protected]>
> ---
>  net/sunrpc/xdr.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> Hi-
>
> I had a few minutes yesterday afternoon to look into this one. The
> following one-liner seems to address the issue and passes my smoke
> tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?
>
>
> diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
> index f87a2d8f23a7..916659be2774 100644
> --- a/net/sunrpc/xdr.c
> +++ b/net/sunrpc/xdr.c
> @@ -987,7 +987,7 @@ static noinline __be32
> *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
>         if (space_left - nbytes >= PAGE_SIZE)
>                 xdr->end = p + PAGE_SIZE;
>         else
> -               xdr->end = p + space_left - frag1bytes;
> +               xdr->end = p + min_t(int, space_left - frag1bytes,
> PAGE_SIZE);

Since we know that frag1bytes <= nbytes (that is determined in
xdr_reserve_space()), isn't this effectively the same thing as changing
the condition to

if (space_left - frag1bytes >= PAGE_SIZE)
xdr->end = p + PAGE_SIZE;
else
xdr->end = p + space_left - frag1bytes;

?
>  
>         xdr->buf->page_len += frag2bytes;
>         xdr->buf->len += nbytes;
>
>

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]


2022-06-30 18:14:00

by Anna Schumaker

[permalink] [raw]
Subject: Re: [PATCH v1] SUNRPC: Fix READ_PLUS crasher

On Thu, Jun 30, 2022 at 2:03 PM Trond Myklebust <[email protected]> wrote:
>
> On Thu, 2022-06-30 at 13:24 -0400, Chuck Lever wrote:
> > Looks like there are still cases when "space_left - frag1bytes" can
> > legitimately exceed PAGE_SIZE. Ensure that xdr->end always remains
> > within the current encode buffer.
> >
> > Reported-by: Bruce Fields <[email protected]>
> > Reported-by: Zorro Lang <[email protected]>
> > Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
> > Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in
> > xdr_get_next_encode_buffer()")
> > Signed-off-by: Chuck Lever <[email protected]>
> > ---
> > net/sunrpc/xdr.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > Hi-
> >
> > I had a few minutes yesterday afternoon to look into this one. The
> > following one-liner seems to address the issue and passes my smoke
> > tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?
> >
> >
> > diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
> > index f87a2d8f23a7..916659be2774 100644
> > --- a/net/sunrpc/xdr.c
> > +++ b/net/sunrpc/xdr.c
> > @@ -987,7 +987,7 @@ static noinline __be32
> > *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
> > if (space_left - nbytes >= PAGE_SIZE)
> > xdr->end = p + PAGE_SIZE;
> > else
> > - xdr->end = p + space_left - frag1bytes;
> > + xdr->end = p + min_t(int, space_left - frag1bytes,
> > PAGE_SIZE);
>
> Since we know that frag1bytes <= nbytes (that is determined in
> xdr_reserve_space()), isn't this effectively the same thing as changing
> the condition to
>
> if (space_left - frag1bytes >= PAGE_SIZE)
> xdr->end = p + PAGE_SIZE;
> else
> xdr->end = p + space_left - frag1bytes;

I added some printk's without this patch, and I'm seeing space_left
larger than PAGE_SIZE and frag1bytes set to 0 in that branch right
before the crash. So the min_t() will choose the PAGE_SIZE option
sometimes.

Anna

>
> ?
> >
> > xdr->buf->page_len += frag2bytes;
> > xdr->buf->len += nbytes;
> >
> >
>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [email protected]
>
>

2022-06-30 18:34:14

by Anna Schumaker

[permalink] [raw]
Subject: Re: [PATCH v1] SUNRPC: Fix READ_PLUS crasher

On Thu, Jun 30, 2022 at 2:09 PM Anna Schumaker <[email protected]> wrote:
>
> On Thu, Jun 30, 2022 at 2:03 PM Trond Myklebust <[email protected]> wrote:
> >
> > On Thu, 2022-06-30 at 13:24 -0400, Chuck Lever wrote:
> > > Looks like there are still cases when "space_left - frag1bytes" can
> > > legitimately exceed PAGE_SIZE. Ensure that xdr->end always remains
> > > within the current encode buffer.
> > >
> > > Reported-by: Bruce Fields <[email protected]>
> > > Reported-by: Zorro Lang <[email protected]>
> > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
> > > Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in
> > > xdr_get_next_encode_buffer()")
> > > Signed-off-by: Chuck Lever <[email protected]>
> > > ---
> > > net/sunrpc/xdr.c | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > Hi-
> > >
> > > I had a few minutes yesterday afternoon to look into this one. The
> > > following one-liner seems to address the issue and passes my smoke
> > > tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?
> > >
> > >
> > > diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
> > > index f87a2d8f23a7..916659be2774 100644
> > > --- a/net/sunrpc/xdr.c
> > > +++ b/net/sunrpc/xdr.c
> > > @@ -987,7 +987,7 @@ static noinline __be32
> > > *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
> > > if (space_left - nbytes >= PAGE_SIZE)
> > > xdr->end = p + PAGE_SIZE;
> > > else
> > > - xdr->end = p + space_left - frag1bytes;
> > > + xdr->end = p + min_t(int, space_left - frag1bytes,
> > > PAGE_SIZE);
> >
> > Since we know that frag1bytes <= nbytes (that is determined in
> > xdr_reserve_space()), isn't this effectively the same thing as changing
> > the condition to
> >
> > if (space_left - frag1bytes >= PAGE_SIZE)
> > xdr->end = p + PAGE_SIZE;
> > else
> > xdr->end = p + space_left - frag1bytes;
>
> I added some printk's without this patch, and I'm seeing space_left
> larger than PAGE_SIZE and frag1bytes set to 0 in that branch right
> before the crash. So the min_t() will choose the PAGE_SIZE option
> sometimes.

Actually, ignore me. I initially misread what Trond said. After a
reread I saw he changed "space_left - nbytes" to "space_left -
frag1bytes" in the if condition. Trond's way fixes the crash for me,
too.

Anna
>
> Anna
>
> >
> > ?
> > >
> > > xdr->buf->page_len += frag2bytes;
> > > xdr->buf->len += nbytes;
> > >
> > >
> >
> > --
> > Trond Myklebust
> > Linux NFS client maintainer, Hammerspace
> > [email protected]
> >
> >

2022-06-30 18:53:49

by Trond Myklebust

[permalink] [raw]
Subject: Re: [PATCH v1] SUNRPC: Fix READ_PLUS crasher

On Thu, 2022-06-30 at 14:09 -0400, Anna Schumaker wrote:
> On Thu, Jun 30, 2022 at 2:03 PM Trond Myklebust
> <[email protected]> wrote:
> >
> > On Thu, 2022-06-30 at 13:24 -0400, Chuck Lever wrote:
> > > Looks like there are still cases when "space_left - frag1bytes"
> > > can
> > > legitimately exceed PAGE_SIZE. Ensure that xdr->end always
> > > remains
> > > within the current encode buffer.
> > >
> > > Reported-by: Bruce Fields <[email protected]>
> > > Reported-by: Zorro Lang <[email protected]>
> > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
> > > Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in
> > > xdr_get_next_encode_buffer()")
> > > Signed-off-by: Chuck Lever <[email protected]>
> > > ---
> > >  net/sunrpc/xdr.c |    2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > Hi-
> > >
> > > I had a few minutes yesterday afternoon to look into this one.
> > > The
> > > following one-liner seems to address the issue and passes my
> > > smoke
> > > tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?
> > >
> > >
> > > diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
> > > index f87a2d8f23a7..916659be2774 100644
> > > --- a/net/sunrpc/xdr.c
> > > +++ b/net/sunrpc/xdr.c
> > > @@ -987,7 +987,7 @@ static noinline __be32
> > > *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
> > >         if (space_left - nbytes >= PAGE_SIZE)
> > >                 xdr->end = p + PAGE_SIZE;
> > >         else
> > > -               xdr->end = p + space_left - frag1bytes;
> > > +               xdr->end = p + min_t(int, space_left -
> > > frag1bytes,
> > > PAGE_SIZE);
> >
> > Since we know that frag1bytes <= nbytes (that is determined in
> > xdr_reserve_space()), isn't this effectively the same thing as
> > changing
> > the condition to
> >
> >         if (space_left - frag1bytes >= PAGE_SIZE)
> >                 xdr->end = p + PAGE_SIZE;
> >         else
> >                 xdr->end = p + space_left - frag1bytes;
>
> I added some printk's without this patch, and I'm seeing space_left
> larger than PAGE_SIZE and frag1bytes set to 0 in that branch right
> before the crash. So the min_t() will choose the PAGE_SIZE option
> sometimes.
>

Sure. My point is that it makes the test for space_left - nbytes
redundant (and confusing).

>

--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]


2022-06-30 18:57:02

by Chuck Lever III

[permalink] [raw]
Subject: Re: [PATCH v1] SUNRPC: Fix READ_PLUS crasher



> On Jun 30, 2022, at 2:40 PM, Trond Myklebust <[email protected]> wrote:
>
> On Thu, 2022-06-30 at 14:09 -0400, Anna Schumaker wrote:
>> On Thu, Jun 30, 2022 at 2:03 PM Trond Myklebust
>> <[email protected]> wrote:
>>>
>>> On Thu, 2022-06-30 at 13:24 -0400, Chuck Lever wrote:
>>>> Looks like there are still cases when "space_left - frag1bytes"
>>>> can
>>>> legitimately exceed PAGE_SIZE. Ensure that xdr->end always
>>>> remains
>>>> within the current encode buffer.
>>>>
>>>> Reported-by: Bruce Fields <[email protected]>
>>>> Reported-by: Zorro Lang <[email protected]>
>>>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
>>>> Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in
>>>> xdr_get_next_encode_buffer()")
>>>> Signed-off-by: Chuck Lever <[email protected]>
>>>> ---
>>>> net/sunrpc/xdr.c | 2 +-
>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> Hi-
>>>>
>>>> I had a few minutes yesterday afternoon to look into this one.
>>>> The
>>>> following one-liner seems to address the issue and passes my
>>>> smoke
>>>> tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?
>>>>
>>>>
>>>> diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
>>>> index f87a2d8f23a7..916659be2774 100644
>>>> --- a/net/sunrpc/xdr.c
>>>> +++ b/net/sunrpc/xdr.c
>>>> @@ -987,7 +987,7 @@ static noinline __be32
>>>> *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
>>>> if (space_left - nbytes >= PAGE_SIZE)
>>>> xdr->end = p + PAGE_SIZE;
>>>> else
>>>> - xdr->end = p + space_left - frag1bytes;
>>>> + xdr->end = p + min_t(int, space_left -
>>>> frag1bytes,
>>>> PAGE_SIZE);
>>>
>>> Since we know that frag1bytes <= nbytes (that is determined in
>>> xdr_reserve_space()), isn't this effectively the same thing as
>>> changing
>>> the condition to
>>>
>>> if (space_left - frag1bytes >= PAGE_SIZE)
>>> xdr->end = p + PAGE_SIZE;
>>> else
>>> xdr->end = p + space_left - frag1bytes;
>>
>> I added some printk's without this patch, and I'm seeing space_left
>> larger than PAGE_SIZE and frag1bytes set to 0 in that branch right
>> before the crash. So the min_t() will choose the PAGE_SIZE option
>> sometimes.
>>
>
> Sure. My point is that it makes the test for space_left - nbytes
> redundant (and confusing).

I didn't immediately see how to collapse the checks together, as
I had tried something like that while working on 6c254bf3b637.
But I will try out your suggestion. If it tests OK, I will post
a v2 of this patch.


--
Chuck Lever



2022-06-30 21:21:58

by J. Bruce Fields

[permalink] [raw]
Subject: Re: [PATCH v1] SUNRPC: Fix READ_PLUS crasher

Looks like the crash I saw no longer reproduces after that, fwiw.

--b.

On Thu, Jun 30, 2022 at 01:24:20PM -0400, Chuck Lever wrote:
> Looks like there are still cases when "space_left - frag1bytes" can
> legitimately exceed PAGE_SIZE. Ensure that xdr->end always remains
> within the current encode buffer.
>
> Reported-by: Bruce Fields <[email protected]>
> Reported-by: Zorro Lang <[email protected]>
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=216151
> Fixes: 6c254bf3b637 ("SUNRPC: Fix the calculation of xdr->end in xdr_get_next_encode_buffer()")
> Signed-off-by: Chuck Lever <[email protected]>
> ---
> net/sunrpc/xdr.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> Hi-
>
> I had a few minutes yesterday afternoon to look into this one. The
> following one-liner seems to address the issue and passes my smoke
> tests with NFSv4.1/TCP and NFSv3/RDMA. Any thoughts?
>
>
> diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
> index f87a2d8f23a7..916659be2774 100644
> --- a/net/sunrpc/xdr.c
> +++ b/net/sunrpc/xdr.c
> @@ -987,7 +987,7 @@ static noinline __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
> if (space_left - nbytes >= PAGE_SIZE)
> xdr->end = p + PAGE_SIZE;
> else
> - xdr->end = p + space_left - frag1bytes;
> + xdr->end = p + min_t(int, space_left - frag1bytes, PAGE_SIZE);
>
> xdr->buf->page_len += frag2bytes;
> xdr->buf->len += nbytes;
>