2020-10-02 14:49:18

by Dan Aloni

Subject: [PATCH] svcrdma: fix bounce buffers for non-zero page offsets

This was discovered using O_DIRECT and small unaligned file offsets
at the client side.

Fixes: e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
Signed-off-by: Dan Aloni <[email protected]>
---
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
index 7b94d971feb3..c991eb1fd4e3 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
@@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
while (remaining) {
len = min_t(u32, PAGE_SIZE - pageoff, remaining);

- memcpy(dst, page_address(*ppages), len);
+ memcpy(dst, page_address(*ppages) + pageoff, len);
remaining -= len;
dst += len;
pageoff = 0;
--
2.26.2

2020-10-02 15:00:39

by Chuck Lever

Subject: Re: [PATCH] svcrdma: fix bounce buffers for non-zero page offsets

Hi Dan-

> On Oct 2, 2020, at 10:48 AM, Dan Aloni <[email protected]> wrote:
>
> This was discovered using O_DIRECT and small unaligned file offsets
> at the client side.
>
> Fixes: e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
> Signed-off-by: Dan Aloni <[email protected]>
> ---
> net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> index 7b94d971feb3..c991eb1fd4e3 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> @@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
> while (remaining) {
> len = min_t(u32, PAGE_SIZE - pageoff, remaining);
>
> - memcpy(dst, page_address(*ppages), len);
> + memcpy(dst, page_address(*ppages) + pageoff, len);

I'm assuming the only relevant place that sets xdr->page_base
is nfsd_splice_actor() ?


> remaining -= len;
> dst += len;
> pageoff = 0;
> --
> 2.26.2
>

--
Chuck Lever

2020-10-02 15:19:06

by Dan Aloni

Subject: Re: [PATCH] svcrdma: fix bounce buffers for non-zero page offsets

On Fri, Oct 02, 2020 at 10:54:28AM -0400, Chuck Lever wrote:
> Hi Dan-
>
> > On Oct 2, 2020, at 10:48 AM, Dan Aloni <[email protected]> wrote:
> >
> > This was discovered using O_DIRECT and small unaligned file offsets
> > at the client side.
> >
> > Fixes: e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
> > Signed-off-by: Dan Aloni <[email protected]>
> > ---
> > net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> > index 7b94d971feb3..c991eb1fd4e3 100644
> > --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> > +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> > @@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
> > while (remaining) {
> > len = min_t(u32, PAGE_SIZE - pageoff, remaining);
> >
> > - memcpy(dst, page_address(*ppages), len);
> > + memcpy(dst, page_address(*ppages) + pageoff, len);
>
> I'm assuming the only relevant place that sets xdr->page_base
> is nfsd_splice_actor() ?

Yes, and traces at the server indeed indicate that splicing happened.
This works for both tmpfs and ext4 as host FSes.

--
Dan Aloni

2020-10-02 15:26:09

by Chuck Lever

Subject: Re: [PATCH] svcrdma: fix bounce buffers for non-zero page offsets

> On Oct 2, 2020, at 11:18 AM, Dan Aloni <[email protected]> wrote:
>
> On Fri, Oct 02, 2020 at 10:54:28AM -0400, Chuck Lever wrote:
>> Hi Dan-
>>
>>> On Oct 2, 2020, at 10:48 AM, Dan Aloni <[email protected]> wrote:
>>>
>>> This was discovered using O_DIRECT and small unaligned file offsets
>>> at the client side.
>>>
>>> Fixes: e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
>>> Signed-off-by: Dan Aloni <[email protected]>
>>> ---
>>> net/sunrpc/xprtrdma/svc_rdma_sendto.c | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
>>> index 7b94d971feb3..c991eb1fd4e3 100644
>>> --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
>>> +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
>>> @@ -638,7 +638,7 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
>>> while (remaining) {
>>> len = min_t(u32, PAGE_SIZE - pageoff, remaining);
>>>
>>> - memcpy(dst, page_address(*ppages), len);
>>> + memcpy(dst, page_address(*ppages) + pageoff, len);
>>
>> I'm assuming the only relevant place that sets xdr->page_base
>> is nfsd_splice_actor() ?
>
> Yes, and traces at the server indeed indicate that splicing happened.
> This works for both tmpfs and ext4 as host FSes.

Seems plausible, thanks. I'll throw it into my test environment.


--
Chuck Lever

2020-10-02 19:35:45

by Dan Aloni

Subject: [PATCH v2] svcrdma: fix bounce buffers for unaligned offsets and multiple pages

This was discovered using O_DIRECT at the client side, with small
unaligned file offsets or IOs that span multiple file pages.

Fixes: e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
Signed-off-by: Dan Aloni <[email protected]>
---
net/sunrpc/xprtrdma/svc_rdma_sendto.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Extended testing found another issue with the loop.

diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
index 7b94d971feb3..c3d588b149aa 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
@@ -638,10 +638,11 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
while (remaining) {
len = min_t(u32, PAGE_SIZE - pageoff, remaining);

- memcpy(dst, page_address(*ppages), len);
+ memcpy(dst, page_address(*ppages) + pageoff, len);
remaining -= len;
dst += len;
pageoff = 0;
+ ppages++;
}
}

--
2.26.2

2020-10-15 04:10:50

by J. Bruce Fields

Subject: Re: [PATCH v2] svcrdma: fix bounce buffers for unaligned offsets and multiple pages

Thanks, applying for 5.10.--b.

On Fri, Oct 02, 2020 at 10:33:43PM +0300, Dan Aloni wrote:
> This was discovered using O_DIRECT at the client side, with small
> unaligned file offsets or IOs that span multiple file pages.
>
> Fixes: e248aa7be86 ("svcrdma: Remove max_sge check at connect time")
> Signed-off-by: Dan Aloni <[email protected]>
> ---
> net/sunrpc/xprtrdma/svc_rdma_sendto.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> Extended testing found another issue with the loop.
>
> diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> index 7b94d971feb3..c3d588b149aa 100644
> --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
> @@ -638,10 +638,11 @@ static int svc_rdma_pull_up_reply_msg(struct svcxprt_rdma *rdma,
> while (remaining) {
> len = min_t(u32, PAGE_SIZE - pageoff, remaining);
>
> - memcpy(dst, page_address(*ppages), len);
> + memcpy(dst, page_address(*ppages) + pageoff, len);
> remaining -= len;
> dst += len;
> pageoff = 0;
> + ppages++;
> }
> }
>
> --
> 2.26.2