2023-11-03 09:56:51

by Zhijian Li (Fujitsu)

Subject: [PATCH RFC V2 1/6] RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr

rxe_set_page() only store one PAGE_SIZE page by the step of page_size.
when page_size != PAGE_SIZE, we cannot restore the address with wrong
index and page_offset.

Let's take a look how current the xarray is being used.

0. offset = iova & (page_size -1); // offset is less than page_size
but may not PAGE_SIZE
1. index = (iova - mr.iova) >> page_shift;
2. page = xa_load(&mr->page_list, index);
3. page_va = kmap_local_page(page) // map one page only, that means only
memory [page_va, page_va + PAGE_SIZE)
is valid for this mapping.
4. memcpy(addr, page_va + offset, bytes);

- when page_size > PAGE_SIZE, the offset could be beyond PAGE_SIZE,
then page_va + offset may be invalid.
- when page_size < PAGE_SIZE, the offset may get lost.

Note that this patch will break some ULPs that try to register 4K
MR when PAGE_SIZE is not 4K. SRP and nvme over RXE is known to be
impacted.

Signed-off-by: Li Zhijian <[email protected]>

---
---
drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index f54042e9aeb2..3755e530e6dc 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -234,6 +234,12 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl,
struct rxe_mr *mr = to_rmr(ibmr);
unsigned int page_size = mr_page_size(mr);

+ if (page_size != PAGE_SIZE) {
+ rxe_err_mr(mr, "Unsupport mr page size %x, expect PAGE_SIZE(%lx)\n",
+ page_size, PAGE_SIZE);
+ return -EINVAL;
+ }
+
mr->nbuf = 0;
mr->page_shift = ilog2(page_size);
mr->page_mask = ~((u64)page_size - 1);
--
2.41.0

2023-11-03 10:14:27

by Greg Sword

Subject: Re: [PATCH RFC V2 1/6] RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr

On Fri, Nov 3, 2023 at 5:56 PM Li Zhijian <[email protected]> wrote:
>
> rxe_set_page() only store one PAGE_SIZE page by the step of page_size.
> when page_size != PAGE_SIZE, we cannot restore the address with wrong
> index and page_offset.
>
> Let's take a look how current the xarray is being used.
>
> 0. offset = iova & (page_size -1); // offset is less than page_size
> but may not PAGE_SIZE
> 1. index = (iova - mr.iova) >> page_shift;
> 2. page = xa_load(&mr->page_list, index);
> 3. page_va = kmap_local_page(page) // map one page only, that means only
> memory [page_va, page_va + PAGE_SIZE)
> is valid for this mapping.
> 4. memcpy(addr, page_va + offset, bytes);
>
> - when page_size > PAGE_SIZE, the offset could be beyond PAGE_SIZE,
> then page_va + offset may be invalid.
> - when page_size < PAGE_SIZE, the offset may get lost.
>
> Note that this patch will break some ULPs that try to register 4K
> MR when PAGE_SIZE is not 4K. SRP and nvme over RXE is known to be
> impacted.
>
> Signed-off-by: Li Zhijian <[email protected]>
>
> ---
> ---
> drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
> index f54042e9aeb2..3755e530e6dc 100644
> --- a/drivers/infiniband/sw/rxe/rxe_mr.c
> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c
> @@ -234,6 +234,12 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl,
> struct rxe_mr *mr = to_rmr(ibmr);
> unsigned int page_size = mr_page_size(mr);
>
> + if (page_size != PAGE_SIZE) {
> + rxe_err_mr(mr, "Unsupport mr page size %x, expect PAGE_SIZE(%lx)\n",
> + page_size, PAGE_SIZE);
> + return -EINVAL;
> + }

Are you kidding us? What problem you are fixing? Do you make tests in your host?

A rubbish patch.

> +
> mr->nbuf = 0;
> mr->page_shift = ilog2(page_size);
> mr->page_mask = ~((u64)page_size - 1);
> --
> 2.41.0
>