Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
From:   Li Zhijian <lizhijian@fujitsu.com>
To:     zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org,
        linux-rdma@vger.kernel.org
Cc:     linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com,
        matsuda-daisuke@fujitsu.com, bvanassche@acm.org,
        yi.zhang@redhat.com, Li Zhijian <lizhijian@fujitsu.com>
Subject: [PATCH RFC V2 1/6] RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr
Date:   Fri,  3 Nov 2023 17:55:44 +0800
Message-ID: <20231103095549.490744-2-lizhijian@fujitsu.com>
In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com>
References: <20231103095549.490744-1-lizhijian@fujitsu.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

rxe_set_page() only store one PAGE_SIZE page by the step of page_size.
when page_size != PAGE_SIZE, we cannot restore the address with wrong
index and page_offset.

Let's take a look how current the xarray is being used.

0. offset = iova & (page_size -1); // offset is less than page_size
                                      but may not PAGE_SIZE
1. index = (iova - mr.iova) >> page_shift;
2. page = xa_load(&mr->page_list, index);
3. page_va = kmap_local_page(page) // map one page only, that means only
                                      memory [page_va, page_va + PAGE_SIZE)
                                      is valid for this mapping.
4. memcpy(addr, page_va + offset, bytes);

- when page_size > PAGE_SIZE, the offset could be beyond PAGE_SIZE,
  then page_va + offset may be invalid.
- when page_size < PAGE_SIZE, the offset may get lost.

Note that this patch will break some ULPs that try to register 4K
MR when PAGE_SIZE is not 4K. SRP and nvme over RXE is known to be
impacted.

Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>

---
---
 drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
index f54042e9aeb2..3755e530e6dc 100644
--- a/drivers/infiniband/sw/rxe/rxe_mr.c
+++ b/drivers/infiniband/sw/rxe/rxe_mr.c
@@ -234,6 +234,12 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl,
 	struct rxe_mr *mr = to_rmr(ibmr);
 	unsigned int page_size = mr_page_size(mr);
 
+	if (page_size != PAGE_SIZE) {
+		rxe_err_mr(mr, "Unsupport mr page size %x, expect PAGE_SIZE(%lx)\n",
+			   page_size, PAGE_SIZE);
+		return -EINVAL;
+	}
+
 	mr->nbuf = 0;
 	mr->page_shift = ilog2(page_size);
 	mr->page_mask = ~((u64)page_size - 1);
-- 
2.41.0