From: Rao Shoaib <[email protected]>
Incorportaed suggestions from Jason. There are two patches.
Patch #1 introduces max WQE size as suggested by Jason
Patch #2 allocates resources requested and makes sure that the buffer size
is same for SG entries and inline data, maximum of the two values
requested is used.
Rao Shoaib (2):
RDMA/rxe: use RXE_MAX_WQE_SIZE to enforce limits
RDMA/rxe: SGE buffer and max_inline data must have same size
drivers/infiniband/sw/rxe/rxe_param.h | 7 ++++++-
drivers/infiniband/sw/rxe/rxe_qp.c | 28 ++++++++++++++++------------
2 files changed, 22 insertions(+), 13 deletions(-)
--
1.8.3.1
From: Rao Shoaib <[email protected]>
SGE buffer size and max_inline data should be same. Maximum of the two
values requested is used.
Signed-off-by: Rao Shoaib <[email protected]>
---
drivers/infiniband/sw/rxe/rxe_qp.c | 28 ++++++++++++++++------------
1 file changed, 16 insertions(+), 12 deletions(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
index e2c6d1c..d29d1a8 100644
--- a/drivers/infiniband/sw/rxe/rxe_qp.c
+++ b/drivers/infiniband/sw/rxe/rxe_qp.c
@@ -238,18 +238,22 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp,
qp->src_port = RXE_ROCE_V2_SPORT +
(hash_32_generic(qp_num(qp), 14) & 0x3fff);
- qp->sq.max_wr = init->cap.max_send_wr;
- qp->sq.max_sge = init->cap.max_send_sge;
- qp->sq.max_inline = init->cap.max_inline_data;
-
- wqe_size = max_t(int, sizeof(struct rxe_send_wqe) +
- qp->sq.max_sge * sizeof(struct ib_sge),
- sizeof(struct rxe_send_wqe) +
- qp->sq.max_inline);
-
- qp->sq.queue = rxe_queue_init(rxe,
- &qp->sq.max_wr,
- wqe_size);
+ /* values for max_send_sge and max_inline_data have already been
+ * checked in rxe_qp_chk_cap() to make sure they are within limits.
+ */
+ wqe_size = max_t(int, init->cap.max_send_sge * sizeof(struct ib_sge),
+ init->cap.max_inline_data);
+ qp->sq.max_sge = wqe_size/sizeof(struct ib_sge);
+ qp->sq.max_inline = wqe_size;
+ init->cap.max_inline_data = qp->sq.max_inline;
+ init->cap.max_send_sge = qp->sq.max_sge;
+
+ wqe_size += sizeof(struct rxe_send_wqe);
+
+ qp->sq.max_wr = init->cap.max_send_wr;
+
+ qp->sq.queue = rxe_queue_init(rxe, &qp->sq.max_wr, wqe_size);
+
if (!qp->sq.queue)
return -ENOMEM;
--
1.8.3.1
From: Rao Shoaib <[email protected]>
Introduce RXE_MAX_WQE_SIZE to impose limits on max SGE's and inline data
size
Signed-off-by: Rao Shoaib <[email protected]>
---
drivers/infiniband/sw/rxe/rxe_param.h | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h
index 353c666..f59616b 100644
--- a/drivers/infiniband/sw/rxe/rxe_param.h
+++ b/drivers/infiniband/sw/rxe/rxe_param.h
@@ -34,6 +34,8 @@
#ifndef RXE_PARAM_H
#define RXE_PARAM_H
+#include <uapi/rdma/rdma_user_rxe.h>
+
static inline enum ib_mtu rxe_mtu_int_to_enum(int mtu)
{
if (mtu < 256)
@@ -64,7 +66,6 @@ enum rxe_device_param {
RXE_PAGE_SIZE_CAP = 0xfffff000,
RXE_MAX_QP = 0x10000,
RXE_MAX_QP_WR = 0x4000,
- RXE_MAX_INLINE_DATA = 400,
RXE_DEVICE_CAP_FLAGS = IB_DEVICE_BAD_PKEY_CNTR
| IB_DEVICE_BAD_QKEY_CNTR
| IB_DEVICE_AUTO_PATH_MIG
@@ -77,6 +78,10 @@ enum rxe_device_param {
| IB_DEVICE_MEM_MGT_EXTENSIONS
| IB_DEVICE_ALLOW_USER_UNREG,
RXE_MAX_SGE = 32,
+ RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) +
+ sizeof(struct ib_sge) * RXE_MAX_SGE,
+ RXE_MAX_INLINE_DATA = RXE_MAX_WQE_SIZE -
+ sizeof(struct rxe_send_wqe),
RXE_MAX_SGE_RD = 32,
RXE_MAX_CQ = 16384,
RXE_MAX_LOG_CQE = 15,
--
1.8.3.1
On Thu, Jan 16, 2020 at 10:30:10AM -0800, rao Shoaib wrote:
> From: Rao Shoaib <[email protected]>
>
> Incorportaed suggestions from Jason. There are two patches.
> Patch #1 introduces max WQE size as suggested by Jason
> Patch #2 allocates resources requested and makes sure that the buffer size
> is same for SG entries and inline data, maximum of the two values
> requested is used.
>
> Rao Shoaib (2):
> RDMA/rxe: use RXE_MAX_WQE_SIZE to enforce limits
> RDMA/rxe: SGE buffer and max_inline data must have same size
I already took v3, with modifications
Jason