From: Konstantin Taranov <[email protected]>
This patch series implements creation and destruction of CQs
which can be used with RC QPs.
Patches with RC QPs will be sent in the next patch series.
To create a CQ for RNIC, mana_ib requires creation of EQs
within mana_ib device. An EQ of mana ethernet cannot be used.
To make the implementation of create_cq cleaner, this series
also introduces minor changes to mana_cq structure (cqe->buf_size)
and adds a helper to remove CQ callbacks.
Mana ethernet and mana_ib CQs are different entities which are
created in different isolation zones (ethernet vs rnic).
As a result, RNIC cannot use ethenet CQs and ethernet cannot
use RNIC CQs.
That is why, we use existing udata request for creation of
ethernet CQs. If the request has extra fields, then we create
an RNIC CQ. The kernel-level CQs will be RNIC CQs (in future
patches).
To preserve backward and forward compatibility with RDMA-CORE,
we will make the following changes to mana provider in RDMA-CORE:
The rdma-core will request RNIC CQs by default, with the proposed
request format.
If the mana has installed an allocator with manadv_set_context_attr,
then the rdma-core undestands that this is a DPDK use-case and
requests an ethernet CQ, using old short request format.
Konstantin Taranov (6):
RDMA/mana_ib: create EQs for RNIC CQs
RDMA/mana_ib: create and destroy RNIC cqs
RDMA/mana_ib: replace duplicate cqe with buf_size
RDMA/mana_ib: introduce a helper to remove cq callbacks
RDMA/mana_ib: boundary check before installing cq callbacks
RDMA/mana_ib: implement uapi for creation of rnic cq
drivers/infiniband/hw/mana/cq.c | 77 ++++++++++++++++++++----
drivers/infiniband/hw/mana/main.c | 88 +++++++++++++++++++++++++++-
drivers/infiniband/hw/mana/mana_ib.h | 36 +++++++++++-
drivers/infiniband/hw/mana/qp.c | 30 ++--------
include/uapi/rdma/mana-abi.h | 7 +++
5 files changed, 200 insertions(+), 38 deletions(-)
--
2.43.0
From: Konstantin Taranov <[email protected]>
Replace cqe with buf_size in struct mana_ib_cq.
The cqe field is already present in struct ib_cq and can be accessed there.
The buf_size field is required for mana RNIC CQs.
Signed-off-by: Konstantin Taranov <[email protected]>
---
drivers/infiniband/hw/mana/cq.c | 4 ++--
drivers/infiniband/hw/mana/mana_ib.h | 2 +-
drivers/infiniband/hw/mana/qp.c | 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c
index dc931b9..0467ee8 100644
--- a/drivers/infiniband/hw/mana/cq.c
+++ b/drivers/infiniband/hw/mana/cq.c
@@ -33,8 +33,8 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
return -EINVAL;
}
- cq->cqe = attr->cqe;
- err = mana_ib_create_queue(mdev, ucmd.buf_addr, cq->cqe * COMP_ENTRY_SIZE, &cq->queue);
+ cq->buf_size = attr->cqe * COMP_ENTRY_SIZE;
+ err = mana_ib_create_queue(mdev, ucmd.buf_addr, cq->buf_size, &cq->queue);
if (err) {
ibdev_dbg(ibdev, "Failed to create queue for create cq, %d\n", err);
return err;
diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h
index 9162f29..9c07021 100644
--- a/drivers/infiniband/hw/mana/mana_ib.h
+++ b/drivers/infiniband/hw/mana/mana_ib.h
@@ -90,7 +90,7 @@ struct mana_ib_mr {
struct mana_ib_cq {
struct ib_cq ibcq;
struct mana_ib_queue queue;
- int cqe;
+ u32 buf_size;
u32 comp_vector;
mana_handle_t cq_handle;
};
diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c
index 280e85a..c4fb8b4 100644
--- a/drivers/infiniband/hw/mana/qp.c
+++ b/drivers/infiniband/hw/mana/qp.c
@@ -196,7 +196,7 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd,
wq_spec.queue_size = wq->wq_buf_size;
cq_spec.gdma_region = cq->queue.gdma_region;
- cq_spec.queue_size = cq->cqe * COMP_ENTRY_SIZE;
+ cq_spec.queue_size = cq->buf_size;
cq_spec.modr_ctx_id = 0;
eq = &mpc->ac->eqs[cq->comp_vector];
cq_spec.attached_eq = eq->eq->id;
@@ -355,7 +355,7 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd,
wq_spec.queue_size = ucmd.sq_buf_size;
cq_spec.gdma_region = send_cq->queue.gdma_region;
- cq_spec.queue_size = send_cq->cqe * COMP_ENTRY_SIZE;
+ cq_spec.queue_size = send_cq->buf_size;
cq_spec.modr_ctx_id = 0;
eq_vec = send_cq->comp_vector;
eq = &mpc->ac->eqs[eq_vec];
--
2.43.0
> Subject: [PATCH rdma-next 3/6] RDMA/mana_ib: replace duplicate cqe with
> buf_size
I don't understand this commit message on "duplicate" cqe. I couldn't find a duplicate of it in the existing code.
> From: Long Li <[email protected]>
> Sent: Wednesday, 24 April 2024 01:35
> To: Konstantin Taranov <[email protected]>; Konstantin
> Taranov <[email protected]>; [email protected];
> [email protected]; [email protected]
> Cc: [email protected]; [email protected]
> Subject: RE: [PATCH rdma-next 3/6] RDMA/mana_ib: replace duplicate cqe
> with buf_size
>
> > Subject: [PATCH rdma-next 3/6] RDMA/mana_ib: replace duplicate cqe
> > with buf_size
>
> I don't understand this commit message on "duplicate" cqe. I couldn't find a
> duplicate of it in the existing code.
If we need cqe, we could use it at cq->ibcq.cqe. The patch does not assign it as
it is not used, but if you want I can add "cq->ibcq.cqe = attr->cqe;" in v2.
- Konstantin
> Subject: RE: [PATCH rdma-next 3/6] RDMA/mana_ib: replace duplicate cqe with
> buf_size
>
> > From: Long Li <[email protected]>
> > Sent: Wednesday, 24 April 2024 01:35
> > To: Konstantin Taranov <[email protected]>; Konstantin
> > Taranov <[email protected]>; [email protected];
> > [email protected]; [email protected]
> > Cc: [email protected]; [email protected]
> > Subject: RE: [PATCH rdma-next 3/6] RDMA/mana_ib: replace duplicate cqe
> > with buf_size
> >
> > > Subject: [PATCH rdma-next 3/6] RDMA/mana_ib: replace duplicate cqe
> > > with buf_size
> >
> > I don't understand this commit message on "duplicate" cqe. I couldn't
> > find a duplicate of it in the existing code.
>
> If we need cqe, we could use it at cq->ibcq.cqe. The patch does not assign it as it
> is not used, but if you want I can add "cq->ibcq.cqe = attr->cqe;" in v2.
>
> - Konstantin
I see. We don't need buf_size because it can be computed from cq->ibcq.cqe?
The commit message is confusing enough to make people think cqe is a duplicate of buf_size.
Long