2022-10-24 03:37:18

by Zhijian Li (Fujitsu)

Subject: [PATCH for-next v3 1/2] RDMA/rxe: Remove unnecessary mr testing

Before the testing, we already passed it to rxe_mr_copy() where mr could
be dereferenced. so this checking is not needed.

The only way that mr is NULL is when it reaches below line 780 with
'qp->resp.mr = NULL', which is not possible in Bob's explanation[1].

778 if (res->state == rdatm_res_state_new) {
779 if (!res->replay) {
780 mr = qp->resp.mr;
781 qp->resp.mr = NULL;
782 } else {

[1] https://lore.kernel.org/lkml/[email protected]/

CC: Bob Pearson <[email protected]>
Signed-off-by: Li Zhijian <[email protected]>
---
V3: remove WARN_ON # Yanjun and Bob
---
drivers/infiniband/sw/rxe/rxe_resp.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index ed5a09e86417..b02639cf8cba 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -811,8 +811,7 @@ static enum resp_states read_reply(struct rxe_qp *qp,

rxe_mr_copy(mr, res->read.va, payload_addr(&ack_pkt),
payload, RXE_FROM_MR_OBJ);
- if (mr)
- rxe_put(mr);
+ rxe_put(mr);

if (bth_pad(&ack_pkt)) {
u8 *pad = payload_addr(&ack_pkt) + payload;
--
1.8.3.1

2022-10-24 05:21:54

by Zhijian Li (Fujitsu)

Subject: [PATCH for-next v3 2/2] RDMA/rxe: Fix mr leak in RESPST_ERR_RNR

rxe_recheck_mr() will increase mr's ref_cnt, so we should call rxe_put(mr)
to drop mr's ref_cnt in RESPST_ERR_RNR to avoid below warning:
[ 633.447883] WARNING: CPU: 0 PID: 4156 at drivers/infiniband/sw/rxe/rxe_pool.c:259 __rxe_cleanup+0x1df/0x240 [rdma_rxe]
...
[ 633.509482] Call Trace:
[ 633.510246] <TASK>
[ 633.510962] rxe_dereg_mr+0x4c/0x60 [rdma_rxe]
[ 633.512123] ib_dereg_mr_user+0xa8/0x200 [ib_core]
[ 633.513444] ib_mr_pool_destroy+0x77/0xb0 [ib_core]
[ 633.514763] nvme_rdma_destroy_queue_ib+0x89/0x240 [nvme_rdma]
[ 633.516230] nvme_rdma_free_queue+0x40/0x50 [nvme_rdma]
[ 633.517577] nvme_rdma_teardown_io_queues.part.0+0xc3/0x120 [nvme_rdma]
[ 633.519204] nvme_rdma_error_recovery_work+0x4d/0xf0 [nvme_rdma]
[ 633.520695] process_one_work+0x582/0xa40
[ 633.522987] ? pwq_dec_nr_in_flight+0x100/0x100
[ 633.524227] ? rwlock_bug.part.0+0x60/0x60
[ 633.525372] worker_thread+0x2a9/0x700
[ 633.526437] ? process_one_work+0xa40/0xa40
[ 633.527589] kthread+0x168/0x1a0
[ 633.528518] ? kthread_complete_and_exit+0x20/0x20
[ 633.529792] ret_from_fork+0x22/0x30

CC: Bob Pearson <[email protected]>
Fixes: 8a1a0be894da ("RDMA/rxe: Replace mr by rkey in responder resources")
Signed-off-by: Li Zhijian <[email protected]>
V2: remove mr testing
---
drivers/infiniband/sw/rxe/rxe_resp.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index b02639cf8cba..41250154a478 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -806,8 +806,10 @@ static enum resp_states read_reply(struct rxe_qp *qp,

skb = prepare_ack_packet(qp, &ack_pkt, opcode, payload,
res->cur_psn, AETH_ACK_UNLIMITED);
- if (!skb)
+ if (!skb) {
+ rxe_put(mr);
return RESPST_ERR_RNR;
+ }

rxe_mr_copy(mr, res->read.va, payload_addr(&ack_pkt),
payload, RXE_FROM_MR_OBJ);
--
2.31.1

2022-10-24 12:42:24

by Leon Romanovsky

Subject: Re: [PATCH for-next v3 2/2] RDMA/rxe: Fix mr leak in RESPST_ERR_RNR

On Mon, Oct 24, 2022 at 01:20:49PM +0800, Li Zhijian wrote:
> rxe_recheck_mr() will increase mr's ref_cnt, so we should call rxe_put(mr)
> to drop mr's ref_cnt in RESPST_ERR_RNR to avoid below warning:
> [ 633.447883] WARNING: CPU: 0 PID: 4156 at drivers/infiniband/sw/rxe/rxe_pool.c:259 __rxe_cleanup+0x1df/0x240 [rdma_rxe]
> ...
> [ 633.509482] Call Trace:
> [ 633.510246] <TASK>
> [ 633.510962] rxe_dereg_mr+0x4c/0x60 [rdma_rxe]
> [ 633.512123] ib_dereg_mr_user+0xa8/0x200 [ib_core]
> [ 633.513444] ib_mr_pool_destroy+0x77/0xb0 [ib_core]
> [ 633.514763] nvme_rdma_destroy_queue_ib+0x89/0x240 [nvme_rdma]
> [ 633.516230] nvme_rdma_free_queue+0x40/0x50 [nvme_rdma]
> [ 633.517577] nvme_rdma_teardown_io_queues.part.0+0xc3/0x120 [nvme_rdma]
> [ 633.519204] nvme_rdma_error_recovery_work+0x4d/0xf0 [nvme_rdma]
> [ 633.520695] process_one_work+0x582/0xa40
> [ 633.522987] ? pwq_dec_nr_in_flight+0x100/0x100
> [ 633.524227] ? rwlock_bug.part.0+0x60/0x60
> [ 633.525372] worker_thread+0x2a9/0x700
> [ 633.526437] ? process_one_work+0xa40/0xa40
> [ 633.527589] kthread+0x168/0x1a0
> [ 633.528518] ? kthread_complete_and_exit+0x20/0x20
> [ 633.529792] ret_from_fork+0x22/0x30
>
> CC: Bob Pearson <[email protected]>
> Fixes: 8a1a0be894da ("RDMA/rxe: Replace mr by rkey in responder resources")
> Signed-off-by: Li Zhijian <[email protected]>
> V2: remove mr testing

This should be after ---

> ---
> drivers/infiniband/sw/rxe/rxe_resp.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> index b02639cf8cba..41250154a478 100644
> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> @@ -806,8 +806,10 @@ static enum resp_states read_reply(struct rxe_qp *qp,
>
> skb = prepare_ack_packet(qp, &ack_pkt, opcode, payload,
> res->cur_psn, AETH_ACK_UNLIMITED);
> - if (!skb)
> + if (!skb) {
> + rxe_put(mr);
> return RESPST_ERR_RNR;
> + }
>
> rxe_mr_copy(mr, res->read.va, payload_addr(&ack_pkt),
> payload, RXE_FROM_MR_OBJ);
> --
> 2.31.1
>

2022-10-24 16:48:43

by Bob Pearson

Subject: Re: [PATCH for-next v3 2/2] RDMA/rxe: Fix mr leak in RESPST_ERR_RNR

On 10/24/22 00:20, Li Zhijian wrote:
> rxe_recheck_mr() will increase mr's ref_cnt, so we should call rxe_put(mr)
> to drop mr's ref_cnt in RESPST_ERR_RNR to avoid below warning:
> [ 633.447883] WARNING: CPU: 0 PID: 4156 at drivers/infiniband/sw/rxe/rxe_pool.c:259 __rxe_cleanup+0x1df/0x240 [rdma_rxe]
> ...
> [ 633.509482] Call Trace:
> [ 633.510246] <TASK>
> [ 633.510962] rxe_dereg_mr+0x4c/0x60 [rdma_rxe]
> [ 633.512123] ib_dereg_mr_user+0xa8/0x200 [ib_core]
> [ 633.513444] ib_mr_pool_destroy+0x77/0xb0 [ib_core]
> [ 633.514763] nvme_rdma_destroy_queue_ib+0x89/0x240 [nvme_rdma]
> [ 633.516230] nvme_rdma_free_queue+0x40/0x50 [nvme_rdma]
> [ 633.517577] nvme_rdma_teardown_io_queues.part.0+0xc3/0x120 [nvme_rdma]
> [ 633.519204] nvme_rdma_error_recovery_work+0x4d/0xf0 [nvme_rdma]
> [ 633.520695] process_one_work+0x582/0xa40
> [ 633.522987] ? pwq_dec_nr_in_flight+0x100/0x100
> [ 633.524227] ? rwlock_bug.part.0+0x60/0x60
> [ 633.525372] worker_thread+0x2a9/0x700
> [ 633.526437] ? process_one_work+0xa40/0xa40
> [ 633.527589] kthread+0x168/0x1a0
> [ 633.528518] ? kthread_complete_and_exit+0x20/0x20
> [ 633.529792] ret_from_fork+0x22/0x30
>
> CC: Bob Pearson <[email protected]>
> Fixes: 8a1a0be894da ("RDMA/rxe: Replace mr by rkey in responder resources")
> Signed-off-by: Li Zhijian <[email protected]>
> V2: remove mr testing
> ---
> drivers/infiniband/sw/rxe/rxe_resp.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> index b02639cf8cba..41250154a478 100644
> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> @@ -806,8 +806,10 @@ static enum resp_states read_reply(struct rxe_qp *qp,
>
> skb = prepare_ack_packet(qp, &ack_pkt, opcode, payload,
> res->cur_psn, AETH_ACK_UNLIMITED);
> - if (!skb)
> + if (!skb) {
> + rxe_put(mr);
> return RESPST_ERR_RNR;
> + }
>
> rxe_mr_copy(mr, res->read.va, payload_addr(&ack_pkt),
> payload, RXE_FROM_MR_OBJ);

This is correct. Good catch. Needs cleanup per Leon otherwise it's good.

Bob

2022-10-24 17:17:57

by Bob Pearson

Subject: Re: [PATCH for-next v3 1/2] RDMA/rxe: Remove unnecessary mr testing

On 10/23/22 22:31, Li Zhijian wrote:
> Before the testing, we already passed it to rxe_mr_copy() where mr could
> be dereferenced. so this checking is not needed.
>
> The only way that mr is NULL is when it reaches below line 780 with
> 'qp->resp.mr = NULL', which is not possible in Bob's explanation[1].
>
> 778 if (res->state == rdatm_res_state_new) {
> 779 if (!res->replay) {
> 780 mr = qp->resp.mr;
> 781 qp->resp.mr = NULL;
> 782 } else {
>
> [1] https://lore.kernel.org/lkml/[email protected]/
>
> CC: Bob Pearson <[email protected]>
> Signed-off-by: Li Zhijian <[email protected]>
> ---
> V3: remove WARN_ON # Yanjun and Bob
> ---
> drivers/infiniband/sw/rxe/rxe_resp.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> index ed5a09e86417..b02639cf8cba 100644
> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> @@ -811,8 +811,7 @@ static enum resp_states read_reply(struct rxe_qp *qp,
>
> rxe_mr_copy(mr, res->read.va, payload_addr(&ack_pkt),
> payload, RXE_FROM_MR_OBJ);
> - if (mr)
> - rxe_put(mr);
> + rxe_put(mr);
>
> if (bth_pad(&ack_pkt)) {
> u8 *pad = payload_addr(&ack_pkt) + payload;

Reviewed-by: Bob Pearson <[email protected]>

2022-10-24 18:41:01

by Jason Gunthorpe

Subject: Re: [PATCH for-next v3 1/2] RDMA/rxe: Remove unnecessary mr testing

On Mon, Oct 24, 2022 at 03:31:54AM +0000, Li Zhijian wrote:
> Before the testing, we already passed it to rxe_mr_copy() where mr could
> be dereferenced. so this checking is not needed.
>
> The only way that mr is NULL is when it reaches below line 780 with
> 'qp->resp.mr = NULL', which is not possible in Bob's explanation[1].
>
> 778 if (res->state == rdatm_res_state_new) {
> 779 if (!res->replay) {
> 780 mr = qp->resp.mr;
> 781 qp->resp.mr = NULL;
> 782 } else {
>
> [1] https://lore.kernel.org/lkml/[email protected]/
>
> CC: Bob Pearson <[email protected]>
> Signed-off-by: Li Zhijian <[email protected]>
> Reviewed-by: Bob Pearson <[email protected]>
> ---
> V3: remove WARN_ON # Yanjun and Bob
> ---
> drivers/infiniband/sw/rxe/rxe_resp.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)

Applied to for-next

Thanks,
Jason

2022-10-24 18:59:06

by Jason Gunthorpe

Subject: Re: [PATCH for-next v3 2/2] RDMA/rxe: Fix mr leak in RESPST_ERR_RNR

On Mon, Oct 24, 2022 at 01:20:49PM +0800, Li Zhijian wrote:
> rxe_recheck_mr() will increase mr's ref_cnt, so we should call rxe_put(mr)
> to drop mr's ref_cnt in RESPST_ERR_RNR to avoid below warning:
> [ 633.447883] WARNING: CPU: 0 PID: 4156 at drivers/infiniband/sw/rxe/rxe_pool.c:259 __rxe_cleanup+0x1df/0x240 [rdma_rxe]
> ...
> [ 633.509482] Call Trace:
> [ 633.510246] <TASK>
> [ 633.510962] rxe_dereg_mr+0x4c/0x60 [rdma_rxe]
> [ 633.512123] ib_dereg_mr_user+0xa8/0x200 [ib_core]
> [ 633.513444] ib_mr_pool_destroy+0x77/0xb0 [ib_core]
> [ 633.514763] nvme_rdma_destroy_queue_ib+0x89/0x240 [nvme_rdma]
> [ 633.516230] nvme_rdma_free_queue+0x40/0x50 [nvme_rdma]
> [ 633.517577] nvme_rdma_teardown_io_queues.part.0+0xc3/0x120 [nvme_rdma]
> [ 633.519204] nvme_rdma_error_recovery_work+0x4d/0xf0 [nvme_rdma]
> [ 633.520695] process_one_work+0x582/0xa40
> [ 633.522987] ? pwq_dec_nr_in_flight+0x100/0x100
> [ 633.524227] ? rwlock_bug.part.0+0x60/0x60
> [ 633.525372] worker_thread+0x2a9/0x700
> [ 633.526437] ? process_one_work+0xa40/0xa40
> [ 633.527589] kthread+0x168/0x1a0
> [ 633.528518] ? kthread_complete_and_exit+0x20/0x20
> [ 633.529792] ret_from_fork+0x22/0x30
>
> CC: Bob Pearson <[email protected]>
> Fixes: 8a1a0be894da ("RDMA/rxe: Replace mr by rkey in responder resources")
> Signed-off-by: Li Zhijian <[email protected]>
> V2: remove mr testing
> ---
> drivers/infiniband/sw/rxe/rxe_resp.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)

Sigh, please try to avoid sending patches in a way that causes
patchworks to become confused. I updated things to remove the if as in
this v2.

Jason

2022-10-25 02:15:53

by Zhijian Li (Fujitsu)

Subject: Re: [PATCH for-next v3 2/2] RDMA/rxe: Fix mr leak in RESPST_ERR_RNR

On 25/10/2022 01:13, Jason Gunthorpe wrote:
> On Mon, Oct 24, 2022 at 01:20:49PM +0800, Li Zhijian wrote:
>> rxe_recheck_mr() will increase mr's ref_cnt, so we should call rxe_put(mr)
>> to drop mr's ref_cnt in RESPST_ERR_RNR to avoid below warning:
>> [ 633.447883] WARNING: CPU: 0 PID: 4156 at drivers/infiniband/sw/rxe/rxe_pool.c:259 __rxe_cleanup+0x1df/0x240 [rdma_rxe]
>> ...
>> [ 633.509482] Call Trace:
>> [ 633.510246] <TASK>
>> [ 633.510962] rxe_dereg_mr+0x4c/0x60 [rdma_rxe]
>> [ 633.512123] ib_dereg_mr_user+0xa8/0x200 [ib_core]
>> [ 633.513444] ib_mr_pool_destroy+0x77/0xb0 [ib_core]
>> [ 633.514763] nvme_rdma_destroy_queue_ib+0x89/0x240 [nvme_rdma]
>> [ 633.516230] nvme_rdma_free_queue+0x40/0x50 [nvme_rdma]
>> [ 633.517577] nvme_rdma_teardown_io_queues.part.0+0xc3/0x120 [nvme_rdma]
>> [ 633.519204] nvme_rdma_error_recovery_work+0x4d/0xf0 [nvme_rdma]
>> [ 633.520695] process_one_work+0x582/0xa40
>> [ 633.522987] ? pwq_dec_nr_in_flight+0x100/0x100
>> [ 633.524227] ? rwlock_bug.part.0+0x60/0x60
>> [ 633.525372] worker_thread+0x2a9/0x700
>> [ 633.526437] ? process_one_work+0xa40/0xa40
>> [ 633.527589] kthread+0x168/0x1a0
>> [ 633.528518] ? kthread_complete_and_exit+0x20/0x20
>> [ 633.529792] ret_from_fork+0x22/0x30
>>
>> CC: Bob Pearson <[email protected]>
>> Fixes: 8a1a0be894da ("RDMA/rxe: Replace mr by rkey in responder resources")
>> Signed-off-by: Li Zhijian <[email protected]>
>> V2: remove mr testing
>> ---
>> drivers/infiniband/sw/rxe/rxe_resp.c | 4 +++-
>> 1 file changed, 3 insertions(+), 1 deletion(-)
> Sigh, please try to avoid sending patches in a way that causes
> patchworks to become confused.

Understood
> I updated things to remove the if as in
> this v2.

thanks a lot.




>
> Jason