Message-ID: <d5453f77-1e23-003b-6706-ce1f9c71a668@gmail.com>
Date:   Wed, 15 Mar 2023 13:07:53 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.8.0
Subject: Re: [PATCH 07/13] RDMA/rxe: Rename kfree_rcu() to
 kfree_rcu_mightsleep()
To:     Joel Fernandes <joel@joelfernandes.org>
Cc:     Uladzislau Rezki <urezki@gmail.com>,
        Zhu Yanjun <zyjzyj2000@gmail.com>,
        Jason Gunthorpe <jgg@ziepe.ca>,
        Leon Romanovsky <leon@kernel.org>, linux-rdma@vger.kernel.org,
        Jason Gunthorpe <jgg@nvidia.com>,
        LKML <linux-kernel@vger.kernel.org>, RCU <rcu@vger.kernel.org>,
        "Paul E . McKenney" <paulmck@kernel.org>,
        Oleksiy Avramchenko <oleksiy.avramchenko@sony.com>,
        Jens Axboe <axboe@kernel.dk>,
        Philipp Reisner <philipp.reisner@linbit.com>,
        Bryan Tan <bryantan@vmware.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Eric Dumazet <edumazet@google.com>,
        Ariel Levkovich <lariel@nvidia.com>,
        Theodore Ts'o <tytso@mit.edu>, Julian Anastasov <ja@ssi.bg>
References: <20230201150815.409582-1-urezki@gmail.com>
 <20230201150815.409582-8-urezki@gmail.com> <ZAnjnRC1wY3RIFhM@pc636>
 <ZAnpdKV/VvvX0TZz@pc636> <20230310005529.GA339498@google.com>
 <67fbe385-3682-be4e-15fe-f26cc56fd56b@gmail.com>
 <20230315115053.GA3784687@google.com>
Content-Language: en-US
From:   Bob Pearson <rpearsonhpe@gmail.com>
In-Reply-To: <20230315115053.GA3784687@google.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 3/15/23 06:50, Joel Fernandes wrote:
> On Mon, Mar 13, 2023 at 02:43:43PM -0500, Bob Pearson wrote:
>> On 3/9/23 18:55, Joel Fernandes wrote:
>>> On Thu, Mar 09, 2023 at 03:13:08PM +0100, Uladzislau Rezki wrote:
>>>>> On Wed, Feb 01, 2023 at 04:08:13PM +0100, Uladzislau Rezki (Sony) wrote:
>>>>>> The kfree_rcu()'s single argument name is deprecated therefore
>>>>>> rename it to kfree_rcu_mightsleep() variant. The goal is explicitly
>>>>>> underline that it is for sleepable contexts.
>>>>>>
>>>>>> Please check the RXE driver in a way that a single argument can
>>>>>> be used. Briefly looking at it and rcu_head should be embed to
>>>>>> free an obj over RCU-core. The context might be atomic.
>>>>>>
>>>>>> Cc: Bob Pearson <rpearsonhpe@gmail.com>
>>>>>> Cc: Jason Gunthorpe <jgg@nvidia.com>
>>>>>> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
>>>>>> ---
>>>>>>  drivers/infiniband/sw/rxe/rxe_pool.c | 2 +-
>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>> Could you please add you reviwed-by or Acked-by tags so we can bring
>>>>> our series with renaming for the next merge window?
>>>>>
>>>>> Thanks!
>>>>>
>>>> __rxe_cleanup() can be called in two contexts, sleepable and not.
>>>> Therefore usage of a single argument of the kvfree_rcu() is not correct
>>>> here.
>>>>
>>>> Could you please fix and check your driver? If my above statement
>>>> is not correct, please provide Acked-by or Reviwed-by tags to the
>>>> path that is in question.
>>>>
>>>> Otherwise please add an rcu_head in your data to free objects over
>>>> kvfree_rcu() using double argument API.
>>>>
>>>> Could you please support?
>>>
>>> Also this one needs renaming? It came in because of the commit in 6.3-rc1:
>>> 72a03627443d ("RDMA/rxe: Remove rxe_alloc()")
>>>
>>> It could be squashed into this patch itself since it is infiniband related.
>>>
>>> Paul noticed that this breaks dropping the old API on -next, so it is
>>> blocking the renaming.
>>>
>>> ---8<-----------------------
>>>
>>> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
>>> index b10aa1580a64..ae3a100e18fb 100644
>>> --- a/drivers/infiniband/sw/rxe/rxe_mr.c
>>> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c
>>> @@ -731,7 +731,7 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
>>>  		return -EINVAL;
>>>  
>>>  	rxe_cleanup(mr);
>>> -	kfree_rcu(mr);
>>> +	kfree_rcu_mightsleep(mr);
>>>  	return 0;
>>>  }
>>>  
>> I just got back from a 1 week vacation and missed all this.
>>
>> The "RDMA/rxe: Remove rxe_alloc()" patch just moved the memory allocation
>> for MR (verbs) objects outside of the rxe_pool code since it only applied
>> to MRs and not the other verbs objects (AH, QP, CQ, ...).  That code has to
>> handle a unique situation for AH objects which can be created or destroyed
>> by connection manager code in atomic context while all the other ones
>> including MRs are always created/destroyed in process context. All objects
>> other than MR's are created/destroyed in the rdma-core code
>> (drivers/infiniband/core).
>>
>> The rxe driver keeps xarray's of pointers to the various objects which are
>> protected by rcu locking and so it made sense to use kfree_rcu to delete
>> the object with a delay. In the MR case ..._mightsleep seems harmless and
>> should not be an issue.
>>
>> However on reflection, all the references to the MR objects are ref counted
>> and they have been dropped before reaching the kfree and so there really
>> never was a good reason to use kfree_rcu in the first place. So a better
>> solution would be to replace kfree_rcu with kfree. There is a timeout in
>> completion_done() that triggers a WARN_ON() and this is only seen if the
>> driver is broken for some reason but that is equivalent to getting a seg
>> fault so no reason to further delay the kfree.
>>
>> Reviewed-by: Bob Pearson <rpearsonhpe@gmail.com>
> 
> Thanks, I am planning to send the following patch for 6.4 consideration,
> please let me know if you disagree. Still testing it.
> 
> ----8<---
> 
> From: Joel Fernandes (Google) <joel@joelfernandes.org>
> Subject: [PATCH] RDMA/rxe: Rename kfree_rcu() to kvfree_rcu_mightsleep()
> 
> The k[v]free_rcu() macro's single-argument form is deprecated.
> Therefore switch to the new k[v]free_rcu_mightsleep() variant. The goal
> is to avoid accidental use of the single-argument forms, which can
> introduce functionality bugs in atomic contexts and latency bugs in
> non-atomic contexts.
> 
> There is no functionality change with this patch.
> 
> Link: https://lore.kernel.org/rcu/20230201150815.409582-1-urezki@gmail.com
> Acked-by: Zhu Yanjun <zyjzyj2000@gmail.com>
> Reviewed-by: Bob Pearson <rpearsonhpe@gmail.com>
> Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
> Fixes: 72a03627443d ("RDMA/rxe: Remove rxe_alloc()")
> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
> ---
>  drivers/infiniband/sw/rxe/rxe_mr.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c
> index b10aa1580a64..ae3a100e18fb 100644
> --- a/drivers/infiniband/sw/rxe/rxe_mr.c
> +++ b/drivers/infiniband/sw/rxe/rxe_mr.c
> @@ -731,7 +731,7 @@ int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
>  		return -EINVAL;
>  
>  	rxe_cleanup(mr);
> -	kfree_rcu(mr);
> +	kfree_rcu_mightsleep(mr);
>  	return 0;
>  }
>  

I would prefer just

-	kfree_rcu(mr);
+	kfree(mr);

but either one will work.

Bob