Received-SPF: pass (google.com: domain of linux-kernel+bounces-40073-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <5bc6ed6d-31e9-4b44-aa91-5f9d0f3d92c8@sangfor.com.cn>
Date: Fri, 26 Jan 2024 10:25:01 +0800
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] RDMA/device: Fix a race between mad_client and cm_client
 init
To: Ding Hui <dinghui@sangfor.com.cn>, Jason Gunthorpe <jgg@ziepe.ca>
Cc: leon@kernel.org, wenglianfa@huawei.com, gustavoars@kernel.org,
 linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
 Shifeng Li <lishifeng1992@126.com>
References: <20240102034335.34842-1-lishifeng@sangfor.com.cn>
 <20240103184804.GB50608@ziepe.ca>
 <80cac9fd-7fed-403e-8889-78e2fc7a49b0@sangfor.com.cn>
 <20240104123728.GC50608@ziepe.ca>
 <e029db0a-c515-e61c-d34e-f7f054d51e88@sangfor.com.cn>
 <20240115134707.GZ50608@ziepe.ca>
 <354e2bf7-a8b4-629d-3d2d-35951a52e8bd@sangfor.com.cn>
From: Shifeng Li <lishifeng@sangfor.com.cn>
In-Reply-To: <354e2bf7-a8b4-629d-3d2d-35951a52e8bd@sangfor.com.cn>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 2024/1/16 10:12, Ding Hui wrote:
> On 2024/1/15 21:47, Jason Gunthorpe wrote:
>> On Sat, Jan 06, 2024 at 10:12:17AM +0800, Ding Hui wrote:
>>> On 2024/1/4 20:37, Jason Gunthorpe wrote:
>>>> On Thu, Jan 04, 2024 at 02:48:14PM +0800, Shifeng Li wrote:
>>>>
>>>>> The root cause is that mad_client and cm_client may init concurrently
>>>>> when devices_rwsem write semaphore is downgraded in enable_device_and_get() like:
>>>>
>>>> That can't be true, the module loader infrastructue ensures those two
>>>> things are sequential.
>>>>
>>>
>>> Please consider the sequence again and notice that:
>>>
>>> 1. We agree that dependencies ensure mad_client be registered before cm_client.
>>> 2. But the mad_client.add() is not invoked in ib_register_client(), since
>>>     there is no DEVICE_REGISTERED device at that time.
>>>     Instead, it will be delayed until the device driver init (e.g. mlx5_core)
>>>     in enable_device_and_get().
>>> 3. The ib_cm and mlx5_core can be loaded concurrently, after setting DEVICE_REGISTERED
>>>     and downgrade_write(&devices_rwsem) in enable_device_and_get(), there is a chance
>>>     that cm_client.add() can be invoked before mad_client.add().
>>>
>>>
>>>          T1(ib_core init)      |      T2(device driver init)        |        T3(ib_cm init)
>>> ---------------------------------------------------------------------------------------------------
>>> ib_register_client mad_client
>>>    assign_client_id
>>>      add clients CLIENT_REGISTERED
>>>      (with clients_rwsem write)
>>>    down_read(&devices_rwsem);
>>>    xa_for_each_marked (&devices, DEVICE_REGISTERED)
>>>      nop # no devices
>>>    up_read(&devices_rwsem);
>>>
>>>                                ib_register_device
>>>                                  enable_device_and_get
>>>                                    down_write(&devices_rwsem);
>>>                                    set DEVICE_REGISTERED
>>>                                    downgrade_write(&devices_rwsem);
>>>                                                                      ib_register_client cm_client
>>>                                                                        assign_client_id
>>>                                                                          add clients CLIENT_REGISTERED
>>>                                                                          (with clients_rwsem write)
>>>                                                                        down_read(&devices_rwsem);
>>>                                                                        xa_for_each_marked (&devices, DEVICE_REGISTERED)
>>>                                                                          add_client_context
>>>                                                                            down_write(&device->client_data_rwsem);
>>>                                                                            get CLIENT_DATA_REGISTERED
>>>                                                                            downgrade_write(&device->client_data_rwsem);
>>>                                                                            cm_client.add
>>>                                                                              cm_add_one
>>>                                                                                ib_register_mad_agent
>>>                                                                                  ib_get_mad_port
>>>                                                                                    __ib_get_mad_port return NULL!
>>>                                                                            set CLIENT_DATA_REGISTERED
>>>                                                                            up_read(&device->client_data_rwsem);
>>>                                                                        up_read(&devices_rwsem);
>>>                                  down_read(&clients_rwsem);
>>>                                  xa_for_each_marked (&clients, CLIENT_REGISTERED)
>>>                                    add_client_context [mad]
>>>                                      mad_client.add
>>>                                    add_client_context [cm]
>>>                                      nop # already CLIENT_DATA_REGISTERED
>>>                                  up_read(&clients_rwsem);
>>>                                  up_read(&devices_rwsem);
>>
>> Take the draft I sent previously and use down_write(&devices_rwsem) in
>> ib_register_client()
>>
> 
> I believe this modification is effective, rather than expanding the clients_rwsem range,
> the key point is down_write(&devices_rwsem), which prevents ib_register_client() from
> being executed in the gap of ib_register_device().
> 
> However, this may cause a little confusion, as ib_register_client() does not modify
> anything related to devices, but it is protected by a write lock.
> 

Hi Jason，

Do you have any differing opinions about above?