Received-SPF: pass (google.com: domain of linux-kernel+bounces-214353-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <af05925c-cf5c-49fe-96f7-7841024cef7e@oracle.com>
Date: Fri, 14 Jun 2024 10:41:30 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v2] RDMA/mlx5 : Reclaim max 50K pages at once
To: Leon Romanovsky <leon@kernel.org>
Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
        netdev@vger.kernel.org, rama.nichanamatlu@oracle.com,
        manjunath.b.patil@oracle.com
References: <20240613121252.93315-1-anand.a.khoje@oracle.com>
 <20240613190309.GI4966@unreal>
Content-Language: en-US
From: Anand Khoje <anand.a.khoje@oracle.com>
In-Reply-To: <20240613190309.GI4966@unreal>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?VjhaRUhoSzV0b2tSRlM1VUtGRngxUnp1V2N1ek5zN0JJTE9kYVk2TFRHV0Fl?=
 =?utf-8?B?ei9RMVF6TnJTYU1DclFBQXh1UTJQT2dDcktFMkpUcFRnOS9aekwyUVpqcFF4?=
 =?utf-8?B?aFR5SitkZXFrekNndUZDajBjK3dZMzRMRXRyeU12UHAxTkF4d1ZBMkRmUjBs?=
 =?utf-8?B?Q1NnV1JDNU9TeHFDRWpmbGZCSmk0YWlDYmYxT3hHMWdLMCtqTHh0QWJMR1RN?=
 =?utf-8?B?RTRIYkFJNFg4UHlvSVlSOWZoZnBidm9QVENiSVcrSmxROE04L3ZlQWZucHNp?=
 =?utf-8?B?ZXdxNVA1Ym0zSDRHRUJKYUJpbE8rSkg1L3l2MzkxbTdrZmZlNCtob3lDbkJ0?=
 =?utf-8?B?K2w5ZVNTcGsrV3hLNnBTemxYbVhyekc4SzMzQ0Z2VjNwaStSTkYvN0VZK1cy?=
 =?utf-8?B?enhuMlRoaEJHeWlLellkVXE5WXdxUFhLSTA2Q0RnTjFLVXFmc29FV2lWTG1J?=
 =?utf-8?B?NWkzRndZWllzenduWTVIZ0dkc3E2R2FTblJwYlVkaG1xb3E4dThCMXFQbHAy?=
 =?utf-8?B?L1BYb3kyR0ZFeGllZmpDNUFVTi9wMzJOeGpkQ0RVMGhyZEQ3MXV6WXJ2emNE?=
 =?utf-8?B?Z2EyZzQ0SklUQ0NmZjNhaGZtTG5RZ0NNMldZTE9FT09yRE9Bc0dkSHRnMCtv?=
 =?utf-8?B?aElvUGcyeGlpQ3R2RDBoVkh6dys1ckswcG1hTG1uazVNbmxZN084WlZIOWVv?=
 =?utf-8?B?TGtzN0V1bk5OR012dEY0cGZTVGFPSHIyclFRS08wNi80dmh0Mng2dmVzaVJs?=
 =?utf-8?B?a0lmZzNhdnh3aFRTaTZ1d0FSQStWdklPVEZBTUxpVEtHaDdNbnlZNjVKbGtj?=
 =?utf-8?B?VW50VmRoVkk3U2FjMDNMU0lCU3A1S2hTakxZRTRXNk5xaDJtUE50SDQ1dEFr?=
 =?utf-8?B?VDZOMWhIa1BoL0EwNHdSeDlrUEZiYkt4elpscm02b01xZ0NFWDh3Ti9DSnZ6?=
 =?utf-8?B?VnNDWFFRaXpLaFZ3Tk04c1pLcW5hT3czWi9lUkxaNkxhTXg3Nll0ZmhKdzZN?=
 =?utf-8?B?QSs3NDBKSmo4UUdlTng3L3BXY2t6c1dYZGdxMUtoL2pJNU5iT0NyZjB0NUx1?=
 =?utf-8?B?VDMrdS9XWmNDS2JoNi9tRWVOQ1FJbjdmWXJwSTRZaGRGa1ZDZDk2eW53b1oz?=
 =?utf-8?B?OHJUdi9admphZXh5NzY4b1NZS25jVjBGU2RlWmdxQm9wamx4MnFyT3g4TXNv?=
 =?utf-8?B?N3QxQWdjeW1GdEV5UmRvOFUwUWwyYlFxZm1VWkk4M3ZwYjlCOWg3bTFJTFUy?=
 =?utf-8?B?YkNTREFkT25uaUlpRkk4Y1lTMjY0WmU2d2M0b1dBREpsRTVGODdEbWRSMEkw?=
 =?utf-8?B?ZGlqekpPdjNPcUt5TlZZWUh4MkRsZWgzVU5SU2lqN1RyV2ZGT0Yybnp2NDdD?=
 =?utf-8?B?NjF6LzNBUE5BNVJ3UDk3cVZLY2tBTm5UZjIvWi84RzkybUFBWmZQNDlmZE9z?=
 =?utf-8?B?RGhoaHV2L0wwNHlvMTBEaHZFbDFodWtpdWxHWXppMXJEUjJaVGY2cStuYkUy?=
 =?utf-8?B?Z3V6MmdDR0poM1V4N29pM0g5RUJuMXkvanRNSm1zcFZXTEV5ejh6SWRFVEZ1?=
 =?utf-8?B?c1pjMmdWclJrMWlQZTdhNFNyYzlUWFd0K2ljZHJxNlEzU1VxQVBCeW9IRDBP?=
 =?utf-8?B?Z3k2aDJOWXhHeERiYTFNSlNUR1F2aUhMazU3MlVSNWhsZzh4ZW16ZFJqbUd6?=
 =?utf-8?B?enU0OXpqZTZoTmhqQVR0Tk9qSkpQa05QWW9zOEdRQXpnZjF1UnNjUjJ5K3ZE?=
 =?utf-8?B?OUdUb3BSVitQemh4TnZ0NURxSTNCWXArUEFQVzRCdkhxT25SRXc5WFpzdWRw?=
 =?utf-8?B?Ly9oM3dCTm80YTR2N0loNUVUdlRQL3MwcTRBbE5HS0l3a240RHF0OWxLWGtW?=
 =?utf-8?B?MUFwMEtoQURCMmVnQno2R2xsVHhzRXRYOTlya3ExalVKRmU3QzBBdDNMcnBu?=
 =?utf-8?B?RGhqc3lRcmNoenBPLzJCSWJDN3ZCSkt3ZTl0aWhCeWVDdEp4TzU4OW5wTzY0?=
 =?utf-8?B?TUJkWTdjMGhLakRpd3JSSXdpc3JMKy9taGpKN0ZmbzlZWlN0MEpEbWJxU1l4?=
 =?utf-8?B?K1Q0UUJJYytMNjhOdXVLUHN3NXRmdEJJYUFFT1k0dGZ6ZGlITTRJcVoxbE1i?=
 =?utf-8?B?VWcyZEVSS2M1QUpLUm16cXE3Z2F4SWkvV3VkOFNHU29qV1hXQmxWYlBFTU9z?=
 =?utf-8?B?M1dTYnJMOXlPRHJ4M2psVnUzc0t4ZzNiY3dsc0lRekdLV21ISDA1L0p4V1Zr?=
 =?utf-8?B?VUpFUTRaR0Zqd2l6eTVoODFwdHZBPT0=?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	lt4q65QsTV17zmycT8DsBWpkewZxZ28NMBv51Rtt2ZuaprJHzSP3/oEjWGGv5VclbYchgrboQ4q/cMmFQiVbC5Dkk2QmwM3v7xk2bUtNLmQJNt8nwMlPNrZA44IpmuxQjnR/EvVAyuGf3LqEVNOhLohnSflzON6dcpzbG/x3w6gIJaQ9arwERP8+c4H6kiWb+Xj38n2HR5Lf/2fi6FejgTXSWc7PqnqP/JGOnRZPz23OTzTYq3QvpuhkpIJ/QJTpZ3lRFODJONTGtN9FgmMMPfYg5MGOXTls0YPKNgVk+Oy4pDbCoMOtqIFcOoqsdHBTvBvT9Nx0a9lkoQumTzBw2KSJLarMGdN3A706Ls2vUUZZ9j2zxJFsOkHzpeXo1tYCd7BJU35h6w5O4XHL2oUksWO1dEkbyKSlAnY+mV2FybZuuX4ljCCKRbIfVKTviaeW80Cq79cer1vqDlceKajZPD+sq3qJLM9CKWdnuEhl9BNFKv7mNQv3dRT4NsXxwPD2nrQWqMbsCW7HzetxYAt5giTWyVi/3V18olrJs7MzyI52r2xouCh3E0h8llYz+N3EAyIaYEHaLDYhjPByRljvCVygnPXpl/z99ay/t91OBZY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1499c4a3-7bc4-4475-9448-08dc8c3078e0
X-MS-Exchange-CrossTenant-AuthSource: DM4PR10MB6111.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2024 05:11:39.7438
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YqDZvNwmR7eGq2y/5rOUFypYDTlPCOTZqBe+XOGzFikW1i4rw7xsbtLKwefKRSLGCXeamlsV8spqsTgbzRd+txHSTs7EdXfckmfotsbsaxo=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA6PR10MB8206
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16
 definitions=2024-06-13_15,2024-06-13_02,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0
 mlxlogscore=999 bulkscore=0 suspectscore=0 spamscore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2405010000 definitions=main-2406140032
X-Proofpoint-GUID: GsY4XdDsukz-90ehp1ttKZYftwxK8UOA
X-Proofpoint-ORIG-GUID: GsY4XdDsukz-90ehp1ttKZYftwxK8UOA


On 6/14/24 00:33, Leon Romanovsky wrote:
> On Thu, Jun 13, 2024 at 05:42:52PM +0530, Anand Khoje wrote:
>> In non FLR context, at times CX-5 requests release of ~8 million FW pages.
>> This needs humongous number of cmd mailboxes, which to be released once
>> the pages are reclaimed. Release of humongous number of cmd mailboxes is
>> consuming cpu time running into many seconds. Which with non preemptible
>> kernels is leading to critical process starving on that cpu’s RQ.
>> To alleviate this, this change restricts the total number of pages
>> a worker will try to reclaim maximum 50K pages in one go.
>> The limit 50K is aligned with the current firmware capacity/limit of
>> releasing 50K pages at once per MLX5_CMD_OP_MANAGE_PAGES + MLX5_PAGES_TAKE
>> device command.
>>
>> Our tests have shown significant benefit of this change in terms of
>> time consumed by dma_pool_free().
>> During a test where an event was raised by HCA
>> to release 1.3 Million pages, following observations were made:
>>
>> - Without this change:
>> Number of mailbox messages allocated was around 20K, to accommodate
>> the DMA addresses of 1.3 million pages.
>> The average time spent by dma_pool_free() to free the DMA pool is between
>> 16 usec to 32 usec.
>>             value  ------------- Distribution ------------- count
>>               256 |                                         0
>>               512 |@                                        287
>>              1024 |@@@                                      1332
>>              2048 |@                                        656
>>              4096 |@@@@@                                    2599
>>              8192 |@@@@@@@@@@                               4755
>>             16384 |@@@@@@@@@@@@@@@                          7545
>>             32768 |@@@@@                                    2501
>>             65536 |                                         0
>>
>> - With this change:
>> Number of mailbox messages allocated was around 800; this was to
>> accommodate DMA addresses of only 50K pages.
>> The average time spent by dma_pool_free() to free the DMA pool in this case
>> lies between 1 usec to 2 usec.
>>             value  ------------- Distribution ------------- count
>>               256 |                                         0
>>               512 |@@@@@@@@@@@@@@@@@@                       346
>>              1024 |@@@@@@@@@@@@@@@@@@@@@@                   435
>>              2048 |                                         0
>>              4096 |                                         0
>>              8192 |                                         1
>>             16384 |                                         0
>>
>> Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
>> ---
>> Changes in v2:
>>   - In v1, CPUs were yielded if more than 2 msec are spent in
>>     mlx5_free_cmd_msg(). The approach to limit the time spent is changed
>>     in this version.
>> ---
>>   drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c b/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c
>> index 1b38397..b1cf97d 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c
>> @@ -482,12 +482,16 @@ static int reclaim_pages(struct mlx5_core_dev *dev, u32 func_id, int npages,
>>   	return err;
>>   }
>>   
>> +#define MAX_RECLAIM_NPAGES -50000
>>   static void pages_work_handler(struct work_struct *work)
>>   {
>>   	struct mlx5_pages_req *req = container_of(work, struct mlx5_pages_req, work);
>>   	struct mlx5_core_dev *dev = req->dev;
>>   	int err = 0;
>>   
>> +	if (req->npages < MAX_RECLAIM_NPAGES)
>> +		req->npages = MAX_RECLAIM_NPAGES;
> I like this change more than previous variant with yield.
> Regarding the patch:
> 1. Please limit the number of pages in req_pages_handler() and not int pages_work_handler().
> 2. Patch title should be "net/mlx5: Reclaim max 50K pages at once" and not "RDMA...".
> 3. You should run get_maintainer.pl script to find the right maintainers and add them to the TO or CC list.
>
> And I still think that you will get better performance by parallelizing the reclaim process.
>
> Thanks

Hi Leon,

Thanks for the comments. I will make the changes and resend a v3.

-Anand

>> +
>>   	if (req->release_all)
>>   		release_all_pages(dev, req->func_id, req->ec_function);
>>   	else if (req->npages < 0)
>> -- 
>> 1.8.3.1
>>
>>