Received-SPF: pass (google.com: domain of linux-kernel+bounces-196282-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <f6d81694-c321-470e-8b53-dcdf24d67c9b@oracle.com>
Date: Fri, 31 May 2024 10:21:39 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/1] RDMA/mlx5: Release CPU for other processes in
 mlx5_free_cmd_msg()
To: Leon Romanovsky <leon@kernel.org>
Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
        rama.nichanamatlu@oracle.com, manjunath.b.patil@oracle.com,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>
References: <20240522033256.11960-1-anand.a.khoje@oracle.com>
 <20240522033256.11960-2-anand.a.khoje@oracle.com>
 <20240530171440.GE3884@unreal>
Content-Language: en-US
From: Anand Khoje <anand.a.khoje@oracle.com>
In-Reply-To: <20240530171440.GE3884@unreal>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?cjR4ZG1pQlcxN1UvSDhvdUdLVmRLV0k5NkIrZlRIVEpNMGViVS9GUndjK2ly?=
 =?utf-8?B?UmNGNCthYXRadmk0bGZocVQ2Wm5abGZ1Yy8yamlLN0RncGhuREtlZTRyZGdY?=
 =?utf-8?B?RE8rVG94MmN2dUFNTEtaRExJK2l5RnYvQ1VHZWRGTVBSYyszUFdEZ1VUN1M1?=
 =?utf-8?B?cmZrWGZmendMbnF2NmljcGFUWk1heS9PVWxrRnZtNnJJZUhkcDBYUFJ2ZjBn?=
 =?utf-8?B?U1VuMCtCYWloNTVFeFFUZ0VnTjhveHRhdkNSU2FTM0R0WTFlV0dnRnYvUk4z?=
 =?utf-8?B?alZWVWEzM0szRk1MVkhiS0h4NWNzRVc0L0NRQUMvanM3OHdhMG8xSi9adXV1?=
 =?utf-8?B?eXlIdDJYUVI2TGRMd0RnUDM2VmtSNFhHWDd6YU1iaWRjSW93ajJKaUI4UjBW?=
 =?utf-8?B?eXFNWk9iN1ZmekhZUDRUbElua2hNQjJ3cGdneTNWdHBCbEZieG5LR0hOQ05N?=
 =?utf-8?B?NEdMWi9helBjSDJSSHR3clJwM21xWHJXTkpJSTNUc245ejhWMkxWckVjQ0ZC?=
 =?utf-8?B?VHg5Y3BnK3Y4OGZLMzZkb2NmbXo1TjQ1M2dsK2dlSERrYzl2VHNicCtKQjcy?=
 =?utf-8?B?cEdhdmtiQVhPWkNzZklXTGZJWDdlSmMrZ0ZFYkJsSHNrTWFJNUpCRFNsVGJJ?=
 =?utf-8?B?SVV0UEFUK2xLdlExNjcrN1ZadE5zZHdlTGlYNjdSMUp6eVE3TzZCSlBKcW5R?=
 =?utf-8?B?NEQ0TFgvbk9hKytJeHBwYWlrais1MTI1UThhakFsZWdLbnhBcEY2M1EzaHpQ?=
 =?utf-8?B?N2ZoYXdyU3lFcktkNXFnZFlrS2d5UjBQWFFDMnFFeUErVE5DaW5UeldsYzRz?=
 =?utf-8?B?Q3VFYXNEZmdIeCtSNm91QXdLYnc4UWRNVGY0M21aNHVzUzdOYXp5aTdZaFoz?=
 =?utf-8?B?Nkc1UkN2OE4xWEFSajJGUTF2ODdjTFJqYlBrRXMyWWpyd2tpN2VZdEd0d0c4?=
 =?utf-8?B?a1BDaTZGQTJOdG50bUFiRkk3RkRvbEhRc0pyRW5IbE1Tc0tnbGdONUlHeHBM?=
 =?utf-8?B?ayt2UFljVEpicEFuazBicis0akhZck1LWkQrZnZQQjdUTkhRckl3eW5MSmlE?=
 =?utf-8?B?RGtYZG5EZ25Hd0t0QkloSzd2dmhYMS9GSm52MHBPT3FNUjNXUUN1bDk1Ukxy?=
 =?utf-8?B?SHNIeVVTdW9TTVZBM05RWUZ5L1BUMkJwa3cvUVJZeGwzUHN5UlF4b0lpRjBV?=
 =?utf-8?B?UlNpa3hMODRPUmNBeldJdVpMOGNJVFJlNUx0RGJiUjhpQURsSk9lcUpnZVND?=
 =?utf-8?B?UDNVK09BVkNYY3RyOHBZQ1M1YlY5ajJqMkRVR1R4NkMwdVRGVFpXK2Njd3hK?=
 =?utf-8?B?TzFFcEZTT3NqUWRFMFRrRW9XOHJZeS9JYnhQYUFUVStvS2NDZW9WZmtPNWtm?=
 =?utf-8?B?WnlHUWdHS0ZaS1hnOTlkN0hBMEJ6dHhJSzdUd3QyeTJvYjR3K0crY1lNcTRn?=
 =?utf-8?B?bG5NVStDS01JbjZCTzVvSHdWQnpER1ZhYjdoenV0bFZrUDlkcENzcE1lL09B?=
 =?utf-8?B?Zm5IYi9wUGZMcno0d2NrRURsaE4yOXd0YlZrZjRJWDQveEkvb0RyMXRlZURJ?=
 =?utf-8?B?TUZWYzduV1FyYmwzeWVJSC9iSTNtcnhReEcyZjhXcE9TTXNwTzZMaTMrOWcr?=
 =?utf-8?B?Wk13emlEdGpDRUM5ZlA1emJ5NkRyMVc5ZVBpMlE5TW53UHNBaTV4ZDZ4TTNM?=
 =?utf-8?B?OTk1M0tHS3pvOHgvU0tqeFQ0bENER0VVczN3d0JnUGFRamUvemp4ZFlDcDZv?=
 =?utf-8?B?T3VZdW9oYnZUWVNBTVhlSzhCRlFmUjR0aDhJUTR5dEs3Sjl0Zm5ZeTJuRTlT?=
 =?utf-8?B?clRxSmMyQWFsWVJLaktjM1hmWTYwLzNBTGgyanFLQmlsV2lzWHNGUyt0UTh0?=
 =?utf-8?B?cGhLeFVhYUFObUp0Q0J6bGVWZ0ZzWVdYVnZTS3ZBUEJnbS9HR1ZlTG5wTWxP?=
 =?utf-8?B?ZjRGUXNTK1VvbmxCelVoRHFoaW1rdlUrZzNFWUxLWHp1OVl3M1I3cWU0RWhZ?=
 =?utf-8?B?QnpEbVpYQlFwL0Y5UmZYbXliNjJ2ZFh5N3kySHNoRHFGOTNZKzVBbkVRWWwy?=
 =?utf-8?B?WHplZ3V0THF5cnJiZWFGc2M4bW9xYzI4a1RIZlRPb0FYNFlrSHIzZ09aV3ps?=
 =?utf-8?B?a1l0alZGVEhtUDJ1LzFleG1XWkdsaGIwYjdyWStsbzBkVGZjbmVKZXZ5ZEJv?=
 =?utf-8?B?WWxRVUhZK0hoVEs0WWtuditQMFBadUliK0lLNCt3T3VTYm1oZEJPbXphNUhp?=
 =?utf-8?B?RzVYaFcxNzQrbkdTekQyc1c5SUxBPT0=?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	J23+T1DfQZzzj+F1Uof3KS80HPyfXAk0pxB+reHwr15mNpwfE7nFbffi6yVnOhidTteSChNIL9vlXKUGDb5I1YLDOr8RxcTKLWb+sNLA3wJO+6+cPCImq6mqoWPevmWDcCdvyi/cDSv+rT3jfY9l7/n0OZlNO+N7wJ6x9UMxlbqKK4pey1ZhaS66GuWWbnJwDlR1JsQ/SP5TA2YXkGlhZ7M2qvF1u6p4+R/NxgGKijWnKe1EWkO+Sp+BV7DzhwNw7Wu0eLz2zNXNVJWDICwUSCS87kE5s8FjwZETvhrm45lAwUhgnEuJNMl+UqFmxmRmddAPSrqo6uzPxZlBlerhIeZAtQB2cN1W7tidgZHQ4MKxDHEmII+T7tuKB2Vt9PVFT17st/JdJOYB69evYMsrryCT6nPytLNfE2U5fed1jD2RJgvLnpdtC8HnI5/27DkjEnKUydvRL2fb2mL7dpNoQLkiyk8yznwADaZ48dtg3BGE3hGG4mcdetEwXhXHJuPLufnji/vg/14WitpSZSwct8IPHqD+bCCohvxezCJ1zpo+sd14RJfsKUFgN58QQwhzdq9Q6iSRDaQ9npYhkKOfWiJZbPiuQupVkL4roer6ByE=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7a9e1b07-a7bb-4fdc-8513-08dc812d6045
X-MS-Exchange-CrossTenant-AuthSource: DM4PR10MB6111.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2024 04:51:47.1519
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oN+vlg1haJGA5Qd9xX4A/ZBfRp745/d1BeD250Az7Et0DzAoc4gtlTd3RfLWzRrK4J0s5BF7xI8UiT0fWodr4/sYVhDo00XHcvCv6Vn/vDU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4564
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-31_02,2024-05-30_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 adultscore=0
 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000
 definitions=main-2405310035
X-Proofpoint-ORIG-GUID: zRG_wEvuBRZTIh19rR4SUxGwQXEgj55Q
X-Proofpoint-GUID: zRG_wEvuBRZTIh19rR4SUxGwQXEgj55Q


On 5/30/24 22:44, Leon Romanovsky wrote:
> On Wed, May 22, 2024 at 09:02:56AM +0530, Anand Khoje wrote:
>> In non FLR context, at times CX-5 requests release of ~8 million device pages.
>> This needs humongous number of cmd mailboxes, which to be released once
>> the pages are reclaimed. Release of humongous number of cmd mailboxes
>> consuming cpu time running into many secs, with non preemptable kernels
>> is leading to critical process starving on that cpu’s RQ. To alleviate
>> this, this patch relinquishes cpu periodically but conditionally.
>>
>> Orabug: 36275016
>>
>> Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
>> ---
>>   drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>> index 9c21bce..9fbf25d 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>> @@ -1336,16 +1336,23 @@ static struct mlx5_cmd_msg *mlx5_alloc_cmd_msg(struct mlx5_core_dev *dev,
>>   	return ERR_PTR(err);
>>   }
>>   
>> +#define RESCHED_MSEC 2
>>   static void mlx5_free_cmd_msg(struct mlx5_core_dev *dev,
>>   			      struct mlx5_cmd_msg *msg)
>>   {
>>   	struct mlx5_cmd_mailbox *head = msg->next;
>>   	struct mlx5_cmd_mailbox *next;
>> +	unsigned long start_time = jiffies;
>>   
>>   	while (head) {
>>   		next = head->next;
>>   		free_cmd_box(dev, head);
> Did you consider to make this function asynchronous and parallel?
>
> Thanks

Hi Leon,

Thanks for reviewing this patch.

Here, all page related methods 
give_pages/reclaim_pages/release_all_pages are executed in a worker 
thread through pages_work_handler().

Doesn't that mean it is already asynchronous?

When the worker thread, in this case it is processing reclaim_pages(), 
is taking a long time - it is starving other processes on the processor 
that it is running on. Oracle UEK being a non-preemptible kernel, these 
other processes that are getting starved do not get CPU until the worker 
relinquishes the CPU. This applies to even processes that are time 
critical and high priority. These processes when starved of CPU for a 
long time, trigger a kernel panic.

Hence, this patch implements a time based relinquish of CPU using 
cond_resched().

Shay Dori, had a suggestion to tune the time (which we have made 2 
msec), to reduce too frequent context switching and find a balance in 
processing of these mailbox objects. I am presently running some tests 
on the basis of this suggestion.

Thanks,

Anand

>>   		head = next;
>> +		if (time_after(jiffies, start_time + msecs_to_jiffies(RESCHED_MSEC))) {
>> +			mlx5_core_warn_rl(dev, "Spent more than %d msecs, yielding CPU\n", RESCHED_MSEC);
>> +			cond_resched();
>> +			start_time = jiffies;
>> +		}
>>   	}
>>   	kfree(msg);
>>   }
>> -- 
>> 1.8.3.1
>>
>>