Received-SPF: pass (google.com: domain of linux-kernel+bounces-194049-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <04591dbf-57de-4d21-8009-5f462fb59c73@oracle.com>
Date: Wed, 29 May 2024 17:31:02 +0530
User-Agent: Mozilla Thunderbird
From: Anand Khoje <anand.a.khoje@oracle.com>
Subject: Re: [PATCH 1/1] RDMA/mlx5: Release CPU for other processes in
 mlx5_free_cmd_msg()
To: Shay Drori <shayd@nvidia.com>, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org, moshe@nvidia.com
Cc: rama.nichanamatlu@oracle.com, manjunath.b.patil@oracle.com,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>
References: <20240522033256.11960-1-anand.a.khoje@oracle.com>
 <20240522033256.11960-2-anand.a.khoje@oracle.com>
 <a26f1947-58fc-48c4-a8f3-4fe2a274afa6@nvidia.com>
Content-Language: en-US
In-Reply-To: <a26f1947-58fc-48c4-a8f3-4fe2a274afa6@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?SkpVQ0JHTVFpWFA3bjFTeEEzRVlFL1FiOE1BeklXM2lOekdXZGZXTUY1cHNO?=
 =?utf-8?B?U2VsazVOcEI2Qm9aSS9UUnM5cW9vdGNYY2U5Ump0enRrY2RQVEFReWN3U252?=
 =?utf-8?B?ckNQN05qNG95T1lLMW5aMWRXRXVaOUcwV0w2K1l1ejdPaUY3dUtzTTd4bmVN?=
 =?utf-8?B?M2FNMVVaVFozODlsaVk4RlhZNG9Gdm84endOaC8wRThCUzZSd2RhZVFJUVRh?=
 =?utf-8?B?aWYwQlRQU0FpMlZDNEoxM3hVaWE4ZUt1ZmppNVZ1c0FEUUZ3NFpNVkNmZER0?=
 =?utf-8?B?bkxqOVJ4REJ2MWMyYWVZaFR1WnRFaVR5d0dSbTVZQkdPcW9ZRUw5dDhQcWpT?=
 =?utf-8?B?UDViTm45cTV5S2FFTGFuK2dKM0xXYVNORWpnUVBUR1hqTlE0Z05NNENXVWh3?=
 =?utf-8?B?Q2dDQkpIa2MyNXdFd3d1cmhNZUhSa1pWQlJCeXFjbE4yRDB1Q0w4bmhzdDZW?=
 =?utf-8?B?Vk9CdXZxTFg0L0ZGQTZsb2w3c1pKTTlKL3Q2c1Fsck1JM3drQkZGUHJONWow?=
 =?utf-8?B?cXhrazlsMEo2cEZXdVMwOU5mTU4vTStpSXF2cVdROHE5QS9jRjJwajZvZ0Zq?=
 =?utf-8?B?a3dzQ0lCc1VPaHRhK0h4dnZ0S2d6RHdWckJld3dUY1VYZWIvbzNxeWMxeVBk?=
 =?utf-8?B?Mm5IYW1YQXFWaHFRM0pqVGhFUEd5YVphV3g1WGRYdFN1T3J6Q2cvSmlpVTVu?=
 =?utf-8?B?Um5tRmZRUDR6SWNUSThIdmY0WlBZbGhab1VGNkVNTjhobVZLaHEvZ1FBcTlB?=
 =?utf-8?B?TEt3ZjlGeVFYSlFzejArQjYwLzhCMkFldzZoMndsRXhKTjZnRFRvV1E1ZWwr?=
 =?utf-8?B?VVlyNVBYQ3FwMUZBYk5ZUmxJUEt1ZWZIdEdrN1RXMWw1L2tPM2VmMHhMSDhj?=
 =?utf-8?B?NEtIOFlmUnNUbHhQVDJUYm82VkFSaTFDWDRhZTRVUFQ2SUZFb01mdUhuNVJR?=
 =?utf-8?B?N3JCQkhrTTMzZGpsTUU5N3h5eFBhK2E1K25XNHkxMDlLRmZNMFduWEwwQ0N3?=
 =?utf-8?B?RTBOY1VNYUJVbkZVbGFKUG84ZW96VXZZaS9pUWJENFBjaEd3K0w2dkRLWHZX?=
 =?utf-8?B?RzluWVk3VzZBbnhidFdVckd3dFQ5NzdncVVhVkd6eFl4NHFJSGprWis2cGlC?=
 =?utf-8?B?RE9Sek8yQ2Y1djJXYjMxNmcvbjM4YUpGdVJKblBEYzZJY1ZLSUhaWjExWHZP?=
 =?utf-8?B?T2o4VFpmMUdubTZVUEpzZkRlcDduNkYyaHd4bjVHUlZLRlNFMjVmZEdKQW8v?=
 =?utf-8?B?VnF4QWRFRHBLVTZ4WlZLYVVNNlJCdVhzSlZLN25sVEdSVHFoR01BRlFCRkRt?=
 =?utf-8?B?eEZIZGdPbWNyQlJWZXZIaEh5a1BTeUIvVHI4dHBsLzZTamNwSlhUOUQ1VmJr?=
 =?utf-8?B?OU4xeU1KYXphOFluOGxUcDhsbFhLMWhqdzVKS1pSRS92UGJYYTBhQU15WjlY?=
 =?utf-8?B?Y3FiZjJKamM5YWZXQ0tKaGEzUlJJMURUT1JacFNPMG5UNTh6eDNHZkFYczB6?=
 =?utf-8?B?S2dWZ2NCRDM3UmpaZXNBcUEvUVd6UnorTThnOWVFQk5ZUXVTQldqbFU0YW45?=
 =?utf-8?B?czhzd1kxSm4wNGM1OTNuZ1AwZ0hoZW1RTkJ5anB0VHpjR0pJMUExTkpJcUp5?=
 =?utf-8?B?a0gxQTBLa2VMYXJUeldrWGdldE0wSG1pVjNwdTJOeVQ3b3NtYmt4MU1CZlN0?=
 =?utf-8?B?V21ldVlPK2RZRHlheEFNWTRYZkVQV2RvbXpHbTAybHlIYXE3S3pVcEdvUURk?=
 =?utf-8?B?ZU04TW1EeFZxSENpWGVWRTJPa3NHR3RvTDV0bTNZSERYaXhQYloySkg0dktn?=
 =?utf-8?B?YllYbkhhWUNGeUJ5MC92YmJSYjlJZHZPaGh3QzRlTzNvaWlMN29uWHdpRkZB?=
 =?utf-8?B?Yk1WejVXSW4wUlg2cUxSZElCRUFDUVlpTmpaSGYvWGZmRXh2enBkUm9mL3Qz?=
 =?utf-8?B?cVFlUk1jckVPYTM1WHkwMklpMlhSbldCdFBJa3JaeG84clNaVkRXUytoZkpO?=
 =?utf-8?B?U2orS1dpdDd4TkN5clFFcFc0c1ZzSnBxVWg2bDE1dDdCcGp0U0E5bTJVeE50?=
 =?utf-8?B?Wk8xK2xaVC8zRXBjb1ZrV2NSMFo1UlJoRE5ab1lSWk5QcC9Fb0dpdWpIcjUx?=
 =?utf-8?B?N1NPMFdmMVgxczQyVHY0d1VnTFRIclF3d3l2NGo5K1J2MUIwN0ZBMENsbTU5?=
 =?utf-8?B?cVh2bUVtbHcwSWNYRXZOSStiaFJuSWxYZVprNU52aHl4SUlVUHJYKzVNaGNI?=
 =?utf-8?B?WXZ5UzNReHJLdVVKcHRONXp2bm1RPT0=?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	fB9xzZxea4vmDf7mawiWKj+p4SP9IxO/UL2zWWmtu/E/ChNVJBXX+SzQIH+L7KyC5u29/1wnAK5WlW9EP63GHzOttlck6LYtTsRRgGgL2rXRl+sKtV818wM+4iEc//kyE/3P6BOsdmZloUnLHX9WJirpVY0zfh3m+y8nuwJRjwUtd0w0weRCL9vKd4n1rBrBCmt5pSzKwMi5g+ueyNCBT9caXtIxt5DKdbqfFTWeqA7cdyiN32Dnq23wRI+ud8fGvGgmkC8Ntu3Q93gVWYaLmP2fNGrMWYR0iCnNiHvYIn0sB9/UNIgy4LSGPAh0gXzlNdls0oG968ZqSyWPswkqgdDCLBQWIuHyy7BxygspAFnwW5UYlWDgl72MaJdXlWCrNQjytmh44f+407WajpSSEuJGcboS/bI2DQFSZg9wJop6ia56rRKXbH2sqzDh/PhDc5m5CLvktWfzQtvo+5YwnbRFZNPG1Hmya3PaZm6Vtin9mljIp6Vom5Dpj8IdVR9+KiIO+i46XYX+HQ1d2Ggu2rykp9zHZFj+kSX96UPfWqz+37c2WVwnI9/SiJ/OgsT5SyxTnlW3OMcrB2rcOGL43u1mlg+ezBOBtv74iTUV5O4=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2fcb316a-c4da-43af-3e92-08dc7fd7093d
X-MS-Exchange-CrossTenant-AuthSource: DM4PR10MB6111.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 May 2024 12:01:13.2905
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: MIKsURfK0bV2c4ojuOhFwcuT9+sOWFPq++KLQh9XhwcFh2Tej4i80sdskyQyFGhC+Hm+oPvW+f9qv2U4/++Go4bJWikj6uLT4yaEoUsoQjY=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6560
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.12.28.16
 definitions=2024-05-29_07,2024-05-28_01,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 mlxscore=0
 mlxlogscore=999 phishscore=0 spamscore=0 adultscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000
 definitions=main-2405290082
X-Proofpoint-ORIG-GUID: I0KtqyjEKtlQgYhfpIP2TgKUDDfZG5pF
X-Proofpoint-GUID: I0KtqyjEKtlQgYhfpIP2TgKUDDfZG5pF


On 5/26/24 20:53, Shay Drori wrote:
> Hi Anand.
>
> First, the correct Mailing list for this patch is
> netdev@vger.kernel.org, please send there the next version.
>
> On 22/05/2024 6:32, Anand Khoje wrote:
>> In non FLR context, at times CX-5 requests release of ~8 million 
>> device pages.
>> This needs humongous number of cmd mailboxes, which to be released once
>> the pages are reclaimed. Release of humongous number of cmd mailboxes
>> consuming cpu time running into many secs, with non preemptable kernels
>> is leading to critical process starving on that cpu’s RQ. To alleviate
>> this, this patch relinquishes cpu periodically but conditionally.
>>
>> Orabug: 36275016
>
> this doesn't seem relevant
>
>>
>> Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
>> ---
>>   drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c 
>> b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>> index 9c21bce..9fbf25d 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>> @@ -1336,16 +1336,23 @@ static struct mlx5_cmd_msg 
>> *mlx5_alloc_cmd_msg(struct mlx5_core_dev *dev,
>>       return ERR_PTR(err);
>>   }
>>   +#define RESCHED_MSEC 2
>
>
> What if you add cond_resched() on every iteration of the loop ? Does it
> take much more time to finish 8 Million pages or same ?
> If it does matter, maybe 2 ms is too high freq ? 20 ms ? 200 ms ?
>
Shay,


There is no rule we could use, but can use only guidance/suggestions here.
Delay if too short/often relinquish leads to thrashing and high context 
switch costs,
while keeping it long/infrequent relinquish leads to RQ starvation.
This observation is based  on our applications / workload, using which a 
middle ground was chosen as 2 msecs.
But your suggestions are also very viable. Hence we are reconsidering it.

This was very helpful. thank you! I will resend a v2 after more testing.

Thanks,

Anand


> Thanks
>
>>   static void mlx5_free_cmd_msg(struct mlx5_core_dev *dev,
>>                     struct mlx5_cmd_msg *msg)
>>   {
>>       struct mlx5_cmd_mailbox *head = msg->next;
>>       struct mlx5_cmd_mailbox *next;
>> +    unsigned long start_time = jiffies;
>>         while (head) {
>>           next = head->next;
>>           free_cmd_box(dev, head);
>>           head = next;
>> +        if (time_after(jiffies, start_time + 
>> msecs_to_jiffies(RESCHED_MSEC))) {
>> +            mlx5_core_warn_rl(dev, "Spent more than %d msecs, 
>> yielding CPU\n", RESCHED_MSEC);
>> +            cond_resched();
>> +            start_time = jiffies;
>> +        }
>>       }
>>       kfree(msg);
>>   }