Received-SPF: pass (google.com: domain of linux-kernel+bounces-98448-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <293eb402-73c9-441a-97ec-a5939ed44256@oracle.com>
Date: Sun, 10 Mar 2024 21:13:18 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3] RDMA/cm: add timeout to cm_destroy_id wait
To: Leon Romanovsky <leon@kernel.org>
Cc: dledford@redhat.com, jgg@ziepe.ca, linux-rdma@vger.kernel.org,
        linux-kernel@vger.kernel.org, rama.nichanamatlu@oracle.com
References: <20240309063323.458102-1-manjunath.b.patil@oracle.com>
 <20240310111744.GE12921@unreal>
Content-Language: en-US
From: Manjunath Patil <manjunath.b.patil@oracle.com>
In-Reply-To: <20240310111744.GE12921@unreal>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?KzRKbkdxU3hzeW9hdkVURVkzdmcyMEFLR0FSNUw5OUVPY0I5RVo2M0JHQ3FM?=
 =?utf-8?B?TWc3RXozaGhJRXpnT0RZM28vTTVzTWIwYWxiQ2VhK2xwN25kRGlsQ1Z3Z0Z5?=
 =?utf-8?B?eThRcVRmSngyREgzQ1lXdXBVQkY5WGVObzk5aElVRkNhTEU4WUlOcnVkc0xV?=
 =?utf-8?B?cGh1VzRYeHQvd2dNWUhNZTVNcG5DR1JDRnl3Q0xuUERqUjdRcTVtbEtFREEz?=
 =?utf-8?B?Rmc5MlI4a2ZETWFvMjNiZnA1UnBjK2w1MXl3c0lqRHVma2pXQUk2d2hWc291?=
 =?utf-8?B?UWtMWUhNWGFnMG9tMEE0bS8wdnpwWkM5dG4vMFpRb1grSGFTZlBaVzlkQ2Rp?=
 =?utf-8?B?RDFZWVc0R1VRNkZza1V4aGQvQUZVenBXQjltSkFLcFVzTCtrRCtyQ0J6OHBM?=
 =?utf-8?B?eUZIL2tZRjhsRnBMZEpibTBYZURZdUdjTnFLVGIvMzk5WGUxZWZ1U24raUgz?=
 =?utf-8?B?L1JhNDlxOUdSOEFRcDgxUEFhUnVzVTFYVmQyZkJMelhTbS8xbmxDNFEwWXFG?=
 =?utf-8?B?RjN5ZUl3US93c3BPejErZERrZWl6MHRUdHIvYXV6RnczcXJ0VmZDazV4NllD?=
 =?utf-8?B?RFZLbm5UR250YVNnWm5vWFE4TzVZUHFHQnhTNGtkdTEyeWN6ZVYzWFIwQTBN?=
 =?utf-8?B?WXBqcW1PdVNQcjZGNzdLZTJFRTRrcU1wRm12RzdzcFp3OVh2dXRhTDg2Vlhi?=
 =?utf-8?B?T0ZGdE55Z1hYazI2aDBzTkJ4b3d1Y1NRSVkyR2ZLaEhPaXVpS1lCUTdWbnhw?=
 =?utf-8?B?dzFlSnFJR0ZCL2tBb0ZsMk1Iblc1RStuUmZwL3ZZeGducUNVeExuTWpHeFZD?=
 =?utf-8?B?RUI5RlNJaWtxMmk2WDJqblNCbmpvZlpSVDkwditMay9xcUFzaEFJL3VPeGJ1?=
 =?utf-8?B?V2hkT3kxVVl5STVtWklodkRlMUxJQTBZZU9DaEE1NWNBVWg2cUgwTkU4ZzVw?=
 =?utf-8?B?YjdGQXJ3NUNLRkNRbkkxdFpHbk9zL25qVmRXK3lXSnNUc00rUmdOdzVBMDNs?=
 =?utf-8?B?VTRRWUhlL25kOWszaUp1ZHZQUDh6SFpzcjZYSm45Ni9DcUI5azFuU3hMMXBm?=
 =?utf-8?B?cllzVEE5MXVWTHlPTG5Fay9lVUVmSytVTFNKWWFtOWN2cDhLaGNnWU1SU0tv?=
 =?utf-8?B?ZnBtWFBkUWdpNzRWSCtwdEtsNGl6ZlgraXd4Q09mTGdYb04yYU96WW5CeFM5?=
 =?utf-8?B?eUErcWc0Mk5USmpXR1RTaCtLc3lrZGVPektBVGcyTFE2TmZrMTAzYjRrR2RP?=
 =?utf-8?B?RFBkOHRMZWNkdS8xdzhhdmNLTk1rSVpTL3RVZTVPZTdYOVU4bmFVZWczN2Zy?=
 =?utf-8?B?MUsxOEMyRGJFUER6ZHlsQlpSdUhQNHJyaFZVTnlJVWlEc05xSmNaR3NoQzU0?=
 =?utf-8?B?M3YzS2l1dDQ1WWpIM1EzUElzNVYvWmtaWFJod29LWlo1VThoa0hGOVF0L2Jn?=
 =?utf-8?B?d25ETGd4UU1BR0paTnZFZURxS1hMYmhnbmZKajZPcHRBb0tyTlRSaVhpcXJS?=
 =?utf-8?B?Tzk5bVVJNlRkUjJIYzduNVFIM3lueTB5cE9kN3BxbkVjVGo5L3N4Mm5zMkpv?=
 =?utf-8?B?YVhUMk1ZeitxRmdUU2VKY0VkaUpiZ2VHekZqWElrbGh0eVJsN2tyb0MyWHVU?=
 =?utf-8?B?TTBMRWV4R2s2aHkyTTY1a0RoZWt5b0FQMlBGZDlQdTBlL3h6VlZSdXhUS2dM?=
 =?utf-8?B?TkJRN2pVbkRKZkZIMTl5TGRHanlGNnhVOWgwTDhQUGNCNHJwY05QeFV2aGJH?=
 =?utf-8?B?a1BXeTE1QUVsV0lPRGYwNGM3a1M1ZG1XdTRPNEllUDNvQk5xaFhMZVBoNFhj?=
 =?utf-8?B?UXZ5aEJScW5sTVQrQktHc3lMeGw3WmVpc0hPQWdGQ2w2eGJabG1POGRXdzk2?=
 =?utf-8?B?NkNrbFozR3NYa1Z0azRsS3k5SWlZTEZZYU1YUVBXdTJZNUxnclRGalhET0Nq?=
 =?utf-8?B?L1ZZZ3JvMXp4bVhlTWdyVlJUekRLRWIrTE5pZ2luNEFLN1lDUG1JNVRzaW1J?=
 =?utf-8?B?d21SSHk2KzUyK3hhRW85LzQ2emczVGdHZmJoM0NnNnBEdVEvRHNrWVhyK3pw?=
 =?utf-8?B?Z0NPSGRocmcrVDlrRDRRZDB6a3NtWUZnOFNZTzVWSmRrc0V3Y2t0dC9ZTEFM?=
 =?utf-8?B?WjFDdkh5YnI5bGF4RkEzY3N4ODVOZHE5Z09ud09HNDZGbWVsNytvMUJ3dHZE?=
 =?utf-8?B?c3c9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	DhUbexHr5Rn4goNg/tkiEFMJ2tqlbfObBP4Y4i7OtolltTyOrf3MePIG334orfgakmzC9U6Yg2lwqqb8pTgRFxKleGowIePJDT/g0nmvH46u83rngJ+Hj2RqtAPTMd/xxzn1jK+GYmj0refFAbcPSe3QPjtAyXMEBuecna9lHG2c5onhLY5eBMlrp15MHOaoGOW9/IT++l77FNgl8HE29syGxSGP7hoepTZP3ws0tAxMAbwfu0Vg53g1e3Svn33H5UXQT/rHWx8qdD6qxFHxUas3pgFVx1Yp82zA+gfy1CN/e3Z/m/3vWzyT5k/akvBE+a+A4IuDQtpIs3tyLMDegiGBYEAK29GRJuPx3uGe4n3oB2UnjYBYNHVtkDIoN5QkiYS5SMWboueUQh8gg1ZT0E9nQwABNDeO/b1eUo56rMPvzPh7QMix/RQMxYbJZfeNbEsfbdFnTJs/WWiE/dHVnySOzhnhJGNXqXLr9feOXmGFD+n/ywGaKasDecIZglRE2+IBHsQGdJlx0oEcbJEqEwDFK43pJ6NhzRyCTPsV1Nf7mfK9g1XwN88adzgwYuHQEcmigqFGVfvfEOV1QiHGO1KZY+KIL35FcKIlJ/21fic=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6a3881e8-483d-4170-0b66-08dc418195c0
X-MS-Exchange-CrossTenant-AuthSource: CY5PR10MB6216.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2024 04:13:20.0465
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: aWuRCfIcIhspBtrDP8ziLBBBS/pHj2wPUucUwO1B4V2I0Cq3XlXDFAmGScVNOBMPpf77CmTvrqBmtGEzVtertxb7nqE9YPPVAzRer/2Qt+c=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4940
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-03-11_01,2024-03-06_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0
 mlxlogscore=999 bulkscore=0 phishscore=0 malwarescore=0 mlxscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2403110030
X-Proofpoint-GUID: XqCFzxaiiU7R95mo8NgVfFzaVhYs8bMM
X-Proofpoint-ORIG-GUID: XqCFzxaiiU7R95mo8NgVfFzaVhYs8bMM


On 3/10/24 4:17 AM, Leon Romanovsky wrote:
> On Fri, Mar 08, 2024 at 10:33:23PM -0800, Manjunath Patil wrote:
>> Add timeout to cm_destroy_id, so that userspace can trigger any data
>> collection that would help in analyzing the cause of delay in destroying
>> the cm_id.
>>
>> New noinline function helps dtrace/ebpf programs to hook on to it.
>> Existing functionality isn't changed except triggering a probe-able new
>> function at every timeout interval.
>>
>> We have seen cases where CM messages stuck with MAD layer (either due to
>> software bug or faulty HCA), leading to cm_id getting stuck in the
>> following call stack. This patch helps in resolving such issues faster.
>>
>> kernel: ... INFO: task XXXX:56778 blocked for more than 120 seconds.
>> ...
>> 	Call Trace:
>> 	__schedule+0x2bc/0x895
>> 	schedule+0x36/0x7c
>> 	schedule_timeout+0x1f6/0x31f
>>   	? __slab_free+0x19c/0x2ba
>> 	wait_for_completion+0x12b/0x18a
>> 	? wake_up_q+0x80/0x73
>> 	cm_destroy_id+0x345/0x610 [ib_cm]
>> 	ib_destroy_cm_id+0x10/0x20 [ib_cm]
>> 	rdma_destroy_id+0xa8/0x300 [rdma_cm]
>> 	ucma_destroy_id+0x13e/0x190 [rdma_ucm]
>> 	ucma_write+0xe0/0x160 [rdma_ucm]
>> 	__vfs_write+0x3a/0x16d
>> 	vfs_write+0xb2/0x1a1
>> 	? syscall_trace_enter+0x1ce/0x2b8
>> 	SyS_write+0x5c/0xd3
>> 	do_syscall_64+0x79/0x1b9
>> 	entry_SYSCALL_64_after_hwframe+0x16d/0x0
>>
>> Orabug: 36280065
> 
> Not related to the upstream.
> 
>>
>> Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
>> ---
>> v3:
>>   - added #define
>>
>> v2:
>>   - removed sysctl related code
>>
>>   drivers/infiniband/core/cm.c | 20 +++++++++++++++++++-
>>   1 file changed, 19 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
>> index ff58058aeadc..793103cf8152 100644
>> --- a/drivers/infiniband/core/cm.c
>> +++ b/drivers/infiniband/core/cm.c
>> @@ -34,6 +34,7 @@ MODULE_AUTHOR("Sean Hefty");
>>   MODULE_DESCRIPTION("InfiniBand CM");
>>   MODULE_LICENSE("Dual BSD/GPL");
>>   
>> +#define CM_DESTORY_ID_WAIT_TIMEOUT 10000 /* msecs */
> 
> CM_DESTORY_ID_WAIT_TIMEOUT -> CM_DESTROY_ID_WAIT_TIMEOUT
> 
> Fixed and applied.
Thank you fixing and applying. Sorry for Orabug and not being careful about the typo.

-Manjunath
> 
> Thanks