Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Message-ID: <89062478-fa97-c265-3b18-de55eeae3c1f@oracle.com>
Date:   Tue, 10 Oct 2023 15:58:43 +1100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.15.1
Subject: Re: [PATCH smp,csd] Throw an error if a CSD lock is stuck for too
 long
Content-Language: en-US
To:     paulmck@kernel.org
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Valentin Schneider <vschneid@redhat.com>,
        Juergen Gross <jgross@suse.com>,
        Leonardo Bras <leobras@redhat.com>,
        Kernel Mailing List <linux-kernel@vger.kernel.org>
References: <bc7cc8b0-f587-4451-8bcd-0daae627bcc7@paulmck-laptop>
 <c87d7939-a88c-ad2d-82f2-866e04692882@oracle.com>
 <a77da319-3161-4e42-894e-521d29fa8348@paulmck-laptop>
From:   Imran Khan <imran.f.khan@oracle.com>
In-Reply-To: <a77da319-3161-4e42-894e-521d29fa8348@paulmck-laptop>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?N0I2YjV6aEhsS1YzWWtRRXFXcUpmUGxyd0ExOXc1RlZSakhwNng0NkQ0ZkRa?=
 =?utf-8?B?d3lmcE5Sb0pML3NxWHk1Q25STXhlUVhQWE51SldhdDVWbUhnWlhJT1hvYmtz?=
 =?utf-8?B?ejBDbGwzSkR6ejQ3N0lVYnZod2hRcmYrRVRHN3dwZ0hyb00xVEVOL3JlYm5m?=
 =?utf-8?B?ZkJWWDBzbTlOdG51a21XYTllZmxXckx1SzlRYTNyVURrUnVBcldPTEJzSXJG?=
 =?utf-8?B?SGVGOFFQTk9nem9DLzN6Y210ZFFRazJyY0tBSjZlQlM1MDhKYSs1WTlNWFRO?=
 =?utf-8?B?Q21EK3E5bnFuU2FWR0VmQlZ0YjdlUzQ5Z0M2K1lXMVRzeEJPNWYyYzZNckd3?=
 =?utf-8?B?RUNzS0RnTFgrVGpRTlZpVEEvdkRKK0wyN1lmYWFyRmw2dUJCNHFLMnBqWWpq?=
 =?utf-8?B?Qm90K245TGloQ2RCL0ZOL3F6ZEhPRHd5Z2FLTkdRYVdWSUUxTjErM1VocHdi?=
 =?utf-8?B?WVBZNFVhdHk5Yi80Rno3RFd5QkhoTnZaaU52dFcyVmVSWFhmTitBYkRXWVNm?=
 =?utf-8?B?ZTNUMUhpK2M3STdSc1FLTStUZVdhV1pxOTVsSUJLYnhxYS9INVRxb1pidEFL?=
 =?utf-8?B?OWwxb3pmcHppcHROMS9ReUtJb2tFbUdJLy9VR1Rtclg2b0pCVFovOHdmbVAr?=
 =?utf-8?B?VFllMHU2QVNGdUduMEs3d0VmTWkvRzZtdUF6bWdzQ2ZGNTZnemZCL1FGMG9C?=
 =?utf-8?B?bDUrMzM3dDdHd2VOUy8rWVpXem5vTXF0cllTZTAwWDY5aGwrUktFNlBKNHpz?=
 =?utf-8?B?YkM5TUFLbjFBbkxoUVc1MHlzOFhQcm1vbGVLQ2JsejVwb3FpaUFiN2QxRnRw?=
 =?utf-8?B?RXJlVHdaVU1qS3lDQVhLcHl1U3RIOTRxMHBhNDJmYmw3WGM5b01TWW85cHph?=
 =?utf-8?B?Y1FvLzJCWlczdDhWdXVyQUFwU0ZNQnFKOTNlL29wSjIzeDF5OWxWdytvS0Iy?=
 =?utf-8?B?Rmx6Q1dHeHdKUlZvL2QyQTNFQjBRQ0duNnIzZU9icUVubTRWQTQ0bmNhTjU4?=
 =?utf-8?B?R1hhNDVoK2toKzFJQmRoVXk4ZytpSW4yaDN6YVR6QVVTWnVsZlVod1JjNkkv?=
 =?utf-8?B?UXFSMGR6N2dGT1JMVGF3cHdiS1VBSUNGQ3JybldhVjMrbU5NQjNMMmpWUkNj?=
 =?utf-8?B?WDJmM0hlK3pHNmNiWUlhaVJHQTNwL2NZbVIrYlZRNlF6Vi9ISW5NbytDWkNX?=
 =?utf-8?B?ZmIwNVQxUTNnSTVaeUhSeWJNK1hka2NUTm5qNWxPZVh3RGJmREtwazRUS0dH?=
 =?utf-8?B?L1Z3U1IrUHRzZzhLQ3ROMVh6bHpBQlRCTTNTR25YS0ZzbDJjWTNxUXFIaFZZ?=
 =?utf-8?B?UjRkWmN4VU9rdVZickYwbHNnTm5UazdIbWFyTVlIcmgycStOaWhPczJCTWRW?=
 =?utf-8?B?MmFVOGhXSE10NzZCL1BWbDJ4SUhxeFkrMGNqN3ZwRlhjckozWlkwV0hndTE0?=
 =?utf-8?B?NEtkVzVzSzVOLzM0TkRaZHBWcmVsS2loUnNzNWRsM0x1ZlJMUlB2a3IxaFgz?=
 =?utf-8?B?Y0lBbk5McHhERUxSbXFnc25ZbUVnN3ZTZW91d2cxcDE3VWRzNXl3U3BwRzNk?=
 =?utf-8?B?RTVYeDNuUWpTbjJ0dTFNano0eTlkZXVwendRNVRxdnNoUmNyeDhLVWlvVnJM?=
 =?utf-8?B?MXRqa2ovbDlPYWhHUW9rQ0pnVlhUeVo4NU5oRlloeVEvMVo1Q3grbnZUazlv?=
 =?utf-8?B?V0dpOUV1VytwMER6cHh1WnVlbmpMTUJQeGNYMGR2QzFEME4rTGxJdDI2aXdY?=
 =?utf-8?B?bktOTE1iVEljM0orYTgxWmd6aDBaK092dWJmbzdBaEp1aEtvMmNDeFlzU2ht?=
 =?utf-8?B?NE1PZHZDcTVaVUVqMXdLQXd4NnBIRFRiVllxUXV2WjQ4a05sZ3Y4aThmaVZy?=
 =?utf-8?B?RlFNWFhEQ0VvcmdtbGZwdjlBbHBDTEVwV2J2c1F5WVRQQXJSeEdrT3QxWndM?=
 =?utf-8?B?V256citIb3pYOGdyZndhcU5OK3ZjaTFXa1ByN3pRVURFTC9wWis5dlNwRk9C?=
 =?utf-8?B?c2ZtVWhIWGxGaUlYcDBRdnQ2SUdidjg2Z2sxb0wvVHJLRU1EODJjbGNMSCtN?=
 =?utf-8?B?cHNRY3YxR0pndGFqaHBMaWh4dDBYVDJJVVB0N2FWMlV2YWJ0ekpSVGJiQzQ3?=
 =?utf-8?B?Sy8zV3I4aDZ4V2pIem5UV2lDaHVwODNwL1V1YjlTbHpZbW1HdzgrOTVXclhX?=
 =?utf-8?B?bEE9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: CYX/L7t9FgPPlzfhk/SOKog2WlsmTgJ6WCo7yYCPUC5PGIAcQh7+cd2b7FvrvnrA3miW798O4e038Og+EdC3d/EfuXraO684LzkaaqRAaoFAbJQ4IRgKDGjb2NIPzGTm+oIyS/h1xT3e4SueocIm014G3FKB7MEjIk3YCuGUwkxKLPfLwY/0UJU1WstKvMWy0xaZsrmFc6uu6lH2U4q1XaWAKi2cCHQW0ijaawM31C80vTJBG/VjTD1jsJ+tWwtPsaYaCs4nHwQAjyYadEBfcbXSD3P2RejSBUFRHcSaxnvvPBPi1uBea9/nRDC/7atYHtnYoFbm8iJZEU1hWJD1Z1+ZaUZvJi7VPGEbd3D8ZUwtFq1lveetihxPUwKutIRwKfTZcPzMQaVDz8ANim6+WPdv6D1v/rzQBqJQzefq2DI82G4Gp3j6BmHEuMEwLkIWwv3PhoKdH3NXyr79u2LKoM5MlWKWPwDt4wxVs8eJePVyWQgjmjl7J+XbOxGdq4EI0WA6ir//kKPWVSdgD3HYhUdeftJIpdl6hnSZBH2Qk9amJ+1v33+wf+guaKyPWeuwP19Th5slQ/yebIR+3NoPKd5SftkkzcRsaTGw55fC9XwHcVkCY8ILxDR+Aj8Yc9fJD2ixflwLpRwQGqPMS6e7huHFU7SWYHJQKPbJNFVnaVy1jODaN8AAp6o9JfB+zypxJsG7ddpg3ecV41OXg78bLPusnYqYUB9TZ1zQPG1FVoa8lEFr5HSfYmHZoTtPogQ2tlbYVf9eiw19SAtTxtsz/d/Xce+FfpKEcy+a10BCCgeHkaotmmpryIR2GoMj/TTrhqdx5gEuqs1HvTifLeF3Pt9fVjdbl3sOGdBVVoYCiLJEcc9LjideUz3v10ntJ1Hk
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 247d80e5-e3a6-4ca2-4dc5-08dbc94d98c6
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4468.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Oct 2023 04:58:51.8600
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: L7Q6E7bjk1H4ErrnH+wndgNJuxVArK4R/gYg4eRfgMTyMfg+i4FJpw9ld9I1HxMGOBOkwkiGfX0DPtW5yyE8xA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR10MB7608
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-10_03,2023-10-09_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 suspectscore=0
 phishscore=0 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2309180000 definitions=main-2310100036
X-Proofpoint-ORIG-GUID: Zl86mALMwhFPvRqGnvTOWUIbzbgsbtFq
X-Proofpoint-GUID: Zl86mALMwhFPvRqGnvTOWUIbzbgsbtFq
X-Spam-Status: No, score=-0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	NICE_REPLY_A,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 09 Oct 2023 21:59:39 -0700 (PDT)

Hello Paul,

On 10/10/2023 3:39 am, Paul E. McKenney wrote:
> On Fri, Oct 06, 2023 at 10:32:07AM +1100, Imran Khan wrote:
>> Hello Paul,
>>
>> On 6/10/2023 3:48 am, Paul E. McKenney wrote:
>>> The CSD lock seems to get stuck in 2 "modes". When it gets stuck
>>> temporarily, it usually gets released in a few seconds, and sometimes
>>> up to one or two minutes.
>>>
>>> If the CSD lock stays stuck for more than several minutes, it never
>>> seems to get unstuck, and gradually more and more things in the system
>>> end up also getting stuck.
>>>
>>> In the latter case, we should just give up, so the system can dump out
>>> a little more information about what went wrong, and, with panic_on_oops
>>> and a kdump kernel loaded, dump a whole bunch more information about
>>> what might have gone wrong.
>>>
>>> Question: should this have its own panic_on_ipistall switch in
>>> /proc/sys/kernel, or maybe piggyback on panic_on_oops in a different
>>> way than via BUG_ON?
>>>
>> panic_on_ipistall (set to 1 by default) looks better option to me. For systems
>> where such delay is acceptable and system can eventually get back to sane state,
>> this option (set to 0 after boot) would prevent crashing the system for
>> apparently benign CSD hangs of long duration.
> 
> Good point!  How about like the following?
> 

Yes, this looks good.
Just realized that keeping panic_on_ipistall set by default(as suggested earlier
by me) would not follow convention of other similar switches like
hard/softlockup_panic etc. which are 0 by deafault.
So default value of 0 looks better choice for panic_on_ipistall as well.

> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> commit 6bcf3786291b86f13b3e13d51e998737a8009ec3
> Author: Rik van Riel <riel@surriel.com>
> Date:   Mon Aug 21 16:04:09 2023 -0400
> 
>     smp,csd: Throw an error if a CSD lock is stuck for too long
>     
>     The CSD lock seems to get stuck in 2 "modes". When it gets stuck
>     temporarily, it usually gets released in a few seconds, and sometimes
>     up to one or two minutes.
>     
>     If the CSD lock stays stuck for more than several minutes, it never
>     seems to get unstuck, and gradually more and more things in the system
>     end up also getting stuck.
>     
>     In the latter case, we should just give up, so the system can dump out
>     a little more information about what went wrong, and, with panic_on_oops
>     and a kdump kernel loaded, dump a whole bunch more information about what
>     might have gone wrong.  In addition, there is an smp.panic_on_ipistall
>     kernel boot parameter that by default retains the old behavior, but when
>     set enables the panic after the CSD lock has been stuck for more than
>     five minutes.
>     
>     [ paulmck: Apply Imran Khan feedback. ]
>     
>     Link: https://urldefense.com/v3/__https://lore.kernel.org/lkml/bc7cc8b0-f587-4451-8bcd-0daae627bcc7@paulmck-laptop/__;!!ACWV5N9M2RV99hQ!PDFpjgGTCPjxqCyusua5IZWkvdWEMf51igFDc-yb9cVK9PYr7FpEE1oGpWp09YK4lc15C2taMdcuEOqyH8k$ 
>     Signed-off-by: Rik van Riel <riel@surriel.com>
>     Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>     Cc: Peter Zijlstra <peterz@infradead.org>
>     Cc: Valentin Schneider <vschneid@redhat.com>
>     Cc: Juergen Gross <jgross@suse.com>
>     Cc: Jonathan Corbet <corbet@lwn.net>
>     Cc: Randy Dunlap <rdunlap@infradead.org>
>
Reviewed-by: Imran Khan <imran.f.khan@oracle.com>

> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 0a1731a0f0ef..592935267ce2 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -5858,6 +5858,11 @@
>  			This feature may be more efficiently disabled
>  			using the csdlock_debug- kernel parameter.
>  
> +	smp.panic_on_ipistall= [KNL]
> +			If a csd_lock_timeout extends for more than
> +			five minutes, panic the system.  By default, let
> +			CSD-lock acquisition take as long as they take.
> +
>  	smsc-ircc2.nopnp	[HW] Don't use PNP to discover SMC devices
>  	smsc-ircc2.ircc_cfg=	[HW] Device configuration I/O port
>  	smsc-ircc2.ircc_sir=	[HW] SIR base I/O port
> diff --git a/kernel/smp.c b/kernel/smp.c
> index 8455a53465af..b6a0773a7015 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -170,6 +170,8 @@ static DEFINE_PER_CPU(void *, cur_csd_info);
>  
>  static ulong csd_lock_timeout = 5000;  /* CSD lock timeout in milliseconds. */
>  module_param(csd_lock_timeout, ulong, 0444);
> +static bool panic_on_ipistall;
> +module_param(panic_on_ipistall, bool, 0444);
>  
>  static atomic_t csd_bug_count = ATOMIC_INIT(0);
>  
> @@ -230,6 +232,7 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *
>  	}
>  
>  	ts2 = sched_clock();
> +	/* How long since we last checked for a stuck CSD lock.*/
>  	ts_delta = ts2 - *ts1;
>  	if (likely(ts_delta <= csd_lock_timeout_ns || csd_lock_timeout_ns == 0))
>  		return false;
> @@ -243,9 +246,17 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *
>  	else
>  		cpux = cpu;
>  	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
> +	/* How long since this CSD lock was stuck. */
> +	ts_delta = ts2 - ts0;
>  	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %llu ns for CPU#%02d %pS(%ps).\n",
> -		 firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), ts2 - ts0,
> +		 firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), ts_delta,
>  		 cpu, csd->func, csd->info);
> +	/*
> +	 * If the CSD lock is still stuck after 5 minutes, it is unlikely
> +	 * to become unstuck. Use a signed comparison to avoid triggering
> +	 * on underflows when the TSC is out of sync between sockets.
> +	 */
> +	BUG_ON(panic_on_ipistall && (s64)ts_delta > 300000000000LL);
>  	if (cpu_cur_csd && csd != cpu_cur_csd) {
>  		pr_alert("\tcsd: CSD lock (#%d) handling prior %pS(%ps) request.\n",
>  			 *bug_id, READ_ONCE(per_cpu(cur_csd_func, cpux)),