Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <c87d7939-a88c-ad2d-82f2-866e04692882@oracle.com>
Date:   Fri, 6 Oct 2023 10:32:07 +1100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.15.1
Subject: Re: [PATCH smp,csd] Throw an error if a CSD lock is stuck for too
 long
To:     paulmck@kernel.org
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Valentin Schneider <vschneid@redhat.com>,
        Juergen Gross <jgross@suse.com>,
        Leonardo Bras <leobras@redhat.com>,
        Kernel Mailing List <linux-kernel@vger.kernel.org>
References: <bc7cc8b0-f587-4451-8bcd-0daae627bcc7@paulmck-laptop>
Content-Language: en-US
From:   Imran Khan <imran.f.khan@oracle.com>
In-Reply-To: <bc7cc8b0-f587-4451-8bcd-0daae627bcc7@paulmck-laptop>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Ry9kbjdXbFZpVkhmaUFtaktnRmdEeUlQRzlhSFp5WEJ5cFlqM3JWSmxYZkJm?=
 =?utf-8?B?SDgxSXQ1c3BzcEI5M2NvQlpjZEdPY3pLNTRsK1c0QzJ5bDVjUmVHaUZscDlz?=
 =?utf-8?B?YkswMU1kT1UyT3ozN2R6ZXdDaFMzamd4ZDZlSEZON0pDRy8xWW8rZ01nVFFB?=
 =?utf-8?B?YnZhRGFya3RwN3ZreWIxZjIyMnd6TlV0ZjBVY24zNEU3ekhTRCtVUWVTKzNz?=
 =?utf-8?B?MS9taTYxMGxpeEJYQUpkbjRHR01XbXRPOHNYZmJlelM2TlpvamVKbGVVa3dv?=
 =?utf-8?B?QjhuR3g0dXBuTTJYUUpwZmQ3S3p4SE1WYmpkM2p2dXp2ekx6cEowZkZyTXBP?=
 =?utf-8?B?akd1RUtjTUNkc3Jpb1IvV1pBOXh4YzdJV0w5VG5xOGp4bFBvQnJMR0hBem1R?=
 =?utf-8?B?RENadHBmN3ZPcHZIaVZlT2NPK3ZxQXllc0pJK0FwKzJJMGkvbEJnSW5EYkNH?=
 =?utf-8?B?TXd5bERESlhtZFBYM3ArUFQzaCtkWjFQK1hFYno4bkVITDMxT3RUNG8xWnFV?=
 =?utf-8?B?bDhmTDFjdXVWRVowWk4yYjE2T3NvNThhMUlDZDZPdUdkZmMvQmJtc1VQU0Zo?=
 =?utf-8?B?QzVaanhteU5wOWlVYmxSaEtVVWNuTDN6TmVibXhVbW8wY3VGWUlxV2NKZ3lq?=
 =?utf-8?B?cE1WNkZpQVZ2Um9BcGhjdlBIbmZPU0wwUDFlVkoyUUdzN1ZhMVZlYlRrSlpQ?=
 =?utf-8?B?WlNqMXowaGRsSjNEMDdmcHprcjhlb0lNMW1GbWtLM0prOGFXMUozSTRONENn?=
 =?utf-8?B?YWg2SU9pWG52MUdOTStZeHQrSUxXeTFNQ1BvZ0tyelpqSG9jL0JxZ0xoWFFk?=
 =?utf-8?B?b2I4Qm5USUN2SFBjYWJ3SW5VMmZNUnA1cW1meDF5SFljQngyb09TOEYrbjBZ?=
 =?utf-8?B?TUlqenBlNFRpcko1bEdBSzdoQ3NkNzkyenlBYVFzaThxcTc0UytKWmFmRFBY?=
 =?utf-8?B?YVF3OFpDeFdKbWx1akFuNkJEa1BKWlYvV0xLQzJZbURQc3I0S2dYN1huQy9r?=
 =?utf-8?B?YWJEa29FVmZKVUYrUjdKeWVxYjdmRTNLS2RqZUxoM2NhczFNRGdkZnZkcHRY?=
 =?utf-8?B?WHNUY3c3cjlvYlNJSE1pOGFienZGR0RRbkNBdDBZVFA0TlhlWkFsRXNmdVl0?=
 =?utf-8?B?NVpYVVQvVXNNaXJPMDZyb0tISCsyZnE5MGhoUmRwR0tvcGpGVkhRd09kSCtE?=
 =?utf-8?B?ZVJoMmxycllibnlrekVLbFlLdG45ZU1teWgxcGVYUzdNTXlkclZkUE1TUnp4?=
 =?utf-8?B?VXBpeldEVExCdTJXR2R6Q3NDTGdBUDZHU2ZMajFCWFlaUEdzajVyek02azY3?=
 =?utf-8?B?RlBCWFZnbUVPdEU4cmN4aHJiaEZCUERJR3lKSjFKRFVWN3NjbThFSEtJa2pM?=
 =?utf-8?B?cmFRK21pMFVYdzFmbXNyay9BbXR5UUhLdEtmdVlQQ3NtQ29tRm1YbnZBN3R4?=
 =?utf-8?B?azRjQk4xT3JIRlFHUUxRN2pYTXlMMzNlSHI4ZjQ3c1ZwTnlsSGIxNEUvREtN?=
 =?utf-8?B?Y3BBVnNmQlRHcm9zOHRNaDBQTUFSclA5QkZhSWxOSERWOWdqd1JJVzl6MUF5?=
 =?utf-8?B?SUxIUXBCQ1NOYlNSeUNsSjdNQ214Y2RadTYxTXlBcEcyQjBvUS9DNG5Gczhl?=
 =?utf-8?B?Q3V5d3MyNFJHUTd2UGdOZTBiMWpQNmZrMkQ3NU1DZWJYeVdNczk1UTN5V2N6?=
 =?utf-8?B?MlZjQW1FdmxEY214bWcyaURkQi8vemdLelZzdHVHMk4vakp1eVNJUXV6VFNQ?=
 =?utf-8?B?V3BENUZUQTVmaXFiTFF3NFFtcFIzbktkNndsZFM3YUFKWWpVczFBd2hLSzM3?=
 =?utf-8?B?OEhkbjdJQWZnblc1WkxYWE4xT29MNE8ycmJqZ3hYWE5IM2RvMkllOW1qMkNM?=
 =?utf-8?B?aFg2em95dHgvRkdUbTlZcVpJNWFUNjhDWDB0Um9VTzNoNGpMdmdRbDR6Y3RO?=
 =?utf-8?B?YW4ralFGZ1E0OVVrRzd4QkxaczZOTlJoaEwwZmRTS2p3TVBVNVQ4ZmN5cWpN?=
 =?utf-8?B?cjMzbzQ2azlBeDg0Ukw0K2ROSFkxK00ybFdET1EvdDNCWUV1YStxWXhwdGM4?=
 =?utf-8?B?SFVVN2FheGY2dXdGYzQ5UnNUZmZQRDJsSmpSREZRWlVwRndJNlVwU09RQWx0?=
 =?utf-8?Q?qnTDGQHW1cXmjM5xsEshoQzRx?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: idjSE9b7ig+h3MSVPGw1+NC4iW7a+0m0p85lpQ7iyUnKnNJQMvo5tvm4WNwUYdOGO1TmmRMcDJcLu4D1EE2fA3AsG6aqiisp77GpolUeOtfE2KRGXbURU6S6P7hD3q/prG5ocabspCa0YoGxkICF8PtprCOfHvJ+835UBFwJ8Gl7J19HZ9K6ZBxhNECwzVW6iN6C5ibtO0jih3ymp53LDlJdSdKYTQfhrbRnfcNbYa5H/om4f1wsHQL8KIJQ7kgDHOUMS5w1rR/5zsOVqRJpBTGhpYgd87VUwwFwVr2lRBm4Pz+77EWy3lWc7OXP0jtNF1O57pk6TqzX7HEif4sydCdwyvKmMl64S8vBXoiHRbEGpRVzSuKkDrIMieHBAN4bdiRMs9B/BMa5wMTjS59crIeCvVL1Zm0eb8AOZBfRSrM0pRMLYc3GLzWS28/V+DMDJsnGKNI5wag1AHGIEk0ddAL57Fca2yCqsruxL1bOsXrCTqHrdXebLHNcHDxnZFj7Sm9gndivXuvOBQZBNMLQ1Vm41QTprfRmKwr5H5D6eszDEk7bIsyRWQu7VrihCAGFCzodoAeVmNgj9+h2TvRzFAU/ieOl9zH6x5SZT75f1C3D1CMPnyfLe1P5ynBiqd/UwfZnXN+oWxHnPOTTN1EZIm5we4ykQef6vQnZswNHrvLdbBITzo+PFiKTwCUzsrNxbnU3+ym/nsVCNiRYR+ZwsCLc4sYo/l5ZLk6SROBAdkHnK8wFG7Ih47yhZ2ruP9+/4nMdAAdOxxTeqHnpmUulCGQSZczdbFY2KeNcbuJheSJVYRwIG5DCOUHE5Shk9WLjevb9CT4SQQtG79l56lQGvQOmFiVG7Svm7P8gLORCDbO81CEAHYTlzDsbpghG2M3W
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fb9c5d1a-d3ea-47e8-4020-08dbc5fb4d70
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4479.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Oct 2023 23:32:13.1124
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: KSu+xedySEjZLHiOaLolDHHDFAAKF7++MbVJGaYyWl23p/HVL+s+pANao5RKpvdcW6ldRgZLE3hCJCKMtx/cSg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4768
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 bulkscore=0
 adultscore=0 mlxlogscore=999 phishscore=0 spamscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2310050179
X-Proofpoint-GUID: bWfZIAJcB5WRje3KsoONo35GSm4U-1T5
X-Proofpoint-ORIG-GUID: bWfZIAJcB5WRje3KsoONo35GSm4U-1T5
X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 05 Oct 2023 16:34:16 -0700 (PDT)

Hello Paul,

On 6/10/2023 3:48 am, Paul E. McKenney wrote:
> The CSD lock seems to get stuck in 2 "modes". When it gets stuck
> temporarily, it usually gets released in a few seconds, and sometimes
> up to one or two minutes.
> 
> If the CSD lock stays stuck for more than several minutes, it never
> seems to get unstuck, and gradually more and more things in the system
> end up also getting stuck.
> 
> In the latter case, we should just give up, so the system can dump out
> a little more information about what went wrong, and, with panic_on_oops
> and a kdump kernel loaded, dump a whole bunch more information about
> what might have gone wrong.
> 
> Question: should this have its own panic_on_ipistall switch in
> /proc/sys/kernel, or maybe piggyback on panic_on_oops in a different
> way than via BUG_ON?
> 
panic_on_ipistall (set to 1 by default) looks better option to me. For systems
where such delay is acceptable and system can eventually get back to sane state,
this option (set to 0 after boot) would prevent crashing the system for
apparently benign CSD hangs of long duration.

> Signed-off-by: Rik van Riel <riel@surriel.com>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> 
> diff --git a/kernel/smp.c b/kernel/smp.c
> index 8455a53465af..059f1f53fc6b 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -230,6 +230,7 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *
>  	}
>  
>  	ts2 = sched_clock();
> +	/* How long since we last checked for a stuck CSD lock.*/
>  	ts_delta = ts2 - *ts1;
>  	if (likely(ts_delta <= csd_lock_timeout_ns || csd_lock_timeout_ns == 0))
>  		return false;
> @@ -243,9 +244,17 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *
>  	else
>  		cpux = cpu;
>  	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
> +	/* How long since this CSD lock was stuck. */
> +	ts_delta = ts2 - ts0;
>  	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %llu ns for CPU#%02d %pS(%ps).\n",
> -		 firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), ts2 - ts0,
> +		 firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), ts_delta,
>  		 cpu, csd->func, csd->info);
> +	/*
> +	 * If the CSD lock is still stuck after 5 minutes, it is unlikely
> +	 * to become unstuck. Use a signed comparison to avoid triggering
> +	 * on underflows when the TSC is out of sync between sockets.
> +	 */
> +	BUG_ON((s64)ts_delta > 300000000000LL);
Can we make this a module_param (default value 5 mins), so that if needed it can
be tweaked to a bigger/smaller value?
>  	if (cpu_cur_csd && csd != cpu_cur_csd) {
>  		pr_alert("\tcsd: CSD lock (#%d) handling prior %pS(%ps) request.\n",
>  			 *bug_id, READ_ONCE(per_cpu(cur_csd_func, cpux)),

Thanks,
Imran