Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Petr Mladek <pmladek@suse.com>
Cc:     linux-kernel@vger.kernel.org, Steven Rostedt <rostedt@goodmis.org>,
        John Ogness <john.ogness@linutronix.de>,
        Stephen Brennan <stephen.s.brennan@oracle.com>
Subject: [PATCH v4 2/4] printk: disable optimistic spin during panic
Date:   Wed,  2 Feb 2022 09:18:19 -0800
Message-Id: <20220202171821.179394-3-stephen.s.brennan@oracle.com>
In-Reply-To: <20220202171821.179394-1-stephen.s.brennan@oracle.com>
References: <20220202171821.179394-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?hIDzoiTeTla1otpw0CDESV8t/woDUqlqRAqy6lM8c6yZMF1hrDNcbzEjf26/?=
 =?us-ascii?Q?wzEuKW70L94k0DjpeOug6j3wEMiTjtUQZAmU4aDTNaiLJrj5Y/DlI2rcB+zH?=
 =?us-ascii?Q?oYWYfTex5nfQrgJXLvOYKwZaIlW52pDBxPa8nYoeHNcX0lX+9KxTPW1w5ePj?=
 =?us-ascii?Q?HNPfpFl8m2vKBZJ5v8oXInpXt0CCrO7zl8MdnSe1rvV+uCOODdWOYUWJ/0zg?=
 =?us-ascii?Q?t4yaHhIrDUK/YeTx2PiZoTN4qHpYUnr1svwUw8sl8zv99VShEsxQAzQP3ZPb?=
 =?us-ascii?Q?UNNiBIVO0UBC3lL1L23eLSfq5a4PytdbUSgw2bUZhcqff2NnlFf8KB+52aXF?=
 =?us-ascii?Q?nN72SPU6MU3EPtcSSC+9RzSeolycNsJyzI95tvfCWS1kAQbEyMDbF8frr4yR?=
 =?us-ascii?Q?AQ8dBJRo6LghdR4FU/0rUA0bbTO8nmMoX71sXW5eDWwxRt5E1U0BYZJ94Q/r?=
 =?us-ascii?Q?6oMo5QY+aPR2irH8cotbG+M9DBI4SClPr+gHwhc0soG6vaYa4fdK1osgixyS?=
 =?us-ascii?Q?slwHoEmlq9Pl+9V4KlUGQ86p4WnoYVbOlWIFHtEfbjAE6aCzbN+s2qniIFDn?=
 =?us-ascii?Q?BJYY9Far8dlkOWH/z+rFN7bYvoP7HZu0gN/QJBzMZvs03stpbCtffE2XpuXG?=
 =?us-ascii?Q?EHkcQdL3Q7cAtRrt5/oEvd55Liez7wmCRRQ3wIneLMCNwj7Kl+fS031AA4ly?=
 =?us-ascii?Q?7+oFRH75ceQ+BIutAUEa7fAFVc0T9Gl7d6TSmBOxdRH//sL1lg21d3RgzAAM?=
 =?us-ascii?Q?BV9mPS9caz9SeD4HSniiuhujPswsrISnmJ7Nz8lgw24kljVC8VsSVmgP6QNq?=
 =?us-ascii?Q?y3UNRdClAtHFA0aWx+XxiLKXiiELLWammsHLaURh1RtT0hfLxg/Z8wYApQUA?=
 =?us-ascii?Q?RAMoxXQUHrWExDAX/bFJLzyUKzNQ1rkeSY62x7msYkg/qDmZn+K4aGYjhIyG?=
 =?us-ascii?Q?DOm936XU4kvogThXrP4rC1XHVRe0JHWDob8mfgv+zsLCdV96UiRJ33RBccn3?=
 =?us-ascii?Q?0eYz5VGeIkQt11cjIFhtMk5u5T02VZl8CCDD3U8Bl8jojs7/vFpoMEyZ2Zmt?=
 =?us-ascii?Q?jZMh1dexA5+qgOlpliMcXkSqx9Bz0nOza/nrcvbcAbvanokuKprH+hkQDdcU?=
 =?us-ascii?Q?sUJr+5VBIbB+UKXQ4R6CbWSDo1pxldbozU0LM0UtX7mAs2f5pGLkDV0aBE9o?=
 =?us-ascii?Q?rwHjXapRH96BKXBuUnkhc6wL8VE47Zc1QFMiK9gx7PM/70Gfz4Jis24SV+rt?=
 =?us-ascii?Q?kCfXa4ByybLmCEkW7c3HkZPCkyuyQgh+4x/FAfB4rgLkNljAmlpZnuKzsvM6?=
 =?us-ascii?Q?hiXz7L4tTGd2qDa9eQWNFRjkV2T/xGWchQzbbpKJqGumxXLkxg9qehWA9RPF?=
 =?us-ascii?Q?SRlmX6Z7wDUEOwu9mAkLoNYT9D0UGep5SljxZZK6S1tk7pxv7EpxIKPa/NV+?=
 =?us-ascii?Q?D7HfiHRbsgwx4z53YHHnvDJ64Qtx6VL1lw5TjlgDS8WJykhebCpGTn3g3F6z?=
 =?us-ascii?Q?/7aPtjAOI7Jt6ndWzupYSIyugi6Uo8G3xe7ThHoznCjVvzobIQUbA9HUrbco?=
 =?us-ascii?Q?9b2H2SN3Bn37yj+WIeiWktXajcQUpPIYGskY63rZ1Y8gMVkVpspv3unB+SPO?=
 =?us-ascii?Q?9lyUpZFJmKS12RDM1sRdNNEpt4FeKKmS8Ze1HOmgF/oEetRiwwbtKRdY8qTm?=
 =?us-ascii?Q?dA96Tg=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 672ad6d1-3f26-4324-eaf7-08d9e67008df
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2022 17:18:31.0713
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dhn6hM0d6oCWfKbSY9Xu5dlmSJ1mV73pWfXZTQOvOQpoVzmHI60wfMhN+uT5Or2TTpX0i+uJWLM7yDh98aUYgoTp1Rv7ht6oEE6jcH0/eKw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB4236
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10246 signatures=673430
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 phishscore=0
 mlxscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2202020097
X-Proofpoint-GUID: wWoL8pl5TaFwS9tOUh_JVsToIdvai8fA
X-Proofpoint-ORIG-GUID: wWoL8pl5TaFwS9tOUh_JVsToIdvai8fA
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

A CPU executing with console lock spinning enabled might be halted
during a panic. Before the panicking CPU calls console_flush_on_panic(),
it may call console_trylock(), which attempts to optimistically spin,
deadlocking the panic CPU:

CPU 0 (panic CPU)             CPU 1
-----------------             ------
                              printk() {
                                vprintk_func() {
                                  vprintk_default() {
                                    vprintk_emit() {
                                      console_unlock() {
                                        console_lock_spinning_enable();
                                        ... printing to console ...
panic() {
  crash_smp_send_stop() {
    NMI  -------------------> HALT
  }
  atomic_notifier_call_chain() {
    printk() {
      ...
      console_trylock_spinnning() {
        // optimistic spin infinitely

This hang during panic can be induced when a kdump kernel is loaded, and
crash_kexec_post_notifiers=1 is present on the kernel command line. The
following script which concurrently writes to /dev/kmsg, and triggers a
panic, can result in this hang:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

To avoid this deadlock, ensure that console_trylock_spinning() does not
allow spinning once a panic has begun.

Fixes: dbdda842fe96 ("printk: Add console owner and waiter logic to load balance console writes")

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
---
 kernel/printk/printk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index e3d986dd49a1..dfac84876e7a 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1848,6 +1848,16 @@ static int console_trylock_spinning(void)
 	if (console_trylock())
 		return 1;
 
+	/*
+	 * It's unsafe to spin once a panic has begun. If we are the
+	 * panic CPU, we may have already halted the owner of the
+	 * console_sem. If we are not the panic CPU, then we should
+	 * avoid taking console_sem, so the panic CPU has a better
+	 * chance of cleanly acquiring it later.
+	 */
+	if (panic_in_progress())
+		return 0;
+
 	printk_safe_enter_irqsave(flags);
 
 	raw_spin_lock(&console_owner_lock);
-- 
2.30.2