Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Petr Mladek <pmladek@suse.com>
Cc:     Steven Rostedt <rostedt@goodmis.org>, linux-kernel@vger.kernel.org,
        John Ogness <john.ogness@linutronix.de>,
        Stephen Brennan <stephen.s.brennan@oracle.com>
Subject: [PATCH v3 2/4] printk: disable optimistic spin during panic
Date:   Tue,  1 Feb 2022 10:58:00 -0800
Message-Id: <20220201185802.98345-3-stephen.s.brennan@oracle.com>
In-Reply-To: <20220201185802.98345-1-stephen.s.brennan@oracle.com>
References: <20220201185802.98345-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?TPodl36N7iFlTTbhCMiu5HkpG2fxEb3df+cO6fiYzBcmS9mAQYRhwownCoB+?=
 =?us-ascii?Q?8N0z5C8SHw0+v1cL6ofdU/YwB+Ljnn3z8RwUDEVyI85uw65wu84MWT1aB162?=
 =?us-ascii?Q?i/0xethrUDeuBHASBv2si5V1c+JXlOxsoHUEa5GUKPLSOLcB3lIAnWAymQdb?=
 =?us-ascii?Q?GP3Zqb5tM1oo1UP6C+vN9vGd2+/YJM4Q7f6iD/5jmJtseOivIMhtso97uUuO?=
 =?us-ascii?Q?VZzQPx4M8SlBcSOcNgPrUq71rj1iScCMUyB5cj99wVXvZNQ2e/RlMQzkmugY?=
 =?us-ascii?Q?N41BdHcvvUAuTtN38GFTI33M5cLf+Fml90aBYUBJZeJHh5PtcKE0zHwbhZvr?=
 =?us-ascii?Q?cOp+tlmBSW4rwxi/6g6Od9q3flv3zJho1IVEQ5NshnIj7gNmf2b7rlQj3jEc?=
 =?us-ascii?Q?6XI1rS5o4k/sUh6nmMgvGddz5TFlQ9Y0CtULqG3Y1a60uYYqWAttCo79GbR3?=
 =?us-ascii?Q?fsuEy+hbOf4R9gJ2HKC0V5WiZO0Ux8x2jjyma/Qb7ruZU4O5YgASiMKyvFQU?=
 =?us-ascii?Q?tbwC12aVsYoMQaYvMEsUfufsCD5gO+tnR7sh0iy1/ejdoCONxeywmUQZLF+7?=
 =?us-ascii?Q?MiznEP9F5QmMZJsjJN0tEJhaq7eodPknxDT+mEuqK4I1uGvkH3e+TWAfcfRQ?=
 =?us-ascii?Q?X/AQgWuuYn327KsnkBPUFBDw6N4Vv+1JJZ4+7gn1DoEqs/nzTMLPI8WhZ7aP?=
 =?us-ascii?Q?/blyIBO+QFPBLUz6d0V7QHfe7sdwd/BQ4vNVYo2Vjxgq6GliSpNz7fxe9uRw?=
 =?us-ascii?Q?HN9MTieQW+stILV7JuhNWW8M07iBhGgBtb3xc6H10+4AlwhySVpcxySu/36D?=
 =?us-ascii?Q?nspsorZ0IpLzSWpBAIWaXbPW42rMsDQTiI1DQ8aqtu/6NWjjb4mz1sKqKse/?=
 =?us-ascii?Q?GLd/Y2Q8b18ZpscRCyMTxH4WZcnK8AiPt/+PnL3SuPm1R1zjPYEqhMxy6cCw?=
 =?us-ascii?Q?QP4JhIEwjKrofJ9yBV7VaQR1XK9HHgYd1s4c7gIHXYj6wIct16uiN/vnDgzw?=
 =?us-ascii?Q?RlJA01vuGpVP3d+CCb9871QvVVjHcHp/+hrWVOjv2EmSCCDOc7UHXhes+nUV?=
 =?us-ascii?Q?He2mQ5CXfulaGMoIFvN3wYFPt0Axqy4cyeGVE892py9DrHfYYPq9gpbZb4dS?=
 =?us-ascii?Q?AoC29sU/cWu+iyNBPp8384VavsFn+SNvqBi5DQn26iI+DmeFT+4XLekaNJUi?=
 =?us-ascii?Q?8rP0A49Zsb9wPk+Gg428cpH5vdyHNqN0IY7jP7nPcP76zBNc/gA46Jt1LM9y?=
 =?us-ascii?Q?V9XdnY1MRFjuOF18BPmHSLmENZsgI9m+veS740zweBCpGZVHl9Z9e3w8/PSA?=
 =?us-ascii?Q?R/WwGGgbiAfMVNM8knjRjs5i9KqpZgh8V11bNsK7LUxDboe34oqGAjPCTNq1?=
 =?us-ascii?Q?uk1BMjRSWyjzLCujC3cIR04EkJIrGYsaH5uDKWTl5oBb2c3TNPAUV/BVXRaV?=
 =?us-ascii?Q?1oJ+BKJZzQ79t7O8pv/nBif76vbPR5xInw/rHoWQNHIw/1Boo/yq2h4mOQ8n?=
 =?us-ascii?Q?3UXP5mzXtYawDw7XiqPYJoLY0KqUaBxTIYIHFgPR1SlDrMjY368LSsSwSsNU?=
 =?us-ascii?Q?bIW5VDKeuln1+TaqmAfcTXrktvpBCUVxK/oLjUeAPhpCrvDXcTVxMD2sSLTh?=
 =?us-ascii?Q?7tG4MYOYJcW4QcnzcFwPt8OpUnuIkoDZomAdn2sq9wclrBEEkgvJwqTX/CUG?=
 =?us-ascii?Q?+7eiuQ=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6979920c-06df-4293-054b-08d9e5b4cb6e
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2022 18:58:12.1369
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gXPX+GpM2wsKlrDcJD6z8fy9iSpuLqq45oenEqKFO/1iaC/31nrJFA8EeupSOhWbNIzXehPFunD/prT9KAZbqq5ZBEmLjKU/I7K/EN8Dzjw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5156
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10245 signatures=673430
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 suspectscore=0 malwarescore=0 bulkscore=0 mlxscore=0 spamscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2202010106
X-Proofpoint-ORIG-GUID: njTyVR6Vx4K2mQB0cWT7ggtpV6I7EBMa
X-Proofpoint-GUID: njTyVR6Vx4K2mQB0cWT7ggtpV6I7EBMa
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

A CPU executing with console lock spinning enabled might be halted
during a panic. Before the panicking CPU calls console_flush_on_panic(),
it may call console_trylock(), which attempts to optimistically spin,
deadlocking the panic CPU:

CPU 0 (panic CPU)             CPU 1
-----------------             ------
                              printk() {
                                vprintk_func() {
                                  vprintk_default() {
                                    vprintk_emit() {
                                      console_unlock() {
                                        console_lock_spinning_enable();
                                        ... printing to console ...
panic() {
  crash_smp_send_stop() {
    NMI  -------------------> HALT
  }
  atomic_notifier_call_chain() {
    printk() {
      ...
      console_trylock_spinnning() {
        // optimistic spin infinitely

This hang during panic can be induced when a kdump kernel is loaded, and
crash_kexec_post_notifiers=1 is present on the kernel command line. The
following script which concurrently writes to /dev/kmsg, and triggers a
panic, can result in this hang:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

To avoid this deadlock, ensure that console_trylock_spinning() does not
allow spinning once a panic has begun.

Fixes: dbdda842fe96 ("printk: Add console owner and waiter logic to load balance console writes")

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
---
 kernel/printk/printk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index b33c2861a8fc..1b96166eea35 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1848,6 +1848,16 @@ static int console_trylock_spinning(void)
 	if (console_trylock())
 		return 1;
 
+	/*
+	 * It's unsafe to spin once a panic has begun. If we are the
+	 * panic CPU, we may have already halted the owner of the
+	 * console_sem. If we are not the panic CPU, then we should
+	 * avoid taking console_sem, so the panic CPU has a better
+	 * chance of cleanly acquiring it later.
+	 */
+	if (panic_in_progress())
+		return 0;
+
 	printk_safe_enter_irqsave(flags);
 
 	raw_spin_lock(&console_owner_lock);
-- 
2.30.2