Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Petr Mladek <pmladek@suse.com>
Cc:     Arnd Bergmann <arnd@arndb.de>,
        Steven Rostedt <rostedt@goodmis.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Stephen Brennan <stephen.s.brennan@oracle.com>,
        Sebastian Reichel <sre@kernel.org>,
        John Ogness <john.ogness@linutronix.de>,
        Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
        Luis Chamberlain <mcgrof@kernel.org>,
        linux-kernel@vger.kernel.org
Subject: [PATCH v2 2/4] printk: disable optimistic spin during panic
Date:   Wed, 26 Jan 2022 15:02:34 -0800
Message-Id: <20220126230236.750229-3-stephen.s.brennan@oracle.com>
In-Reply-To: <20220126230236.750229-1-stephen.s.brennan@oracle.com>
References: <20220126230236.750229-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?UaG0Zt8nXxTRvSaJSSeBxaiKT9DJo4QssFjqLwxV8h04KlXwMDXEsIod1QhS?=
 =?us-ascii?Q?smnqBs86beITcahZ0hzvyxTMnD7oHwHaJIloJp+CoA+f1apmUx4Ht8lt2VJr?=
 =?us-ascii?Q?fZvX+OrIKlctc66fMSfsyla9xztocClqWWXt+G/fLcaAde3n8MUimi2UKSR/?=
 =?us-ascii?Q?8nmSHXBsgrLWULn6/0Qop19TFfzqlv+e25FPTqlKkNg6hXTSD1LEKfwO+GaH?=
 =?us-ascii?Q?xPHdDBdUhlu4CbsOhn0EvaQFDznbfw9w8xUkbEP/ZeznI2WJdKHbQaSqHRXy?=
 =?us-ascii?Q?REo6MOkybBQVnyIoeaYlgSHKgZP8yZBncU2YoUTZNEZrumyJMCgXRzsWJBKa?=
 =?us-ascii?Q?k1BHlncjsI0m8jn02IHM+6y52v62eQH8lksEiMG84aUeC6KVErE155yBqQ+A?=
 =?us-ascii?Q?V2HI+n/CZ3+5FCLAICN2BiTQI+FA3dCD/Tsnd4Nb1nVlhFhYzhiESN5Odt0K?=
 =?us-ascii?Q?p/pHEUgE+pT6pI9oEmsff432Kr8xyRR4pwAkdx/f0d8MWVnCf2gmP4kC9mwQ?=
 =?us-ascii?Q?Ddz/YLnXzW1eEtUZCUraG1luvcb3OzcycLZ7PT5v3XIIBda84drhSNck+MHN?=
 =?us-ascii?Q?DVyqASh/JKsJRkUVaWuhCgb5KZLoxFsDYkaNf1j4ZtQE8jq9guSLuIRd8FFv?=
 =?us-ascii?Q?Fr9wNnOG1/yu9TIlmqZka27pyfyMoEjvqIMCi9zD8lMENtXMDOSpIM1X1hnc?=
 =?us-ascii?Q?0CBIY85wYxmp+DCQrGWpl0wOndzvrGqnGVdYjkHen9vOd2T4eKcRDz6j7+e1?=
 =?us-ascii?Q?B+LmIdXXzQqlK8MagKiRpQygzYUr4VCJP7iv6oFv/km98CgGssYiUphlfzgz?=
 =?us-ascii?Q?+8itS0WpkKqjXi4+Lb4RMpqjr4rMgVQCUGZmwwp7v8f9lHHwoQNxhrLTbM7r?=
 =?us-ascii?Q?Eu0+YDNSPaVMTo/XrpqRYhdoPRG0cVB3kEVQQ9gMyktm220A+e7AdRcTUehJ?=
 =?us-ascii?Q?wVqN12I76w+QqZu2giaSVjKxHLB7/J/UbmtHGMAVa90ObZtVn3Z7j/uWwix6?=
 =?us-ascii?Q?VNrk5sscUZdIEyBJPTQH7G8xd+XKgmUvuVbbuvknwr+32VdVm6KCF4rx/UmM?=
 =?us-ascii?Q?S5xoomJ0UGxGf9eM6/qIiWjpHgVzGiWqgTxXzXo5YWwvQ2c95l7a++Eq0u0u?=
 =?us-ascii?Q?GnTK2nfip8pYAGSGubieq5US1Sy1OSla158YPCVur/FS5h/Q2M6WquukHOBJ?=
 =?us-ascii?Q?U0rjNJQf248SqR8XVfZknDt0jIgeuVsJJIrsR2gZdqOEbr+VOSVmej4hSNKK?=
 =?us-ascii?Q?LqhdxiXbU7m28ipB8A/4JkUhl1ar91oD4NkBayIjmFYRmcaaw4vNqfG95J6k?=
 =?us-ascii?Q?Ds4kcTFtMfhmd/AaQRuITC9b1nZfFa5LabA6agoLcieejDFCNWVNB9jUp3B5?=
 =?us-ascii?Q?FoyPBd826xS1prXqMbUJ4/pSQdyZ9tiEtt1xT0tSa6/fttfwSrOFPkBMz3k6?=
 =?us-ascii?Q?BCCTN7XvGLB5nQaG6QwcfKY6TU0iucGg9A4HUnCjuDKi+e92xfelnkLgotWy?=
 =?us-ascii?Q?jL3AoZHXXIGXscGafJgGLRLD05wjUUltLpxrbdN4CB759g4ZK13zM389b1hY?=
 =?us-ascii?Q?Q78yFiPNOu1QilOVMM9yqrliRDzE3bzyNyr5TgiEnJzjxh1BJ5mF2zHNu6l8?=
 =?us-ascii?Q?7zAkz5l4Ct9YQRcNUpU3nomYBF2Tj1aSJ7+b+w+V6wh/3ovmYHAJZOpZH8eu?=
 =?us-ascii?Q?xjgbZA=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e02f06f6-d864-40a3-c3b7-08d9e11ff309
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2022 23:02:38.8937
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8Kf0Fl8PTUN+AB00qfB5l3vGoTlfui7jRTcqCgwcWmphgwfC16a3krwBq5hvQWWaKZytxWat+BJzeesSu5INmjrODH+vOCWeBbDnFm7ZTu4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1615
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10239 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 malwarescore=0
 spamscore=0 phishscore=0 suspectscore=0 adultscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2201260131
X-Proofpoint-GUID: lRpGq5qzcR_cATYwJguxenTXFvafJaKO
X-Proofpoint-ORIG-GUID: lRpGq5qzcR_cATYwJguxenTXFvafJaKO
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

A CPU executing with console lock spinning enabled might be halted
during a panic. Before the panicking CPU calls console_flush_on_panic(),
it may call console_trylock(), which attempts to optimistically spin,
deadlocking the panic CPU:

CPU 0 (panic CPU)             CPU 1
-----------------             ------
                              printk() {
                                vprintk_func() {
                                  vprintk_default() {
                                    vprintk_emit() {
                                      console_unlock() {
                                        console_lock_spinning_enable();
                                        ... printing to console ...
panic() {
  crash_smp_send_stop() {
    NMI  -------------------> HALT
  }
  atomic_notifier_call_chain() {
    printk() {
      ...
      console_trylock_spinnning() {
        // optimistic spin infinitely

This hang during panic can be induced when a kdump kernel is loaded, and
crash_kexec_post_notifiers=1 is present on the kernel command line. The
following script which concurrently writes to /dev/kmsg, and triggers a
panic, can result in this hang:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

To avoid this deadlock, ensure that console_trylock_spinning() does not
allow spinning once a panic has begun.

Fixes: dbdda842fe96 ("printk: Add console owner and waiter logic to load balance console writes")

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
---
 kernel/printk/printk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 57b132b658e1..20b4b71a1a07 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1843,6 +1843,16 @@ static int console_trylock_spinning(void)
 	if (console_trylock())
 		return 1;
 
+	/*
+	 * It's unsafe to spin once a panic has begun. If we are the
+	 * panic CPU, we may have already halted the owner of the
+	 * console_sem. If we are not the panic CPU, then we should
+	 * avoid taking console_sem, so the panic CPU has a better
+	 * chance of cleanly acquiring it later.
+	 */
+	if (panic_in_progress())
+		return 0;
+
 	printk_safe_enter_irqsave(flags);
 
 	raw_spin_lock(&console_owner_lock);
-- 
2.30.2