From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Petr Mladek <pmladek@suse.com>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        John Ogness <john.ogness@linutronix.de>
Cc:     Stephen Brennan <stephen.s.brennan@oracle.com>,
        Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
        linux-kernel@vger.kernel.org
Subject: [PATCH] panic: disable optimistic spin after halting CPUs
Date:   Wed, 12 Jan 2022 16:54:25 -0800
Message-Id: <20220113005425.74988-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?xH1eD9ff1geC5KA4gn98O/qyQR8j1fWEPFFB48TIgeWJ/kthybVBLs4INZNk?=
 =?us-ascii?Q?ea+Bvkx4g4GgMLS7+UVrs50dhzypM3hrdYy99Gw78X4QRdMfuyr34ItRG+qo?=
 =?us-ascii?Q?q0YSUB+4Bkv4Urgq6xH1TEnn08edoCQ2Z72ufQYE0lDD60YI2PSLwPH+BNBt?=
 =?us-ascii?Q?GoED9D8Vqmip1aBwq2XIyqZVSTJT/qJ9VUf4NOFk4tq+5LBH4N/sGW/y61FX?=
 =?us-ascii?Q?GHExFRMCrCoOMWmfJBiJ9g7fr/yysEmaS1bXaNt6gHrPrza4Qs6LdFDDZRFQ?=
 =?us-ascii?Q?Lfl1bfoe0OSmvQj755KTY1dBS4L7/9Ceed2+7qCYrXYbbkCiB2yTbse2gNWc?=
 =?us-ascii?Q?7ZU2WkxyFrfbKFLQtmZGB9etTfu+KkpJJaF/ZGianau2sYdTJtVssGTXT+EC?=
 =?us-ascii?Q?oVgLPAmoosZUlkLCX60HlWHJ6eqaGJTih+rePqMWxdM/h4cUK+e/3p1NuI7B?=
 =?us-ascii?Q?Vj2oM5VgaOktN/v11S2+bzi5pgvIQs9dFIdRvmSezSV7HpMHPRaomIZe84/8?=
 =?us-ascii?Q?iMMTgEOnhvB4s407o9A91KdnUh13gfc25B6F1Zy1dHYQyzEHI7IYxM8hm1sm?=
 =?us-ascii?Q?hhV/oeLZIPiVtkY6B6qZ3Uv8Kst+YZdN4G579FWzE1qPfYrdm53iKUGIX12w?=
 =?us-ascii?Q?ovtnWbH6PSbG07JZb1hM0m9wAOou/kZtA9QUJrgiBwbk0uekEOBQVrdu4YJ4?=
 =?us-ascii?Q?Umc7WktVxXRmU2e7q7yM9RwQsBM4i286a2nqBcu04h6EK5GQkJmOSrkylTr3?=
 =?us-ascii?Q?pE6cq+mwGz19oMYvvtl1uoSmLT0DlcMP6mQGYx6wBkP2eKdY6vsee6Uj/jRn?=
 =?us-ascii?Q?RgGRuwgLI/2aLmClaFv7u1OqBqusaxx9CxEELu2L6Rf72bo18vHFCWs7NNFG?=
 =?us-ascii?Q?pk6Lv5zwZ8ohTPXp32dRTvtUAwdCVGOCIkV9B98IranJqNDHMoiZxQKkAxLZ?=
 =?us-ascii?Q?/aQI+vqZiv/fzfUVuvgKmRxFaDmxH5lhlgI2pxAtaDsl04IsFHp8TIw/PNHT?=
 =?us-ascii?Q?V97FxQLDmO/jU8xmLADqv+zNInAs8XtAhqyP+VpmkaOF4SaD+033h+y5cejI?=
 =?us-ascii?Q?2qwcIGxr2FIzQyU0UgMgjk39N1Sdc0t9Zdbpf/vqa4G9Hyr9SWxxKP9dwBPt?=
 =?us-ascii?Q?2SCMEWK1ZhpKbbgDcEL64hMJlk69qa7sHdhHJ9PpDf6aLKUItIb6rJXU+x38?=
 =?us-ascii?Q?zXobB5sUr0d4M015btytLjf7s4BGIPbl4mBwjTJw8MCDrO/c8IRcn2wF9uLf?=
 =?us-ascii?Q?pFYFERB3siKpiT9kalV9hCLo0q9QPY74gJzWaQLPONaxOylJsVUWxDysa8Kt?=
 =?us-ascii?Q?Cq/nqOa4KxOloXlz4quZsDoe/oCqXGt5iEpcI/IV7mZjW1PJN0+BtFiEoVML?=
 =?us-ascii?Q?Vrh2crv+S69xyd1pw98YywyJFD4HwbR431Bs0vfLlDCp6yQESMPIDsJ4x+Om?=
 =?us-ascii?Q?MdYhPWDCjaveFptb1bdtxAIDTnnNlgi3+LTHPvKCUTFBpXc/qOxSqEjUF71G?=
 =?us-ascii?Q?j1zgUKgL/yfsyoF+A8XE3Jko8yr+adr7HqaAeUFmv9KdmdI5fox/5Jb4ZlNX?=
 =?us-ascii?Q?GaR8ACLgljnCqECLd9VuP3CVzrUVAklbAcmawVOfszrXqTnYJ7BGSJA1aR6j?=
 =?us-ascii?Q?iokjU50Xq0bG6LNujxpG/E2qJfaHOOFxTFHyLp5u2yLdDTQfCFdwKT2NGAYx?=
 =?us-ascii?Q?G6vltw=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8ec77593-f50e-4f7e-a797-08d9d62f40c0
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jan 2022 00:54:28.9685
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: soMjMWrBD9zey/jebBErXhlPe5ZVOcQwBq5FqgqjSZ+oHC0b9B0R0FlVSDQJTEgyfj3nVm122mUlmAK+87DEtUYvMIXZDplJje2zo4LhFLg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4908
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10225 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0
 phishscore=0 adultscore=0 suspectscore=0 mlxscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201130000
X-Proofpoint-GUID: qMMAcFuuVgIE1rX1jFKZ-openFWhKF0g
X-Proofpoint-ORIG-GUID: qMMAcFuuVgIE1rX1jFKZ-openFWhKF0g
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

A CPU executing with console lock spinning enabled might be halted
during a panic. Before console_flush_on_panic(), it's possible for
console_trylock() to attempt optimistic spinning, deadlocking the panic
CPU:

CPU 0 (panic CPU)             CPU 1
-----------------             ------
                              printk() {
                                vprintk_func() {
                                  vprintk_default() {
                                    vprintk_emit() {
                                      console_unlock() {
                                        console_lock_spinning_enable();
                                        ... printing to console ...
panic() {
  crash_smp_send_stop() {
    NMI  -------------------> HALT
  }
  atomic_notifier_call_chain() {
    printk() {
      ...
      console_trylock_spinnning() {
        // optimistic spin infinitely

This hang during panic can be induced when a kdump kernel is loaded, and
crash_kexec_post_notifiers=1 is present on the kernel command line. The
following script which concurrently writes to /dev/kmsg, and triggers a
panic, can result in this hang:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

Below are the hang rates for the above test case, based on v5.16-rc8
before and after this patch:

before:  197 hangs / 340 trials - 57.9%
after :    0 hangs / 245 trials -  0.0%

Fixes: dbdda842fe96 ("printk: Add console owner and waiter logic to load balance console writes")

Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Reviewed-by: Junxiao Bi <junxiao.bi@oracle.com>
---
 include/linux/console.h |  1 +
 kernel/panic.c          |  7 +++++++
 kernel/printk/printk.c  | 17 +++++++++++++++++
 3 files changed, 25 insertions(+)

diff --git a/include/linux/console.h b/include/linux/console.h
index a97f277cfdfa..4eeb46927d96 100644
--- a/include/linux/console.h
+++ b/include/linux/console.h
@@ -179,6 +179,7 @@ extern void console_unlock(void);
 extern void console_conditional_schedule(void);
 extern void console_unblank(void);
 extern void console_flush_on_panic(enum con_flush_mode mode);
+extern void console_lock_spinning_disable_on_panic(void);
 extern struct tty_driver *console_device(int *);
 extern void console_stop(struct console *);
 extern void console_start(struct console *);
diff --git a/kernel/panic.c b/kernel/panic.c
index cefd7d82366f..a9340e580b20 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -265,6 +265,13 @@ void panic(const char *fmt, ...)
 		crash_smp_send_stop();
 	}
 
+	/*
+	 * Now that we've halted other CPUs, disable optimistic spinning in
+	 * printk(). This avoids deadlocking in console_trylock(), until we call
+	 * console_flush_on_panic().
+	 */
+	console_lock_spinning_disable_on_panic();
+
 	/*
 	 * Run any panic handlers, including those that might need to
 	 * add information to the kmsg dump output.
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 57b132b658e1..d824fdf7d3fd 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1823,6 +1823,23 @@ static int console_lock_spinning_disable_and_check(void)
 	return 1;
 }
 
+/**
+ * console_lock_spinning_disable_on_panic - disable spinning so that
+ *	a panic CPU does not enter an infinite loop
+ *
+ * This is called once all CPUs are halted. A CPU halted during a section which
+ * allowed spinning, could trigger an infinite loop in console_trylock. To avoid
+ * this, mark console_owner as NULL.
+ */
+void console_lock_spinning_disable_on_panic(void)
+{
+	WRITE_ONCE(console_owner, NULL);
+	if (raw_spin_is_locked(&console_owner_lock)) {
+		debug_locks_off();
+		raw_spin_lock_init(&console_owner_lock);
+	}
+}
+
 /**
  * console_trylock_spinning - try to get console_lock by busy waiting
  *
-- 
2.30.2