Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Petr Mladek <pmladek@suse.com>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        John Ogness <john.ogness@linutronix.de>
Cc:     Stephen Brennan <stephen.s.brennan@oracle.com>,
        Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
        linux-kernel@vger.kernel.org
Subject: [PATCH 2/4] printk: disable optimistic spin during panic
Date:   Fri, 21 Jan 2022 11:02:20 -0800
Message-Id: <20220121190222.572694-3-stephen.s.brennan@oracle.com>
In-Reply-To: <20220121190222.572694-1-stephen.s.brennan@oracle.com>
References: <20220121190222.572694-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LTEqNGCFerC5KPbos8mlgNpI2IqdnIJXl1mNOrqD6OcIfymtcqw5dTVIhlZk?=
 =?us-ascii?Q?KBpBwPMuF7Xqq7/SXzH4Y3o1G2WzvdlALNl7aBAxH73qb/5ZDaargX+grHCk?=
 =?us-ascii?Q?L43rpclwTN1VweyrdhIXAmbsESMhHi/EN6NnkuwhgpllHuffaOkG1pBAS8mf?=
 =?us-ascii?Q?HOx/Ll3U3VgDV8pKgqwJhdRg1FCshnO02gwTEntEnYiZXu7e0+9PtKDm0NjW?=
 =?us-ascii?Q?xGAjv6tCL7y7ba3RvkbV9sw++DxZtMnzCI5tB+zCw9skepy4BA963G875XN1?=
 =?us-ascii?Q?lj4jDoy35fgVpiWfBxtCzrwbBJUNe7+Vaw0G3oHqy1sOGES4FTWzRnSt4UpE?=
 =?us-ascii?Q?oReFb9BhZQP0rgDSs9C9mwsBzzpGHHrjLuVrJuFk86iHWA3zdkrQWpeMsYK0?=
 =?us-ascii?Q?3OI6J1vFp8q1Rn/MK8PdxkLKx7NK4v5FrGbT3iRcBV6rchaa3C9QCsQqNEhS?=
 =?us-ascii?Q?ZJVT0Pr1wYBycMpW9qT7w2BjQc5ZnLhBxPHIXbMPoLDyn2x19Z8mYvOMa6tr?=
 =?us-ascii?Q?FtI3l50fYEk2l7aknVFs7O5ux79Ck4c6GwxJ5OH34++A5Z9g32ciMRpQ4whk?=
 =?us-ascii?Q?L75fwZ+zyvQIqzXu7CyCc8J6uGI4vQA1Q6NetcAQ0PwaRg8xn4VimDtReoWH?=
 =?us-ascii?Q?UqMRchLadKxaB++bp4M7vDJDppg+DRIDIWbQWVUuiVd8V92C6Dg5ijUdne/j?=
 =?us-ascii?Q?LepPqpNVhweqkHGDrKjXRRTsrJ6VVolDHloASy/2gdHL3X5P6eWAc35m0XQg?=
 =?us-ascii?Q?wcU40sXGEdV5siOW4OOxTCEWg6IPAaK/SkPS0YUc3yBcEpHrKpjOAix/2Av4?=
 =?us-ascii?Q?IQJZ2JGbsmVkvbmIDAFhY/kK+gJwRw4TnXzmnZjl+bdhQEc3Yl8CIl2+1df3?=
 =?us-ascii?Q?UO6YRreQvtCnBROYXzXQd1/vb9iWseQgugC27AmBHV4Q9Jm2Wk40R95gfjKi?=
 =?us-ascii?Q?cX22Du1Tp9Mqy3VC8fHFYrhfmjTgaAGlJUiI+K321vGhzmdbV50QlUTlQ9t1?=
 =?us-ascii?Q?YZPkKDvFzis+eu47++ZdNs3o5QQK4mYQumLInmxvS8AL4XJ9CqxO3gbDe+s8?=
 =?us-ascii?Q?JhzobakhLcssnAcIra9rMstjxh7yEi936mR84OCRgT14uivySdIR6kN5rH8a?=
 =?us-ascii?Q?mBwWc8DWtMHbaBve+ujJZiJ+04Jfz4kyyc20Pjnd+TTbaRiDAZ1W51N3cOXG?=
 =?us-ascii?Q?p2STGJpQkfHsRcrC2m+MWgS8pRQ8ZLvZfJJn8+QJgEeG9Rqyh/77u9fF0xHt?=
 =?us-ascii?Q?62k7C8sCm7mR7LYDvQBsldz+OggYJjfenDPR4FhGXfe1Nbp8vd72EV2IgI3p?=
 =?us-ascii?Q?olMSny44vNXUESkals75kMLScXOai4SP45vmrVm2zIpfeHoOoLO7AbbGoRy2?=
 =?us-ascii?Q?L05DVemhIAIuV8ApDnd3vaFxtiD/oFlBsCMbXneb1Ukpi9pzLGkkh1m7OIRl?=
 =?us-ascii?Q?tvJsnf0Hx3dlQtgcBE+Bj0S4cDrW4ONZwdfyVFCd5Do9X2DLE9IldMX4EZNF?=
 =?us-ascii?Q?veqx8ESGEUfrdC29rMVE1zuNcOva1XCNZzbuNtxPHgPsHcOj3hdiRAUUMr5o?=
 =?us-ascii?Q?ASBg5eLp6Ili+2+wRFQkOWmcFQ0EoQ6lQmch/tGDpwD2jDWR5D4J/iahfjH9?=
 =?us-ascii?Q?8RCrhjrj7XqouAdKrx71PmqPglnlNPEgzgTA0HjPDkn12A12rv4cMuc6PUQP?=
 =?us-ascii?Q?iaBntw=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8a3de880-1c10-4e53-e630-08d9dd1093e5
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jan 2022 19:02:32.2043
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 08CNw2gVVhvjbRzcIy0UCZ3ffs9eGeOXVSMjSF75Y17vDpO/Q0LGcsWjyj9WUiVQfvRkTwTR1shwPg2EKkDskPcCjtP5DI3MLYyWl+Ke7Gw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3434
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10234 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 suspectscore=0
 spamscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2201210124
X-Proofpoint-ORIG-GUID: TfVJSRHuiPbJuyFKrp3OSEtSREKq6IMP
X-Proofpoint-GUID: TfVJSRHuiPbJuyFKrp3OSEtSREKq6IMP
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

A CPU executing with console lock spinning enabled might be halted
during a panic. Before the panicking CPU calls console_flush_on_panic(),
it may call console_trylock(), which attempts to optimistically spin,
deadlocking the panic CPU:

CPU 0 (panic CPU)             CPU 1
-----------------             ------
                              printk() {
                                vprintk_func() {
                                  vprintk_default() {
                                    vprintk_emit() {
                                      console_unlock() {
                                        console_lock_spinning_enable();
                                        ... printing to console ...
panic() {
  crash_smp_send_stop() {
    NMI  -------------------> HALT
  }
  atomic_notifier_call_chain() {
    printk() {
      ...
      console_trylock_spinnning() {
        // optimistic spin infinitely

This hang during panic can be induced when a kdump kernel is loaded, and
crash_kexec_post_notifiers=1 is present on the kernel command line. The
following script which concurrently writes to /dev/kmsg, and triggers a
panic, can result in this hang:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

To avoid this deadlock, ensure that console_trylock_spinning() does not
allow spinning once a panic has begun.

Fixes: dbdda842fe96 ("printk: Add console owner and waiter logic to load balance console writes")

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
---
 kernel/printk/printk.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 57b132b658e1..20b4b71a1a07 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1843,6 +1843,16 @@ static int console_trylock_spinning(void)
 	if (console_trylock())
 		return 1;
 
+	/*
+	 * It's unsafe to spin once a panic has begun. If we are the
+	 * panic CPU, we may have already halted the owner of the
+	 * console_sem. If we are not the panic CPU, then we should
+	 * avoid taking console_sem, so the panic CPU has a better
+	 * chance of cleanly acquiring it later.
+	 */
+	if (panic_in_progress())
+		return 0;
+
 	printk_safe_enter_irqsave(flags);
 
 	raw_spin_lock(&console_owner_lock);
-- 
2.30.2