Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Petr Mladek <pmladek@suse.com>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        John Ogness <john.ogness@linutronix.de>
Cc:     Stephen Brennan <stephen.s.brennan@oracle.com>,
        Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
        linux-kernel@vger.kernel.org
Subject: [PATCH 3/4] printk: Avoid livelock with heavy printk during panic
Date:   Fri, 21 Jan 2022 11:02:21 -0800
Message-Id: <20220121190222.572694-4-stephen.s.brennan@oracle.com>
In-Reply-To: <20220121190222.572694-1-stephen.s.brennan@oracle.com>
References: <20220121190222.572694-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?VGXlcd4DicS6bIVlrx0TnEv48qPvz+iOb07PL3DdWvT58DZjIMR0omDEXVkP?=
 =?us-ascii?Q?H5TQqsHI5cICO9lfaeFS5ehW48db43iRmQOsoROnwfLD864Qc/smfodAL7Sb?=
 =?us-ascii?Q?GaL+KBu5IHg1/2D9+LSwhbIrlt7FJvXPtrxX1uHhy83pwDZ0+MoXzniYov1l?=
 =?us-ascii?Q?6khmu9/d2vqbbTgVP6qJnrADcKuGdDgK/DLkyLAbznn1vrKftMmCiAQH1+GA?=
 =?us-ascii?Q?0mFua5Og8fyg/ahli++Bpflopp6aQwB07NFqogUccvDwRGaaFyvfx8bkpIXD?=
 =?us-ascii?Q?cKZQdvgZsj2CamcF0Yy0UWrJiSYoK87ytN8HIcD59qvn5rpAvaUBS2wsDyyL?=
 =?us-ascii?Q?fmCQh0Aq79ODmwDaXG4eVgQcl/JdqLEMYD1GrgM7GQ40cSsGI+59x9iudExP?=
 =?us-ascii?Q?+inOJbczM8Nqzkf16pzEBfpBaZhK7MwivXP64MYgX0tKQfjH3UbKiIphjbyh?=
 =?us-ascii?Q?teUco86HUd6USmI37lbmekKEYDpw7Rn42qivOZZcsKxbMiDKRiBkneHC72Qk?=
 =?us-ascii?Q?aRxINFMN5wAKZvl314VOVi4tpgw73Y5ys8Y4d5qFh+jjlU5vUlOOXPFo1oPE?=
 =?us-ascii?Q?0u8MQiqIPAJ3H1H1yB3BNb5u4a7Usmk2nhiTGUBThkMHjOEcQjz87SpmKxp7?=
 =?us-ascii?Q?zIkUi7gw7RjnZmphY9VLMjten9NL2qkIyser5WAgLCam4QmW/86QF+wQADM1?=
 =?us-ascii?Q?Gn4qqKLCky/av0fE4+o7ntOfQu5eSdJ6ybcYizzvUeUqFDaqDb+qh4wCKJPA?=
 =?us-ascii?Q?xCQrwUJbtn3fFhctW/uAhcZvR7YuhqR9zCeHF0xJ4yvvTuHMEPUk+Z3+pLC4?=
 =?us-ascii?Q?oVkUeVbTfLmdGc/akX4PR/jQ4IeSZQohGqIZsXfGYxCrsNBKujo9bgJIUdO1?=
 =?us-ascii?Q?UUKUVHsbP1Gc+jp0FAEQV083OskOJV8GoN85wjVVL3J62fh9+8JSF0U9fvEj?=
 =?us-ascii?Q?Zbv5bT0Fw9xQcR4E7cQKcilu5cOjkp0Z9rD4NoxjNF5gjUC/QJduY1FO8lah?=
 =?us-ascii?Q?hT1cRdfQ/8BaKXHWSAZIPoZ3GnFumzSFw3Shm2nQpVS4BRIDlQhbYfscYwmR?=
 =?us-ascii?Q?G1Rdf4qlDVX07cCZuw8AKuv4S1kysoiUn5g0qs/+3nikjsiPVPPhkfK+WcdM?=
 =?us-ascii?Q?BBKKBCUp29SYIdS24BReJOpWVYCCzwmb8lpLo0HBYcavHYcO0XsAE1bZD+9t?=
 =?us-ascii?Q?egNWJgcktbxo6cMaCCn546eLWxwX3mCtFqxCdksi7eSnrLyzQZ6Vww6xABbB?=
 =?us-ascii?Q?uBFPOX34NYn3G/2+W7zrNGZ0mt+6bM62DNTUD3my4wWZDA8qYHt2NH/Cr4/t?=
 =?us-ascii?Q?+SuEyFK9q5CQ+oiCnEcpIgMLorD/56k9sRqgItUdTuQ4hRVp6uHlW64XsAEm?=
 =?us-ascii?Q?Gj5123dkyv2RQBxVaAhyAGqgH0IPbxe2Ng0dGvi9ucJB8Fztk99u1JenVQGD?=
 =?us-ascii?Q?hKqFnndl9O0bar8N6zL/G1/Zi7zEb/ZfU0TIRhnTuatAY8sAnLBv/H5fIqmP?=
 =?us-ascii?Q?jfE0PfdiNKxST6I4YrvbeN7ssNb58npBL0rbqIeCDMYxGcMMMdmxNRO8IIpT?=
 =?us-ascii?Q?T4RKJn8s1etPQXDijOVsPearB8NuKC1RZdhwRPu6OoS/yJThF6WZPEjliGvB?=
 =?us-ascii?Q?6eehUt3ZdmJ0x+nPZfPOeNduVwkhMZyzvyFlQLdgNy5YjK/Ynm4AXyA7FBIz?=
 =?us-ascii?Q?qeupXw=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ee4a1600-ea23-4466-278a-08d9dd109582
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jan 2022 19:02:34.9216
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: wsc4zBIH8tNwwbo8UIEHjRM3DP9fOMRiPMcPN3d8ElTazEkt3IefUzKvJU8+M0n1gpJzo4M+HNgfzD5GnUkGglRV31fxlXmR4w2M87wIUXM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3434
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10234 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0
 malwarescore=0 phishscore=0 suspectscore=0 bulkscore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2201210124
X-Proofpoint-GUID: DJKxD6TSlBX9Ytjb6oB4INJRS9OZB_oB
X-Proofpoint-ORIG-GUID: DJKxD6TSlBX9Ytjb6oB4INJRS9OZB_oB
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

During a panic(), if another CPU is writing heavily the kernel log (e.g.
via /dev/kmsg), then the panic CPU may livelock writing out its messages
to the console. Note when too many messages are dropped during panic and
suppress further printk, except from the panic CPU.

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
---
 kernel/printk/printk.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 20b4b71a1a07..ca253ac07615 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -93,6 +93,12 @@ EXPORT_SYMBOL_GPL(console_drivers);
  */
 int __read_mostly suppress_printk;
 
+/*
+ * During panic, heavy printk by other CPUs can delay the
+ * panic and risk deadlock on console resources.
+ */
+int __read_mostly suppress_panic_printk;
+
 #ifdef CONFIG_LOCKDEP
 static struct lockdep_map console_lock_dep_map = {
 	.name = "console_lock"
@@ -2228,6 +2234,10 @@ asmlinkage int vprintk_emit(int facility, int level,
 	if (unlikely(suppress_printk))
 		return 0;
 
+	if (unlikely(suppress_panic_printk) &&
+	    atomic_read(&panic_cpu) != raw_smp_processor_id())
+		return 0;
+
 	if (level == LOGLEVEL_SCHED) {
 		level = LOGLEVEL_DEFAULT;
 		in_sched = true;
@@ -2613,6 +2623,7 @@ void console_unlock(void)
 {
 	static char ext_text[CONSOLE_EXT_LOG_MAX];
 	static char text[CONSOLE_LOG_MAX];
+	static int panic_console_dropped;
 	unsigned long flags;
 	bool do_cond_resched, retry;
 	struct printk_info info;
@@ -2667,6 +2678,8 @@ void console_unlock(void)
 		if (console_seq != r.info->seq) {
 			console_dropped += r.info->seq - console_seq;
 			console_seq = r.info->seq;
+			if (panic_in_progress() && panic_console_dropped++ > 10)
+				suppress_panic_printk = 1;
 		}
 
 		if (suppress_message_printing(r.info->level)) {
-- 
2.30.2