Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Petr Mladek <pmladek@suse.com>
Cc:     Steven Rostedt <rostedt@goodmis.org>, linux-kernel@vger.kernel.org,
        John Ogness <john.ogness@linutronix.de>
Subject: [PATCH v3 0/4] printk: reduce deadlocks during panic
Date:   Tue,  1 Feb 2022 10:57:58 -0800
Message-Id: <20220201185802.98345-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?hkDS4cpFgOXnwpkUUhcj86CtPk45uATB0fsIGP/PUQpqinJEWby3O7GBK+eo?=
 =?us-ascii?Q?eEzYNiGhl4RwkH4e7M2PoLGcyWEC4pnZOGnePk4ZzrF5+1bvIsK1//l63OSA?=
 =?us-ascii?Q?2ieKGbYQXA+uBHKHQK6xh2avuo5ntUKjJzA70zrp/eciATYFZXS+KS2DVl/R?=
 =?us-ascii?Q?axA5LlAh/P95PkfhxteLA7q4jANWz+3D4THuBrK9jgxrNdui2olKrVarEpo6?=
 =?us-ascii?Q?p9Ysf0lBqEncaXKWv5J/wrW9D8zICXGFnpzkzIm8CJ8oH8px+lBcpYiZUrwY?=
 =?us-ascii?Q?UP9BtFovF3LP4Q+Dla/uNVq9xZDipWIyWS6Ilc9nSHnmOj1j7YmE6Q6ey34y?=
 =?us-ascii?Q?X+fjf14SNqU3u6o3/LVhEpPUzkWPlA9sHOx8a6JkmT8dGnoRfbp4kijHNOFg?=
 =?us-ascii?Q?PkPww6SGWplFRiY2eAd0cRRuUp+WTjYiWDW48moPNYNkCunI5AS7JSWCa13V?=
 =?us-ascii?Q?hol/7ckl3oAHnVF78pkNOdf3vw+dNV1bJYFqDALCNpMHGgUQdqEsIKKmwFZl?=
 =?us-ascii?Q?oP32jiQAYSo6adBr+SRlJKEiKolSjtW+C/Vj/tJI27q6dLUQIhKqjUeRx9Tm?=
 =?us-ascii?Q?kLRw9Stebge2eztm2Ome+8mkMzFDPCBdBwR4GHIrrmhweBJL3iK7X5PRTTIl?=
 =?us-ascii?Q?cP17qcqVeora9AZDLjeRoOwtTSUDjRZLHtuN0GxUf52zB96G5apE27XMQxQx?=
 =?us-ascii?Q?8BeBBB3FnhZJ52iy3k9AuOgLXJY6BQWC3fKMcBKJryLCkrxe4ekesS7Xxiem?=
 =?us-ascii?Q?Yov1+kjbOmdWQIhojmnmrSxqve5L8ix9BKfh/kZ7ddlZQypA8oyTvXI/lxFx?=
 =?us-ascii?Q?Kl7v/UkLXAltr5g3Yj8+JJg1fqirkn6c1FAvl1UvFXHCdtqQSNi2yOrVY4F1?=
 =?us-ascii?Q?rcCjN2GT+oCvpDuSNtwQxSpe4l4c8UgVghvqp7XBqKOyIXQVwJ3I97qEk1Wo?=
 =?us-ascii?Q?Z4Lsbk3Va/mAkYYw5dAd096qck5WRsxpCFjUZDFyamJj0SkhWlFQKG572OfE?=
 =?us-ascii?Q?hibGExL7KPLouGxE8tpv0ra6rXk+v7Q3GtpZBrzXw+hZiUqN5nBNQ+1Hs4xD?=
 =?us-ascii?Q?XPj6jh0fRc5/r8ptNjClt7hqZxV7mkYNdXkvuXGjeO80Bfpe2LxU+RMFnYcl?=
 =?us-ascii?Q?rqKE42RQRf40i5rEspOqZD8UdIYiV1fvPcm8KnrwLQm2LdU2RYIyjfNBrM5O?=
 =?us-ascii?Q?dOTv2tHeIWdrA4aUixr35aU0fglqqFL/7zAzQO751NwSoe9wuRKzJ/Pii4es?=
 =?us-ascii?Q?/2WmYvh2p79i/bi+Bm+wge9acllAmM8FT+LcvplkcxbPFMJ+DuRxPEYLcuXT?=
 =?us-ascii?Q?uy6YfEu4abuXO8GByjRJxVaqTtzUPEeLwiIy28OvWtz0R2/gBi9QmhYIPHby?=
 =?us-ascii?Q?GEFNY85HDDkLCT/9xJCYn67Vl/nK7yCtX5mnGBi6C/QTf9FQuXFdw2AWm9Ek?=
 =?us-ascii?Q?atcpe2ZEV+VYn5J4X5okkuVCYAcgbwRCErbyXB4O0Ql0EoNH/qlu4NbzF9zU?=
 =?us-ascii?Q?keMpBJXWvme3KsubmVZjIiHCjJUcZahOHRVMJ7Ktl8AFYNvWetDcBUg8OZkQ?=
 =?us-ascii?Q?f5/4EKEuz81k04N+ROezBN/sYa7PELJyXqnD7eA+gKOtBGFSbYUrsNl8udQj?=
 =?us-ascii?Q?G3PwxPxfVxxSEEQLn2L4F+wTk8h7KhmyDomM4hs8p8iBa3FvrFs2o8X2j+f9?=
 =?us-ascii?Q?lbt2VA00/ZZCwsBR3ejRrBGBQ/Y=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4431498e-09f0-4d9c-fabf-08d9e5b4c7a5
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2022 18:58:05.8240
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZZH4dGHGgJ8sBvQJ+T+s/Z3Zy9JcgmVSWYrcYoPb5K+Fk54/jfOUXOwl4AalwRdA8+UM2EtPWvWtX1rSEe3yWHgawlnibsCDhH+mQ820a1o=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5156
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10245 signatures=673430
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 spamscore=0
 bulkscore=0 adultscore=0 mlxlogscore=999 phishscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2202010106
X-Proofpoint-ORIG-GUID: zQA7NN0G9szhfBokyyRr6md30gGKQktN
X-Proofpoint-GUID: zQA7NN0G9szhfBokyyRr6md30gGKQktN
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

When a caller writes heavily to the kernel log (e.g. writing to
/dev/kmsg in a loop) while another panics, there's currently a high
likelihood of a deadlock (see patch 2 for the full description of this
deadlock).

The principle fix is to disable the optimistic spin once panic_cpu is
set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
the console_sem.

However, this exposed us to a livelock situation, where the panic CPU
holds the console_sem, and another CPU could fill up the log buffer
faster than the consoles could drain it, preventing the panic from
progressing and halting the other CPUs. To avoid this, patch 3 adds a
mechanism to suppress printk (from non-panic-CPU) during panic, if we
reach a threshold of dropped messages.

A major goal with all of these patches is to try to decrease the
likelihood that another CPU is holding the console_sem when we halt it
in panic(). This reduces the odds of needing to break locks and
potentially encountering further deadlocks with the console drivers.

To test, I use the following script, kmsg_panic.sh:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

I defined a hang as any time the system did not reboot to a login prompt
on the serial console within 60 seconds. Here are the statistics on
hangs using this script, before and after the patch.

before:  776 hangs / 1484 trials - 52.3%
after :    0 hangs /  15k trials -  0.0%

v3:
    Some mild style changes, none of which affect testing (which has run
    continuously and is now over 15 thousand trials without a hang!)
v2:
    Each patch has minor updates from code reviews. I've re-done testing and
    updated the above statistics. Exact changes are in each patch.

Stephen Brennan (4):
  printk: Add panic_in_progress helper
  printk: disable optimistic spin during panic
  printk: Avoid livelock with heavy printk during panic
  printk: Drop console_sem during panic

 kernel/printk/printk.c | 55 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

-- 
2.30.2