Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Petr Mladek <pmladek@suse.com>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        John Ogness <john.ogness@linutronix.de>
Cc:     Stephen Brennan <stephen.s.brennan@oracle.com>,
        Sergey Senozhatsky <sergey.senozhatsky@gmail.com>,
        linux-kernel@vger.kernel.org
Subject: [PATCH 0/4] printk: reduce deadlocks during panic
Date:   Fri, 21 Jan 2022 11:02:18 -0800
Message-Id: <20220121190222.572694-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YJ9LSiWpa4Dj0BmcMrjbDXtbPU+vvyczMu4eMV5VHuEiDcnZ+7i/KutwkzCF?=
 =?us-ascii?Q?kHgQsouqZAByv8bqEqUNSPGPZh/ZbJpPSe93bBOpOa5KjK2S2DblDy8BG6k+?=
 =?us-ascii?Q?A9ogZ3UJbAvP+hCF5AFiAiPzsO1n/hwMVUMJAbDa85zdPXsp1QiIowNin5Cb?=
 =?us-ascii?Q?VqDMvDH1UL6EIB7eQJxJ9EdEQhr8x0sOUEYWatvlj3ZCMNHRWpyEvhP8VYqY?=
 =?us-ascii?Q?dDuugXsf0hGGoi3ZpV5zw/jkuVrxCFk9RdDHhFEXtjJr4r0TuExmwUTiX2OO?=
 =?us-ascii?Q?+vBIuyfxwZ1KFhjW0aLSi7/TOwhchqdklSwK/6hqkXVHFovMkYEGVmoeVjC+?=
 =?us-ascii?Q?D73Mi/cJh3nllK5w5v/dMFxP3Whh59IYQ2PkwHbv1IpLwdBNIT52xyVGPZ7D?=
 =?us-ascii?Q?KJZp2gHWjvW5DGSArsSjJL88xgmAX6R/+n5Xj0S9Wts0GM0K2zdmhjvvGdmc?=
 =?us-ascii?Q?ZRlfNFr3xubwfbHGIk7BdL+4JbCSOwW5uwmBrjQ5q1qomyAjC99uz48wARUb?=
 =?us-ascii?Q?bZe71fT4u6kVA1XOYG9/R11UPabB9Q3euq7UJjPGR9uj3fJB5ciFawlPQ+5W?=
 =?us-ascii?Q?uZFeqVi95YUvC3AnFg3Lr642OE99A1i+n+aHE2BDeseUMvccsLLm46xL23HK?=
 =?us-ascii?Q?33clzPWQdo+Ouk2J8Y9llEX+fVlLcO55hH2+1Dl+2+VrqVgS7RIQkJsq2Nle?=
 =?us-ascii?Q?1C5XMyy0ZiMdHhPthwUJ4rGsSeqpQbOMbp5gWOt/6J8c7nlzDemVYh2DxHuF?=
 =?us-ascii?Q?+yHWafgP3ZWcjCL/jDld2nt5b4/EslWTkOeoeP7ikaOllHa9Wm45DBjfzCXt?=
 =?us-ascii?Q?zxUsFxSW93+r/cAef79jhCgsM5jozLc/f+ZH27BSbK1pLkQBe+P9OwoQ7GUX?=
 =?us-ascii?Q?Ba1cxrxSfX7E51+eHGT0kiRhsyKaCLqO/ei4I3RYoH/Pkk0dgkNr383+0g2D?=
 =?us-ascii?Q?1dteo/m3uPVoOoe3FwNgD6ZEgL0E3QqJCKIwNgz82XZu7S8mFj66vpMLVpdp?=
 =?us-ascii?Q?c1AhAL+tAnfOfQLnb86YhQe7h5AeV19zsmtuAmN3Ryh7CpQG0jmFv/MLehCO?=
 =?us-ascii?Q?sX0LIRRnhMQ5ocY5bDejhf5YNuqo15QBlyjOKa44df6WuC8dRDNfl6K3jqEk?=
 =?us-ascii?Q?PlyiB8ibslaF3gHNMECTRtv3iuUjzQbDz6v1UiHcsbgYayt5pzuu0mQfH5kz?=
 =?us-ascii?Q?nGUTd4agD27tLat21qhutlpW6XMC7NOfwjkrZiZCGRx38lYoqV/bmwP/dCEZ?=
 =?us-ascii?Q?aPh2ZUl+yTQYHf/BTXWK//0ZN7iGFvq2oqukB+QQ//Zf++96oFl926hZInSn?=
 =?us-ascii?Q?p81z7pYnYujn3/+l2dHdo1cZ8pf8R73DjGNyCJ8DslWXM1IY4IjB5VgH2hGD?=
 =?us-ascii?Q?ewMRpnsrjybFFksiCoiSoulByLsEr1Hx0zQAsQMkg//COcG7OazYJzOW8WkL?=
 =?us-ascii?Q?XvekKNDI8xUUixnYC5Rtln7dHFaixDkcQi4b4jljHWhwfs3ejneAQe/rpGcR?=
 =?us-ascii?Q?+hJX5Cx1ADizThneOZtZ7k9wOWvfoFSt/bYJ9W+xecmy44oBNPsfuO2zgWoD?=
 =?us-ascii?Q?7u5JA3vhmbED+QENvvAJptvDhrHYdpnFeUb/lbN6kFPq3UaAjbHyzr+koYWa?=
 =?us-ascii?Q?OYPBBGXstb4IV7hzRe4ypBWRmzvwTzPl1+HTnGUCqkPIjSrPQKLhqd+bEr8C?=
 =?us-ascii?Q?QOcoAQ=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2db6d2df-7cac-427f-c850-08d9dd109076
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jan 2022 19:02:26.6147
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: v3mSF+5L2NvlbkNVkPu6GfMbkjitIC6lhYdX6s92nGmkwYMljzEYLWtyfnIz5FhmXN2/3zB33999837+/zcVD97wxz2z8qikYCnJPCIr3Go=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3434
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10234 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 spamscore=0
 suspectscore=0 mlxlogscore=953 phishscore=0 malwarescore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2201210124
X-Proofpoint-ORIG-GUID: gJSEsHWXwKoEDmbX6vxku-r1W_jn5crN
X-Proofpoint-GUID: gJSEsHWXwKoEDmbX6vxku-r1W_jn5crN
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

When a caller writes heavily to the kernel log (e.g. writing to
/dev/kmsg in a loop) while another panics, there's currently a high
likelihood of a deadlock (see patch 2 for the full description of this
deadlock).

The principle fix is to disable the optimistic spin once panic_cpu is
set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
the console_sem.

However, this exposed us to a livelock situation, where the panic CPU
holds the console_sem, and another CPU could fill up the log buffer
faster than the consoles could drain it, preventing the panic from
progressing and halting the other CPUs. To avoid this, patch 3 adds a
mechanism to suppress printk (from non-panic-CPU) during panic, if we
reach a threshold of dropped messages.

A major goal with all of these patches is to try to decrease the
likelihood that another CPU is holding the console_sem when we halt it
in panic(). This reduces the odds of needing to break locks and
potentially encountering further deadlocks with the console drivers.

To test, I use the following script, kmsg_panic.sh:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

I defined a hang as any time the system did not reboot to a login prompt
on the serial console within 60 seconds. Here are the statistics on
hangs using this script, before and after the patch.

before:  776 hangs / 1484 trials - 52.3%
after :    0 hangs / 1238 trials -  0.0%


Stephen Brennan (4):
  panic: Add panic_in_progress helper
  printk: disable optimistic spin during panic
  printk: Avoid livelock with heavy printk during panic
  printk: Drop console_sem during panic

 include/linux/panic.h  |  3 +++
 kernel/printk/printk.c | 31 ++++++++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)

-- 
2.30.2