Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Andrew Morton <akpm@linux-foundation.org>,
        Sergey Senozhatsky <senozhatsky@chromium.org>,
        Petr Mladek <pmladek@suse.com>
Cc:     Arnd Bergmann <arnd@arndb.de>,
        Steven Rostedt <rostedt@goodmis.org>,
        Stephen Brennan <stephen.s.brennan@oracle.com>,
        Sebastian Reichel <sre@kernel.org>,
        John Ogness <john.ogness@linutronix.de>,
        Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
        Luis Chamberlain <mcgrof@kernel.org>,
        linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/4] printk: reduce deadlocks during panic
Date:   Wed, 26 Jan 2022 15:02:32 -0800
Message-Id: <20220126230236.750229-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?1h7lxRZKRptALKZOHRHalxIJSFeZQpyLGr5zNBlnaU9NVIRJlqEnfk23oWYq?=
 =?us-ascii?Q?KxGRgM0ZQMk3D9jextESaBwF2ZFLwqTYywkAJghk7Bir/UZ/uMMeAxv/Jgk2?=
 =?us-ascii?Q?MFV2nCRFk4tZBLlNul0aj8mSQsDhgmCluoRvAxDFCj3aSVujtmREjLtaMzwZ?=
 =?us-ascii?Q?Bewi3okELWEiWVKR00lZA2UckNIwXCdBUriObRKOwpwlrtnwGbD2HgRkaf3c?=
 =?us-ascii?Q?PJnLWKqxriVz+7gPkmTkG5HqMWz0B2A8kqfckD6vq0AckBD6GEiwOJDpkp9I?=
 =?us-ascii?Q?9ABvrsVwUxxQ06BoWIlMAE8Tev+NKoDuY+PSv0GRPfRG8IVgSaCDEBB1xwB4?=
 =?us-ascii?Q?PQMuuMipEPNNmmiejOQ5uf4hRVoFgiQG9Q9H1SIUelxCswWZIDxGwTYJhyo2?=
 =?us-ascii?Q?5YJngBqLob9/3Icp1hd7CnuNemFgp2SmFcRIFjLaIYNFEHWHGmpa11dN0GSO?=
 =?us-ascii?Q?xdxPQ1VTNwbzzVtckKdjbmnG5XdSyhGv7AHRng+lHTMZ8ZMVOJcCb03lwLqm?=
 =?us-ascii?Q?R+gsYYNQNuIpWQ56to2LcZ5KgApt4NTb+W940rdlxEt5ZI/RF/BXl/gaeAGg?=
 =?us-ascii?Q?rKaJ9Cz+t4N1PodgC9oQ5vm3xOA+X1iSKu7BgGYBQpiAjRI37jowj6NuFptc?=
 =?us-ascii?Q?V/P8OBJcrktRCyeyAVtgqlj58i9CSG+drOhT8bSqxfAGlVnTYqYi99b6QJgP?=
 =?us-ascii?Q?xrOcuw7BchRe15lUDkh/DdvGfGa1wfkF0MG5QxgGIF0PIzOyD0h0BUEG4tlk?=
 =?us-ascii?Q?UTmhH7b3BOI0eZTJbu3ABVXIpLYjf+4D81qBZSqmN7WnFbV01dhcCc7tzC4V?=
 =?us-ascii?Q?F8+qkgE6FX81Iev/H0ZasHT+W3e5R2GJRBI0cMm0FKSe6vV7Ovn9DoDjygkH?=
 =?us-ascii?Q?HWcq3uqlUZuy8tu5W9JV1hEa4SlK1m3YzPAjDqOB5oz80tF9z/g0xhWtHq3g?=
 =?us-ascii?Q?Fo7uTsdLT/j689NTGljOD0S7hF4f3ze/A+nJY7nqTZdTh2AZ/U3RczvFq8hJ?=
 =?us-ascii?Q?HC+eQozNT0t+hpGCCDEv3A6+xHUV1i+Fw9rlIg1Yau+UHgvuiEQMUgQhYDYr?=
 =?us-ascii?Q?lwALPbYUgXf6W+VkMXg+DTKF/dKJt0ALkBwUfx6QLM7NIVUh0uJsld475dqS?=
 =?us-ascii?Q?2hjwPDG7wf0ryWkUhbn+PyqlT3PBaqz9c372BTOjhzgh2sK6C0W2QCVP3e1C?=
 =?us-ascii?Q?N04pOR52A1hz9ygnYRPMtdv9xXfyi6okH0srJIfJp5mQdB231Rgu6B2z+An7?=
 =?us-ascii?Q?QtJELPYIMSzphAPFJqwM3YU9caRDAmVOKg6YhCsY9JeF3IwxN+75gLApPCDK?=
 =?us-ascii?Q?0oNAtnJ4W/Xd+lFEI6RuwbZtWw4T1qwYCR3cwMtbEHrbf0FOiPmDCJePfa5h?=
 =?us-ascii?Q?p1IGnymvB/Xmu9vX7W+NF75WhRSWTsiXRJVZ2cA4xfOR3AreGRR/OjjnyVml?=
 =?us-ascii?Q?v7/ZUZAo5J53f68e7Y5aw6Eiez93JCfz+BIBsY5rzazoXcnTqBttfyvmDs5a?=
 =?us-ascii?Q?9p2efX8VR0ZeP+TmFu3qFc0if861D1/eBU44beMIvp5tisqYZa7Xp/fS/WGN?=
 =?us-ascii?Q?/1zw3rY5hcWwtzKDPwt6jVcZyF5NBNW2hTROUE7szFE5O+hfkCKyLCJnOu7A?=
 =?us-ascii?Q?ExaIsVCsscpjdtNC1izWkz0OJ+vZaU9lyJgF7DGd5cUDiOb+TsgvSaE4Bqu0?=
 =?us-ascii?Q?4p2l0g=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6117f5ca-6a02-4c2d-b26c-08d9e11ff015
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2022 23:02:34.0793
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fbKi5zqRTRbx3PC5BOtpJ3PTy7eagwGH7PEQnxJOo1hKgVN5LQvkgWfcTqi/zaTGqVmvQ3sdK5kJkCTlg0EabW2BeJHnqcplYo2mC0XC4f4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1615
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10239 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 malwarescore=0
 spamscore=0 phishscore=0 suspectscore=0 adultscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2201260131
X-Proofpoint-GUID: _6aGNX_lN00LnvAcdcON-fbVMf-kvcCZ
X-Proofpoint-ORIG-GUID: _6aGNX_lN00LnvAcdcON-fbVMf-kvcCZ
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

When a caller writes heavily to the kernel log (e.g. writing to
/dev/kmsg in a loop) while another panics, there's currently a high
likelihood of a deadlock (see patch 2 for the full description of this
deadlock).

The principle fix is to disable the optimistic spin once panic_cpu is
set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
the console_sem.

However, this exposed us to a livelock situation, where the panic CPU
holds the console_sem, and another CPU could fill up the log buffer
faster than the consoles could drain it, preventing the panic from
progressing and halting the other CPUs. To avoid this, patch 3 adds a
mechanism to suppress printk (from non-panic-CPU) during panic, if we
reach a threshold of dropped messages.

A major goal with all of these patches is to try to decrease the
likelihood that another CPU is holding the console_sem when we halt it
in panic(). This reduces the odds of needing to break locks and
potentially encountering further deadlocks with the console drivers.

To test, I use the following script, kmsg_panic.sh:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

I defined a hang as any time the system did not reboot to a login prompt
on the serial console within 60 seconds. Here are the statistics on
hangs using this script, before and after the patch.

before:  776 hangs / 1484 trials - 52.3%
after :    0 hangs /  408 trials -  0.0%

v2:
    Each patch has minor updates from code reviews. I've re-done testing and
    updated the above statistics. Exact changes are in each patch.

Stephen Brennan (4):
  panic: Add panic_in_progress helper
  printk: disable optimistic spin during panic
  printk: Avoid livelock with heavy printk during panic
  printk: Drop console_sem during panic

 include/linux/panic.h  |  6 +++++
 kernel/printk/printk.c | 51 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 56 insertions(+), 1 deletion(-)

-- 
2.30.2