Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Petr Mladek <pmladek@suse.com>
Cc:     linux-kernel@vger.kernel.org, Steven Rostedt <rostedt@goodmis.org>,
        John Ogness <john.ogness@linutronix.de>
Subject: [PATCH v4 0/4] printk: reduce deadlocks during panic
Date:   Wed,  2 Feb 2022 09:18:17 -0800
Message-Id: <20220202171821.179394-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nKMws4f5Z116jhcudT2+fiBIKByhDr5h7EWNBh+aUc5sWD+HIfl5jk6y1VYs?=
 =?us-ascii?Q?51FBmp4xICbCH/12w/yigt67cJn7e2/amLu8BV6UTbOXNiEWUGqU55xDkh0G?=
 =?us-ascii?Q?zvwWi5dBXraQsra8hVJ40yIoOc2OAi4OIwX9yS5beLaQUc28aarau7ADCvFi?=
 =?us-ascii?Q?YuVgffZ9RRxHM4JYpdEEdLPHmDWYgtudBYCSujMswh6RZsgcuCmElueZra3d?=
 =?us-ascii?Q?rHxO/kiGz3grBWINFQh4fY2ihVXfOvO6N6zeyx1FSl9LG7lreV7GU6SmLfvL?=
 =?us-ascii?Q?rVA2hV3bAElgRLSG7BDM9Gch7Iv7INDO9i3trS8gePWgPua9kWYNh0o5lOug?=
 =?us-ascii?Q?391PbO9sTqhbtjlsH9x5aODhx9z+qkCTXPnBgO7Wk5VHObaWvWlen8n9QAKb?=
 =?us-ascii?Q?AOVWyIiDOKdFy8q49QS2GdbxpyASd/373xQ1PF65ACU+ZijYvO6hgsEuXDW+?=
 =?us-ascii?Q?C0QMcZPu6YiuHZHMFkfh30S6tN6jOIPiNaYLuBgG1APeJmNstX+2xSgm+b57?=
 =?us-ascii?Q?BibCHmOdlEVqt88mOxqZRceJRoolzywRVVsHsOJs2WHrilZb/XeyyqbwfK2h?=
 =?us-ascii?Q?ZhZso80rNg/j63tVEK7WYFnqJa6f68B3kr+swav+JmGOiUpRvd7gSOaixeOB?=
 =?us-ascii?Q?ui4FZp7JjtAHfMufC2wiOkxsGBajfFvSOydLdB5/Mpi2Qg8NqWI7o8VtysM9?=
 =?us-ascii?Q?naLLJzp88VRteM6/Tg8A8nR9g00qQiczDh/bUYblkAAZmyHnkKzAAKNL1EwE?=
 =?us-ascii?Q?M2HZpY38KGvwybkBwYjwhKU9G8a/J59zPO7BZe3ehWAVlquRMcQe93jkFi4T?=
 =?us-ascii?Q?d+lWkOu6b+rk2ZceKt/trnim5HGnhQRQDnnRzJ12O26dQL5iCQKxoksBunUA?=
 =?us-ascii?Q?rzCoahv3jY0i15Gtn6m35Gob71pyz2pHOvG0Jdksma/jua2/BEdpy9JL5oqE?=
 =?us-ascii?Q?hQcWkVW/Kiz84CFuyQhC+Ml+0bc2iqh2BFPAGXk2Gpbu3boH3JlHY3nUTPmV?=
 =?us-ascii?Q?rUFCjErmMaEZBq6u9lRtqJGa7iMGE0cWNSUlULnRYxmxCDwzsCGvt8CiYDFt?=
 =?us-ascii?Q?5Tm5ZQFOvV5ZD8MOs5UgdMLZH8K8snHRPuEBPmNK/7DndQ/yhIO+8Ti4Q1ro?=
 =?us-ascii?Q?qM9aWuiiDarNHHqxQAKjcMRXfH9f5D/met2zb7hFIx/CgItGCFffKfrQ0646?=
 =?us-ascii?Q?sWb/1qaZPyBUET4zMuNYEsnwG5vnjrD9kDTTRFnYLUqsgcQ/GrVZ/0q6ZnkN?=
 =?us-ascii?Q?/6Fx9Ch6qz37H39L/TqyQJhKKeslfL3hK89aGCFtXgsllEVvMmUy2I8sr09k?=
 =?us-ascii?Q?Yk1zD1NY9QA2S8zLEDeYUaUgTukrPodcIYlBJUyAvbe5skV2mbK3L43EiPKh?=
 =?us-ascii?Q?KsEyWcwH6CgmPEguNJhqtZHIO2UfpaPUa4TY56j/QrPYnHjB4pQiOUSwfqQY?=
 =?us-ascii?Q?HVppbL8ODhPQM0P0ww5kc43HEYmd73QD5fon2JNuoKCfHWEYcfv7vm7W4Vmh?=
 =?us-ascii?Q?07lWi6QkuLX16CJy8LVorMhjy7WIDD6Ow4ItoAefMyvq59lNvYxHz9A7R0jS?=
 =?us-ascii?Q?79gkZEENtogP4UD9t1UGZGVauA6kWG4EKmzUkLhFxjqU8X/VGroq7fB+CDbO?=
 =?us-ascii?Q?2205OS0Jsfm/7+0tAyvj+j8CiuwzeyjhAHhItIJPNWfbipoxyuYzDvg2W2ay?=
 =?us-ascii?Q?ZhGuJQ=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6923948c-08a7-4c3f-d453-08d9e6700582
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2022 17:18:25.4540
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: j0qAbQ//8Ndkdc8V4lG4Lh/kI+hmgNddlupG5iFxlpa7TCPRl9f4n1B3dzooHTLny/O7ZJGziRT6EBJcMVv9drwdJtQa7WWiLEXJMhWy4s8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB4236
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10246 signatures=673430
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999
 adultscore=0 malwarescore=0 bulkscore=0 suspectscore=0 mlxscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2202020097
X-Proofpoint-ORIG-GUID: X_07Ee2ILLyEyxOzFXIRVYUa9PvFcU7m
X-Proofpoint-GUID: X_07Ee2ILLyEyxOzFXIRVYUa9PvFcU7m
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

When a caller writes heavily to the kernel log (e.g. writing to
/dev/kmsg in a loop) while another panics, there's currently a high
likelihood of a deadlock (see patch 2 for the full description of this
deadlock).

The principle fix is to disable the optimistic spin once panic_cpu is
set, so the panic CPU doesn't spin waiting for a halted CPU to hand over
the console_sem.

However, this exposed us to a livelock situation, where the panic CPU
holds the console_sem, and another CPU could fill up the log buffer
faster than the consoles could drain it, preventing the panic from
progressing and halting the other CPUs. To avoid this, patch 3 adds a
mechanism to suppress printk (from non-panic-CPU) during panic, if we
reach a threshold of dropped messages.

A major goal with all of these patches is to try to decrease the
likelihood that another CPU is holding the console_sem when we halt it
in panic(). This reduces the odds of needing to break locks and
potentially encountering further deadlocks with the console drivers.

To test, I use the following script, kmsg_panic.sh:

    #!/bin/bash
    date
    # 991 chars (based on log buffer size):
    chars="$(printf 'a%.0s' {1..991})"
    while :; do
        echo $chars > /dev/kmsg
    done &
    echo c > /proc/sysrq-trigger &
    date
    exit

I defined a hang as any time the system did not reboot to a login prompt
on the serial console within 60 seconds. Here are the statistics on
hangs using this script, before and after the patch.

before:  776 hangs / 1484 trials - 52.3%
after :    0 hangs /  15k trials -  0.0%

v4: Moved panic_in_progress() out of #define CONFIG_PRINTK
v3:
    Some mild style changes, none of which affect testing (which has run
    continuously and is now over 15 thousand trials without a hang!)
v2:
    Each patch has minor updates from code reviews. I've re-done testing
    and updated the above statistics. Exact changes are in each patch.


Stephen Brennan (4):
  printk: Add panic_in_progress helper
  printk: disable optimistic spin during panic
  printk: Avoid livelock with heavy printk during panic
  printk: Drop console_sem during panic

 kernel/printk/printk.c | 55 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

-- 
2.30.2