Received: by 2002:ab2:6857:0:b0:1ef:ffd0:ce49 with SMTP id l23csp3157749lqp; Tue, 26 Mar 2024 00:41:38 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCXwT5n6dSIxijeBMnmRDhMoZWnMsxc3QgPlBtjshmbU4j5T/Z/PvlRBaAi59yifCvcYvd5JC5qCU8gb4Y8qYpfLdlU7iiiorxz82COSYg== X-Google-Smtp-Source: AGHT+IEKHOyzpRFZp8Y7/MWzIMeaSNTRwb9d53m4WzbjABjDeHbKsAu8FxQPA0xzbAYjrTnud7pK X-Received: by 2002:a17:907:20b2:b0:a46:617e:d3a7 with SMTP id pw18-20020a17090720b200b00a46617ed3a7mr1259960ejb.60.1711438898716; Tue, 26 Mar 2024 00:41:38 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1711438898; cv=pass; d=google.com; s=arc-20160816; b=vyZyKt2HxFyb8hqqUImirEckrzq7ajnlc1UOm2fX6SYgWXd9c8SCA+ZJO+2Ac1rNMm lou2s+sbVitqRvbTVk5neEwSlY3WEV0lp6kHQRT0zi5HT+p5RDTxviYFgWJ+aWrObrYP BFB4GAO9wliHbKqHQkAYTq/tXjPRl3sJfWvz1KEO+HGKosc+voyJ8+GSgr49/oOic01q Acgok/1t7we35G03TOs6EiLJzzH8hWdemYW0995KLyaFhzJ68RHkZCCiJ1LDDkeUh3h7 sF+2MrpVCYsUF0QvEqVCo6sIh863j04JTNA6cGwJH9+Yy30Kl/B/vKRkZDrkDOkiOZPc hh+Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=LKgcwe8CYMngn2SkWhaS4FVU4kwAh/r4Fz/He/iANA0=; fh=zdOMAZnBW6EIESt0EX/+avsk4/H625ZPwCc9+pZbT5k=; b=ISHX7c+3fQNBuuMYV6zFIyPMOoHbL4O+PQNm4+2m7kQAsIN/nmMJXqufC+BnQWvGLo MFNgUl8h4P9Bb/tHFD6n3yiDQGLnk+A6MuvpOVMlKnLLEgCFP5flBIZL9wWsf6zr31k6 6KObdeJu1tB7tQYvNmwbafFHqrplwcwrNNwjEpcN0z5hqHG4lOr2fUEnpwkuXXOoZS3w vbRziZnaL+viXkCAeImTcNBs8Xb1oNOuO/1K2NNWgQkZMs2tEseZC4TMr3roj6yeeMDO 9oyGn+yqip9aOacre72MxmcefAzerkCSUBWeexCRrkxG8625uoIMYOHC6OGfxL5xlM6r U7dA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=h3c.com); spf=pass (google.com: domain of linux-kernel+bounces-118479-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-118479-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id gb42-20020a170907962a00b00a474e73906bsi2377747ejc.796.2024.03.26.00.41.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 00:41:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-118479-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=h3c.com); spf=pass (google.com: domain of linux-kernel+bounces-118479-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-118479-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 44F6A1F38291 for ; Tue, 26 Mar 2024 07:41:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 56B2B132C15; Tue, 26 Mar 2024 07:41:28 +0000 (UTC) Received: from h3cspam02-ex.h3c.com (smtp.h3c.com [60.191.123.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E45DE839E2; Tue, 26 Mar 2024 07:41:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=60.191.123.50 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711438887; cv=none; b=hQ7egACHV/hgo3rz+AZosqXdruMTdqbTy+Qksm2zWqzLArIa6mqQ2oUIvCOCeeV6xt5No0J58rqy2kuaGTl61/DhVrCzVp7EjDrG1JtycCD7FvjS7r5a28Gt3Io6PSKq3nlp2dPdY2paZyPydPw0NNaQr/lCUDaZQYjQRfPEYEU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711438887; c=relaxed/simple; bh=rdKXw9XGhrMvPHlFoEDeAwMNHHTBwz8aECPqVByQ+Oo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oNUStFB3XXZwx+SqL1VbVA0d3RxZ/8pDg4TUpj2XcLDjG19MZIJ9afw+0aPbEsJ4Mos+W9z7f6zYeAveWy4/ctVk8sjntGiNNFPNAh+PNv9WMeIcczJG1BZMVVmZ8smcg+Q4j1RCpvZ6uVxJo40iVscIVHSonw0PQFYKUPRC18I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=h3c.com; spf=pass smtp.mailfrom=h3c.com; arc=none smtp.client-ip=60.191.123.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=h3c.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=h3c.com Received: from mail.maildlp.com ([172.25.15.154]) by h3cspam02-ex.h3c.com with ESMTP id 42Q7eJKv022798; Tue, 26 Mar 2024 15:40:19 +0800 (GMT-8) (envelope-from liu.yeC@h3c.com) Received: from DAG6EX02-IMDC.srv.huawei-3com.com (unknown [10.62.14.11]) by mail.maildlp.com (Postfix) with ESMTP id 1F6A72004BB3; Tue, 26 Mar 2024 15:42:09 +0800 (CST) Received: from localhost.localdomain (10.114.186.34) by DAG6EX02-IMDC.srv.huawei-3com.com (10.62.14.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1258.27; Tue, 26 Mar 2024 15:40:21 +0800 From: To: , CC: , , , , , , , LiuYe Subject: [PATCH V6] kdb: Fix the deadlock issue in KDB debugging. Date: Tue, 26 Mar 2024 15:40:14 +0800 Message-ID: <20240326074014.1905023-1-liu.yec@h3c.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240325165436.GA485978@aspen.lan> References: <20240325165436.GA485978@aspen.lan> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BJSMTP01-EX.srv.huawei-3com.com (10.63.20.132) To DAG6EX02-IMDC.srv.huawei-3com.com (10.62.14.11) X-DNSRBL: X-SPAM-SOURCE-CHECK: pass X-MAIL:h3cspam02-ex.h3c.com 42Q7eJKv022798 From: LiuYe Currently, if CONFIG_KDB_KEYBOARD is enabled, then kgdboc will attempt to use schedule_work() to provoke a keyboard reset when transitioning out of the debugger and back to normal operation. This can cause deadlock because schedule_work() is not NMI-safe. The stack trace below shows an example of the problem. In this case the master cpu is not running from NMI but it has parked the slave CPUs using an NMI and the parked CPUs is holding spinlocks needed by schedule_work(). example: BUG: spinlock lockup suspected on CPU#0, namex/10450 lock: 0xffff881ffe823980, .magic: dead4ead, .owner: namexx/21888, .owner_cpu: 1 ffff881741d00000 ffff881741c01000 0000000000000000 0000000000000000 ffff881740f58e78 ffff881741cffdd0 ffffffff8147a7fc ffff881740f58f20 Call Trace: [] ? __schedule+0x16d/0xac0 [] ? schedule+0x3c/0x90 [] ? schedule_hrtimeout_range_clock+0x10a/0x120 [] ? mutex_unlock+0xe/0x10 [] ? ep_scan_ready_list+0x1db/0x1e0 [] ? schedule_hrtimeout_range+0x13/0x20 [] ? ep_poll+0x27a/0x3b0 [] ? wake_up_q+0x70/0x70 [] ? SyS_epoll_wait+0xb8/0xd0 [] ? entry_SYSCALL_64_fastpath+0x12/0x75 CPU: 0 PID: 10450 Comm: namex Tainted: G O 4.4.65 #1 Hardware name: Insyde Purley/Type2 - Board Product Name1, BIOS 05.21.51.0036 07/19/2019 0000000000000000 ffff881ffe813c10 ffffffff8124e883 ffff881741c01000 ffff881ffe823980 ffff881ffe813c38 ffffffff810a7f7f ffff881ffe823980 000000007d2b7cd0 0000000000000001 ffff881ffe813c68 ffffffff810a80e0 Call Trace: <#DB> [] dump_stack+0x85/0xc2 [] spin_dump+0x7f/0x100 [] do_raw_spin_lock+0xa0/0x150 [] _raw_spin_lock+0x15/0x20 [] try_to_wake_up+0x176/0x3d0 [] wake_up_process+0x15/0x20 [] insert_work+0x81/0xc0 [] __queue_work+0x135/0x390 [] queue_work_on+0x46/0x90 [] kgdboc_post_exp_handler+0x48/0x70 [] kgdb_cpu_enter+0x598/0x610 [] kgdb_handle_exception+0xf2/0x1f0 [] __kgdb_notify+0x71/0xd0 [] kgdb_notify+0x35/0x70 [] notifier_call_chain+0x4a/0x70 [] notify_die+0x3d/0x50 [] do_int3+0x89/0x120 [] int3+0x44/0x80 We fix the problem by using irq_work to call schedule_work() instead of calling it directly. irq_work is an NMI-safe deferred work framework that performs the requested work from a hardirq context (usually an IPI but it can be timer interrupt on some architectures). Note that we still need to a workqueue since we cannot resync the keyboard state from the hardirq context provided by irq_work. That must be done from task context for the calls into the input subystem. Hence we must defer the work twice. First to safely switch from the debug trap (NMI-like context) to hardirq and then, secondly, to get from hardirq to the system workqueue. Signed-off-by: LiuYe Co-authored-by: Daniel Thompson Signed-off-by: Daniel Thompson --- V5 -> V6: Replace with a more professional and accurate answer. V4 -> V5: Answer why schedule another work in the irq_work and not do the job directly. V3 -> V4: Add changelogs V2 -> V3: Add description information V1 -> V2: using irq_work to solve this properly. --- --- drivers/tty/serial/kgdboc.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/tty/serial/kgdboc.c b/drivers/tty/serial/kgdboc.c index 7ce7bb164..161b25ecc 100644 --- a/drivers/tty/serial/kgdboc.c +++ b/drivers/tty/serial/kgdboc.c @@ -22,6 +22,7 @@ #include #include #include +#include #define MAX_CONFIG_LEN 40 @@ -99,10 +100,17 @@ static void kgdboc_restore_input_helper(struct work_struct *dummy) static DECLARE_WORK(kgdboc_restore_input_work, kgdboc_restore_input_helper); +static void kgdboc_queue_restore_input_helper(struct irq_work *unused) +{ + schedule_work(&kgdboc_restore_input_work); +} + +static DEFINE_IRQ_WORK(kgdboc_restore_input_irq_work, kgdboc_queue_restore_input_helper); + static void kgdboc_restore_input(void) { if (likely(system_state == SYSTEM_RUNNING)) - schedule_work(&kgdboc_restore_input_work); + irq_work_queue(&kgdboc_restore_input_irq_work); } static int kgdboc_register_kbd(char **cptr) @@ -133,6 +141,7 @@ static void kgdboc_unregister_kbd(void) i--; } } + irq_work_sync(&kgdboc_restore_input_irq_work); flush_work(&kgdboc_restore_input_work); } #else /* ! CONFIG_KDB_KEYBOARD */ -- 2.25.1