Received: by 2002:ab2:1149:0:b0:1f3:1f8c:d0c6 with SMTP id z9csp2513892lqz; Tue, 2 Apr 2024 23:12:48 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCWl0CHew61iFb1Ck1T1vqKZ8IpJvBPKrUFZYNnydpE4JzwW4+RxdQE/MvZpxDH89x13ISgFJaM4Le7+ZGyceSeojDlUCmzFw2LcOWs8gg== X-Google-Smtp-Source: AGHT+IF13HR/pYw90mkeY7hRQ4WQ2dzBihZ6nf3r4MYP/a+qEwH0mpjvbVrDvv1V8LiAKYQhitW7 X-Received: by 2002:a05:6402:40c2:b0:567:fb9b:37d5 with SMTP id z2-20020a05640240c200b00567fb9b37d5mr9114918edb.32.1712124768410; Tue, 02 Apr 2024 23:12:48 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1712124768; cv=pass; d=google.com; s=arc-20160816; b=gJABUFE/FD5mM3tzKmeZVK8+rz4C8tiG0P/u4bgVdazqB5DbwNSLxFn1pEotlIF2VK 5Y0eKk3PfDQBMP7RzRAUf+3zTlktoPMFG3DXk72biRZ6Mew5lY/RB7fmKkCrvGVTDKI/ ND4OFyLA/w/ZTUExblYV+9fbr9XvRMWPUkez2knmscHpmPttFHx7J/AeafCDxMUThb+7 YpSNjZp6Z2HWQN1NmQOgL0/LFXO6jkQ9C4DfLdnt4MRfpRySQLSrnp/cV+vUqtX6yvwT AGykJ16g6EBMI1iDMhYQ+F6u/9Gkf3diqRcqe6VnglFClk9xsTafmn5aBrjSpuSfdUZf K1hQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=oweCgn/pW/RDZfAQ88EY8fgmq1qUw5mM+wana337D0c=; fh=qHCRA4TnzMBEE6omRnMmvafBBXnpRKK6O+as6iMLK88=; b=lHl3XSB5bQuHCmmaFhHWvU5yw0u1/1ESBX87DLvqOHLZsT8ke3jt0un/Qb1ldY3I2L +JD+pQwBS9txvhYmKR1U4ttIyLM7s8kaiwfGnt0GOJaeyMj+kUn1UVzbkS2Y26XM/DTH Dh4cQc3hjQJTINXx7Phjbd3MuGqtQqX4JlJrmOzYsIDaL8HpekYhniTNFbEO0MSU2YnT HWGrVw3/8uStCggZXyGhj5PyK/Xe4Zwd6D2bugA/NfYPt1gEgLpTJNLuYwAZtv18Bl0I n2o+8LxPGwOq1UPjT2Mer9LMC9guFJGgfFWXKFXaAxpS0tl48i9H5Fg+I2uwbLOSZpWo XsSw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=h3c.com); spf=pass (google.com: domain of linux-kernel+bounces-129076-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-129076-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id x3-20020a50ba83000000b0056df50f2746si1021340ede.408.2024.04.02.23.12.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 23:12:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-129076-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=h3c.com); spf=pass (google.com: domain of linux-kernel+bounces-129076-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-129076-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id DEE331F22A4A for ; Wed, 3 Apr 2024 06:12:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0AD274E1D9; Wed, 3 Apr 2024 06:12:40 +0000 (UTC) Received: from h3cspam02-ex.h3c.com (smtp.h3c.com [60.191.123.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7044645; Wed, 3 Apr 2024 06:12:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=60.191.123.50 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712124759; cv=none; b=ubd3YycaAXAkPI6TDr6AYFBVPUXjsEnyBDIPNC+SYLUT9Y05HVnouv49xSxSEZ/VFLim3APuMPkKnGlmoiFDh5OqzW/NB6KThxp0o6UFhl9iGDOD1e80YwJD42JBcTLb0XSYz0kw4iFgxWXfDs+iyYb8SwHk+JE5gEZQoTz1sWY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712124759; c=relaxed/simple; bh=cDJ+56UgBDjll1l3QqIhy5OF77+XfU4Ieoj5qmZhn5o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OVGL+x8h/8vQG2fpasN5St/G0TXODGeJ7Pw/69ozQOMVdYigmFtUFn4ezJ6EiNaQ52LWgEFONorUHfB9pcYzgHI0lRjG6X/a6OBIwWq2Q1yci6FIgHT9T3RPJ1n1sHMczhHARL+EtIi9APbOdB8oCDTeqxrkHdrNfrFD0xgtQi8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=h3c.com; spf=pass smtp.mailfrom=h3c.com; arc=none smtp.client-ip=60.191.123.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=h3c.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=h3c.com Received: from mail.maildlp.com ([172.25.15.154]) by h3cspam02-ex.h3c.com with ESMTP id 4336BGO9024811; Wed, 3 Apr 2024 14:11:16 +0800 (GMT-8) (envelope-from liu.yeC@h3c.com) Received: from DAG6EX02-IMDC.srv.huawei-3com.com (unknown [10.62.14.11]) by mail.maildlp.com (Postfix) with ESMTP id 987942004BA1; Wed, 3 Apr 2024 14:13:16 +0800 (CST) Received: from localhost.localdomain (10.114.186.34) by DAG6EX02-IMDC.srv.huawei-3com.com (10.62.14.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1258.27; Wed, 3 Apr 2024 14:11:17 +0800 From: To: CC: , , , , , , , , LiuYe Subject: [PATCH V8] kdb: Fix the deadlock issue in KDB debugging. Date: Wed, 3 Apr 2024 14:11:09 +0800 Message-ID: <20240403061109.3142580-1-liu.yec@h3c.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240402125802.GC25200@aspen.lan> References: <20240402125802.GC25200@aspen.lan> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BJSMTP01-EX.srv.huawei-3com.com (10.63.20.132) To DAG6EX02-IMDC.srv.huawei-3com.com (10.62.14.11) X-DNSRBL: X-SPAM-SOURCE-CHECK: pass X-MAIL:h3cspam02-ex.h3c.com 4336BGO9024811 From: LiuYe Currently, if CONFIG_KDB_KEYBOARD is enabled, then kgdboc will attempt to use schedule_work() to provoke a keyboard reset when transitioning out of the debugger and back to normal operation. This can cause deadlock because schedule_work() is not NMI-safe. The stack trace below shows an example of the problem. In this case the master cpu is not running from NMI but it has parked the slave CPUs using an NMI and the parked CPUs is holding spinlocks needed by schedule_work(). example: BUG: spinlock lockup suspected on CPU#0, namex/10450 lock: 0xffff881ffe823980, .magic: dead4ead, .owner: namexx/21888, .owner_cpu: 1 ffff881741d00000 ffff881741c01000 0000000000000000 0000000000000000 ffff881740f58e78 ffff881741cffdd0 ffffffff8147a7fc ffff881740f58f20 Call Trace: [] ? __schedule+0x16d/0xac0 [] ? schedule+0x3c/0x90 [] ? schedule_hrtimeout_range_clock+0x10a/0x120 [] ? mutex_unlock+0xe/0x10 [] ? ep_scan_ready_list+0x1db/0x1e0 [] ? schedule_hrtimeout_range+0x13/0x20 [] ? ep_poll+0x27a/0x3b0 [] ? wake_up_q+0x70/0x70 [] ? SyS_epoll_wait+0xb8/0xd0 [] ? entry_SYSCALL_64_fastpath+0x12/0x75 CPU: 0 PID: 10450 Comm: namex Tainted: G O 4.4.65 #1 Hardware name: Insyde Purley/Type2 - Board Product Name1, BIOS 05.21.51.0036 07/19/2019 0000000000000000 ffff881ffe813c10 ffffffff8124e883 ffff881741c01000 ffff881ffe823980 ffff881ffe813c38 ffffffff810a7f7f ffff881ffe823980 000000007d2b7cd0 0000000000000001 ffff881ffe813c68 ffffffff810a80e0 Call Trace: <#DB> [] dump_stack+0x85/0xc2 [] spin_dump+0x7f/0x100 [] do_raw_spin_lock+0xa0/0x150 [] _raw_spin_lock+0x15/0x20 [] try_to_wake_up+0x176/0x3d0 [] wake_up_process+0x15/0x20 [] insert_work+0x81/0xc0 [] __queue_work+0x135/0x390 [] queue_work_on+0x46/0x90 [] kgdboc_post_exp_handler+0x48/0x70 [] kgdb_cpu_enter+0x598/0x610 [] kgdb_handle_exception+0xf2/0x1f0 [] __kgdb_notify+0x71/0xd0 [] kgdb_notify+0x35/0x70 [] notifier_call_chain+0x4a/0x70 [] notify_die+0x3d/0x50 [] do_int3+0x89/0x120 [] int3+0x44/0x80 We fix the problem by using irq_work to call schedule_work() instead of calling it directly. This is because we cannot resynchronize the keyboard state from the hardirq context provided by irq_work. This must be done from the task context in order to call the input subsystem. Therefore, we have to defer the work twice. First, safely switch from the debug trap context (similar to NMI) to the hardirq, and then switch from the hardirq to the system work queue. Signed-off-by: LiuYe Co-authored-by: Daniel Thompson Signed-off-by: Daniel Thompson --- V7 -> V8: Update the description information and comments in the code. : Submit this patch based on version linux-6.9-rc2. V6 -> V7: Add comments in the code. V5 -> V6: Replace with a more professional and accurate answer. V4 -> V5: Answer why schedule another work in the irq_work and not do the job directly. V3 -> V4: Add changelogs V2 -> V3: Add description information V1 -> V2: using irq_work to solve this properly. --- --- drivers/tty/serial/kgdboc.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/drivers/tty/serial/kgdboc.c b/drivers/tty/serial/kgdboc.c index 7ce7bb164..d6ce945f0 100644 --- a/drivers/tty/serial/kgdboc.c +++ b/drivers/tty/serial/kgdboc.c @@ -22,6 +22,7 @@ #include #include #include +#include #define MAX_CONFIG_LEN 40 @@ -82,6 +83,19 @@ static struct input_handler kgdboc_reset_handler = { static DEFINE_MUTEX(kgdboc_reset_mutex); +/* + * This code ensures that the keyboard state, which is changed during kdb + * execution, is resynchronized when we leave the debug trap. The resync + * logic calls into the input subsystem to force a reset. The calls into + * the input subsystem must be executed from normal task context. + * + * We need to trigger the resync from the debug trap, which executes in an + * NMI (or similar) context. To make it safe to call into the input + * subsystem we end up having use two deferred execution techniques. + * Firstly, we use irq_work, which is NMI-safe, to provoke a callback from + * hardirq context. Then, from the hardirq callback we use the system + * workqueue to provoke the callback that actually performs the resync. + */ static void kgdboc_restore_input_helper(struct work_struct *dummy) { /* @@ -99,10 +113,17 @@ static void kgdboc_restore_input_helper(struct work_struct *dummy) static DECLARE_WORK(kgdboc_restore_input_work, kgdboc_restore_input_helper); +static void kgdboc_queue_restore_input_helper(struct irq_work *unused) +{ + schedule_work(&kgdboc_restore_input_work); +} + +static DEFINE_IRQ_WORK(kgdboc_restore_input_irq_work, kgdboc_queue_restore_input_helper); + static void kgdboc_restore_input(void) { if (likely(system_state == SYSTEM_RUNNING)) - schedule_work(&kgdboc_restore_input_work); + irq_work_queue(&kgdboc_restore_input_irq_work); } static int kgdboc_register_kbd(char **cptr) @@ -133,6 +154,7 @@ static void kgdboc_unregister_kbd(void) i--; } } + irq_work_sync(&kgdboc_restore_input_irq_work); flush_work(&kgdboc_restore_input_work); } #else /* ! CONFIG_KDB_KEYBOARD */ -- 2.25.1