Received: by 2002:a25:ad19:0:0:0:0:0 with SMTP id y25csp11156067ybi; Thu, 25 Jul 2019 10:59:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqzUox35dWT2tEMkQrSUJDjkdOfTOcJ0XGXqMREQ24bwyFDM6ayOI/QJ2FT3oL7l+0IpcZgi X-Received: by 2002:a17:902:8696:: with SMTP id g22mr89847068plo.249.1564077540265; Thu, 25 Jul 2019 10:59:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1564077540; cv=none; d=google.com; s=arc-20160816; b=WQcwSlT0mmQpDt1sS9wZFFsHiD8ROIXvM7OcoAbDiRrmKbXw557I25YDcr3mus+HSS kmg3wp/gsXdoo2q8uf/iVOc7EfygSa7UmkM5LY0VjpbTbtAPfrJjIeBQUUCdQ50NhwEt a7GUdk9fndYAFfINQza+EszLZb8EqL33xQnDxgurA0LwWNbHf8NpOfyQxaqgUxMAzdoI aqSJ7vHZBzK3vFnhJyMUI/Z50jiSn8W486RswE8BwMMJ/Twvnenk3/rxhAOZ+e3F/aRu pEnyPrUWtSqKiMza2j6pmwciyhOuPmamDp/04nFQgVb9TZleg4OhIPk6Zl8duI4WmUBe Ifgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition :content-transfer-encoding:mime-version:robot-unsubscribe:robot-id :git-commit-id:subject:to:references:in-reply-to:reply-to:cc :message-id:from:date:dkim-signature:dkim-filter; bh=hoq7Vf7QmgDgfxpj/d14CcgXrqJQtHOYpk3ad7VQZEk=; b=tOBbSbS5+JZMbh6NMnuF1JalPgOFCViFctE5ctBfe/WbI6gzhGKTevzT74RuMJvTJB 0nMXJD+7y/7Qp4xmwZYmqoCPg6/oq5ZwLVqC7mWf05K6w7wYNObt64+xgLCxQ/cZuf5H t8wxtnP0KxjHBuojDBChpI2hY0dJx0b8s7GjXD2yXGsSMSx2M6H6+9fCCZo3Y0Cx+X6N JMPmvxxe7TdBPL1/kHOSF8LqI8/5Ngu+FKD/fHcha9JP7wzxtDGETbwkuAqxIU43aKjt H8wApyyjgCC2By4Ql6BEqKtcK/3ecASFODNHvyn0xRsjVB8WHaA/PiX5tME1lSn2X6KV dRRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zytor.com header.s=2019071901 header.b="P/W6FXPE"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=zytor.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x32si15807215pld.252.2019.07.25.10.58.45; Thu, 25 Jul 2019 10:59:00 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@zytor.com header.s=2019071901 header.b="P/W6FXPE"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=zytor.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728143AbfGYOTu (ORCPT + 99 others); Thu, 25 Jul 2019 10:19:50 -0400 Received: from terminus.zytor.com ([198.137.202.136]:56979 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725944AbfGYOTu (ORCPT ); Thu, 25 Jul 2019 10:19:50 -0400 Received: from terminus.zytor.com (localhost [127.0.0.1]) by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id x6PEJf9J1036943 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 25 Jul 2019 07:19:41 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 terminus.zytor.com x6PEJf9J1036943 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2019071901; t=1564064382; bh=hoq7Vf7QmgDgfxpj/d14CcgXrqJQtHOYpk3ad7VQZEk=; h=Date:From:Cc:Reply-To:In-Reply-To:References:To:Subject:From; b=P/W6FXPE4ng12iHPMAfZcR4BRlFuy0bB4dW/HVCmvFc/SBVyhi3UWX4fKOyoX/xIe su23HKvZkwSAwx3AjARbiVwlgXlUDWWME2UWFpA7p1dSoRBKzWM2no61YCf4U8tpXD vHEA1V7ND8Q2+flSVF0kKgwSoA7Q76R71Jr1NkY788criRueys7lx3Z+44NbOHD+sb gOfGaen6tZshph/F35uKzA4ObZcozYgOc/fp7rZ8dAi2fdALu7qRRRsCv/9939kF4P AsgVzkAVRCDFv44KjbbpoU0nYqFRDa7B+2WqYK+TyyvzJIkKTgvHte1CZUFh2jVJLh K87a+MgO9zr3A== Received: (from tipbot@localhost) by terminus.zytor.com (8.15.2/8.15.2/Submit) id x6PEJe6S1036940; Thu, 25 Jul 2019 07:19:40 -0700 Date: Thu, 25 Jul 2019 07:19:40 -0700 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@zytor.com using -f From: tip-bot for Grzegorz Halat Message-ID: Cc: mingo@kernel.org, ghalat@redhat.com, linux-kernel@vger.kernel.org, hpa@zytor.com, dzickus@redhat.com, tglx@linutronix.de Reply-To: tglx@linutronix.de, hpa@zytor.com, dzickus@redhat.com, ghalat@redhat.com, linux-kernel@vger.kernel.org, mingo@kernel.org In-Reply-To: <20190628122813.15500-1-ghalat@redhat.com> References: <20190628122813.15500-1-ghalat@redhat.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:x86/apic] x86/reboot: Always use NMI fallback when shutdown via reboot vector IPI fails Git-Commit-ID: 747d5a1bf293dcb33af755a6d285d41b8c1ea010 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline X-Spam-Status: No, score=-0.3 required=5.0 tests=ALL_TRUSTED,BAYES_00, DATE_IN_FUTURE_96_Q,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on terminus.zytor.com Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 747d5a1bf293dcb33af755a6d285d41b8c1ea010 Gitweb: https://git.kernel.org/tip/747d5a1bf293dcb33af755a6d285d41b8c1ea010 Author: Grzegorz Halat AuthorDate: Fri, 28 Jun 2019 14:28:13 +0200 Committer: Thomas Gleixner CommitDate: Thu, 25 Jul 2019 16:09:24 +0200 x86/reboot: Always use NMI fallback when shutdown via reboot vector IPI fails A reboot request sends an IPI via the reboot vector and waits for all other CPUs to stop. If one or more CPUs are in critical regions with interrupts disabled then the IPI is not handled on those CPUs and the shutdown hangs if native_stop_other_cpus() is called with the wait argument set. Such a situation can happen when one CPU was stopped within a lock held section and another CPU is trying to acquire that lock with interrupts disabled. There are other scenarios which can cause such a lockup as well. In theory the shutdown should be attempted by an NMI IPI after the timeout period elapsed. Though the wait loop after sending the reboot vector IPI prevents this. It checks the wait request argument and the timeout. If wait is set, which is true for sys_reboot() then it won't fall through to the NMI shutdown method after the timeout period has finished. This was an oversight when the NMI shutdown mechanism was added to handle the 'reboot IPI is not working' situation. The mechanism was added to deal with stuck panic shutdowns, which do not have the wait request set, so the 'wait request' case was probably not considered. Remove the wait check from the post reboot vector IPI wait loop and enforce that the wait loop in the NMI fallback path is invoked even if NMI IPIs are disabled or the registration of the NMI handler fails. That second wait loop will then hang if not all CPUs shutdown and the wait argument is set. [ tglx: Avoid the hard to parse line break in the NMI fallback path, add comments and massage the changelog ] Fixes: 7d007d21e539 ("x86/reboot: Use NMI to assist in shutting down if IRQ fails") Signed-off-by: Grzegorz Halat Signed-off-by: Thomas Gleixner Cc: Don Zickus Link: https://lkml.kernel.org/r/20190628122813.15500-1-ghalat@redhat.com --- arch/x86/kernel/smp.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index 96421f97e75c..231fa230ebc7 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -179,6 +179,12 @@ asmlinkage __visible void smp_reboot_interrupt(void) irq_exit(); } +static int register_stop_handler(void) +{ + return register_nmi_handler(NMI_LOCAL, smp_stop_nmi_callback, + NMI_FLAG_FIRST, "smp_stop"); +} + static void native_stop_other_cpus(int wait) { unsigned long flags; @@ -212,39 +218,41 @@ static void native_stop_other_cpus(int wait) apic->send_IPI_allbutself(REBOOT_VECTOR); /* - * Don't wait longer than a second if the caller - * didn't ask us to wait. + * Don't wait longer than a second for IPI completion. The + * wait request is not checked here because that would + * prevent an NMI shutdown attempt in case that not all + * CPUs reach shutdown state. */ timeout = USEC_PER_SEC; - while (num_online_cpus() > 1 && (wait || timeout--)) + while (num_online_cpus() > 1 && timeout--) udelay(1); } - - /* if the REBOOT_VECTOR didn't work, try with the NMI */ - if ((num_online_cpus() > 1) && (!smp_no_nmi_ipi)) { - if (register_nmi_handler(NMI_LOCAL, smp_stop_nmi_callback, - NMI_FLAG_FIRST, "smp_stop")) - /* Note: we ignore failures here */ - /* Hope the REBOOT_IRQ is good enough */ - goto finish; - - /* sync above data before sending IRQ */ - wmb(); - pr_emerg("Shutting down cpus with NMI\n"); + /* if the REBOOT_VECTOR didn't work, try with the NMI */ + if (num_online_cpus() > 1) { + /* + * If NMI IPI is enabled, try to register the stop handler + * and send the IPI. In any case try to wait for the other + * CPUs to stop. + */ + if (!smp_no_nmi_ipi && !register_stop_handler()) { + /* Sync above data before sending IRQ */ + wmb(); - apic->send_IPI_allbutself(NMI_VECTOR); + pr_emerg("Shutting down cpus with NMI\n"); + apic->send_IPI_allbutself(NMI_VECTOR); + } /* - * Don't wait longer than a 10 ms if the caller - * didn't ask us to wait. + * Don't wait longer than 10 ms if the caller didn't + * reqeust it. If wait is true, the machine hangs here if + * one or more CPUs do not reach shutdown state. */ timeout = USEC_PER_MSEC * 10; while (num_online_cpus() > 1 && (wait || timeout--)) udelay(1); } -finish: local_irq_save(flags); disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info));