Received: by 2002:ac0:946b:0:0:0:0:0 with SMTP id j40csp73846imj; Wed, 13 Feb 2019 05:00:52 -0800 (PST) X-Google-Smtp-Source: AHgI3IY5MgLV815ia30iBkNjGnFZO/CDfJo39RTG05tWPDU4PYW5ziGU/bVt3vp3ddXu5NsudQx9 X-Received: by 2002:a17:902:925:: with SMTP id 34mr436845plm.14.1550062852814; Wed, 13 Feb 2019 05:00:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550062852; cv=none; d=google.com; s=arc-20160816; b=qZ8MumQ3smGgRddPPLzCqFWoMtjuYcSdVeU1RYz68FBVo6ZQpGdTJzEeu+WW1SlKrR sB0kwFNFoN+m/pr90W1/WqveFE5U8kzAGvPzlecq1ygbdoqh2+Ib/AQn4KT0ASMOQpoK GHNndk7avxwqnvFTTddX66cP408ttEJM19Tb+E4qXnuJn3adzfGbujQPD/o6Qq5O4RcY DJb8wqpPGKQlMQfoQgXsNWqs5MDg0EnsMEafNv69+XgvU98JDYsSuuwjKt0ryKhoGnpX NdCbvcWN7bEBsyYQaYMDTUj24+wmVJC7Zig7u3plUatCzYtfWNwpyCa0aDEiAXKaUH/q IMyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition :content-transfer-encoding:mime-version:robot-unsubscribe:robot-id :git-commit-id:subject:to:references:in-reply-to:reply-to:cc :message-id:from:date; bh=SO3JgGP3LvpeJbfjn17eszuhFrRQ+zzzxMGPdMDiiug=; b=conTOWzq19FbIGPvPK9z7zU0JzncnOMlbbQs2GxlgUEviFQyj7DBfUaNqiZMk3Sv9h H1/HrkoLk0Rb1Gadfh70SGqe8n8SirS9IVPrqA9JA/29QYjbdrbzqvRO8vVHZf3d5wNq 2kJgReRH+VjmzFDeY3VlpO5vxPiWl4sodWNqsoZLAVNVbb8rwfRlnuSMLGoopu9cyO6M iYLGVvLasXPy+Vh7TkVzGYTC4+ctIlbQWUinFHS0q9n0YTKvz4jTKUGelSRvFxf/6RlX p0lbdMySS8RmKb/ZcRW5OmUNTQeHw0GBzc8ii+3CyyJ8BpWxOP96NhEvEtaehB4bRQo7 x2Ng== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 27si8227208pgp.135.2019.02.13.05.00.35; Wed, 13 Feb 2019 05:00:52 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2403788AbfBMIa7 (ORCPT + 99 others); Wed, 13 Feb 2019 03:30:59 -0500 Received: from terminus.zytor.com ([198.137.202.136]:34321 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733281AbfBMIa7 (ORCPT ); Wed, 13 Feb 2019 03:30:59 -0500 Received: from terminus.zytor.com (localhost [127.0.0.1]) by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id x1D8Ulic189339 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 13 Feb 2019 00:30:47 -0800 Received: (from tipbot@localhost) by terminus.zytor.com (8.15.2/8.15.2/Submit) id x1D8UlRX189336; Wed, 13 Feb 2019 00:30:47 -0800 Date: Wed, 13 Feb 2019 00:30:47 -0800 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@zytor.com using -f From: "tip-bot for Zhang, Jun" Message-ID: Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, bo.he@intel.com, hpa@zytor.com, tglx@linutronix.de, jun.zhang@intel.com, jin.xiao@intel.com, paulmck@linux.ibm.com, jie.a.bai@intel.com Reply-To: paulmck@linux.ibm.com, jie.a.bai@intel.com, jin.xiao@intel.com, hpa@zytor.com, tglx@linutronix.de, jun.zhang@intel.com, linux-kernel@vger.kernel.org, mingo@kernel.org, bo.he@intel.com In-Reply-To: References: To: linux-tip-commits@vger.kernel.org Subject: [tip:core/rcu] rcu: Do RCU GP kthread self-wakeup from softirq and interrupt Git-Commit-ID: 1d1f898df6586c5ea9aeaf349f13089c6fa37903 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, T_DATE_IN_FUTURE_96_Q autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on terminus.zytor.com Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 1d1f898df6586c5ea9aeaf349f13089c6fa37903 Gitweb: https://git.kernel.org/tip/1d1f898df6586c5ea9aeaf349f13089c6fa37903 Author: Zhang, Jun AuthorDate: Tue, 18 Dec 2018 06:55:01 -0800 Committer: Paul E. McKenney CommitDate: Fri, 25 Jan 2019 15:29:59 -0800 rcu: Do RCU GP kthread self-wakeup from softirq and interrupt The rcu_gp_kthread_wake() function is invoked when it might be necessary to wake the RCU grace-period kthread. Because self-wakeups are normally a useless waste of CPU cycles, if rcu_gp_kthread_wake() is invoked from this kthread, it naturally refuses to do the wakeup. Unfortunately, natural though it might be, this heuristic fails when rcu_gp_kthread_wake() is invoked from an interrupt or softirq handler that interrupted the grace-period kthread just after the final check of the wait-event condition but just before the schedule() call. In this case, a wakeup is required, even though the call to rcu_gp_kthread_wake() is within the RCU grace-period kthread's context. Failing to provide this wakeup can result in grace periods failing to start, which in turn results in out-of-memory conditions. This race window is quite narrow, but it actually did happen during real testing. It would of course need to be fixed even if it was strictly theoretical in nature. This patch does not Cc stable because it does not apply cleanly to earlier kernel versions. Fixes: 48a7639ce80c ("rcu: Make callers awaken grace-period kthread") Reported-by: "He, Bo" Co-developed-by: "Zhang, Jun" Co-developed-by: "He, Bo" Co-developed-by: "xiao, jin" Co-developed-by: Bai, Jie A Signed-off: "Zhang, Jun" Signed-off: "He, Bo" Signed-off: "xiao, jin" Signed-off: Bai, Jie A Signed-off-by: "Zhang, Jun" [ paulmck: Switch from !in_softirq() to "!in_interrupt() && !in_serving_softirq() to avoid redundant wakeups and to also handle the interrupt-handler scenario as well as the softirq-handler scenario that actually occurred in testing. ] Signed-off-by: Paul E. McKenney Link: https://lkml.kernel.org/r/CD6925E8781EFD4D8E11882D20FC406D52A11F61@SHSMSX104.ccr.corp.intel.com --- kernel/rcu/tree.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9ceb93f848cd..21775eebb8f0 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1593,15 +1593,23 @@ static bool rcu_future_gp_cleanup(struct rcu_node *rnp) } /* - * Awaken the grace-period kthread. Don't do a self-awaken, and don't - * bother awakening when there is nothing for the grace-period kthread - * to do (as in several CPUs raced to awaken, and we lost), and finally - * don't try to awaken a kthread that has not yet been created. If - * all those checks are passed, track some debug information and awaken. + * Awaken the grace-period kthread. Don't do a self-awaken (unless in + * an interrupt or softirq handler), and don't bother awakening when there + * is nothing for the grace-period kthread to do (as in several CPUs raced + * to awaken, and we lost), and finally don't try to awaken a kthread that + * has not yet been created. If all those checks are passed, track some + * debug information and awaken. + * + * So why do the self-wakeup when in an interrupt or softirq handler + * in the grace-period kthread's context? Because the kthread might have + * been interrupted just as it was going to sleep, and just after the final + * pre-sleep check of the awaken condition. In this case, a wakeup really + * is required, and is therefore supplied. */ static void rcu_gp_kthread_wake(void) { - if (current == rcu_state.gp_kthread || + if ((current == rcu_state.gp_kthread && + !in_interrupt() && !in_serving_softirq()) || !READ_ONCE(rcu_state.gp_flags) || !rcu_state.gp_kthread) return;