Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp4626564pxa; Mon, 10 Aug 2020 13:57:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzmQKHmdFMQl7GCXjYey0gzBlL25/AWhh1PasaA6K/4WFDxCw3LwtYDRI2zVI3bMnBTFtH1 X-Received: by 2002:a17:906:2b57:: with SMTP id b23mr22791305ejg.26.1597093069965; Mon, 10 Aug 2020 13:57:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597093069; cv=none; d=google.com; s=arc-20160816; b=TufmU1qMcQXewTC8Flr9Rw8dBI0fluwZTjM+JNsM7qPclpalN9+4SUoSVbTGrgy3r7 dK55oNRLNLaeuyINlnzEywUKUL2PxQgqWzmOIAOtA/xza7rffq3yJo8lB62NRqIwHSKh gUWlUYHjEjXDqjWdQnPn/GeTZ7aK5xHdlSDskwbFIDuomsVFGAbF++vf2Bs5ZxL8LwE7 S7EFDzpATKB2ycs79T1QpXh7q33vm1pqC+wjpp/k3gHRLZqWK96ZBROflU2n0hRh4sa8 xPHnI/E9UfDVF5nacfAJglBctKDShU8pjQLn21SmzUGaZWFQv5R+lX9hHF2sGhnQCgu8 9s4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:reply-to:message-id :subject:cc:to:from:date:dkim-signature; bh=o8+JAN+msDrhzUBrPCH+KwFatDeJoCqCBgHCeUCorE4=; b=LQiHSt0+CXS6Fe5ebIQd89V4sZCrVJmLVfPgFXHSyIDvahFFkJOVGHvjCFH5iV0NtH yJYXCCL5AlJLLBA4djsF9ncF6jtXHQqP6es5qPcFOejVcpYmxQTkNDo86kQUlyJ4ROl3 Kq52/tFtqfxNVv6BrhRlGb8eWYYfQ1vAUUMp1OtB9nUqHH40tACtFOcEMmQ6tLequUPB tKG0/OxgXEsCokhM45QS8+ub7W3j7E9/P2rFoY7we7YmC+p6FR3wjFaxDQz91+V9/i0p DMzFK6n4TMrgwAuGyrbeAIEeLghrJYX9JUIYyJkXVRRXA+iE2MoKNoKMNGaQ4d5fxxPz BO/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=tKm15NRi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id ly14si11297813ejb.369.2020.08.10.13.57.26; Mon, 10 Aug 2020 13:57:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=tKm15NRi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726690AbgHJU45 (ORCPT + 99 others); Mon, 10 Aug 2020 16:56:57 -0400 Received: from mail.kernel.org ([198.145.29.99]:49734 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726115AbgHJU44 (ORCPT ); Mon, 10 Aug 2020 16:56:56 -0400 Received: from paulmck-ThinkPad-P72.home (unknown [50.45.173.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9D80B20716; Mon, 10 Aug 2020 20:56:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597093015; bh=pum/l0HZs+kYzU3nToNYXCbY1WcpAwrT1CrspgU1jXY=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=tKm15NRiXzWMySEGdhssl0BukbdYKt/AJVy91KvXvmuA8mihfBAteoT3MGn2wh/EB kKbKVje9DyeNu18RWZ4GpQXjUw9HxXomk8N3xWGhQduo6/gKrQA5owDzuY4Ak4iIhL iXdKaxSPvGTSA/ffzlX3p8FORoJOnnKAqi9zLv68= Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 50DDD35228C7; Mon, 10 Aug 2020 13:56:55 -0700 (PDT) Date: Mon, 10 Aug 2020 13:56:55 -0700 From: "Paul E. McKenney" To: "Joel Fernandes (Google)" Cc: linux-kernel@vger.kernel.org, Davidlohr Bueso , Jonathan Corbet , Josh Triplett , Lai Jiangshan , linux-doc@vger.kernel.org, Mathieu Desnoyers , Mauro Carvalho Chehab , neeraju@codeaurora.org, peterz@infradead.org, Randy Dunlap , rcu@vger.kernel.org, Steven Rostedt , tglx@linutronix.de, vineethrp@gmail.com Subject: Re: [PATCH v4 3/5] rcu/tree: Make FQS complaining about offline CPU more aggressive Message-ID: <20200810205655.GA17709@paulmck-ThinkPad-P72> Reply-To: paulmck@kernel.org References: <20200807170722.2897328-1-joel@joelfernandes.org> <20200807170722.2897328-4-joel@joelfernandes.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200807170722.2897328-4-joel@joelfernandes.org> User-Agent: Mutt/1.9.4 (2018-02-28) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Aug 07, 2020 at 01:07:20PM -0400, Joel Fernandes (Google) wrote: > The FQS loop detecting that an offline CPU has not yet reported a > quiescent state, is a serious problem. The current interaction between > RCU quiescent-state reporting and CPU-hotplug operations means that the > FQS loop should never find that an offline CPU (by RCU's understanding > of what is offline or not) has not yet reported a quiescent state. > > First, the outgoing CPU explicitly reports a quiescent state if needed > in rcu_report_dead(). Second, the race where the CPU is leaving just as > RCU is initializing a new grace period is handled by an explicit check > for this condition in rcu_gp_init(). Third, the CPU's leaf rcu_node > structure's ->lock serializes these modifications and checks. > > At the moment, the FQS loop detections this anomaly but only reports it > after a second has elapsed. > > This commit therefore makes the warning more aggressive such that we fail > immediately when the FQS loop scan happens and bring the problem to > everyone's attention. > > Light testing with TREE03 and hotplug shows no warnings. Converted the > warning as well to WARN_ON_ONCE() to reduce log spam. > > Signed-off-by: Joel Fernandes (Google) Looks good, queued, thank you! I did a bit of editing as shown below. Please let me know if I messed anything up. Thanx, Paul ------------------------------------------------------------------------ commit 61b82b349d0089120a9705240ece6ecf2b176fd5 Author: Joel Fernandes (Google) Date: Fri Aug 7 13:07:20 2020 -0400 rcu: Make FQS more aggressive in complaining about offline CPUs The RCU grace-period kthread's force-quiescent state (FQS) loop should never see an offline CPU that has not yet reported a quiescent state. After all, the offline CPU should have reported a quiescent state during the CPU-offline process, or, failing that, by rcu_gp_init() if it ran concurrently with either the CPU going offline or the last task on a leaf rcu_node structure exiting its RCU read-side critical section while all CPUs corresponding to that structure are offline. The FQS loop should therefore complain if it does see an offline CPU that has not yet reported a quiescent state. And it does, but only once the grace period has been in force for a full second. This commit therefore makes this warning more aggressive, so that it will trigger as soon as the condition makes its appearance. Light testing with TREE03 and hotplug shows no warnings. This commit also converts the warning to WARN_ON_ONCE() in order to stave off possible log spam. Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 59e1943..f79827b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1215,13 +1215,28 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) return 1; } - /* If waiting too long on an offline CPU, complain. */ - if (!(rdp->grpmask & rcu_rnp_online_cpus(rnp)) && - time_after(jiffies, rcu_state.gp_start + HZ)) { + /* + * Complain if a CPU that is considered to be offline from RCU's + * perspective has not yet reported a quiescent state. After all, + * the offline CPU should have reported a quiescent state during + * the CPU-offline process, or, failing that, by rcu_gp_init() + * if it ran concurrently with either the CPU going offline or the + * last task on a leaf rcu_node structure exiting its RCU read-side + * critical section while all CPUs corresponding to that structure + * are offline. This added warning detects bugs in any of these + * code paths. + * + * The rcu_node structure's ->lock is held here, which excludes + * the relevant portions the CPU-hotplug code, the grace-period + * initialization code, and the rcu_read_unlock() code paths. + * + * For more detail, please refer to the "Hotplug CPU" section + * of RCU's Requirements documentation. + */ + if (WARN_ON_ONCE(!(rdp->grpmask & rcu_rnp_online_cpus(rnp)))) { bool onl; struct rcu_node *rnp1; - WARN_ON(1); /* Offline CPUs are supposed to report QS! */ pr_info("%s: grp: %d-%d level: %d ->gp_seq %ld ->completedqs %ld\n", __func__, rnp->grplo, rnp->grphi, rnp->level, (long)rnp->gp_seq, (long)rnp->completedqs);