Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 15 Sep 2022 20:50:44 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     "Paul E. McKenney" <paulmck@kernel.org>
Cc:     Joel Fernandes <joel@joelfernandes.org>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        linux-kernel@vger.kernel.org, Boqun Feng <boqun.feng@gmail.com>,
        "Rafael J. Wysocki" <rjw@rjwysocki.net>
Subject: Re: RCU vs NOHZ
Message-ID: <YyN0BKEoDbe4hcIl@hirez.programming.kicks-ass.net>
References: <YyLksEr05QTNo05Q@hirez.programming.kicks-ass.net>
 <20220915160600.GA246308@paulmck-ThinkPad-P17-Gen-1>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220915160600.GA246308@paulmck-ThinkPad-P17-Gen-1>
Precedence: bulk

On Thu, Sep 15, 2022 at 09:06:00AM -0700, Paul E. McKenney wrote:
> On Thu, Sep 15, 2022 at 10:39:12AM +0200, Peter Zijlstra wrote:
> > Hi,
> > 
> > After watching Joel's talk about RCU and idle ticks I was wondering
> > about why RCU doesn't have NOHZ hooks -- that is regular NOHZ, not the
> > NOHZ_FULL stuff.
> 
> It actually does, but they have recently moved into the context-tracking
> code, courtesy of Frederic's recent patch series.

afair that's idle and that is not nohz.

> > These deep idle states are only feasible during NOHZ idle, and the NOHZ
> > path is already relatively expensive (which is offset by then mostly
> > staying idle for a long while).
> > 
> > Specifically my thinking was that when a CPU goes NOHZ it can splice
> > it's callback list onto a global list (cmpxchg), and then the
> > jiffy-updater CPU can look at and consume this global list (xchg).
> > 
> > Before you say... but globals suck (they do), NOHZ already has a fair
> > amount of global state, and as said before, it's offset by the CPU then
> > staying idle for a fair while. If there is heavy contention on the NOHZ
> > data, the idle governor is doing a bad job by selecting deep idle states
> > whilst we're not actually idle for long.
> > 
> > The above would remove the reason for RCU to inhibit NOHZ.
> > 
> > 
> > Additionally; when the very last CPU goes idle (I think we know this
> > somewhere, but I can't reaily remember where) we can insta-advance the
> > QS machinery and run the callbacks before going (NOHZ) idle.
> > 
> > 
> > Is there a reason this couldn't work? To me this seems like a much
> > simpler solution than the whole rcu-cb thing.
> 
> To restate Joel's reply a bit...
> 
> Maybe.
> 
> Except that we need rcu_nocbs anyway for low latency and HPC applications.
> Given that we have it, and given that it totally eliminates RCU-induced
> idle ticks, how would it help to add cmpxchg-based global offloading?

Because that nocb stuff isn't default enabled?