Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 30 Jul 2019 18:43:10 +0200
From:   Frederic Weisbecker <frederic@kernel.org>
To:     "Paul E. McKenney" <paulmck@linux.ibm.com>
Cc:     fweisbec@gmail.com, tglx@linutronix.de, mingo@kernel.org,
        linux-kernel@vger.kernel.org, rcu@vger.kernel.org,
        peterz@infradead.org, paulmckrcu@gmail.com
Subject: Re: How to turn scheduler tick on for current nohz_full CPU?
Message-ID: <20190730164309.GA962@lenoir>
References: <20190724115331.GA29059@linux.ibm.com>
 <20190724132257.GA1029@lenoir>
 <20190724135219.GY14271@linux.ibm.com>
 <20190724143012.GB1029@lenoir>
 <20190725011243.GZ14271@linux.ibm.com>
 <20190729223238.GF14271@linux.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20190729223238.GF14271@linux.ibm.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Jul 29, 2019 at 03:32:38PM -0700, Paul E. McKenney wrote:
> On Wed, Jul 24, 2019 at 06:12:43PM -0700, Paul E. McKenney wrote:
> 
> The patch below (which includes your patch) does help considerably.
> However, it does have some shortcomings:
> 
> 1.	Adds an atomic operation (albeit a cache-local one) to
> 	the scheduler fastpath.  One approach would be to have
> 	a way of testing this bit and clearing it only if set.
> 
> 	Another approach would be to instead clear it on the
> 	transition to nohz_full userspace or to idle.

Well, the latter would be costly as it is going to restart the tick on every
user -> kernel transitions.

> 
> 2.	There are a lot of other places in the kernel that are in
> 	need of this bit being set.  I am therefore considering making
> 	multi_cpu_stop() or its callers set this bit on all CPUs upon
> 	entry and clear it upon exit.  While in this state, it is
> 	likely necessary to disable clearing this bit.  Or it would
> 	be necessary to make multi_cpu_stop() repeat clearing the bit
> 	every so often.
> 
> 	As it stands, I have CPU hotplug removal operations taking
> 	more than 400 seconds.
> 
> 3.	It was tempting to ask for this bit to be tracked on a per-task
> 	basis, but from what I can see that adds at least as much
> 	complexity as it removes.

Yeah I forgot to answer, you can use tick_dep_set_task() for that.

> 
> Thoughts?
> 
> 							Thanx, Paul
> 
> PS.  Outage on @linux.ibm.com, hence the CC of my gmail address.
> 
> ------------------------------------------------------------------------
> 
> diff --git a/include/linux/tick.h b/include/linux/tick.h
> index 196a0a7bfc4f..0dea6fb33a11 100644
> --- a/include/linux/tick.h
> +++ b/include/linux/tick.h
> @@ -108,7 +108,8 @@ enum tick_dep_bits {
>  	TICK_DEP_BIT_POSIX_TIMER	= 0,
>  	TICK_DEP_BIT_PERF_EVENTS	= 1,
>  	TICK_DEP_BIT_SCHED		= 2,
> -	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3
> +	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3,
> +	TICK_DEP_BIT_RCU		= 4
>  };
>  
>  #define TICK_DEP_MASK_NONE		0
> @@ -116,6 +117,7 @@ enum tick_dep_bits {
>  #define TICK_DEP_MASK_PERF_EVENTS	(1 << TICK_DEP_BIT_PERF_EVENTS)
>  #define TICK_DEP_MASK_SCHED		(1 << TICK_DEP_BIT_SCHED)
>  #define TICK_DEP_MASK_CLOCK_UNSTABLE	(1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
> +#define TICK_DEP_MASK_RCU		(1 << TICK_DEP_BIT_RCU)
>  
>  #ifdef CONFIG_NO_HZ_COMMON
>  extern bool tick_nohz_enabled;
> @@ -258,6 +260,9 @@ static inline bool tick_nohz_full_enabled(void) { return false; }
>  static inline bool tick_nohz_full_cpu(int cpu) { return false; }
>  static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
>  
> +static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
> +static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }

And I gave you the wrong APIs. Please consider_using tick_dep_set_cpu()
and tick_dep_clear_cpu() that first check if the CPU uses nohz_full.

Those should have the !CONFIG_NO_HZ_FULL stub implemented as well.

Thanks.