Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 4 Jul 2023 13:39:46 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Cruz Zhao <CruzZhao@linux.alibaba.com>
CC:     <gregkh@linuxfoundation.org>, <jirislaby@kernel.org>,
        <mingo@redhat.com>, <peterz@infradead.org>,
        <juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
        <dietmar.eggemann@arm.com>, <rostedt@goodmis.org>,
        <bsegall@google.com>, <mgorman@suse.de>, <bristot@redhat.com>,
        <vschneid@redhat.com>, <paulmck@kernel.org>,
        <quic_neeraju@quicinc.com>, <joel@joelfernandes.org>,
        <josh@joshtriplett.org>, <boqun.feng@gmail.com>,
        <mathieu.desnoyers@efficios.com>, <jiangshanlai@gmail.com>,
        <qiang1.zhang@intel.com>, <jstultz@google.com>,
        <clingutla@codeaurora.org>, <nsaenzju@redhat.com>,
        <tglx@linutronix.de>, <frederic@kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v3] sched/core: introduce sched_core_idle_cpu()
Message-ID: <20230704053946.GA658436@ziqianlu-dell>
References: <1688011324-42406-1-git-send-email-CruzZhao@linux.alibaba.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <1688011324-42406-1-git-send-email-CruzZhao@linux.alibaba.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bXWtfkLSwPE0mTTy5r4BIhQ1rGYnR1M1UVmaU3AMBWLbNRAlXAHRI+fluyrY?=
 =?us-ascii?Q?5BTD1iy2E7pNJoSAg/oXW5Vwjp4JMeBgzzHpOd+yRDoa31rvFj/iUSPI4sSj?=
 =?us-ascii?Q?n1FJHGDYItJayLDHZKSJhzU7gYYC3SJMnlt2ZSHsQ7LzVcB1MjFCui2NXUbY?=
 =?us-ascii?Q?74p5+8XisppOl/z1EOHkDJTQJajNY5ZmUJ1MwXLBS+k/lBbgIcJQKwRmuwzA?=
 =?us-ascii?Q?rN7ERXIvJO7eQKO8whiHG4rsBVba6B7LaJ60hk9dR2EPQjO1Wdz7beFYn8r4?=
 =?us-ascii?Q?1TVB87daLO/3UJ4cEZgBdENsOGu/pJCGBDg1ARDo2VYbHl1nbd3EbWXu05mG?=
 =?us-ascii?Q?e/LsfDCGFKblRWJeSgRLt8sFulIU70xPUwyB8WlpU9RHBkTF7yytv6mOQknz?=
 =?us-ascii?Q?4nczlfw0fd5+fckDu2+quJ3mCYHucXXxk+33ZyQF5Qhk2Vh9oxraVXd3+h+V?=
 =?us-ascii?Q?wcfYc7X1uUNp9UYj7XRpNwOoov8PS2TWRHIq7VW/H3qK7fgHSDBg/uRYWKh4?=
 =?us-ascii?Q?oT08tvtk+4GG8PGBQ5csL8Mqk3ixSZRQw7uigNa+e4YVDiW/fK4cZxN2fBcH?=
 =?us-ascii?Q?ja5T5E5bJE8QJPqBuk0XJBbkiVq/60Wy8jW1hJvCQ7uYoLall1ESs8IF9SuT?=
 =?us-ascii?Q?BBzg1+u6Ms3Pkl1pIKDl14AVQL8pFQThqeKANsoJ2lQoM7eRel1zXhuHdu1X?=
 =?us-ascii?Q?TdlWC3yxctClho4+CkOZK1UcRK/kXpmbBSgQhpnvK1lTWmtkBBxJoFgWOmxO?=
 =?us-ascii?Q?rr1tIVyDN8uaY+B3gig6ml16/wdHCpXM+t38Cv9ugE3N9jtddkB/NjOg1doD?=
 =?us-ascii?Q?gH9ZVIvlY4yL1cBU2vMNxtVaulAS/aGz0n7iaMKLkusNbQw96wnR5peyCkp6?=
 =?us-ascii?Q?DPBU/UcU5QTzQ8J00kegI3T6riTXQxdQaGWhIwVmj/sDDIClyoYCmMux69ZG?=
 =?us-ascii?Q?WV8dsqUQbSDO4vd3m1i4gAmUhfotvsRwyXkcgt1+LueieSexg3vo+ajhqMxG?=
 =?us-ascii?Q?E/ALegwmixYYdg0/rTeJa3RMx++rbIqAxbjgubt+2qCqSSz0peBtsuifRiDZ?=
 =?us-ascii?Q?jyHMAHG3wZ+khNtJCLjXnbNv92E/0+PVehnHbynSTg4q6dolO1M7t7OsVU5F?=
 =?us-ascii?Q?tKpj3PZecmuwqtb7iwnVwrAx995YJClV8Ul0c5M8MEGjcafHmIt8IjVYizNP?=
 =?us-ascii?Q?cvrlusPo40gWsQQ/llLyVhh9cEPi1eOf2LtGvK1J5+L6TtVjZn4FEBMq6qxB?=
 =?us-ascii?Q?Yye+gg0H/h1YKi/igmO+/3ANCvQTcxN0UUggzZhGO6nWLlDQHpv5icrihXIf?=
 =?us-ascii?Q?tfyip5CF5u9YHOxgDN25VQWufc5CBjW15YkFFLP3xJinN5rPZRE3LNKKJVuu?=
 =?us-ascii?Q?n8CMiMD8TyCKFyu8g/uHojsHA5S35FphFrSZyg/N24IFaF+7I1sKitmB8fRs?=
 =?us-ascii?Q?ql74G1qgRvI4mRHJBu/4tDuEiQ16Pc5562eq2uPzcBoiPiqEEFaaVlNp9kfW?=
 =?us-ascii?Q?A194RNgdluL/v1IVS+Cf5J++MXMNvlGm1jTvwcp2mnPSNwI6ulTBXvYCY7Ch?=
 =?us-ascii?Q?LoDRD9ZR/d+51Zwb29VVhDk8qWPH46EqRM2l2cyj?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 42efb470-7d05-442d-74dc-08db7c511cab
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 05:40:02.1001
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 42GVEP+8LEvcySDfPyd9jAwyzsQjCowECa55wA9YNPH/AlwNKnnwjaVgFyT9OEC9HwJyD6sceQZGGGWGfNM87A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB8397
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jun 29, 2023 at 12:02:04PM +0800, Cruz Zhao wrote:
> As core scheduling introduced, a new state of idle is defined as
> force idle, running idle task but nr_running greater than zero.
> 
> If a cpu is in force idle state, idle_cpu() will return zero. This
> result makes sense in some scenarios, e.g., load balance,
> showacpu when dumping, and judge the RCU boost kthread is starving.
> 
> But this will cause error in other scenarios, e.g., tick_irq_exit():
> When force idle, rq->curr == rq->idle but rq->nr_running > 0, results
> that idle_cpu() returns 0. In function tick_irq_exit(), if idle_cpu()
> is 0, tick_nohz_irq_exit() will not be called, and ts->idle_active will
> not become 1, which became 0 in tick_nohz_irq_enter().
> ts->idle_sleeptime won't update in function update_ts_time_stats(), if
> ts->idle_active is 0, which should be 1. And this bug will result that
> ts->idle_sleeptime is less than the actual value, and finally will
> result that the idle time in /proc/stat is less than the actual value.
> 
> To solve this problem, we introduce sched_core_idle_cpu(), which
> returns 1 when force idle. We audit all users of idle_cpu(), and
> change idle_cpu() into sched_core_idle_cpu() in function
> tick_irq_exit().
> 
> v2-->v3: Only replace idle_cpu() with sched_core_idle_cpu() in
> function tick_irq_exit(). And modify the corresponding commit log.
> 
> Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
> Reviewed-by: Peter Zijlstra <peterz@infradead.org>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> Reviewed-by: Joel Fernandes <joel@joelfernandes.org>
> Link: https://lore.kernel.org/lkml/1687631295-126383-1-git-send-email-CruzZhao@linux.alibaba.com
> ---
>  include/linux/sched.h |  2 ++
>  kernel/sched/core.c   | 13 +++++++++++++
>  kernel/softirq.c      |  2 +-
>  3 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index b09a83bfad8b..73e61c0f10a7 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2430,9 +2430,11 @@ extern void sched_core_free(struct task_struct *tsk);
>  extern void sched_core_fork(struct task_struct *p);
>  extern int sched_core_share_pid(unsigned int cmd, pid_t pid, enum pid_type type,
>  				unsigned long uaddr);
> +extern int sched_core_idle_cpu(int cpu);
>  #else
>  static inline void sched_core_free(struct task_struct *tsk) { }
>  static inline void sched_core_fork(struct task_struct *p) { }
> +static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); }
>  #endif
>  
>  extern void sched_set_stop_task(int cpu, struct task_struct *stop);
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 71c1a0f232b4..c80088956987 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7286,6 +7286,19 @@ struct task_struct *idle_task(int cpu)
>  	return cpu_rq(cpu)->idle;
>  }
>  
> +#ifdef CONFIG_SCHED_CORE
> +int sched_core_idle_cpu(int cpu)
> +{
> +	struct rq *rq = cpu_rq(cpu);
> +
> +	if (sched_core_enabled(rq) && rq->curr == rq->idle)
> +		return 1;

If the intention is to consider forced idle cpus as idle, then should
the above condition written as:

	if (sched_core_enabled(rq) && rq->core->core_forceidle_count)
		return 1;
?

Or as long as a single cookied task is running, all normal idle cpus are
regarded forced idle here and 1 is returned while previously, idle_cpu()
is called for those cpus and if they have wakeup task pending, they are
not regarded as idle so looks like a behaviour change.

Thanks,
Aaron

> +
> +	return idle_cpu(cpu);
> +}
> +
> +#endif
> +
>  #ifdef CONFIG_SMP
>  /*
>   * This function computes an effective utilization for the given CPU, to be
> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index c8a6913c067d..98b98991ce45 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -630,7 +630,7 @@ static inline void tick_irq_exit(void)
>  	int cpu = smp_processor_id();
>  
>  	/* Make sure that timer wheel updates are propagated */
> -	if ((idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
> +	if ((sched_core_idle_cpu(cpu) && !need_resched()) || tick_nohz_full_cpu(cpu)) {
>  		if (!in_hardirq())
>  			tick_nohz_irq_exit();
>  	}
> -- 
> 2.27.0
>