Received-SPF: pass (google.com: domain of linux-kernel+bounces-179768-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 15 May 2024 11:41:42 +0100
From: Qais Yousef <qyousef@layalina.io>
To: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>,
	Viresh Kumar <viresh.kumar@linaro.org>,
	Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Juri Lelli <juri.lelli@redhat.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Daniel Bristot de Oliveira <bristot@redhat.com>,
	Valentin Schneider <vschneid@redhat.com>,
	Christian Loehle <christian.loehle@arm.com>,
	linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] sched: Consolidate cpufreq updates
Message-ID: <20240515104142.5ixmhzaomva2a2u7@airbuntu>
References: <20240512190018.531820-1-qyousef@layalina.io>
 <9e845146-8a31-407c-a5ee-e2e32f1655e5@arm.com>
 <20240513220903.no2j6zl4tk7lr6um@airbuntu>
 <09717ad7-2a4b-486c-a4f5-e3f09a212add@arm.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <09717ad7-2a4b-486c-a4f5-e3f09a212add@arm.com>

On 05/15/24 12:00, Dietmar Eggemann wrote:
> On 14/05/2024 00:09, Qais Yousef wrote:
> > On 05/13/24 14:43, Dietmar Eggemann wrote:
> >> On 12/05/2024 21:00, Qais Yousef wrote:
> >>
> >> [...]
> >>
> >>> @@ -4682,7 +4659,7 @@ static void attach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *s
> >>>  
> >>>  	add_tg_cfs_propagate(cfs_rq, se->avg.load_sum);
> >>>  
> >>> -	cfs_rq_util_change(cfs_rq, 0);
> >>> +	cpufreq_update_util(rq_of(cfs_rq), 0);
> >>
> >> Isn't this slighlty different now?
> >>
> >> before:
> >>
> >>    if (&rq->cfs == cfs_rq) {
> >>        cpufreq_update_util(rq, ....)
> >>    }
> >>
> >> now:
> >>
> >>    cpufreq_update_util(rq_of(cfs_rq), ...)
> >>
> >> You should get way more updates from attach/detach now.
> > 
> > Yes, well spotted!
> > 
> > Looking at the path more closely, I can see this is called from
> > enqueue_task_fair() path when a task migrates to new CPU. And when
> > attach_task_cfs_rq() which is called when we switch_to_fair(), which I already
> > cover in the policy change for the RUNNING task, or when
> > task_change_group_fair() which what I originally understood Vincent was
> > referring to. I moved the update to this function after the detach/attach
> > operations with better guards to avoid unnecessary update.
> 
> Yeah, all !root cfs_rq attach or detach wouldn't change anything since
> the util_avg wouldn't have propagated to the root cfs_rq yet. So
> sugov_get_util() wouldn't see a difference.
> 
> Yes, enqueue_entity() sets DO_ATTACH unconditionally.
> 
> And dequeue_entity() sets DO_DETACH for a migrating (!wakeup migrating)
> task.
> 
> For a wakeup migrating task we have remove_entity_load_avg() but this
> can't remove util_avg from the cfs_rq. This is deferred to
> update_cfs_rq_load_avg() in update_load_avg() or __update_blocked_fair().
> 
> And switched_{to,from}_fair() (check_class_changed()) and
> task_change_group_fair() are the other 2 users of
> {attach,detach}_entity_load_avg(). (plus online_fair_sched_group() for
> attach).
> 
> > I understood this will lead to big change and better apply immediately vs
> > wait for the next context switch. But I'll ask the question again, can we drop
> > this and defer to context switch?
> 
> Hard to say really, probably we can. All benchmarks with score numbers
> will create plenty of context switches so you wont see a diff. And for
> more lighter testcases you would have to study the differences in trace
> files and reason about the implications of potentially kick CPUfreq a
> little bit later.

I lean to drop this and let the CPU state considered to be 'settled' on next
context switch.

But I'll wait to hear more opinions before I post a new version.

Thanks!