Received: by 2002:ac0:a5b6:0:0:0:0:0 with SMTP id m51-v6csp722694imm; Fri, 8 Jun 2018 04:19:53 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKhGqWbuQCH245qfjHtbe5VbE8Dxa4x/ErctmZyMBxIiMyIJcKdru+XwMfffPid/dIZ4oo+ X-Received: by 2002:aa7:8148:: with SMTP id d8-v6mr23458pfn.78.1528456793848; Fri, 08 Jun 2018 04:19:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528456793; cv=none; d=google.com; s=arc-20160816; b=S2ipGTl/jgBXvM2q2WXHB94xTvUWyRWemwg/FQ4tPIHRBXUQX8Z+3UHOFxV8jGBJxY aqy0sNljncnXesPqiYeq5EtjqzR0Tl8zYVEIMJmR/U4i1f781iuNjz7k/qHtdnXIkr8B BUJ6ewbYTzobsRZmAgaX7FuArcz74xcwrN6MxmWpolKTepxEb5BL7CQGZcX620R1lHVI +NbYJuN1umdKOSkr2oZliZ9lLQjS+mJ7cqAHWwYN0C9uQDVmKx/T8AraxPnkIb+TwQob o1OEYVTsHTaths1+46/t5CU5bChuS7kWwv7btLXZthbEH2EM4XfItGAykm4Ho4hWJjW2 fN/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:arc-authentication-results; bh=b/d1KWD68aRmPrSQwEfJBLe/WNq8BeW2oezpkTrTpfs=; b=dcrxKSmJO6Qxy1Cn7DyYfyfSfV+Jj1zQdSwgveaHMea1JiTgT/fjd2pjv6dC+HVOU9 oSLLkjCU8P/v8Ji5cHEqH0X4Me+QA/KTHcQSRL8tcKDduUWSE1zjHgVV9cWuwQWXyxWA bETWrW/QAU9FRnbl6/W8SdYc1T6L8KOYch4LaWb0LmCOPaSoe3ebf5e+CXhOkwxEeJXf RKCaTHaNv+U/Pdcpy1aOZbzNiwrhxJEVdHTSPm7ZEpzgreR8tTelxajk4Plf+q3t6HfJ QHHKeFPLeIAftU/0xjdZd3oT2GJycJlUvmCcPrcNMxzz4yM96/0i2UoOZMnFP2pOmVoZ FYdA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g7-v6si9079486pgu.152.2018.06.08.04.19.39; Fri, 08 Jun 2018 04:19:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751483AbeFHLTQ (ORCPT + 99 others); Fri, 8 Jun 2018 07:19:16 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:32794 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750993AbeFHLTP (ORCPT ); Fri, 8 Jun 2018 07:19:15 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 06E2E1529; Fri, 8 Jun 2018 04:19:15 -0700 (PDT) Received: from e108498-lin.cambridge.arm.com (e108498-lin.cambridge.arm.com [10.1.211.46]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 12AB93F557; Fri, 8 Jun 2018 04:19:10 -0700 (PDT) Date: Fri, 8 Jun 2018 12:19:09 +0100 From: Quentin Perret To: Juri Lelli Cc: peterz@infradead.org, rjw@rjwysocki.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, mingo@redhat.com, dietmar.eggemann@arm.com, morten.rasmussen@arm.com, chris.redpath@arm.com, patrick.bellasi@arm.com, valentin.schneider@arm.com, vincent.guittot@linaro.org, thara.gopinath@linaro.org, viresh.kumar@linaro.org, tkjos@google.com, joelaf@google.com, smuckle@google.com, adharmap@quicinc.com, skannan@quicinc.com, pkondeti@codeaurora.org, edubezval@gmail.com, srinivas.pandruvada@linux.intel.com, currojerez@riseup.net, javi.merino@kernel.org Subject: Re: [RFC PATCH v3 09/10] sched/fair: Select an energy-efficient CPU on task wake-up Message-ID: <20180608111909.GC418@e108498-lin.cambridge.arm.com> References: <20180521142505.6522-1-quentin.perret@arm.com> <20180521142505.6522-10-quentin.perret@arm.com> <20180608102446.GE658@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180608102446.GE658@localhost.localdomain> User-Agent: Mutt/1.8.3 (2017-05-23) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Friday 08 Jun 2018 at 12:24:46 (+0200), Juri Lelli wrote: > Hi, > > On 21/05/18 15:25, Quentin Perret wrote: > > [...] > > > +static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) > > +{ > > + unsigned long cur_energy, prev_energy, best_energy, cpu_cap, task_util; > > + int cpu, best_energy_cpu = prev_cpu; > > + struct sched_energy_fd *sfd; > > + struct sched_domain *sd; > > + > > + sync_entity_load_avg(&p->se); > > + > > + task_util = task_util_est(p); > > + if (!task_util) > > + return prev_cpu; > > + > > + /* > > + * Energy-aware wake-up happens on the lowest sched_domain starting > > + * from sd_ea spanning over this_cpu and prev_cpu. > > + */ > > + sd = rcu_dereference(*this_cpu_ptr(&sd_ea)); > > + while (sd && !cpumask_test_cpu(prev_cpu, sched_domain_span(sd))) > > + sd = sd->parent; > > + if (!sd) > > + return -1; > > Shouldn't this be return prev_cpu? Well, you shouldn't be entering this function without an sd_ea pointer, so this case is a sort of bug I think. By returning -1 I think we should end-up picking a CPU using select_fallback_rq(), which sort of makes sense ? > > > + > > + if (cpumask_test_cpu(prev_cpu, &p->cpus_allowed)) > > + prev_energy = best_energy = compute_energy(p, prev_cpu); > > + else > > + prev_energy = best_energy = ULONG_MAX; > > + > > + for_each_freq_domain(sfd) { > > + unsigned long spare_cap, max_spare_cap = 0; > > + int max_spare_cap_cpu = -1; > > + unsigned long util; > > + > > + /* Find the CPU with the max spare cap in the freq. dom. */ > > I undestand this being a heuristic to cut some overhead, but shouldn't > the model tell between packing vs. spreading? Ah, that's a very interesting one :-) ! So, with only active costs of the CPUs in the model, we can't really tell what's best between packing or spreading between identical CPUs if the migration of the task doesn't change the OPP request. In a frequency domain, all the "best" CPU candidates for a task are those for which we'll request a low OPP. When there are several CPUs for which the OPP request will be the same, we just don't know which one to pick from an energy standpoint, because we don't have other energy costs (for idle states for ex) to break the tie. With this EM, the interesting thing is that if you assume that OPP requests follow utilization, you are _guaranteed_ that the CPU with the max spare capacity in a freq domain will always be among the best candidates of this freq domain. And since we don't know how to differentiate those candidates, why not using this one ? Yes, it _might_ be better from an energy standpoint to pack small tasks on a CPU in order to let other CPUs go in deeper idle states. But that also hurts your chances to go cluster idle. Which solution is the best ? It depends, and we have no ways to tell with this EM. This approach basically favors cluster-packing, and spreading inside a cluster. That should at least be a good thing for latency, and this is consistent with the idea that most of the energy savings come from the asymmetry of the system, and not so much from breaking the tie between identical CPUs. That's also the reason why EAS is enabled only if your system has SD_ASYM_CPUCAPACITY set, as we already discussed for patch 05/10 :-). Does that make sense ? Thanks, Quentin