Received: by 2002:ac0:a5b6:0:0:0:0:0 with SMTP id m51-v6csp2516442imm; Thu, 7 Jun 2018 12:00:27 -0700 (PDT) X-Google-Smtp-Source: ADUXVKImZo1h9N8sIW3vW6ZKeWgEJrBgr5S8J42KcXfzr8PNIzkB8nXz7PvUdMDS/Sopa+i/QYNU X-Received: by 2002:a17:902:4b:: with SMTP id 69-v6mr3288663pla.178.1528398027019; Thu, 07 Jun 2018 12:00:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528398026; cv=none; d=google.com; s=arc-20160816; b=0uiWOtb7X1m587Op6tgTPblWU2yXwqYAmiQ2I3p3vrrtoQNxx4PIUovtEbmNHohAD6 aEBdgaQyCRZjOiCJQ05r1lDUBgdrwi27HI15bUiF6OTpStXz7G13ON6H1PD5ojCunMgk vY/qsYceUpf/9z6XM6Gu/VQQZ25Uh79cE4jMHy2ZMxZlRlJsdTReEN2DDg+fuWCjGGND zjDwUZx2sWojDa5xU8L5CvpsNvNHM4lKanQ0S1aHdO7MAzW0/rcyOkJoxNNqTv/igi3T tb6sRhC3uyJZ/k1TgEhGqnvC5jqYCneJwl2PqcXmG+pg2Y9wI1S3yOz2v/GV97fYwWJR UBZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:arc-authentication-results; bh=nqlO5w2tjEIZY7Ebl3CH/ChhTZ4vcqk9K6prvXRyZO4=; b=m2y8UuPNck56VuDIYA1d3F5ug3PbkW2MrwB6os+Kag6zxDDAG5s2AjVK+XhCrtR6qm SlyOGrtcyE1P4rse6g9tnN1wly6O9kPuNtwTYLjV9GYGdclOKen6LVEiFxdFvlleXiqN OxHv7NRAEBuy03xh9P1y1Y07ILX8vzmt3EKSjI4EI1dJp0muRZXB3SNo68WBYr6a0TS+ vYCExrdxB4KHVtZYYj9Q0BFO2hpiMEpWEAaMxo7YU0xYOEocPcOiWmszpvWlxuwdxRYM Mjlwq0gUIqJdAlZgdfhiAhWNjoXnCY53hE98vNFEH0OfM8c1IYczsIi4Q2GxgvQeiMZI Z7eA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h11-v6si14128086pls.399.2018.06.07.12.00.13; Thu, 07 Jun 2018 12:00:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932674AbeFGQ3S (ORCPT + 99 others); Thu, 7 Jun 2018 12:29:18 -0400 Received: from mail-wm0-f68.google.com ([74.125.82.68]:34757 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753750AbeFGQ3P (ORCPT ); Thu, 7 Jun 2018 12:29:15 -0400 Received: by mail-wm0-f68.google.com with SMTP id q4-v6so3851003wmq.1 for ; Thu, 07 Jun 2018 09:29:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=nqlO5w2tjEIZY7Ebl3CH/ChhTZ4vcqk9K6prvXRyZO4=; b=UkB+ro+YfpUKjtIx8EP+4NJ2Ag4xJacVBzNfzPh2DhP1Yo4rKDjTvImgJPIEYn4I7/ 3zr0blvGDnXa4Xf+XGUM6AYarxUl6kprYT3e+Hppi+BjDK5AqSuc11K7YY++fMWyoUZa se2a0T4LSGBKbHv9OU5C1LRXj2ZZZz5vmrdZLTy83apKeVixsgkyGoXDj5yJ5pHHDMWs WsoZ9zNcG0+fT8fNETZmo0K++uCnE5GUC8KEvwystgl1yvNw0Letuh2RdQpLusZAiHZd Gtlw96SRVjrhcTFoltWjO/THK1KTbQYxDy4Mt3/ckgke35WAkjvYPo9BIi0lfIYNxpRF 7dZQ== X-Gm-Message-State: APt69E1oEBVcUdnuZxJCPMmOkGFYUD4EVdfa9iFPt1bjCd/JuhhomiPC vK+Idso3z2myDcvlFGH0MfVbAQ== X-Received: by 2002:a1c:ce0e:: with SMTP id e14-v6mr1936421wmg.87.1528388954408; Thu, 07 Jun 2018 09:29:14 -0700 (PDT) Received: from localhost.localdomain ([151.15.207.242]) by smtp.gmail.com with ESMTPSA id o9-v6sm10944505wrn.73.2018.06.07.09.29.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Jun 2018 09:29:13 -0700 (PDT) Date: Thu, 7 Jun 2018 18:29:10 +0200 From: Juri Lelli To: Quentin Perret Cc: peterz@infradead.org, rjw@rjwysocki.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, mingo@redhat.com, dietmar.eggemann@arm.com, morten.rasmussen@arm.com, chris.redpath@arm.com, patrick.bellasi@arm.com, valentin.schneider@arm.com, vincent.guittot@linaro.org, thara.gopinath@linaro.org, viresh.kumar@linaro.org, tkjos@google.com, joelaf@google.com, smuckle@google.com, adharmap@quicinc.com, skannan@quicinc.com, pkondeti@codeaurora.org, edubezval@gmail.com, srinivas.pandruvada@linux.intel.com, currojerez@riseup.net, javi.merino@kernel.org Subject: Re: [RFC PATCH v3 05/10] sched/topology: Reference the Energy Model of CPUs when available Message-ID: <20180607162910.GE3311@localhost.localdomain> References: <20180521142505.6522-1-quentin.perret@arm.com> <20180521142505.6522-6-quentin.perret@arm.com> <20180607144422.GA17216@localhost.localdomain> <20180607160200.GB3597@e108498-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180607160200.GB3597@e108498-lin.cambridge.arm.com> User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/06/18 17:02, Quentin Perret wrote: > On Thursday 07 Jun 2018 at 16:44:22 (+0200), Juri Lelli wrote: > > Hi, > > > > On 21/05/18 15:25, Quentin Perret wrote: > > > In order to use EAS, the task scheduler has to know about the Energy > > > Model (EM) of the platform. This commit extends the scheduler topology > > > code to take references on the frequency domains objects of the EM > > > framework for all online CPUs. Hence, the availability of the EM for > > > those CPUs is guaranteed to the scheduler at runtime without further > > > checks in latency sensitive code paths (i.e. task wake-up). > > > > > > A (RCU-protected) private list of online frequency domains is maintained > > > by the scheduler to enable fast iterations. Furthermore, the availability > > > of an EM is notified to the rest of the scheduler with a static key, > > > which ensures a low impact on non-EAS systems. > > > > > > Energy Aware Scheduling can be started if and only if: > > > 1. all online CPUs are covered by the EM; > > > 2. the EM complexity is low enough to keep scheduling overheads low; > > > 3. the platform has an asymmetric CPU capacity topology (detected by > > > looking for the SD_ASYM_CPUCAPACITY flag in the sched_domain > > > hierarchy). > > > > Not sure about this. How about multi-freq domain same max capacity > > systems. I understand that most of the energy saving come from selecting > > the right (big/LITTLE) cluster, but EM should still be useful to drive > > OPP selection (that was one of the use-cases we discussed lately IIRC) > > and also to decide between packing or spreading, no? > > So, let's discuss the usage of the EM for frequency selection first, > and its usage for task placement after. > > For frequency selection, schedutil could definitely use the EM as > provided by the framework introduced in patch 03/10. We could definitely > use that to make policy decisions (jump faster to the so called "knee" > if there is one for ex). This is true for symmetric and asymmetric > system. And I consider that independent from this patch. This patch is > about providing the scheduler with an EM to biais _task placement_. > > So, about the task placement ... There are cases (at least theoretical > ones) where EAS _could_ help on symmetric systems, but I have never > been able to measure any real benefits in practice. Most of the time, > it's a good idea from an energy standpoint to just spread the tasks > and to keep the OPPs as low as possible on symmetric systems, which is > already what CFS does. Of course you can come-up with specific counter > examples, but the question is whether or not these (corner) cases are > that important. They might or might not, it's not so easy to tell. > > On asymmetric systems, it is pretty clear that there is a massive > potential for saving energy with a different task placement strategy. > So, since the big savings are there, our idea was basically to address > that first, while we minimize the risk of hurting others (server folks > for ex). I guess that enabling EAS for asymmetric systems can be seen as > an incremental step. We should be able to extend the scope of EAS to > symmetric systems later, if proven useful. > > Another thing is that, if you are using an asymmetric system (e.g. > big.LITTLE), it is a good indication that energy/battery life is probably > important for your use-case, and that you might be ready to "pay" the > cost of EAS to save energy. This isn't that obvious for symmetric > systems. Ok, I buy the step by step approach starting from the use case that seems to fit most. But I still feel that having something like 3. stated (or in the code) might stop people from trying to see if having an EM around might help other cases (freq, sym, etc.). Also, if no EM data is present should equally result in disabling the whole thing, so not much (at all?) overhead for who is simply not providing data, no? [...] > > > + list_for_each_entry_safe(sfd, tmp, &sched_energy_fd_list, next) { > > > + if (cpumask_intersects(freq_domain_span(sfd), > > > + cpu_online_mask)) { > > > + nr_opp += em_fd_nr_cap_states(sfd->fd); > > > + nr_fd++; > > > + continue; > > > + } > > > + > > > + /* Remove the unused frequency domains */ > > > + list_del_rcu(&sfd->next); > > > + call_rcu(&sfd->rcu, free_sched_energy_fd); > > > > Unused because of? Hotplug? > > Yes. The list of frequency domains is just convenient because we need to > iterate over them in the wake-up path. Now, if you hotplug out all the > CPUs of a frequency domain, it is safe to remove it from the list > because the scheduler shouldn't migrate task to/from those CPUs while > they're offline. And that's one less element in the list, so iterating > over the entire list is faster. OK, I mainly asked to be sure that I understood the comment. I guess some stress test involving hotplug and iterating over the list would best answer which way is the safest. :)