Received: by 2002:a25:c205:0:0:0:0:0 with SMTP id s5csp870419ybf; Thu, 27 Feb 2020 01:03:04 -0800 (PST) X-Google-Smtp-Source: APXvYqxdOKKxcY/8ckk15t1IcaKYbo/AnKfEVehXLuw9g0ab/IKY46eZo/gaP8LIE95fSBInra7z X-Received: by 2002:a9d:5e8b:: with SMTP id f11mr2543231otl.110.1582794184249; Thu, 27 Feb 2020 01:03:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1582794184; cv=none; d=google.com; s=arc-20160816; b=EPbrcjjORXsXGHGsW+/l3TEiLjkSY9Jp0AJSPGrSAi9wuGF3dd5xjwhqVR+ZC7eixr TgLIfY6EbyNPaHGUWqY+z7oP5+O0ja8fur7ig36zeiDS5H1SNhCISLtTf29Cr2LdB8eN 67ZaVDU5TxB8gzVUeUCMhnmWZ4eQbpn3NhAzQItDmlAf/8xElOiNRRN1/3QRHpTAv89E j8gPTLsvdKRvEEh0d5iTI/yVGejlEAudXhLeu2ZEuFoiLmUgxnn4HrG6/nnekcj/5lpP tIqqpItA5hdg0VrmdOdttzTeBW9p9DqpnZCcpxwZSa44zl0Hi2ShdfGY6QuxBaqmWeJD oaeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=9Ym3+LivfToOu0cDcirhGLkZLPrt03CVZkRGzcx6ji4=; b=M9ZsaWato+QFzqTYHxx88Q8QKSVnCSPlWv7VTH010MBGkTouLHXDGVKWxJmudelgZb e+QH185JIG7NENm1Q4cxG499hYqbJV2AFGkofeZY3yKbd+icg/YtbwqJCY7XbpT6byu6 r0Sa+ddFBYSdOkw2d1Ay65JwjZnIyD+D+m3CKeoWmtMgbjZB9k9fMftlcWgWjg8Pg/pN UmEJE1AkuUZYuKr4myegm06nPnVHm39SZaYjAaAokaY3QiQEop4HSjqraCWO2ocb/CjI Zn4GFswvpUc7MKko0cI7ui2dwjW2XXqcJIKSCgg1GNGi7jw//iECMZn2meNgv1dwGGcB Oszw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@verdurent-com.20150623.gappssmtp.com header.s=20150623 header.b=QSgP8qus; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h11si1220614otr.197.2020.02.27.01.02.52; Thu, 27 Feb 2020 01:03:04 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@verdurent-com.20150623.gappssmtp.com header.s=20150623 header.b=QSgP8qus; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728642AbgB0JCM (ORCPT + 99 others); Thu, 27 Feb 2020 04:02:12 -0500 Received: from mail-vs1-f66.google.com ([209.85.217.66]:43526 "EHLO mail-vs1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728620AbgB0JCM (ORCPT ); Thu, 27 Feb 2020 04:02:12 -0500 Received: by mail-vs1-f66.google.com with SMTP id 7so1324278vsr.10 for ; Thu, 27 Feb 2020 01:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=verdurent-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9Ym3+LivfToOu0cDcirhGLkZLPrt03CVZkRGzcx6ji4=; b=QSgP8qusfjMI3a22VeuMrNYxGrB7iZszxeGRLFN01+GT4/KVIiLhpwqCA4LWi9Acq5 4tBCYVC5uBGsx++PEUgYwcKI2tgaPZhNl5Vmf0cX6PPOxA9C58QG0Zf+lq4c+XBOzJv9 i9HLHMD8UtrLTh2brlhZZ63ooaH26MVH4Z+5mfsdpD4IyVizonZ5fAeVZdlf5CT0jCDp Q5MfGgZPi79G+e2QCYRJfLDEIsILhar1cAOgv9YTVszJskHz9tART1ZsEO+s5U3/50w4 xh3+8cz2Ehht1f3BcXQIrYXemESiBUQZpbeEYLTZky7kd0ClecC6yJd1YrBkFaTeHma/ Ccmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9Ym3+LivfToOu0cDcirhGLkZLPrt03CVZkRGzcx6ji4=; b=jJm05dnBa3k15cDjcgQf2+C9WBYFpefDLXvj5pbig04fKHG2F6z0GVM9jMmlhOIZh9 10HoJuQdcOp03oA+LWqPtOqaYXG1a+6DqAvZzqfn/rcKjAaNzWpqBIgtSoLd4cTw7zAa i7aIKRjUpCAzpoOdoq3RQhZ90SMMB/zVhkL2YYsgFlGfdcTpWAnh6znt1y+QUhiWfwBE wysuCrO4W3r8PRFok3HJSNsss3TfV2tz+YJc5G12zsdKDeE1YkBm/OtUEjG6aY54vZjT R87+x0Z4jfjS8FATDgHb4XGjaw2iXG9D6oiUzAlq1WoA3LlHtjuJP02Ds6L1txuuGmD3 DRbQ== X-Gm-Message-State: APjAAAXXDOTv4dcJu7fSOZlvJriKAu74ejij3qC7a7xKZu+8qA6ZH/+p 9S7EjXvIC5iW1/TlZeY/YT0aFaGJmlwUITTfQIIMJg== X-Received: by 2002:a67:de85:: with SMTP id r5mr2031791vsk.9.1582794129961; Thu, 27 Feb 2020 01:02:09 -0800 (PST) MIME-Version: 1.0 References: <20200222005213.3873-1-thara.gopinath@linaro.org> In-Reply-To: <20200222005213.3873-1-thara.gopinath@linaro.org> From: Amit Kucheria Date: Thu, 27 Feb 2020 14:31:58 +0530 Message-ID: Subject: Re: [Patch v10 0/9] Introduce Thermal Pressure To: Thara Gopinath Cc: Ingo Molnar , Peter Zijlstra , ionela.voinescu@arm.com, Vincent Guittot , Dietmar Eggemann , Zhang Rui , qperret@google.com, Daniel Lezcano , Viresh Kumar , Steven Rostedt , Will Deacon , Catalin Marinas , Sudeep Holla , Juri Lelli , Jonathan Corbet , LKML , Amit Daniel Kachhap , Javi Merino Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Feb 22, 2020 at 6:22 AM Thara Gopinath wrote: > > Thermal governors can respond to an overheat event of a cpu by > capping the cpu's maximum possible frequency. This in turn > means that the maximum available compute capacity of the > cpu is restricted. But today in the kernel, task scheduler is > not notified of capping of maximum frequency of a cpu. > In other words, scheduler is unaware of maximum capacity > restrictions placed on a cpu due to thermal activity. > This patch series attempts to address this issue. > The benefits identified are better task placement among available > cpus in event of overheating which in turn leads to better > performance numbers. > > The reduction in the maximum possible capacity of a cpu due to a > thermal event can be considered as thermal pressure. Instantaneous > thermal pressure is hard to record and can sometime be erroneous > as there can be mismatch between the actual capping of capacity > and scheduler recording it. Thus solution is to have a weighted > average per cpu value for thermal pressure over time. > The weight reflects the amount of time the cpu has spent at a > capped maximum frequency. Since thermal pressure is recorded as > an average, it must be decayed periodically. Exisiting algorithm > in the kernel scheduler pelt framework is re-used to calculate > the weighted average. This patch series also defines a sysctl > inerface to allow for a configurable decay period. > > Regarding testing, basic build, boot and sanity testing have been > performed on db845c platform with debian file system. > Further, dhrystone and hackbench tests have been > run with the thermal pressure algorithm. During testing, due to > constraints of step wise governor in dealing with big little systems, > trip point 0 temperature was made assymetric between cpus in little > cluster and big cluster; the idea being that > big core will heat up and cpu cooling device will throttle the > frequency of the big cores faster, there by limiting the maximum available > capacity and the scheduler will spread out tasks to little cores as well. > > Test Results > > Hackbench: 1 group , 30000 loops, 10 runs > Result SD > (Secs) (% of mean) > No Thermal Pressure 14.03 2.69% > Thermal Pressure PELT Algo. Decay : 32 ms 13.29 0.56% > Thermal Pressure PELT Algo. Decay : 64 ms 12.57 1.56% > Thermal Pressure PELT Algo. Decay : 128 ms 12.71 1.04% > Thermal Pressure PELT Algo. Decay : 256 ms 12.29 1.42% > Thermal Pressure PELT Algo. Decay : 512 ms 12.42 1.15% > > Dhrystone Run Time : 20 threads, 3000 MLOOPS > Result SD > (Secs) (% of mean) > No Thermal Pressure 9.452 4.49% > Thermal Pressure PELT Algo. Decay : 32 ms 8.793 5.30% > Thermal Pressure PELT Algo. Decay : 64 ms 8.981 5.29% > Thermal Pressure PELT Algo. Decay : 128 ms 8.647 6.62% > Thermal Pressure PELT Algo. Decay : 256 ms 8.774 6.45% > Thermal Pressure PELT Algo. Decay : 512 ms 8.603 5.41% > I've tested this series with a patch to artificially reducing the capacity of big cores on the QCOM sdm845 by reducing the temperature at which it starts throttling (thereby introducing thermal pressure earlier) and can see the tasks being migrated to the LITTLE cores. FWIW, Tested-by: Amit Kucheria > A Brief History > > The first version of this patch-series was posted with resuing > PELT algorithm to decay thermal pressure signal. The discussions > that followed were around whether intanteneous thermal pressure > solution is better and whether a stand-alone algortihm to accumulate > and decay thermal pressure is more appropriate than re-using the > PELT framework. > Tests on Hikey960 showed the stand-alone algorithm performing slightly > better than resuing PELT algorithm and V2 was posted with the stand > alone algorithm. Test results were shared as part of this series. > Discussions were around re-using PELT algorithm and running > further tests with more granular decay period. > > For some time after this development was impeded due to hardware > unavailability, some other unforseen and possibly unfortunate events. > For this version, h/w was switched from hikey960 to db845c. > Also Instantaneous thermal pressure was never tested as part of this > cycle as it is clear that weighted average is a better implementation. > The non-PELT algorithm never gave any conclusive results to prove that it > is better than reusing PELT algorithm, in this round of testing. > Also reusing PELT algorithm means thermal pressure tracks the > other utilization signals in the scheduler. > > v3->v4: > - "Patch 3/7:sched: Initialize per cpu thermal pressure structure" > is dropped as it is no longer needed following changes in other > other patches. > - rest of the change log mentioned in specific patches. > > v5->v6: > - "Added arch_ interface APIs to access and update thermal pressure. > Moved declaration of per cpu thermal_pressure valriable and > infrastructure to update the variable to topology files. > > v6->v7: > - Added CONFIG_HAVE_SCHED_THERMAL_PRESSURE to stub out > update_thermal_load_avg in unsupported architectures as per > review comments from Peter, Dietmar and Quentin. > - Renamed arch_scale_thermal_capacity to arch_cpu_thermal_pressure > as per review comments from Peter, Dietmar and Ionela. > - Changed the input argument in arch_set_thermal_pressure from > capped capacity to delta capacity(thermal pressure) as per > Ionela's review comments. Hence the calculation for delta > capacity(thermal pressure) is moved to cpufreq_cooling.c. > - Fixed a bunch of spelling typos. > > v7->v8: > - Fixed typo in defining update_thermal_load_avg which was > causing build errors (reported by kbuild test report) > > v8->v9: > - Defined thermal_load_avg to read rq->avg_thermal.load_avg and > avoid cacheline miss in unsupported cases as per Peter's > suggestion. > - Moved periodic triggering of thermal pressure averaging from CFS > tick function to generic scheduler core tick function. > - Moved rq_clock_thermal from fair.c to sched.h to enable using > the function from multiple files. > - Initialized the __shift to 0 in setup_sched_thermal_decay_shift > as per Quentin's suggestion > - Added an extra patch enabling CONFIG_HAVE_SCHED_THERMAL_PRESSURE > as per Dietmar's request. > > v9->v10: > - Renamed arch_cpu_thermal_pressure to arch_scale_thermal_pressure > as per review comments from Dietmar. > - Split "[Patch v9 3/8] arm,arm64,drivers:Add infrastructure to > store and update instantaneous thermal pressure" into 3 thus > separating out arch/arm and arch/arm64 specific code into > individual patches as suggested by Amit Kucheria. > - Added description for sched_thermal_decay_shift in > kernel-parameters.txt following Randy's review comments. > - Fixed typos in comments as per Amit Kucheria's review comments. > > Thara Gopinath (9): > sched/pelt: Add support to track thermal pressure > sched/topology: Add hook to read per cpu thermal pressure. > drivers/base/arch_topology: Add infrastructure to store and update > instantaneous thermal pressure > arm64/topology: Populate arch_cpu_thermal_pressure for arm64 platforms > arm/topology: Populate arch_cpu_thermal_pressure for arm platforms > sched/fair: Enable periodic update of average thermal pressure > sched/fair: update cpu_capacity to reflect thermal pressure > thermal/cpu-cooling: Update thermal pressure in case of a maximum > frequency capping > sched/fair: Enable tuning of decay period > > .../admin-guide/kernel-parameters.txt | 16 ++++++++++ > arch/arm/include/asm/topology.h | 3 ++ > arch/arm64/include/asm/topology.h | 3 ++ > drivers/base/arch_topology.c | 11 +++++++ > drivers/thermal/cpufreq_cooling.c | 19 ++++++++++-- > include/linux/arch_topology.h | 10 ++++++ > include/linux/sched/topology.h | 8 +++++ > include/trace/events/sched.h | 4 +++ > init/Kconfig | 4 +++ > kernel/sched/core.c | 3 ++ > kernel/sched/fair.c | 27 ++++++++++++++++ > kernel/sched/pelt.c | 31 +++++++++++++++++++ > kernel/sched/pelt.h | 31 +++++++++++++++++++ > kernel/sched/sched.h | 21 +++++++++++++ > 14 files changed, 189 insertions(+), 2 deletions(-) > > -- > 2.20.1 >