Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20221003144914.160547-1-kajetan.puchalski@arm.com>
 <CAJZ5v0hoe=8nY9vR=+Bjvexrg+E6fcO-S=W+PDkfD=Li6Uy__g@mail.gmail.com> <Y0fymW5LOoIHstE2@e126311.manchester.arm.com>
In-Reply-To: <Y0fymW5LOoIHstE2@e126311.manchester.arm.com>
From:   Doug Smythies <dsmythies@telus.net>
Date:   Thu, 13 Oct 2022 15:12:53 -0700
Message-ID: <CAAYoRsW+5xbW_Zd7Mtbo4VMi4RZFXRr7mf4NAU=Le7GhQzNJvg@mail.gmail.com>
Subject: Re: [RFC PATCH v2 0/1] cpuidle: teo: Introduce optional util-awareness
To:     Kajetan Puchalski <kajetan.puchalski@arm.com>
Cc:     "Rafael J. Wysocki" <rafael@kernel.org>, daniel.lezcano@linaro.org,
        lukasz.luba@arm.com, Dietmar.Eggemann@arm.com,
        yu.chen.surf@gmail.com, linux-pm@vger.kernel.org,
        linux-kernel@vger.kernel.org, Doug Smythies <dsmythies@telus.net>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi All,

On Thu, Oct 13, 2022 at 4:12 AM Kajetan Puchalski
<kajetan.puchalski@arm.com> wrote:
> On Wed, Oct 12, 2022 at 08:50:39PM +0200, Rafael J. Wysocki wrote:
> > On Mon, Oct 3, 2022 at 4:50 PM Kajetan Puchalski
> > <kajetan.puchalski@arm.com> wrote:
...

> On the Intel & power usage angle you might have seen in the discussion,
> Doug sent me some interesting data privately. As far as I can tell the
> main issue there is that C0 on Intel doesn't actually do power saving so
> moving the state selection down to it is a pretty bad idea because C1
> could be very close in terms of latency and save much more power.
>
> A potential solution could be altering the v2 to only decrease the state
> selection by 1 if it's above 1, ie 2->1 but not 1->0. It's fine for us
> because arm systems with 2 states use the early exit path anyway. It'd
> just amount to changing this hunk:
>
> +       if (cpu_data->utilized && idx > 0 && !dev->states_usage[idx-1].disable)
> +               idx--;
>
> to:
>
> +       if (cpu_data->utilized && idx > 1 && !dev->states_usage[idx-1].disable)
> +               idx--;
>
> What would you think about that? Should make it much less intense for
> Intel systems.

I tested the above, which you sent me as patch version v2-2.

By default, my Intel i5-10600K has 4 idle states:

$ grep . /sys/devices/system/cpu/cpu7/cpuidle/state*/name
/sys/devices/system/cpu/cpu7/cpuidle/state0/name:POLL
/sys/devices/system/cpu/cpu7/cpuidle/state1/name:C1_ACPI
/sys/devices/system/cpu/cpu7/cpuidle/state2/name:C2_ACPI
/sys/devices/system/cpu/cpu7/cpuidle/state3/name:C3_ACPI

Idle driver governor legend:
teo: the normal teo idle governor
menu: the normal menu idle governor
util or v1: the original patch
util-v2 or v2: V2 of the patch
util-v2-2 or v2-2: the suggestion further up in this thread.

Test 1: Timer based periodic:

A load sweep from 0 to 100%, then 100% to 0, first 73 hertz, then 113,
211,347 and finally 401 hertz work/sleep frequency. Single thread.

http://smythies.com/~doug/linux/idle/teo-util/consume/idle-1/

Summary, average processor package powers (watts):

teo              menu          v1               v2             v2-2
10.19399    10.74804    22.12791    21.0431    11.27865
                     5.44%      117.07%     106.43%     10.64%

There is no performance measurement for this test, it just has to
finish the work packet before the next period starts. Note that
overruns do occur as the workload approaches 100%, but I do not record
that data, as typically the lower workload percentages are the area of
interest.

Test 2: Ping-pong test rotating through 6 different cores, with a
variable packet of work to do at each stop. This test goes gradually
through different idle states and is not timer based. A different 2
core test (which I have not done) is used to better explore the idle
state 0 to idle state 1 transition. This test has a performance
measurement. The CPU scaling governor was set to performance. HWP was
disabled.

http://smythies.com/~doug/linux/idle/teo-util/ping-sweep/6-1/loop-times.png
http://smythies.com/~doug/linux/idle/teo-util/ping-sweep/6-1/loop-times-detail-a.png
http://smythies.com/~doug/linux/idle/teo-util/ping-sweep/6-1/

Summary:

Average processor package power (watts):
teo            v2-2            menu
27.3881    29.98293    28.04096
                 9.47%        2.38%

Execution time for the entire test (minutes):
teo   v2-2        menu
56    54.667    55.333
        -2.38%    -1.19%

However, notice that in the idle-state 0 and 1 region, V2-2 uses more
power and its loop time is longer (less is better), but also in the
deeper idle states regions V2-2 uses more power and also runs faster.

teo: 36.4 watts and 10.3533 usecs/loop.
menu: 36.8 watts and 10.1604 usecs/loop.
util-v2-2: 38.8 watts and 11.2358 usecs/loop.

and

teo: 15.2 watts and 1,777.6 usecs/loop.
menu: 15.6 watts and 1767.4 usecs/loop.
util-v2-2: 17.4 watts and 1618.7 usecs/loop.

... Doug