Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Message-ID: <2725211932028cf24ab34da9eb9d19190848cceb.camel@linux.vnet.ibm.com>
Subject: Re: [RFC v3 0/2] CPU-Idle latency selftest framework
From:   Aboorva Devarajan <aboorvad@linux.vnet.ibm.com>
Reply-To: 20230911053620.87973-1-aboorvad@linux.vnet.ibm.com
To:     mpe@ellerman.id.au, npiggin@gmail.com, rmclure@linux.ibm.com,
        arnd@arndb.de, joel@jms.id.au, shuah@kernel.org,
        linux-kselftest@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
        linux-kernel@vger.kernel.org, pratik.r.sampat@gmail.com,
        aboorvad@linux.vnet.ibm.com
Cc:     sshegde@linux.vnet.ibm.com, srikar@linux.vnet.ibm.com,
        rafael@kernel.org, daniel.lezcano@linaro.org,
        linux-pm@vger.kernel.org
Date:   Mon, 25 Sep 2023 10:36:19 +0530
In-Reply-To: <20230911053620.87973-1-aboorvad@linux.vnet.ibm.com>
References: <20230911053620.87973-1-aboorvad@linux.vnet.ibm.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Precedence: bulk

On Mon, 2023-09-11 at 11:06 +0530, Aboorva Devarajan wrote:

CC'ing CPUidle lists and maintainers,

Patch Summary:

The patchset introduces a kernel module and userspace driver designed
for estimating the wakeup latency experienced when waking up from
various CPU idle states. It primarily measures latencies related to two
types of events: Inter-Processor Interrupts (IPIs) and Timers.

Background:

Initially, these patches were introduced as a generic self-test.
However, it was later discovered that Intel platforms incorporate
timer-based wakeup optimizations. These optimizations allow CPUs to
perform a pre-wakeup, which limits the effectiveness of latency
observation in certain scenarios because it only measures the optimized
wakeup latency [1]. 

Therefore, in this RFC, the self-test is specifically integrated into
PowerPC, as it has been tested and used in PowerPC so far.

Another proposal is to introduce these patches as a generic cpuilde IPI
and timer wake-up test. While this method may not give us an exact
measurement of latency variations at the hardware level, it can still
help us assess this metric from a software observability standpoint.

Looking forward to hearing what you think and any suggestions you may
have regarding this. Thanks.

[1] 
https://lore.kernel.org/linux-pm/20200914174625.GB25628@in.ibm.com/T/#m5c004b9b1a918f669e91b3d0f33e2e3500923234

> Changelog: v2 -> v3
> 
> * Minimal code refactoring
> * Rebased on v6.6-rc1
> 
> RFC v1: 
> https://lore.kernel.org/all/20210611124154.56427-1-psampat@linux.ibm.com/
> 
> RFC v2:
> https://lore.kernel.org/all/20230828061530.126588-2-aboorvad@linux.vnet.ibm.com/
> 
> Other related RFC:
> https://lore.kernel.org/all/20210430082804.38018-1-psampat@linux.ibm.com/
> 
> Userspace selftest:
> https://lkml.org/lkml/2020/9/2/356
> 
> ----
> 
> A kernel module + userspace driver to estimate the wakeup latency
> caused by going into stop states. The motivation behind this program
> is
> to find significant deviations behind advertised latency and
> residency
> values.
> 
> The patchset measures latencies for two kinds of events. IPIs and
> Timers
> As this is a software-only mechanism, there will be additional
> latencies
> of the kernel-firmware-hardware interactions. To account for that,
> the
> program also measures a baseline latency on a 100 percent loaded CPU
> and the latencies achieved must be in view relative to that.
> 
> To achieve this, we introduce a kernel module and expose its control
> knobs through the debugfs interface that the selftests can engage
> with.
> 
> The kernel module provides the following interfaces within
> /sys/kernel/debug/powerpc/latency_test/ for,
> 
> IPI test:
>     ipi_cpu_dest = Destination CPU for the IPI
>     ipi_cpu_src = Origin of the IPI
>     ipi_latency_ns = Measured latency time in ns
> Timeout test:
>     timeout_cpu_src = CPU on which the timer to be queued
>     timeout_expected_ns = Timer duration
>     timeout_diff_ns = Difference of actual duration vs expected timer
> 
> Sample output is as follows:
> 
> # --IPI Latency Test---
> # Baseline Avg IPI latency(ns): 2720
> # Observed Avg IPI latency(ns) - State snooze: 2565
> # Observed Avg IPI latency(ns) - State stop0_lite: 3856
> # Observed Avg IPI latency(ns) - State stop0: 3670
> # Observed Avg IPI latency(ns) - State stop1: 3872
> # Observed Avg IPI latency(ns) - State stop2: 17421
> # Observed Avg IPI latency(ns) - State stop4: 1003922
> # Observed Avg IPI latency(ns) - State stop5: 1058870
> #
> # --Timeout Latency Test--
> # Baseline Avg timeout diff(ns): 1435
> # Observed Avg timeout diff(ns) - State snooze: 1709
> # Observed Avg timeout diff(ns) - State stop0_lite: 2028
> # Observed Avg timeout diff(ns) - State stop0: 1954
> # Observed Avg timeout diff(ns) - State stop1: 1895
> # Observed Avg timeout diff(ns) - State stop2: 14556
> # Observed Avg timeout diff(ns) - State stop4: 873988
> # Observed Avg timeout diff(ns) - State stop5: 959137
> 
> Aboorva Devarajan (2):
>   powerpc/cpuidle: cpuidle wakeup latency based on IPI and timer
> events
>   powerpc/selftest: Add support for cpuidle latency measurement
> 
>  arch/powerpc/Kconfig.debug                    |  10 +
>  arch/powerpc/kernel/Makefile                  |   1 +
>  arch/powerpc/kernel/test_cpuidle_latency.c    | 154 ++++++
>  tools/testing/selftests/powerpc/Makefile      |   1 +
>  .../powerpc/cpuidle_latency/.gitignore        |   2 +
>  .../powerpc/cpuidle_latency/Makefile          |   6 +
>  .../cpuidle_latency/cpuidle_latency.sh        | 443
> ++++++++++++++++++
>  .../powerpc/cpuidle_latency/settings          |   1 +
>  8 files changed, 618 insertions(+)
>  create mode 100644 arch/powerpc/kernel/test_cpuidle_latency.c
>  create mode 100644
> tools/testing/selftests/powerpc/cpuidle_latency/.gitignore
>  create mode 100644
> tools/testing/selftests/powerpc/cpuidle_latency/Makefile
>  create mode 100755
> tools/testing/selftests/powerpc/cpuidle_latency/cpuidle_latency.sh
>  create mode 100644
> tools/testing/selftests/powerpc/cpuidle_latency/settings
>