Received: by 2002:a05:6a10:9848:0:0:0:0 with SMTP id x8csp2440458pxf; Sun, 4 Apr 2021 01:36:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyPlFaTCfhQE51oaDI0zWN1XaIztNYe6lovmU8cwnP4tWwiO6bbEM682X41fLGcAQBoOp/h X-Received: by 2002:a05:6e02:198f:: with SMTP id g15mr16142767ilf.200.1617525361425; Sun, 04 Apr 2021 01:36:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1617525361; cv=none; d=google.com; s=arc-20160816; b=NThC5Il9mMjJk/WyMDcymAFz7Fl832gy5fd87kHPo0snBWAjghZTy0X0m2nUWVlhbA kBGzIkFwdDL2p0gQzf5TSAsTj9ZdLMVgKTi80ZehTeNf6vZuknxYRbaj//Jmfh0XQ6eF nuWUehsCYrnHDR/PNKJzE6a8fKW0WOE03WzXzk9pUp2YA/wHUMM4J4z3Hy4nrWV/DjSY e8E4+O9Dn38EIAINeS8pThTEoHYKHDKNwZdEdaaqg2+iT8OIyFGC6DsNPNej2P0hluR5 5GZNo2e4PDjW/gUje6EBFgC4G4fqudpdokZ2cq0eHC3D09iLV2DSiG95QBP0PQ26RVTj GHGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=XVtw3e/O49gnpckVQFrAiS5Jf1enaz3UqNsrJVyUPSQ=; b=H9vyLhuuCxIJEBp2jh4VgrTlz2rO63sFLRqHWrFJYldLXWOKNdCQszKxIT4NB+K4wF yCInSZqjaoeUCKQHY48OK8xhcHwQwsBKHJclahZOeUMSNDotWKmY1HiCqqfE+Is9C1ph TX3OPZrvcTlD/ef+pYiO0oZOdDYWC3gNS1ya2MyzPYPW0Wdc7dxSdLjSZrSekaLF41d+ 0edXDhpTAQWqsy8vVIAX8ePYZJyi5QqfCN++5ooMLlaNpy1gAF0r7BdRf2l7iFRx/S16 dgQBjJDiEsO5Xry9ejNz3LcpTaJhodfC7cTR1UJ6X1TAse59UNIdjdnVEAvyG8g1M9gf mLiw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=N3B5MSy8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id c5si12320864jam.17.2021.04.04.01.35.48; Sun, 04 Apr 2021 01:36:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=N3B5MSy8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230337AbhDDIeS (ORCPT + 99 others); Sun, 4 Apr 2021 04:34:18 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:14358 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230331AbhDDIeQ (ORCPT ); Sun, 4 Apr 2021 04:34:16 -0400 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1348WuMf160271; Sun, 4 Apr 2021 04:34:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=XVtw3e/O49gnpckVQFrAiS5Jf1enaz3UqNsrJVyUPSQ=; b=N3B5MSy8aMHnF9Oa0kYHSEKfKRtlrucRoMvFZ6sO8R3idluD52Hgc/Ir+Sa2xZRegcSR VYdAnq0Q/ul7My+hHGksln2FG5lHywGjjnMzvk7V0klDft03BKZf9KbipS6A3HijtjYP Ee8hBbP8mM8qFqNnsmZu/z2Ye/jluFyJRm7byoK7ghOZEfIRfulSnPr2Ulhb/bcAdL5o fIPuTEELtOGS0CuDFQvgoDGqBBkVsiQu08nYkcMP6P+uagNet7Xq7QCD5lxERgHCMfFC zFsbFzrta4pW8xsB3oPPlFk0AWIfE99fTaMi5i0Cy7BvtkhBoWorLDZqgrKbSSg22LKO 4A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 37q593vsms-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 04 Apr 2021 04:34:09 -0400 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 1348Y8UI162150; Sun, 4 Apr 2021 04:34:08 -0400 Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 37q593vsmd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 04 Apr 2021 04:34:08 -0400 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.0.43/8.16.0.43) with SMTP id 1348Y69G001153; Sun, 4 Apr 2021 08:34:06 GMT Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by ppma03ams.nl.ibm.com with ESMTP id 37q2y9g7hm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 04 Apr 2021 08:34:06 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1348Y3KY6553886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 4 Apr 2021 08:34:03 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 67D724C040; Sun, 4 Apr 2021 08:34:03 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3E9A24C046; Sun, 4 Apr 2021 08:34:01 +0000 (GMT) Received: from pratiks-thinkpad.ibmuc.com (unknown [9.85.90.174]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Sun, 4 Apr 2021 08:34:01 +0000 (GMT) From: Pratik Rajesh Sampat To: rjw@rjwysocki.net, daniel.lezcano@linaro.org, shuah@kernel.org, dsmythies@telus.net, ego@linux.vnet.ibm.com, svaidy@linux.ibm.com, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, pratik.r.sampat@gmail.com, psampat@linux.ibm.com Subject: [RFC v3 2/2] selftest/cpuidle: Add support for cpuidle latency measurement Date: Sun, 4 Apr 2021 14:03:54 +0530 Message-Id: <20210404083354.23060-3-psampat@linux.ibm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210404083354.23060-1-psampat@linux.ibm.com> References: <20210404083354.23060-1-psampat@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: c98N6g8845OxoSKGg22-1_UZQAWN1xGb X-Proofpoint-ORIG-GUID: ns2Ize-2EZf-LvcesV9J_531UcYMfm-6 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369,18.0.761 definitions=2021-04-04_01:2021-04-01,2021-04-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 impostorscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 clxscore=1015 priorityscore=1501 suspectscore=0 mlxscore=0 lowpriorityscore=0 adultscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104040058 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The cpuidle latency selftest provides support to systematically extract, analyse and present IPI and timer based wakeup latencies for each CPU and each idle state available on the system. The selftest leverages test-cpuidle_latency module's debugfs interface to interact and extract latency information from the kernel. The selftest inserts the module if already not inserted, disables all the idle states and enables them one by one testing the following: 1. Keeping source CPU constant, iterate through all the CPUS measuring IPI latency for baseline (CPU is busy with cat /dev/random > /dev/null workload) and then when the CPU is allowed to be at rest 2. Iterating through all the CPUs, sending expected timer durations to be equivalent to the residency of the deepest idle state enabled and extracting the difference in time between the time of wakeup and the expected timer duration The timer based test produces run to run variance on some intel based systems that sport a mechansim "C-state pre-wake" which can pre-wake a CPU from an idle state when timers are armed. For systems and architectures that don't have this mechansim can leverage timer tests with the -i option. To run this test specifically: $ sudo make -C tools/testing/selftests TARGETS="cpuidle" run_tests There are a few optional arguments too that the script can take [-h ] [-i ] [-m ] [-o ] [-v (run on all cpus)] Default Output location in: tools/testing/selftest/cpuidle/cpuidle.log To run the test without re-compiling: $ cd tools/testing/selftest/cpuidle/ $ sudo ./cpuidle.sh Signed-off-by: Pratik Rajesh Sampat --- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/cpuidle/Makefile | 6 + tools/testing/selftests/cpuidle/cpuidle.sh | 326 +++++++++++++++++++++ tools/testing/selftests/cpuidle/settings | 2 + 4 files changed, 335 insertions(+) create mode 100644 tools/testing/selftests/cpuidle/Makefile create mode 100755 tools/testing/selftests/cpuidle/cpuidle.sh create mode 100644 tools/testing/selftests/cpuidle/settings diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 6c575cf34a71..561eb67a4839 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -7,6 +7,7 @@ TARGETS += cgroup TARGETS += clone3 TARGETS += core TARGETS += cpufreq +TARGETS += cpuidle TARGETS += cpu-hotplug TARGETS += drivers/dma-buf TARGETS += efivarfs diff --git a/tools/testing/selftests/cpuidle/Makefile b/tools/testing/selftests/cpuidle/Makefile new file mode 100644 index 000000000000..cbe13feced34 --- /dev/null +++ b/tools/testing/selftests/cpuidle/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 +all: + +TEST_PROGS := cpuidle.sh + +include ../lib.mk \ No newline at end of file diff --git a/tools/testing/selftests/cpuidle/cpuidle.sh b/tools/testing/selftests/cpuidle/cpuidle.sh new file mode 100755 index 000000000000..e3d0c853be8f --- /dev/null +++ b/tools/testing/selftests/cpuidle/cpuidle.sh @@ -0,0 +1,326 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +LOG=cpuidle.log +MODULE=/lib/modules/$(uname -r)/kernel/drivers/cpuidle/test-cpuidle_latency.ko + +# Kselftest framework requirement - SKIP code is 4. +ksft_skip=4 + +SMT=$(lscpu | grep "Thread(s) per core" | awk '{print $4}') +RUN_TIMER_TEST=0 + +DISABLE=1 +ENABLE=0 + +helpme() +{ + printf "Usage: $0 [-h] [-todg args] + [-h ] + [-i ] + [-m ] + [-o ] + [-v ] + \n" + exit 2 +} + +parse_arguments() +{ + while getopts ht:m:o:vt:it: arg + do + case $arg in + h) # --help + helpme + ;; + i) # run timer tests + RUN_TIMER_TEST=1 + ;; + m) # --mod-file + MODULE=$OPTARG + ;; + o) # output log files + LOG=$OPTARG + ;; + v) # Verbose mode + SMT=1 + ;; + \?) + helpme + ;; + esac + done +} + +ins_mod() +{ + debugfs_file=/sys/kernel/debug/latency_test/ipi_latency_ns + # Check if the module is already loaded + if [ -f "$debugfs_file" ]; then + printf "Module already loaded\n\n" + return 0 + fi + # Try to load the module + if [ ! -f "$MODULE" ]; then + printf "$MODULE module does not exist. Exitting\n" + exit $ksft_skip + fi + printf "Inserting $MODULE module\n\n" + insmod $MODULE + if [ $? != 0 ]; then + printf "Insmod $MODULE failed\n" + exit $ksft_skip + fi +} + +compute_average() +{ + arr=("$@") + sum=0 + size=${#arr[@]} + if [ $size == 0 ]; then + avg=0 + return 1 + fi + for i in "${arr[@]}" + do + sum=$((sum + i)) + done + avg=$((sum/size)) +} + +cpu_is_online() +{ + cpu=$1 + if [ ! -f "/sys/devices/system/cpu/cpu$cpu/online" ]; then + echo 1 + return + fi + status=$(cat /sys/devices/system/cpu/cpu$cpu/online) + echo $status +} + +# Perform operation on each CPU for the given state +# $1 - Operation: enable (0) / disable (1) +# $2 - State to enable +op_state() +{ + for ((cpu=0; cpu /sys/devices/system/cpu/cpu$cpu/cpuidle/state$2/disable + done +} + +cpuidle_enable_state() +{ + state=$1 + op_state $ENABLE $state +} + +cpuidle_disable_state() +{ + state=$1 + op_state $DISABLE $state +} + +# Enable/Disable all stop states for all CPUs +# $1 - Operation: enable (0) / disable (1) +op_cpuidle() +{ + for ((state=0; state /dev/null & + task_pid=$! + # Wait for the workload to achieve 100% CPU usage + sleep 1 + fi + taskset 0x1 echo $dest_cpu > /sys/kernel/debug/latency_test/ipi_cpu_dest + ipi_latency=$(cat /sys/kernel/debug/latency_test/ipi_latency_ns) + src_cpu=$(cat /sys/kernel/debug/latency_test/ipi_cpu_src) + if [ "$1" = "baseline" ]; then + kill $task_pid + wait $task_pid 2>/dev/null + fi +} + +# Incrementally Enable idle states one by one and compute the latency +run_ipi_tests() +{ + extract_latency + # Disable idle states for CPUs + op_cpuidle $DISABLE + + declare -a avg_arr + echo -e "--IPI Latency Test---" | tee -a $LOG + + echo -e "--Baseline IPI Latency measurement: CPU Busy--" >> $LOG + printf "%s %10s %12s\n" "SRC_CPU" "DEST_CPU" "IPI_Latency(ns)" >> $LOG + for ((cpu=0; cpu> $LOG + avg_arr+=($ipi_latency) + done + compute_average "${avg_arr[@]}" + echo -e "Baseline Avg IPI latency(ns): $avg" | tee -a $LOG + + for ((state=0; state> $LOG + cpuidle_enable_state $state + printf "%s %10s %12s\n" "SRC_CPU" "DEST_CPU" "IPI_Latency(ns)" >> $LOG + for ((cpu=0; cpu> $LOG + avg_arr+=($ipi_latency) + done + compute_average "${avg_arr[@]}" + echo -e "Expected IPI latency(ns): ${latency_arr[$state]}" >> $LOG + echo -e "Observed Avg IPI latency(ns) - State $state: $avg" | tee -a $LOG + cpuidle_disable_state $state + done +} + +# Extract the residency in microseconds and convert to nanoseconds. +# Add 100 ns so that the timer stays for a little longer than the residency +extract_residency() +{ + for ((state=0; state /dev/null & + task_pid=$! + # Wait for the workload to achieve 100% CPU usage + sleep 1 + fi + taskset -c $dest_cpu echo $3 > /sys/kernel/debug/latency_test/timeout_expected_ns + # Wait for the result to populate + sleep 0.1 + timeout_diff=$(cat /sys/kernel/debug/latency_test/timeout_diff_ns) + src_cpu=$(cat /sys/kernel/debug/latency_test/timeout_cpu_src) + if [ "$1" = "baseline" ]; then + kill $task_pid + wait $task_pid 2>/dev/null + fi +} + +run_timeout_tests() +{ + extract_residency + # Disable idle states for all CPUs + op_cpuidle $DISABLE + + declare -a avg_arr + echo -e "\n--Timeout Latency Test--" | tee -a $LOG + + echo -e "--Baseline Timeout Latency measurement: CPU Busy--" >> $LOG + printf "%s %10s %10s\n" "Wakeup_src" "Baseline_delay(ns)">> $LOG + for ((cpu=0; cpu> $LOG + avg_arr+=($timeout_diff) + done + compute_average "${avg_arr[@]}" + echo -e "Baseline Avg timeout diff(ns): $avg" | tee -a $LOG + + for ((state=0; state> $LOG + cpuidle_enable_state $state + printf "%s %10s %10s\n" "Wakeup_src" "Baseline_delay(ns)" "Delay(ns)" >> $LOG + for ((cpu=0; cpu> $LOG + avg_arr+=($timeout_diff) + done + compute_average "${avg_arr[@]}" + echo -e "Expected timeout(ns): ${residency_arr[$state]}" >> $LOG + echo -e "Observed Avg timeout diff(ns) - State $state: $avg" | tee -a $LOG + cpuidle_disable_state $state + done +} + +declare -a residency_arr +declare -a latency_arr + +parse_arguments $@ + +rm -f $LOG +touch $LOG +NUM_CPUS=$(nproc --all) +NUM_STATES=$(ls -1 /sys/devices/system/cpu/cpu0/cpuidle/ | wc -l) + +ins_mod $MODULE + +run_ipi_tests +if [ $RUN_TIMER_TEST == 1 ]; then + run_timeout_tests +fi + +# Enable all idle states for all CPUs +op_cpuidle $ENABLE +printf "Removing $MODULE module\n" +printf "Full Output logged at: $LOG\n" +rmmod $MODULE diff --git a/tools/testing/selftests/cpuidle/settings b/tools/testing/selftests/cpuidle/settings new file mode 100644 index 000000000000..a26c38a70d77 --- /dev/null +++ b/tools/testing/selftests/cpuidle/settings @@ -0,0 +1,2 @@ +timeout=0 + -- 2.17.1