Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp2975414pxa; Tue, 25 Aug 2020 08:13:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJykn5ghvzgcfyV+GaCGLWwP2bwDTj5H4asCes8IoYWniup8OLFRVFFBuB9pvPFdMAsRFm2P X-Received: by 2002:a17:907:aa2:: with SMTP id bz2mr10783560ejc.521.1598368436121; Tue, 25 Aug 2020 08:13:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1598368436; cv=none; d=google.com; s=arc-20160816; b=kGejVJMa/e6w6xAQNWgIezLiJ0dLJiX81iC2fNKWQVEQ5bXiGrunoFGsi3+H8tZpEo +EMLDSwJIftHa9f2/Xfjd+/P6dJUxtnZasRWJmhYI+D0V2hpnsMbvxwDs6lDguAVuQ1P PbrhsmFcf2NdCke8UjwAhXSgoxZnIlm2lJe3PlvKH9gCtjosAPEdRKHxJf/g0+QzhxWP 7kJvXFVb1+YVMWjdI0OfjwtQcFCv61D65WF4SPsHs0IOe9pFK+buxElIyYlAU5v67myw YztrWH2Y832lHJPZ4FWFTltg1NEEGFZeuSQGOCKAbeR7QcK7I1jYZtjtBJig0J+sjaX1 wrHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:date:cc:to:from:subject :message-id:ironport-sdr:ironport-sdr; bh=TxN9zRBEBKtX8igMpABOBuUA3tGcm4+L5H6goaSvaEE=; b=vebMH7hTOKJq3wtA1S7zTKzx1DsHsoc8u6irBSUpekZgJcb0nDhjucSKq83cXQhfYU LzV9ricZpyLz6eWLK+gX+BUFmha1mjjKh1kDMNOnnsQY28fhim+QyIJYfRvI+3Vwtp0D N1XTHbW5YC/YNdEYw/tad60l331vALFh2gXGfhZUDSzFkYl4qjPYz8SUxhKtVbj9LSx6 92iO1NZ72rI697iLtvUQVOH7qhD2BKRQq0GCs1NG/qMm8LRqyY4517a63NbBOMvLiqFh KEZCaW5S9pEDe8L1MPc7V8ny4rw1qRKbtuv4LjkH+v+wZ9k85g37GULFBDnvQcSahfB7 9AZA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id yh22si10604748ejb.583.2020.08.25.08.13.32; Tue, 25 Aug 2020 08:13:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726876AbgHYPMO (ORCPT + 99 others); Tue, 25 Aug 2020 11:12:14 -0400 Received: from mga17.intel.com ([192.55.52.151]:11547 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726673AbgHYPMK (ORCPT ); Tue, 25 Aug 2020 11:12:10 -0400 IronPort-SDR: UFXd8xJ2a/e69RsTXe9lJh/+yloy76bkUtHrTRxzTQfe69hkYpWQGQ5mJH6ZtVwlKYsMGs6zAy IVfCYjXiF7ZA== X-IronPort-AV: E=McAfee;i="6000,8403,9723"; a="136191158" X-IronPort-AV: E=Sophos;i="5.76,353,1592895600"; d="scan'208";a="136191158" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Aug 2020 08:11:54 -0700 IronPort-SDR: uvR6zCLMdAk2LJ51webBU5H9odU+fT7f0cGUCpaLI7k+n3vyxlKmPF/riiHNDfBYnMl0Q+XP06 HIZbnWbNXUeg== X-IronPort-AV: E=Sophos;i="5.76,353,1592895600"; d="scan'208";a="474383039" Received: from spandruv-mobl.amr.corp.intel.com ([10.251.25.243]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Aug 2020 08:11:54 -0700 Message-ID: Subject: Re: [PATCH v2 0/5] cpufreq: intel_pstate: Address some HWP-related oddities From: Srinivas Pandruvada To: Doug Smythies , "'Rafael J. Wysocki'" Cc: 'LKML' , 'Linux PM' Date: Tue, 25 Aug 2020 08:11:53 -0700 In-Reply-To: <002001d67a7b$2b46e1c0$81d4a540$@net> References: <4169555.5IIHXK4Dsd@kreacher> <5cf44a75c9f73740d2a22dbfc5c7a57489b1a3ca.camel@linux.intel.com> <002001d67a7b$2b46e1c0$81d4a540$@net> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Doug, On Mon, 2020-08-24 at 18:00 -0700, Doug Smythies wrote: > Hi Srinivas, > > I think there is a disconnect between your written > description of what is going on and your supporting MSR reads. > I reproduced again. I see the copy paste individual at the first place swapped. I pasted the full output by direct copy - paste from the screen. But the issues are still there. [labuser@otcpl-perf-test-skx-i9 ~]$ sudo -s [root@otcpl-perf-test-skx-i9 labuser]# rdmsr -a 0x774 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002b0c 80002b0c 80002d0d [root@otcpl-perf-test-skx-i9 labuser]# cd /sys/devices/system/cpu/cpu1 [root@otcpl-perf-test-skx-i9 cpu1]# cd cpufreq/ [root@otcpl-perf-test-skx-i9 cpufreq]# echo 127 > energy_performance_preference [root@otcpl-perf-test-skx-i9 cpufreq]# cat energy_performance_preference 127 [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774 80002b0c 7f002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002b0c 80002b0c 80002d0d [root@otcpl-perf-test-skx-i9 cpufreq]# echo 0 > /sys/devices/system/cpu/cpu1/online [root@otcpl-perf-test-skx-i9 cpufreq]# echo 1 > /sys/devices/system/cpu/cpu1/online [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002d0c 80002b0c 80002b0c 80002b0c 80002b0c 80002b0c 80002d0d 7f002b0c [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -p 1 0x774 7f002b0c [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -p 19 0x774 80002d0d [root@otcpl-perf-test-skx-i9 cpufreq]# rtcwake -m mem -s 10 rtcwake: wakeup from "mem" using /dev/rtc0 at Tue Aug 25 15:04:02 2020 [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774 80002b0c 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 8000ff00 [root@otcpl-perf-test-skx-i9 cpufreq]# dmesg > ~/temp/dmesg.txt [root@otcpl-perf-test-skx-i9 cpufreq]# cat energy_performance_preference 127 [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -p 1 0x774 8000ff00 [root@otcpl-perf-test-skx-i9 cpufreq]# echo passive > /sys/devices/system/ clockevents/ clocksource/ container/ cpu/ edac/ m achinecheck/ memory/ node/ [root@otcpl-perf-test-skx-i9 cpufreq]# echo passive > /sys/devices/system/cpu/intel_pstate/status [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774 80002b0c 7f000000 80000000 80002d0c 80002b0c 80000000 80000000 80002b0c 80000000 80002d0c 80000000 80002b0c 80002b0d 80002d0c 80000000 80000000 80000000 80002b0d 80000000 80000000 [root@otcpl-perf-test-skx-i9 cpufreq]# rdmsr -a 0x774 80002b0c 7f000000 80000000 80002d0c 80002b0c 80000000 80000000 80002b0c 80000000 80002d0c 80000000 80002b0c 80002b0d 80002d0c 80000000 80002b0c 80000000 80002b0d 80000000 80000000 > On 2020.08.24 16:56 Srinivas Pandruvada wrote: > > On Mon, 2020-08-24 at 19:39 +0200, Rafael J. Wysocki wrote: > > > Hi All, > > > > > > The v2 is here to address feedback from Doug and one issue found > > > by > > > me. > > > > > > The purpose of this series is to address some peculiarities > > > related > > > to > > > taking CPUs offline/online and switching between different > > > operation > > > modes with HWP enabled that have become visible after allowing > > > the > > > driver to work in the passive mode with HWP enabled in 5.9-rc1 > > > (and > > > one that was there earlier, but can be addressed easily after the > > > changes madein 5.9-rc1). > > > > > > Please refer to the patch changelogs for details. > > > > > > For easier testing/review, the series is available from the git > > > branch at: > > > > > > git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux- > > > pm.git \ > > > intel_pstate-testing > > > > > > > Applied these patches to 5.9-rc2 > > So did I, and the issues I reported the other day are fine now. > I did try a few of the things you were doing. > > > - After s3 limits got messed up. > > # cat /sys/power/mem_sleep > > s2idle [deep] > > > > - In the dmesg unchecked MSR for HWP register > > > > 1. > > Before test > > > > sudo rdmsr -a 0x774 > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0d > > 7f002b0c > > ?? This looks like the MSR > read for further below, and for > CPU 19 instead of 1. > > > cd /sys/devices/system/cpu/intel_pstate/ > > [root@otcpl-perf-test-skx-i9 intel_pstate]# grep . * > > hwp_dynamic_boost:0 > > max_perf_pct:100 > > min_perf_pct:27 > > no_turbo:0 > > num_pstates:32 > > status:active > > turbo_pct:32 > > > > cd ../cpu1/cpufreq/ > > [root@otcpl-perf-test-skx-i9 cpufreq]# grep . * > > affected_cpus:1 > > base_frequency:3300000 > > cpuinfo_max_freq:4300000 > > cpuinfo_min_freq:1200000 > > cpuinfo_transition_latency:0 > > energy_performance_available_preferences:default performance > > balance_performance balance_power power > > energy_performance_preference:balance_performance > > related_cpus:1 > > scaling_available_governors:performance powersave > > scaling_cur_freq:1200000 > > scaling_driver:intel_pstate > > scaling_governor:powersave > > scaling_max_freq:4300000 > > scaling_min_freq:1200000 > > scaling_setspeed: > > > > > > 2. Now change the EPP > > > > # echo 127 > energy_performance_preference > > sudo rdmsr -a 0x774 > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0d > > This looks like the original MSR read. > > > Good here > > > > 3. Offline/online good > > > > [root@otcpl-perf-test-skx-i9 cpufreq]# echo 0 > > > /sys/devices/system/cpu/cpu1/online > > [root@otcpl-perf-test-skx-i9 cpufreq]# echo ` > > > /sys/devices/system/cpu/cpu1/online > > > echo ` > /sys/devices/system/cpu/cpu1/online ^C > > [root@otcpl-perf-test-skx-i9 cpufreq]# echo 1 > > > /sys/devices/system/cpu/cpu1/online > > > > sudo rdmsr -a 0x774 > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0d > > 7f002b0c > > O.K. > > > Good. Online restored the setting > > > > 4. Now S3 > > > > rtcwake -m mem -s 10 > > Cool command. I did not know about it. > I tried it. > > All limits are now messed up > > > > sudo rdmsr -a 0x774 > > 80002b0c > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > > > Yes, I got the same: > > # /home/doug/c/msr-decoder (edited) > 6.) 0x774: IA32_HWP_REQUEST: CPU 0-5 : > raw: 80002E08 : 8000FF01 : 8000FF01 : 8000FF01 : 8000FF01 : > 8000FF01 : > min: 8 : 1 : 1 : 1 : 1 > : 1 : > max: 46 : 255 : 255 : 255 : 255 > : 255 : > des: 0 : 0 : 0 : 0 : 0 > : 0 : > epp: 128 : 128 : 128 : 128 : 128 > : 128 : > act: 0 : 0 : 0 : 0 : 0 > : 0 : > > > 5. Now switch to passive > > Again bad, some CPU max/min is 0 > > > > sudo rdmsr -a 0x774 > > 80002b0d > > 7f002b0f > > Hmmm... Now seems to be CPU 1 > > > 80002b0c > > 80002d0e > > 80002b0c > > 80002b0d > > 80002b0f > > 80002b2b > > 80002b0c > > 80002d1d > > 80000000 > > 80002b0c > > 80002b0c > > 80000000 > > 80000000 > > 80000000 > > 80000000 > > 80000000 > > 80000000 > > 80000000 > > MSR 774 was good for me, but in general my decoder was having > troubles. > > 0x774: IA32_HWP_REQUEST: CPU 0-5 : > sh: 0: getcwd() failed: No such file or directory > raw: 80002E2E : 7F002E2E : 80002E2E : 80002E2E : 80002E2E : > 80002E2E : > min: 46 : 46 : 46 : 46 : 46 > : 46 : > max: 46 : 46 : 46 : 46 : 46 > : 46 : > des: 0 : 0 : 0 : 0 : 0 > : 0 : > epp: 128 : 127 : 128 : 128 : 128 > : 128 : > act: 0 : 0 : 0 : 0 : 0 > : 0 : > > > 6. > > Switched back to active to restore back > > > > Lost EPP setting but rest are good. > > > > sudo rdmsr -a 0x774 > > 80002b0c > > 7f002b0c > > And again, now seems to be CPU1. > > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002b0c > > 80002d0d > > > > 7. S3 again > > > > rtcwake -m mem -s 10 > > > > Again messed up > > > > sudo rdmsr -a 0x774 > > 80002b0c > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > 8000ff00 > > > > This time dmesg didn't have unchecked MSR in dmesg (I think because > > I > > didn't change EPP before) > > > > Thanks, > > Srinivas > > > > > Thanks, > > > Rafael > > > > > > > > >