Message-ID: <4dfcb8af-be95-9665-7fad-47a088d5f0c3@amd.com>
Date:   Wed, 1 Mar 2023 23:57:24 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH] perf/x86/rapl: Enable Core RAPL for AMD
Content-Language: en-US
To:     Stephane Eranian <eranian@google.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Jiri Olsa <jolsa@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, hpa@zytor.com,
        x86@kernel.org, linux-perf-users@vger.kernel.org,
        linux-kernel@vger.kernel.org, gautham.shenoy@amd.com,
        ananth.narayan@amd.com
References: <20230217161354.129442-1-wyes.karny@amd.com>
 <Y/NdUjLiQu+D0q7b@hirez.programming.kicks-ass.net>
 <CABPqkBTU_gnTr9ayvg220T4pO5NAguvu_UJAkZO2RUATWuMJ3A@mail.gmail.com>
 <Y/ROTsCSCFtrwwTK@beas>
 <CABPqkBQ_bSTC-OEe_LrgUrpj2VsseX1ThvO-yLcEtF8vb4+AAw@mail.gmail.com>
From:   Wyes Karny <wyes.karny@amd.com>
In-Reply-To: <CABPqkBQ_bSTC-OEe_LrgUrpj2VsseX1ThvO-yLcEtF8vb4+AAw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dW1rVEJYanhlWXZEM1hlSFhvYXJaemVuRVN1Q2x3V21HNHVKQjFNV2I1SUoy?=
 =?utf-8?B?OUYxeUhoUnIwMHZEKzFnRm1Ud2FZWW1oS3BTRkxEUm1NRkh1QWQ1am1VbjJG?=
 =?utf-8?B?elFWZGlDTlFlYk9mNDM5QU5WOEc5ZHhEQ21UQmd4RkcrSG84K2F0VlFmNE9M?=
 =?utf-8?B?QVJFSGxsUEtzY2FsMFVXK3U1SmxHQnVrcVRzTHJ3dGUrSy96aUVqSUZVUXdp?=
 =?utf-8?B?VmFydVEybkV4cGw3RUNtUlZHTzYrVnVYaFoxSmlMQnJkUVcrZmUvNThZdkg3?=
 =?utf-8?B?RUxkOWhXMDVMQThPaUZ1dmtuMEc2aDVaeVpkV3puRlBlTTBhNGM3NHJwVm4r?=
 =?utf-8?B?SW0zcUU0UnJrcFRUZ1RZUWMvOFpjMDV1dXlzanphbFZqazNxZjFmcUIycWpH?=
 =?utf-8?B?SUQ3a2hnNm82Mk1FUWxlMXBpS0hWM3ZTMVhMTUhEU05aTTNxVlNGMkNIZUI4?=
 =?utf-8?B?TGMyLzJMeFFobU94QVRBY2p4T0dOYys4bmFNWnhIbWRUZmtzMWhmQndhNFBs?=
 =?utf-8?B?SEVhSG9FakJyUXhaUjh4d3RxaWJNK041TFhLLzRjQ1B2bDhHeUFvZldzSE41?=
 =?utf-8?B?VHA0dWN5MzhrdEp5eGNoK3NET21PcXJPaUE3WnYvMkRhM2NTZGhBUGdXdlU5?=
 =?utf-8?B?MERseU85MDN4bmgwdXh5bFYrUU54Q3V3K2M5NTErZHJuU1NUblJBZmtnR0lX?=
 =?utf-8?B?RUhCQjBhSFdzV1RkeTQ2dHBZaXdnLzRkcmo2bE1SS3NMdllBY1RjWVI4a1ht?=
 =?utf-8?B?cW9JOHdpOVVpYm1ZcldRL3ZKWDNUcXdzSGdvTTVSK2R1Y3p1OFZMdzliM293?=
 =?utf-8?B?Ti9LdlBsREEwNTljVkNTZmVaSTMzYXl4RkRwU2Z2RmxSeDZUMSs0b3p3VTc2?=
 =?utf-8?B?SE5MKzJoRjliRmw5elBKOG92eUxoTUdMV3JCRlJTb3VPNXJoTGpoS2NTanVn?=
 =?utf-8?B?YlBrcjYzc2FNVUxvYmVTMVZyTmlxL2kweFVySnp2eXZ1UXRqaVNyN010WGhY?=
 =?utf-8?B?NTVsZDVzWkVBK2V1Z0Raa2w0VTBndkdMUWttNUUvdm83R0x0MmRnNTdsOTZS?=
 =?utf-8?B?SVkyb1FlZmViSExNTFhmQkZrcTExMC9CeWtBeEtFUVBkZkE3MndnT3JwMEkz?=
 =?utf-8?B?ZnJyTlhBWWhyRTJKaE5iR04yaUphT3VLQlNnd293dXNnVzlPcUtESXZSRHls?=
 =?utf-8?B?UldzREk0ZXZiaG5yakFQKzlLUG90SENrVjFQOENzNDdCUjNSQi81dERoSUla?=
 =?utf-8?B?dnBDSzZ2TFlRMXIzR010dXJiT3FWWjZvWmdrUjFyb3B5ZjNrdzh1OFIybWdX?=
 =?utf-8?B?UU9DalpBNitsMDlkSmd4MFl1eUpzR3Fyb1dvclpzSFNaRjRuOThwRnZGY0RM?=
 =?utf-8?B?YjhIdDBrTXNibmVDZGhhdE5wdkhLUEp4S3ZQaVZxdyttbWZTR0JENHBBSXZW?=
 =?utf-8?B?Z0IyN3dtNGxwdmZ3VmhkUkg3SzFnaGYzM01NM3Zoc09lZ3dqV1R1YVgzL2pj?=
 =?utf-8?B?YWpoQmtjQ3ZNa3lWMWo2UUNsVnR3S2pHa2lrWDQ4VzgrR210TmdnVmM4V1B0?=
 =?utf-8?B?MTNmVG5PSFF2blo1OE1oS1V0RmNKS0c1cW5BNUVxTThjLzJtYVRRY3IyaSs5?=
 =?utf-8?B?MEduTlBNbWcvQWZqVTdaQ2NzblNMZTc0WlRrYUVIN2U2MVhtd2prTVk4RzNZ?=
 =?utf-8?B?Q0w3MFdpNEZRVFdlRFNwem9aL1Q0NnQybytyem5vTVVHbVFFQzJMNzZCbE91?=
 =?utf-8?B?NFBrVXVrNFJHK3JoSmphWDhiL0s1Z3lnNThDeHJpam5kVTZxU1J6dFhMby84?=
 =?utf-8?B?S29aWlVxamlwMWNOMFg5WnAxYWFjVElhK0NhRWpFVjdzRUpsVVZDd1lmT2NE?=
 =?utf-8?B?L1N4TVRpTGJtYVhoVnN6WFpKV3c5ZU1meVRzcVp0S00zU2hEM091SFFQZlI0?=
 =?utf-8?B?NGtqVHJvVTA5Q004dTROWS9FK3k5RWg3UktGZFFFblBLeGJsU3FlS0dNOTZz?=
 =?utf-8?B?Z2kwRHEyUm9ZYUQ0UWdOWlZhS1R2aG1NSFpDYVNpbUE5TUFzOU8vWnFiTGlW?=
 =?utf-8?B?dGQxWURCZ3dmWkl2VVhWQzFPSEllNlRCZy9OdThBM1BGZWl4YVVQRjA1Vnls?=
 =?utf-8?Q?XMY7nOxa2KqtZinUBccJ51c4Y?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 882a703d-3f0f-4d1f-7e75-08db1a82a44a
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB3876.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Mar 2023 18:27:41.3097
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: P1ApB1aeoiZ3NA29PLtApeQT//L1Qtu2xGtEU2ysH3wraRUfFnua29LgHXNlaAKcFMWQTmsQa1a/41YwjAZLBw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR12MB6570
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Stephane,

On 2/21/2023 2:20 PM, Stephane Eranian wrote:
> Hi,
> 
> Testing Wyes' patch for energy-cores on Zen3 server loaded with triad
> bench on socket0:
> 
> $ perf stat --per-core -a -C0-63 -I 1000 -e
> power/energy-cores/,power/energy-pkg/
> #           time core               cpus             counts   unit events
>      1.001019203 S0-D0-C0              1               1.28 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C0              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C1              1   4,294,967,130.96 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C1              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C2              1   4,294,967,126.23 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C2              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C3              1   4,294,967,122.50 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C3              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C4              1   4,294,967,129.92 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C4              1             231.38 Joules
> power/energy-pkg/
>      1.001019203 S0-D0-C5              1   4,294,967,121.49 Joules
> power/energy-cores/
>      1.001019203 S0-D0-C5              1             231.39 Joules
> power/energy-pkg/
> 
> I think the result of energy-cores is not reliable and I think that is
> why I did not
> include it in the patch.
> 
> Could also be a problem with the kernel code, but I don't know why it would only
> impact energy-cores given energy-pkg looks reasonable here.

This is a kernel issue. I've addressed this in v2.

Thanks,
Wyes
> 
> 
> On Mon, Feb 20, 2023 at 8:53 PM Wyes Karny <wyes.karny@amd.com> wrote:
>>
>> On 20 Feb 13:29, Stephane Eranian wrote:
>>> On Mon, Feb 20, 2023 at 3:45 AM Peter Zijlstra <peterz@infradead.org> wrote:
>>>>
>>>> On Fri, Feb 17, 2023 at 04:13:54PM +0000, Wyes Karny wrote:
>>>>> AMD processors support per-package and per-core energy monitoring
>>>>> through RAPL counters which can be accessed by users running in
>>>>> supervisor mode.
>>>>>
>>>>> Core RAPL counters gives power consumption information per core.  For
>>>>> AMD processors the package level RAPL counter are already exposed to
>>>>> perf. Expose the core level RAPL counters also.
>>>>>
>>>>> sudo perf stat -a --per-core -C 0-127 -e power/energy-cores/
>>>>>
>>>>> Output:
>>>>> S0-D0-C0           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C1           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C2           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C3           2               8.73 Joules power/energy-cores/
>>>>> S0-D0-C4           2               8.73 Joules power/energy-cores/
>>>>>
>>>>> Signed-off-by: Wyes Karny <wyes.karny@amd.com>
>>>>> ---
>>>>>  arch/x86/events/rapl.c | 5 +++--
>>>>>  1 file changed, 3 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
>>>>> index 52e6e7ed4f78..d301bbbc3b93 100644
>>>>> --- a/arch/x86/events/rapl.c
>>>>> +++ b/arch/x86/events/rapl.c
>>>>> @@ -537,7 +537,7 @@ static struct perf_msr intel_rapl_spr_msrs[] = {
>>>>>   * - want to use same event codes across both architectures
>>>>>   */
>>>>>  static struct perf_msr amd_rapl_msrs[] = {
>>>>> -     [PERF_RAPL_PP0]  = { 0, &rapl_events_cores_group, 0, false, 0 },
>>>>> +     [PERF_RAPL_PP0]  = { MSR_AMD_CORE_ENERGY_STATUS, &rapl_events_cores_group, test_msr, false, RAPL_MSR_MASK },
>>>>
>>>> Stephane, this was an oversight?
>>>>
>>> I think it may depend on the CPU model. I remember it returning either
>>> 0 or bogus values on my systems. They may have improved that.
>>> The commit msg does not show which CPU model this is run on.
>>
>> I've tested this on Zen 2, 3 and 4 server systems.
>>
>> Thanks,
>> Wyes
>>>
>>>>
>>>>>       [PERF_RAPL_PKG]  = { MSR_AMD_PKG_ENERGY_STATUS,  &rapl_events_pkg_group,   test_msr, false, RAPL_MSR_MASK },
>>>>>       [PERF_RAPL_RAM]  = { 0, &rapl_events_ram_group,   0, false, 0 },
>>>>>       [PERF_RAPL_PP1]  = { 0, &rapl_events_gpu_group,   0, false, 0 },
>>>>> @@ -764,7 +764,8 @@ static struct rapl_model model_spr = {
>>>>>  };
>>>>>
>>>>>  static struct rapl_model model_amd_hygon = {
>>>>> -     .events         = BIT(PERF_RAPL_PKG),
>>>>> +     .events         = BIT(PERF_RAPL_PP0) |
>>>>> +                       BIT(PERF_RAPL_PKG),
>>>>>       .msr_power_unit = MSR_AMD_RAPL_POWER_UNIT,
>>>>>       .rapl_msrs      = amd_rapl_msrs,
>>>>>  };
>>>>> --
>>>>> 2.34.1
>>>>>

-- 
Thanks & Regards,
Wyes