Received-SPF: pass (google.com: domain of linux-kernel+bounces-17358-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Thu, 4 Jan 2024 16:48:00 -0800
From: Vanshidhar Konda <vanshikonda@os.amperecomputing.com>
To: "lihuisong (C)" <lihuisong@huawei.com>
Cc: Ionela Voinescu <ionela.voinescu@arm.com>, 
	linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, 
	rafael@kernel.org, beata.michalska@arm.com, sumitg@nvidia.com, zengheng4@huawei.com, 
	yang@os.amperecomputing.com, will@kernel.org, sudeep.holla@arm.com, liuyonglong@huawei.com, 
	zhanjie9@hisilicon.com
Subject: Re: [PATCH] cpufreq: CPPC: Resolve the large frequency discrepancy
 from cpuinfo_cur_freq
Message-ID: <lnocwcitdbmgcyhd2dlczgdlhtfw4pfot2br2i3hqscnvr3xgq@nuxlauxum3nr>
References: <20231212072617.14756-1-lihuisong@huawei.com>
 <ZZWfJOsDlEXWYHA5@arm.com>
 <9428a1ed-ba4d-1fe6-63e8-11e152bf1f09@huawei.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <9428a1ed-ba4d-1fe6-63e8-11e152bf1f09@huawei.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ejZTclUyRGFVZnRFVDdXdStJWHFvZHBMdlE2SUM2R08xbzJkZWhLbzVjcVBv?=
 =?utf-8?B?M2Rwc0hib0kvaXBJdGZMcE5SNGE2bGlTUWV6ZnRQYldnWHNLbEZyOGFDVFhn?=
 =?utf-8?B?UFY3UkE0a2xOVFhVUUduZ0trakdjRXE5MFZvVU1WV3ZHTU1aVHpwUEIwWkdT?=
 =?utf-8?B?ai9hOTAxZ0VXQVdNS3dqUUNjQVM5SUt0SEFXS3p1SnNXR0FkTlkvUnEzYU45?=
 =?utf-8?B?UlNqeisxM0tvalVoYVZ4dEhTSS80NDNLVHZsQnpjN0d4SFJmU2F3bnRyTE9W?=
 =?utf-8?B?aVFxem5BZzA4RU1DUlFsUTdBT3d3RjlFVmpvZS94N3ZsM0NVRmZaUDVPWTU1?=
 =?utf-8?B?eXFuYStNUDZEMUU2dUhGOFBMdllmWnpZOHlTWURPd2E4Ny9KeXB6YVpsMnJK?=
 =?utf-8?B?MHhPMm5nSE96LzQ5RGthbmdQdkEvdGVFQ29Fbm9maVhZYi9BVGpodU9JSG0r?=
 =?utf-8?B?WGNqaXM4eEJyOTM4T3kvb3dvaUVTTkdnNzlkMlZQYnBCQzZpZzlobHA3a0pS?=
 =?utf-8?B?bFhXUEhTdkwvWjVXZVdSYy9sblJzVDhuVGROT3lSSnd5TUNYMkVuRkdnWjNQ?=
 =?utf-8?B?cSsvdHZuL1kvbXFUbGpmWXhRZDN6WS8zUXFyR0JyVWlvR0VHU0VzdEFqSFJi?=
 =?utf-8?B?MzJ1aFBtb09wUDRUM3BmaisyY2l1dTJtOUZXNEFpWmhDWnVlTVl6UDhNQVB3?=
 =?utf-8?B?NHppS3dQd2YyeHQwbGpDanZvTjF3M3hmQnp1K2NVcG1BNGVYVUdPRFZod0Fq?=
 =?utf-8?B?Y1BIRkNEQWRvN3g3Q3pTN2E3ZEJpYS9XZGVxVG9GYUlFL3NOQ0srYlhUZGVl?=
 =?utf-8?B?UlArWjBvYU8xYVExOW9hQThYbURXN0tYSGRZcDlGdnhRcEJWalJqVXZDUi9E?=
 =?utf-8?B?eDhtS3FEYktuYk5hWWxLaDU2RzVxNkZlYy92YmRoZ050UEc4OVVuaGlpUnNu?=
 =?utf-8?B?Z1YrdDFYRDhVRytPdGxBbHEwNklaalV4b214ZHhiSnI0THlhOHJoS1p3cnpq?=
 =?utf-8?B?SFdBMmtZYitNNVY4YWsyRGJoQjZRTGZKbjNkMGhBTVUrVlgxWkc3b1B4dzdS?=
 =?utf-8?B?TGlOcWtNalprd2xoMWpCUzQrbjZDMnVKbDRnbU55dVg2U2dkYzdvZ0svLzZ0?=
 =?utf-8?B?Q2FVMlFvMnRWYzZhR0o1YlhUWFA1Y2dMTjk0TWZieG9lMVZjNkxFZFdlbzNZ?=
 =?utf-8?B?TWNEdnE0TWRlR2JhZ2NnTy9US3RQMTdTaHRCZGwzKy80K0x3RTRIZFZHM25N?=
 =?utf-8?B?aVZEWjgwN2FRbm81L2o3SkIxeFBmTUZKRzNrczNuU2VLaC9lSHFIbUNLNmQ1?=
 =?utf-8?B?Y1FVcjFiMWR5UURCZTdGUFdkVkJOalNpL2lrNFkvOUJEenRzMFVyOFFvSHRY?=
 =?utf-8?B?b25OVzZ6MGpYY1cvT05pSDV2cGxHVHRveVZXYUxwNzdlNWl5M3UyZDZBc0hN?=
 =?utf-8?B?ZjNHQit5ckNxZlRZR2U2RUNSUjkwSll0SFhDTDNSL24yZjN3eWViNlJSSFhR?=
 =?utf-8?B?NE8zeHByeDVkUThlS2dDZGluZC8xK0ppYzJMWFBBMUd1UkZzUGZCUHJRNnlz?=
 =?utf-8?B?eE9EZkVTb2NHbmRwdXBFT2d2OHd5d2RzdmZxWUNMbG5JUmJyZXVTREtFTFpq?=
 =?utf-8?B?eEh4RTN4ZjZpbjVWbzI1eVcvdGxsOHg5Z0g4d3hxUkIwcnJkTFdMUnZEK2tN?=
 =?utf-8?B?RnR5UkN1anJkTlcxZ2pJTHVyRVh5RjFuWm5BR0czOUMvbU5FbkZXdy9VcVRa?=
 =?utf-8?B?aXF1cStRRzdienpYL3pFc2hpMEVVekhpNGR5RVVWZGl5MEdobWxGYzl6RjdZ?=
 =?utf-8?B?eENvY0dCaEZhYWNwV2s3MUJBZXl1RURHU2hGOCtEN0ZhY29Id0RaMlo5Yk84?=
 =?utf-8?B?VHF2S0srLythQ01ob3VYNW03UUsyZHV1azBNc1JSbER0WUFlNkV3SUVZa013?=
 =?utf-8?B?VGYrQk00MlZod3o4aGl2djFxOGFEa1hrNk91OU00cWhzQnpmd1Bva2RLdDAx?=
 =?utf-8?B?RGp2eHRTWXBLUXp2OU5FYXUwV3BNRzRkeFJ5bnpHSWN4ZFBpWnE5cW00Zm16?=
 =?utf-8?B?d2F2MGpCYTVrV3lSZjV1UmRaNnZxY01WM3h0cDh0Z3RJcXpZZFJ0OUt4TjlO?=
 =?utf-8?B?VVRkTkFUMkhFYkV4RSs5KzlWZTE2TFh4bCtybkZ1QktVLzVPV0ZHOUt6RElJ?=
 =?utf-8?Q?6dZaSHJpLiDWWU5ppUQ/uHKq5yreQqYbdnpFmLbpINir?=
X-OriginatorOrg: os.amperecomputing.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bc143487-4ec2-4d65-e4f7-08dc0d87f94c
X-MS-Exchange-CrossTenant-AuthSource: MW4PR01MB6498.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 00:48:03.6279
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gPyhmPCnDPKtsyrS0p7GWqbr5RveO+92aWiLrkaLSx87u+h7ctXZjSwU1K7K9lvshcEY1M1IcVl3WpgBaAS5i82X5U6Hahz4qcjDpnR7VlXYVuZ0kDzVanbPS717+gtY
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR01MB7761

On Thu, Jan 04, 2024 at 05:36:51PM +0800, lihuisong (C) wrote:
>
>在 2024/1/4 1:53, Ionela Voinescu 写道:
>>Hi,
>>
>>On Tuesday 12 Dec 2023 at 15:26:17 (+0800), Huisong Li wrote:
>>>Many developers found that the cpu current frequency is greater than
>>>the maximum frequency of the platform, please see [1], [2] and [3].
>>>
>>>In the scenarios with high memory access pressure, the patch [1] has
>>>proved the significant latency of cpc_read() which is used to obtain
>>>delivered and reference performance counter cause an absurd frequency.
>>>The sampling interval for this counters is very critical and is expected
>>>to be equal. However, the different latency of cpc_read() has a direct
>>>impact on their sampling interval.
>>>
>>Would this [1] alternative solution work for you?
>It would work for me AFAICS.
>Because the "arch_freq_scale" is also from AMU core and constant 
>counter, and read together.
>But, from their discuss line, it seems that there are some tricky 
>points to clarify or consider.

I think the changes in [1] would work better when CPUs may be idle. With this
patch we would have to wake any core that is in idle state to read the AMU
counters. Worst case, if core 0 is trying to read the CPU frequency of all
cores, it may need to wake up all the other cores to read the AMU counters.
For systems with 128 cores or more, this could be very expensive and happen
very frequently.

AFAICS, the approach in [1] would avoid this cost.

Thanks,
Vanshi

>>
>>[1] https://lore.kernel.org/lkml/20231127160838.1403404-1-beata.michalska@arm.com/
>>
>>Thanks,
>>Ionela.
>>
>>>This patch adds a interface, cpc_read_arch_counters_on_cpu, to read
>>>delivered and reference performance counter together. According to my
>>>test[4], the discrepancy of cpu current frequency in the scenarios with
>>>high memory access pressure is lower than 0.2% by stress-ng application.
>>>
>>>[1] https://lore.kernel.org/all/20231025093847.3740104-4-zengheng4@huawei.com/
>>>[2] https://lore.kernel.org/all/20230328193846.8757-1-yang@os.amperecomputing.com/
>>>[3] https://lore.kernel.org/all/20230418113459.12860-7-sumitg@nvidia.com/
>>>
>>>[4] My local test:
>>>The testing platform enable SMT and include 128 logical CPU in total,
>>>and CPU base frequency is 2.7GHz. Reading "cpuinfo_cur_freq" for each
>>>physical core on platform during the high memory access pressure from
>>>stress-ng, and the output is as follows:
>>>   0: 2699133     2: 2699942     4: 2698189     6: 2704347
>>>   8: 2704009    10: 2696277    12: 2702016    14: 2701388
>>>  16: 2700358    18: 2696741    20: 2700091    22: 2700122
>>>  24: 2701713    26: 2702025    28: 2699816    30: 2700121
>>>  32: 2700000    34: 2699788    36: 2698884    38: 2699109
>>>  40: 2704494    42: 2698350    44: 2699997    46: 2701023
>>>  48: 2703448    50: 2699501    52: 2700000    54: 2699999
>>>  56: 2702645    58: 2696923    60: 2697718    62: 2700547
>>>  64: 2700313    66: 2700000    68: 2699904    70: 2699259
>>>  72: 2699511    74: 2700644    76: 2702201    78: 2700000
>>>  80: 2700776    82: 2700364    84: 2702674    86: 2700255
>>>  88: 2699886    90: 2700359    92: 2699662    94: 2696188
>>>  96: 2705454    98: 2699260   100: 2701097   102: 2699630
>>>104: 2700463   106: 2698408   108: 2697766   110: 2701181
>>>112: 2699166   114: 2701804   116: 2701907   118: 2701973
>>>120: 2699584   122: 2700474   124: 2700768   126: 2701963
>>>
>>>Signed-off-by: Huisong Li <lihuisong@huawei.com>
>>>---
>>>  arch/arm64/kernel/topology.c | 43 ++++++++++++++++++++++++++++++++++--
>>>  drivers/acpi/cppc_acpi.c     | 22 +++++++++++++++---
>>>  include/acpi/cppc_acpi.h     |  5 +++++
>>>  3 files changed, 65 insertions(+), 5 deletions(-)
>>>
>>>diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
>>>index 7d37e458e2f5..c3122154d738 100644
>>>--- a/arch/arm64/kernel/topology.c
>>>+++ b/arch/arm64/kernel/topology.c
>>>@@ -299,6 +299,11 @@ core_initcall(init_amu_fie);
>>>  #ifdef CONFIG_ACPI_CPPC_LIB
>>>  #include <acpi/cppc_acpi.h>
>>>+struct amu_counters {
>>>+	u64 corecnt;
>>>+	u64 constcnt;
>>>+};
>>>+
>>>  static void cpu_read_corecnt(void *val)
>>>  {
>>>  	/*
>>>@@ -322,8 +327,27 @@ static void cpu_read_constcnt(void *val)
>>>  		      0UL : read_constcnt();
>>>  }
>>>+static void cpu_read_amu_counters(void *data)
>>>+{
>>>+	struct amu_counters *cnt = (struct amu_counters *)data;
>>>+
>>>+	/*
>>>+	 * The running time of the this_cpu_has_cap() might have a couple of
>>>+	 * microseconds and is significantly increased to tens of microseconds.
>>>+	 * But AMU core and constant counter need to be read togeter without any
>>>+	 * time interval to reduce the calculation discrepancy using this counters.
>>>+	 */
>>>+	if (this_cpu_has_cap(ARM64_WORKAROUND_2457168)) {
>>>+		cnt->corecnt = read_corecnt();
>>>+		cnt->constcnt = 0;
>>>+	} else {
>>>+		cnt->corecnt = read_corecnt();
>>>+		cnt->constcnt = read_constcnt();
>>>+	}
>>>+}
>>>+
>>>  static inline
>>>-int counters_read_on_cpu(int cpu, smp_call_func_t func, u64 *val)
>>>+int counters_read_on_cpu(int cpu, smp_call_func_t func, void *data)
>>>  {
>>>  	/*
>>>  	 * Abort call on counterless CPU or when interrupts are
>>>@@ -335,7 +359,7 @@ int counters_read_on_cpu(int cpu, smp_call_func_t func, u64 *val)
>>>  	if (WARN_ON_ONCE(irqs_disabled()))
>>>  		return -EPERM;
>>>-	smp_call_function_single(cpu, func, val, 1);
>>>+	smp_call_function_single(cpu, func, data, 1);
>>>  	return 0;
>>>  }
>>>@@ -364,6 +388,21 @@ bool cpc_ffh_supported(void)
>>>  	return true;
>>>  }
>>>+int cpc_read_arch_counters_on_cpu(int cpu, u64 *delivered, u64 *reference)
>>>+{
>>>+	struct amu_counters cnts = {0};
>>>+	int ret;
>>>+
>>>+	ret = counters_read_on_cpu(cpu, cpu_read_amu_counters, &cnts);
>>>+	if (ret)
>>>+		return ret;
>>>+
>>>+	*delivered = cnts.corecnt;
>>>+	*reference = cnts.constcnt;
>>>+
>>>+	return 0;
>>>+}
>>>+
>>>  int cpc_read_ffh(int cpu, struct cpc_reg *reg, u64 *val)
>>>  {
>>>  	int ret = -EOPNOTSUPP;
>>>diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c
>>>index 7ff269a78c20..f303fabd7cfe 100644
>>>--- a/drivers/acpi/cppc_acpi.c
>>>+++ b/drivers/acpi/cppc_acpi.c
>>>@@ -1299,6 +1299,11 @@ bool cppc_perf_ctrs_in_pcc(void)
>>>  }
>>>  EXPORT_SYMBOL_GPL(cppc_perf_ctrs_in_pcc);
>>>+int __weak cpc_read_arch_counters_on_cpu(int cpu, u64 *delivered, u64 *reference)
>>>+{
>>>+	return 0;
>>>+}
>>>+
>>>  /**
>>>   * cppc_get_perf_ctrs - Read a CPU's performance feedback counters.
>>>   * @cpunum: CPU from which to read counters.
>>>@@ -1313,7 +1318,8 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs)
>>>  		*ref_perf_reg, *ctr_wrap_reg;
>>>  	int pcc_ss_id = per_cpu(cpu_pcc_subspace_idx, cpunum);
>>>  	struct cppc_pcc_data *pcc_ss_data = NULL;
>>>-	u64 delivered, reference, ref_perf, ctr_wrap_time;
>>>+	u64 delivered = 0, reference = 0;
>>>+	u64 ref_perf, ctr_wrap_time;
>>>  	int ret = 0, regs_in_pcc = 0;
>>>  	if (!cpc_desc) {
>>>@@ -1350,8 +1356,18 @@ int cppc_get_perf_ctrs(int cpunum, struct cppc_perf_fb_ctrs *perf_fb_ctrs)
>>>  		}
>>>  	}
>>>-	cpc_read(cpunum, delivered_reg, &delivered);
>>>-	cpc_read(cpunum, reference_reg, &reference);
>>>+	if (cpc_ffh_supported()) {
>>>+		ret = cpc_read_arch_counters_on_cpu(cpunum, &delivered, &reference);
>>>+		if (ret) {
>>>+			pr_debug("read arch counters failed, ret=%d.\n", ret);
>>>+			ret = 0;
>>>+		}
>>>+	}
>>>+	if (!delivered || !reference) {
>>>+		cpc_read(cpunum, delivered_reg, &delivered);
>>>+		cpc_read(cpunum, reference_reg, &reference);
>>>+	}
>>>+
>>>  	cpc_read(cpunum, ref_perf_reg, &ref_perf);
>>>  	/*
>>>diff --git a/include/acpi/cppc_acpi.h b/include/acpi/cppc_acpi.h
>>>index 6126c977ece0..07d4fd82d499 100644
>>>--- a/include/acpi/cppc_acpi.h
>>>+++ b/include/acpi/cppc_acpi.h
>>>@@ -152,6 +152,7 @@ extern bool cpc_ffh_supported(void);
>>>  extern bool cpc_supported_by_cpu(void);
>>>  extern int cpc_read_ffh(int cpunum, struct cpc_reg *reg, u64 *val);
>>>  extern int cpc_write_ffh(int cpunum, struct cpc_reg *reg, u64 val);
>>>+extern int cpc_read_arch_counters_on_cpu(int cpu, u64 *delivered, u64 *reference);
>>>  extern int cppc_get_epp_perf(int cpunum, u64 *epp_perf);
>>>  extern int cppc_set_epp_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls, bool enable);
>>>  extern int cppc_get_auto_sel_caps(int cpunum, struct cppc_perf_caps *perf_caps);
>>>@@ -209,6 +210,10 @@ static inline int cpc_write_ffh(int cpunum, struct cpc_reg *reg, u64 val)
>>>  {
>>>  	return -ENOTSUPP;
>>>  }
>>>+static inline int cpc_read_arch_counters_on_cpu(int cpu, u64 *delivered, u64 *reference)
>>>+{
>>>+	return -EOPNOTSUPP;
>>>+}
>>>  static inline int cppc_set_epp_perf(int cpu, struct cppc_perf_ctrls *perf_ctrls, bool enable)
>>>  {
>>>  	return -ENOTSUPP;
>>>-- 
>>>2.33.0
>>>
>>.
>
>_______________________________________________
>linux-arm-kernel mailing list
>linux-arm-kernel@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/linux-arm-kernel