Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 12 Dec 2022 17:02:24 +0800
From:   Huang Rui <ray.huang@amd.com>
To:     "Yuan, Perry" <Perry.Yuan@amd.com>
Cc:     "rafael.j.wysocki@intel.com" <rafael.j.wysocki@intel.com>,
        "Limonciello, Mario" <Mario.Limonciello@amd.com>,
        "viresh.kumar@linaro.org" <viresh.kumar@linaro.org>,
        "Sharma, Deepak" <Deepak.Sharma@amd.com>,
        "Fontenot, Nathan" <Nathan.Fontenot@amd.com>,
        "Deucher, Alexander" <Alexander.Deucher@amd.com>,
        "Huang, Shimmer" <Shimmer.Huang@amd.com>,
        "Du, Xiaojian" <Xiaojian.Du@amd.com>,
        "Meng, Li (Jassmine)" <Li.Meng@amd.com>,
        "Karny, Wyes" <Wyes.Karny@amd.com>,
        "linux-pm@vger.kernel.org" <linux-pm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v7 06/13] cpufreq: amd-pstate: implement amd pstate cpu
 online and offline callback
Message-ID: <Y5buIL08ovNxYp5f@amd.com>
References: <20221208111852.386731-1-perry.yuan@amd.com>
 <20221208111852.386731-7-perry.yuan@amd.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20221208111852.386731-7-perry.yuan@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CpyMNu4M9ms7J5hYKYgzZPsuMZLMDM2RdOKQ0Y1BZbFkzdFiR1jkdq8wARjS?=
 =?us-ascii?Q?zjvCyWYPsyFo2oO1Jj7mInjcdbdZ8MaOKb2sx1bVZ0GLXJHMVY/jhtdwfh+x?=
 =?us-ascii?Q?DoTjIOZTlLSt7nAHcRb+9D8xPimN+yNSajNtUPLzpcrL5MDm5HThhASuz0AY?=
 =?us-ascii?Q?hISCtwG9ThAZgwhArYpLiOOfvSgLB6sMXwEbnw61ZIb4RxyiP6+lMpiqPdZB?=
 =?us-ascii?Q?4xoKUotydhTDbmMKAcQtj5mHewHsP03sfljH0efuH7govgemqjBKT3kNZkux?=
 =?us-ascii?Q?TjVlj3mzAUfsWDnXNiYZsBO2v6mXONHOfgWJDWiLvITreU3amSdi1GS5B+o8?=
 =?us-ascii?Q?a7FY6jx0cDSw9265W4BZj0asM5yzb8JvtNN/Acu8ru/kMO48fHjpCz/ZCNMa?=
 =?us-ascii?Q?/5OtP8Qw5+gp8JXMgUac8EdPSHHQQ1YmU1pCYwnOkPFVSytP1vLso11v+q3c?=
 =?us-ascii?Q?pEPOK99go3UX8aRav+LF0MTiA+HeR37ywvEzlaN/jei2VDp3TUVTZrWJ2LNz?=
 =?us-ascii?Q?SsmyehzuyIUDs77YosMj1GF4C67PynRowLoIbJEamzkoCxD0aTVTiITMdy5U?=
 =?us-ascii?Q?LW60xdBIRNXOuZCRHn4LCqEHOxxS8z1+CTP5tUiSWgnnhCU6h5j0D2tGspnw?=
 =?us-ascii?Q?n2Vx57mlnRCxakj1FV9MbG+tIshMgVoO7GDB7ZWhigvd1JyxM88hZMaB1O5R?=
 =?us-ascii?Q?KqPuJkJgDWgcrN/plJ/rLo0kteVHouXOUQck1TkoIclmD12GxA72m1p2R9gQ?=
 =?us-ascii?Q?Fx+wnd3+ty6cZWotX6ciRszIQR1vNlJWGANOi1s/w2ByRD0uEzW8acH0fzVr?=
 =?us-ascii?Q?WKZDn++R8R7BOsLoGqr687M7FnPyp3mEvjD/pXpx0+qgHU1VVmeq05BM4Z76?=
 =?us-ascii?Q?+FC1mJuX6FcyQkOQzfot/2TsX7lYg4jZrbqzaI63X6GJiclPUakn6SFgD140?=
 =?us-ascii?Q?oyt56HLTEjJsZPszWALe/i4iNChzYOonGs3vIvnwmHzBSwnqCecYFcZb2kP4?=
 =?us-ascii?Q?eYfae9jP+GpohlpAj3ncdQ8TkUhKXgAqI+cIlovyM0G1jvQjcOLYBwwDAGVf?=
 =?us-ascii?Q?XVnZQX8uTuZwvqIBEFxjtOW1McR4V5NsaQEjKnXsfsimNU+8xmIrM561w6mp?=
 =?us-ascii?Q?CnP4v0zqbDJPdcpMWxzflAZpeIpeTuAa+S1q6gHSQQSG7GQmnzP1SRstyBcX?=
 =?us-ascii?Q?8Ggb70Vs2i2GKhCnjFkW1MY71sxE0p/DeJqUoNEPrwPG0/RoM0zir16gwsg+?=
 =?us-ascii?Q?nTKHvn6VYG+cVNveDChy5JZIDGOLahFUM/z2bREHC21QBqxxaMuIx6p9NuFU?=
 =?us-ascii?Q?+Kn+frYV5LyYPqqIDxJqTfCtSYUujUYgi07X9YEbhnGOcTS3ONk2ZkIUYQ9l?=
 =?us-ascii?Q?S+i4aFEWDLCgq4ZPFIjCqS0wcXLK+kV7k6j5vt5BGWnWw5Iex2jcotEPHIyf?=
 =?us-ascii?Q?0NQU++T/tjnTUsU0hEDHq/koMjRcKVPDvqEo6mThRMqUoy8ry2sVJK6YWy/p?=
 =?us-ascii?Q?YzqFWrSqsloCh6i/SRvVIsPgoVVaoXVtAzc9cEA+P8ZEJ6dza5Nn6q2qkrJJ?=
 =?us-ascii?Q?nCWA0/clnGps0M7d+qYRe9okPRaSOdPl1TOK/QC4?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a5689f8-7ddb-403f-5647-08dadc1fa397
X-MS-Exchange-CrossTenant-AuthSource: DM5PR12MB2504.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2022 09:02:47.5234
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lYI79ZUuhnx7I/R25mNGGTwW6TjgC4FIb7wYfd5QbeFXj6X5atTxqIUdkc3yRJ/25Hj24dDRv3AS/YizLfejHw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6957
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Dec 08, 2022 at 07:18:45PM +0800, Yuan, Perry wrote:
> From: Perry Yuan <Perry.Yuan@amd.com>
> 
> Adds online and offline driver callback support to allow cpu cores go
> offline and help to restore the previous working states when core goes
> back online later for EPP driver mode.
> 
> Signed-off-by: Perry Yuan <Perry.Yuan@amd.com>
> ---
>  drivers/cpufreq/amd-pstate.c | 89 ++++++++++++++++++++++++++++++++++++
>  include/linux/amd-pstate.h   |  1 +
>  2 files changed, 90 insertions(+)
> 
> diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
> index 0a521be1be8a..412accab7bda 100644
> --- a/drivers/cpufreq/amd-pstate.c
> +++ b/drivers/cpufreq/amd-pstate.c
> @@ -1186,6 +1186,93 @@ static int amd_pstate_epp_set_policy(struct cpufreq_policy *policy)
>  	return 0;
>  }
>  
> +static void amd_pstate_epp_reenable(struct amd_cpudata *cpudata)
> +{
> +	struct cppc_perf_ctrls perf_ctrls;
> +	u64 value, max_perf;
> +	int ret;
> +
> +	ret = amd_pstate_enable(true);
> +	if (ret)
> +		pr_err("failed to enable amd pstate during resume, return %d\n", ret);
> +
> +	value = READ_ONCE(cpudata->cppc_req_cached);
> +	max_perf = READ_ONCE(cpudata->highest_perf);
> +
> +	if (boot_cpu_has(X86_FEATURE_CPPC)) {
> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value);
> +	} else {
> +		perf_ctrls.max_perf = max_perf;
> +		perf_ctrls.energy_perf = AMD_CPPC_ENERGY_PERF_PREF(cpudata->epp_cached);
> +		cppc_set_perf(cpudata->cpu, &perf_ctrls);
> +	}
> +}
> +
> +static int amd_pstate_epp_cpu_online(struct cpufreq_policy *policy)
> +{
> +	struct amd_cpudata *cpudata = all_cpu_data[policy->cpu];
> +
> +	pr_debug("AMD CPU Core %d going online\n", cpudata->cpu);
> +
> +	if (cppc_active) {
> +		amd_pstate_epp_reenable(cpudata);
> +		cpudata->suspended = false;
> +	}
> +
> +	return 0;
> +}
> +
> +static void amd_pstate_epp_offline(struct cpufreq_policy *policy)
> +{
> +	struct amd_cpudata *cpudata = all_cpu_data[policy->cpu];
> +	struct cppc_perf_ctrls perf_ctrls;
> +	int min_perf;
> +	u64 value;
> +
> +	min_perf = READ_ONCE(cpudata->lowest_perf);
> +	value = READ_ONCE(cpudata->cppc_req_cached);
> +
> +	mutex_lock(&amd_pstate_limits_lock);
> +	if (boot_cpu_has(X86_FEATURE_CPPC)) {
> +		cpudata->epp_policy = CPUFREQ_POLICY_UNKNOWN;
> +
> +		/* Set max perf same as min perf */
> +		value &= ~AMD_CPPC_MAX_PERF(~0L);
> +		value |= AMD_CPPC_MAX_PERF(min_perf);
> +		value &= ~AMD_CPPC_MIN_PERF(~0L);
> +		value |= AMD_CPPC_MIN_PERF(min_perf);
> +		wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value);
> +	} else {
> +		perf_ctrls.desired_perf = 0;
> +		perf_ctrls.max_perf = min_perf;
> +		perf_ctrls.energy_perf = AMD_CPPC_ENERGY_PERF_PREF(HWP_EPP_POWERSAVE);
> +		cppc_set_perf(cpudata->cpu, &perf_ctrls);
> +	}

Could you double confirm whether these registers will be cleared or
modified while the CPU cores enter/exit online/offline? I remember Joe gave
a test before, the register value will be saved even it gets back to
idle/offline.

Thanks,
Ray

> +	mutex_unlock(&amd_pstate_limits_lock);
> +}
> +
> +static int amd_pstate_cpu_offline(struct cpufreq_policy *policy)
> +{
> +	struct amd_cpudata *cpudata = all_cpu_data[policy->cpu];
> +
> +	pr_debug("AMD CPU Core %d going offline\n", cpudata->cpu);
> +
> +	if (cpudata->suspended)
> +		return 0;
> +
> +	if (cppc_active)
> +		amd_pstate_epp_offline(policy);
> +
> +	return 0;
> +}
> +
> +static int amd_pstate_epp_cpu_offline(struct cpufreq_policy *policy)
> +{
> +	amd_pstate_clear_update_util_hook(policy->cpu);
> +
> +	return amd_pstate_cpu_offline(policy);
> +}
> +
>  static void amd_pstate_verify_cpu_policy(struct amd_cpudata *cpudata,
>  					   struct cpufreq_policy_data *policy)
>  {
> @@ -1220,6 +1307,8 @@ static struct cpufreq_driver amd_pstate_epp_driver = {
>  	.init		= amd_pstate_epp_cpu_init,
>  	.exit		= amd_pstate_epp_cpu_exit,
>  	.update_limits	= amd_pstate_epp_update_limits,
> +	.offline	= amd_pstate_epp_cpu_offline,
> +	.online		= amd_pstate_epp_cpu_online,
>  	.name		= "amd_pstate_epp",
>  	.attr		= amd_pstate_epp_attr,
>  };
> diff --git a/include/linux/amd-pstate.h b/include/linux/amd-pstate.h
> index 888af62040f1..3dd26a3d104c 100644
> --- a/include/linux/amd-pstate.h
> +++ b/include/linux/amd-pstate.h
> @@ -99,6 +99,7 @@ struct amd_cpudata {
>  	u64	cppc_cap1_cached;
>  	struct	update_util_data update_util;
>  	struct	amd_aperf_mperf sample;
> +	bool suspended;
>  };
>  
>  /**
> -- 
> 2.34.1
>