Received-SPF: pass (google.com: domain of linux-kernel+bounces-201534-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
References: <20240430183730.561960-1-ankur.a.arora@oracle.com>
 <20240430183730.561960-9-ankur.a.arora@oracle.com>
 <41a184a3f5695061487e86a6da5eb87c140dbe3c.camel@amazon.com>
User-agent: mu4e 1.4.10; emacs 27.2
From: Ankur Arora <ankur.a.arora@oracle.com>
To: "Okanovic, Haris" <harisokn@amazon.com>
Cc: "linux-arm-kernel@lists.infradead.org"
 <linux-arm-kernel@lists.infradead.org>,
        "kvm@vger.kernel.org"
 <kvm@vger.kernel.org>,
        "linux-pm@vger.kernel.org"
 <linux-pm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org"
 <linux-kernel@vger.kernel.org>,
        "ankur.a.arora@oracle.com"
 <ankur.a.arora@oracle.com>,
        "joao.m.martins@oracle.com"
 <joao.m.martins@oracle.com>,
        "boris.ostrovsky@oracle.com"
 <boris.ostrovsky@oracle.com>,
        "konrad.wilk@oracle.com"
 <konrad.wilk@oracle.com>,
        "wanpengli@tencent.com" <wanpengli@tencent.com>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "catalin.marinas@arm.com"
 <catalin.marinas@arm.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "daniel.lezcano@linaro.org"
 <daniel.lezcano@linaro.org>,
        "lenb@kernel.org" <lenb@kernel.org>, "arnd@arndb.de" <arnd@arndb.de>,
        "will@kernel.org" <will@kernel.org>, "hpa@zytor.com" <hpa@zytor.com>,
        "peterz@infradead.org"
 <peterz@infradead.org>,
        "vkuznets@redhat.com" <vkuznets@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>, "rafael@kernel.org" <rafael@kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "mark.rutland@arm.com"
 <mark.rutland@arm.com>
Subject: Re: [PATCH 8/9] arm64: support cpuidle-haltpoll
In-reply-to: <41a184a3f5695061487e86a6da5eb87c140dbe3c.camel@amazon.com>
Date: Tue, 04 Jun 2024 16:09:06 -0700
Message-ID: <87frts2tm5.fsf@oracle.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?zip6yOScip0xjuMjgm1AXr4EQe+/llVsKFvO7qJL6RVHe3iVp7MIcAyrioEB?=
 =?us-ascii?Q?TmwtJcuIJ7/9t8JaEutoRyUjKAekj7djYh+WPIy9fgxfZLb8InqgI+j11n4v?=
 =?us-ascii?Q?KrIy7GKPtj/YaLrojzhoEqKYouXh5wVQNHIK8//6coBHghnNolZ8WSZGPvqC?=
 =?us-ascii?Q?HrwDfwO/1N76z36iaapOdOgGtJB2+HP4NYMUmrceSZBHDBOdpJk5Te7BLTZo?=
 =?us-ascii?Q?9FLrbn1t3cPAgu8UX7e7NNQqCwrY1sVC/ND8WD7dggvrHPWgTildjwVvfyPz?=
 =?us-ascii?Q?ikzk+dDzXpLu8fuqbp/33WFveviED5Pc/pn+7/OVLhare2+hZey4VNt+9mYq?=
 =?us-ascii?Q?9x1QC7/Az9aDIoQVbZvTRmlng9WeClL9RYAsDuGK+Z4AcGPnSSHV0eApKYdB?=
 =?us-ascii?Q?veEogfvbSszPhs9oOIwdSWqcqT1f3pog5VG+AtgABqIeXkBCPuIUolPT0jsj?=
 =?us-ascii?Q?tTSTHcR8iGK8p8oBHCsNJcFsbgrbOEuvZLwri5DIbITgdKaEgqFnFlNqe/9e?=
 =?us-ascii?Q?7oDevFVpqcO21m1p3ZJSyB0a4wqAaokzNjbcOKaipzgKexC0uECgytYzGXnx?=
 =?us-ascii?Q?H76HaYhmvLWZ4J8oahnCINHLI3yDqDbyenY8WA819GukE8edVMvGMSmVA54k?=
 =?us-ascii?Q?l3BcHYA0H8xgRdSpz0Q7Oft5wbtY/yk3FMWwqWEHiyi1Ut1h5XFPFvvWIErl?=
 =?us-ascii?Q?WAlw9iJiUIvqbpcj2aaJbFbZUvbUSPvw7cRs9xrErB/B9oLv0j9pUMtOmwMk?=
 =?us-ascii?Q?HzITzSmaUMLGw178XUtamN14Sym/sxiU3fuseaYEqJ5oJDebDCy/0FtpWocd?=
 =?us-ascii?Q?6C+bE+aUo3b+BiPqutgBZlrmt0JlobpJD+M2d7lM3SXufHWKRVTpJaonzeYI?=
 =?us-ascii?Q?sluJxGzRuvRT0nPbfS4aRflwVM7WHw/Ov2DlWej+1xu2/LL9V4bNEzrjxUlN?=
 =?us-ascii?Q?q6mO0L4cZ5rm2V3gO+yaQAlHqZ721M0o643BTbuViixg1dyIjoqnIDhVwW2b?=
 =?us-ascii?Q?NTDc9AuVM6igRxXLstEFvse6VKVA1TAh4hO+4WguBts4Hfs3UnwlDfAeawJS?=
 =?us-ascii?Q?Hv3G6KkBSRCOVq9QY+I+xqry+UFCUIGEmIyr+Zdoy6wKPTMfFoNerZGPGG9A?=
 =?us-ascii?Q?AneLHlca6MugKczkMHxqtobcCo9SJK/otpq+9hBPE26Vr1OrqVG6KIAYRCLX?=
 =?us-ascii?Q?a2VqZFehZhdNDdJ7jWaB8bfP/dYPOTKMeKEZ2AsMCjxFwL71pyR6o8i8BAR9?=
 =?us-ascii?Q?YX61LF7MxLd/69h2mQJK+z1nPtf2U4hiLeMbaguogsEYknvV3Cx2FAmHLVDe?=
 =?us-ascii?Q?I9V6HqwbUFjWozSAuri+Nhbab49BgjDFeRg/Qs41/r0jop6ZnMpIeYxIKCYV?=
 =?us-ascii?Q?MAm4uv25bUBjsOckpn8NOgXfDgvol7ooVCRFK0vQQ3vCxe43nmntFb4QmKV0?=
 =?us-ascii?Q?fP1oofODLOqTV+2NjuSCAmV318FK/WicHG1PTMukkfxAsR0Rcz4sJYUuy6gc?=
 =?us-ascii?Q?SGq2joDtVicvO0aBnVTj+HiUP2D5qEPaHdfR5fOBHRrdZyWcEvbZ+OLrao4+?=
 =?us-ascii?Q?pnjaNtN0gNCezxqG6844YInIeU6DF0ZV59YoFHL+jqRCTWwyOeaBl4ulxsVI?=
 =?us-ascii?Q?pw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	zODW3NIvZuVpguSR/FgtGjHX0oflqBXcSbh7K8NJpGaDmsDaIr3jWW2KrS42iSbqFGEEHh2SdrOvTDhFQsZVYQjThU+CuZqUardBlbUg668NVbw/O9lDtHeAYxUFNWwqUTI9LKBq/A6AAZ+EzjP3Z61lgMVGC1/m/JmDGqqzi6lOfBIfoERS3RXvkmADZ91pG+n79jPxt9hnAvnNJoSZ6awIGFI5eYqUfdXfFtjUbA8SGdIBsZJ0AZUA96WbuRhEESiOEr9Zcq/gw/6/hDMu3BmtBKU4E65mzH9Wl59mIPrVyXljIeodQT5u/Atm+2NoKX4P5zp/k1a0vKvJJrs0SCWggSNYcLgLvhMh0UxFics80yiLJgbmLkPw73sxcZn6AWM9cH2IMARUv5Rr4cb+1nB+Z4KX+BXUGcZBobcEKooKlOYi0QK6/WmLiLrbANJ2bXDGIMin6nQst/w88fjQliqmUPy2qPsUISs2dId39RukT59oyRyRx7yeppRKlLxc77aAIptw52GpO4om9k+29zkMPdfOzJR1YthXdqKXfug7uSsVs3S0DimrJvnHWqzXG51wK3rCk2AwtBwZN0ChyVrdSGxkrC5nqnng2re0L7I=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 983193c4-6792-4936-0354-08dc84eb57ad
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jun 2024 23:09:10.5991
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /BDy73nQDNMwN8b5UehKXNiYMzC47ijUcLTfN6wzHj8uIt4mF5ckEP6unT04H7fBOvvl7b/t1YT/WEGWPN0uLgvHPb0YcedJo1Xp4H23LL4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6688
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16
 definitions=2024-06-04_11,2024-06-04_02,2024-05-17_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 malwarescore=0
 suspectscore=0 spamscore=0 mlxlogscore=999 adultscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2405010000
 definitions=main-2406040187
X-Proofpoint-ORIG-GUID: MiMBWa_LUNGJrEE9kyDtXzboVSP5YC-z
X-Proofpoint-GUID: MiMBWa_LUNGJrEE9kyDtXzboVSP5YC-z


Okanovic, Haris <harisokn@amazon.com> writes:

> On Tue, 2024-04-30 at 11:37 -0700, Ankur Arora wrote:
>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>
>>
>>
>> Add architectural support for the cpuidle-haltpoll driver by defining
>> arch_haltpoll_*(). Also select ARCH_HAS_OPTIMIZED_POLL since we have
>> an optimized polling mechanism via smp_cond_load*().
>>
>> Add the configuration option, ARCH_CPUIDLE_HALTPOLL to allow
>> cpuidle-haltpoll to be selected.
>>
>> Note that we limit cpuidle-haltpoll support to when the event-stream is
>> available. This is necessary because polling via smp_cond_load_relaxed()
>> uses WFE to wait for a store which might not happen for an prolonged
>> period of time. So, ensure the event-stream is around to provide a
>> terminating condition.
>>
>> Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>> ---
>>  arch/arm64/Kconfig                        | 10 ++++++++++
>>  arch/arm64/include/asm/cpuidle_haltpoll.h | 21 +++++++++++++++++++++
>>  2 files changed, 31 insertions(+)
>>  create mode 100644 arch/arm64/include/asm/cpuidle_haltpoll.h
>>
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 7b11c98b3e84..6f2df162b10e 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -34,6 +34,7 @@ config ARM64
>>         select ARCH_HAS_MEMBARRIER_SYNC_CORE
>>         select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
>>         select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
>> +       select ARCH_HAS_OPTIMIZED_POLL
>>         select ARCH_HAS_PTE_DEVMAP
>>         select ARCH_HAS_PTE_SPECIAL
>>         select ARCH_HAS_HW_PTE_YOUNG
>> @@ -2331,6 +2332,15 @@ config ARCH_HIBERNATION_HEADER
>>  config ARCH_SUSPEND_POSSIBLE
>>         def_bool y
>>
>> +config ARCH_CPUIDLE_HALTPOLL
>> +       bool "Enable selection of the cpuidle-haltpoll driver"
>> +       default n
>> +       help
>> +         cpuidle-haltpoll allows for adaptive polling based on
>> +         current load before entering the idle state.
>> +
>> +         Some virtualized workloads benefit from using it.
>> +
>>  endmenu # "Power management options"
>>
>>  menu "CPU Power Management"
>> diff --git a/arch/arm64/include/asm/cpuidle_haltpoll.h b/arch/arm64/include/asm/cpuidle_haltpoll.h
>> new file mode 100644
>> index 000000000000..a79bdec7f516
>> --- /dev/null
>> +++ b/arch/arm64/include/asm/cpuidle_haltpoll.h
>> @@ -0,0 +1,21 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _ASM_HALTPOLL_H
>> +#define _ASM_HALTPOLL_H
>> +
>> +static inline void arch_haltpoll_enable(unsigned int cpu)
>> +{
>> +}
>> +
>> +static inline void arch_haltpoll_disable(unsigned int cpu)
>> +{
>> +}
>> +
>> +static inline bool arch_haltpoll_supported(void)
>> +{
>> +       /*
>> +        * Ensure the event stream is available to provide a terminating
>> +        * condition to the WFE in the poll loop.
>> +        */
>> +       return arch_timer_evtstrm_available();
>
> Note this fails build when CONFIG_HALTPOLL_CPUIDLE=m (module):
>
> ERROR: modpost: "arch_cpu_idle" [drivers/cpuidle/cpuidle-haltpoll.ko]
> undefined!
> ERROR: modpost: "arch_timer_evtstrm_available"
> [drivers/cpuidle/cpuidle-haltpoll.ko] undefined!
> make[2]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1
> make[1]: *** [/home/ubuntu/linux/Makefile:1886: modpost] Error 2
> make: *** [Makefile:240: __sub-make] Error 2

Thanks for trying it out. Missed that.

> You could add EXPORT_SYMBOL_*()'s on the above helpers or restrict
> HALTPOLL_CPUIDLE module to built-in (remove "tristate" Kconfig).

Yeah AFAICT this is the only cpuidle driver providing the module
option. Unfortunately can't remove the tristate thing. People might
already be using it as a module on x86.

I think the arch_cpu_idle() makes sense to export. For
arch_timer_evtstrm_available(), eventually the arch_haltpoll_*()
in any case need to move out of a header file. I'll just do that
now.

> Otherwise, everything worked for me when built-in (=y) atop 6.10.0
> (4a4be1a). I see similar performance gains in `perf bench` on AWS
> Graviton3 c7g.16xlarge.

Excellent. Thanks for checking.

--
ankur