Received-SPF: pass (google.com: domain of linux-kernel+bounces-160592-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
 <74ffd154-5d92-4303-9977-6ddc9accdf01@linux.ibm.com>
 <CAHk-=whrTE30xzuq4mcXidT4M0z_HRYJW9i8M9gD9VG7d29Qpg@mail.gmail.com>
 <63314d92-9226-4005-a166-37c8648c5351@linux.ibm.com>
User-agent: mu4e 1.4.10; emacs 27.2
From: Ankur Arora <ankur.a.arora@oracle.com>
To: Shrikanth Hegde <sshegde@linux.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
        Ankur Arora
 <ankur.a.arora@oracle.com>,
        Thomas Gleixner <tglx@linutronix.de>, peterz@infradead.org,
        paulmck@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
        willy@infradead.org, mgorman@suse.de, jpoimboe@kernel.org,
        mark.rutland@arm.com, jgross@suse.com, andrew.cooper3@citrix.com,
        bristot@kernel.org, mathieu.desnoyers@efficios.com,
        geert@linux-m68k.org, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@aculab.com, richard@nod.at, mjguzik@gmail.com,
        jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        LKML
 <linux-kernel@vger.kernel.org>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Nicholas Piggin <npiggin@gmail.com>
Subject: Re: [PATCH 00/30] PREEMPT_AUTO: support lazy rescheduling
In-reply-to: <63314d92-9226-4005-a166-37c8648c5351@linux.ibm.com>
Date: Fri, 26 Apr 2024 12:00:00 -0700
Message-ID: <87r0esc5lb.fsf@oracle.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?61dPWyP9hi5nD7KlbVxdVCnL2cobgi91aEwwrYwOUCBMYCdmEY7Pk7dIpeSY?=
 =?us-ascii?Q?I9t4tqhvw93jUjN7tYp72gCQzc3VJ0/ryGS/WIUlJNdl+nQrpmiV7VZrN1Un?=
 =?us-ascii?Q?G25JfZEwW+r3FeLMHv+jUnkp2ssvYNu1nVr/Tf3/YJmoWXaeKYxZdlitjVAg?=
 =?us-ascii?Q?/ZAzBlH2MgI9At9j2Op/bgYge0wKIiN6ZW+bJAX9BLau6hnPxTzF0M3pHtYh?=
 =?us-ascii?Q?pD8MiKKp3n5XmdnARde6u24NId/rbZkmTR/K29Xb/NZ8akHPjNnDpekrBWsu?=
 =?us-ascii?Q?42TIGChWA5zrBQjSKEK16rgREkty56EP2+PeQu3DwnTjxgi6JCcWmpiyfLd5?=
 =?us-ascii?Q?4tSh6qeZ+/VwPP0U26LYXbETR3vFmz8pnQeS1n2EqvCDoLfH1r6UkzVA4ekf?=
 =?us-ascii?Q?xWFY7TLHyN1CEltdPSmePJJf8lI+eEiBxpgkjjjbTE3Rbu3+9rUBFSjewzJh?=
 =?us-ascii?Q?jOM5L6OWKoRDX3ZCOwB0VQgG+0IBv89SUBmFsv/ZEFt+F6JTB+zOyzs/ufqV?=
 =?us-ascii?Q?+3M1N9VbLAB753TF5LRTb0QMSCt6sSp3k+1sysq9cbawTIzKlIg6Voib+t29?=
 =?us-ascii?Q?kuxz9Eetgo0KTx9NMyG7YEGFXsSZG9X/mQ2+CX/+QUy8vlxvc+qwmsEru5vV?=
 =?us-ascii?Q?QXgqU+nYsy8u73JO5Czt2TIbIUQEwsKjsP9QuhIFt3KedHBc0O+dac96hiwr?=
 =?us-ascii?Q?YGzdeGbHi8lNEc/cQv+UBFFdFvPsLhltBTSLNG5DV9kXyqTWTI7CqWae0Epz?=
 =?us-ascii?Q?UIYE2V5zuxGY1nrVpmtfzMPTPImJZAxhXfaYg8hqI5yK08lH5lLrgIVrf6hL?=
 =?us-ascii?Q?Kx9lGEpgBZGiMNzSsydXjO5+O3UI8ljCK0BhwGSjhZy4NUtOnzb94DUNfFI+?=
 =?us-ascii?Q?lpJDXME8wXEJMd40VN9b72bfCViek/NGSPDkVpZOyjUiNSn57khLnueyxNOA?=
 =?us-ascii?Q?G+m3FAQMkFxgeBlg8LxpDUIjKjSMiiptBDgPbeaLXSiPXfw43McEmu3W8i++?=
 =?us-ascii?Q?m7zue+HOfEYa7iBRW7sI6MByi3k2HIL8Yq1tJdKJU8ts5lP6SZrqjIynJLvT?=
 =?us-ascii?Q?HlCtXnecC4kUSnOCiXTlh69xBkCckW8vRgGwCpa5FAH8dSLKwHiHB4bNbAZ8?=
 =?us-ascii?Q?srKb7ECKjw4a6msnbz9Agp8GftUHdEDjFGSHzCnRs4ZNFcVgwk4+IFqBbcyx?=
 =?us-ascii?Q?TPOsEStc0h033FdCxkgIPbrOy8N54/xTgi6PBkm+qN0Nu8U3KuUp6XRV/W8a?=
 =?us-ascii?Q?neX6PTIjK0JZTCymRALQ8OXFsz6Ls4bI/5Thd1h075f+CxcqeK5s3qPRojgq?=
 =?us-ascii?Q?XvLjxdlXrybMjuYdKKPyxf/qkgDweElQOXdrDF9S7EuDfe8j92LcwwRKQAI5?=
 =?us-ascii?Q?8dwEa9uVym56tCcIhfFK7emERvTOSrTFDqlyWUkMM8P7HMcOpw+X9aS2p/Ro?=
 =?us-ascii?Q?DWsgqUfOH9fTUHb5hFq97JadOljk/Th6ISgkyimEQTnMug+Dm/f4Rip8S2ZC?=
 =?us-ascii?Q?ry1eO6feM0lZ+MDOYJETSBC31/4JnKXvYkLglQ6G3prPPCV72CZVOprlVMt4?=
 =?us-ascii?Q?Ff5U9zGO/izwMII6x3IBhoehNf07h7FNbko9Wwh2QJ3h5Fx4nhRD9IDLcU24?=
 =?us-ascii?Q?yw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	QjmYRpA/wdSpyPFS7rQuz1l9ng5ddzoiOBT0UaCIQpveV+ewPPVuFctzEYoA60Hvr1/CnhaDTyHRghlmEMgly5CGaUS9zcFUNoJw7gvvTBm7RDCN4rCl9uD6e79s+U/KN8jlsXFcSOTv+3UH6TYUPm+vEVPlaxTwKBSABukBXNu2wM/RUEc+wOPJMl8hZ2dFW/oO73jdlBjQJQZLzz7tldp5xdcAsYWBtJEO+lyPn0DcyGNr4iBfDAQVc54uXCCeut2TmKdHhVmyVmx3WJOljlKNj6ser3tZ1JsuFFN4YfVZgfLhwHL5Eo+jQG+XRB53K59MJ2HQYR3lYjM0HAfcZd8x5ZZyneGvlfUMx3u4e2QP0xpATEhlIElmg6RmNyr9dnQxR5J0dei0x010p5QXrz1mfH/Zd6f9L89f7FrJ3qTlLViMf/kTaB1zPxuhTMjG5RQU0r9J2n+GBxJ/2p536B5Cxxnk/MW/ctjkBHg2RiDm1N6o9/V0XdPKAfRsyISi07Y2gxhPs1ayasMQbcKvPTBEZ4Hdpwfknu62PcSw3KPhCiUD+ly4rdsJOaXE1k1oWpJZfUhvr1460WAyaLQTktQJikUmC7oV3w3qwd+uzSY=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9c717ee1-c642-420e-c805-08dc662313f8
X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Apr 2024 19:00:02.6671
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iwGyia7VcPF8OypgcOvhAu7WtHjxtYGYFsH5PwdH9ndoZJSyfOQz1CTuC7wvTQcPgOZ2au+VmZtMosvS/1apY+ocz6pwm4R89HQ/dQRA+rc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB7604
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-04-26_16,2024-04-26_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=887
 mlxscore=0 phishscore=0 spamscore=0 adultscore=0 bulkscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2404260130
X-Proofpoint-GUID: tbvHfBAUQYkv0VElvbMuB4qAYT2L2kjf
X-Proofpoint-ORIG-GUID: tbvHfBAUQYkv0VElvbMuB4qAYT2L2kjf


Shrikanth Hegde <sshegde@linux.ibm.com> writes:

> On 4/23/24 9:43 PM, Linus Torvalds wrote:
>> On Tue, 23 Apr 2024 at 08:23, Shrikanth Hegde <sshegde@linux.ibm.com> wrote:
>>>
>>>
>>> Are these the only arch bits that need to be defined? am I missing something very
>>> basic here? will try to debug this further. Any inputs?
>>
>> I don't think powerpc uses the generic *_exit_to_user_mode() helper
>> functions, so you'll need to also add that logic to the low-level
>> powerpc code.
>>
>> IOW, on x86, with this patch series, patch 06/30 did this:
>>
>> -               if (ti_work & _TIF_NEED_RESCHED)
>> +               if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY))
>>                         schedule();
>>
>> in kernel/entry/common.c exit_to_user_mode_loop().
>>
>> But that works on x86 because it uses the irqentry_exit_to_user_mode().
>>
>> On PowerPC, I think you need to at least fix up
>>
>>     interrupt_exit_user_prepare_main()
>>
>> similarly (and any other paths like that - I used to know the powerpc
>> code, but that was long long LOOONG ago).
>>
>>                 Linus
>
> Thank you Linus for the pointers. That indeed did the trick.
>
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index eca293794a1e..f0f38bf5cea9 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -185,7 +185,7 @@ interrupt_exit_user_prepare_main(unsigned long ret, struct pt_regs *regs)
>         ti_flags = read_thread_flags();
>         while (unlikely(ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) {
>                 local_irq_enable();
> -               if (ti_flags & _TIF_NEED_RESCHED) {
> +               if (ti_flags & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY) ) {
>                         schedule();
>                 } else {
>
>
> By adding LAZY checks in interrupt_exit_user_prepare_main, softlockup is no longer seen and
> hackbench results are more or less same on smaller system(96CPUS).

Great. I'm guessing these tests are when running in voluntary preemption
mode (under PREEMPT_AUTO).

If you haven't, could you also try full preemption? There you should see
identical results unless something is horribly wrong.

> However, I still see 20-50%
> regression on the larger system(320 CPUS). I will continue to debug why.

Could you try this patch? This is needed because PREEMPT_AUTO turns on
CONFIG_PREEMPTION, but not CONFIG_PREEMPT:

diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index eca293794a1e..599410050f6b 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -396,7 +396,7 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs)
                /* Returning to a kernel context with local irqs enabled. */
                WARN_ON_ONCE(!(regs->msr & MSR_EE));
 again:
-               if (IS_ENABLED(CONFIG_PREEMPT)) {
+               if (IS_ENABLED(CONFIG_PREEMPTION)) {
                        /* Return to preemptible kernel context */
                        if (unlikely(read_thread_flags() & _TIF_NEED_RESCHED)) {
                                if (preempt_count() == 0)


--
ankur