Received-SPF: pass (google.com: domain of linux-kernel+bounces-159658-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <63314d92-9226-4005-a166-37c8648c5351@linux.ibm.com>
Date: Fri, 26 Apr 2024 13:16:10 +0530
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 00/30] PREEMPT_AUTO: support lazy rescheduling
To: Linus Torvalds <torvalds@linux-foundation.org>,
        Ankur Arora <ankur.a.arora@oracle.com>
Cc: Thomas Gleixner <tglx@linutronix.de>, peterz@infradead.org,
        paulmck@kernel.org, akpm@linux-foundation.org, luto@kernel.org,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        mingo@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org,
        willy@infradead.org, mgorman@suse.de, jpoimboe@kernel.org,
        mark.rutland@arm.com, jgross@suse.com, andrew.cooper3@citrix.com,
        bristot@kernel.org, mathieu.desnoyers@efficios.com,
        geert@linux-m68k.org, glaubitz@physik.fu-berlin.de,
        anton.ivanov@cambridgegreys.com, mattst88@gmail.com,
        krypton@ulrich-teichert.org, rostedt@goodmis.org,
        David.Laight@aculab.com, richard@nod.at, mjguzik@gmail.com,
        jon.grimm@amd.com, bharata@amd.com, raghavendra.kt@amd.com,
        boris.ostrovsky@oracle.com, konrad.wilk@oracle.com,
        LKML <linux-kernel@vger.kernel.org>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Nicholas Piggin <npiggin@gmail.com>
References: <20240213055554.1802415-1-ankur.a.arora@oracle.com>
 <74ffd154-5d92-4303-9977-6ddc9accdf01@linux.ibm.com>
 <CAHk-=whrTE30xzuq4mcXidT4M0z_HRYJW9i8M9gD9VG7d29Qpg@mail.gmail.com>
From: Shrikanth Hegde <sshegde@linux.ibm.com>
Content-Language: en-US
In-Reply-To: <CAHk-=whrTE30xzuq4mcXidT4M0z_HRYJW9i8M9gD9VG7d29Qpg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit


On 4/23/24 9:43 PM, Linus Torvalds wrote:
> On Tue, 23 Apr 2024 at 08:23, Shrikanth Hegde <sshegde@linux.ibm.com> wrote:
>>
>>
>> Are these the only arch bits that need to be defined? am I missing something very
>> basic here? will try to debug this further. Any inputs?
> 
> I don't think powerpc uses the generic *_exit_to_user_mode() helper
> functions, so you'll need to also add that logic to the low-level
> powerpc code.
> 
> IOW, on x86, with this patch series, patch 06/30 did this:
> 
> -               if (ti_work & _TIF_NEED_RESCHED)
> +               if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY))
>                         schedule();
> 
> in kernel/entry/common.c exit_to_user_mode_loop().
> 
> But that works on x86 because it uses the irqentry_exit_to_user_mode().
> 
> On PowerPC, I think you need to at least fix up
> 
>     interrupt_exit_user_prepare_main()
> 
> similarly (and any other paths like that - I used to know the powerpc
> code, but that was long long LOOONG ago).
> 
>                 Linus

Thank you Linus for the pointers. That indeed did the trick. 

diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index eca293794a1e..f0f38bf5cea9 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -185,7 +185,7 @@ interrupt_exit_user_prepare_main(unsigned long ret, struct pt_regs *regs)
        ti_flags = read_thread_flags();
        while (unlikely(ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) {
                local_irq_enable();
-               if (ti_flags & _TIF_NEED_RESCHED) {
+               if (ti_flags & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY) ) {
                        schedule();
                } else {


By adding LAZY checks in interrupt_exit_user_prepare_main, softlockup is no longer seen and 
hackbench results are more or less same on smaller system(96CPUS). However, I still see 20-50% 
regression on the larger system(320 CPUS). I will continue to debug why.