Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <8691a8ec-410d-afe8-f468-eefe698c6751@gmail.com>
Date:   Fri, 15 Oct 2021 21:08:58 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.1.0
Subject: Re: Performance regression: thread wakeup time (latency) increased up
 to 3x
Content-Language: en-US
To:     Steven Rostedt <rostedt@goodmis.org>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Yunfeng Ye <yeyunfeng@huawei.com>
References: <035c23b4-118e-6a35-36d9-1b11e3d679f8@gmail.com>
 <YWlBUVDy9gOMiXls@hirez.programming.kicks-ass.net>
 <5fe0ffa5-f2db-ca79-5a10-305310066ff9@gmail.com>
 <20211015100401.45833169@gandalf.local.home>
From:   Norbert <nbrtt01@gmail.com>
In-Reply-To: <20211015100401.45833169@gandalf.local.home>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 10/15/21 07:04, Steven Rostedt wrote:
> On Fri, 15 Oct 2021 02:36:03 -0700
> Norbert <nbrtt01@gmail.com> wrote:
> 
>> On 10/15/21 01:52, Peter Zijlstra wrote:
>>> On Fri, Oct 15, 2021 at 12:43:45AM -0700, Norbert wrote:
>>>> Performance regression: thread wakeup time (latency) increased up to 3x.
>>>>
>>>> Happened between 5.13.8 and 5.14.0. Still happening at least on 5.14.11.
>>>
>>> Could you git-bisect this?
>>>    
>>
>> So far I haven't built a kernel yet, I'm quite new to Linux in that way,
>> so it may take me some time to figure it all out, but yes.
> 
> If you do test a kernel you build yourself, you may want to use
> "localmodconfig". Which will look at all the modules that are loaded on the
> local system and turn off the configs for modules that are not loaded. This
> will save a lot of time compiling the kernel. Especially if your basing off
> of a distro config.
> 
>    $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>    $ cd linux
>    $ cp /boot/config-[whatever your kernel is] .config
>    $ yes '' | make localmodconfig
> 
> Then build your kernel.
> 
>    $ make -j10
>    $ sudo make modules_install install
> 
>>
>> (By the way, of course I meant that throughput *de*creases, not increases.)
> 
> of course ;-)
> 
> -- Steve
> 

Thank you, localmodconfig is an enormous timesaver. Once I got some 
local build problems/modules out of the way, this method worked very well.

So git-bisect finally identified the following commit.
The performance difference came in a single step. Times were consistent 
with my first post either the slow time or the fast time,
as far as I could tell during the bisection.

It is a bit unfortunate that this comes from an attempt to reduce OS noise.

-----------------------------------------------------
commit a5183862e76fdc25f36b39c2489b816a5c66e2e5
Author: Yunfeng Ye <yeyunfeng@huawei.com>
Date:   Thu May 13 01:29:16 2021 +0200

     tick/nohz: Conditionally restart tick on idle exit

     In nohz_full mode, switching from idle to a task will unconditionally
     issue a tick restart. If the task is alone in the runqueue or is the
     highest priority, the tick will fire once then eventually stop. But 
that
     alone is still undesired noise.

     Therefore, only restart the tick on idle exit when it's strictly
     necessary.

     Signed-off-by: Yunfeng Ye <yeyunfeng@huawei.com>
     Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
     Signed-off-by: Ingo Molnar <mingo@kernel.org>
     Acked-by: Peter Zijlstra <peterz@infradead.org>
     Link: 
https://lore.kernel.org/r/20210512232924.150322-3-frederic@kernel.org
-----------------------------------------------------

Is there anything else to do to complete this report?