Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Fri, 15 Oct 2021 10:04:01 -0400
From:   Steven Rostedt <rostedt@goodmis.org>
To:     Norbert <nbrtt01@gmail.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>
Subject: Re: Performance regression: thread wakeup time (latency) increased
 up to 3x
Message-ID: <20211015100401.45833169@gandalf.local.home>
In-Reply-To: <5fe0ffa5-f2db-ca79-5a10-305310066ff9@gmail.com>
References: <035c23b4-118e-6a35-36d9-1b11e3d679f8@gmail.com>
        <YWlBUVDy9gOMiXls@hirez.programming.kicks-ass.net>
        <5fe0ffa5-f2db-ca79-5a10-305310066ff9@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Fri, 15 Oct 2021 02:36:03 -0700
Norbert <nbrtt01@gmail.com> wrote:

> On 10/15/21 01:52, Peter Zijlstra wrote:
> > On Fri, Oct 15, 2021 at 12:43:45AM -0700, Norbert wrote:  
> >> Performance regression: thread wakeup time (latency) increased up to 3x.
> >>
> >> Happened between 5.13.8 and 5.14.0. Still happening at least on 5.14.11.  
> > 
> > Could you git-bisect this?
> >   
> 
> So far I haven't built a kernel yet, I'm quite new to Linux in that way, 
> so it may take me some time to figure it all out, but yes.

If you do test a kernel you build yourself, you may want to use
"localmodconfig". Which will look at all the modules that are loaded on the
local system and turn off the configs for modules that are not loaded. This
will save a lot of time compiling the kernel. Especially if your basing off
of a distro config.

  $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  $ cd linux
  $ cp /boot/config-[whatever your kernel is] .config
  $ yes '' | make localmodconfig

Then build your kernel.

  $ make -j10
  $ sudo make modules_install install

> 
> (By the way, of course I meant that throughput *de*creases, not increases.)

of course ;-)

-- Steve