Received-SPF: pass (google.com: domain of linux-kernel+bounces-134658-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Sun, 7 Apr 2024 15:52:52 -0700
From: Bjorn Andersson <andersson@kernel.org>
To: Linux regressions mailing list <regressions@lists.linux.dev>
Cc: Tejun Heo <tj@kernel.org>, LKML <linux-kernel@vger.kernel.org>
Subject: Re: Bug 218665 - nohz_full=0 prevents kernel from booting
Message-ID: <enqg6mcuhvff7gujjbapdiclicl3z6f2vnggcsg65pexipyr3o@4men5fhyt3vb>
References: <5be248c6-cdda-4d2e-8fae-30fc2cc124c0@leemhuis.info>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5be248c6-cdda-4d2e-8fae-30fc2cc124c0@leemhuis.info>

On Tue, Apr 02, 2024 at 10:17:16AM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
> Hi, Thorsten here, the Linux kernel's regression tracker.
> 
> I noticed a regression report in bugzilla.kernel.org. As many (most?)
> kernel developers don't keep an eye on it, I decided to forward it by mail.
> 
> Tejun, apparently it's cause by a change of yours.
> 
> Note, you have to use bugzilla to reach the reporter, as I sadly[1] can
> not CCed them in mails like this.
> 
> Quoting from https://bugzilla.kernel.org/show_bug.cgi?id=218665 :
> 
> > booting the current kernel (6.9.0-rc1, master/712e1425) on x86_64
> > with nohz_full=0 cause a page fault and prevents the kernel from
> > booting.
> > 
> > Steps to reproduce:
> > - make defconfig
> > - set CONFIG_NO_HZ_FULL=y
> > - set CONFIG_SUSPEND=n and CONFIG_HIBERNATION=n (to get CONFIG_PM_SLEEP_SMP=n)
> > - make
> > - qemu-system-x86_64 -nographic -cpu qemu64 -smp cores=2 -m 1024 -kernel arch/x86/boot/bzImage -append "earlyprintk=ttyS0 console=ttyS0 root=/dev/dummy rootwait nohz_full=0"
> > 
> > I have attached the output of a failed nohz_full=0 boot as
> > nohz_full_0.txt and - for reference - the output of a nohz_full=1
> > boot as nohz_full_1.txt.
> > 
> > Interestingly enough, using the deprecated isolcpus parameter to
> > enable NO_HZ for cpu0 works. I've attached the output as
> > isolcpus_nohz_0.txt.
> > 
> > Bisecting showed 5797b1c18919cd9c289ded7954383e499f729ce0 as first bad commit.
> 
> See the ticket for more details.
> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
> 
> [1] because bugzilla.kernel.org tells users upon registration their
> "email address will never be displayed to logged out users"
> 
> #regzbot introduced: 5797b1c18919cd9c289ded7954383e499f729ce0
> #regzbot from: Friedrich Oslage
> #regzbot duplicate https://bugzilla.kernel.org/show_bug.cgi?id=218665
> #regzbot title: workqueue: nohz_full=0 prevents booting
> #regzbot ignore-activity

In addition to this report, I have finally bisected another regression
to the same commit:

I start neovim, send SIGSTOP (i.e. ^Z) to it, start another neovim
instance and upon sending SIGSTOP to that instance all of userspace
locks up - 100% reproducible.

The kernel seems to continue to operate, and tapping the power button
dislodge the lockup and I get a clean shutdown.

This is seen on multiple Arm64 (Qualcomm) machines with upstream
defconfig since commit '5797b1c18919 ("workqueue: Implement system-wide
nr_active enforcement for unbound workqueues")'.

Regards,
Bjorn