Received-SPF: pass (google.com: domain of linux-kernel+bounces-100929-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Feedback-ID: iad51458e:Fastmail
Date: Tue, 12 Mar 2024 16:48:05 -0700
From: Boqun Feng <boqun.feng@gmail.com>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Florian Fainelli <f.fainelli@gmail.com>, linux-kernel@vger.kernel.org,
	kernel-team@meta.com, paulmck@kernel.org, mingo@kernel.org,
	tglx@linutronix.de, rcu@vger.kernel.org, joel@joelfernandes.org,
	neeraj.upadhyay@amd.com, urezki@gmail.com,
	qiang.zhang1211@gmail.com, frederic@kernel.org,
	bigeasy@linutronix.de, anna-maria@linutronix.de,
	chenzhongjin@huawei.com, yangjihong1@huawei.com,
	rostedt@goodmis.org
Subject: Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for
 v6.9]
Message-ID: <ZfDptafiK0jns050@boqun-archlinux>
References: <ZetHwrCb0KXE0xFI@tardis>
 <4274be61-60bd-4e1e-9c16-26e6e5e06f65@gmail.com>
 <ZfDEIs63EBIYBJIC@boqun-archlinux>
 <c5f9c640-4c06-495e-9c7e-0c208b914fa7@gmail.com>
 <CAHk-=wgP=9JxdOJ5oYtVO5yM6pFi5+3FPxfCQa4ezpagJuXq3g@mail.gmail.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAHk-=wgP=9JxdOJ5oYtVO5yM6pFi5+3FPxfCQa4ezpagJuXq3g@mail.gmail.com>

On Tue, Mar 12, 2024 at 02:44:14PM -0700, Linus Torvalds wrote:
> On Tue, 12 Mar 2024 at 14:34, Florian Fainelli <f.fainelli@gmail.com> wrote:
> >
> > and here is a log where this fails:
> >
> > https://gist.github.com/ffainelli/ed08a2b3e853f59343786ebd20364fc8
> 
> You could try the 'initcall_debug' kernel command line.
> 

Right, that'll be helpful.

Besides I took a look at the config Florian shared, no TASKS_RCU,
RCU_LAZY or RCU nocb is enabled. So probably the only left changes in
the PR are around RCU expedited. Florian, could you see if you can build
and test with CONFIG_PROVE_RCU=y (you need to select
CONFIG_PROVE_LOCKING for that)? That'll call synchronize_rcu() +
synchronize_rcu_expedited() before and after we switch
rcu_scheduler_active, and it may provide more information. Thanks!

Regards,
Boqun

> It will make the above *much* noisier, but it might - thanks to all
> the new noise - show exactly *what* is being crazy slow to initialize.
> 
> Because right now it's just radio silence in between those
> 
>   [    1.926435] bcmgenet f0480000.ethernet: GENET 5.0 EPHY: 0x0000
>   [  162.148135] unimac-mdio unimac-mdio.0: Broadcom UniMAC MDIO bus
> 
> things, and that's presumably because some random initcall there just
> takes forever to time out.
> 
>              Linus