Received-SPF: pass (google.com: domain of linux-kernel+bounces-100851-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <95467e67-16f1-4c75-a609-0295ac45db43@gmail.com>
Date: Tue, 12 Mar 2024 15:05:37 -0700
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for
 v6.9]
Content-Language: en-US
To: Frederic Weisbecker <frederic@kernel.org>
Cc: Boqun Feng <boqun.feng@gmail.com>, torvalds@linux-foundation.org,
 linux-kernel@vger.kernel.org, kernel-team@meta.com, paulmck@kernel.org,
 mingo@kernel.org, tglx@linutronix.de, rcu@vger.kernel.org,
 joel@joelfernandes.org, neeraj.upadhyay@amd.com, urezki@gmail.com,
 qiang.zhang1211@gmail.com, bigeasy@linutronix.de, anna-maria@linutronix.de,
 chenzhongjin@huawei.com, yangjihong1@huawei.com, rostedt@goodmis.org
References: <ZetHwrCb0KXE0xFI@tardis>
 <4274be61-60bd-4e1e-9c16-26e6e5e06f65@gmail.com>
 <ZfDCjBFfyWpTK-I1@pavilion.home>
From: Florian Fainelli <f.fainelli@gmail.com>
In-Reply-To: <ZfDCjBFfyWpTK-I1@pavilion.home>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 3/12/24 14:01, Frederic Weisbecker wrote:
> Le Tue, Mar 12, 2024 at 01:32:03PM -0700, Florian Fainelli a écrit :
>> Hi Boqun,
>>
>> On 3/8/24 09:15, Boqun Feng wrote:
>>> Hi Linus,
>>>
>>> Please pull this for the RCU changes of v6.9:
>>>
>>> The following changes since commit 41bccc98fb7931d63d03f326a746ac4d429c1dd3:
>>>
>>>     Linux 6.8-rc2 (2024-01-28 17:01:12 -0800)
>>>
>>> are available in the Git repository at:
>>>
>>>     git://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git tags/rcu.next.v6.9
>>>
>>> for you to fetch changes up to 3add00be5fe5810d7aa5ec3af8b6a245ef33144b:
>>>
>>>     Merge branches 'rcu-doc.2024.02.14a', 'rcu-nocb.2024.02.14a', 'rcu-exp.2024.02.14a', 'rcu-tasks.2024.02.26a' and 'rcu-misc.2024.02.14a' into rcu.2024.02.26a (2024-02-26 17:37:25 -0800)
>>>
>>>
>>> Two merge conflicts were detected by linux-next:
>>>
>>> * https://lore.kernel.org/lkml/20240226135745.12ac854d@canb.auug.org.au/
>>> * https://lore.kernel.org/lkml/20240227125522.2bdbe6be@canb.auug.org.au/
>>>
>>> These conflict resolutions from linux-next look good to me, plus I made
>>> my own resolutions at branch merge/rcu.2024.02.27a for your reference.
>>>
>>>
>>> Some highlights of the changes:
>>>
>>> * Eliminates deadlocks involving do_exit() and RCU tasks, by Paul:
>>>     Instead of SRCU read side critical sections, now a percpu list is used
>>>     in do_exit() for scaning yet-to-exit tasks.
>>>
>>> * Fixes a deadlock due to the dependency between workqueue and RCU
>>>     expedited grace period, reported by Anna-Maria Behnsen and Thomas
>>>     Gleixner and fixed by Frederic: Now RCU expedited always uses its own
>>>     kthread worker instead of a workqueue.
>>
>> At least one device in my test farm (ARM 32-bit) has consistently shown a
>> very long boot, and some others are intermittently affected. This
>> consistently looks like this on most of my devices:
>>
>> [    2.450351] bcmgenet f0480000.ethernet: GENET 5.0 EPHY: 0x0000
>> [    2.547562] ata1: SATA link down (SStatus 0 SControl 300)
>> [  162.107264] unimac-mdio unimac-mdio.0: Broadcom UniMAC MDIO bus
>>
>> this gets flagged by my boot script as a boot failure since we exceeded the
>> 30 seconds timeout given to boot a kernel to a prompt.
>>
>> It has been somewhat difficult to get a reliable bisection going on, but
>> what I am sure of is that e5a3878c947ceef7b6ab68fdc093f3848059842c~1 does
>> not expose the problem for 10 consecutive boots, while I *might* see it at
>> e5a3878c947ceef7b6ab68fdc093f3848059842c and beyond.
>>
>> Any clues what is going on here?
> 
> I understand it doesn't reliably reproduce. But is there any chance you could
> test that alone?
> 
>       0bb11a372fc8 (rcu-tasks: Maintain real-time response in rcu_tasks_postscan())
> 
> And if it reproduces there try a bisection?
> 
> git bisect bad 0bb11a372fc8d7006b4d0f42a2882939747bdbff
> git bisect good 41bccc98fb7931d63d03f326a746ac4d429c1dd3

None of them failed unfortunately, I need to narrow down the window 
between v6.8 and 65d287c7eb1d14e0f4d56f19cec30d97fc7e8f66.

Thanks!
-- 
Florian