Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Joel Fernandes <joel@joelfernandes.org>
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH 5.15 00/92] 5.15.126-rc1 review
Date:   Thu, 10 Aug 2023 19:13:14 -0400
Message-Id: <C45ECBF0-6573-47BC-B127-0B2BEF3E4425@joelfernandes.org>
References: <3d23da0b-696b-5ace-0ea7-5ac4a38cd46b@roeck-us.net>
Cc:     "Paul E. McKenney" <paulmck@kernel.org>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        stable@vger.kernel.org, patches@lists.linux.dev,
        linux-kernel@vger.kernel.org, torvalds@linux-foundation.org,
        akpm@linux-foundation.org, shuah@kernel.org, patches@kernelci.org,
        lkft-triage@lists.linaro.org, pavel@denx.de, jonathanh@nvidia.com,
        f.fainelli@gmail.com, sudipm.mukherjee@gmail.com,
        srw@sladewatkins.net, rwarsow@gmx.de, conor@kernel.org
In-Reply-To: <3d23da0b-696b-5ace-0ea7-5ac4a38cd46b@roeck-us.net>
To:     Guenter Roeck <linux@roeck-us.net>
Precedence: bulk


> On Aug 10, 2023, at 6:55 PM, Guenter Roeck <linux@roeck-us.net> wrote:
>=20
> =EF=BB=BFOn 8/10/23 14:54, Joel Fernandes wrote:
>>> On Thu, Aug 10, 2023 at 10:55:16AM -0700, Paul E. McKenney wrote:
>>> On Wed, Aug 09, 2023 at 02:45:44PM -0700, Guenter Roeck wrote:
>>>> On 8/9/23 13:39, Joel Fernandes wrote:
>>>>> On Wed, Aug 9, 2023 at 4:38=E2=80=AFPM Guenter Roeck <linux@roeck-us.n=
et> wrote:
>>>>>>=20
>>>>>> On 8/9/23 13:14, Joel Fernandes wrote:
>>>>>>> On Wed, Aug 09, 2023 at 12:25:48PM -0700, Guenter Roeck wrote:
>>>>>>>> On Wed, Aug 09, 2023 at 02:35:59PM -0400, Joel Fernandes wrote:
>>>>>>>>> On Wed, Aug 9, 2023 at 12:18=E2=80=AFPM Guenter Roeck <linux@roeck=
-us.net> wrote:
>>>>>>>>>>=20
>>>>>>>>>> On 8/9/23 06:53, Joel Fernandes wrote:
>>>>>>>>>>> On Wed, Aug 09, 2023 at 12:40:36PM +0200, Greg Kroah-Hartman wro=
te:
>>>>>>>>>>>> This is the start of the stable review cycle for the 5.15.126 r=
elease.
>>>>>>>>>>>> There are 92 patches in this series, all will be posted as a re=
sponse
>>>>>>>>>>>> to this one.  If anyone has any issues with these being applied=
, please
>>>>>>>>>>>> let me know.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Responses should be made by Fri, 11 Aug 2023 10:36:10 +0000.
>>>>>>>>>>>> Anything received after that time might be too late.
>>>>>>>>>>>>=20
>>>>>>>>>>>> The whole patch series can be found in one patch at:
>>>>>>>>>>>>        https://www.kernel.org/pub/linux/kernel/v5.x/stable-revi=
ew/patch-5.15.126-rc1.gz
>>>>>>>>>>>> or in the git tree and branch at:
>>>>>>>>>>>>        git://git.kernel.org/pub/scm/linux/kernel/git/stable/lin=
ux-stable-rc.git linux-5.15.y
>>>>>>>>>>>> and the diffstat can be found below.
>>>>>>>>>>>=20
>>>>>>>>>>> Not necesscarily new with 5.15 stable but 3 of the 19 rcutorture=
 scenarios
>>>>>>>>>>> hang with this -rc: TREE04, TREE07, TASKS03.
>>>>>>>>>>>=20
>>>>>>>>>>> 5.15 has a known stop machine issue where it hangs after 1.5 hou=
rs with cpu
>>>>>>>>>>> hotplug rcutorture testing. Me and tglx are continuing to debug t=
his. The
>>>>>>>>>>> issue does not show up on anything but 5.15 stable kernels and n=
either on
>>>>>>>>>>> mainline.
>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> Do you by any have a crash pattern that we could possibly use to f=
ind the crash
>>>>>>>>>> in ChromeOS crash logs ? No idea if that would help, but it could=
 provide some
>>>>>>>>>> additional data points.
>>>>>>>>>=20
>>>>>>>>> The pattern shows as a hard hang, the system is unresponsive and a=
ll CPUs
>>>>>>>>> are stuck in stop_machine. Sometimes it recovers on its own from t=
he
>>>>>>>>> hang and then RCU immediately gives stall warnings. It takes 1.5 h=
our
>>>>>>>>> to reproduce and sometimes never happens for several hours.
>>>>>>>>>=20
>>>>>>>>> It appears related to CPU hotplug since gdb showed me most of the C=
PUs
>>>>>>>>> are spinning in multi_cpu_stop() / stop machine after the hang.
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> Hmm, we do see lots of soft lockups with multi_cpu_stop() in the ba=
cktrace,
>>>>>>>> but not with v5.15.y but with v5.4.y. The actual hang is in stop_ma=
chine_yield().
>>>>>>>=20
>>>>>>> Interesting. It looks similar as far as the stack dump in gdb goes, h=
ere are
>>>>>>> the stacks I dumped with the hang I referred to:
>>>>>>> https://paste.debian.net/1288308/
>>>>>>>=20
>>>>>>=20
>>>>>> That link gives me "Entry not found".
>>>>>=20
>>>>> Yeah that was weird. Here it is again: https://pastebin.com/raw/L3nv1k=
H2
>>>>=20
>>>> I found a couple of crash reports from chromeos-5.10, one of them compl=
aining
>>>> about RCU issues. I sent you links via IM. Nothing from 5.15 or later, t=
hough.
>>>=20
>>> Is the crash showing the eternally refiring timer fixed by this commit?
>>>=20
>>> 53e87e3cdc15 ("timers/nohz: Last resort update jiffies on nohz_full IRQ e=
ntry")
>> Ah I was just replying, I have been seeing really good results after appl=
ying
>> the following 3 commits since yesterday:
>> 53e87e3cdc15 ("timers/nohz: Last resort update jiffies on nohz_full IRQ e=
ntry")
>> 5417ddc1cf1f ("timers/nohz: Switch to ONESHOT_STOPPED in the low-res hand=
ler when the tick is stopped")
>> a1ff03cd6fb9 ("tick: Detect and fix jiffies update stall")
>=20
> Would those also apply to v5.10.y, or just 5.15.y ?

All apply to 5.10 but one. I am currently testing with it more and will post=
 to stable for 5.10 as well.

Thanks,

 - Joel


>=20
> Thanks,
> Guenter
>=20
>> 5417ddc1cf1f also mentioned a "tick storm" which is exactly what I was
>> seeing.
>> I did a lengthy test and everything is looking good. I'll send these out t=
o
>> the stable list.
>> thanks,
>>  - Joel
>=20