Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Sender: Guenter Roeck <groeck7@gmail.com>
Message-ID: <3d23da0b-696b-5ace-0ea7-5ac4a38cd46b@roeck-us.net>
Date:   Thu, 10 Aug 2023 15:55:09 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [PATCH 5.15 00/92] 5.15.126-rc1 review
Content-Language: en-US
To:     Joel Fernandes <joel@joelfernandes.org>,
        "Paul E. McKenney" <paulmck@kernel.org>
Cc:     Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        stable@vger.kernel.org, patches@lists.linux.dev,
        linux-kernel@vger.kernel.org, torvalds@linux-foundation.org,
        akpm@linux-foundation.org, shuah@kernel.org, patches@kernelci.org,
        lkft-triage@lists.linaro.org, pavel@denx.de, jonathanh@nvidia.com,
        f.fainelli@gmail.com, sudipm.mukherjee@gmail.com,
        srw@sladewatkins.net, rwarsow@gmx.de, conor@kernel.org
References: <20230809103633.485906560@linuxfoundation.org>
 <20230809135326.GE3031656@google.com>
 <f47340c6-3c41-1f91-d0f9-fe0b59a23aac@roeck-us.net>
 <CAEXW_YQ4GqPwvUF8=8CWmdj=cD56v_eEVK-EirsObQXyBDFVpg@mail.gmail.com>
 <35e4b770-2ead-4a19-ad01-fa75996adef4@roeck-us.net>
 <20230809201413.GA3374446@google.com>
 <6b05a082-41a7-f0cf-c0a4-1cced8d5a230@roeck-us.net>
 <CAEXW_YT-7epvu4uUS19aDAcM0X63j9_L2aa-XGGFGSLceLu8bA@mail.gmail.com>
 <4dbe72a3-50ea-051c-96ba-d709b33d3a98@roeck-us.net>
 <882a606f-5776-46d1-a423-2c5ea0a1cd8f@paulmck-laptop>
 <20230810215416.GA562211@google.com>
From:   Guenter Roeck <linux@roeck-us.net>
In-Reply-To: <20230810215416.GA562211@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 8/10/23 14:54, Joel Fernandes wrote:
> On Thu, Aug 10, 2023 at 10:55:16AM -0700, Paul E. McKenney wrote:
>> On Wed, Aug 09, 2023 at 02:45:44PM -0700, Guenter Roeck wrote:
>>> On 8/9/23 13:39, Joel Fernandes wrote:
>>>> On Wed, Aug 9, 2023 at 4:38 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>>>
>>>>> On 8/9/23 13:14, Joel Fernandes wrote:
>>>>>> On Wed, Aug 09, 2023 at 12:25:48PM -0700, Guenter Roeck wrote:
>>>>>>> On Wed, Aug 09, 2023 at 02:35:59PM -0400, Joel Fernandes wrote:
>>>>>>>> On Wed, Aug 9, 2023 at 12:18 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>>>>>>>
>>>>>>>>> On 8/9/23 06:53, Joel Fernandes wrote:
>>>>>>>>>> On Wed, Aug 09, 2023 at 12:40:36PM +0200, Greg Kroah-Hartman wrote:
>>>>>>>>>>> This is the start of the stable review cycle for the 5.15.126 release.
>>>>>>>>>>> There are 92 patches in this series, all will be posted as a response
>>>>>>>>>>> to this one.  If anyone has any issues with these being applied, please
>>>>>>>>>>> let me know.
>>>>>>>>>>>
>>>>>>>>>>> Responses should be made by Fri, 11 Aug 2023 10:36:10 +0000.
>>>>>>>>>>> Anything received after that time might be too late.
>>>>>>>>>>>
>>>>>>>>>>> The whole patch series can be found in one patch at:
>>>>>>>>>>>         https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.126-rc1.gz
>>>>>>>>>>> or in the git tree and branch at:
>>>>>>>>>>>         git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
>>>>>>>>>>> and the diffstat can be found below.
>>>>>>>>>>
>>>>>>>>>> Not necesscarily new with 5.15 stable but 3 of the 19 rcutorture scenarios
>>>>>>>>>> hang with this -rc: TREE04, TREE07, TASKS03.
>>>>>>>>>>
>>>>>>>>>> 5.15 has a known stop machine issue where it hangs after 1.5 hours with cpu
>>>>>>>>>> hotplug rcutorture testing. Me and tglx are continuing to debug this. The
>>>>>>>>>> issue does not show up on anything but 5.15 stable kernels and neither on
>>>>>>>>>> mainline.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you by any have a crash pattern that we could possibly use to find the crash
>>>>>>>>> in ChromeOS crash logs ? No idea if that would help, but it could provide some
>>>>>>>>> additional data points.
>>>>>>>>
>>>>>>>> The pattern shows as a hard hang, the system is unresponsive and all CPUs
>>>>>>>> are stuck in stop_machine. Sometimes it recovers on its own from the
>>>>>>>> hang and then RCU immediately gives stall warnings. It takes 1.5 hour
>>>>>>>> to reproduce and sometimes never happens for several hours.
>>>>>>>>
>>>>>>>> It appears related to CPU hotplug since gdb showed me most of the CPUs
>>>>>>>> are spinning in multi_cpu_stop() / stop machine after the hang.
>>>>>>>>
>>>>>>>
>>>>>>> Hmm, we do see lots of soft lockups with multi_cpu_stop() in the backtrace,
>>>>>>> but not with v5.15.y but with v5.4.y. The actual hang is in stop_machine_yield().
>>>>>>
>>>>>> Interesting. It looks similar as far as the stack dump in gdb goes, here are
>>>>>> the stacks I dumped with the hang I referred to:
>>>>>> https://paste.debian.net/1288308/
>>>>>>
>>>>>
>>>>> That link gives me "Entry not found".
>>>>
>>>> Yeah that was weird. Here it is again: https://pastebin.com/raw/L3nv1kH2
>>>
>>> I found a couple of crash reports from chromeos-5.10, one of them complaining
>>> about RCU issues. I sent you links via IM. Nothing from 5.15 or later, though.
>>
>> Is the crash showing the eternally refiring timer fixed by this commit?
>>
>> 53e87e3cdc15 ("timers/nohz: Last resort update jiffies on nohz_full IRQ entry")
> 
> Ah I was just replying, I have been seeing really good results after applying
> the following 3 commits since yesterday:
> 
> 53e87e3cdc15 ("timers/nohz: Last resort update jiffies on nohz_full IRQ entry")
> 5417ddc1cf1f ("timers/nohz: Switch to ONESHOT_STOPPED in the low-res handler when the tick is stopped")
> a1ff03cd6fb9 ("tick: Detect and fix jiffies update stall")
> 

Would those also apply to v5.10.y, or just 5.15.y ?

Thanks,
Guenter

> 5417ddc1cf1f also mentioned a "tick storm" which is exactly what I was
> seeing.
> 
> I did a lengthy test and everything is looking good. I'll send these out to
> the stable list.
> 
> thanks,
> 
>   - Joel
> 
>