Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f5d2cc62-4aae-2579-1468-2e6e389f28dc@amazon.com>
Date:   Wed, 14 Jun 2023 09:57:56 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.11.2
Subject: Re: Observing RCU stalls in kernel 5.4/5.10/5.15/6.1 stable trees
Content-Language: en-US
To:     Sven-Haegar Koch <haegar@sdinet.de>
CC:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "Bhatnagar, Rishabh" <risbhat@amazon.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "sashal@kernel.org" <sashal@kernel.org>, <abuehaze@amazon.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>,
        <minipli@grsecurity.net>
References: <12c6f9a3-d087-b824-0d05-0d18c9bc1bf3@amazon.com>
 <c4724b40-89f6-5aa7-720d-c4a4af57cf45@amazon.com>
 <2023061428-compacter-economic-b648@gregkh>
 <20230614092045.tNY8USjq@linutronix.de>
 <4c4178a1-1050-ced4-e6fb-f95c3bdefc98@amazon.com>
 <2a3fa097-8ba0-5b0e-f506-779fee5b8fef@sdinet.de>
From:   Luiz Capitulino <luizcap@amazon.com>
In-Reply-To: <2a3fa097-8ba0-5b0e-f506-779fee5b8fef@sdinet.de>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 2023-06-14 09:45, Sven-Haegar Koch wrote:

> 
> 
> 
> On Wed, 14 Jun 2023, Luiz Capitulino wrote:
> 
>> On 2023-06-14 05:20, Sebastian Andrzej Siewior wrote:
>>
>>> On 2023-06-14 11:14:49 [+0200], gregkh@linuxfoundation.org wrote:
>>>> Oops, missed this.
>>>>
>>>> Yes, there might be, can you do 'git bisect' and track down the patch
>>>> that fixed this?
>>>
>>> There was a report of a lockup during boot in VMs yesterday. If I
>>> remember correctly this still exists and might be related to this
>>> report. I'm going to have a look.
>>
>> Thanks, Sebastian. Do you have a link for the discussion?
> 
> May be this, talking about the same commit as cause as this thread:
> 
> Subject: Re: [PATCH] timekeeping: Align tick_sched_timer() with the HZ
> tick. -- regression report
> https://lore.kernel.org/lkml/5a56290d-806e-b9a5-f37c-f21958b5a8c0@grsecurity.net/

Thank you, Sven.

Sebastian, except for the detailed analysis which we haven't done yet, the
issue described by Mathias matches 100% what we're observing. Also, we do
observe this on bare-metal instances which could mean that the initial
reports are against VMs because those are rebooted more often (our quick
reproducer boots hundreds of instances in AWS and only 1 or 2 reproduces this).

IMHO, I'd suggest we revert this for now from Linus tree and stable trees.
We can help testing for the fix maybe for the next merge window.

- Luiz

> 
> May not have been the best idea to respond with such big analysis to a 3
> months old dead thread, gets lost extremely easy.
> 
> c'ya
> sven-haegar
> 
> --
> Three may keep a secret, if two of them are dead.
> - Ben F.