Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: id4a34324:Fastmail
User-Agent: Cyrus-JMAP/3.7.0-alpha0-841-g7899e99a45-fm-20220811.002-g7899e99a
Mime-Version: 1.0
Message-Id: <3df6bb82-1951-455d-a768-e9e1513eb667@www.fastmail.com>
In-Reply-To: <20220815031549-mutt-send-email-mst@kernel.org>
References: <20220814212610.GA3690074@roeck-us.net>
 <CAHk-=wgf2EfLHui6A5NbWoaVBB2f8t-XBUiOMkyjN2NU41t6eA@mail.gmail.com>
 <20220814223743.26ebsbnrvrjien4f@awork3.anarazel.de>
 <CAHk-=wi6raoJE-1cyRU0YxJ+9ReO1eXmOAq0FwKAyZS7nhvk9w@mail.gmail.com>
 <1c057afa-92df-ee3c-5978-3731d3db9345@kernel.dk>
 <20220815013651.mrm7qgklk6sgpkbb@awork3.anarazel.de>
 <CAHk-=wikzU4402P-FpJRK_QwfVOS+t-3p1Wx5awGHTvr-s_0Ew@mail.gmail.com>
 <20220815071143.n2t5xsmifnigttq2@awork3.anarazel.de>
 <20220815031549-mutt-send-email-mst@kernel.org>
Date:   Mon, 15 Aug 2022 00:46:36 -0700
From:   "Andres Freund" <andres@anarazel.de>
To:     "Michael S. Tsirkin" <mst@redhat.com>
Cc:     "Linus Torvalds" <torvalds@linux-foundation.org>,
        "Jens Axboe" <axboe@kernel.dk>,
        "James Bottomley" <James.Bottomley@hansenpartnership.com>,
        "Martin K. Petersen" <martin.petersen@oracle.com>,
        "Guenter Roeck" <linux@roeck-us.net>, linux-kernel@vger.kernel.org,
        "Greg Kroah-Hartman" <gregkh@linuxfoundation.org>
Subject: Re: upstream kernel crashes
Content-Type: text/plain
Precedence: bulk

Hi,

On Mon, Aug 15, 2022, at 00:29, Michael S. Tsirkin wrote:
> On Mon, Aug 15, 2022 at 12:11:43AM -0700, Andres Freund wrote:
>> Hi,
>> 
>> On 2022-08-14 20:18:44 -0700, Linus Torvalds wrote:
>> > On Sun, Aug 14, 2022 at 6:36 PM Andres Freund <andres@anarazel.de> wrote:
>> > >
>> > > Some of the symptoms could be related to the issue in this thread, hence
>> > > listing them here
>> > 
>> > Smells like slab corruption to me, and the problems may end up being
>> > then largely random just depending on who ends up using the allocation
>> > that gets trampled on.
>> > 
>> > I wouldn't be surprised if it's all the same thing - including your
>> > network issue.
>> 
>> Yea. As I just wrote in
>> https://postgr.es/m/20220815070203.plwjx7b3cyugpdt7%40awork3.anarazel.de I
>> bisected it down to one commit (762faee5a267). With that commit I only see the
>> networking issue across a few reboots, but with ebcce4926365 some boots oops
>> badly and other times it' "just" network not working.
>> 
>> 
>> [oopses]

>> If somebody knowledgeable staring at 762faee5a267 doesn't surface somebody I
>> can create a kernel with some more debugging stuff enabled, if somebody tells
>> me what'd work best here.
>> 
>> 
>> Greetings,
>> 
>> Andres Freund
>
> Thanks a lot for the work!
> Just a small clarification:
>
> So IIUC you see several issues, right?

Yes, although they might be related, as theorized by Linus upthread.

> With 762faee5a2678559d3dc09d95f8f2c54cd0466a7 you see networking issues.

Yes.


> With ebcce492636506443e4361db6587e6acd1a624f9 you see crashes.

Changed between rebooting. Sometimes the network issue, sometimes the crashes in the email you're replying to.


> MST