Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220814212610.GA3690074@roeck-us.net> <CAHk-=wgf2EfLHui6A5NbWoaVBB2f8t-XBUiOMkyjN2NU41t6eA@mail.gmail.com>
 <20220814223743.26ebsbnrvrjien4f@awork3.anarazel.de> <CAHk-=wi6raoJE-1cyRU0YxJ+9ReO1eXmOAq0FwKAyZS7nhvk9w@mail.gmail.com>
 <1c057afa-92df-ee3c-5978-3731d3db9345@kernel.dk> <20220815013651.mrm7qgklk6sgpkbb@awork3.anarazel.de>
In-Reply-To: <20220815013651.mrm7qgklk6sgpkbb@awork3.anarazel.de>
From:   Linus Torvalds <torvalds@linux-foundation.org>
Date:   Sun, 14 Aug 2022 20:18:44 -0700
Message-ID: <CAHk-=wikzU4402P-FpJRK_QwfVOS+t-3p1Wx5awGHTvr-s_0Ew@mail.gmail.com>
Subject: Re: upstream kernel crashes
To:     Andres Freund <andres@anarazel.de>
Cc:     Jens Axboe <axboe@kernel.dk>,
        James Bottomley <James.Bottomley@hansenpartnership.com>,
        "Martin K. Petersen" <martin.petersen@oracle.com>,
        Guenter Roeck <linux@roeck-us.net>,
        linux-kernel@vger.kernel.org, Greg KH <gregkh@linuxfoundation.org>,
        "Michael S. Tsirkin" <mst@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Sun, Aug 14, 2022 at 6:36 PM Andres Freund <andres@anarazel.de> wrote:
>
> Some of the symptoms could be related to the issue in this thread, hence
> listing them here

Smells like slab corruption to me, and the problems may end up being
then largely random just depending on who ends up using the allocation
that gets trampled on.

I wouldn't be surprised if it's all the same thing - including your
network issue.

Both you and Guenter seem to be seeing this on that google cloud VM
setup, which may or may not be related. I haven't seen any reports
from anybody else, but that might still be just coincidence.

But it makes me go back to your original report, where you suspected
the virtio pull.

Looking at that original report, and doing

    gitk 7ebfc85e2cd7..69dac8e431af

nothing else looks all that relevant, although obviously any random
error path could end up doing a double free and messing up the slab
cache that way. But a lot of the non-virtio stuff there is things like
RISC-V and devicetree that definitely shouldn't have any effect.

Adding Michael Tsirkin to this thread too, just so he knows.

            Linus