Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CA+wXwBRGab3UqbLqsr8xG=ZL2u9bgyDNNea4RGfTDjqB=J3geQ@mail.gmail.com>
 <CA+wXwBR6S3StBwJJmo8Fu6KdPW5Q382N7FwnmfckBJo4e6ZD_A@mail.gmail.com> <ZL7w9dEH8BSXRzyu@dread.disaster.area>
In-Reply-To: <ZL7w9dEH8BSXRzyu@dread.disaster.area>
From:   Daniel Dao <dqminh@cloudflare.com>
Date:   Mon, 24 Jul 2023 23:04:13 +0100
Message-ID: <CA+wXwBQmusp49-b6cU-hPAoOnpTiuvA2QrjaOSyb-EvKigC_Ug@mail.gmail.com>
Subject: Re: Kernel NULL pointer deref and data corruptions with xfs on 6.1
To:     Dave Chinner <david@fromorbit.com>
Cc:     linux-fsdevel@vger.kernel.org,
        Matthew Wilcox <willy@infradead.org>,
        kernel-team <kernel-team@cloudflare.com>,
        linux-kernel <linux-kernel@vger.kernel.org>, djwong@kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, Jul 24, 2023 at 10:45=E2=80=AFPM Dave Chinner <david@fromorbit.com>=
 wrote:
>
> On Mon, Jul 24, 2023 at 12:23:31PM +0100, Daniel Dao wrote:
> > Hi again,
> >
> > We had another example of xarray corruption involving xfs and zsmalloc.=
 We are
> > running zram as swap. We have 2 tasks deadlock waiting for page to be r=
eleased
>
> Do your problems on 6.1 go away if you stop using zram as swap?

We had xarray corruptions even on nodes without swap, so I'm not sure
if swap matters.
The corruption on those nodes were noted in the first email with the
following trace

 BUG: kernel NULL pointer dereference, address: 0000000000000036
    #PF: supervisor read access in kernel mode
    #PF: error_code(0x0000) - not-present page
    PGD 18806c5067 P4D 18806c5067 PUD 188ed48067 PMD 0
    Oops: 0000 [#1] PREEMPT SMP NOPTI
    CPU: 73 PID: 3579408 Comm: prometheus Tainted: G           O
6.1.34-cloudflare-2023.6.7 #1
    Hardware name: GIGABYTE R162-Z12-CD1/MZ12-HD4-CD, BIOS M03 11/19/2021
    RIP: 0010:__filemap_get_folio (arch/x86/include/asm/atomic.h:29
include/linux/atomic/atomic-arch-fallback.h:1242
include/linux/atomic/atomic-arch-fallback.h:1267
include/linux/atomic/atomic-instrumented.h:608
include/linux/page_ref.h:238 include/linux/page_ref.h:247
include/linux/page_ref.h:280 include/linux/page_ref.h:313
mm/filemap.c:1863 mm/filemap.c:1915)

It's hard for us to run tests without zram swap at scale since the
benefits are significant with a lot of
workloads.

Daniel.