Received-SPF: pass (google.com: domain of linux-kernel+bounces-112079-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Precedence: bulk
MIME-Version: 1.0
References: <1ca6e688950ee82b1526bb3098852af99b75e6ba.1710551459.git.tavianator@tavianator.com>
 <20240322192108.GK14596@twin.jikos.cz>
In-Reply-To: <20240322192108.GK14596@twin.jikos.cz>
From: Tavian Barnes <tavianator@tavianator.com>
Date: Fri, 22 Mar 2024 18:50:07 -0400
Message-ID: <CABg4E-n=iWjM3K0dvrrwQ9BOEiqTawJ4YUVL6RPWaq2DT2AKvQ@mail.gmail.com>
Subject: Re: [PATCH] btrfs: Fix race in read_extent_buffer_pages()
To: dsterba@suse.cz
Cc: linux-btrfs@vger.kernel.org, Christoph Hellwig <hch@lst.de>, Qu Wenruo <wqu@suse.com>, 
	Chris Mason <clm@fb.com>, Josef Bacik <josef@toxicpanda.com>, David Sterba <dsterba@suse.com>, 
	linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Mar 22, 2024 at 3:28=E2=80=AFPM David Sterba <dsterba@suse.cz> wrot=
e:
> On Fri, Mar 15, 2024 at 09:14:29PM -0400, Tavian Barnes wrote:
> > To prevent concurrent reads for the same extent buffer,
> > read_extent_buffer_pages() performs these checks:
> >
> >     /* (1) */
> >     if (test_bit(EXTENT_BUFFER_UPTODATE, &eb->bflags))
> >         return 0;
> >
> >     /* (2) */
> >     if (test_and_set_bit(EXTENT_BUFFER_READING, &eb->bflags))
> >         goto done;
> >
> > At this point, it seems safe to start the actual read operation. Once
> > that completes, end_bbio_meta_read() does
> >
> >     /* (3) */
> >     set_extent_buffer_uptodate(eb);
> >
> >     /* (4) */
> >     clear_bit(EXTENT_BUFFER_READING, &eb->bflags);
> >
> > Normally, this is enough to ensure only one read happens, and all other
> > callers wait for it to finish before returning.  Unfortunately, there i=
s
> > a racey interleaving:
> >
> >     Thread A | Thread B | Thread C
> >     ---------+----------+---------
> >        (1)   |          |
> >              |    (1)   |
> >        (2)   |          |
> >        (3)   |          |
> >        (4)   |          |
> >              |    (2)   |
> >              |          |    (1)
> >
> > When this happens, thread B kicks of an unnecessary read. Worse, thread
> > C will see UPTODATE set and return immediately, while the read from
> > thread B is still in progress.  This race could result in tree-checker
> > errors like this as the extent buffer is concurrently modified:
> >
> >     BTRFS critical (device dm-0): corrupted node, root=3D256
> >     block=3D8550954455682405139 owner mismatch, have 118582055676422943=
56
> >     expect [256, 18446744073709551360]
> >
> > Fix it by testing UPTODATE again after setting the READING bit, and if
> > it's been set, skip the unnecessary read.
> >
> > Fixes: d7172f52e993 ("btrfs: use per-buffer locking for extent_buffer r=
eading")
> > Link: https://lore.kernel.org/linux-btrfs/CAHk-=3DwhNdMaN9ntZ47XRKP6DBe=
s2E5w7fi-0U3H2+PS18p+Pzw@mail.gmail.com/
> > Link: https://lore.kernel.org/linux-btrfs/f51a6d5d7432455a6a858d51b49ec=
ac183e0bbc9.1706312914.git.wqu@suse.com/
> > Link: https://lore.kernel.org/linux-btrfs/c7241ea4-fcc6-48d2-98c8-b5ea7=
90d6c89@gmx.com/
> > Signed-off-by: Tavian Barnes <tavianator@tavianator.com>
>
> Thank you very much for taking the time to debug the issue and for the
> fix. It is a rare occurrence that a tough bug is followed by a fix from
> the same person (outside of the developer group) and is certainly
> appreciated.

Thank you!

Sorry to nitpick, but the paragraph you added to the commit message
[1] has a typo:

> There are reports from tree-checker that detects corrupted nodes,
> without any obvious pattern so possibly an overwrite in memory.
> After some debugging it turns out there's a race when reading an extent
> buffer the uptodate status is can be missed.

s/is can/can/

[1]: https://github.com/btrfs/linux/commit/402887e0e9ad76d72496aefebd37bd72=
9748be79

--=20
Tavian Barnes