Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley
        Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United
        Kingdom.
        Registered in England and Wales under Company Registration No. 3798903
From:   David Howells <dhowells@redhat.com>
In-Reply-To: <Yl7EyMLnqqDv63yW@rabbit.intern.cm-ag>
References: <Yl7EyMLnqqDv63yW@rabbit.intern.cm-ag> <YlWWbpW5Foynjllo@rabbit.intern.cm-ag> <454834.1650373340@warthog.procyon.org.uk>
To:     Max Kellermann <mk@cm4all.com>
Cc:     dhowells@redhat.com, linux-cachefs@redhat.com,
        linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: fscache corruption in Linux 5.17?
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <508602.1650385022.1@warthog.procyon.org.uk>
Content-Transfer-Encoding: quoted-printable
Date:   Tue, 19 Apr 2022 17:17:02 +0100
Message-ID: <508603.1650385022@warthog.procyon.org.uk>
Precedence: bulk

Max Kellermann <mk@cm4all.com> wrote:

> At least one web server is still in this broken state right now.  So
> if you need anything from that server, tell me, and I'll get it.

Can you turn on:

echo 65536 >/sys/kernel/debug/tracing/buffer_size_kb
echo 1 >/sys/kernel/debug/tracing/events/cachefiles/cachefiles_read/enable
echo 1 >/sys/kernel/debug/tracing/events/cachefiles/cachefiles_write/enabl=
e
echo 1 >/sys/kernel/debug/tracing/events/cachefiles/cachefiles_trunc/enabl=
e
echo 1 >/sys/kernel/debug/tracing/events/cachefiles/cachefiles_io_error/en=
able
echo 1 >/sys/kernel/debug/tracing/events/cachefiles/cachefiles_vfs_error/e=
nable

Then try and trigger the bug if you can.  The trace can be viewed with:

cat /sys/kernel/debug/tracing/trace | less

The problem very likely happens on write rather than read.  If you know of=
 a
file that's corrupt, turn on the tracing above and read that file.  Then l=
ook
in the trace buffer and you should see the corresponding lines and they sh=
ould
have the backing inode in them, marked "B=3Diiii" where "iiii" is the inod=
e
number of the file in hex.  You should be able to examine the backing file=
 by
finding it with something like:

	find /var/cache/fscache -inum $((0xiiii))

and see if you can see the corruption in there.  Note that there may be bl=
ocks
of zeroes corresponding to unfetched file blocks.

Also, what filesystem is backing your cachefiles cache?  It could be usefu=
l to
dump the extent list of the file.  You should be able to do this with
"filefrag -e".

As to why this happens, a write that's misaligned by 31 bytes should cause=
 DIO
to a disk to fail - so it shouldn't be possible to write that.  However, I=
'm
doing fallocate and truncate on the file to shape it so that DIO will work=
 on
it, so it's possible that there's a bug there.  The cachefiles_trunc trace
lines may help catch that.

David