Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sat, 08 May 2021 22:29:57 -0400
From:   "Alex Xu (Hello71)" <alex_y_xu@yahoo.ca>
Subject: Re: regression: data corruption with ext4 on LUKS on nvme with
 torvalds master
To:     linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org,
        dm-crypt@saout.de, linux-nvme@lists.infradead.org,
        linux-block@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
        Changheun Lee <nanich.lee@samsung.com>, bvanassche@acm.org,
        yi.zhang@redhat.com, ming.lei@redhat.com, bgoncalv@redhat.com,
        hch@lst.de, jaegeuk@kernel.org
References: <1620493841.bxdq8r5haw.none@localhost>
In-Reply-To: <1620493841.bxdq8r5haw.none@localhost>
MIME-Version: 1.0
Message-Id: <1620526887.tg1zx7w5np.none@localhost>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Content-Length: 2267
Precedence: bulk

Excerpts from Alex Xu (Hello71)'s message of May 8, 2021 1:54 pm:
> Hi all,
>=20
> Using torvalds master, I recently encountered data corruption on my ext4=20
> volume on LUKS on NVMe. Specifically, during heavy writes, the system=20
> partially hangs; SysRq-W shows that processes are blocked in the kernel=20
> on I/O. After forcibly rebooting, chunks of files are replaced with=20
> other, unrelated data. I'm not sure exactly what the data is; some of it=20
> is unknown binary data, but in at least one case, a list of file paths=20
> was inserted into a file, indicating that the data is misdirected after=20
> encryption.
>=20
> This issue appears to affect files receiving writes in the temporal=20
> vicinity of the hang, but affects both new and old data: for example, my=20
> shell history file was corrupted up to many months before.
>=20
> The drive reports no SMART issues.
>=20
> I believe this is a regression in the kernel related to something merged=20
> in the last few days, as it consistently occurs with my most recent=20
> kernel versions, but disappears when reverting to an older kernel.
>=20
> I haven't investigated further, such as by bisecting. I hope this is=20
> sufficient information to give someone a lead on the issue, and if it is=20
> a bug, nail it down before anybody else loses data.
>=20
> Regards,
> Alex.
>=20

I found the following test to reproduce a hang, which I guess may be the=20
cause:

host$ cd /tmp
host$ truncate -s 10G drive
host$ qemu-system-x86_64 -drive format=3Draw,file=3Ddrive,if=3Dnone,id=3Ddr=
ive -device nvme,drive=3Ddrive,serial=3D1 [... more VM setup options]
guest$ cryptsetup luksFormat /dev/nvme0n1
[accept warning, use any password]
guest$ cryptsetup open /dev/nvme0n1
[enter password]
guest$ mkfs.ext4 /dev/mapper/test
[normal output...]
Creating journal (16384 blocks): [hangs forever]

I bisected this issue to:

cd2c7545ae1beac3b6aae033c7f31193b3255946 is the first bad commit
commit cd2c7545ae1beac3b6aae033c7f31193b3255946
Author: Changheun Lee <nanich.lee@samsung.com>
Date:   Mon May 3 18:52:03 2021 +0900

    bio: limit bio max size

I didn't try reverting this commit or further reducing the test case.=20
Let me know if you need my kernel config or other information.

Regards,
Alex.