Received-SPF: pass (google.com: domain of linux-ext4-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Sun, 09 May 2021 10:47:26 -0400
From:   "Alex Xu (Hello71)" <alex_y_xu@yahoo.ca>
Subject: Re: regression: data corruption with ext4 on LUKS on nvme with
 torvalds master
To:     Jens Axboe <axboe@kernel.dk>, bgoncalv@redhat.com,
        bvanassche@acm.org, dm-crypt@saout.de, hch@lst.de,
        jaegeuk@kernel.org, linux-block@vger.kernel.org,
        linux-ext4@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, ming.lei@redhat.com,
        Changheun Lee <nanich.lee@samsung.com>, yi.zhang@redhat.com
References: <1620493841.bxdq8r5haw.none@localhost>
        <1620526887.tg1zx7w5np.none@localhost>
        <f0c90fc0-c239-df68-371d-a5c74c8f32eb@kernel.dk>
In-Reply-To: <f0c90fc0-c239-df68-371d-a5c74c8f32eb@kernel.dk>
MIME-Version: 1.0
Message-Id: <1620571445.2k94orj8ee.none@localhost>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Content-Length: 4163
Precedence: bulk

Excerpts from Jens Axboe's message of May 8, 2021 11:51 pm:
> On 5/8/21 8:29 PM, Alex Xu (Hello71) wrote:
>> Excerpts from Alex Xu (Hello71)'s message of May 8, 2021 1:54 pm:
>>> Hi all,
>>>
>>> Using torvalds master, I recently encountered data corruption on my ext=
4=20
>>> volume on LUKS on NVMe. Specifically, during heavy writes, the system=20
>>> partially hangs; SysRq-W shows that processes are blocked in the kernel=
=20
>>> on I/O. After forcibly rebooting, chunks of files are replaced with=20
>>> other, unrelated data. I'm not sure exactly what the data is; some of i=
t=20
>>> is unknown binary data, but in at least one case, a list of file paths=20
>>> was inserted into a file, indicating that the data is misdirected after=
=20
>>> encryption.
>>>
>>> This issue appears to affect files receiving writes in the temporal=20
>>> vicinity of the hang, but affects both new and old data: for example, m=
y=20
>>> shell history file was corrupted up to many months before.
>>>
>>> The drive reports no SMART issues.
>>>
>>> I believe this is a regression in the kernel related to something merge=
d=20
>>> in the last few days, as it consistently occurs with my most recent=20
>>> kernel versions, but disappears when reverting to an older kernel.
>>>
>>> I haven't investigated further, such as by bisecting. I hope this is=20
>>> sufficient information to give someone a lead on the issue, and if it i=
s=20
>>> a bug, nail it down before anybody else loses data.
>>>
>>> Regards,
>>> Alex.
>>>
>>=20
>> I found the following test to reproduce a hang, which I guess may be the=
=20
>> cause:
>>=20
>> host$ cd /tmp
>> host$ truncate -s 10G drive
>> host$ qemu-system-x86_64 -drive format=3Draw,file=3Ddrive,if=3Dnone,id=
=3Ddrive -device nvme,drive=3Ddrive,serial=3D1 [... more VM setup options]
>> guest$ cryptsetup luksFormat /dev/nvme0n1
>> [accept warning, use any password]
>> guest$ cryptsetup open /dev/nvme0n1
>> [enter password]
>> guest$ mkfs.ext4 /dev/mapper/test
>> [normal output...]
>> Creating journal (16384 blocks): [hangs forever]
>>=20
>> I bisected this issue to:
>>=20
>> cd2c7545ae1beac3b6aae033c7f31193b3255946 is the first bad commit
>> commit cd2c7545ae1beac3b6aae033c7f31193b3255946
>> Author: Changheun Lee <nanich.lee@samsung.com>
>> Date:   Mon May 3 18:52:03 2021 +0900
>>=20
>>     bio: limit bio max size
>>=20
>> I didn't try reverting this commit or further reducing the test case.=20
>> Let me know if you need my kernel config or other information.
>=20
> If you have time, please do test with that reverted. I'd be anxious to
> get this revert queued up for 5.13-rc1.
>=20
> --=20
> Jens Axboe
>=20
>=20

I tested reverting it on top of b741596468b010af2846b75f5e75a842ce344a6e=20
("Merge tag 'riscv-for-linus-5.13-mw1' of=20
git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux"), causing it=20
to no longer hang. I didn't check if this fixes the data corruption, but=20
I assume so.

I also tested a 1 GB image (works either way), and a virtio-blk=20
interface (works either way)

The Show Blocked State from the VM (without revert):

sysrq: Show Blocked State
task:kworker/u2:0    state:D stack:    0 pid:    7 ppid:     2 flags:0x0000=
4000
Workqueue: kcryptd/252:0 kcryptd_crypt
Call Trace:
 __schedule+0x1a2/0x4f0
 schedule+0x63/0xe0
 schedule_timeout+0x6a/0xd0
 ? lock_timer_base+0x80/0x80
 io_schedule_timeout+0x4c/0x70
 mempool_alloc+0xfc/0x130
 ? __wake_up_common_lock+0x90/0x90
 kcryptd_crypt+0x291/0x4e0
 process_one_work+0x1b1/0x300
 worker_thread+0x48/0x3d0
 ? process_one_work+0x300/0x300
 kthread+0x129/0x150
 ? __kthread_create_worker+0x100/0x100
 ret_from_fork+0x22/0x30
task:mkfs.ext4       state:D stack:    0 pid:  979 ppid:   964 flags:0x0000=
4000
Call Trace:
 __schedule+0x1a2/0x4f0
 ? __schedule+0x1aa/0x4f0
 schedule+0x63/0xe0
 schedule_timeout+0x99/0xd0
 io_schedule_timeout+0x4c/0x70
 wait_for_completion_io+0x74/0xc0
 submit_bio_wait+0x46/0x60
 blkdev_issue_zeroout+0x118/0x1f0
 blkdev_fallocate+0x125/0x180
 vfs_fallocate+0x126/0x2e0
 __x64_sys_fallocate+0x37/0x60
 do_syscall_64+0x61/0x80
 ? do_syscall_64+0x6e/0x80
 entry_SYSCALL_64_after_hwframe+0x44/0xae

Regards,
Alex.