Received-SPF: pass (google.com: domain of linux-kernel+bounces-51369-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Subject: Re: [PATCH v5 00/14] dm-raid/md/raid: fix v6.7 regressions
To: Benjamin Marzinski <bmarzins@redhat.com>,
 Yu Kuai <yukuai1@huaweicloud.com>
Cc: mpatocka@redhat.com, heinzm@redhat.com, xni@redhat.com,
 blazej.kucman@linux.intel.com, agk@redhat.com, snitzer@kernel.org,
 dm-devel@lists.linux.dev, song@kernel.org, jbrassow@f14.redhat.com,
 neilb@suse.de, shli@fb.com, akpm@osdl.org, linux-kernel@vger.kernel.org,
 linux-raid@vger.kernel.org, yi.zhang@huawei.com, yangerkun@huawei.com,
 "yukuai (C)" <yukuai3@huawei.com>
References: <20240201092559.910982-1-yukuai1@huaweicloud.com>
 <Zb2wxIpf7uYV6Vya@bmarzins-01.fast.eng.rdu2.dc.redhat.com>
From: Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <528ce926-6f17-c1ea-8e77-c7d5d7f56022@huaweicloud.com>
Date: Sun, 4 Feb 2024 09:35:09 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <Zb2wxIpf7uYV6Vya@bmarzins-01.fast.eng.rdu2.dc.redhat.com>
Content-Type: text/plain; charset=gbk; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

?? 2024/02/03 11:19, Benjamin Marzinski д??:
> On Thu, Feb 01, 2024 at 05:25:45PM +0800, Yu Kuai wrote:
>> From: Yu Kuai <yukuai3@huawei.com>
>> I apply this patchset on top of v6.8-rc1, and run lvm2 tests suite with
>> folling cmd for 24 round(for about 2 days):
>>
>> for t in `ls test/shell`; do
>>          if cat test/shell/$t | grep raid &> /dev/null; then
>>                  make check T=shell/$t
>>          fi
>> done
>>
>> failed count                             failed test
>>        1 ###       failed: [ndev-vanilla] shell/dmsecuretest.sh
>>        1 ###       failed: [ndev-vanilla] shell/dmsetup-integrity-keys.sh
>>        1 ###       failed: [ndev-vanilla] shell/dmsetup-keyring.sh
>>        5 ###       failed: [ndev-vanilla] shell/duplicate-pvs-md0.sh
>>        1 ###       failed: [ndev-vanilla] shell/duplicate-vgid.sh
>>        2 ###       failed: [ndev-vanilla] shell/duplicate-vgnames.sh
>>        1 ###       failed: [ndev-vanilla] shell/fsadm-crypt.sh
>>        1 ###       failed: [ndev-vanilla] shell/integrity.sh
>>        6 ###       failed: [ndev-vanilla] shell/lvchange-raid1-writemostly.sh
>>        2 ###       failed: [ndev-vanilla] shell/lvchange-rebuild-raid.sh
>>        5 ###       failed: [ndev-vanilla] shell/lvconvert-raid-reshape-stripes-load-reload.sh
>>        4 ###       failed: [ndev-vanilla] shell/lvconvert-raid-restripe-linear.sh
>>        1 ###       failed: [ndev-vanilla] shell/lvconvert-raid1-split-trackchanges.sh
>>       20 ###       failed: [ndev-vanilla] shell/lvconvert-repair-raid.sh
>>       20 ###       failed: [ndev-vanilla] shell/lvcreate-large-raid.sh
>>       24 ###       failed: [ndev-vanilla] shell/lvextend-raid.sh
>>
>> And I ramdomly pick some tests verified by hand that these test will
>> fail in v6.6 as well(not all tests):
>>
>> shell/lvextend-raid.sh
>> shell/lvcreate-large-raid.sh
>> shell/lvconvert-repair-raid.sh
>> shell/lvchange-rebuild-raid.sh
>> shell/lvchange-raid1-writemostly.sh
> 
> In my testing with this patchset on top of the head of linus's tree
> (5c24e4e9e708) I am seeing failures in
> shell/lvconvert-raid-reshape-stripes-load-reload.sh and
> shell/lvconvert-repair-raid.sh in about 20% of my runs. I have never
> seen either of these these fail running on the 6.6 kernel (ffc253263a13).

This sounds quite different in my testing, as I said, the test

shell/lvconvert-repair-raid.sh is very likely to fail in v6.6 already,
I don't know why it never fail in your testing, test log in v6.6:

| [ 1:38.162] #lvconvert-repair-raid.sh:1+ aux teardown
| [ 1:38.162] ## teardown.......## removing stray mapped devices with 
names beginning with LVMTEST3474:
| [ 1:39.207] .set +vx; STACKTRACE; set -vx
| [ 1:41.448] ##lvconvert-repair-raid.sh:1+ set +vx
| [ 1:41.448] ## - /mnt/test/lvm2/test/shell/lvconvert-repair-raid.sh:1
| [ 1:41.449] ## 1 STACKTRACE() called from 
/mnt/test/lvm2/test/shell/lvconvert-repair-raid.sh:1
| [ 1:41.449] ## ERROR: The test started dmeventd (3718) unexpectedly.

And the same in v6.8-rc1. Perhaps do you know how to fix this error?

Thanks,
Kuai

> 
> lvconvert-repair-raid.sh creates a raid array and then disables one if
> its drives before there's enough time to finish the initial sync and
> tries to repair it. This is supposed to fail (it uses dm-delay devices
> to slow down the sync). When the test succeeds, I see things like this:
> 
> [ 0:13.469] #lvconvert-repair-raid.sh:161+ lvcreate --type raid10 -m 1 -i 2 -L 64 -n LV1 LVMTEST191946vg /tmp/LVMTEST191946.ImUMG6dyqB/dev/mapper/LVMTEST191946pv1 /tmp/LVMTEST191946.ImUMG6dyqB/dev/mapper/LVMTEST191946pv2 /tmp/LVMTEST191946.ImUMG6dyqB/dev/mapper/LVMTEST191946pv3 /tmp/LVMTEST191946.ImUMG6dyqB/dev/mapper/LVMTEST191946pv4
> [ 0:13.469]   Using default stripesize 64.00 KiB.
> [ 0:13.483]   Logical volume "LV1" created.
> [ 0:14.042] 6,8908,1194343108,-;device-mapper: raid: Superblocks created for new raid set
> [ 0:14.042] 5,8909,1194348704,-;md/raid10:mdX: not clean -- starting background reconstruction
> [ 0:14.042] 6,8910,1194349443,-;md/raid10:mdX: active with 4 out of 4 devices
> [ 0:14.042] 4,8911,1194459161,-;mdX: bitmap file is out of date, doing full recovery
> [ 0:14.042] 6,8912,1194563810,-;md: resync of RAID array mdX
> [ 0:14.042]   WARNING: This metadata update is NOT backed up.
> [ 0:14.042] aux disable_dev "$dev4"
> [ 0:14.058] #lvconvert-repair-raid.sh:163+ aux disable_dev /tmp/LVMTEST191946.ImUMG6dyqB/dev/mapper/LVMTEST191946pv4
> [ 0:14.058] Disabling device /tmp/LVMTEST191946.ImUMG6dyqB/dev/mapper/LVMTEST191946pv4 (253:5)
> [ 0:14.101] not lvconvert -y --repair $vg/$lv1
> 
> When it fails, I see:
> 
> [ 0:13.831] #lvconvert-repair-raid.sh:161+ lvcreate --type raid10 -m 1 -i 2 -L 64 -n LV1 LVMTEST192248vg /tmp/LVMTEST192248.ATcecgSGfE/dev/mapper/LVMTEST192248pv1 /tmp/LVMTEST192248.ATcecgSGfE/dev/mapper/LVMTEST192248pv2 /tmp/LVMTEST192248.ATcecgSGfE/dev/mapper/LVMTEST192248pv3 /tmp/LVMTEST192248.ATcecgSGfE/dev/mapper/LVMTEST192248pv4
> [ 0:13.831]   Using default stripesize 64.00 KiB.
> [ 0:13.847]   Logical volume "LV1" created.
> [ 0:14.499]   WARNING: This metadata update is NOT backed up.
> [ 0:14.499] 6,8925,1187444256,-;device-mapper: raid: Superblocks created for new raid set
> [ 0:14.499] 5,8926,1187449525,-;md/raid10:mdX: not clean -- starting background reconstruction
> [ 0:14.499] 6,8927,1187450148,-;md/raid10:mdX: active with 4 out of 4 devices
> [ 0:14.499] 6,8928,1187452472,-;md: resync of RAID array mdX
> [ 0:14.499] 6,8929,1187453016,-;md: mdX: resync done.
> [ 0:14.499] 4,8930,1187555486,-;mdX: bitmap file is out of date, doing full recovery
> [ 0:14.499] aux disable_dev "$dev4"
> [ 0:14.515] #lvconvert-repair-raid.sh:163+ aux disable_dev /tmp/LVMTEST192248.AT
> cecgSGfE/dev/mapper/LVMTEST192248pv4
> [ 0:14.515] Disabling device /tmp/LVMTEST192248.ATcecgSGfE/dev/mapper/LVMTEST192
> 248pv4 (253:5)
> [ 0:14.554] not lvconvert -y --repair $vg/$lv1
> 
> To me the important looking difference (and I admit, I'm no RAID expert), is that in the
> case where the test passes (where lvconvert fails as expected), I see
> 
> [ 0:14.042] 4,8911,1194459161,-;mdX: bitmap file is out of date, doing full recovery
> [ 0:14.042] 6,8912,1194563810,-;md: resync of RAID array mdX
> 
> When it fails I see:
> 
> [ 0:14.499] 6,8928,1187452472,-;md: resync of RAID array mdX
> [ 0:14.499] 6,8929,1187453016,-;md: mdX: resync done.
> [ 0:14.499] 4,8930,1187555486,-;mdX: bitmap file is out of date, doing full recovery
> 
> Which appears to show a resync that takes no time, presumable because it happens before
> the device notices that the bitmaps are wrong and schedules a full recovery.
> 
> 
> lvconvert-raid-reshape-stripes-load-reload.sh repeatedly reloads the
> device table during a raid reshape, and then tests the filesystem for
> corruption afterwards. With this patchset, the filesystem is
> occasionally corrupted.  I do not see this with the 6.6 kernel.
> 
> -Ben
>   
>> Xiao Ni also test the last version on a real machine, see [1].
>>
>> [1] https://lore.kernel.org/all/CALTww29QO5kzmN6Vd+jT=-8W5F52tJjHKSgrfUc1Z1ZAeRKHHA@mail.gmail.com/
>>
>> Yu Kuai (14):
>>    md: don't ignore suspended array in md_check_recovery()
>>    md: don't ignore read-only array in md_check_recovery()
>>    md: make sure md_do_sync() will set MD_RECOVERY_DONE
>>    md: don't register sync_thread for reshape directly
>>    md: don't suspend the array for interrupted reshape
>>    md: fix missing release of 'active_io' for flush
>>    md: export helpers to stop sync_thread
>>    md: export helper md_is_rdwr()
>>    dm-raid: really frozen sync_thread during suspend
>>    md/dm-raid: don't call md_reap_sync_thread() directly
>>    dm-raid: add a new helper prepare_suspend() in md_personality
>>    md/raid456: fix a deadlock for dm-raid456 while io concurrent with
>>      reshape
>>    dm-raid: fix lockdep waring in "pers->hot_add_disk"
>>    dm-raid: remove mddev_suspend/resume()
>>
>>   drivers/md/dm-raid.c |  78 +++++++++++++++++++--------
>>   drivers/md/md.c      | 126 +++++++++++++++++++++++++++++--------------
>>   drivers/md/md.h      |  16 ++++++
>>   drivers/md/raid10.c  |  16 +-----
>>   drivers/md/raid5.c   |  61 +++++++++++----------
>>   5 files changed, 192 insertions(+), 105 deletions(-)
>>
>> -- 
>> 2.39.2
>>
> 
> .
>