Received-SPF: pass (google.com: domain of linux-kernel+bounces-90476-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Precedence: bulk
MIME-Version: 1.0
References: <20240301095657.662111-1-yukuai1@huaweicloud.com>
 <CALTww2-GV-YBX1ey4juaVHjHz-wNS0xRBqBn=ucDXEb4WNbSOg@mail.gmail.com>
 <0091f7d1-2273-16ff-8285-5fa3f7e2e0f7@huaweicloud.com> <35feaa54-db9e-f0d6-d5a5-a10a45bb90a5@huaweicloud.com>
 <CALTww2-BmudurHbsbbqBMq+KgZs+hokqOJnovS5KDGEidHqZzA@mail.gmail.com> <CALTww2_tP9kZEeReystMVLT+UqE7gpHvLO6yrSJ_bjsfTOuaxQ@mail.gmail.com>
In-Reply-To: <CALTww2_tP9kZEeReystMVLT+UqE7gpHvLO6yrSJ_bjsfTOuaxQ@mail.gmail.com>
From: Xiao Ni <xni@redhat.com>
Date: Mon, 4 Mar 2024 19:06:21 +0800
Message-ID: <CALTww2_8B1XMaFEBtPeWae0Gse7ngqZuuRZMn32BdfW2-M8uYA@mail.gmail.com>
Subject: Re: [PATCH -next 0/9] dm-raid, md/raid: fix v6.7 regressions part2
To: Yu Kuai <yukuai1@huaweicloud.com>
Cc: zkabelac@redhat.com, agk@redhat.com, snitzer@kernel.org, 
	mpatocka@redhat.com, dm-devel@lists.linux.dev, song@kernel.org, 
	heinzm@redhat.com, neilb@suse.de, jbrassow@redhat.com, 
	linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, yi.zhang@huawei.com, 
	yangerkun@huawei.com, "yukuai (C)" <yukuai3@huawei.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Mar 4, 2024 at 4:27=E2=80=AFPM Xiao Ni <xni@redhat.com> wrote:
>
> On Mon, Mar 4, 2024 at 9:25=E2=80=AFAM Xiao Ni <xni@redhat.com> wrote:
> >
> > On Mon, Mar 4, 2024 at 9:24=E2=80=AFAM Yu Kuai <yukuai1@huaweicloud.com=
> wrote:
> > >
> > > Hi,
> > >
> > > =E5=9C=A8 2024/03/04 9:07, Yu Kuai =E5=86=99=E9=81=93:
> > > > Hi,
> > > >
> > > > =E5=9C=A8 2024/03/03 21:16, Xiao Ni =E5=86=99=E9=81=93:
> > > >> Hi all
> > > >>
> > > >> There is a error report from lvm regression tests. The case is
> > > >> lvconvert-raid-reshape-stripes-load-reload.sh. I saw this error wh=
en I
> > > >> tried to fix dmraid regression problems too. In my patch set,  aft=
er
> > > >> reverting ad39c08186f8a0f221337985036ba86731d6aafe (md: Don't regi=
ster
> > > >> sync_thread for reshape directly), this problem doesn't appear.
> > > >
> >
> > Hi Kuai
> > > > How often did you see this tes failed? I'm running the tests for ov=
er
> > > > two days now, for 30+ rounds, and this test never fail in my VM.
> >
> > I ran 5 times and it failed 2 times just now.
> >
> > >
> > > Take a quick look, there is still a path from raid10 that
> > > MD_RECOVERY_FROZEN can be cleared, and in theroy this problem can be
> > > triggered. Can you test the following patch on the top of this set?
> > > I'll keep running the test myself.
> >
> > Sure, I'll give the result later.
>
> Hi all
>
> It's not stable to reproduce this. After applying this raid10 patch it
> failed once 28 times. Without the raid10 patch, it failed once 30
> times, but it failed frequently this morning.

Hi all

After running 152 times with kernel 6.6, the problem can appear too.
So it can return the state of 6.6. This patch set can make this
problem appear quickly.

Best Regards
Xiao


>
> Regards
> Xiao
> >
> > Regards
> > Xiao
> > >
> > > diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> > > index a5f8419e2df1..7ca29469123a 100644
> > > --- a/drivers/md/raid10.c
> > > +++ b/drivers/md/raid10.c
> > > @@ -4575,7 +4575,8 @@ static int raid10_start_reshape(struct mddev *m=
ddev)
> > >          return 0;
> > >
> > >   abort:
> > > -       mddev->recovery =3D 0;
> > > +       if (mddev->gendisk)
> > > +               mddev->recovery =3D 0;
> > >          spin_lock_irq(&conf->device_lock);
> > >          conf->geo =3D conf->prev;
> > >          mddev->raid_disks =3D conf->geo.raid_disks;
> > >
> > > Thanks,
> > > Kuai
> > > >
> > > > Thanks,
> > > > Kuai
> > > >
> > > >>
> > > >> I put the log in the attachment.
> > > >>
> > > >> On Fri, Mar 1, 2024 at 6:03=E2=80=AFPM Yu Kuai <yukuai1@huaweiclou=
d.com> wrote:
> > > >>>
> > > >>> From: Yu Kuai <yukuai3@huawei.com>
> > > >>>
> > > >>> link to part1:
> > > >>> https://lore.kernel.org/all/CAPhsuW7u1UKHCDOBDhD7DzOVtkGemDz_QnJ4=
DUq_kSN-Q3G66Q@mail.gmail.com/
> > > >>>
> > > >>>
> > > >>> part1 contains fixes for deadlocks for stopping sync_thread
> > > >>>
> > > >>> This set contains fixes:
> > > >>>   - reshape can start unexpected, cause data corruption, patch 1,=
5,6;
> > > >>>   - deadlocks that reshape concurrent with IO, patch 8;
> > > >>>   - a lockdep warning, patch 9;
> > > >>>
> > > >>> I'm runing lvm2 tests with following scripts with a few rounds no=
w,
> > > >>>
> > > >>> for t in `ls test/shell`; do
> > > >>>          if cat test/shell/$t | grep raid &> /dev/null; then
> > > >>>                  make check T=3Dshell/$t
> > > >>>          fi
> > > >>> done
> > > >>>
> > > >>> There are no deadlock and no fs corrupt now, however, there are s=
till
> > > >>> four
> > > >>> failed tests:
> > > >>>
> > > >>> ###       failed: [ndev-vanilla] shell/lvchange-raid1-writemostly=
sh
> > > >>> ###       failed: [ndev-vanilla] shell/lvconvert-repair-raid.sh
> > > >>> ###       failed: [ndev-vanilla] shell/lvcreate-large-raid.sh
> > > >>> ###       failed: [ndev-vanilla] shell/lvextend-raid.sh
> > > >>>
> > > >>> And failed reasons are the same:
> > > >>>
> > > >>> ## ERROR: The test started dmeventd (147856) unexpectedly
> > > >>>
> > > >>> I have no clue yet, and it seems other folks doesn't have this is=
sue.
> > > >>>
> > > >>> Yu Kuai (9):
> > > >>>    md: don't clear MD_RECOVERY_FROZEN for new dm-raid until resum=
e
> > > >>>    md: export helpers to stop sync_thread
> > > >>>    md: export helper md_is_rdwr()
> > > >>>    md: add a new helper reshape_interrupted()
> > > >>>    dm-raid: really frozen sync_thread during suspend
> > > >>>    md/dm-raid: don't call md_reap_sync_thread() directly
> > > >>>    dm-raid: add a new helper prepare_suspend() in md_personality
> > > >>>    dm-raid456, md/raid456: fix a deadlock for dm-raid456 while io
> > > >>>      concurrent with reshape
> > > >>>    dm-raid: fix lockdep waring in "pers->hot_add_disk"
> > > >>>
> > > >>>   drivers/md/dm-raid.c | 93 ++++++++++++++++++++++++++++++++++---=
-------
> > > >>>   drivers/md/md.c      | 73 ++++++++++++++++++++++++++--------
> > > >>>   drivers/md/md.h      | 38 +++++++++++++++++-
> > > >>>   drivers/md/raid5.c   | 32 ++++++++++++++-
> > > >>>   4 files changed, 196 insertions(+), 40 deletions(-)
> > > >>>
> > > >>> --
> > > >>> 2.39.2
> > > >>>
> > > >
> > > >
> > > > .
> > > >
> > >