Received-SPF: pass (google.com: domain of linux-kernel+bounces-45585-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
MIME-Version: 1.0
References: <ZbMnZnvyIyoWeIro@fisica.ufpr.br> <20240126154610.24755-1-dan@danm.net>
 <20240130172524.0000417b@linux.intel.com>
In-Reply-To: <20240130172524.0000417b@linux.intel.com>
From: Song Liu <song@kernel.org>
Date: Tue, 30 Jan 2024 17:26:45 -0800
Message-ID: <CAPhsuW5S7H-P9NiBxk=EVrzYSC6esBxiEg-pR=QMeASWh_S5GA@mail.gmail.com>
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
To: Blazej Kucman <blazej.kucman@linux.intel.com>, Yu Kuai <yukuai1@huaweicloud.com>
Cc: Dan Moulding <dan@danm.net>, carlos@fisica.ufpr.br, gregkh@linuxfoundation.org, 
	junxiao.bi@oracle.com, linux-kernel@vger.kernel.org, 
	linux-raid@vger.kernel.org, regressions@lists.linux.dev, 
	stable@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Update my findings so far.

On Tue, Jan 30, 2024 at 8:27=E2=80=AFAM Blazej Kucman
<blazej.kucman@linux.intel.com> wrote:
[...]
> Our daily tests directed at mdadm/md also detected a problem with
> identical symptoms as described in the thread.
>
> Issue detected with IMSM metadata but it also reproduces with native
> metadata.
> NVMe disks under VMD controller were used.
>
> Scenario:
> 1. Create raid10:
> mdadm --create /dev/md/r10d4s128-15_A --level=3D10 --chunk=3D128
> --raid-devices=3D4 /dev/nvme6n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme0n1
> --size=3D7864320 --run
> 2. Create FS
> mkfs.ext4 /dev/md/r10d4s128-15_A
> 3. Set faulty one raid member:
> mdadm --set-faulty /dev/md/r10d4s128-15_A /dev/nvme3n1

With a failed drive, md_thread calls md_check_recovery() and kicks
off mddev->sync_work, which is =EF=BB=BF=EF=BB=BFmd_start_sync().
md_check_recovery() also sets MD_RECOVERY_RUNNING.

md_start_sync() calls mddev_suspend() and waits for
mddev->active_io to become zero.

> 4. Stop raid devies:
> mdadm -Ss

This command calls stop_sync_thread() and waits for
MD_RECOVERY_RUNNING to be cleared.

Given we need a working file system to reproduce the issue, I
suspect the problem comes from active_io.

Yu Kuai, I guess we missed this case in the recent refactoring.
I don't have a good idea to fix this. Please also take a look into
this.

Thanks,
Song