Received-SPF: pass (google.com: domain of linux-kernel+bounces-45628-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
To: Song Liu <song@kernel.org>, Blazej Kucman
 <blazej.kucman@linux.intel.com>, Yu Kuai <yukuai1@huaweicloud.com>
Cc: Dan Moulding <dan@danm.net>, carlos@fisica.ufpr.br,
 gregkh@linuxfoundation.org, junxiao.bi@oracle.com,
 linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org,
 regressions@lists.linux.dev, stable@vger.kernel.org,
 "yukuai (C)" <yukuai3@huawei.com>
References: <ZbMnZnvyIyoWeIro@fisica.ufpr.br>
 <20240126154610.24755-1-dan@danm.net>
 <20240130172524.0000417b@linux.intel.com>
 <CAPhsuW5S7H-P9NiBxk=EVrzYSC6esBxiEg-pR=QMeASWh_S5GA@mail.gmail.com>
From: Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <cb0e6b82-1a09-4539-2d72-f0f1b07e8ea8@huaweicloud.com>
Date: Wed, 31 Jan 2024 10:13:29 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAPhsuW5S7H-P9NiBxk=EVrzYSC6esBxiEg-pR=QMeASWh_S5GA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

在 2024/01/31 9:26, Song Liu 写道:
>> Scenario:
>> 1. Create raid10:
>> mdadm --create /dev/md/r10d4s128-15_A --level=10 --chunk=128
>> --raid-devices=4 /dev/nvme6n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme0n1
>> --size=7864320 --run
>> 2. Create FS
>> mkfs.ext4 /dev/md/r10d4s128-15_A
>> 3. Set faulty one raid member:
>> mdadm --set-faulty /dev/md/r10d4s128-15_A /dev/nvme3n1
> With a failed drive, md_thread calls md_check_recovery() and kicks
> off mddev->sync_work, which is ﻿﻿md_start_sync().
> md_check_recovery() also sets MD_RECOVERY_RUNNING.
> 
> md_start_sync() calls mddev_suspend() and waits for
> mddev->active_io to become zero.
> 
>> 4. Stop raid devies:
>> mdadm -Ss
> This command calls stop_sync_thread() and waits for
> MD_RECOVERY_RUNNING to be cleared.
> 
> Given we need a working file system to reproduce the issue, I
> suspect the problem comes from active_io.

I'll look into this. But I don't understand the root cause yet.
Who grab the 'active_io' and why doesn't release it?

Thanks,
Kuai

> 
> Yu Kuai, I guess we missed this case in the recent refactoring.
> I don't have a good idea to fix this. Please also take a look into
> this.