Received-SPF: pass (google.com: domain of linux-kernel+bounces-75350-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <c9b3466c-7972-47c0-b590-9c1012b53a82@oracle.com>
Date: Wed, 21 Feb 2024 11:15:35 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
Content-Language: en-US
To: Mateusz Kusiak <mateusz.kusiak@linux.intel.com>,
        Dan Moulding <dan@danm.net>
Cc: gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
        linux-raid@vger.kernel.org, regressions@lists.linux.dev,
        song@kernel.org, stable@vger.kernel.org
References: <20240123005700.9302-1-dan@danm.net>
 <20240220230658.11069-1-dan@danm.net>
 <f32cd478-a905-4e98-a46c-0612bc10c38e@oracle.com>
 <2b5daa31-af66-4297-932b-d2fd341b63e6@linux.intel.com>
From: junxiao.bi@oracle.com
In-Reply-To: <2b5daa31-af66-4297-932b-d2fd341b63e6@linux.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?Q0I4ZnlKMkpOWU1saGQ1TXoza3RFa2tWUVUyeDZSREZqSU9uWjl2THZnVng1?=
 =?utf-8?B?VGVGUWs5cTNEUHRud1R3UFhKWTNneUpkMDBDY1ByWkcrSEptVEpHam14QVJh?=
 =?utf-8?B?bFpsanIybkk3YndqWmlwTzBVdzRyZndNYy9FdC9UQ1ZMSEpFNW9CQUNUU1Jr?=
 =?utf-8?B?c2d3bjh2VzRVMk9MamdIL09rVzBrblBaV3ZxMFhaUGNEaDBoMTRwNUhxREwy?=
 =?utf-8?B?UkNCRkNVbHFYRTNrTjVCWnlNekZNRmU1ZVhNL0N4cWJ6ZFlOSlJrKzZ0MTdW?=
 =?utf-8?B?NHhNUjZlZy8rR081UDRBN1pGRTVLVjRQaWEzbUs3Q1dBbzNheVJpSmNIbFBT?=
 =?utf-8?B?WG5YaHQ3ODcvakpCRUpDbGlJYWhMZXFmSUhmWUF5NjQ2eWhwRXhERHVrTjB5?=
 =?utf-8?B?RFArM3F4OUlWZHNoK2JlNW00SFZzRnF6RzRGMGt4NFR2ajZJVXVXd1k2ZFZ2?=
 =?utf-8?B?bGFhbEZxTlpTN0xpQ1c2Q29jSHhuWHVJeURGcExYYVZzK0lSSTVpZmVZNExT?=
 =?utf-8?B?MEM3MmwzUXBFSlc2R0JrOUhJb1BLRlhaYW04ejdZVWppM3B5emRBM00zVXo4?=
 =?utf-8?B?bldtc0NuaFIyUUVSeHBJSDROOW53M2V0aG5zSDkxZ1hWcGptM0x5bHljSDEw?=
 =?utf-8?B?V25rcFRnTnMxcVVveWxOMGlnbGdmU3JZRmc3Rkl1V0JFV1kxdnZFVTlKbVlI?=
 =?utf-8?B?TG1CT1Z5a3Jyem9rc0g3aW9ndTJVQldwaUxWUjFzbWFXQ2d1b0d5S1djMUFX?=
 =?utf-8?B?bHlpdWdaaHVPd1FiMHV6T1ZFL3VWMzhJUnlIdjVoT01xQmI2QXI0SWNnV0s1?=
 =?utf-8?B?a3dER0Exd2pOOXEwMFNOOXB5VFFhRThyL3BCSUZuTk5CUzE4SjVtL1MzWWJx?=
 =?utf-8?B?bFZiOVJEZnhZbFZXL1FsUzU3djNmSTg2eDJJMlFZeW96ajEzdHJVblpiVk1m?=
 =?utf-8?B?OVM3WVBCeHkyc1pROE9OUlBPZkpRWTdqK3pwTCtVZlFqWTN0WHZjak0xbGxk?=
 =?utf-8?B?eExadFU2Z2c0bi9ycFZkVzJiNVR4RytQWFFSeS9PREdoKzgvNlpvZnVpdGt6?=
 =?utf-8?B?Yk84cWxydnZzN21VYklXa2FWVDRoWWMzNmc0MEU3Y2pYdnNwcnJNeDBPVHhG?=
 =?utf-8?B?dGo4RVV6dzFmQ05ZWXBndGF2dVRhREI4NXU0d2N1WDBKMitJY3YwOTV0ZVMr?=
 =?utf-8?B?R2tKVFlWdUNzUmgvSDB3enk3NTlRRDNRTlA3LzhBU2xVQjJPaWUzdkxNV3cv?=
 =?utf-8?B?bWg3NFo0b0k2SXdqSkhvZElwM0RWUVdGQ1M1dGc2RUlveFFpSUowVjRBQTVy?=
 =?utf-8?B?b25uMlBxdW1WVm9sYVNoMEdudzNVSWkwT1poYVpQWG50QWFwUWJaam1qQnJW?=
 =?utf-8?B?T2tFYWNaWWZKdlZYaXhMUEVsSlBPcVFaSTl3NWxkSGdkamhDcE9lbm1qOGo1?=
 =?utf-8?B?TnpRWXcvanZsSmN5V3FKWHNkc1R3WWxFYjhLQTFOdXUwbGEvVXRWb3UyWmR1?=
 =?utf-8?B?MDhvNHlsUXJSSlB2QUV2dDhuV21xNW1ObDR4RTBSYW1LY0FOb3k5bVh3RW5Z?=
 =?utf-8?B?bVhudUsva213MmZWT241WDNQdTdlWTV3SG1PanhxZGNSRERqYnRMaW8vNThm?=
 =?utf-8?B?eGM1K1VPNm9wcjhZaXRqenlxQk0vQ1BxTXZDNmxydjdJWVFQaVpjZCtzV3Bh?=
 =?utf-8?B?SHhSOGJ0aHJvWkNNWnhvdXZLYlJRMldJbnN4cG1UOW81R1UrRmY5WVVJUDF5?=
 =?utf-8?B?QzJvcEc4U20wWWhaZjZYc0w3anIvZkgzTVhZMXEzbno1Um9PK2hNQ05WbGVO?=
 =?utf-8?B?SE8rclN3bG5udXozZ1c0TlBFQ0FiZ0QycjdZUmRESmxSaTdxR3o0TSs1NHJ6?=
 =?utf-8?B?WkhKNnJwZ0wwd1RmUWdwM0lSQkFFL1hGYUVXcDBYKzU0OTBtUmFzdnphZUMv?=
 =?utf-8?B?OFd6VFN4UExYUjltYm16azZWR1RMYU9Vdkh5bVlxUDhMOUhKWU5PQndjTTl5?=
 =?utf-8?B?enFvQjNtUkNqMzlXU1drUTRRc29CQnJBYWVWOXdqVWlUY2xQSUV3Wjh5ZVBt?=
 =?utf-8?B?K3FGODdyTGlqQzI3My95K3ZyYlFEc3Q5VUhWVkpPdzkrWFJSeEJUZlZ3UlpQ?=
 =?utf-8?Q?tIPOOZR8HRHPKyPyo/naS+yM+?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	fjNh04JIq5RIhIJGO7zWLQ5PjyOoednv+oOPpbBwlQTVXForIhHIcIC7TlWWQyO5sH4tYOkAdaXL9weOcBgo8t2Zcd9o5/RKUn8insFmjF7zJTIBOKJ+2iet8Gkm7/2n1mbJhZ/l2f3KNcWUyiyo0wPgSS6BKNJUT5Ju446VjYU3QNfurJa8+hwgy/sFtKPgg3zpE45Zub1s7kfKvDVNN5i3vOFb8m2lw9am4EtrvTXXylyXWy6NbL36nBYlnKPt9YXPKuzyGkXz8n9AgRf8QRsAMjFMRYp02mP4C0O80a++JSfMrK+yQAF5xcmucyApCn32kiVhXfreW0ug18s+ZVkIEIxejajTXCruoiGSM3aRcKiVqxrkfeniotlN87w1QqWFts0InUmnEspdh1SHsfzarC8hgW+h/kmyLrg/3tEU4HucFZvLhR7z9HbEHoOJvqHLWf7IodW5/iEogFXE9agrGqJkFtJa4AXzwYoq2lx+gr0ElWfjm8RX32U7G5zZ2n4Nz8QcllRoQ54WffXXqhqTJyaRHrm7HpE/ZHY9YOowlm/aQros38sdLfgbfMY5fKx7km3Y5jlLwkZ0J+06L4++FmiK0G7FgMPb6AA7xLc=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 396490bf-a0fc-467b-1264-08dc33117c06
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4752.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Feb 2024 19:15:37.0211
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1ieyWhqAS7dXMDQx71clVxPduxWp0+Q0Gxtd9St77sSI1x7fB99581UrnN01vlTz2D4aw6OpZgW+SWX8CvaqoQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5234
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-21_06,2024-02-21_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 malwarescore=0
 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2402210149
X-Proofpoint-GUID: tJJmCJknyKsTqVo80g8Bg1PbZCfthfht
X-Proofpoint-ORIG-GUID: tJJmCJknyKsTqVo80g8Bg1PbZCfthfht

On 2/21/24 6:50 AM, Mateusz Kusiak wrote:
> On 21.02.2024 00:15, junxiao.bi@oracle.com wrote:
>>
>> The thing is we can't reproduce this issue at all. If you can 
>> generate a vmcore when the hung happened, then we can review which 
>> processes are stuck.
>>
> Hi,
> don't know if that be any of help, but I run below scenario with SATA 
> and NVMe drives. For me, the issue is reproducible on NVMe drives only.
>
> Scenario:
> 1. Create R5D3 with native metadata
>     # mdadm -CR /dev/md/vol -l5 -n3 /dev/nvme[0-2]n1 --assume-clean
> 2. Create FS on the array
>     # mkfs.ext4 /dev/md/vol -F
> 3. Remove single member drive via "--incremental --fail"
>     # mdadm -If nvme0n1
>
> The result is almost instant.

This is not the same issue that Dan reported, it looks like another 
regression that Yu Kuai fixed , can you please try this patch?

https://lore.kernel.org/lkml/95f2e08e-2daf-e298-e696-42ebfa7b9bbf@huaweicloud.com/

Thanks,

Junxiao.

>
> Thanks,
> Mateusz