Received-SPF: pass (google.com: domain of linux-kernel+bounces-46912-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <86a8e266-110e-49cb-a156-faa950df3a62@oracle.com>
Date: Wed, 31 Jan 2024 09:37:23 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
Content-Language: en-US
To: Dan Moulding <dan@danm.net>
Cc: gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
        linux-raid@vger.kernel.org, regressions@lists.linux.dev,
        song@kernel.org, stable@vger.kernel.org, yukuai1@huaweicloud.com
References: <2ef7d741-3df8-402a-967f-53ec77c73e2c@oracle.com>
 <20240125203130.28187-1-dan@danm.net>
From: junxiao.bi@oracle.com
In-Reply-To: <20240125203130.28187-1-dan@danm.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?OXFubWFuUys2SFN0Mzl1LzF2c0w4cFlaUkVqT2VzNTlZeVdVc0gxWi9XaC9h?=
 =?utf-8?B?YUNSVkN4MmdiWEhub0hVbXg4SmRRc0pDSHczRkdXS2xWYUZYcFNYaUk3YnlY?=
 =?utf-8?B?NWZDTC9MVHp2a001ZlJoa1ZFME5QcW1vODRoVXd0ZDl3YW93OWs1RGlhbmNN?=
 =?utf-8?B?SHMzNjlGVkFVMTROaUhmejhUZEtUNFYrd1JmTUpYNExWMzlqYk10cUUvQ2tj?=
 =?utf-8?B?SFIrWXdGcUw4YTR4TlVONWlsU0UyYlg3YlVLZUhwL3E2N0c0TnVMVDJUMUpz?=
 =?utf-8?B?SDRycjI4M1lSc0VCai9kUWoyNjRSRFlqQVRBek1tS2RlczR6cHUyaDBsU092?=
 =?utf-8?B?OGg0NDkydjBneTM1SDRjTmN4VDJiMGJUM09HaWNhU0t5b0tQbURGT1hUSnhZ?=
 =?utf-8?B?VmMvY0J1VUJEL3M4b09sQWNjYTBiS2haUE5GVnRFOUdITDFaeS9zeFpvZTJO?=
 =?utf-8?B?V1VIdlNyVFI3Y3d5bjlBVVFZZUM0MFo3KzlXZWU1T1RkeERnelBEZ09rT3RE?=
 =?utf-8?B?U3B2RjVyc2F0QWQxYVducjY3bFZtNXpJeE5WU1dSUE1DR3FkdGNxZUhRZUI0?=
 =?utf-8?B?YjcyZ1U2eXpSVUxxekhQdDY3bHBKeUplSm4wSDFQa3ZJcnc0WXprY2xMcHBu?=
 =?utf-8?B?V21BY2E1MFVKb21FZ3JuZGJiYnhIMDBhbTF5OHVmekx4aHNhQmdneC8rV3pr?=
 =?utf-8?B?RkNYeTdtR1l4WDN5bjZjZGVlQlc4QmxncTY0OE1CWGRERTE0RHZmUWFpditX?=
 =?utf-8?B?dHkwNEcxU1pIci9UbGg0Q0pWdGZmQmc1UFVjVEZnNmZzUWltUTZma3ZQMEgv?=
 =?utf-8?B?VXp5ckNOQ3FDd2Vta1E1OCtteld5QkNpNnZKWCtXSkpvUmFmRXZKZ3M4T2xU?=
 =?utf-8?B?RDFhazhCR1Z1aTEzNlhiVmt4d1FORG00S0x5NWtPL3BmNEhvZmNWbVh0MXlj?=
 =?utf-8?B?T1BqMmZ2UnIraE0ya3prZXhVNFZrWUx1QXZEc1ZzZk0rNThPZC9yUUkzeTJo?=
 =?utf-8?B?M0dJR214NDUxTUVTcWJQYStpOGpmTTU0cDQyNHJaQ0lzS2pYWnpmZ1I3Z2tz?=
 =?utf-8?B?MFcyeVI5cm91M1d5d0VqZzkvWk5IL0NSRnBNaXdzV2ZiNWEydDJEazJyZit0?=
 =?utf-8?B?Y1IraG1TcmpPd3Azc1MvVCtPZmorbE55a0hmN1BkWTA2Ung0Z1dtdmR0eFZD?=
 =?utf-8?B?WEJKNGtacEZjYk5XVFhOMlo1eFI0T0JvZEtVNnJFRWJsUnpzM2RqQVJILzZM?=
 =?utf-8?B?aitHMW9MK3JQV2gvVGtZUHhMOXh6Z1VuZWdlQ01sRExQZjhuZnRvTUN3TUlE?=
 =?utf-8?B?aXlIaCtNOGhFQVNBaGk0Mm5zYmdPNzdwTFc0S3M2V3ZkZlpNakxPWWRlUUFO?=
 =?utf-8?B?NFpueXhIbGNHanRaUWdrTE1XRE5JY0tXVitoRlFzdCs3bTV5cVJ0ZkF3SitX?=
 =?utf-8?B?Nnc5b0YzcVByYjBvdXlLNGtSU0VRQy9FVitUMWdqUEl1S0NrK3RMVHhSYjhh?=
 =?utf-8?B?NTV0ajBJOU82aCtmOGZUYlQ1ZFFpYUM4bTZnOHdSajRtb205Z2h6a1lhMTht?=
 =?utf-8?B?RDJ3NzFBOFJmMzRDaU9XRzN3SUtkWjg2V1AyUldJVDhFS0QzckkyNVhHMVpD?=
 =?utf-8?B?bzFFTWNzS3l4UXhDc0xwZElIdTBFMUhNODJ1WXBOUU5WVFJOWUF6QmZSU2Q5?=
 =?utf-8?B?Z2JRMEMrcW1rK2FFWXNRMHRXYzZCQWgzQXN5S0dvTEY4cnRyN0JJT3owcDBx?=
 =?utf-8?B?VnZadmZpYWcvOWMzR0JMcDA5NTVpQS8veVlKcEt4UGxxbGY1OWVBVnNoYWdi?=
 =?utf-8?B?dGJnRk5ING13OXJKcHMvd1NqdEEwUUV1Q2NOMCtreVlYbTFkRTY5TTVNS1Ex?=
 =?utf-8?B?NjNVc0cvakt6SlArZ3ptOW9EdDU0ckhOdzhFMUg2VE9NVG1JUHZHRTJyOUlD?=
 =?utf-8?B?ZUdqTTQ0SkMyS2RTZnlnTTlMZkRJeEhyWURXdDZGaFBCZ3p6L3FGbE8xeHB5?=
 =?utf-8?B?bXVTYlpSWVVIa2xyMFZtRFRaRitXSjJwRU5IclpUQ0I5UzNPZ3F0bVNNcXRp?=
 =?utf-8?B?V2lJVFBLSTZseVdiVERGa3ZEakYxbFNQS0UyN0p6b0xjcTNTdlZWbHZ5SE5o?=
 =?utf-8?Q?Q1RiIUoMvl1LHXA0ArQFH7P46?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	XAc1W0cgW2lSBrQtwUmM+GhXQwVd9URgwSJFQT3kxi22/kHhry4hZBxspSS1QTT4qqbB10eGtbJNwA+v/SnHQXfWgw+bCDoyZlGrXWRF2x5fjXBTgRkiv3reN4LehKe7rnWLZCmCkn4Ofl+OqDXo0C3uue6zRMwaC+NOvznSYXKBsikWo6RDNKpajZrT6b8vy/kkYt+FrlpmGW9uAPuuZi2bImlgdWDgNHSjB8z4N2OW/ykFthF1LsLXgMI8BDfU4tF/Vn4bcv1UmuhgavrVCnFpnyj3ARLwzh+h403paTthZ6f+B+rxxV6Fyh4sWoiNlYOIqSebJ38csXyvNdQPPN/zPwZ7JG2xuqPnPI2nzCRqVhIIg3d9vVZlsiozExGtufKldwaJxX5MZK+ZOuBnBCxIy67zKS+/Id0vM4yicA7meEk0nWdjIljYmiLdzZ/BujlIzFc4d6R+ZWnqsrvrmRQE7ptPEdbQ6qLw+lBr9c2tZ85ADjfzSlQzZqUuFVdRSFfXz2JV9NaOqsJfUMuXT0mrekCnn3h1HmIE8ytEiPVnUdKiu7Cc0L/Lo37LwicZj6UXfyYgE/J95cy3oqtFonKzCzK9bvHdzDIUsvHs23k=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c206edf2-0d00-48aa-4db9-08dc228349ca
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4752.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2024 17:37:25.5018
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZYuvjtwpm9/NK1fqu5hfqppEZ+0civZinVf/022Gmik0+QezIsehCCSmNItAn/OQJtXgQwkEr4Mj7uKGi6+cSA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6897
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-31_10,2024-01-31_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 phishscore=0 malwarescore=0
 spamscore=0 mlxlogscore=831 adultscore=0 bulkscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2401310136
X-Proofpoint-GUID: RhLseUEWfQXkkDRjTBEcCtvz6t6QWp7R
X-Proofpoint-ORIG-GUID: RhLseUEWfQXkkDRjTBEcCtvz6t6QWp7R

Hi Dan,

On 1/25/24 12:31 PM, Dan Moulding wrote:
> On this Fedora 39 VM, I created a 1GiB LVM volume to use as the RAID-5
> journal from space on the "boot" disk. Then I attached 3 additional
> 100 GiB virtual disks and created the RAID-5 from those 3 disks and
> the write-journal device. I then created a new LVM volume group from
> the md0 array and created one LVM logical volume named "data", using
> all but 64GiB of the available VG space. I then created an ext4 file
> system on the "data" volume, mounted it, and used "dd" to copy 1MiB
> blocks from /dev/urandom to a file on the "data" file system, and just
> let it run. Eventually "dd" hangs and top shows that md0_raid5 is
> using 100% CPU.

I can't reproduce this issue with this test case running over night, dd 
is making progress well. I can see dd is very busy, closing to 100%, 
sometimes it stay in D status, but just for a moment. md5_raid5 is 
staying around 60%, never 100%.

I am wondering your case is a performance issue or a dead hung, if it's 
a hung, i suppose we should see some hung task call trace of dd in dmesg 
if you didn't disable kernel.hung_task_timeout_secs.

Also are you able to configure kdump and trigger a core dump when issue 
reproduced.

Thanks,

Junxiao.