Received-SPF: pass (google.com: domain of linux-kernel+bounces-38932-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <2ef7d741-3df8-402a-967f-53ec77c73e2c@oracle.com>
Date: Thu, 25 Jan 2024 08:44:07 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
To: Song Liu <song@kernel.org>, Dan Moulding <dan@danm.net>
Cc: gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
        linux-raid@vger.kernel.org, regressions@lists.linux.dev,
        stable@vger.kernel.org, yukuai1@huaweicloud.com
References: <CAPhsuW7KMLHHrcyZhKS_m_fwWSKM66VFXaLj9fmY+ab5Mu3pvA@mail.gmail.com>
 <20240123235803.8298-1-dan@danm.net>
 <CAPhsuW49L8B9K8QFg68v=zG9ywMehUTD18DaG4PexEt-3mzQqQ@mail.gmail.com>
Content-Language: en-US
From: junxiao.bi@oracle.com
In-Reply-To: <CAPhsuW49L8B9K8QFg68v=zG9ywMehUTD18DaG4PexEt-3mzQqQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?a2ZaOW5MdXRXTU9OZFNuL3VacFBYNEZvclNRRmhPYWpjeXZHVjNBQ2NhZkUw?=
 =?utf-8?B?V3JaRkF1blBSRHh4WG9kYzVSWjh0dzg5VEV5Zml3VkcyRndlQWwwM2dCdVg2?=
 =?utf-8?B?R1dDUGNLTXdoOTdIOFdhWDNhcjZ5a29KaGVNTHBZeXFtbDhmZUZPZCs3aWFK?=
 =?utf-8?B?N0NRZWQ4cGcvWXNDSTdPODNhemFiTkg3QUg5eEh3eEVDUXAyc2Y4bGVuK0Y3?=
 =?utf-8?B?eFBxWFUrbHhWa1RLK0Y3RWNLcENZazhIQ3ZpTzZ1OHk2UDlDdHRkRXgxSzdo?=
 =?utf-8?B?dk0yZUJjZkhqNkR4MWVWWFBLSDJadUNHa2RkbzNNM21HRFhwY2lHR2t4Rm1C?=
 =?utf-8?B?VlRDcDAzbnRhQ2cybmNWSlIvWFI5REZkVnoyY2tQL0hCRFV6QzVKU1F0K01W?=
 =?utf-8?B?UHJzUmVGVTZDZ2tTRTVmSkc4cXcyUmlCeXAzdGppcFl2OUlZZXlpc3dEdUZ2?=
 =?utf-8?B?dFlkRFA2eEVlek52dUM4bWp0djJtQ2JUWjVKWjg2Z1cvRitvalROcjIwWXFX?=
 =?utf-8?B?dkhiZDJBRWRsc0xYYUpWQmJ0dXcvc1p6Y2VmbUpBM2tjMmF4bFBidEVETEpp?=
 =?utf-8?B?OTRhS2s5VkVicHlyUUxmRUo3WFFCVWVtVzIvQXB2YWFPM01xQklnTWVxV0tG?=
 =?utf-8?B?V0diMlBTWDJack9lU293c3NmNlpMbGREcjV0MDN4NWhGRk9DYXpUOXNZTjFy?=
 =?utf-8?B?WGJvZFliejBMbEdITUw2SDgrTHZJRzJPaEpMWEMxUlBEZlVyYXNmbkJKRTkz?=
 =?utf-8?B?elB3QlAyTUVEN096YktYcFAyK1k4aUxNckZ1NHZWVXdwbDlGUnZobVJrQnlm?=
 =?utf-8?B?V3VDQ1lJUlBNdjZJOFhBc3Rob01IbUQ0eURxbldsZW5JeXhreHplODFSVmhl?=
 =?utf-8?B?SURwbzloWHhwUkNqZXdYWGhyS2ZkQWxJV091bUZxU1Zpb2RENzVxNkxON3hH?=
 =?utf-8?B?MDc1eCtoUFVLcDdTQU1lNnVMeHI4ekdwVm0xbmVKQi9DRS9LcG1uVkJKUzc3?=
 =?utf-8?B?SUdzRThBeXJYZ29QTzhhdkZHd3NMbzN2aWxpVWF1cFp6YUlhVjNLWDRNY2hz?=
 =?utf-8?B?aElOZ0Q0VlZRTXhYbzlZRlA4VTl0VlJHTmgrWm5pcFdoTHlBYjlEbWZ2NEph?=
 =?utf-8?B?N2REWlArUnFIVk1aYmRyMFVWNDJScVhaQTNKVTFJVFJrMjVCN3pwYWFmRmFa?=
 =?utf-8?B?TVNwNTIwNkxpOXltUTVkOEdEZXpKaWNBd05odUE1TVM5NExNczJEWUsxMGwz?=
 =?utf-8?B?QU9FR0NHajViaHdJRU93d2VjcDFOemNMT3YyME9TK0sza0I4WGlNeVQweDRi?=
 =?utf-8?B?RENhWVFuditLZndISzhVdkY4Yzh1NzVCcUhIYkJmQ29EblpsZlBOVEhLaEkz?=
 =?utf-8?B?RlNuZkdUellQV1AxS0NGdXQvMlVncldPQ1YwdlVXdXcybmZYczdDZ1E4enI5?=
 =?utf-8?B?U0t5RFVHVml2REFFaG5hcEpnSHc0b210TnhPN3hrZTU4aWVHMUZnSzY0MFpt?=
 =?utf-8?B?cG5sK0hmRlhraGszMEFCazd2WkY1cG8yaVgvZ0tBeGFXZkc3OUYzRGlMVjdr?=
 =?utf-8?B?NGhaR3hoUUlCeVR3TTZpUzBIYTVETUxvMmxVaFJPUEhVdU9YZU1NMFE3c21D?=
 =?utf-8?B?MlFLeTh3UVo4YzgxbWQ5eDY0UEhsRFRtWHBieURNV0k5Y0UyWUY0dmc4U3R4?=
 =?utf-8?B?VVB1NzFMQjVjWDJaSWIwa2k5c3NyWGxuQThBZldHbVFmMndCNnZVM214RENK?=
 =?utf-8?B?UnZPTWhFeDJjZkM4Yjk3S0xXUG12VkNaS0oxd0RxYjAraC9waWxzWTZQU3l3?=
 =?utf-8?B?Z0hyMmNaY09iYmpuQ3g3aUN1N1hteVhaWXduZm9yYkVsemZQRkwzcGR0T2lO?=
 =?utf-8?B?aElNMHJ6eUtyNnhvbjN1bGdkN3N2WnROY0FJNjN0amwra1N0eVBZdGJBNHV4?=
 =?utf-8?B?Nkpkelp6ckVuZnhRazFKVVQ0L3YwblpKbDNEaGk2ZThON3RYR2NzVStXUmZq?=
 =?utf-8?B?UjAxZXpnRklqQzJ3dGU1ZnRsRFI0S21zNjlNejNtaG5jRmVtaVdvWHdVN2dZ?=
 =?utf-8?B?OGVYMWdPbnh3UWZJTWYvWmJBTkdCN3FndDl1WEdzc3ZGZnoxK3h3d2g5OElM?=
 =?utf-8?Q?MmOIlHikMCF+CIc81j1lcHRUa?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	iCbMKdWz5V864+D4qeEGqCw8j8I09oKxVONcQowZmhq3Gc0k+K72jtCVeuVJTspuzIRLjhEmZIDMJxgWGHNo1aDU2zsUxxVprDqhLFAvsS9K0Z2TzKaWssJescEfGLkXhScj3yh/FNhP8sx4h45xOyJZzXjUlRy9UYaeSULs+41Tfvrqjk7SBHW007db5p7VnFe39pE5juHumQk49tkI/dj9HiBQPL+xpSPsK+W24zWVWepb7CGC//NQr9Xv+CUnTM3F/Z03agvtQjzCIdkmWwIMakSvEOR2KpVU9KMJzPw4FYmrAzPRHx57xcrrU03YHahmY34kK/DvudSEL1rRPs4vYiL3x5J3g3VtHP38dMTAPgh4DnokkwSoeTKKufB7ckMmaBRCbB0jnBFNPHkZ8+vbOs875vJxP9JoE1v4QpAy5mR+uOjPLLJ1lrBUqX/XoMRjZkHjBowWtKHeUl7u94r1mEiDAXELMFoifnLTqj73oF5zg9DptNKOt1TIlElZXefLG7JA92ulQc3UsDpyvd4Hb1uzPmQ4zJYqQ67r0QVRO25H+ypchyWsjewHQjclQUe/lksMKCC0ND9fsD6yeMiFATi/W6vGkAPMIF+8r+8=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 346a6f74-07ab-420b-7f6b-08dc1dc4da57
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4752.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 16:44:09.5173
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /5To3TVzzjxjpVG80iQg5I3WHVKlwWBQ0StXPKyEonDUks7bAThE18jinzBLvTWnO2ko+8SDK/yei1gyTPePUg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4873
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-25_10,2024-01-25_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 malwarescore=0
 mlxlogscore=999 suspectscore=0 bulkscore=0 adultscore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2401250119
X-Proofpoint-ORIG-GUID: 0nSTJ6IHrDQQJ6H0VMlo5sPbNSlPwVt5
X-Proofpoint-GUID: 0nSTJ6IHrDQQJ6H0VMlo5sPbNSlPwVt5

Hi Dan,

Thanks for the report.

Can you define the hung? No hung task or other error from dmesg, any 
process in D status and what is the call trace if there is? From the 
perf result, looks like the raid thread is doing some real job, it may 
be issuing io since ops_run_io() took around 20% cpu, please share 
"iostat -xz 1" while the workload is running, i am wondering is this 
some performance issue with the workload?

Thanks,

Junxiao.

On 1/24/24 4:01 PM, Song Liu wrote:
> Thanks for the information!
>
>
> On Tue, Jan 23, 2024 at 3:58 PM Dan Moulding <dan@danm.net> wrote:
>>> This appears the md thread hit some infinite loop, so I would like to
>>> know what it is doing. We can probably get the information with the
>>> perf tool, something like:
>>>
>>> perf record -a
>>> perf report
>> Here you go!
>>
>> # Total Lost Samples: 0
>> #
>> # Samples: 78K of event 'cycles'
>> # Event count (approx.): 83127675745
>> #
>> # Overhead  Command          Shared Object                   Symbol
>> # ........  ...............  ..............................  ...................................................
>> #
>>      49.31%  md0_raid5        [kernel.kallsyms]               [k] handle_stripe
>>      18.63%  md0_raid5        [kernel.kallsyms]               [k] ops_run_io
>>       6.07%  md0_raid5        [kernel.kallsyms]               [k] handle_active_stripes.isra.0
>>       5.50%  md0_raid5        [kernel.kallsyms]               [k] do_release_stripe
>>       3.09%  md0_raid5        [kernel.kallsyms]               [k] _raw_spin_lock_irqsave
>>       2.48%  md0_raid5        [kernel.kallsyms]               [k] r5l_write_stripe
>>       1.89%  md0_raid5        [kernel.kallsyms]               [k] md_wakeup_thread
>>       1.45%  ksmd             [kernel.kallsyms]               [k] ksm_scan_thread
>>       1.37%  md0_raid5        [kernel.kallsyms]               [k] stripe_is_lowprio
>>       0.87%  ksmd             [kernel.kallsyms]               [k] memcmp
>>       0.68%  ksmd             [kernel.kallsyms]               [k] xxh64
>>       0.56%  md0_raid5        [kernel.kallsyms]               [k] __wake_up_common
>>       0.52%  md0_raid5        [kernel.kallsyms]               [k] __wake_up
>>       0.46%  ksmd             [kernel.kallsyms]               [k] mtree_load
>>       0.44%  ksmd             [kernel.kallsyms]               [k] try_grab_page
>>       0.40%  ksmd             [kernel.kallsyms]               [k] follow_p4d_mask.constprop.0
>>       0.39%  md0_raid5        [kernel.kallsyms]               [k] r5l_log_disk_error
>>       0.37%  md0_raid5        [kernel.kallsyms]               [k] _raw_spin_lock_irq
>>       0.33%  md0_raid5        [kernel.kallsyms]               [k] release_stripe_list
>>       0.31%  md0_raid5        [kernel.kallsyms]               [k] release_inactive_stripe_list
> It appears the thread is indeed doing something. I haven't got luck to
> reproduce this on my hosts. Could you please try whether the following
> change fixes the issue (without reverting 0de40f76d567)? I will try to
> reproduce the issue on my side.
>
> Junxiao,
>
> Please also help look into this.
>
> Thanks,
> Song