Received-SPF: pass (google.com: domain of linux-kernel+bounces-79020-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <f6640cc8-2b66-44e7-b2f7-8ba17bde2281@oracle.com>
Date: Fri, 23 Feb 2024 11:18:02 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
Content-Language: en-US
To: Dan Moulding <dan@danm.net>
Cc: gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org,
        linux-raid@vger.kernel.org, regressions@lists.linux.dev,
        song@kernel.org, stable@vger.kernel.org
References: <f32cd478-a905-4e98-a46c-0612bc10c38e@oracle.com>
 <20240223174452.10209-1-dan@danm.net>
From: junxiao.bi@oracle.com
In-Reply-To: <20240223174452.10209-1-dan@danm.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?aEpHM3hubCt0aC8zbFdRNjVuU3hsT2RiWVlYVTRoaWkwd3lMY3JNU3hoa1Nu?=
 =?utf-8?B?WHRjd2xuNG9tSnlMQ3lSU2p0cWdCalVxOVVVWGtRWlpYQlhHZzJUaXZoN3lM?=
 =?utf-8?B?b2dlSEhJQjhpYlJVMHJIc3NxUlRPMzIzRHdiYU9aTVZYWjYvZWV0ajN5Z0Rv?=
 =?utf-8?B?d1o1a2Q1aWxSRTdlc0MzM1YrU3hBdEVUdDhKeWh6V2U4YU95QS8vSmdnaWph?=
 =?utf-8?B?cnFrTWIyWWRTRFg0OVZSOEcxZXJZVEFuUTZrSFIzUEJCeTRJWk1LbmNuY0F5?=
 =?utf-8?B?Y0xVbXRZYmdRN1UrUkxzMmxTaTVhb3dlb2xuZGErT1VLOU9hWldTd0hpTm1C?=
 =?utf-8?B?bHVxT3hFM3dPeTRkVURwN2V0VFIxREhtYU9MSjhMYlk3cEZjMUJhZ2J4RTlH?=
 =?utf-8?B?cHFJMVZpM2NmQW5KWmpid0RuN0REM2JJaWljQkRreERQM3pRRDRGRVBhZkto?=
 =?utf-8?B?QldaTWo0SkNIMXhIMDkzcXdISi9wTlJwM094TldDSjdSQTdVOWtuSW96MXBr?=
 =?utf-8?B?TzBSNENlK3N2TytyNnRYemZTcmN4MCt3TDJnd25MM0Z0b3cxZ2xPYlY1cHFp?=
 =?utf-8?B?TGVSMmM4MFRlckRGemlKZndmTDhydEF6OXJUMWZPcTEvVytEUVRkbXhsT2NR?=
 =?utf-8?B?Rm9IcEVvZm15K1l3V1ZQbEtsaWxiL20vb2ExM3VVNFFpdU9xV2pSQ0ZNZ3Nw?=
 =?utf-8?B?U25mMHpNdEZhYWxyS0NNWURncXI2bHYyVkl4L0Y5Um9xbjB5UTI5d2dZb0VL?=
 =?utf-8?B?a1FUVmdxeWcwdEduTURBSTFQYWZTZlVZVlFDYzh3WW11Smh1TDhwK3lwMWlh?=
 =?utf-8?B?L3JyQ0J0WUVOajJNUkZnLzZoMWdHckFHOVBIeDZ6QjN1NzV4dU04b290VWdq?=
 =?utf-8?B?b2w0QTRrd1VKZk5VY0J5QmNHZHYzaWJSN0dhaXMxRElMM05DZlg2TnBuOUlY?=
 =?utf-8?B?c0pYcTZrN0ViM2hkeitXclpLcksrSktRRzNFNnNsUVp6NmppMWEySHN0ZjI4?=
 =?utf-8?B?RTQ5SDhtaFJFY0w5Zy9sLy9uc2JQRkxNY1VYcDFuOVdoeEg2a1ZDZlU3ZFdz?=
 =?utf-8?B?Ukt6WURBNUk5dmRGSWFhcTVjKzc5Z0h5cVdQY1ZqUlRHUXY4bmNwTWV5bjdF?=
 =?utf-8?B?UmRPYlE2bkgxbzhYMy9uQ2lZSVlscjBwSXZQT01aSWpiSXNDUjhSNVpvT3VQ?=
 =?utf-8?B?K3JlamxrS3NaV3lMdzlLMUxxNGMvWm1FOTNnb3VVZ05hZ3E5cmdkbTUvcTNB?=
 =?utf-8?B?TlVJREVwYXZycVVwNTRoN1p4UHhnMGpsZW9HMU1uOEJ5TzNGK3VGcGJnUE1L?=
 =?utf-8?B?TDI1cTVrYTNSc3JSbXgzOVNkNTN6aGg4L3hmVllKR0Fmc2dTc2tLWVBobmZX?=
 =?utf-8?B?MndpdDZ6UXhEYTVnM3ZqbUFTZS9FSmVhQjZsc0tZNUN3U1YvUytUelFrQmR1?=
 =?utf-8?B?c2ZrREcybmpJNkl2NENzSEpSbVlIaklVdHBjalh4RGxwVXlVSEVrYlRPcUVz?=
 =?utf-8?B?dmM5d1c2TElodml5MDlRdXc0TkRHbTVpUnFGNUh6V2VqZ3F1SXBKMUM5bCsw?=
 =?utf-8?B?V01rQjlickdpRXBTODFvTm4vYTQvVUdURVQ3Y0pCdkhuWmwzSlVhdWRJQ0hy?=
 =?utf-8?B?VDdoQ29xWW1Zc085M21oc1dVQXlXWTJSMzZhd2ZaU3c4QVg3QlJHWGlGYmd4?=
 =?utf-8?B?Ny95eW02d2RpdGt2QXpyb29CK2k4VVY2MlhCeU5uaXp3T2hFRXd2VldYci9X?=
 =?utf-8?B?YzIvSVVJNGxpaktDeFJldEh4WjhkejNjVnpycnVoT1BFTERVL0hwNTh6SUJF?=
 =?utf-8?B?OXNiQ1lvT3ZWbk9wYW5XbzEwV2NKejlnS3d1WmJCdFlxK0h6SHhJT0pobUpr?=
 =?utf-8?B?U2JsL2FidVFiUFNwM3FYOVI4akJ1QnV6YnAzSEtYVTlaSGZsbWFTV1VrVW9u?=
 =?utf-8?B?dC9GcHVOWm5NaDU5aEZwMFBkb2NyZms1eVZoSG01ZkNrM0kxVTZtbHdnTldF?=
 =?utf-8?B?NGNUNTMwWWRBdGxtUG1Ick5nWmpGVkpmTU94RGJHWUhVYTFRT1h3NGZZcDE2?=
 =?utf-8?B?YloyNEFhNXNjSjdHTzRQMEJZSWdCMXJwRllmQldXcm93TzR5TmI4VG9HRnl6?=
 =?utf-8?Q?tTokN7MBMxKLlXvQY0h8488+0?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	Crx271UmgGD3Vqtsb7vtV6bTeDv9N5ReKBzUdRd9LcOIY89kD98RztQ/pVqHp697Uy7Oa7EgdlV/iwD57/K0IH8Rym+bQpPyIZUyDIbBZxZfIXPyubBPfNn78euDbvLAqna50sVqwQ3+Ujs9tNraEzrDbQamz8FN0XGwk6OtQgx+tuSTHgGbklNWXfW3hKZ2cjX00Mmx6yAbRwGkj6G4HPUrIz4epUyKtHa6+LoMpVW1dWXj7ofFKz4zR8s7g/UxoqpT7hFZDAT7qzEfRBoValtOKHgK0lKrQh4hCYWHq+a47FwS+EmVL/lBSl5BBzIDyn3oiP8XMVentau1mEC/KDl1fzBK0BzqacbPdpANmu7xhwgVfDBQUMn+jb0hrkEpbRRyxP240uhYkPl20+Exq+GVxsaAtCnuc3FD/fdijeAT02BHOFSm6Sf2J0vNG9tTvDsiyGDTQfaC1FZ6nUadAOCzZCugovJaGUJczdO5QvOafupYcVtKtCgyUn3yx+uFhN5DMrqQ+7fzEYmaDtOPONAi4hNuhaz2AjK8rR85KwWn1lB1N+ynAfAQYaB3QnL60kVJYvfWJhvnRRsTISep+WXDjqmjo6vDX/2pK8ZQG7o=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5199c850-4353-499f-150e-08dc34a428d2
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4752.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2024 19:18:04.5474
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: j9hFGeBEiM13b2LM8+7j1QeIEZDxIz8+c0IQ+wauFGDediSFZCCd/c8Rgmoo153+BAgJ2yEIaoDMUrdSweWhnQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4608
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-23_04,2024-02-23_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 adultscore=0
 mlxlogscore=999 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000
 definitions=main-2402230142
X-Proofpoint-GUID: WJKQmPj4JYy-7zCMnhK8r2EzMn8UFo5S
X-Proofpoint-ORIG-GUID: WJKQmPj4JYy-7zCMnhK8r2EzMn8UFo5S

Thanks Dan.

Before we know how to share vmcore, can you run below cmds from crash first:

1. ps -m | grep UN

2. foreach UN bt

3. ps -m | grep md

4. bt each md process

Thanks,

Junxiao.

On 2/23/24 9:44 AM, Dan Moulding wrote:
> Hi Junxiao,
>
> Thanks for your time so far on this problem. It took some time,
> because I've never had to generate a vmcore before, but I have one now
> and it looks usable from what I've seen using crash and gdb on
> it. It's a bit large, 1.1GB. How can I share it? Also, I'm assuming
> you'll also need the vmlinux image that it came from? It's also a bit
> big, 251MB.
>
> -- Dan