Received-SPF: pass (google.com: domain of linux-kernel+bounces-37810-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Precedence: bulk
MIME-Version: 1.0
References: <CAPhsuW7KMLHHrcyZhKS_m_fwWSKM66VFXaLj9fmY+ab5Mu3pvA@mail.gmail.com>
 <20240123235803.8298-1-dan@danm.net>
In-Reply-To: <20240123235803.8298-1-dan@danm.net>
From: Song Liu <song@kernel.org>
Date: Wed, 24 Jan 2024 16:01:47 -0800
Message-ID: <CAPhsuW49L8B9K8QFg68v=zG9ywMehUTD18DaG4PexEt-3mzQqQ@mail.gmail.com>
Subject: Re: [REGRESSION] 6.7.1: md: raid5 hang and unresponsive system;
 successfully bisected
To: Dan Moulding <dan@danm.net>, junxiao.bi@oracle.com
Cc: gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, 
	linux-raid@vger.kernel.org, regressions@lists.linux.dev, 
	stable@vger.kernel.org, yukuai1@huaweicloud.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks for the information!


On Tue, Jan 23, 2024 at 3:58=E2=80=AFPM Dan Moulding <dan@danm.net> wrote:
>
> > This appears the md thread hit some infinite loop, so I would like to
> > know what it is doing. We can probably get the information with the
> > perf tool, something like:
> >
> > perf record -a
> > perf report
>
> Here you go!
>
> # Total Lost Samples: 0
> #
> # Samples: 78K of event 'cycles'
> # Event count (approx.): 83127675745
> #
> # Overhead  Command          Shared Object                   Symbol
> # ........  ...............  ..............................  ............=
......................................
> #
>     49.31%  md0_raid5        [kernel.kallsyms]               [k] handle_s=
tripe
>     18.63%  md0_raid5        [kernel.kallsyms]               [k] ops_run_=
io
>      6.07%  md0_raid5        [kernel.kallsyms]               [k] handle_a=
ctive_stripes.isra.0
>      5.50%  md0_raid5        [kernel.kallsyms]               [k] do_relea=
se_stripe
>      3.09%  md0_raid5        [kernel.kallsyms]               [k] _raw_spi=
n_lock_irqsave
>      2.48%  md0_raid5        [kernel.kallsyms]               [k] r5l_writ=
e_stripe
>      1.89%  md0_raid5        [kernel.kallsyms]               [k] md_wakeu=
p_thread
>      1.45%  ksmd             [kernel.kallsyms]               [k] ksm_scan=
_thread
>      1.37%  md0_raid5        [kernel.kallsyms]               [k] stripe_i=
s_lowprio
>      0.87%  ksmd             [kernel.kallsyms]               [k] memcmp
>      0.68%  ksmd             [kernel.kallsyms]               [k] xxh64
>      0.56%  md0_raid5        [kernel.kallsyms]               [k] __wake_u=
p_common
>      0.52%  md0_raid5        [kernel.kallsyms]               [k] __wake_u=
p
>      0.46%  ksmd             [kernel.kallsyms]               [k] mtree_lo=
ad
>      0.44%  ksmd             [kernel.kallsyms]               [k] try_grab=
_page
>      0.40%  ksmd             [kernel.kallsyms]               [k] follow_p=
4d_mask.constprop.0
>      0.39%  md0_raid5        [kernel.kallsyms]               [k] r5l_log_=
disk_error
>      0.37%  md0_raid5        [kernel.kallsyms]               [k] _raw_spi=
n_lock_irq
>      0.33%  md0_raid5        [kernel.kallsyms]               [k] release_=
stripe_list
>      0.31%  md0_raid5        [kernel.kallsyms]               [k] release_=
inactive_stripe_list

It appears the thread is indeed doing something. I haven't got luck to
reproduce this on my hosts. Could you please try whether the following
change fixes the issue (without reverting 0de40f76d567)? I will try to
reproduce the issue on my side.

Junxiao,

Please also help look into this.

Thanks,
Song