Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <e38aa76d-6034-4dde-8624-df1745bb17fc@www.fastmail.com>
 <YvPvghdv6lzVRm/S@localhost.localdomain> <2220d403-e443-4e60-b7c3-d149e402c13e@www.fastmail.com>
 <cb1521d5-8b07-48d8-8b88-ca078828cf69@www.fastmail.com> <ad78a32c-7790-4e21-be9f-81c5848a4953@www.fastmail.com>
 <e36fe80f-a33b-4750-b593-3108ba169611@www.fastmail.com>
In-Reply-To: <e36fe80f-a33b-4750-b593-3108ba169611@www.fastmail.com>
From:   Josef Bacik <josef@toxicpanda.com>
Date:   Fri, 12 Aug 2022 13:59:19 -0400
Message-ID: <CAEzrpqe3rRTvH=s+-aXTtupn-XaCxe0=KUe_iQfEyHWp-pXb5w@mail.gmail.com>
Subject: Re: stalling IO regression since linux 5.12, through 5.18
To:     Chris Murphy <lists@colorremedies.com>
Cc:     Paolo Valente <paolo.valente@linaro.org>,
        Btrfs BTRFS <linux-btrfs@vger.kernel.org>,
        Linux-RAID <linux-raid@vger.kernel.org>,
        linux-block <linux-block@vger.kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Fri, Aug 12, 2022 at 12:05 PM Chris Murphy <lists@colorremedies.com> wro=
te:
>
>
>
> On Wed, Aug 10, 2022, at 3:34 PM, Chris Murphy wrote:
> > Booted with cgroup_disable=3Dio, and confirmed cat
> > /sys/fs/cgroup/cgroup.controllers does not list io.
>
> The problem still reproduces with the cgroup IO controller disabled.
>
> On a whim, I decided to switch the IO scheduler from Fedora's default bfq=
 for rotating drives to mq-deadline. The problem does not reproduce for 15+=
 hours, which is not 100% conclusive but probably 99% conclusive. I then sw=
itched live while running the workload to bfq on all eight drives, and with=
in 10 minutes the system cratered, all new commands just hang. Load average=
 goes to triple digits, i/o wait increasing, i/o pressure for the workload =
tasks to 100%, and IO completely stalls to zero. I was able to switch only =
two of the drive queues back to mq-deadline and then lost responsivness in =
that shell and had to issue sysrq+b...
>
> Before that I was able to extra sysrq+w and sysrq+t.
> https://drive.google.com/file/d/16hdQjyBnuzzQIhiQT6fQdE0nkRQJj7EI/view?us=
p=3Dsharing
>
> I can't tell if this is a bfq bug, or if there's some negative interactio=
n between bfq and scsi or megaraid_sas. Obviously it's rare because otherwi=
se people would have been falling over this much sooner. But at this point =
there's strong correlation that it's bfq related and is a kernel regression=
 that's been around since 5.12.0 through 5.18.0, and I suspect also 5.19.0 =
but it's being partly masked by other improvements.

This matches observations we've had internally (inside Facebook) as
well as my continual integration performance testing.  It should
probably be looked into by the BFQ guys as it was working previously.
Thanks,

Josef