Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\))
Subject: Re: Injecting delays into block layer
From:   Paolo Valente <paolo.valente@linaro.org>
In-Reply-To: <3D695D19-B226-4093-9C27-CE561ED08CB7@linaro.org>
Date:   Fri, 6 Dec 2019 17:17:45 +0100
Cc:     linux-kernel <linux-kernel@vger.kernel.org>,
        linux-block <linux-block@vger.kernel.org>,
        SIMONE RICHETTI <206161@studenti.unimore.it>
Content-Transfer-Encoding: quoted-printable
Message-Id: <942604AE-5A91-4E05-869F-74A7EAC5A247@linaro.org>
References: <d7ee69fc368db16fa96a05643083674a@natalenko.name>
 <3D695D19-B226-4093-9C27-CE561ED08CB7@linaro.org>
To:     Oleksandr Natalenko <oleksandr@natalenko.name>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> Il giorno 21 nov 2019, alle ore 09:00, Paolo Valente =
<paolo.valente@linaro.org> ha scritto:
>=20
>=20
>=20
>> Il giorno 21 nov 2019, alle ore 08:13, Oleksandr Natalenko =
<oleksandr@natalenko.name> ha scritto:
>>=20
>> Hi Paolo et al.
>>=20
>=20
> Hi
>=20
>> I have a strong suspect that something is going wrong when the =
underlying block device responds with a large delay. What makes me =
thinking so is that I use a VM on some cloud provider, and they have =
substantial block device latency resulting in permanently high (~20%) =
iowait. It spikes occasionally when their cluster is overloaded, and =
when that happens, the I/O in my VM may stop and never recover. This is =
a rare occasion, but it really happens.
>>=20
>> What's worse, so far I've seen such a behaviour with BFQ only. I'm =
still testing other schedulers though.
>>=20
>> Important note: I have no strict evidences that this is *the* case, =
thus I'm asking for some suggestions. My idea is to fire up a local VM =
and inject delays to a block device while performing some I/O from =
within the VM.
>>=20
>> So the question is: how can those delays be injected? Using dm-delay? =
Can those delays be random?
>>=20
>=20
> So far I have used scsi_debug [1] for this kind of tests.  In my S
> suite [2], it boils down to setting SCSI_DEBUG=3Dyes in the S config
> file, and then launching any of the benchmarks.  Unfortunately, AFAIK
> scsi_debug gives you only constant delays; but you can emulate delay
> spikes very easily, by changing the delay parameter manually during
> the test.
>=20
> If this option sounds reasonable to you, then I'm willing to help you
> for every step.
>=20

Hi Oleksandr,
Simone (in CC) and I have worked a little bit on reproducing the I/O
freeze you report.  Simone made a small change in SCSI_debug, which
makes the latter serve I/O with a highly varying random delay (100ms -
1s), about twice a second.

Then, to generate some fluctuating and heavy I/O, he ran the
comm_startup_lat.sh script of my S suite with SCSI_debug a few times.
Unfortunately, he didn't succeed in reproducing the problem.  If you
want, we can send you a patch with his change for SCSI_debug.

Any news on your side?

Thanks,
Simone

> Thanks,
> Paolo
>=20
> [1] http://sg.danny.cz/sg/sdebug26.html
> [2] https://github.com/Algodev-github/S
>=20
>> Thanks in advance.
>>=20
>> --=20
>> Oleksandr Natalenko (post-factum)