Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\))
Subject: Re: Injecting delays into block layer
From:   Paolo Valente <paolo.valente@linaro.org>
In-Reply-To: <d7ee69fc368db16fa96a05643083674a@natalenko.name>
Date:   Thu, 21 Nov 2019 09:00:33 +0100
Cc:     linux-kernel <linux-kernel@vger.kernel.org>,
        linux-block@vger.kernel.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <3D695D19-B226-4093-9C27-CE561ED08CB7@linaro.org>
References: <d7ee69fc368db16fa96a05643083674a@natalenko.name>
To:     Oleksandr Natalenko <oleksandr@natalenko.name>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> Il giorno 21 nov 2019, alle ore 08:13, Oleksandr Natalenko =
<oleksandr@natalenko.name> ha scritto:
>=20
> Hi Paolo et al.
>=20

Hi

> I have a strong suspect that something is going wrong when the =
underlying block device responds with a large delay. What makes me =
thinking so is that I use a VM on some cloud provider, and they have =
substantial block device latency resulting in permanently high (~20%) =
iowait. It spikes occasionally when their cluster is overloaded, and =
when that happens, the I/O in my VM may stop and never recover. This is =
a rare occasion, but it really happens.
>=20
> What's worse, so far I've seen such a behaviour with BFQ only. I'm =
still testing other schedulers though.
>=20
> Important note: I have no strict evidences that this is *the* case, =
thus I'm asking for some suggestions. My idea is to fire up a local VM =
and inject delays to a block device while performing some I/O from =
within the VM.
>=20
> So the question is: how can those delays be injected? Using dm-delay? =
Can those delays be random?
>=20

So far I have used scsi_debug [1] for this kind of tests.  In my S
suite [2], it boils down to setting SCSI_DEBUG=3Dyes in the S config
file, and then launching any of the benchmarks.  Unfortunately, AFAIK
scsi_debug gives you only constant delays; but you can emulate delay
spikes very easily, by changing the delay parameter manually during
the test.

If this option sounds reasonable to you, then I'm willing to help you
for every step.

Thanks,
Paolo

[1] http://sg.danny.cz/sg/sdebug26.html
[2] https://github.com/Algodev-github/S

> Thanks in advance.
>=20
> --=20
>  Oleksandr Natalenko (post-factum)