Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
DMARC-Filter: OpenDMARC Filter v1.3.2 vulcan.natalenko.name 88D1A2F8C77
From:   Oleksandr Natalenko <oleksandr@natalenko.name>
To:     "David S. Miller" <davem@davemloft.net>
Cc:     Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
        Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        Eric Dumazet <edumazet@google.com>,
        Soheil Hassas Yeganeh <soheil@google.com>,
        Neal Cardwell <ncardwell@google.com>,
        Yuchung Cheng <ycheng@google.com>,
        Van Jacobson <vanj@google.com>, Jerry Chu <hkchu@google.com>
Subject: Re: TCP and BBR: reproducibly low cwnd and bandwidth
Date:   Fri, 16 Feb 2018 16:15:51 +0100
Message-ID: <2189487.nPhU5NAnbi@natalenko.name>
In-Reply-To: <1697118.nv5eASg0nx@natalenko.name>
References: <1697118.nv5eASg0nx@natalenko.name>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi, David, Eric, Neal et al.

On =C4=8Dtvrtek 15. =C3=BAnora 2018 21:42:26 CET Oleksandr Natalenko wrote:
> I've faced an issue with a limited TCP bandwidth between my laptop and a
> server in my 1 Gbps LAN while using BBR as a congestion control mechanism.
> To verify my observations, I've set up 2 KVM VMs with the following
> parameters:
>=20
> 1) Linux v4.15.3
> 2) virtio NICs
> 3) 128 MiB of RAM
> 4) 2 vCPUs
> 5) tested on both non-PREEMPT/100 Hz and PREEMPT/1000 Hz
>=20
> The VMs are interconnected via host bridge (-netdev bridge). I was running
> iperf3 in the default and reverse mode. Here are the results:
>=20
> 1) BBR on both VMs
>=20
> upload: 3.42 Gbits/sec, cwnd ~ 320 KBytes
> download: 3.39 Gbits/sec, cwnd ~ 320 KBytes
>=20
> 2) Reno on both VMs
>=20
> upload: 5.50 Gbits/sec, cwnd =3D 976 KBytes (constant)
> download: 5.22 Gbits/sec, cwnd =3D 1.20 MBytes (constant)
>=20
> 3) Reno on client, BBR on server
>=20
> upload: 5.29 Gbits/sec, cwnd =3D 952 KBytes (constant)
> download: 3.45 Gbits/sec, cwnd ~ 320 KBytes
>=20
> 4) BBR on client, Reno on server
>=20
> upload: 3.36 Gbits/sec, cwnd ~ 370 KBytes
> download: 5.21 Gbits/sec, cwnd =3D 887 KBytes (constant)
>=20
> So, as you may see, when BBR is in use, upload rate is bad and cwnd is lo=
w.
> If using real HW (1 Gbps LAN, laptop and server), BBR limits the throughp=
ut
> to ~100 Mbps (verifiable not only by iperf3, but also by scp while
> transferring some files between hosts).
>=20
> Also, I've tried to use YeAH instead of Reno, and it gives me the same
> results as Reno (IOW, YeAH works fine too).
>=20
> Questions:
>=20
> 1) is this expected?
> 2) or am I missing some extra BBR tuneable?
> 3) if it is not a regression (I don't have any previous data to compare
> with), how can I fix this?
> 4) if it is a bug in BBR, what else should I provide or check for a proper
> investigation?

I've played with BBR a little bit more and managed to narrow the issue down=
 to=20
the changes between v4.12 and v4.13. Here are my observations:

v4.12 + BBR + fq_codel =3D=3D OK
v4.12 + BBR + fq       =3D=3D OK
v4.13 + BBR + fq_codel =3D=3D Not OK
v4.13 + BBR + fq       =3D=3D OK

I think this has something to do with an internal TCP implementation for=20
pacing, that was introduced in v4.13 (commit 218af599fa63) specifically to=
=20
allow using BBR together with non-fq qdiscs. Once BBR relies on fq, the=20
throughput is high and saturates the link, but if another qdisc is in use, =
for=20
instance, fq_codel, the throughput drops. Just to be sure, I've also tried=
=20
pfifo_fast instead of fq_codel with the same outcome resulting in the low=20
throughput.

Unfortunately, I do not know if this is something expected or should be=20
considered as a regression. Thus, asking for an advice.

Ideas?

Thanks.

Regards,
  Oleksandr