Received-SPF: pass (google.com: best guess record for domain of linux-wireless-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: debugging TCP stalls on high-speed wifi
To:     Johannes Berg <johannes@sipsolutions.net>,
        Neal Cardwell <ncardwell@google.com>
Cc:     Eric Dumazet <eric.dumazet@gmail.com>,
        =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= <toke@redhat.com>,
        linux-wireless@vger.kernel.org, Netdev <netdev@vger.kernel.org>
References: <14cedbb9300f887fecc399ebcdb70c153955f876.camel@sipsolutions.net>
 <CADVnQym_CNktZ917q0-9dVY9dhtiJVRRotGTrPNdZUpkjd3vyw@mail.gmail.com>
 <f4670ce0f4399fe82e7168fb9c491d8eb718e8d8.camel@sipsolutions.net>
From:   Eric Dumazet <eric.dumazet@gmail.com>
Message-ID: <99748db5-7898-534b-d407-ed819f07f939@gmail.com>
Date:   Thu, 12 Dec 2019 10:11:14 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.2.2
MIME-Version: 1.0
In-Reply-To: <f4670ce0f4399fe82e7168fb9c491d8eb718e8d8.camel@sipsolutions.net>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-wireless-owner@vger.kernel.org
Precedence: bulk


On 12/12/19 7:47 AM, Johannes Berg wrote:
> Hi Neal,
> 
> On Thu, 2019-12-12 at 10:11 -0500, Neal Cardwell wrote:
>> On Thu, Dec 12, 2019 at 9:50 AM Johannes Berg <johannes@sipsolutions.net> wrote:
>>> If you have any thoughts on this, I'd appreciate it.
>>
>> Thanks for the detailed report!
> 
> Well, it wasn't. For example, I neglected to mention that I have to
> actually use at least 2 TCP streams (but have tried up to 20) in order
> to not run into the gbit link limit on the AP :-)
> 
>> I was curious:
>>
>> o What's the sender's qdisc configuration?
> 
> There's none, mac80211 bypasses qdiscs in favour of its internal TXQs
> with FQ/codel.
> 
>> o Would you be able to log periodic dumps (e.g. every 50ms or 100ms)
>> of the test connection using a recent "ss" binary and "ss -tinm", to
>> hopefully get a sense of buffer parameters, and whether the flow in
>> these cases is being cwnd-limited, pacing-limited,
>> send-buffer-limited, or receive-window-limited?
> 
> Sure, though I'm not sure my ss is recent enough for what you had in
> mind - if not I'll have to rebuild it (it was iproute2-ss190708).
> 
> https://p.sipsolutions.net/3e515625bf13fa69.txt
> 
> Note there are 4 connections (iperf is being used) but two are control
> and two are data. Easy to see the difference really :-)
> 
>> o Would you be able to share a headers-only tcpdump pcap trace?
> 
> I'm not sure how to do headers-only, but I guess -s100 will work.
> 
> https://johannes.sipsolutions.net/files/he-tcp.pcap.xz
> 

Lack of GRO on receiver is probably what is killing performance,
both for receiver (generating gazillions of acks) and sender (to process all these acks)

I had a plan about enabling compressing ACK as I did for SACK
in commit 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d9f4262b7ea41ca9981cc790e37cca6e37c789e

But I have not done it yet.
It is a pity because this would tremendously help wifi I am sure.