Received-SPF: pass (google.com: best guess record for domain of linux-wireless-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Message-ID: <9bcbab4b562669b96198c632f476b1b74956ca09.camel@sipsolutions.net>
Subject: Re: debugging TXQs being empty
From:   Johannes Berg <johannes@sipsolutions.net>
To:     Kan Yan <kyan@google.com>
Cc:     Toke =?ISO-8859-1?Q?H=F8iland-J=F8rgensen?= <toke@redhat.com>,
        linux-wireless <linux-wireless@vger.kernel.org>
Date:   Fri, 06 Dec 2019 12:49:24 +0100
In-Reply-To: <bf0cced86a1466285f74eb845e73fbaea1dff1c0.camel@sipsolutions.net> (sfid-20191206_101305_773178_ACF584A9)
References: <bbc516f28782175b27ac5e19dcdeac13cd6ee76a.camel@sipsolutions.net>
         <fd23a26dea59128ede8c1c4d02fb2f3514ffb5e9.camel@sipsolutions.net>
         <CA+iem5tjTpO_2MKL_pEu7enTa-8=g5vY3=2WJKjg9f=JA2eCEw@mail.gmail.com>
         (sfid-20191206_020554_916514_C4D7D41E) <9b89b3b294295063aec045b9e863a44ad20b8782.camel@sipsolutions.net>
         (sfid-20191206_094144_773877_4A5AF79B) <bf0cced86a1466285f74eb845e73fbaea1dff1c0.camel@sipsolutions.net>
         (sfid-20191206_101305_773178_ACF584A9)
Content-Type: text/plain; charset="UTF-8"
User-Agent: Evolution 3.34.2 (3.34.2-1.fc31) 
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-wireless-owner@vger.kernel.org
Precedence: bulk

On Fri, 2019-12-06 at 10:12 +0100, Johannes Berg wrote:
> On Fri, 2019-12-06 at 09:41 +0100, Johannes Berg wrote:
> > Maybe somehow TSO is interacting badly with the TXQs and the tracking
> > here, since TSO makes the traffic *very* bursty? A 64k packet in the
> > driver will typically expand to 9 or 10 A-MSDUs I think?
> 
> No, that all seems well. Without TSO (with the trivial mac80211 patch to
> let me turn it off with ethtool) I get about 890Mbps, so about 5% less.
> That's not actually *that* bad, I guess due to software A-MSDU in
> mac80211, but it's not really the right direction :)
> 
> Changing wmem_max/tcp_mem to outrageous values also didn't really make
> any difference.
> 
> I guess it's time to see if I can poke into the TCP stack to figure out
> what's going on...

Sadly no functioning kprobes on the system ... bpftrace -l lists them,
but can't actually use them.

If I also change net.ipv4.tcp_limit_output_bytes to an outrageous value
(10x) I can recover a bit more than half of the performance loss with
TSO disabled, but it makes no real difference with TSO enabled.

Either way, what bothers me somewhat is that the backlog fluctuates so
much. Sometimes I see a backlock of 2MB or more, while it *still*
manages to go completely empty.

Shouldn't I expect the steady state to have a somewhat even backlog? Why
does this vary so much?

johannes