Received-SPF: pass (google.com: domain of linux-wireless-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: mac80211 mlme connection probing woes (bisected)
From:   Ben Greear <greearb@candelatech.com>
To:     "linux-wireless@vger.kernel.org" <linux-wireless@vger.kernel.org>,
        Felix Fietkau <nbd@nbd.name>
References: <312fa32e-21f7-fcc0-fef4-55633db53f65@candelatech.com>
Organization: Candela Technologies
Message-ID: <af486fdb-5ac4-8d89-5d5e-554b828435bf@candelatech.com>
Date:   Tue, 30 Mar 2021 12:19:14 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.0
MIME-Version: 1.0
In-Reply-To: <312fa32e-21f7-fcc0-fef4-55633db53f65@candelatech.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-MW
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 3/26/21 5:18 PM, Ben Greear wrote:
> I've been digging into a bug where our ath10k driver shows periodic
> throughput drops on regular intervals.  We've bisected this down to a patch
> where we disable the firmware connection monitor, and so ask mac80211 to
> do the connection monitor.
> 
> This works fine in 5.4 kernel, but in 5.11, it does not work well.
> 
> First, if anyone has an idea what change might have caused this,
> please let me know.  We will try with ath9k, assuming it uses
> the mac80211 connection monitor to see if it has the same issue.

Ok, it took a while, but I bisected to this:

commit 9abf4e49830d606f18a05111cfa96b8f0b724c7d (HEAD, refs/bisect/good-9abf4e49830d606f18a05111cfa96b8f0b724c7d)
Author: Felix Fietkau <nbd@nbd.name>
Date:   Tue Sep 8 14:36:56 2020 +0200

     mac80211: optimize station connection monitor

     Calling mod_timer for every rx/tx packet can be quite expensive.
     Instead of constantly updating the timer, we can simply let it run out
     and check the timestamp of the last ACK or rx packet to re-arm it.

     Signed-off-by: Felix Fietkau <nbd@nbd.name>
     Link: https://lore.kernel.org/r/20200908123702.88454-9-nbd@nbd.name
     Signed-off-by: Johannes Berg <johannes.berg@intel.com>

To do the bisect, I copied my ath10k-ct driver from the 5.4 kernel (well tested driver)
over whatever ath10k code was in the particular kernel commit I was testing.  I tweaked
the driver slightly to compile and work against stock kernel.

The failure case is that when in station mode, and transmitting UDP in upload direction
(with a few packets per second of download traffic too),
the traffic periodically goes to zero throughput every 30 seconds, and stays quiesced
for about 5 seconds, and then resumes.  The station stays connected.

In previous debugging, I noticed this only happens when my driver enables mac80211
connection monitoring.  In a different bisect attempt, my driver hit the issue when
changing how tx-descriptor count was configured, but I am not fully confident that
is a root cause, and changing things a bit made that problem go away.

The problem is not seen with ath9k, nor stock ath10k.  Stock ath10k uses in-firmware
connection monitoring.

Felix, if you have any ideas of likely failure points, please let me know.

Thanks,
Ben