Received-SPF: pass (google.com: domain of linux-wireless-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
From:   Nico Sneck <snecknico@gmail.com>
Date:   Mon, 14 Feb 2022 21:25:55 +0200
Message-ID: <CAO_iuKEL8tHnovpGiQGUxg7JUpZFxHpxhOHbqAMgbt5R4Eftgg@mail.gmail.com>
Subject: rtw_8822ce wifi regression after kernel update from 5.15 to 5.16
To:     linux-wireless@vger.kernel.org
Cc:     snecknico@gmail.com
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

Hi,

I'm running Fedora 35 on a Huawei Matestation S (HUAWEI PUM-WDX9), AMD
Renoir with Realtek rtw_8822ce handling wifi stuff.

Ever since the kernel update from 5.15.13-200.fc35 to 5.16.8-200.fc35
(which I performed Feb 12th), I noticed that my Wifi connection
doesn't really work anymore. I'm connecting to a Zyxel VMG3927-B50A,
and it appears to be using 5 GHz connection always. I also tested that
5.17-rc4 also suffers from this issue.

The issue is that even trying to ping my routers gateway address will
result in connection timeouts, and ping times are in the thousands to
tens of thousands of milliseconds (normally peak ping times are ~3-6
ms), making wireless unusable with 5.16+.
I can also see that in dmesg logs there are two types of rtw_8822ce
driver warnings flooding the logs, which I didn't see with 5.15:

"helmi 13 18:20:03 fedora kernel: rtw_8822ce 0000:06:00.0: timed out
to flush queue {1,2}"
"helmi 13 18:16:23 fedora kernel: rtw_8822ce 0000:06:00.0: failed to
get tx report from firmware"

Some stats:
On kernel 5.15.13-200.fc35 running for 29 days:
[nico@fedora ~]$ journalctl -k -b -18 | grep 'timed out to flush queue' | wc -l
0

[nico@fedora ~]$ journalctl -k -b -18 | grep 'failed to get tx report
from firmware' | wc -l
0

On kernel 5.16.8-200.fc35 running for 4 hours:
[nico@fedora ~]$ journalctl -k -b -17 | grep 'timed out to flush queue' | wc -l
45370

[nico@fedora ~]$ journalctl -k -b -17 | grep 'failed to get tx report
from firmware' | wc -l
502

I tried bisecting which commit introduced this regression, but after
some 12 hours of recompiling and testing, it seems like I failed
somehow. I tried a bisect with first known good revision as
8bb7eca972ad (5.15 release commit), and first known bad revision as
df0cc57e057f (5.16 release commit). I managed to identify that
revision
fc02cb2b37fe Merge tag 'net-next-for-5.16' of
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
is bad, but then all other revisions were good apart from
8a33dcc2f6d5 (refs/bisect/bad) Merge
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
which was also bad.
But here's the baffling part, commit 6b278c0cb378 was good, and it's
the last commit in the merge (8a33dcc2f6d5) which appeared bad.
Now I retested with 8a33dcc2f6d5, and I don't see the issues anymore,
so I guess I tested a wrong kernel version at that point or something.
shrug.

So I can only assume that the regression came in one of the commits inside
fc02cb2b37fe Merge tag 'net-next-for-5.16' of
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
but it'll take me a while to be try bisecting the commits in that merge again.

If anyone has any idea about what could cause these issues I'm seeing,
I can try out patches / test different things. But I'll try
rebisecting this again soon.

- Nico