Hi,
I have an Allwinner-H700-based platform (Anbernic RG35XX+) which I am working on mainline u-boot/kernel support for. The RGL8821CS chip on this board works when configured with a 32MHz clock and 3.3v VCC and the RTW88 kernel driver, however the BT is unstable with extensive logging about LPS state entry failures, and the driver will eventually crash with the following:
[ 830.797592] ------------[ cut here ]------------
[ 830.802268] firmware failed to ack driver for entering Deep Power mode
[ 830.808931] WARNING: CPU: 1 PID: 67 at drivers/net/wireless/realtek/rtw88/ps.c:105 rtw_power_mode_change+0x130/0x180 [rtw88_core]
[ 830.820663] Modules linked in: btrfs blake2b_generic libcrc32c xor xor_neon raid6_pq zstd_compress af_alg rtw88_8821cs rtw88_8821c 6
[ 830.846661] CPU: 1 PID: 67 Comm: kworker/u16:4 Not tainted 6.8.0-rc7-next-20240308-00004-gee0f4864bb48 #8
[ 830.856231] Hardware name: Anbernic RG35XX Plus (DT)
[ 830.861198] Workqueue: phy0 rtw_watch_dog_work [rtw88_core]
[ 830.866831] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 830.873796] pc : rtw_power_mode_change+0x130/0x180 [rtw88_core]
[ 830.879763] lr : rtw_power_mode_change+0x130/0x180 [rtw88_core]
[ 830.885730] sp : ffff800082fabcc0
[ 830.889047] x29: ffff800082fabcc0 x28: 0000000000000000 x27: 0000000000000000
[ 830.896192] x26: ffff00000e670900 x25: ffff0000042a2b80 x24: 0000000000000000
[ 830.903337] x23: 0000000000000001 x22: 0000000000068dbc x21: 0000000000000000
[ 830.910482] x20: 0000000000000000 x19: ffff00000e671fc0 x18: 0000000000000006
[ 830.917627] x17: 0000000000000000 x16: 0000000000000080 x15: 076e076907720765
[ 830.924772] x14: 0774076e07650720 x13: ffff8000825a26c0 x12: 0000000000000954
[ 830.931916] x11: 000000000000031c x10: ffff8000825fa6c0 x9 : ffff8000825a26c0
[ 830.939061] x8 : 00000000ffffefff x7 : ffff8000825fa6c0 x6 : 80000000fffff000
[ 830.946205] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 830.953348] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000003264500
[ 830.960493] Call trace:
[ 830.962943] rtw_power_mode_change+0x130/0x180 [rtw88_core]
[ 830.968565] rtw_sdio_deep_ps+0x8c/0xd8 [rtw88_sdio]
[ 830.973547] rtw_enter_lps+0xec/0x148 [rtw88_core]
[ 830.978387] rtw_watch_dog_work+0x298/0x2a4 [rtw88_core]
[ 830.983746] process_one_work+0x148/0x2a0
[ 830.987771] worker_thread+0x2fc/0x40c
[ 830.991530] kthread+0x110/0x114
[ 830.994768] ret_from_fork+0x10/0x20
[ 830.998352] ---[ end trace 0000000000000000 ]---
FW version is
[ 12.561710] rtw_8821cs mmc1:0001:1: Firmware version 24.11.0, H2C version 12
[ 13.471889] Bluetooth: hci0: RTL: fw version 0x75b8f098
Thanks,
Ryan
> -----Original Message-----
> From: Ryan Walklin <[email protected]>
> Sent: Wednesday, March 13, 2024 6:43 AM
> To: Ping-Ke Shih <[email protected]>; [email protected]
> Cc: [email protected]
> Subject: [BUG] RTL8821CS panic on entering power-save mode
>
> Hi,
>
> I have an Allwinner-H700-based platform (Anbernic RG35XX+) which I am working on mainline u-boot/kernel
> support for. The RGL8821CS chip on this board works when configured with a 32MHz clock and 3.3v VCC and
> the RTW88 kernel driver,
>
Did the extensive logging of LPS state happen always? Or happen after a while?
Try module parameter rtw_disable_lps_deep_mode=1 to see if it works well.
The IO of SDIO is slower than PCIE, so maybe we can try to enlarge timeout time
in rtw_power_mode_change():
diff --git a/drivers/net/wireless/realtek/rtw88/ps.c b/drivers/net/wireless/realtek/rtw88/ps.c
index add5a20b8432..d959e3ebba07 100644
--- a/drivers/net/wireless/realtek/rtw88/ps.c
+++ b/drivers/net/wireless/realtek/rtw88/ps.c
@@ -92,7 +92,7 @@ void rtw_power_mode_change(struct rtw_dev *rtwdev, bool enter)
/* Check firmware get the power requset and ack via cpwm register */
ret = read_poll_timeout_atomic(rtw_read8, polling,
(polling ^ confirm) & BIT_RPWM_TOGGLE,
- 100, 15000, true, rtwdev,
+ 100, 15000 * 100, true, rtwdev,
rtwdev->hci.cpwm_addr);
if (ret) {
/* Hit here means that driver failed to get an ack from firmware.
> however the BT is unstable with extensive logging about LPS state entry failures,
> and the driver will eventually crash with the following:
Not sure why this could lead crash, because it is only a warning.
Anyway, please try above suggestions.
Ping-Ke
> Did the extensive logging of LPS state happen always? Or happen after a while?
> Try module parameter rtw_disable_lps_deep_mode=1 to see if it works well.
No after about ~2mins or so. Thanks I will try that.
> The IO of SDIO is slower than PCIE, so maybe we can try to enlarge timeout time
> in rtw_power_mode_change():
Good to know, I will try that patch also.
> Not sure why this could lead crash, because it is only a warning.
Sorry yes you are right, it is just a warning. The WiFi does continue to work.
Thanks for the reply, will try your suggestions and update.
Ryan
On Wed, 13 Mar 2024, at 7:05 PM, Ping-Ke Shih wrote:
> Try module parameter rtw_disable_lps_deep_mode=1 to see if it works well.
Adding 'options rtw88_core disable_lps_deep=1' did help significantly, and bluetooth connections now seem stable. I still get a lot of
[ 267.074209] rtw_8821cs mmc2:0001:1: timed out to flush queue 1
(1-2 per second every few seconds) in dmesg, particularly during downloads over wifi.
> The IO of SDIO is slower than PCIE, so maybe we can try to enlarge timeout time
> in rtw_power_mode_change():
This didn't make any change without disabling the power-saving modes, I also tried increasing some of the other timeouts without an effect.
Ryan
On Fri, 2024-03-15 at 22:30 +1300, Ryan Walklin wrote:
>
> On Wed, 13 Mar 2024, at 7:05 PM, Ping-Ke Shih wrote:
>
> > Try module parameter rtw_disable_lps_deep_mode=1 to see if it works well.
>
> Adding 'options rtw88_core disable_lps_deep=1' did help significantly, and bluetooth connections
> now seem stable. I still get a lot of
>
> [ 267.074209] rtw_8821cs mmc2:0001:1: timed out to flush queue 1
>
> (1-2 per second every few seconds) in dmesg, particularly during downloads over wifi.
This is because it is trying to enter PS mode but TX queue is still not empty.
If power consumption is not big deal to you, try to turn off PS by
iw wlan0 set power_save off
I think messages can be disappear.
>
> > The IO of SDIO is slower than PCIE, so maybe we can try to enlarge timeout time
> > in rtw_power_mode_change():
>
> This didn't make any change without disabling the power-saving modes, I also tried increasing some
> of the other timeouts without an effect.
>
No idea for now. SDIO looks much different from PCIE.
On Sat, 16 Mar 2024, at 2:47 AM, Ping-Ke Shih wrote:
> This is because it is trying to enter PS mode but TX queue is still not empty.
> If power consumption is not big deal to you, try to turn off PS by
> iw wlan0 set power_save off
>
> I think messages can be disappear.
I would prefer to have power-saving on as it is a portable device but this does stop the remaining messages about the queue.
Finally I get extensive logging of
[ 1137.214502] Bluetooth: hci0: Out-of-order packet arrived (0 != 1)
[ 1142.185255] Bluetooth: hci0: Out-of-order packet arrived (1 != 2)
[ 1142.319504] Bluetooth: hci0: Out-of-order packet arrived (2 != 3)
[ 1142.446742] Bluetooth: hci0: Out-of-order packet arrived (3 != 4)
[ 1142.574117] Bluetooth: hci0: Out-of-order packet arrived (4 != 5)
[ 1143.160501] Bluetooth: hci0: Out-of-order packet arrived (5 != 6)
[ 1143.287232] Bluetooth: hci0: Out-of-order packet arrived (6 != 7)
When I am running a bluetooth scan and/or connecting to a device, this seems harmless but does spam dmesg quite a bit.
Thanks,
Ryan