Date: Thu, 1 Dec 2016 12:39:08 +0530
From: Mohammed Shafi Shajakhan <mohammed@codeaurora.org>
To: Benjamin Morgan <bmorgan@cococorp.com>
Cc: linux-wireless@vger.kernel.org, ath10k@lists.infradead.org,
        lede-dev@lists.infradead.org, agreen@cococorp.com
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880
Message-ID: <20161201070908.GB11953@atheros-ThinkPad-T61> (sfid-20161201_080922_461532_9B550A46)
References: <583DD564.8040704@cococorp.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
In-Reply-To: <583DD564.8040704@cococorp.com>
Sender: linux-wireless-owner@vger.kernel.org

Hi,

On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
> When we try to transmit traffic (ping) between two meshed ath10k
> devices running latest lede we keep experiencing ath10k firmware
> crashes. This seems to only happen when running in 802.11n/ac mode
> but not in 802.11a/g mode. Also, from the station dumps it appears
> that management traffic is flowing between the devices, however when
> we try to send unicast data traffic the firmware crashes
> immediately.

[shafi] Did you get a chance to try with the below firmware as well
https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58

> 
> Platform: Archer C7 AC1750 v2
> Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
> 
> Crypto: wpa_supplicant
> wpa_supplicant-wlan0.conf
> network={
> ssid="bmorgan_lede_mesh"
> key_mgmt=SAE
> mode=5
> frequency=5180
> psk="meshpassword"
> }
> 
> Backports Verstion:
> [    9.818007] Loading modules backported from Linux version
> wt-2016-10-03-1-g6fcb1a6
> [    9.825736] Backport generated by backports.git
> backports-20160324-9-g0e38f5c
> 
> ​​Ath10k Initialization on Station A (dmesg)
> [    9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
> [    9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
> 1 irq_mode 0 reset_mode 0
> [   10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
> ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
> [   10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
> [   10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
> firmware_loading_store: map pages failed
> [   10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [   10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [   10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [   10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
> ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
> [   10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
> [   10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
> firmware_loading_store: map pages failed
> [   11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [   12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> 2 cal file max-sta 128 raw 0 hwcrypto 1
> 
> Station A (wlan0):
> 18:A6:F7:23:6E:66
> 10.230.5.41
> 
> Station B (wlan0):
> 18:a6:f7:26:0f:21
> 10.230.5.42
> 
> Station Dump on Station A before ping:
> Station 18:a6:f7:26:0f:21 (on wlan0)
> inactive time:340 ms
> rx bytes:2472
> rx packets:28
> tx bytes:1204
> tx packets:9
> tx retries:0
> tx failed:0
> rx drop misc:1
> signal: -14 dBm
> signal avg:-14 dBm
> Toffset:18142530 us
> tx bitrate:6.0 MBit/s
> rx bitrate:6.0 MBit/s
> rx duration:1524 us
> mesh llid:0
> mesh plid:0
> mesh plink:ESTAB
> mesh local PS mode:ACTIVE
> mesh peer PS mode:UNKNOWN
> mesh non-peer PS mode:ACTIVE
> authorized:yes
> authenticated:yes
> associated:yes
> preamble:long
> WMM/WME:yes
> MFP:yes
> TDLS peer:no
> DTIM period:2
> beacon interval:1000
> connected time:10 seconds
> 
> ​Crash Log on Station B (10.230.5.42)
> [245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> 2bab5ee9-08ff-4a17-95b1-636d212acebc)
> [245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
> cal file max-sta 128 raw 0 hwcrypto 1
> [245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
> [245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A4577 0x00955B31
> [245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> 0x00000002 0x00439E98
> [245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A3518 0x009A3526
> [245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> 0x00000000 0x00000000
> [245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> 0x00400000 0xC09A4577
> [245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> 0x0044110C 0x00442074
> [245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> 0x0044110C 0x00407120
> [245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> 0x0040AC14 0x00001580
> [245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> 0x009C643C 0x004436CC
> [245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> 0x004208FC 0x00439E4C
> [245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> 0x004208FC 0x00425404
> [245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> 0x00424FE8 0x00000002
> [245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [245.767023] ieee80211 phy0: Hardware restart was requested
> [245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
> vdev 1 stop: -143
> [245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> ​
> With wpa_supplicant turned off:
> 
> Station Dump on Station A before ping:
> Station 18:a6:f7:26:0f:21 (on wlan0)
> inactive time:60 ms
> rx bytes:3874
> rx packets:67
> tx bytes:707
> tx packets:7
> tx retries:0
> tx failed:1
> rx drop misc:3
> signal: -14 dBm
> signal avg:-14 dBm
> Toffset:18446744073684250999 us
> tx bitrate:6.0 MBit/s
> rx bitrate:6.0 MBit/s
> rx duration:1312 us
> mesh llid:24953
> mesh plid:59412
> mesh plink:ESTAB
> mesh local PS mode:ACTIVE
> mesh peer PS mode:UNKNOWN
> mesh non-peer PS mode:ACTIVE
> authorized:yes
> authenticated:yes
> associated:yes
> preamble:long
> WMM/WME:yes
> MFP:no
> TDLS peer:no
> DTIM period:2
> beacon interval:1000
> short slot time:yes
> connected time:33 seconds
> 
> Firmware crash on Station A (10.230.5.41)
> [ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
> 42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
> [ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
> 0x4100016c chip_id 0x043202ff sub 0000:0000
> [ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
> tracing 0 dfs 1 testmode 1
> [ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
> [ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
> crc32 bebc7c08
> [ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
> 2 cal file max-sta 128 raw 0 hwcrypto 1
> [ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
> [ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
> 0x009A4577 0x00955B31
> [ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
> 0x00000001 0x00435354
> [ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
> 0x00407120 0x004436CC
> [ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
> 0x009A43A0 0x009A43DE
> [ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
> 0x00000000 0x00000000
> [ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
> 0x0040AC60 0x0040AC09
> [ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
> 0x00955A00 0xC09A4577
> [ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
> 0x0044110C 0x00442074
> [ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
> 0x0044110C 0x00407120
> [ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
> 0x0040AC10 0x00001580
> [ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
> 0x009C643C 0x004436CC
> [ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
> 0x004208FC 0x00439E4C
> [ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
> 0x004208FC 0x00425758
> [ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
> 0x00424FE8 0x00000002
> [ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
> 0x00000004 0x004039D0
> [ 1041.267025] ieee80211 phy0: Hardware restart was requested
> [ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
> monitor vdev 1 stop: -143
> [ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
> 
> ​
> Thanks in advance for your time and help,
> 
> ~Benjamin
>