Subject: Re: ath10k firmware crashes in mesh mode on QCA9880
To: Mohammed Shafi Shajakhan <mohammed@codeaurora.org>
References: <583DD564.8040704@cococorp.com>
 <20161201070908.GB11953@atheros-ThinkPad-T61>
Cc: linux-wireless@vger.kernel.org, ath10k@lists.infradead.org,
        lede-dev@lists.infradead.org, agreen@cococorp.com
From: Benjamin Morgan <bmorgan@cococorp.com>
Message-ID: <58421FA2.2020004@cococorp.com> (sfid-20161203_022809_673631_3F60F7F0)
Date: Fri, 2 Dec 2016 17:28:02 -0800
MIME-Version: 1.0
In-Reply-To: <20161201070908.GB11953@atheros-ThinkPad-T61>
Content-Type: text/plain; charset=utf-8; format=flowed
Sender: linux-wireless-owner@vger.kernel.org

Just tried 10.2.4.70.58 firmware that you linked to and it still crashes:

[  131.568989] ath10k_pci 0000:01:00.0: firmware crashed! (uuid 
1838347e-9380-4a26-ac9d-2963ee95968b)
[  131.578124] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[  131.587491] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 
tracing 0 dfs 1 testmode 1
[  131.600521] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.58 api 5 
features no-p2p,raw-mode,mfp crc32 e1af076f
[  131.610899] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[  131.618325] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 
cal file max-sta 128 raw 0 hwcrypto 1
[  131.629965] ath10k_pci 0000:01:00.0: firmware register dump:
[  131.635728] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3 
0x009A45AF 0x00955B31
[  131.643761] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130 
0x00000002 0x00439E98
[  131.651806] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074 
0x00407120 0x004436CC
[  131.659852] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000 
0x009A3550 0x009A355E
[  131.667892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A31D6 
0x00000000 0x00000000
[  131.675936] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4 
0x0040AC60 0x0040AC09
[  131.683968] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24 
0x00400000 0xC09A45AF
[  131.692013] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84 
0x0044110C 0x00442074
[  131.700056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4 
0x0044110C 0x00407120
[  131.708100] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4 
0x0040AC14 0x00001580
[  131.716143] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04 
0x009C6458 0x004436CC
[  131.724175] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64 
0x004208FC 0x00439E4C
[  131.732220] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84 
0x004208FC 0x00425874
[  131.740263] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4 
0x00424FE8 0x00000002
[  131.748306] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14 
0x00000004 0x004039D0
[  131.857076] ieee80211 phy0: Hardware restart was requested
[  131.862705] ath10k_pci 0000:01:00.0: failed to synchronize monitor 
vdev 1 stop: -143
[  131.870594] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143

~Benjamin

On 11/30/2016 11:09 PM, Mohammed Shafi Shajakhan wrote:
> Hi,
>
> On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
>> When we try to transmit traffic (ping) between two meshed ath10k
>> devices running latest lede we keep experiencing ath10k firmware
>> crashes. This seems to only happen when running in 802.11n/ac mode
>> but not in 802.11a/g mode. Also, from the station dumps it appears
>> that management traffic is flowing between the devices, however when
>> we try to send unicast data traffic the firmware crashes
>> immediately.
> [shafi] Did you get a chance to try with the below firmware as well
> https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58
>
>> Platform: Archer C7 AC1750 v2
>> Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
>>
>> Crypto: wpa_supplicant
>> wpa_supplicant-wlan0.conf
>> network={
>> ssid="bmorgan_lede_mesh"
>> key_mgmt=SAE
>> mode=5
>> frequency=5180
>> psk="meshpassword"
>> }
>>
>> Backports Verstion:
>> [    9.818007] Loading modules backported from Linux version
>> wt-2016-10-03-1-g6fcb1a6
>> [    9.825736] Backport generated by backports.git
>> backports-20160324-9-g0e38f5c
>>
>> ​​Ath10k Initialization on Station A (dmesg)
>> [    9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
>> [    9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
>> 1 irq_mode 0 reset_mode 0
>> [   10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
>> ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
>> [   10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
>> [   10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
>> firmware_loading_store: map pages failed
>> [   10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [   10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [   10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [   10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
>> ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
>> [   10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
>> [   10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
>> firmware_loading_store: map pages failed
>> [   11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [   12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>
>> Station A (wlan0):
>> 18:A6:F7:23:6E:66
>> 10.230.5.41
>>
>> Station B (wlan0):
>> 18:a6:f7:26:0f:21
>> 10.230.5.42
>>
>> Station Dump on Station A before ping:
>> Station 18:a6:f7:26:0f:21 (on wlan0)
>> inactive time:340 ms
>> rx bytes:2472
>> rx packets:28
>> tx bytes:1204
>> tx packets:9
>> tx retries:0
>> tx failed:0
>> rx drop misc:1
>> signal: -14 dBm
>> signal avg:-14 dBm
>> Toffset:18142530 us
>> tx bitrate:6.0 MBit/s
>> rx bitrate:6.0 MBit/s
>> rx duration:1524 us
>> mesh llid:0
>> mesh plid:0
>> mesh plink:ESTAB
>> mesh local PS mode:ACTIVE
>> mesh peer PS mode:UNKNOWN
>> mesh non-peer PS mode:ACTIVE
>> authorized:yes
>> authenticated:yes
>> associated:yes
>> preamble:long
>> WMM/WME:yes
>> MFP:yes
>> TDLS peer:no
>> DTIM period:2
>> beacon interval:1000
>> connected time:10 seconds
>>
>> ​Crash Log on Station B (10.230.5.42)
>> [245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>> 2bab5ee9-08ff-4a17-95b1-636d212acebc)
>> [245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
>> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
>> cal file max-sta 128 raw 0 hwcrypto 1
>> [245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
>> [245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>> 0x009A4577 0x00955B31
>> [245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>> 0x00000002 0x00439E98
>> [245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>> 0x00407120 0x004436CC
>> [245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>> 0x009A3518 0x009A3526
>> [245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>> 0x00000000 0x00000000
>> [245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>> 0x0040AC60 0x0040AC09
>> [245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>> 0x00400000 0xC09A4577
>> [245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>> 0x0044110C 0x00442074
>> [245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>> 0x0044110C 0x00407120
>> [245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>> 0x0040AC14 0x00001580
>> [245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>> 0x009C643C 0x004436CC
>> [245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>> 0x004208FC 0x00439E4C
>> [245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>> 0x004208FC 0x00425404
>> [245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>> 0x00424FE8 0x00000002
>> [245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>> 0x00000004 0x004039D0
>> [245.767023] ieee80211 phy0: Hardware restart was requested
>> [245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
>> vdev 1 stop: -143
>> [245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>> ​
>> With wpa_supplicant turned off:
>>
>> Station Dump on Station A before ping:
>> Station 18:a6:f7:26:0f:21 (on wlan0)
>> inactive time:60 ms
>> rx bytes:3874
>> rx packets:67
>> tx bytes:707
>> tx packets:7
>> tx retries:0
>> tx failed:1
>> rx drop misc:3
>> signal: -14 dBm
>> signal avg:-14 dBm
>> Toffset:18446744073684250999 us
>> tx bitrate:6.0 MBit/s
>> rx bitrate:6.0 MBit/s
>> rx duration:1312 us
>> mesh llid:24953
>> mesh plid:59412
>> mesh plink:ESTAB
>> mesh local PS mode:ACTIVE
>> mesh peer PS mode:UNKNOWN
>> mesh non-peer PS mode:ACTIVE
>> authorized:yes
>> authenticated:yes
>> associated:yes
>> preamble:long
>> WMM/WME:yes
>> MFP:no
>> TDLS peer:no
>> DTIM period:2
>> beacon interval:1000
>> short slot time:yes
>> connected time:33 seconds
>>
>> Firmware crash on Station A (10.230.5.41)
>> [ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>> 42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
>> [ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>> [ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>> tracing 0 dfs 1 testmode 1
>> [ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>> [ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>> crc32 bebc7c08
>> [ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>> [ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
>> [ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>> 0x009A4577 0x00955B31
>> [ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>> 0x00000001 0x00435354
>> [ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>> 0x00407120 0x004436CC
>> [ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>> 0x009A43A0 0x009A43DE
>> [ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>> 0x00000000 0x00000000
>> [ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>> 0x0040AC60 0x0040AC09
>> [ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>> 0x00955A00 0xC09A4577
>> [ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>> 0x0044110C 0x00442074
>> [ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>> 0x0044110C 0x00407120
>> [ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>> 0x0040AC10 0x00001580
>> [ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>> 0x009C643C 0x004436CC
>> [ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>> 0x004208FC 0x00439E4C
>> [ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>> 0x004208FC 0x00425758
>> [ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>> 0x00424FE8 0x00000002
>> [ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>> 0x00000004 0x004039D0
>> [ 1041.267025] ieee80211 phy0: Hardware restart was requested
>> [ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
>> monitor vdev 1 stop: -143
>> [ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>>
>> ​
>> Thanks in advance for your time and help,
>>
>> ~Benjamin
>>