Return-path: <linux-wireless-owner@vger.kernel.org>
Received: from mail-io0-f179.google.com ([209.85.223.179]:36605 "EHLO
        mail-io0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751645AbcLESeh (ORCPT
        <rfc822;linux-wireless@vger.kernel.org>);
        Mon, 5 Dec 2016 13:34:37 -0500
Received: by mail-io0-f179.google.com with SMTP id m5so473162175ioe.3
        for <linux-wireless@vger.kernel.org>; Mon, 05 Dec 2016 10:34:16 -0800 (PST)
Subject: Re: ath10k firmware crashes in mesh mode on QCA9880
To: Mohammed Shafi Shajakhan <mohammed@codeaurora.org>
References: <583DD564.8040704@cococorp.com>
 <20161201070908.GB11953@atheros-ThinkPad-T61> <58421FA2.2020004@cococorp.com>
 <20161203124358.GB31861@atheros-ThinkPad-T61>
 <20161203124600.GC31861@atheros-ThinkPad-T61>
Cc: linux-wireless@vger.kernel.org, ath10k@lists.infradead.org,
        lede-dev@lists.infradead.org, agreen@cococorp.com
From: Benjamin Morgan <bmorgan@cococorp.com>
Message-ID: <5845B325.6060002@cococorp.com> (sfid-20161205_193522_591448_BA9E9C83)
Date: Mon, 5 Dec 2016 10:34:13 -0800
MIME-Version: 1.0
In-Reply-To: <20161203124600.GC31861@atheros-ThinkPad-T61>
Content-Type: text/plain; charset=utf-8; format=flowed
Sender: linux-wireless-owner@vger.kernel.org
List-ID: <linux-wireless.vger.kernel.org>

Applied the patch and tried with 10.2.4.70.54 firmware and it still crashes:

[  142.438377] ath10k_pci 0000:01:00.0: firmware crashed! (uuid 
a5499582-e220-46d2-9359-0b44219f69ea)
[  142.447512] ath10k_pci 0000:01:00.0: qca988x hw2.0 target 0x4100016c 
chip_id 0x043202ff sub 0000:0000
[  142.456879] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 
tracing 0 dfs 1 testmode 1
[  142.469916] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api 5 
features no-p2p,raw-mode,mfp crc32 9d340dd9
[  142.480295] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A 
crc32 bebc7c08
[  142.487717] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 
cal file max-sta 128 raw 0 hwcrypto 1
[  142.499361] ath10k_pci 0000:01:00.0: firmware register dump:
[  142.505124] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3 
0x009A4577 0x00955B31
[  142.513157] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130 
0x00000002 0x00439E98
[  142.521203] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074 
0x00407120 0x004436CC
[  142.529246] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000 
0x009A3518 0x009A3526
[  142.537285] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A3EA6 
0x00000000 0x00000000
[  142.545324] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4 
0x0040AC60 0x0040AC09
[  142.553356] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24 
0x00400000 0xC09A4577
[  142.561400] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84 
0x0044110C 0x00442074
[  142.569444] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4 
0x0044110C 0x00407120
[  142.577483] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4 
0x0040AC14 0x00001580
[  142.585522] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04 
0x009C643C 0x004436CC
[  142.593554] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64 
0x004208FC 0x00439E4C
[  142.601600] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84 
0x004208FC 0x0042638C
[  142.609642] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4 
0x00424FE8 0x00000002
[  142.617681] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14 
0x00000004 0x004039D0
[  142.727220] ieee80211 phy0: Hardware restart was requested
[  142.732850] ath10k_pci 0000:01:00.0: failed to synchronize monitor 
vdev 1 stop: -143
[  142.740739] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143


~Benjamin

On 12/03/2016 04:46 AM, Mohammed Shafi Shajakhan wrote:
> https://patchwork.kernel.org/patch/9437519/
> (sorry missed this in the previous thread)
>
> On Sat, Dec 03, 2016 at 06:13:58PM +0530, Mohammed Shafi Shajakhan wrote:
>> Hi Benjamin,
>>
>> On Fri, Dec 02, 2016 at 05:28:02PM -0800, Benjamin Morgan wrote:
>>> Just tried 10.2.4.70.58 firmware that you linked to and it still crashes:
>>>
>>> [  131.568989] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>>> 1838347e-9380-4a26-ac9d-2963ee95968b)
>>> [  131.578124] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>> [  131.587491] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>> tracing 0 dfs 1 testmode 1
>>> [  131.600521] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.58
>>> api 5 features no-p2p,raw-mode,mfp crc32 e1af076f
>>> [  131.610899] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>> crc32 bebc7c08
>>> [  131.618325] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>> [  131.629965] ath10k_pci 0000:01:00.0: firmware register dump:
>>> [  131.635728] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>>> 0x009A45AF 0x00955B31
>>> [  131.643761] ath10k_pci 0000:01:00.0: [04]: 0x009A45AF 0x00060130
>>> 0x00000002 0x00439E98
>>> [  131.651806] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>>> 0x00407120 0x004436CC
>>> [  131.659852] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>>> 0x009A3550 0x009A355E
>>> [  131.667892] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x009A31D6
>>> 0x00000000 0x00000000
>>> [  131.675936] ath10k_pci 0000:01:00.0: [20]: 0x409A45AF 0x0040AAC4
>>> 0x0040AC60 0x0040AC09
>>> [  131.683968] ath10k_pci 0000:01:00.0: [24]: 0x809A44F2 0x0040AB24
>>> 0x00400000 0xC09A45AF
>>> [  131.692013] ath10k_pci 0000:01:00.0: [28]: 0x809A3A16 0x0040AB84
>>> 0x0044110C 0x00442074
>>> [  131.700056] ath10k_pci 0000:01:00.0: [32]: 0x809A601A 0x0040ABB4
>>> 0x0044110C 0x00407120
>>> [  131.708100] ath10k_pci 0000:01:00.0: [36]: 0x809A2EA4 0x0040ABF4
>>> 0x0040AC14 0x00001580
>>> [  131.716143] ath10k_pci 0000:01:00.0: [40]: 0x80990F63 0x0040AD04
>>> 0x009C6458 0x004436CC
>>> [  131.724175] ath10k_pci 0000:01:00.0: [44]: 0x80998520 0x0040AD64
>>> 0x004208FC 0x00439E4C
>>> [  131.732220] ath10k_pci 0000:01:00.0: [48]: 0x8099AEA5 0x0040AD84
>>> 0x004208FC 0x00425874
>>> [  131.740263] ath10k_pci 0000:01:00.0: [52]: 0x809BFC39 0x0040AEE4
>>> 0x00424FE8 0x00000002
>>> [  131.748306] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>>> 0x00000004 0x004039D0
>>> [  131.857076] ieee80211 phy0: Hardware restart was requested
>>> [  131.862705] ath10k_pci 0000:01:00.0: failed to synchronize
>>> monitor vdev 1 stop: -143
>>> [  131.870594] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>> [shafi] request to try with the latest backports and ath.git
>> Also worth trying this fix in ath10k (once the next firmware version)
>> is available. I will keep you posted. thank you !
>>
>>> ~Benjamin
>>>
>>> On 11/30/2016 11:09 PM, Mohammed Shafi Shajakhan wrote:
>>>> Hi,
>>>>
>>>> On Tue, Nov 29, 2016 at 11:22:12AM -0800, Benjamin Morgan wrote:
>>>>> When we try to transmit traffic (ping) between two meshed ath10k
>>>>> devices running latest lede we keep experiencing ath10k firmware
>>>>> crashes. This seems to only happen when running in 802.11n/ac mode
>>>>> but not in 802.11a/g mode. Also, from the station dumps it appears
>>>>> that management traffic is flowing between the devices, however when
>>>>> we try to send unicast data traffic the firmware crashes
>>>>> immediately.
>>>> [shafi] Did you get a chance to try with the below firmware as well
>>>> https://github.com/kvalo/ath10k-firmware/blob/master/QCA988X/hw2.0/10.2.4.70/firmware-5.bin_10.2.4.70.58
>>>>
>>>>> Platform: Archer C7 AC1750 v2
>>>>> Software Image: LEDE (HEAD, r2299) Commit: https://github.com/lede-project/source/commit/d596c21ebd5a3e6ce933eff3e51989031e4b1d58
>>>>>
>>>>> Crypto: wpa_supplicant
>>>>> wpa_supplicant-wlan0.conf
>>>>> network={
>>>>> ssid="bmorgan_lede_mesh"
>>>>> key_mgmt=SAE
>>>>> mode=5
>>>>> frequency=5180
>>>>> psk="meshpassword"
>>>>> }
>>>>>
>>>>> Backports Verstion:
>>>>> [    9.818007] Loading modules backported from Linux version
>>>>> wt-2016-10-03-1-g6fcb1a6
>>>>> [    9.825736] Backport generated by backports.git
>>>>> backports-20160324-9-g0e38f5c
>>>>>
>>>>> ​​Ath10k Initialization on Station A (dmesg)
>>>>> [    9.896715] PCI: Enabling device 0000:01:00.0 (0000 -> 0002)
>>>>> [    9.902622] ath10k_pci 0000:01:00.0: pci irq legacy oper_irq_mode
>>>>> 1 irq_mode 0 reset_mode 0
>>>>> [   10.123734] ath10k_pci 0000:01:00.0: Direct firmware load for
>>>>> ath10k/pre-cal-pci-0000:01:00.0.bin failed with error -2
>>>>> [   10.134620] ath10k_pci 0000:01:00.0: Falling back to user helper
>>>>> [   10.287680] firmware ath10k!pre-cal-pci-0000:01:00.0.bin:
>>>>> firmware_loading_store: map pages failed
>>>>> [   10.622789] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>>>> [   10.632184] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>>>> tracing 0 dfs 1 testmode 1
>>>>> [   10.645231] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>>>>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>>>>> [   10.655660] ath10k_pci 0000:01:00.0: Direct firmware load for
>>>>> ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
>>>>> [   10.666264] ath10k_pci 0000:01:00.0: Falling back to user helper
>>>>> [   10.747925] firmware ath10k!QCA988X!hw2.0!board-2.bin:
>>>>> firmware_loading_store: map pages failed
>>>>> [   11.011123] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>>>> crc32 bebc7c08
>>>>> [   12.155224] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>>>>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>>>>
>>>>> Station A (wlan0):
>>>>> 18:A6:F7:23:6E:66
>>>>> 10.230.5.41
>>>>>
>>>>> Station B (wlan0):
>>>>> 18:a6:f7:26:0f:21
>>>>> 10.230.5.42
>>>>>
>>>>> Station Dump on Station A before ping:
>>>>> Station 18:a6:f7:26:0f:21 (on wlan0)
>>>>> inactive time:340 ms
>>>>> rx bytes:2472
>>>>> rx packets:28
>>>>> tx bytes:1204
>>>>> tx packets:9
>>>>> tx retries:0
>>>>> tx failed:0
>>>>> rx drop misc:1
>>>>> signal: -14 dBm
>>>>> signal avg:-14 dBm
>>>>> Toffset:18142530 us
>>>>> tx bitrate:6.0 MBit/s
>>>>> rx bitrate:6.0 MBit/s
>>>>> rx duration:1524 us
>>>>> mesh llid:0
>>>>> mesh plid:0
>>>>> mesh plink:ESTAB
>>>>> mesh local PS mode:ACTIVE
>>>>> mesh peer PS mode:UNKNOWN
>>>>> mesh non-peer PS mode:ACTIVE
>>>>> authorized:yes
>>>>> authenticated:yes
>>>>> associated:yes
>>>>> preamble:long
>>>>> WMM/WME:yes
>>>>> MFP:yes
>>>>> TDLS peer:no
>>>>> DTIM period:2
>>>>> beacon interval:1000
>>>>> connected time:10 seconds
>>>>>
>>>>> ​Crash Log on Station B (10.230.5.42)
>>>>> [245.483888] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>>>>> 2bab5ee9-08ff-4a17-95b1-636d212acebc)
>>>>> [245.493020] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>>>> [245.502384] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>>>> tracing 0 dfs 1 testmode 1
>>>>> [245.515436] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54 api
>>>>> 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>>>>> [245.525812] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>>>> crc32 bebc7c08
>>>>> [245.533232] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
>>>>> cal file max-sta 128 raw 0 hwcrypto 1
>>>>> [245.544876] ath10k_pci 0000:01:00.0: firmware register dump:
>>>>> [245.550633] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>>>>> 0x009A4577 0x00955B31
>>>>> [245.558676] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>>>>> 0x00000002 0x00439E98
>>>>> [245.566715] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>>>>> 0x00407120 0x004436CC
>>>>> [245.574749] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>>>>> 0x009A3518 0x009A3526
>>>>> [245.582793] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>>>>> 0x00000000 0x00000000
>>>>> [245.590836] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>>>>> 0x0040AC60 0x0040AC09
>>>>> [245.598882] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>>>>> 0x00400000 0xC09A4577
>>>>> [245.606923] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>>>>> 0x0044110C 0x00442074
>>>>> [245.614955] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>>>>> 0x0044110C 0x00407120
>>>>> [245.623000] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>>>>> 0x0040AC14 0x00001580
>>>>> [245.631043] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>>>>> 0x009C643C 0x004436CC
>>>>> [245.639086] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>>>>> 0x004208FC 0x00439E4C
>>>>> [245.647129] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>>>>> 0x004208FC 0x00425404
>>>>> [245.655170] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>>>>> 0x00424FE8 0x00000002
>>>>> [245.663198] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>>>>> 0x00000004 0x004039D0
>>>>> [245.767023] ieee80211 phy0: Hardware restart was requested
>>>>> [245.772655] ath10k_pci 0000:01:00.0: failed to synchronize monitor
>>>>> vdev 1 stop: -143
>>>>> [245.780542] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>>>>> ​
>>>>> With wpa_supplicant turned off:
>>>>>
>>>>> Station Dump on Station A before ping:
>>>>> Station 18:a6:f7:26:0f:21 (on wlan0)
>>>>> inactive time:60 ms
>>>>> rx bytes:3874
>>>>> rx packets:67
>>>>> tx bytes:707
>>>>> tx packets:7
>>>>> tx retries:0
>>>>> tx failed:1
>>>>> rx drop misc:3
>>>>> signal: -14 dBm
>>>>> signal avg:-14 dBm
>>>>> Toffset:18446744073684250999 us
>>>>> tx bitrate:6.0 MBit/s
>>>>> rx bitrate:6.0 MBit/s
>>>>> rx duration:1312 us
>>>>> mesh llid:24953
>>>>> mesh plid:59412
>>>>> mesh plink:ESTAB
>>>>> mesh local PS mode:ACTIVE
>>>>> mesh peer PS mode:UNKNOWN
>>>>> mesh non-peer PS mode:ACTIVE
>>>>> authorized:yes
>>>>> authenticated:yes
>>>>> associated:yes
>>>>> preamble:long
>>>>> WMM/WME:yes
>>>>> MFP:no
>>>>> TDLS peer:no
>>>>> DTIM period:2
>>>>> beacon interval:1000
>>>>> short slot time:yes
>>>>> connected time:33 seconds
>>>>>
>>>>> Firmware crash on Station A (10.230.5.41)
>>>>> [ 1040.984599] ath10k_pci 0000:01:00.0: firmware crashed! (uuid
>>>>> 42d4f6dc-0e89-4505-aefb-58cbb70d0dce)
>>>>> [ 1040.993732] ath10k_pci 0000:01:00.0: qca988x hw2.0 target
>>>>> 0x4100016c chip_id 0x043202ff sub 0000:0000
>>>>> [ 1041.003100] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1
>>>>> tracing 0 dfs 1 testmode 1
>>>>> [ 1041.016144] ath10k_pci 0000:01:00.0: firmware ver 10.2.4.70.54
>>>>> api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
>>>>> [ 1041.026523] ath10k_pci 0000:01:00.0: board_file api 1 bmi_id N/A
>>>>> crc32 bebc7c08
>>>>> [ 1041.033940] ath10k_pci 0000:01:00.0: htt-ver 2.1 wmi-op 5 htt-op
>>>>> 2 cal file max-sta 128 raw 0 hwcrypto 1
>>>>> [ 1041.045593] ath10k_pci 0000:01:00.0: firmware register dump:
>>>>> [ 1041.051336] ath10k_pci 0000:01:00.0: [00]: 0x4100016C 0x000015B3
>>>>> 0x009A4577 0x00955B31
>>>>> [ 1041.059383] ath10k_pci 0000:01:00.0: [04]: 0x009A4577 0x00060130
>>>>> 0x00000001 0x00435354
>>>>> [ 1041.067424] ath10k_pci 0000:01:00.0: [08]: 0x0044110C 0x00442074
>>>>> 0x00407120 0x004436CC
>>>>> [ 1041.075463] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0x00000000
>>>>> 0x009A43A0 0x009A43DE
>>>>> [ 1041.083490] ath10k_pci 0000:01:00.0: [16]: 0x00958080 0x0094085D
>>>>> 0x00000000 0x00000000
>>>>> [ 1041.091535] ath10k_pci 0000:01:00.0: [20]: 0x409A4577 0x0040AAC4
>>>>> 0x0040AC60 0x0040AC09
>>>>> [ 1041.099579] ath10k_pci 0000:01:00.0: [24]: 0x809A44BA 0x0040AB24
>>>>> 0x00955A00 0xC09A4577
>>>>> [ 1041.107622] ath10k_pci 0000:01:00.0: [28]: 0x809A39DE 0x0040AB84
>>>>> 0x0044110C 0x00442074
>>>>> [ 1041.115661] ath10k_pci 0000:01:00.0: [32]: 0x809A5FE2 0x0040ABB4
>>>>> 0x0044110C 0x00407120
>>>>> [ 1041.123689] ath10k_pci 0000:01:00.0: [36]: 0x809A2E6C 0x0040ABF4
>>>>> 0x0040AC10 0x00001580
>>>>> [ 1041.131733] ath10k_pci 0000:01:00.0: [40]: 0x80990F6F 0x0040AD04
>>>>> 0x009C643C 0x004436CC
>>>>> [ 1041.139777] ath10k_pci 0000:01:00.0: [44]: 0x80998510 0x0040AD64
>>>>> 0x004208FC 0x00439E4C
>>>>> [ 1041.147820] ath10k_pci 0000:01:00.0: [48]: 0x8099AE95 0x0040AD84
>>>>> 0x004208FC 0x00425758
>>>>> [ 1041.155860] ath10k_pci 0000:01:00.0: [52]: 0x809BFC55 0x0040AEE4
>>>>> 0x00424FE8 0x00000002
>>>>> [ 1041.163888] ath10k_pci 0000:01:00.0: [56]: 0x80940F18 0x0040AF14
>>>>> 0x00000004 0x004039D0
>>>>> [ 1041.267025] ieee80211 phy0: Hardware restart was requested
>>>>> [ 1041.272656] ath10k_pci 0000:01:00.0: failed to synchronize
>>>>> monitor vdev 1 stop: -143
>>>>> [ 1041.280545] ath10k_pci 0000:01:00.0: failed to stop monitor vdev: -143
>>>>>
>>>>> ​
>>>>> Thanks in advance for your time and help,
>>>>>
>>>>> ~Benjamin
>>>>>